2014-01-07 11:50 AM
Our FAS2240-2 has a volume that is growing for unexplained reasons. How can I determine what's using all the space and stop it without deleting the volume?
A week ago four VMware vm's that used the the volume locked up when it ran out of space. One vm was deleted to free up working space. Gradually over the next few days it kept growing, even though the vm's should not have been increasing in size. All vm's have since been moved to other volumes and volume snaps were deleted. While browsing the datastore from Vsphere the volume shows to be empty but OnCommand reports 97% used; 54GB available of 1.8TB.
2014-01-07 01:43 PM
When were the datastores deleted? When were the volume snaps deleted? The actual space reclamation for these takes time; until it is done, ONTAP will still show data in the volume.
2014-01-07 01:56 PM
The problematic volume has not been deleted because I'd like to know why it's behaving this way before deleting the evidence. The volume snaps were deleted two days ago. The volume snapshot detail in OnCommand reports about 1MB Cululative Total Size for each of the six snaps, but the volume reports 28GB available of 2TB.
2014-01-07 03:09 PM
You said the datastore on vmware shows them empty - are these NFS datastores or have you carved out luns in the volume and presented those as datastores? If you're using luns, and they still exist, that is what is taking the space.
2014-01-07 03:22 PM
Ah well. That was the low hanging fruit...
I assume there is free space in the aggregate? How many other volumes in the aggr? Have you mounted the volume up to a host and verified there's nothing there? Can you post:
snap list <vol>
aggr show_space <aggr>
2014-01-08 11:21 AM
The aggr is 97% used, but three of six volumes have more than 50% space available.
Total space WAFL reserve Snap reserve Usable space BSR NVLOG A-SIS Smtape
5160771072KB 516077104KB 0KB 4644693968KB 0KB 99414384KB 0KB
Space allocated to volumes in the aggregate
Volume Allocated Used Guarantee
vol0 191462080KB 4052144KB volume
vol_*** 2159414112KB 437429852KB volume
vol_2*** 26627380KB 14952428KB none
vol_3*** 527200708KB 126126452KB volume
VOL_4*** 1476161988KB 907011340KB volume
vol_5*** 3278448KB 412380KB none
Aggregate Allocated Used Avail
Total space 4384144716KB 1489984596KB 111571540KB
Snap reserve 0KB 49243032KB 0KB
WAFL reserve 516077104KB 59087464KB 456989640KB
snap list vol2
%/used %/total date name
---------- ---------- ------------ --------
31% (31%) 0% ( 0%) Jan 08 08:00 hourly.0
31% ( 0%) 0% ( 0%) Jan 08 00:00 nightly.0
44% (25%) 0% ( 0%) Jan 07 20:00 hourly.1
44% ( 0%) 0% ( 0%) Jan 07 16:00 hourly.2
44% ( 0%) 0% ( 0%) Jan 07 12:00 hourly.3
44% ( 0%) 0% ( 0%) Jan 07 08:00 hourly.4
44% ( 0%) 0% ( 0%) Jan 07 00:00 nightly.1
44% ( 0%) 0% ( 0%) Jan 06 20:00 hourly.5
Thanks - Dan
2014-01-08 12:34 PM
Is the volume in question vol2? vol2 and vol5 have no space guarantee - so they will show as free space the lesser of the aggregate free space, and the difference between the vol size and vol used. Even if the other volumes are at 50% capacity, if they are volume guaranteed the entire size of the volume is considered "used" in the aggregate.
Please post a df of the volume in question, df -A of aggr0.
2014-01-08 03:21 PM
Aggregate kbytes used avail capacity
aggr0 4644693968 4564316928 80377040 98%
aggr0/.snapshot 0 80432548 0 ---%
Both vol2 and vol5 were thin provisioned. Is that why the Guarantee is none instead of volume? Can it be changed from command line?
2014-01-08 03:35 PM
Thin Provisioned == space guarantee of none
That's the problem. The aggregate only has 80G available, and because these volumes have no space guarantee, they can only utilize whatever space is left in the aggregate. Even though they are 1.8TB (virtual), there is only 80G left of physical space, so they report as much.
You can change the space guarantee (vol options vol2 guarantee volume) - but it won't work in your case, because the containing aggregate does not have enough space to cover the request.
You can reduce the size of the other volumes - that will free space in the aggregate. Then you can size the thin volumes appropriately and make them thick. All this of course depends on your needs and projected usage - I'm just saying that you CAN do this....
Does that help?