Df -g not showing correct free space

[tsysgccadmin05]

A Df -g on a volume not showing the correct free space size

Example : vol size is 8544 - used =5824 GB available should be 2720 GB but it is reported having free only 1464GB.

Can someone advice why and what is happening?

 Filesystem             total       used        avail     capacity Mounted on

/vol/VMWARE01 8544GB 5824GB 1464GB 83% /vol/TUAUXVMWARE_4M_01/
/vol/VMWARE01/.snapshot 0GB 0GB 0GB 0% /vol/TUAUXVMWARE_4M_01/.snapshot

filer05> vol options /vol/VMWARE01/
nosnap=on, nosnapdir=off, minra=off, no_atime_update=off, nvfail=off,
fs_size_fixed=off, guarantee=none, svo_enable=off, svo_checksum=off,
svo_allow_rman=off, svo_reject_errors=off, no_i2p=off,
fractional_reserve=0, extent=off, try_first=volume_grow,
read_realloc=off, snapshot_clone_dependency=off, dlog_hole_reserve=off,

/vol/VMWARE01/VMWARE01 5.7t (6227796951040) (r/w, online, mapped)
Serial#: 805bQ]G856hO
Share: none
Space Reservation: enabled (not honored by containing Aggregate)
Multiprotocol Type: vmware
Maps: dmz_backup=0 ms_backup=0 tuauxvmware=0
Occupied Size: 5.3t (5826621575168)
Creation Time: Tue Mar 24 17:06:57 +08 2015
Cluster Shared Volume Information: 0x0
Read-Only: disabled

[TMACMD]

Likely, thin provisioning here.

look at the aggregate the volume is on and see how free space is there. it may line up with the actual free space

[tsysgccadmin05]

Running aggr show_space yielded such result

Volume	Allocated	Used	Guarantee
vol0	492030416	KB	9376304	KB
VOL745RE_G_TEMPDB	39538116	KB	39307080	KB
vol127	10582132	KB	133364	KB
vol423	10582132	KB	2190020	KB
VOL745RE_M	97176	KB	77248	KB
VOL712RE_E	2181600	KB	2071656	KB
VOL745RE_G_LOG	63711508	KB	63395872	KB
VOL745RE_G	5185208	KB	5123004	KB
vol425	1476161988	KB	1058415740	KB
VOL712RE_Q	1068572	KB	983152	KB
VMWARE01	5813972660	KB	5794181116	KB
VOL745RE_Q	1478064	KB	1419260	KB
vol421	33766144	KB	9510076	KB
VOL745RE_G_DATA	318712328	KB	317375116	KB

Aggregate Allocated Used Avail
Total space 8269068044 KB 7303559008 KB
Snap reserve 516077104 KB 393364 KB
WAFL reserve 1146838016 KB 126746100 KB

/vol/VMWARE01/	8544	GB	5824	GB	1464	GB	83%
/vol/VMWARE01/.snapshot	0	GB	0	GB	0	GB	0%

I) Running aggr show_space show the aggr is having WALF reserve about 1 TB, the Aggregate allocated space is about  8TB

2) Problem is with this volume called VMWARE01 which is a volume presented to ESX hosts used to create VMDKs.

from aggr show_status

VMWARE01

5813972660

KB

5794181116

KB

/vol/VMWARE01/

8544

GB

5824

GB

1464

GB

83%

The volume VMWARE01 has volume guarantee=NONE, the lun inside this VMWARE01 has LUN space_guarantee=enabled

from df -g

3) Despite doing some housekeeping such as reducing the size of a VMDK significantly, the free space from df -g in Netapp storage doesn't show much free space released. We checked from ESX, the datastore has free space about 2 TB

Storage	Capacity	Used	Free
SBA-4X-DS-001	5.66 TB	3.61 TB	2.06TB

4) Question will, turning off space_reservation on LUN free up the space on the volume VMWARE01  ?

lun show -v /vol/VMWARE01/VMWARE01
/vol/VMWARE01/VMWARE01 5.7t (6227796951040) (r/w, online, mapped)
Serial#: 805bQ]G856hO
Share: none
Space Reservation: enabled (not honored by containing Aggregate)
Multiprotocol Type: vmware
Maps: dmz_backup=0 ms_backup=0 tuauxvmware=0
Occupied Size: 5.3t (5826621575168)
Creation Time: Tue Mar 24 17:06:57 +08 2015
Cluster Shared Volume Information: 0x0
Read-Only: disabled

[tsysgccadmin05]

Yes, I think it is tie to the free disk space in the Aggregate.

My next question would be for that /vol/VMWARE01 volume, is it safe to turn off LUN space guarantee ?

According to my VMWARE team colleague, the datastore after some housekeeping should have about 2 TB free space but this is not reflected in the volume. This is because the LUN has a space reservation enabled. Is it safe to turn off the space reservation just like that ? And will turning off the space reservation free up the space to the volume ? 

[GidonMarcus]

Note that you have an output saying "Space Reservation: enabled (not honored by containing Aggregate)"  Hence, changing the space reserve on the LUN will not change your free space situation.

There's also indication for it in the output of "Occupied Size", If you reckon this output is inflated (which I suspect it does as on your opening post you referred to a similar figure), it likely to be the ESXi not doing UNMAP to free-out space.

https://kb.vmware.com/s/article/2014849

If you plan to keep using this datastore for long and don't need the space back on the NetApp AGGR - Just leave it, the ESXi will re-use these blocks when it needs to.

[tsysgccadmin05]

Hello,

I summarize my findings:

1) The Aggregate has 9.5 TB capacity. I did a aggregate show_status : I see the aggregate has 8 TB for data, 500GB for snap reserve, 1 TB for WAFL reserve.

2) Some of the volumes have guarantee reserve, some don't.

3) The volume VMWARE01 is showing that about 8.4 TB is allocated (over-provisioning) and having no guarantee reserve for volume, so a aggr show_space only showing that 5.5 TB is allocated..

4) We tried to do housekeep on the datastores in VMWARE01 volume. With the housekeeping, on ESX level it is showing that it is only utilizing 3 TB of disk space. 

My question will be if we "turn off" the  LUN reservation now, will more free space be reduced ? A lun show -v on the VMWARE01 lun showing that the lun is 5.3 TB used...or even if we turn off  LUN reservation now, it will not help ? By the way, customer is not using snapshots. They had turned off snapshots on the volume long time ago

[GidonMarcus]

Sorry. I don't see new information in your recent reply. Have you looked into the VMWare article I attached?

As I also mentioned, Space Reservation on your LUN seems not to be honored per your output. hence, changing it not going to free up any volume space.