2009-02-05 12:37 PM
I have my VMware ESX datastore on a NetApp LUN. The volume associated with the LUN shows 26 GB of free space. However when I display the datastore freespace in VMware it shows 98 GB free. Can you help me understand the discrepancy.
249G 154G 95G 61% /vmfs/volumes/Datastore1
Data ONTAP (netapp.unitedtrust.com)
netapp> Thu Feb 5 14:35:30 CST [telnet_0:info]: root logged in from host: ddock
Filesystem kbytes used avail capacity Mounted on
/vol/vol0/ 140497404 268472 140228932 0% /vol/vol0/
/vol/vol0/.snapshot 35124348 8372 35115976 0% /vol/vol0/.snapsh
/vol/file_shares/ 32715572 29697628 3017944 91% /vol/file_shares/
/vol/file_shares/.snapshot 8178892 438156 7740736 5% /vol/file_
/vol/vol2/ 398458880 371353684 27105196 93% /vol/vol2/
snap reserve 0 76 0 ---% /vol/vol2/..
/vol/vmwaredatastore1/ 262144000 237195360 24948640 90% /vol/vmwaredat
/vol/vmwaredatastore1/.snapshot 65536000 12571328 52964672 19% /vol/
/vol/vmwarepagepool1/ 31457280 4442852 27014428 14% /vol/vmwarepage
snap reserve 0 0 0 ---% /vol/vmwarepagepo
/vol/zantaz_archive/ 47185920 37220924 9964996 79% /vol/zantaz_arch
/vol/zantaz_archive/.snapshot 5242880 3086584 2156296 59% /vol/za
/vol/vol1/ 10485760 357876 10127884 3% /vol/vol1/
snap reserve 2621440 8052 2613388 0% /vol/vol1/..
2009-02-05 02:09 PM
I saw a similiar issue that was caused by the fact their was a previous datastore on a drive with the same device name. I had previously deleted a hardware RAID without deleting the datastore first. The datastore disappeared from the GUI tools and everything appeared to be fine.
Later, I re-created another RAID with the same device name (it was on the same location on the SCSI bus). It appeared with the size of the previously deleted RAID, and started causing extreme stability issues with the ESX Server.
The solution was to use fdisk from the command line to delete the datastore, and then recreate it with the proper size.
2009-02-05 07:29 PM
I just re-read your post, and realized it actually looks like your LUN is smaller than volume. VMWare is reporting the space on the LUN, not on the volume -- there is a difference.
2009-02-06 10:13 AM
Ahh ... I found your descrepency ...I'm still not sure why the numbers don't add up though.
According to the NetApp doc it seems you have a 250 GB volume with a 226 GB VMFS Partition on it. From the NetApp perspective, all 226 GB used by the VMFS Partition (your datastore) are 'in use' ... picture the datastore being a large file on the NetApp -- the NetApp doesn't have any insight into the structure of this file, so it reports it all as being 'used'. From the NetApp perspective, this is true ... the VMFS partition is consuming 226 GB of space on disk.
VMWare, on the other, is reporting 98 GB free in the datastore. That means, out of the 226 GB used for the datastore, 98 GB are free, or avaialble for .vmdk files.
So, your volume size is 262144000.
237195360 out of that 262144000 is 'in use' as a VMFS partition
98GB out of the 262144000 VMFS partition is 'free'.
You might want to try 'df -h' on the NetApp ... it's easier to read.
Attached is a diagram illustrating the relationships between different levels of storage and how they nest together.
2009-02-06 11:50 AM
To make it more confusing, if you have the default Fractional Reservation of 100%, this will also be factored into the volume size, but not clearly shown for the "df" output.
A good way of seeing how much space the filer thinks a LUN is using, (when fractional reservation is still set to 100% and you have snapshots), try using "df -r". This will show the used space within the LUN, and as such, how much space the filer is reserving for overwrites (give or take).
But there is definitely a clear difference between the NetApp volume, and the space within the LUN that is presented to ESX.
2009-02-06 12:25 PM
So bottom line I should just go by what VMware is telling me? What (if anything) happens when/if the NetApp volume free % goes to zero? Right now it is decreasing by almost 1 G per day. It started doing that about 10 days ago and shows no signs of slowing down. Not sure what might be causing that.
2009-02-06 12:29 PM
Well, you really need to monitor both to be honest! If the VMFS datastore runs out of space your running Virtual Machines may be effected. If the filer volume runs out of space, you won't be able to take any more snapshots and it could effect the volumes performance. Best keep the volume below 90% for optimum performance.
If the volume is slowly growing, my guess is that you took a snapshot of the volume about 10 days ago. The rate of change from this snapshot and the active filesystem will grow at a steady rate. I can guess that the rate of change on your volume is about 1g per day
Delete the snapshot (if you don't need it), and this space will get freed up. Check the snapshot schedules if you do need snapshots, you may not need 10 days worth.
2009-02-06 12:30 PM
You have a NetApp Snapshot, and your data is changing at a rate of about 1 G per day -- the space you're 'losing' is the space being consumed by the SnapShot ar vs it saves off changd blocks. I would recommend deleting the SnapShot ... I believe when you reach 0% on your Volume you will recieve an IO error.
use the 'snap' command
'snap list' will list your snapshots.
'snap delete <vol-name> <snapshot-name>' will delete the snapshot in question.