VMware datastore freespace discrepancy

doug_dockter · ‎2009-02-05

I have my VMware ESX datastore on a NetApp LUN. The volume associated with the LUN shows 26 GB of free space. However when I display the datastore freespace in VMware it shows 98 GB free. Can you help me understand the discrepancy.

/vmfs/volumes/471f7838-fcbe3a6f-806b-001c23c68968
249G 154G 95G 61% /vmfs/volumes/Datastore1

Data ONTAP (netapp.unitedtrust.com)
login: root
Password:
netapp> Thu Feb 5 14:35:30 CST [telnet_0:info]: root logged in from host: ddock
ter.unitedtrust.com
df
Filesystem              kbytes       used      avail capacity Mounted on
/vol/vol0/           140497404     268472 140228932       0% /vol/vol0/
/vol/vol0/.snapshot   35124348       8372   35115976       0% /vol/vol0/.snapsh
ot
/vol/file_shares/     32715572   29697628    3017944      91% /vol/file_shares/
/vol/file_shares/.snapshot    8178892     438156    7740736       5% /vol/file_
shares/.snapshot
/vol/vol2/           398458880 371353684   27105196      93% /vol/vol2/
snap reserve                 0         76          0     ---% /vol/vol2/..
/vol/vmwaredatastore1/ 262144000 237195360   24948640      90% /vol/vmwaredat
astore1/
/vol/vmwaredatastore1/.snapshot   65536000   12571328   52964672      19% /vol/
vmwaredatastore1/.snapshot
/vol/vmwarepagepool1/   31457280    4442852   27014428      14% /vol/vmwarepage
pool1/
snap reserve                 0          0          0     ---% /vol/vmwarepagepo
ol1/..
/vol/zantaz_archive/   47185920   37220924    9964996      79% /vol/zantaz_arch
ive/
/vol/zantaz_archive/.snapshot    5242880    3086584    2156296      59% /vol/za
ntaz_archive/.snapshot
/vol/vol1/            10485760     357876   10127884       3% /vol/vol1/
snap reserve           2621440       8052    2613388       0% /vol/vol1/..
netapp>

mshaebanyan · ‎2009-02-05

I saw a similiar issue that was caused by the fact their was a previous datastore on a drive with the same device name. I had previously deleted a hardware RAID without deleting the datastore first. The datastore disappeared from the GUI tools and everything appeared to be fine.

Later, I re-created another RAID with the same device name (it was on the same location on the SCSI bus). It appeared with the size of the previously deleted RAID, and started causing extreme stability issues with the ESX Server.

The solution was to use fdisk from the command line to delete the datastore, and then recreate it with the proper size.

Regards,

Meredith Shaebanyan

mshaebanyan · ‎2009-02-05

Hi

I just re-read your post, and realized it actually looks like your LUN is smaller than volume. VMWare is reporting the space on the LUN, not on the volume -- there is a difference.

Regards,

Meredith

doug_dockter · ‎2009-02-06

My LUN and volume are the same size - 250 GB. I've attached a file that shows this.

mshaebanyan · ‎2009-02-06

Ahh ... I found your descrepency ...I'm still not sure why the numbers don't add up though.
According to the NetApp doc it seems you have a 250 GB volume with a 226 GB VMFS Partition on it. From the NetApp perspective, all 226 GB used by the VMFS Partition (your datastore) are 'in use' ... picture the datastore being a large file on the NetApp -- the NetApp doesn't have any insight into the structure of this file, so it reports it all as being 'used'. From the NetApp perspective, this is true ... the VMFS partition is consuming 226 GB of space on disk.
VMWare, on the other, is reporting 98 GB free in the datastore. That means, out of the 226 GB used for the datastore, 98 GB are free, or avaialble for .vmdk files.

So, your volume size is 262144000.
237195360 out of that 262144000 is 'in use' as a VMFS partition
98GB out of the 262144000 VMFS partition is 'free'.

You might want to try 'df -h' on the NetApp ... it's easier to read.

Attached is a diagram illustrating the relationships between different levels of storage and how they nest together.

Regards,
Meredith

chriskranz · ‎2009-02-06

To make it more confusing, if you have the default Fractional Reservation of 100%, this will also be factored into the volume size, but not clearly shown for the "df" output.

A good way of seeing how much space the filer thinks a LUN is using, (when fractional reservation is still set to 100% and you have snapshots), try using "df -r". This will show the used space within the LUN, and as such, how much space the filer is reserving for overwrites (give or take).

But there is definitely a clear difference between the NetApp volume, and the space within the LUN that is presented to ESX.

doug_dockter · ‎2009-02-06

So bottom line I should just go by what VMware is telling me? What (if anything) happens when/if the NetApp volume free % goes to zero? Right now it is decreasing by almost 1 G per day. It started doing that about 10 days ago and shows no signs of slowing down. Not sure what might be causing that.

chriskranz · ‎2009-02-06

Well, you really need to monitor both to be honest! If the VMFS datastore runs out of space your running Virtual Machines may be effected. If the filer volume runs out of space, you won't be able to take any more snapshots and it could effect the volumes performance. Best keep the volume below 90% for optimum performance.

If the volume is slowly growing, my guess is that you took a snapshot of the volume about 10 days ago. The rate of change from this snapshot and the active filesystem will grow at a steady rate. I can guess that the rate of change on your volume is about 1g per day

Delete the snapshot (if you don't need it), and this space will get freed up. Check the snapshot schedules if you do need snapshots, you may not need 10 days worth.

doug_dockter · ‎2009-02-06

I just had another thought. My VMware datastore is currently set up to snapshot 4 times a day retaining 6 hourly and 2 nightly snapshots. This was set up by the consultant who installed my NetApp and VMware. Does anyone care to comment on the wisdom of taking snapshots of the VMware datastore?

chriskranz · ‎2009-02-06

Snapshotting the VMware datastore is great!

quick and easy full virtual machine recovery
quick and easy cloning of virtual machines
patch testing on clones

amongst others! At the very least, keep a couple of snapshots for just in case!

The retention policies and timing is personal preference really, and depends on what you want to achieve. The longer you keep your backups the more storage you'll use up.

I'd look into SnapManager for VI to snapshot these datastores better though.

mshaebanyan · ‎2009-02-06

You have a NetApp Snapshot, and your data is changing at a rate of about 1 G per day -- the space you're 'losing' is the space being consumed by the SnapShot ar vs it saves off changd blocks. I would recommend deleting the SnapShot ... I believe when you reach 0% on your Volume you will recieve an IO error.

use the 'snap' command

'snap list' will list your snapshots.

'snap delete <vol-name> <snapshot-name>' will delete the snapshot in question.

Regards,

Meredith

doug_dockter · ‎2009-02-06

That you for you very helpful answers. I'll go hunting for snapshots!