Thin Provisioning LUNs

[TMADOCTHOMAS]

Hello,

In an effort to provide as much aggregate free space as possible, I am considering thin provisioning all of our LUNs. Currently we are on an All Flash system and all our our volumes are already thin-provisioned. I find myself getting confused regarding the various combinations of options and wondered if someone can provide clarity. Specifically:

I have a handful of LUNs (29) with space-reserve already disabled but "space-reserve-honored" equals false. Does this mean it ignores the fact that it is thin provisioned? If so, how to override this? <Note: these LUNs are the few I've created via SnapCenter vs. the older SnapDrive tool>.

The vast majority of our LUNs have space-reserve enabled with "space-reserve-honored" also set to false. I had thought at one point this meant the LUNs were just picking up the space-reserve setting from the volume, which would mean all LUNs are already thin provisioned, but I'm not sure that that's the case. Can someone assist?

[SpindleNinja]

"space-reserve-honored" will go to true if the volume is thick (or -space-guarantee volume).    Meaning it will honor the space reserve on the lun (if set true).  

If you're thin volume / thin lun,  it should be false.   

Here's an example from my lab: 

Mother::*> lun show -fields space-reserve-honored,space-reserve
vserver       path                                                   space-reserve space-reserve-honored
------------- ------------------------------------------------------ ------------- ---------------------
Storage_iSCSI /vol/ntap_datastore_vmfs_01/ntap_datastore_vmfs_01.lun disabled      false

Mother::*> lun modify -vserver Storage_iSCSI -path /vol/ntap_datastore_vmfs_01/ntap_datastore_vmfs_01.lun -space-reserve true

Mother::*> lun show -fields space-reserve-honored,space-reserve
vserver       path                                                   space-reserve space-reserve-honored
------------- ------------------------------------------------------ ------------- ---------------------
Storage_iSCSI /vol/ntap_datastore_vmfs_01/ntap_datastore_vmfs_01.lun enabled       false

Mother::*> lun modify -vserver Storage_iSCSI -path /vol/ntap_datastore_vmfs_01/ntap_datastore_vmfs_01.lun -space-reserve false

Mother::*> lun show -fields space-reserve-honored,space-reserve
vserver       path                                                   space-reserve space-reserve-honored
------------- ------------------------------------------------------ ------------- ---------------------
Storage_iSCSI /vol/ntap_datastore_vmfs_01/ntap_datastore_vmfs_01.lun disabled      false

Mother::*> vol modify -vserver Storage_iSCSI -volume ntap_datastore_vmfs_01 -space-guarantee volume
Volume modify successful on volume ntap_datastore_vmfs_01 of Vserver Storage_iSCSI.

Mother::*> lun show -fields space-reserve-honored,space-reserve
vserver       path                                                   space-reserve space-reserve-honored
------------- ------------------------------------------------------ ------------- ---------------------
Storage_iSCSI /vol/ntap_datastore_vmfs_01/ntap_datastore_vmfs_01.lun disabled      true

[TMADOCTHOMAS]

Thank you @SpindleNinja . All of our volumes are thin, so I assume based on what you're saying that every volume should read "false" for "space-reserve-honored". If I'm understanding right, does that mean it essentially doesn't matter what you set the LUN to if the volume is thin? Will the LUN always be thin provisioned if the volume is?

If so, in that case, I'm confused about the volumes I recently created in SnapCenter <all previous LUNs were created in SnapDrive>. I checked the box "Use thin provisioning for the volume hosting this LUN" (even though the volume was already thin provisioned). The result is different - System Manager shows 0 used space on the volume until content is added. For all LUNs created by SnapDrive, it shows the LUN size as being used space on the volume, regardless of usage on the LUN.

For example: if I create a 200 GB thin volume and a 100 GB LUN in SnapDrive, the volume shows 100 GB utilized. If I create the LUN in SnapCenter on the same volume, it shows 0 GB utilized.

Any insights or help will be appreciated!

[SpindleNinja]

All of our volumes are thin, so I assume based on what you're saying that every volume should read "false" for "space-reserve-honored". - Yes, every thin volume. 

If I'm understanding right, does that mean it essentially doesn't matter what you set the LUN to if the volume is thin? Will the LUN always be thin provisioned if the volume is?

-Technically, no.  but I would leave Thin/Thin esp because you're on AFF.  Check out this docs page to see what works.  

https://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.dot-cm-sanag%2FGUID-2F5C9474-FFE9-4E59-84DB-1B9D6D134688.html 

The result is different - System Manager shows 0 used space on the volume until content is added. - how it should be if space-guarantee is disabled.  

For all LUNs created by SnapDrive, it shows the LUN size as being used space on the volume, regardless of usage on the LUN.

For example: if I create a 200 GB thin volume and a 100 GB LUN in SnapDrive, the volume shows 100 GB utilized. If I create the LUN in SnapCenter on the same volume, it shows 0 GB utilized.

-I would verify that all luns are thin (space-guarantee disabled) sounds like SnapDrive is creating thick luns on top of thin volumes maybe?   It's been a long while since I used SnapDrive though.. can't recall what it dose by default off the top of my head.   

[TMADOCTHOMAS]

Thanks @SpindleNinja! That brings clarity to the issue. All the LUNs I created in SnapCenter have space-reserve set to disabled, and all the ones I created in SnapDrive have it set to enabled. So I have thick LUNs created via SnapDrive and thin LUNs created via SnapCenter.

Last question to you, @aborzenkov , and anyone else who would like to comment: I would like to change all of the LUNs created via SnapDrive to space-reserve = disabled for space saving reasons. Any downsides to doing so? The only ones I've seen involve potential performance issues (which I assume would be mitigated by the fact of this being an AFF cluster) and not having enough space in the volume (easily mitigated by leaving volume spaces the same). Any thoughts or suggestions?

[paul_stejskal]

It really depends on the best practices on the types of LUNs. As long as you have sufficient space, you should be fine. Performance will likely not be an issue, but it would be good to monitor because AFF just means SSDs don't hold you up, but CPU does.

[TMADOCTHOMAS]

Good point @paul_stejskal , thank you! CPU is pretty low on all four nodes right now but we would definitely have to watch carefully.

[aborzenkov]

---

@TMADOCTHOMAS wrote:

I would like to change all of the LUNs created via SnapDrive to space-reserve = disabled for space saving reasons. Any downsides to doing so?

---

This does not save any (physical) space. It simply allows you to over-commit logical space in volume by creating LUNs with total size exceeding volume size. If you over-commit you need to monitor for free space both on logical (volume) and physical (aggregate) level. If you do not over-commit, it is entirely equivalent to having space-reserved LUN.

[TMADOCTHOMAS]

Thank you @aborzenkov. I'm starting to think the confusion is around the way System Manager reports things (we are on OnTAP 9.3P15). For example:

I have a 1TB LUN, created in SnapDrive, with 323 GB of used space. In System Manager, I show the volume with 909.23 GB of data space used (not snapshots). I also show 127.14 GB of dedupe / compression savings, which added together roughly equals 1TB, the full size of the LUN, not 323 GB, the amount that's actually used. When I disable space reservation on the LUN, System Manager reports that the used data space for the volume drops from 909.23 GB to 298.48 GB. HOWEVER, the ~600GB in savings doesn't show up in the aggregate used space total, so it doesn't appear that any actual space has been saved.

It's very confusing the say the least. I would expect that if the LUN is truly taking us ~600GB less space in the volume due to thin provisioning, the fact that it is a thin provisioned volume would also pass those savings on to the aggregate, but it doesn't appear to be the case. Or ... is this one of those cases where it takes time for metadata to "catch up" and I will see the savings if I wait awhile? Would love any insight or thoughts on this!

[TMADOCTHOMAS]

Update: no change after several hours. It appears to be a discrepancy in System Manager reporting, in 9.3 at least, from what I can tell. Would love any insight, especially if I am missing something!

[paul_stejskal]

ONTAP may fill the LUN especially if SCSI_UNMAP (TRIM) is disabled. There are overwrites that happen and it will just recycle the used space. I'm not sure if that's the case here.

Keep in mind, a LUN is just a blob file to ONTAP. When you look at space, you have 3 file systems: 1) Aggr's WAFL, 2) Vol's WAFL, and 3) host side NTFS/VMFS/EXT4/etc. When you compare the host side to WAFL, it will never match. You can attempt to reclaim space, but it may not be needed. It may be done by default.

[TMADOCTHOMAS]

Thanks @paul_stejskal . I had thought that the  -space-allocation enable switch in the lun modify command alleviated the difference between OnTAP and the host. I've been issuing the command on all LUNs created for 2012+ servers for that purpose.

At any rate, that's not my issue. The issue is that when you thin provision a LUN, as best as I can tell, System Manager immediately shows more free space in the volume but not in the aggregate. It's confusing, and I'm not entirely certain which is correct. Keep in mind the volume is already thin provisioned and I am just adding the thin provision setting to the LUN inside the volume. For example, if a 100GB volume has a 50GB LUN with 25GB used space, it initially shows 50GB free space. When I enable thin provisioning on the LUN it becomes 75GB free space, but the aggregate shows the same amount of free space.

[aborzenkov]

See https://community.netapp.com/t5/Data-ONTAP-Discussions/Lun-space-reserve/td-p/140196