ONTAP Discussions

Odd Storage issue (possibly Snapshot related ..?)

OldGreyBeard
17,129 Views

Hi All

 

Had a case open with VEEAM and NetApp neither of which is going anywhere pretty quickly, so will throw it out there and see if anyone has any ideas.

We have a large estate broken down in to areas which mirror the configuration of each other so a nice lot of reference points.

There are 16 Tb NetApp LUN and 16TB volumes mapped to Veeam backup servers.

These LUNS and volumes serve no other purpose or function.

The client wrote a lot of data to this back up location and took it offline. So we cleared snapshots down and brought it back online.

 When the mapped drive was browsed it could be seen there was a lot (TBs + TBs) of “Dead” VM backup files VEEAM advised these could be deleted which they were.

However the adjustment hasnt carried through to the storage end 

VEEAM UI thinks there is 11.9 Tb free NetApp only 2 Tb ** After Storage Snapshot removal 

Have stopped the VEEAM jobs 

reset the storage password in VEEAM

Re-scanned the storage in VEEAM

Re-scanned the appropriate repository. in VEEAM.

NetApp side it thinks the volume is at capacity from the command line or from within On Command

Both also show the same Snapshots.

Its like there is a really old Snap out there locking the capacity but its just not showing up anywhere... 

 

Any thoughts at all ..?

 

24 REPLIES 24

Ontapforrum
14,282 Views

Could you share this data with us:

1) Filer ? & Current ontap version ?

2) ::> volume show-footprint -vserver <svm_name> -volume <volume_name>

3) This is to see if there is any activity such as 'container block reclamation' going on on a Node which holds this volume/aggr.

::> node run -node <node_that_holds_aggr(vol)>
node1> priv set advanced
node1*> wafl scan status

4)::> snapshot show -vserver <svm_name> -volume <volume_name>

5) ::> lun show -vserver <svm_name> -path /vol/path -fields space-reserve

OldGreyBeard
14,262 Views

Hi There 
Thanks for the reply.

These are FAS 8200s

All running OnTap 9.7p14

 

Ran these through as recommended. Please see attached i needed to remove some detail.

The numbers are a little out of square as it has just cleared down its latest and largest snapshot but its not moved by a lot.

When running  LUN show

For the LUN to clear down deleted blocks doesn't space reclamation need to be enabled ..?

Looks like it hasn't been set here ?

 

Thanx

P

 

 

 

 

 

 

OldGreyBeard
14,261 Views

By setting 

Space-Allocation  =  Enabled 

currently  Space-Allocation  =  Disabled 

Ontapforrum
14,259 Views

Please go ahead and enable it, No downtine is required. It 

 

Enabling space allocation for thinly provisioned LUNs:
If you set the space-allocation option to enabled, Data ONTAP notifies the host when the volume has run out of space and the LUN in the volume cannot accept writes. Also this option enables Data ONTAP to reclaim space automatically when your host deletes data. The space-allocation option is set to disabled by default.

 

OldGreyBeard
15,903 Views

Thanks sir ok set is there a way \ command to make it rescan the LUN immediately and if poss check the progress

 

Ontapforrum
15,889 Views

For LUNs, re-scan is usually done from the host side. Could  you perform re-scan first. If it does not show any changes then :

 

Could you follow the steps here, looks like the LUN must be taken offline first, enable the feature and bring the lun online and re-scan from host side (Windows/Linux) side.

If you could follow the steps mentioned here:

http://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.dot-cm-sanag%2FGUID-6AD84908-041A-497D-95A7-BB6AFDD1B282.html

OldGreyBeard
15,888 Views

Yeah i read through that article.

It actually let me change the setting with the LUNN and Volume online.

But with it set as Enabled i took the lun offline and re-enabled anyway.

 

Windows Disk re-scan was reran inline

 

Ontapforrum
15,885 Views

ok thx for the update,  usually it should be ok on the fly. But, when I read the article I thought interesting...but as you have now done it so let's see if that makes any difference. I haven't had time to look at all the attachments in the first response. I will have a look and feed you back.

OldGreyBeard
15,884 Views

thanx sir will sit on it over night

Ontapforrum
15,874 Views

No worries, could you also share the output for that volume ?

::> volume show -vserver <vserver> -volume <volume>

Ontapforrum
15,874 Views

Will need detailed info of the volume, so if you could do -instance from advanced cmd, that will be nice.

OldGreyBeard
15,871 Views

Here you go sir

See what i mean the volume is convinced its full and I'm pretty convinced its not ! (well not anymore anyways.)

paul_stejskal
15,839 Views

https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/How_is_space_utilization_managed_in_a_Data_ONTAP_SAN_environment

 

Keep in mind, LUN space on the host side is viewing the client OS file system (probably NTFS). The space will be different than the WAFL file system as they don't talk to each other. The only way to reclaim space is to use something that uses SCSI UNMAP or some kind of space reclamation.

OldGreyBeard
15,835 Views

Hiya 

Presentation is FCoE

File system is presented to Windows as ReFS 

Setting the space-allocation option to enabled hasn't affected it either.

 

Starting to give up on this one now. Lots of help but nothing that seems to do anything.

VEEAM thinks 5.3 Tb is used

Windows thinks the mapped drive linked to the storage has 5.5 TB is use

NetApp continues to think about 14.5 Tb is used

 

That really sounds wrong to me somewhere.

jcolonfzenpr
15,832 Views

Hi,

Can you check from windows side if SCSI UNMAP is enabled?

 

How to verify whether SCSI UNMAP is enabled or disabled on a Windows host?

fsutil behavior query disabledeletenotify

 

Reference:

https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/ONTAP_and_Windows_SCSI_unmap_and_space_reclamation

 

Also try to “Optimize” the drive from windows:

 

fsutil2.png

Reference:

"Windows also supports manual UNMAP, which can be run on-demand or per a schedule. This is performed using the Disk Optimizer tool. Thin virtual disks can be identified in the tool as volume media types of “thin provisioned drive”—these are the volumes that support UNMAP."

 

Hope this helps!

Jonathan Colón | Blog | Linkedin

OldGreyBeard
15,828 Views

Hi there sir you may be on to something  ...

 

fsutil behavior query disabledeletenotify

NTFS DisableDeleteNotify = 1

ReFS DisableDeleteNotify is not currently set

 

Also from the article which I haven't seen elsewhere...

Note: If you change the space_alloc or space-allocation setting AFTER provisioning a LUN to a Windows host, reboot the Windows host for it to discover the changed setting.

 

Tonight's backup run has just kicked off so i will look at getting to enabled for ReFS  it tomorrow.

Not really keen on running a Disk defrag on the NetApp (goes against everything i was told \ learned)

I assume if it does need this option enabled then between Windows and NetApp at some point they will clean u automatically..

 

 

jcolonfzenpr
15,824 Views

Hi,

 

Sorry I misunderstood the question, I didn't see that you were talking about REFS but according to several articles REFS does not support unmap. It would be better if someone from netapp could confirm it.

 

Note:

Ontap Reference:

https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Systems/FAS_Systems/Windows_host_does_not_auto-reclaim_unused_space_for_thin_provisioned...

 

Note

Backup targets include the above supported configurations. Please contact application and storage array vendors for support details on Fiber Channel and iSCSI SANs. For SANs, if features such as thin provisioning, TRIM/UNMAP, or Offloaded Data Transfer (ODX) are required, NTFS must be used.

 

Microsoft Reference:

https://docs.microsoft.com/en-us/windows-server/storage/refs/refs-overview

 

Hope this help!

 

Jonathan Colón | Blog | Linkedin

OldGreyBeard
15,819 Views

Perhaps its just a conspiracy that wont allow me to delete data so i have to buy more disk !! 🙂

jcolonfzenpr
15,774 Views

Hi,

 

If storage space is your main concern you can use sdelete to free up unused space. sadly it is a manual and intensive process.

 

sdelete -z <drive letter> 

 

sdelete can be used to reflect unused space. The storage will catch up with the changes and at some point reflect the actual storage usage of the LUN. I performed some tests in my lab using sdelete to cleanup free space from the REFS volume and I was able to see the claimed space from ontap.

 

https://docs.microsoft.com/en-us/sysinternals/downloads/sdelete

 

Hope this helps. Good luck!

Jonathan Colón | Blog | Linkedin

Ontapforrum
15,804 Views

Interesting, didn't know ReFS does not support UNMAP with 3rd party storage vendors. Thanks for sharing the article Jonathan. Thin provisioning/UNMAP only supported on "Windows Storage Spaces".

 

That sounds like the Microsoft designed the file-system for it's own Storage.

 

No wonder ReFS is an issue with other Storage Arrays as well.
https://forums.veeam.com/veeam-backup-replication-f2/refs-reclaim-space-on-array-t55541.html


Considering a huge locked up space in the LUN, and if that's pushing aggregate to risk then it is worth adding more disks. But, I guess no point in using ReFS as 'Backup target' with 3rd Party Storage Vendor unless it is NTFS. It may be worth going forward, to point the backup target to new NTFS location and let the old backups retire, and then may be just get rid of the LUN to regain space.

 

It  may be worth updating the Support Interoperability Matrix to show supportability for UNMAP/TRIM etc under "Host Feature" when one selects ONTAP & SAN Host to refine the results. This will help make better decisions before deployment.

Public