Had a case open with VEEAM and NetApp neither of which is going anywhere pretty quickly, so will throw it out there and see if anyone has any ideas.
We have a large estate broken down in to areas which mirror the configuration of each other so a nice lot of reference points.
There are 16 Tb NetApp LUN and 16TB volumes mapped to Veeam backup servers.
These LUNS and volumes serve no other purpose or function.
The client wrote a lot of data to this back up location and took it offline. So we cleared snapshots down and brought it back online.
When the mapped drive was browsed it could be seen there was a lot (TBs + TBs) of “Dead” VM backup files VEEAM advised these could be deleted which they were.
However the adjustment hasnt carried through to the storage end
VEEAM UI thinks there is 11.9 Tb free NetApp only 2 Tb ** After Storage Snapshot removal
Have stopped the VEEAM jobs
reset the storage password in VEEAM
Re-scanned the storage in VEEAM
Re-scanned the appropriate repository. in VEEAM.
NetApp side it thinks the volume is at capacity from the command line or from within On Command
Both also show the same Snapshots.
Its like there is a really old Snap out there locking the capacity but its just not showing up anywhere...
Any thoughts at all ..?
Could you share this data with us:
1) Filer ? & Current ontap version ?
2) ::> volume show-footprint -vserver <svm_name> -volume <volume_name>
3) This is to see if there is any activity such as 'container block reclamation' going on on a Node which holds this volume/aggr.
::> node run -node <node_that_holds_aggr(vol)>
node1> priv set advanced
node1*> wafl scan status
4)::> snapshot show -vserver <svm_name> -volume <volume_name>
5) ::> lun show -vserver <svm_name> -path /vol/path -fields space-reserve
Thanks for the reply.
These are FAS 8200s
All running OnTap 9.7p14
Ran these through as recommended. Please see attached i needed to remove some detail.
The numbers are a little out of square as it has just cleared down its latest and largest snapshot but its not moved by a lot.
When running LUN show
For the LUN to clear down deleted blocks doesn't space reclamation need to be enabled ..?
Looks like it hasn't been set here ?
Please go ahead and enable it, No downtine is required. It
Enabling space allocation for thinly provisioned LUNs:
If you set the space-allocation option to enabled, Data ONTAP notifies the host when the volume has run out of space and the LUN in the volume cannot accept writes. Also this option enables Data ONTAP to reclaim space automatically when your host deletes data. The space-allocation option is set to disabled by default.
For LUNs, re-scan is usually done from the host side. Could you perform re-scan first. If it does not show any changes then :
Could you follow the steps here, looks like the LUN must be taken offline first, enable the feature and bring the lun online and re-scan from host side (Windows/Linux) side.
If you could follow the steps mentioned here:
Yeah i read through that article.
It actually let me change the setting with the LUNN and Volume online.
But with it set as Enabled i took the lun offline and re-enabled anyway.
Windows Disk re-scan was reran inline
ok thx for the update, usually it should be ok on the fly. But, when I read the article I thought interesting...but as you have now done it so let's see if that makes any difference. I haven't had time to look at all the attachments in the first response. I will have a look and feed you back.
Keep in mind, LUN space on the host side is viewing the client OS file system (probably NTFS). The space will be different than the WAFL file system as they don't talk to each other. The only way to reclaim space is to use something that uses SCSI UNMAP or some kind of space reclamation.
Presentation is FCoE
File system is presented to Windows as ReFS
Setting the space-allocation option to enabled hasn't affected it either.
Starting to give up on this one now. Lots of help but nothing that seems to do anything.
VEEAM thinks 5.3 Tb is used
Windows thinks the mapped drive linked to the storage has 5.5 TB is use
NetApp continues to think about 14.5 Tb is used
That really sounds wrong to me somewhere.
Can you check from windows side if SCSI UNMAP is enabled?
How to verify whether SCSI UNMAP is enabled or disabled on a Windows host?
fsutil behavior query disabledeletenotify
Also try to “Optimize” the drive from windows:
"Windows also supports manual UNMAP, which can be run on-demand or per a schedule. This is performed using the Disk Optimizer tool. Thin virtual disks can be identified in the tool as volume media types of “thin provisioned drive”—these are the volumes that support UNMAP."
Hope this helps!
Hi there sir you may be on to something ...
fsutil behavior query disabledeletenotify
NTFS DisableDeleteNotify = 1
ReFS DisableDeleteNotify is not currently set
Also from the article which I haven't seen elsewhere...
Note: If you change the space_alloc or space-allocation setting AFTER provisioning a LUN to a Windows host, reboot the Windows host for it to discover the changed setting.
Tonight's backup run has just kicked off so i will look at getting to enabled for ReFS it tomorrow.
Not really keen on running a Disk defrag on the NetApp (goes against everything i was told \ learned)
I assume if it does need this option enabled then between Windows and NetApp at some point they will clean u automatically..
Sorry I misunderstood the question, I didn't see that you were talking about REFS but according to several articles REFS does not support unmap. It would be better if someone from netapp could confirm it.
Backup targets include the above supported configurations. Please contact application and storage array vendors for support details on Fiber Channel and iSCSI SANs. For SANs, if features such as thin provisioning, TRIM/UNMAP, or Offloaded Data Transfer (ODX) are required, NTFS must be used.
Hope this help!
If storage space is your main concern you can use sdelete to free up unused space. sadly it is a manual and intensive process.
sdelete -z <drive letter>
sdelete can be used to reflect unused space. The storage will catch up with the changes and at some point reflect the actual storage usage of the LUN. I performed some tests in my lab using sdelete to cleanup free space from the REFS volume and I was able to see the claimed space from ontap.
Hope this helps. Good luck!
Interesting, didn't know ReFS does not support UNMAP with 3rd party storage vendors. Thanks for sharing the article Jonathan. Thin provisioning/UNMAP only supported on "Windows Storage Spaces".
That sounds like the Microsoft designed the file-system for it's own Storage.
No wonder ReFS is an issue with other Storage Arrays as well.
Considering a huge locked up space in the LUN, and if that's pushing aggregate to risk then it is worth adding more disks. But, I guess no point in using ReFS as 'Backup target' with 3rd Party Storage Vendor unless it is NTFS. It may be worth going forward, to point the backup target to new NTFS location and let the old backups retire, and then may be just get rid of the LUN to regain space.
It may be worth updating the Support Interoperability Matrix to show supportability for UNMAP/TRIM etc under "Host Feature" when one selects ONTAP & SAN Host to refine the results. This will help make better decisions before deployment.
NetApp Wins One Silver and One Bronze Stevie® Award in 2022 Stevie Awards for Sales and Customer Service