ONTAP Discussions

Odd Storage issue (possibly Snapshot related ..?)

OldGreyBeard
30,337 Views

Hi All

 

Had a case open with VEEAM and NetApp neither of which is going anywhere pretty quickly, so will throw it out there and see if anyone has any ideas.

We have a large estate broken down in to areas which mirror the configuration of each other so a nice lot of reference points.

There are 16 Tb NetApp LUN and 16TB volumes mapped to Veeam backup servers.

These LUNS and volumes serve no other purpose or function.

The client wrote a lot of data to this back up location and took it offline. So we cleared snapshots down and brought it back online.

 When the mapped drive was browsed it could be seen there was a lot (TBs + TBs) of “Dead” VM backup files VEEAM advised these could be deleted which they were.

However the adjustment hasnt carried through to the storage end 

VEEAM UI thinks there is 11.9 Tb free NetApp only 2 Tb ** After Storage Snapshot removal 

Have stopped the VEEAM jobs 

reset the storage password in VEEAM

Re-scanned the storage in VEEAM

Re-scanned the appropriate repository. in VEEAM.

NetApp side it thinks the volume is at capacity from the command line or from within On Command

Both also show the same Snapshots.

Its like there is a really old Snap out there locking the capacity but its just not showing up anywhere... 

 

Any thoughts at all ..?

 

24 REPLIES 24

paul_stejskal
3,875 Views
I agree.

OldGreyBeard
3,840 Views

So the below linkage exactly supports our issue as best i can tell (and REFS  is a VEEAM best practice)

https://forums.veeam.com/veeam-backup-replication-f2/refs-reclaim-space-on-array-t55541.html

 

We also tried the below.

HOW TO RECLAIM SPACE FOR THIN PROVISIONED LUNS USING POWERSHELL FOR WINDOWS… – One thing (wordpress.com)

 

Bat as its presented as a mapped drive it cant "see through it" if that makes any sense. We also tried with Disk ID each time it does not detect it as a NetApp disk.

 

Our case is ongoing..

jcolonfzenpr
3,821 Views

Hi,

 

Answering your last post:

 

"So the below linkage exactly supports our issue as best i can tell (and REFS  is a VEEAM best practice)"

 

You are right, we never doubt the configuration. veeem's best practices have nothing to do with your space reclamation problem. In essence the problem is microsoft disabling the UNMAP feature in the REFS file system.

 

when you have time use the veeam best practices site to validate your configuration:

https://bp.veeam.com/vbr/VBP/3_Build_structures/B_Veeam_Components/B_backup_repositories/block.html#refs 

 

ReFS

ReFS is using linked clone technology. This is perfect for synthetic operations and will save dramatic IOs and throughput during operations like merges or creating synthetic fulls.

Follow these best practices when using ReFS:

  • Format the volume with 64KB block size
  • Configure 256 KB block size on LUNs (Storage or RAID controller)
  • Never bring linked clone space savings into your calculation for required storage space
  • “All ReFS supported configurations must use Windows Server Catalog certified hardware” - please contact your hardware vendor
  • Never use any shared LUN concept with ReFS and a Veeam Repository
  • Check the existing driver version of ReFS:
    • The minimum should be ReFS.sys 10.0.14393.2457 on Windows Server 2016.
    • The minimum should be ReFS.sys 10.0.17763.1369 on Windows Server 2019.
    • General recommendation: fully patched with the most recent Windows Updates.

 

Could you explain what you mean by this statement?

"Bat as its presented as a mapped drive it cant "see through it" if that makes any sense. We also tried with Disk ID each time it does not detect it as a NetApp disk."

 

"We also tried the below.

HOW TO RECLAIM SPACE FOR THIN PROVISIONED LUNS USING POWERSHELL FOR WINDOWS… – One thing (wordpress.com)"

 

The command mentioned in the article "Invoke-NaHostVolumeSpaceReclaim" is for ontap 7mode storage.  Even if you use the correct option "Invoke-NcHostVolumeSpaceReclaim", the host operating system and the file system need to support the UNMAP capability (SCSI thin provisioning features).

 

Host support for SCSI thin provisioning

https://docs.netapp.com/ontap-9/topic/com.netapp.doc.dot-cm-sanag/GUID-882DF278-68F2-49DF-AB60-5264DC962864.html

 

When I try to run the cmdlet the following is returned:

 

Identifying the netapp lun:

 

PS C:\Users\administrator> get-disk | Select-Object Number,FriendlyName,OperationalStatus | Where-Object {$_.FriendlyName -eq "NETAPP LUN C-Mode" }

Number FriendlyName      OperationalStatus
------ ------------      -----------------
     1 NETAPP LUN C-Mode Online


PS C:\Users\administrator>

 

 

Identifying the volume in windows:

 

PS C:\Users\administrator> Get-Volume | Where-Object {$_.FileSystem -eq "Refs"}

DriveLetter FileSystemLabel FileSystem  HealthStatus  SizeRemaining     Size
----------- --------------- ----------  ------------  -------------     ----
E           DATA            ReFS           Healthy      4.88 GB       19.81 GB


PS C:\Users\administrator>

 

 

Executing the cmdlet:

 

PS C:\Users\administrator> Invoke-NcHostVolumeSpaceReclaim  e:
Invoke-NcHostVolumeSpaceReclaim : DeviceIoControl failed (FSCTL_GET_NTFS_VOLUME_DATA)
At line:1 char:1
+ Invoke-NcHostVolumeSpaceReclaim  e:
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Invoke-NaHostVolumeSpaceReclaim], Exception
    + FullyQualifiedErrorId : Could not get volume info,DataONTAP.PowerShell.SDK.Cmdlets.Windows.InvokeNaHostVolumeSpaceReclaim

PS C:\Users\administrator>

 

 

As the error indicates, this command is only compatible with NTFS.

"Invoke-NcHostVolumeSpaceReclaim : DeviceIoControl failed (FSCTL_GET_NTFS_VOLUME_DATA)"

 

The cmdlet help says:

get-help Invoke-NcHostVolumeSpaceReclaim

"This cmdlet examines the NTFS file system on a Windows volume, locates all free space, and deallocates the corresponding blocks on the Data ONTAP storage controller. No difference is apparent from Windows' perspective, but all space not consumed by files on the volume is freed on the storage controller."

 

Hope this helps!

Jonathan Colón | Blog | Linkedin

jcolonfzenpr
3,819 Views

Upon further investigation of the problem, it appears that veeam's recommended values for unmap in REFS is to keep it disabled.

 

fsutil behavior set DisableDeleteNotify ReFS 1

 

https://forums.veeam.com/post364022.html#p364022

https://original-network.com/veeam-br-v11-and-refs/

 

 

Jonathan Colón | Blog | Linkedin
Public