Data ONTAP Discussions

Filebased backup from CIFS Vault shares cause high latency and utilization

Hi guys,

 

we use SnapVault to replicate various volumes (NFS and CIFS) from a remote 8.3 Cluster to our internal 8.3 Cluster.

So far so good.

 

Now we would like to Backup the replicated CIFS shares with TSM to tape.

Everytime a backup starts on a read-only Vault share we get warning and ciritcal alerts regarding CIFS latency.

 

For comparison:

~ 140us on "normal" read-write CIFS shares

~ 502992us on the replicated VAULT CIFS shares

 

This also happens if tools like xcopy, robocopy etc. are used to copy the files.

 

 

statistics (just 5 seconds) while doing backups

cluster.cluster: 7/1/2015 21:44:00
cpu cpu total fcache total total data data data cluster cluster cluster disk disk pkts pkts
avg busy ops nfs-ops cifs-ops ops spin-ops recv sent busy recv sent busy recv sent read write recv sent
---- ---- -------- -------- -------- -------- -------- -------- -------- ---- -------- -------- ------- -------- -------- -------- -------- -------- --------
Minimums:
23% 41% 5239 611 4273 0 4241 5.56MB 4.86MB 0% 5.14MB 4.69MB 0% 142KB 120KB 528KB 0B 6302 8296
Averages for 15 samples:
28% 67% 6120 1488 4631 0 5111 7.26MB 49.2MB 3% 6.99MB 49.0MB 0% 282KB 280KB 13.1MB 6.94MB 9862 41362
Maximums:
40% 83% 7554 2927 4774 0 6514 12.2MB 137MB 11% 11.9MB 136MB 0% 664KB 659KB 38.7MB 26.3MB 15809 106834

 

 

Did anyone experience similar issues?

Anything I could try to resolve this? (exchanging the backup software is not an option currently)

 

Many thanks!

6 REPLIES

Re: Filebased backup from CIFS Vault shares cause high latency and utilization

So i'm on the ame page as you, are you using ndmp?

 

And what is your disk setup on the backup side?

Re: Filebased backup from CIFS Vault shares cause high latency and utilization

Unfortunatly it`s just basic file-copying. No NDMP for now.

It does not matter if we use TSM, robocopy, xcopy etc. also it does not matter if we copy it to a VM on the same cluster, a QNAP NAS, or to a physical windows server.

 

NetApp support told us "low-end-systems" like ours (2552) can be affected of the bug  BURT "880471" - and our system is affected.

They said it was caused by a lif that was not on his home port. But we have this issue even if all lifs are on their home ports.

So far we did not get a proper solution.

 

Re: Filebased backup from CIFS Vault shares cause high latency and utilization

Well, that's an ugly way to backup..

 

Ok, what's your disk setup..  How many disks in the aggregate...

 

 

Re: Filebased backup from CIFS Vault shares cause high latency and utilization

I know it`s not ideal and it`s going to be changed but not at this time.

 

We have 2 Shelfs, each have 20x900GB SAS and 4x200GB SSD.

Out of those we have built 2 aggregates with the same assignment.

 

I hope 8.3.1 or a future release will fix it, if this is an hardware issue I hope we get replacements.

 

 

Re: Filebased backup from CIFS Vault shares cause high latency and utilization

Ok i apologize for keep asking but i want to make sure i follow.

Your vaulted shares seem to be going to an aggregate with enough iops based on your above disk layout. I assume you have hybrid aggrs with sas and ssd.
Gimme exact raid group layout of the destination shares underlying aggr. I just want to triple check

Also did someone accidently put qos on?

Re: Filebased backup from CIFS Vault shares cause high latency and utilization

raid type = dp for SAS, 4 for SSD

raid gr. size = 19

raid alloc = 18

1 flashpool with 6 disks

 

the latency during file copys/backups on the vault cifs shares are 300-400x higher than on the normal cifs shares.

And the normal shares are about 3TB (no backups / latency issues) compared to the vault shares with only 50GB.

 

Both reside on the same aggr. So I think this is a software related issue.

QoS is active for all CIFS shares (100MB/s).  Nothing improves when QoS is disabled.

 

Forums