Backup VEEAM on Netapp FAS2720 high latency

[Julien_Mos]

Hello,

We have configured the backups of our VMWare virtual infrastructure with VEEAM and the Netapp integration.

We have quite a few VMs, which means that backups overflow into production hours and employees can experience slowness.

I've noticed that every time the backups are launched, the latency of our Netapp FAS2720 increases considerably, reaching latencies of between 90 ms and 190 ms. As soon as the backup is complete, the latency returns to normal at less than 1 ms.

Has anyone ever been in this situation?
Are these normal latencies for VEEAM backups with Netapp integration on a Netapp FAS 2720 with mechanical disk?

Best,

[keitha]

It certainly sounds like you are saturating the spinning media.  With small IO, the NetApp system accepts the writes into the NVRAM and acks it back the client quickly but when you send a long stream of data, like a Veeam backup, the NVRAM can't keep up and you become bound to the disk speed. You either need more drives to distribute the writes (likely ALOT more drives) or consider something like the C-series system to get the flash performance.

[Julien_Mos]

Is there a way to verify that ?

[elementx]

You may check IOPS and/or throughput during slowness, vs. sized performance and you'll likely see you're maxing out IOPS.

You can run fewer jobs in parallel and/or throttle Veeam until latency drops to say 50ms. It will take longer to complete but it won't be as noticeable.

https://helpcenter.veeam.com/docs/backup/vsphere/setting_network_traffic_throttling.html?ver=120

[Julien_Mos]

Hello,

You an see on the graph that the backup begin around 8-9 PM and after that the latency begins to increase. I'll take a look at your proposed solution ty. The problem is that without throttle, jobs finish very late. But I'll have a look 

[elementx · Accepted solution]

How many NL-SAS disks you have? With less than 20? In that case I wouldn't be surprised.

Maybe a closer look at the Veeam log would reveal which stages coincide with peaks, but I don't think you could do much to make Veeam work differently, other than throttling it.

Of course, adding more disks would help increase IOPS as well as throughput, but judging by the latency, you'd need to add quite a few to get the latency to 20-40 ms.

Alternatively you could - this is where a deep dive into per-volume and per-VM details can come handy - add a small flash tier made of SSDs and move the few busiest VMs over there.

[Julien_Mos]

I have got 12 disks of 9TB ...  

So I think the problem come from here like you and  elementx said .