topic Re: Latency on cluster when we do vmotion in ONTAP Discussions

Latency on cluster when we do vmotion

mulbzh — Thu, 17 Oct 2024 08:20:26 GMT

Hello and sorry for my bad english 🙂

We have 2 cluster C250 (ONTAp 9.12) with new switchs Aruba. We have also Vmware with storage ISCSI on Netapp clusters.

When we do Vmotion storage from Vmware, we can see on cluster that the latency is high 70ms and sometimes more than 100ms. When the vmotion is finish, it come back to less than 1ms.

Do you think is normal this latency when we do vmotion or not ?

thanks a lot

Re: Latency on cluster when we do vmotion

TMACMD — Thu, 17 Oct 2024 10:24:20 GMT

Are things properly setup with iscsi?

You should have two iscsi lifs on the Netapp per node in the vserver as well as two iscsi vmkernels on the ESXi hosts. On ESXi each vmkernel should be bound to a single physical interface.

Re: Latency on cluster when we do vmotion

sanmanjax — Thu, 17 Oct 2024 13:32:08 GMT

Shortest answer: Maybe 😐. Tuning a SAN's performance is why we're all still employed 😉.

Definitely confirm iSCSI configuration, as noted by TMACMD. That is critical to your success. As to latency, it depends on a number of factors, including speed of your iSCSI interface, amount of data being transferred, load on the network during transfers, type of disks, number of hops, etc. Your description of the situation, though, seems to indicate that the storage vMotion is mainly responsible for the increase. Where are you measuring the steady-state latency, and are you just looking at the cluster's reported latency, or are you doing an end-to-end measured value? Are the customers seeing the latency, or are they getting normal performance & the latency is due to VMware prioritizing serving data to them over transferring data? I'll confess the last point is a guess - I am not a VMware engineer, so I'm using SAN & array-based knowledge to help you troubleshoot.

Re: Latency on cluster when we do vmotion

mulbzh — Fri, 18 Oct 2024 12:10:47 GMT

Yes, the configuration has been verified by Netapp Support

Re: Latency on cluster when we do vmotion

mulbzh — Fri, 18 Oct 2024 12:13:28 GMT

I see the latency on cluster dashboard. (see attach).
Customers not see latency
The traffic on switch is not very high, the latency is see only on cluster dashboard

Re: Latency on cluster when we do vmotion

sanmanjax — Fri, 18 Oct 2024 13:33:42 GMT

I think you can safely say that the latency is due to background processing and data movement due to the Storage vMotion activities. If it's not affecting your customers, then it's just the cost of doing business and shouldn't be a concern.

Best wishes!

Re: Latency on cluster when we do vmotion

TMACMD — Fri, 18 Oct 2024 13:36:18 GMT

Are you moving between datastores on the same NetApp cluster? if so, then It is very possible this due to VAAI operations. VMware/ESXi will offload some functions directly to the storage.

https://core.vmware.com/resource/vmware-vsphere-apis-array-integration-vaai#sec9410-sub3

Re: Latency on cluster when we do vmotion

mulbzh — Fri, 18 Oct 2024 13:46:00 GMT

no, i have one volume on cluster and an other volume on other cluster.

Hypervisor are connect to volume trough ISCSI. Then, i move VM from volume1 to volume2 on other cluster

Re: Latency on cluster when we do vmotion

TMACMD — Fri, 18 Oct 2024 13:51:46 GMT

gonna say it again: iscsi configured for BEST practices? meaning

1. jumbo frames throughout

2. two LIFS per node in the cluster

3. two VMKs on each host, binding a VMK to a particular interface?

NetApp support will likley not vet out an entire setup. I am aksing you check all this out.

Re: Latency on cluster when we do vmotion

Sanaman — Sun, 20 Oct 2024 23:47:51 GMT

Have you installed OnTap Tools for VMWrae vSphere? If not install it and add your clusters to it and run the vmotion.

Re: Latency on cluster when we do vmotion

mulbzh — Mon, 21 Oct 2024 13:34:47 GMT

it is installed