ONTAP Discussions

Latency on cluster when we do vmotion

mulbzh
1,514 Views

Hello and sorry for my bad english  🙂

 

We have 2 cluster C250 (ONTAp 9.12) with new switchs Aruba. We have also Vmware with storage ISCSI on Netapp clusters.

When we do Vmotion storage from Vmware, we can see on cluster that the latency is high 70ms and sometimes more than 100ms. When the vmotion is finish, it come back to less than 1ms.

Do you think is normal this latency when we do vmotion or not ?

thanks a lot

10 REPLIES 10

TMACMD
1,462 Views

Are things properly setup with iscsi?

 You should have two iscsi lifs on the Netapp per node in the vserver as well as two iscsi vmkernels on the ESXi hosts. On ESXi each vmkernel should be bound to a single physical interface. 

mulbzh
1,373 Views

Yes, the configuration has been verified by Netapp Support

sanmanjax
1,447 Views

Shortest answer:  Maybe 😐.  Tuning a SAN's performance is why we're all still employed 😉.

 

Definitely confirm iSCSI configuration, as noted by TMACMD.  That is critical to your success.  As to latency, it depends on a number of factors, including speed of your iSCSI interface, amount of data being transferred, load on the network during transfers, type of disks, number of hops, etc.  Your description of the situation, though, seems to indicate that the storage vMotion is mainly responsible for the increase.  Where are you measuring the steady-state latency, and are you just looking at the cluster's reported latency, or are you doing an end-to-end measured value?  Are the customers seeing the latency, or are they getting normal performance & the latency is due to VMware prioritizing serving data to them over transferring data?  I'll confess the last point is a guess - I am not a VMware engineer, so I'm using SAN & array-based knowledge to help you troubleshoot. 

mulbzh
1,373 Views

I see the latency on cluster dashboard. (see attach).
Customers not see latency
The traffic on switch is not very high, the latency is see only on cluster dashboard

sanmanjax
1,343 Views

I think you can safely say that the latency is due to background processing and data movement due to the Storage vMotion activities.  If it's not affecting your customers, then it's just the cost of doing business and shouldn't be a concern.

 

Best wishes!

TMACMD
1,340 Views

Are you moving between datastores on the same NetApp cluster? if so, then It is very possible this due to VAAI operations. VMware/ESXi will offload some functions directly to the storage.

https://core.vmware.com/resource/vmware-vsphere-apis-array-integration-vaai#sec9410-sub3

mulbzh
1,336 Views

no, i have one volume on cluster and an other volume on other cluster.

Hypervisor are connect to volume trough ISCSI. Then, i move VM from volume1 to volume2 on other cluster

TMACMD
1,331 Views

gonna say it again: iscsi configured for BEST practices? meaning

1. jumbo frames throughout

2. two LIFS per node in the cluster

3. two VMKs on each host, binding a VMK to a particular interface?

 

NetApp support will likley not vet out an entire setup. I am aksing you check all this out.

Sanaman
1,184 Views

Have you installed OnTap Tools for VMWrae vSphere? If not install it and add your clusters to it and run the vmotion.

mulbzh
1,140 Views

it is installed

Public