Seeing High NFS Read Latency but very low IOPS

[mdvillanueva]

Hi Expert,

I need your opinion on this issue we are having. One of our NFS datastore, I see constantly showing a read latency above 20 millisec, however the IOPS is below 10 per secs. Then there would be a spurt where the Read Ops will shoot up but the latency will drop to less than 5 millisec.

[ralfaro01zero]

Looks like something else is consuming your network resources, do you have any other workload on that filer (iSCSI, CIFS?). I would check as well if there is any other 3rd party service consuming resources in your network.

Cheers.

[mdvillanueva]

I have alerting enabled for all volumes that reaches 15 ms for a duration of 5 mins. That is the only volume that consistently reaches the treshold. I was able to corelate the spike to a specific vm. What is odd is why the iops is very low.

[jkennedy_4]

Here's a guess.

S/ATA drives can do about 20 ops at 15-20ms latency.  I'm guessing that the high latency line is "other"?  If that's the case, it looks to me like you have something doing small amounts of metadata IO which impact one drive at a time, and thus the high latency.  When you get to that spike of ops but drop in latency, that looks like a full scale data read, which uses more drives in sequence.