What does the other_latency counter mean?

[keahey]

While doing some performance advisor stats with a customer, we noticed a counter other_latency.  What does this counter mean and how do we explain it to our customers.

[arndt]

That is latency for operations other than reads or writes. For example, in an NFS environment, metadata operations such as GETATTR and ACCESS calls would be other_ops that are measured for response time with other_latency.

Mike

[adaikkap]

Basically these are operation done by the system, and not the user.

Regards

adai

[arndt]

No, other_latency and other_ops are not related to system work. They are related to protocol operations initiated by a client, but they are not read or write operations.

Mike

[adaikkap]

Thanks mike for correcting me.So what are they ? if not read or write ?

Regards

adai

[arndt]

I don’t have a full list, but there are a number of different types of operations in a NFS environment that are not reads or writes. Examples would be GETATTR, ACCESS, and LOOKUP calls.

MIke

[reide]

The nfsv3 counter object in DataONTAP tells you what all these 'other' operations are for NFS. I'm pretty sure there is a similar one for CIFS.  Here is a list of the NFSv3 operations other than read & write:

null, getattr, setattr, lookup, access, readlink, create, mkdir, symlink, mknod, remove, rmdir, rename, link, readdir, readdirplus, fsstat, fsinfo, pathconf, commit

There are three counters in the nfsv3 counter object that are extremely useful for monitoring these "other" operations if you need to.  You can create custom views in Performance Advisor that will show each of these counters over time.  Very useful for troubleshooting "chatty" NFS applications that generate a lot of 'other' NFS operations.

Name: nfsv3_op_count
Description: Array of select NFS v3 operation counts
Properties: delta
Unit: none
Size: 22 column array
Column names: null, getattr, setattr, lookup, access, readlink, read, write, create, mkdir, symlink, mknod, remove, rmdir, rename, link, readdir, readdirplus, fsstat, fsinfo, pathconf, commit

Name: nfsv3_op_percent
Description: Array of select NFS v3 operations as a percentage of total NFS v3 operations
Properties: percent
Unit: percent
Size: 22 column array
Column names: null, getattr, setattr, lookup, access, readlink, read, write, create, mkdir, symlink, mknod, remove, rmdir, rename, link, readdir, readdirplus, fsstat, fsinfo, pathconf, commit
        Base Name: nfsv3_ops
        Base Description: Total number of NFS v3 operations per second
        Base Properties: rate
        Base Unit: per_sec

Name: nfsv3_op_latency
Description: Array of latencies of select NFS v3 operations
Properties: average
Unit: microsec
Size: 22 column array
Column names: null, getattr, setattr, lookup, access, readlink, read, write, create, mkdir, symlink, mknod, remove, rmdir, rename, link, readdir, readdirplus, fsstat, fsinfo, pathconf, commit
        Base Name: nfsv3_op_latency_base
        Base Description: Array of select NFS v3 operation counts for latency calculation
        Base Properties: delta,no-display
        Base Unit: none
        Base Size: 22 column array
        Base Column names: null, getattr, setattr, lookup, access, readlink, read, write, create, mkdir, symlink, mknod, remove, rmdir, rename, link, readdir, readdirplus, fsstat, fsinfo, pathconf, commit

[jim_dewaard]

Hey Keahey,

We had the same exact issue.  Other_ops were showing up in perf advisor and completely throwing off our graphs.  After failed troubleshooting with both NetApp and VMware we finally determined that the Veeam Monitoring platform was constantly enumerating our NFS datastores.  After shutting off the Veeam collector service, other_ops went away.  I believe Veeam released an update which fixes this behavior, but I wouldn't doubt that other monitoring platforms could cause the same issue.

Hope that helps.

Thanks,

[mcorbeille]

Does anyone know what kind of operations would be constituted as other_ops and included in other_latency for a volume that is accessed by FCP only?  I have a customer that is seeing sub 5ms latency for both reads and writes but he is concerned about his other_latency spikes of 40ms.  Any assistance would be appreciated.

Thanks,

Mike

[chriszurich]

I am running into this problem as well.  I'm seeing 600 millisecond other_latency spikes within the Volume Latency View in performance advisor. This is a Fibre Channel LUN which is storing Oracle Data files.