Actually for latency the same concept applies - aggregate latency and volume latency are different, disconnected, but related in roundabout ways. So just as with IOPs latency for the aggregate is tracked separately from latency for a volume. Also, we now need to take read/write operations into account more specifically.
Starting with aggregates. When not under high load stress, latency for an aggregate is essentially constant, between 4-10ms per IOP depending on disk type, size, etc. Why? All IOPs are 4K in size. A wide ranging workload tends toward a level of randomness in access despite WAFLs best efforts to leverage sequential access when it can. Thus the latency from disk is mathematical for every aggregat IOP based on degree of randomness, rotational speed, interface speed, etc. That is classic disk math.
But clearly a volume is not always getting 4-10ms per IOP. In fact it is getting wildly different performance from <1ms to 20ms per IOP typically (again assuming no silly loads). Why? First, consider writes. Writes are not sent to disk first. Rather, a write is stored in the NVRAM (or equivalent) on both nodes of an HA pair then acknowledged. The volume latency could easily be 0.2ms for write operations. Periodically NVRAM is flushed to disk as needed. That triggers write IOPs to the aggregate in that 4-10ms range as described above. Thus a long as disk is keeping up bandwidth-wise with the total combined write load of the volumes, volume writes can get really good latency performance in the sub millisecond range while the disks are steady at 4-10ms per backend IOP.
Volume reads have a similar result. The first read of any volume has to go to disk to get data so may be subject to that 4-10ms aggregate latency in addition to internal processing overhead - protocol, space efficiency expansion, etc. But, if the readahead prediction is good, subsequent reads will mostly come from data in RAM that came along for free with that first read. As such, subsequent volume read latencies could be much reduced.
All that holds in isolation. There are numerous other "total system" factors that drive both read and write latency up, the largest of which is the number if volume IOPs that directly require aggregate disk IOPs to meet. There can be multiplication effects due to space efficiencies as well - multiple layers of snapshots, deduplication, cloning can call force extra disk reads to find the key data. Extra or more disk needs due to load drive up aggregate latency which eventually drives up volume latency as well. There are performance counters to track how many reads were satisfied from various layers of the data stack - RAM, FlashPool (if available), SSD (if all Flash), FlashCache, and disk. As you watch periods where more reads are require disk accesses, volume read latencies naturally increase as well.
A key takeaway is that you shouldn't try to add/average volume performance indicators to get the equivalent aggregate performance indicators. Still comparing apples to oranges. Of course they are related, but they are not directly comparable. Rather, you can use one as the indicator to investigate the other. For example, if a volume is getting unacceptable latency, you might then check the aggregate latency where that volume lives. If it is within the calculated mathematical range the aggregate should have based on disk type, then aggregate performance may not be the contributing factor to the volume latency in question. If aggregate latency is either high or spiky in sync with the volume latency, then aggregate performance is more likely to be a contributing factor.
An example - I had a system that would average 60-80ms latency for reads for hours (it was a backup cycle) on a single aggregate measured at the volume level. Protocol IOPs would hit 30K, but backend disk IOPs would be more like 15K - shows the disconnect. Disk utilization would spike to 75%-80% but wasn't steadily high. Disk read latency was crappy though due to the extreme load. Throughput off the system was 15-20 Gbps over multiple links of course. This was bulk data reads, and in the aggregate data was flowing out really well. Measured against a single volume the latency and performance was not great, but being a backup cycle it was the total system performance that was important and how do I complain about systaining 15Gbps out for 6 hours? That's 32TB out in 6 hours give or take. Of course any "normal" application read during that window also saw that terrible response due to the aggregate trying to meet the backup workload.
Hope this helps.
Bob Greenwald
Senior Systems Engineer | cStor
NCIE SAN ONTAP, Data Protection
Kudos and accepted solutions are always appreciated.
