Hi @mzp45
Yes, the points you raise are valid; there is downsampling occuring at two places and each of them uses average by default.
I usually explain that the values in a Grafana panel table are for the datapoints shown in the panel, and Grafana by default displays as many data points as pixels [using Graphite's maxDataPoints API parameter]. So as you zoom in or out the max/min values may change. If you want a true max you can zoom in on the max value you see in the graph a few times until the timespan shows all individual data points.
I suppose you can wrap the metric calls using the the consolidateBy(max) function but I think that result will be misleading too since you essentially get the max sample of the group of data points being consolidated. So now your average is of the consolidated max points, and your overall graph will show much higher values than it should.
Here is an example:

If you zoom in to the last day or so the lines more or less overlap (in my lab), and further so that every data point is displayed natively you get entirely overlapping lines.
If you use storage-aggregation.conf to do the consolidating across retention archives I think you will have the same problem.
You could solve by increasing your retention of your most granular archive to keep the submitted values, then you could run queries without any maxDataPoints set and you will see true max. Or you need to create new metrics for max values. For this you'd have to write a script that would fetch the actual metric datapoints, consolidate, and post the max into some less granular archives (ex: hourly maxes). If you name them in a standard way and use storage-aggregation.conf you could ensure they are rolled up as maxes again as well. Sorry, but other than these [unapealing] ideas I'm not sure what to recommend.
Last thing, even the data points submitted by Harvest are averages of the sample period, which is 60s by default. So your actual single peak second IOPs won't be captured ever. To capture those you need to look at metrics that have histograms [which i don't typically collect] to see unaveraged data.
Cheers,
Chris Madden
Storage Architect, NetApp EMEA (and author of Harvest)
Blog: It all begins with data
If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO or both!