QOS Policy Group counters? Harvest/Grafana/Graphite json file?

[James_Castro]

I am looking to get a detailed performance view of named QOS policy groups rather than just the name of the volume on Grafana.  I've installed the Harvest/Grafana/Graphite OVA along with the NetApp dashboards.  I am looking to create or find a dashboard the lists each QOS policy by name.  Below is an image of the QOS drilll down but it lists the volume and does not tell me what QOS policy group it has been assigned.  If I could see performance based on the group name that woul be great.  any sugestions?

[madden · Accepted solution]

Hi,

I haven't created any default dashboards for stats at the policy group level but they are collected by default.  

Here I can show them in the native Graphite GUI:

So you could 'save as' from the 'volume' dashboard to a new QOS Policy Group dashboard and then edit a bit to show these.

Hope that helps!

Cheers,
Chris Madden

Storage Architect, NetApp EMEA (and author of Harvest)

Blog: It all begins with data

P.S.  Please select “Options” and then “Accept as Solution” if this response answered your question so that others will find it easily!

[James_Castro]

aaahhh.  I see why I was having so many issues.  Are there counters I need to add or anything like that?  That counter is missing on my set-up.  I've installed the ADVA-64 OVA which runs the v1.2.2 verion of Harvest.  Thanks

[James_Castro]

I'm also polling CDOT Realease 8.2.1

[James_Castro]

Okay.  You can disregard my last question.  I figured it out.  While we have named QOS policies we have not set limits on most of them.  Most are set to unlimited for the time being as we gather baselines.  Once a limit has been set (like 1000 iops) a counter will be measured against the specified policy.

This ,of course, is probably a well known fact already so I will just be quiet now.  🙂

[madden]

Hi,

Actually, if you create a QoS policy group, and then modify resources (SVM or vol or lun or file) into it, then IO to those resources will roll-up to the policy group level.  Also, if using OPM with Data ONTAP 8.2 it will will automatically assign any volume which has no policy group to the _performance_monitor_volumes policy group.  This policy group has no limit and is there so that we get workload tracking of each volume so OPM can do it's magic.  So if you don't see a qos_policy folder inside an SVM I suspect either (a) you have no IO occurring in the SVM or (b) the vols are not assigned to a policy group (which would be odd since OPM should do that!).

Hope that helps!

Cheers,

Chris

[JamesIlderton]

I've got a related question here...we've been able to add latency from qos policies to our Dashboards, but I can't seem to find the metric for the VALUE assigned in the QoS Policy.  Ideally we'd like to represent that on the IOPS/Throughput graphs to visually show the "headroom" or to quickly see where the visible workload is in relation to the limits placed on them.  Does this metric exist, or can it be added?

[madden]

Hi @JamesIlderton

The throttle limit (iops and/or throughput) is not something collected right now.  There isn't a counter for it so I'd have to hard code something in netapp-worker.  Also, throttles are applied per policy group and since multiple objects (volumes, luns, etc) can be added to a single policy group I could only report on it at the policy group level, not the object (volune/lun/etc) level.  Would tracking at the policy group level still be sufficient?  The other thing I advise people is to check the 'latency from throttle' graphs to see if any workloads are hitting their limit.  But, the downside of this technique is you can't see how close you are to the limit, only once you hve hit it and latency is being added to keep you from exceeding it.

Cheers,
Chris Madden

Storage Architect, NetApp EMEA (and author of Harvest)

Blog: It all begins with data

If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO or both!

[JamesIlderton]

Yep, you hit the nail on the head - we would LIKE to be able to display the limit fro mthe object level, but that would require a join-type function since the relationship is object -> policy -> limit.  If ONTAP doesn't have the counter directly attached to the object is there a way to do a join-style lookup to display it?

We're using the Latency from Throttle now and while it is handy, it doesn't go quite as far as we'd like yet.  AFAIK OCPM nor OCI can display this either.

[madden]

Hi @JamesIlderton

I will add this to the feature request backlog.  

At the policy group level I could send metrics like:

throttle_limit_throughput: configured limit

throttle_limit_iops: configured limit

throttle_remaining_throughput:  (configured limit - current throughput)

throttle_remaining_iops: (configured limit - current throughput)

And at the object level I could send:

throttle_limit_throughput: configured limit (if applied to only one object, otherwise not sent)

throttle_limit_iops: configured limit (if applied to only one object, otherwise not sent)

throttle_remaining_throughput:  (configured limit - current throughput)

throttle_remaining_iops: (configured limit - current throughput)

Would that work?  Any better ideas?

Cheers,
Chris Madden

Storage Architect, NetApp EMEA (and author of Harvest)

Blog: It all begins with data

If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO or both!