Subscribe
Accepted Solution

Harvest IFGRP Counters

[ Edited ]

I am using Harvest and Grafana for a few months now. I noticed recently that the Ethernet Port Utilization shown in Grafana is near 0% for all ports, though we have quite some load on the nodes. We use IFGRPS/LACP channels and my assumption is that the traffic is counted on the channel ports. Are there different counters for ifgrps? If not, why is the port utilization for all Ports 0% when my SVM shows 500MB/s NFS throughput?

Re: Harvest IFGRP Counters

Hi @moep

 

Harvest submits usage metrics at the physical port and LIF level. It does not track activity at the VLAN or LACP ifgrp level.  Do you see send / recv activity on the physical ports?  

 

The Link Utilization percent, or port utilization percent, is the max ( (send / link_speed), (recv / link_speed) ).  This metric is calculated by a plugin.  If there is an issue calculating I would expect something to be logged in the poller logfile found in /opt/netapp-harvest/log.

 

Maybe you can post a screenshot to help explain the issue you observe?

 

Cheers,
Chris Madden

Storage Architect, NetApp EMEA (and author of Harvest)

Blog: It all begins with data

 

If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO

 

 

Re: Harvest IFGRP Counters

Hi Chris,

 

here is the port utilization, it never shows more than 2% for any port.

grafana_port_util.JPG

 

If you look at the troughput of a hosted SVM, there is lot of traffic. And that isn't the only SVM.

grafana_SVM_throughput.JPG

 

I see the same behaviour for 4 different Clusters. Why is the traffic not counted towards port utilization?

Re: Harvest IFGRP Counters

Hi @moep

 

If I look at my "NetApp Detail: Network Port" dashboard I see this:

port.png

 

So the send, receive, and link util graphs are all correlated.

 

The screenshot you provided looks like it comes from the "NetApp Dashboard: Node" dashboard.  I checked this one and it basically showed the same as above, so it doesn't seem to be a grafana display issue.

 

Can you check if the "NetApp Detail: Network Port" dashboard send and receive panels and see if these match reality?

 

Also, to doublecheck, could this traffic be FCP and thus show up under the FC ports and not Ethernet ports?

 

If things are still not cleared up please provide the following so I can dig deeper:

o) Data ONTAP release

o) Hw model in cluster

o) Explain configuration of vlans and ifgrps

 

 

Cheers,
Chris Madden

Storage Architect, NetApp EMEA (and author of Harvest)

Blog: It all begins with data

 

If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO

 

 

Re: Harvest IFGRP Counters

I was unaware the Detail dashboards existed. Smiley LOL

 

These indeed show a more reasonable overview.

 

grafana_detail_network_util.png

 

It looks like the LACP load balancing works so well, all 10Gb-Links stay below 1% Utilization currently. Thanks for your help!

Re: Harvest IFGRP Counters

Hi @moep

 

I wanted to loop back to you on something I discovered while working on Harvest update this week.  I saw some strangeness with ethernet port counters and after some research learned that internally they were tracked with a 4 byte unsigned int counter.  So that means that every 4096 MB of transfer it would rollover, and when Harvest detects a rollover it skips a poll.  And with a high data rate it could actually rollover multiple times between poll leading to other strange reported transfer rates.  So with a low data rate all appears ok, but a higher one you will see strange behavior. 

 

A bug was created and a fix delivered in cDOT 8.2.4 and CDOT 8.3.2 whereby a new 8 byte counter was added to track port send/recv bytes.  I have adapted Harvest to use this counter and it will be included in the next release (i.e. Harvest 1.2.2 has the issue and > 1.2.2 will not).

 

So in summary there is truth to your initial claim and you shouldn't take the port counters as fact until a fix is available.

 

Cheers,
Chris Madden

Storage Architect, NetApp EMEA (and author of Harvest)

Blog: It all begins with data

 

If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO or both!