Harvest not collecting FabricPool statistics for some nodes of a cluster

GregGuignard · ‎2020-04-24

Harvest is not collecting statistics for FabricPools from some nodes of a cluster but is collecting for other nodes.

We're running Harvest 1.6 in NaBox

NAbox 2.5.1 (2019-09-08)

Debian GNU/Linux 8 (jessie) - NAbox 2.5.1 (8.11)

NetApp SDK v9.7P1 installed
Harvest v1.6 installed
Grafana v6.3.5 installed
Graphite v1.2.0 installed

The Grafana dashboards for Nodes 01-06 report object storage latency, reqs per second, throughput, success/failure count. Data for Nodes 07-10 are either blank or report 'No Data Points'. Latency, for example is from the Graphite tree:

netapp, perf, $Group, $Cluster, node, $Node, fabricpool, average_latency, total-latency

No data is being collected by Harvest for any 'object_store_client_op' with an instance of '0'

# ./perf-counters-utility -host cluster.dns.name -user user -pass secret -in -f object_store_client_op
## Connected successfully to 19.14.1.5 running 97P1
0: node07:kernel:0
0: node08:kernel:0
0: node09:kernel:0
0: node10:kernel:0
1: node02:kernel:1
2: node01:kernel:2
2: node04:kernel:2
2: node05:kernel:2
2: node06:kernel:2
4: node03:kernel:4

These will return the expected metrics for the expected nodes

# ./perf-counters-utility -host cluster.dns.name -user user -pass secret -d -f object_store_client_op -n 2

# ./perf-counters-utility -host cluster.dns.name -user user -pass secret -d -f object_store_client_op -n 4

This returns a syntax error

# ./perf-counters-utility -host cluster.dns.name -user user -pass secret -d -f object_store_client_op -n 0

## Connected successfully to cluster.dns.name running 97P1
ERROR: -d requires the use of -u or -n.

Usage: perf-counters-utility -host <host> -user <user> -pass <pass> [-o|-in|-c|-d] [-f <family>] [-n <name> | -u <uuid>]

This same pattern repeats across multiple clusters, where any non-0 instance data is gathered and instance 0 data is not. Any assistance would be appreciated.

yannb · ‎2020-04-25

@vachagan_gratian, I would take a look at this, but maybe you know off the top of your head ?

Is it possibily a bug in Harvest when -n is zero?

vachagan_gratian · ‎2020-04-28

Hi Greg,

Harvest does not emit counter data if the values are Nulls, most likely this is the case for your node 07-10. For the performance utility: try using uuid instead of name:

# ./perf-counters-utility -host cluster.dns.name  -user user -pass secret -d -f object_store_client_op -u node01:kernel:2

GregGuignard · ‎2020-04-28

@vachagan_gratian Thanks for the reply.

Harvest doesn't report any stats by UUID either ...

# ./perf-counters-utility -host  host -user username -pass password -d -f object_store_client_op -n node07:kernel:0
## Connected successfully to host running 97P1
API call failed with reason: For object_store_client_op object, no instances were found to match the given query

but ONTAP does have stats for the specific object/counter and node/instance pair.

dc1h201::statistics*> show -object object_store_client_op -instance 0 -counter stats -node node07 -raw true

Object: object_store_client_op
Instance: 0
Start-time: 4/28/2020 08:45:50
End-time: 4/28/2020 08:45:50
Scope: node07

    Counter                                                     Value
    -------------------------------- --------------------------------
    stats                                                           -
                           put.total                         71555288
                         put.success                         71555102
                         put.failure                              186
                           get.total                        963444227
                         get.success                        963443696
                         get.failure                              531
                        delete.total                         45992007
                      delete.success                         45992007
                          head.total                            56370
                        head.success                            56370

There are a total of 25 counters in ONTAP for this object and they all have data. I only selected 1 for illustration purposes.

vachagan_gratian · ‎2020-04-28

use the flag -u instead of -n for UUIDs

GregGuignard · ‎2020-04-28

Oops, the '-u node07:kernel:0' does return all the stats.

vachagan_gratian · ‎2020-04-28

are those also non-empty as in the CLI?

GregGuignard · ‎2020-04-28

Yes they are

[stats]=[71555288,71555102,186,963444227,963443696,531,45997783,45997783,0,0,0,0,56370,56370,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]