NetApp Harvest Graphite Issue (Metrics not present in graphite)

raghavp · ‎2019-06-18

Hi,

I setup Netapp Harvest following Administration guide, I can start the pollers and they seem to be sending data to graphite

[2019-06-17 17:02:00] [NORMAL ] [main] Collection of system info from [10.9.0.61] running [NetApp Release 9.5P2] successful.
[2019-06-17 17:02:00] [NORMAL ] [main] Found best-fit monitoring template (older generation or major release): [cdot-9.3.0.conf]
[2019-06-17 17:02:00] [NORMAL ] [main] Added and/or merged monitoring template [/opt/netapp-harvest/template/default/cdot-9.3.0.conf]
[2019-06-17 17:02:01] [NORMAL ] [main] Metrics will be submitted with graphite_root [netapp.perf.perf-test.perf-netapp01]
[2019-06-17 17:02:01] [NORMAL ] [main] Using graphite_meta_metrics_root [netapp.poller.perf.perf-test.perf-netapp01]
[2019-06-17 17:02:01] [NORMAL ] Creating output plugins
[2019-06-17 17:02:01] [NORMAL ] Created output plugins
[2019-06-17 17:02:01] [NORMAL ] [main] Startup complete.  Polling for new data every [60] seconds
[2019-06-17 17:29:51] [DEBUG ] [sysinfo] Cluster name is [perf-netapp01]
[2019-06-17 17:29:52] [DEBUG ] [sysinfo] System info for system_name=[perf-netapp01-01], model=[AFF-A300], cpu=[16], memory=[131072]

Does this mean, metrics are collected correctly but not sent to graphite:

[2019-06-17 17:38:03] [DEBUG  ] M= netapp.poller.perf.perf-test.perf-netapp01.workload_detail_volume.apiTime 0.0698990821838379 1560818283
[2019-06-17 17:38:03] [DEBUG  ] M= netapp.poller.perf.perf-test.perf-netapp01.workload_detail_volume.pluginTime 0.000946998596191406 1560818283
[2019-06-17 17:38:03] [DEBUG  ] Sleeping [57] seconds
[2019-06-17 17:53:00] [WARNING] [cifs:node] Metrics NOT sent to Graphite. Cannot establish/maintain a connection to [10.9.33.208:2003/tcp]: Connection refused

I checked logs inside graphite container but do not see any errors but don't see any data in whisper db, for eg:

22 metrics were recieved by netapp harvest

[2019-06-17 17:38:03] [DEBUG  ] M= netapp.poller.perf.perf-test.perf-netapp01.workload_detail_volume.metricsReceived 22 1560818283
[2019-06-17 17:38:03] [DEBUG  ] M= netapp.poller.perf.perf-test.perf-netapp01.workload_detail_volume.apiTime 0.0698990821838379 1560818283
[2019-06-17 17:38:03] [DEBUG  ] M= netapp.poller.perf.perf-test.perf-netapp01.workload_detail_volume.pluginTime 0.000946998596191406 1560818283
[2019-06-17 17:38:03] [DEBUG  ] Sleeping [57] seconds
[2019-06-17 17:53:00] [WARNING] [cifs:node] Metrics NOT sent to Graphite. Cannot establish/maintain a connection to [10.9.33.208:2003/tcp]: Connection refused

but there are no entries in whisper

/opt/graphite/bin # ./whisper-fetch.py /opt/graphite/storage/whisper/netapp/poller/perf/perf-test/perf-netapp01
/workload_detail_volume/metricsReceived.wsp | grep 15608182
1560818220	None
1560818280	None

Kindly advise, Netapp looks very promising but cannot get the setup working e2e

yannb · ‎2019-06-19

You pasted twice the same log lines, did you mean to post the logs for the graphite container ?

From what I see, Harvest cannot connect to Graphite in your case.

You say it's running in a container, how did you run the container ? Are you sure it's publishing port 2003 ?

raghavp · ‎2019-06-19

My bad for duplicate logs. I was able to resolve the issue, there were 2 problems container running graphite didn't had proper permissions to write to storage volume mapped to the docker host and retention for metrics were not setup correctly in /etc/carbon/storage-schemas.conf (Section Setting frequency and retention in storage-schemas.conf in administrative guide).

After following the above steps, Netapp harvest works like a charm!