Re: NetApp-Harvest 1.6 is out!

vachagan_gratian · ‎2019-09-03

Hi Folks,

We have released Harvest 1.6 and it's available for download on ToolChest. There are three major changes in the new release, here's a short description of each of them:

Harvest Extension Manager (HEM)
We have added an entirely new component to Harvest: HEM allows you to customize and extend your poller without the need to touch Harvest source code. You can add your own scripts in any programming language and Harvest will run them during each polling session.

Harvest 1.6 comes with two extensions (NFS connections and SnapMirror replications collectors), but you can write your extension scripts. You might want to do that if you need additional post-processing on already collected metrics or if you want to collect metrics from some other source, or actually for any purpose. And you don't need to start from scratch: we have included in Harvest 1.6 templates for extension scripts in Python, Perl and Bash.

Check the NetApp Harvest Extension Manager 1.6 guideline for more info.

FlexGroup Capacity Metrics
Until now Harvest did not collect capacity metrics for FlexGroup volumes. This was due to ZAPI not providing sufficient information about FlexGroup volumes (e.g. the Cluster to which the volume belonged). Now we use RestAPI to get this missing data. This feature is supported for OCUM systems 9.4 or higher. By default it's disabled: see our Installation and Administration Guide to find out how to activate it.

Kudos to @Zoltan_Somogyvari for putting a lot of time and energy in testing this feature many many times!

AutoSupport Statistics
Harvest will send poller statistics to AutoSupport every month. This new feature is important to us, Harvest developers. As we move forward and want to make the future Harvest more scalable, resource-efficient and robust, and for that we want to have some statistics about the many thousand of Harvest pollers, such as average API runtimes, fails nodes monitored, etc. This feature is by default enabled, but you can disable it in Harvest's configuration file.

We hope you will upgrade your Harvest soon and look forward to your feedback!

bkamil · ‎2019-09-04

Great news Vachagan, thanks a lot!

2 questions on SnapMirror replications collector:

- is there a corresponding Grafana Dashboard included as well?

- does the collector support SVM-DR or only volume-based SnapMirror?

vachagan_gratian · ‎2019-09-04

Hi! Yes, a dashboard is included and the counters are summed up on node-level.

Chandler · ‎2019-10-01

How can I disable the autosupports each month? You mentioned it's enabled by default but can be disabled in the Harvest config file however I'm unable to locate that option in the config file.

Edit: I believe I found the line that needs to be manually added:

send_autosupport_stats = 0

vachagan_gratian · ‎2019-10-04

yes, that will disable it.

DPS_ICS · ‎2019-10-09

Hi,

Thank you for this new version !

I'd like to use new dashboards on Grafan, but I've got an error message with HEM :

[root@server lib]# pwd
/opt/netapp-harvest/lib
[root@server  lib]# ls
DfmErrno.pm  NaElement.pm  NaErrno.pm  NaServer.pm  OCUMAPI.pm  OCUMClassicAPI.pm  Ontap7ModeAPI.pm  OntapClusterAPI.pm  ONTAPILogParser.pm  ONTAPITestContainer.pm  python  SdkEnv.pm  Test.pm
[root@server lib]# /opt/netapp-harvest/extension/snapmirror_replications.py -host XX.XX.XX.XX user netapp-harvest -pass password -port 443 -graphite_host localhost -graphite_port 2003 -graphite_root default -harvest_dir /opt/netapp-harvest/ -v
[2019-10-09 16:06:25,296] [ERROR] [load_prereqs] Failed to import NaServer: No module named NaServer
Traceback (most recent call last):
  File "/opt/netapp-harvest/extension/snapmirror_replications.py", line 618, in <module>
    main()
  File "/opt/netapp-harvest/extension/snapmirror_replications.py", line 89, in main
    load_prereqs()
  File "/opt/netapp-harvest/extension/snapmirror_replications.py", line 213, in load_prereqs
    'available in {}lib/. Exiting'.format(harvest_dir))
NameError: global name 'harvest_dir' is not defined

I'm with Harvest 1.6, SDK 9.5 (tested with 9.6 too), python 2.7 & perl 5. The host is a RHEL 7.7.

Regards,

Arnaud

MartINC · ‎2019-10-17

Hi Arnaud,

I'm having the same issue as you.

NameError: global name 'harvest_dir' is not defined

vachagan_gratian · ‎2019-10-23

Hi @DPS_ICS and @MartINC,

You have a dependency issue. Be aware that if you use extensions in Harvest, they might have their own dependency requirements.

For this specific case, the snapmirror extension requires the Python libraries of the NetApp SDK. You already have the Perl libs from there. You need to get the latests NetApp SDK, unzip, and copy the folder python to /opt/netapp-harvest/lib/. Hope this helps!

TimBol · ‎2020-01-22

Hi,

Any chance of adding the capability to integrate with Prometheus (https://prometheus.io). We're moving away from graphite, but we will keep Grafana, only with a Prometheus backend.

Cheers,

Guus

vachagan_gratian · ‎2020-01-24

Hi Tim,

Yes, we are in the process of integrating Harvest with Prometheus. See also my post here.

israelmmi · ‎2020-02-04

Does the 1.6 harvest support ONTAP 7.0

vachagan_gratian · ‎2020-02-04

Hi. Do you mean 7-mode? If so, 7.3 and onwards releases are supported.

israelmmi · ‎2020-02-04

No, I mean CDOT ONTAP 9.7. Sorry for not being clear.

vachagan_gratian · ‎2020-02-04

yes, supported.

SChristian · ‎2020-02-20

Hello,

I want to create a query, in Grafana, that would only display the volumes hosted by a node of the cluster, is this supported by the new Netapp Harvest 1.6 ?

jcwms1971 · ‎2020-05-08

I've been wanting to do this as well. Have you figured out how to display volumes by node?

SChristian · ‎2020-05-10

Unfortunately no 😞

vachagan_gratian · ‎2020-05-15

Sorry, I missed this question somehow earlier. Harvest creates a summary of volume metrics for each aggregate, for each node. However the actual volume metrics are only emitted under SVM's. This is only done because we don't want to store the same data twice and in the Whisper/Graphite database there is not other way around. I could modify the plugin that does this (/opt/netapp-harvest/plugin/cdot-volume) to store them under the aggregates as well if you don't mind having redundancy in your database, but I probably won't find time until end of next week.

This kind of stuff will be much better solved in Harvest 2.0 that will use Prometheus as its default database (Graphite/Whispler will be supported as well). In Prometheus we will not need to store the same data twice, we will just use labels to associate volumes with SVMs, nodes, aggregates, etc.

jcwms1971 · ‎2020-05-16

Thank you for the detailed response. How long before Harvest 2.0 is released?

Having Volume stats per node or aggregate would be very helpful when trying to balance multi-node clusters.

Personally, I would not mind the duplication in the database to have this functionality now, but that may not be the case for everyone.

If you can work it in before Harvest 2.0, that would be great. If not, I will look forward to the 2.0 release.

Thank you