Data Infrastructure Management Software Discussions

Highlighted

NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

 

Dear Harvest users,

 

First of all, apologies for not responding to your questions lately, I was too busy but I'll try to get back to unanswered messages during the next week. Here is some good news: we released a Harvest updater to fix some issues in Harvest 1.6 and add some requested features. We don't go for an official release, since that would take a lot (more) time. The updates include:

 

  • Support for SSL authentication in Harvest Extensions
  • Fixing bug in the extension snapmirror_replications.py
  • New extension to collect capacity counters (without OCUM)
  • Caching resolved Graphite hostname (previously sending metrics to Graphite could add pressure on your DNS if caching was not configured in your server/network).

Here is how to run the updater:

  • Download the updater package here
  • Verify MD5 checksum:
$ md5sum harvest_updater_161.tar.gz 
> 1923977dee44366080ca19e724ad4650  harvest_updater_161.tar.gz
  • Unextract the package somewhere on your Harvest server, e.g.:
$ tar -xzvf harvest_updater_161.tar.gz -C /tmp/
  • Stop all harvest pollers
  • Run the updater:
$ cd harvest_updater_161/
$ ./harvest_updater
  • Restart Harvest

The updater adds three Grafana dashboards which you'll need to manually import in the Grafana webgui to use/update them:

/opt/netapp-harvest/grafana/db_netapp-detail-nfs-connections.json
/opt/netapp-harvest/grafana/db_netapp-detail-snapmirror.json
/opt/netapp-harvest/grafana/db_netapp-detail-volume-capacity.json

 

Reversing the update. Before the updater changes any files, it will create a backup in /opt/netapp-harvest/backup/harvest_updater_16100/, so if something goes wrong, you can reverse the update by:

$ ./harvest_updater --reverse

 

Second of all, many of you are asking about Harvest 2.0 and about replacing Graphite. We are well aware of the scalability issues of Graphite, and while we will continue supporting Graphite, our main backend in Harvest 2.0 will be (most likely) Prometheus. Unfortunately I can't give estimation of a release date, since at the moment we are trying to get more manpower behind this project, but I can tell you that Harvest 2.0 is our main focus at the moment.

 

Finally if by any chance you have written a Python module to send performance metrics to Prometheus and you want to contribute to an open-source project (Harvest 2.0 will be on Github!), please get in touch with me.

 

Cheers,

 

Vachagan

35 REPLIES 35
Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

I understand Harvest 2.0 is a WIP and adopting Prometheus as a backend is a very welcome news!

 

I do have two questions about Harvest 2.0:

 

1. Will Harvest and/or the Prometheus exporter be available as a docker image? This would help in deploying it in Kubernetes

2. Will there be flexibility in having Harvest 2.0's exporter expose metrics to another Prometheus instance?

 

Appreciate your focus on getting Harvest 2.0 happen.

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

Will you release an updated RPM for 1.6.1? This hacky patching approach is really bad when it comes to version management.

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

Hi Kumar,

 

1. Yes, that's on our schedule

2. Harvest 2.0 will architecture will allow to send each metric object to different (and multiple) DB instances. We consider also integrating with Thalos.

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

At some point we will do that, but I am not sure when exactly, because usually it takes a lot time to issue an official release.

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

Awesome!  Thanks for the update! 

 

I had a successful update but I'm getting Permission denied when I update the dashboards.  Any ideas or did I miss a step? 

 

[root@HOST harvest_updater_161]# /opt/netapp-harvest/grafana/db_netapp-detail-nfs-connections.json
-bash: /opt/netapp-harvest/grafana/db_netapp-detail-nfs-connections.json: Permission denied

[root@HOST harvest_updater_161]# /opt/netapp-harvest/grafana/db_netapp-detail-snapmirror.json
-bash: /opt/netapp-harvest/grafana/db_netapp-detail-snapmirror.json: Permission denied

 

I noticed for the snapmiorror metric, it's trying to pull from my graphite server this metric but it doesn't exist: netapp.perf.$Group.$Cluster.node.$Node.snapmirror.src.*   I have 10 CDOT clusters ranging from 9.3.P6 to 9.5 and that snapmirror metric doesn't exist.  

 

 

 

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0


@vachagan_gratian wrote:

Hi Kumar,

 

1. Yes, that's on our schedule

2. Harvest 2.0 will architecture will allow to send each metric object to different (and multiple) DB instances. We consider also integrating with Thalos.


Thank you Vachagan!

 

To clarify, did you mean Thanos rather than Thalos?

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

yes, I meant thanos!

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

Hi, it seems like you are trying to execute the json files, which have only read permissions (and write for owner):

 

$ ls -l /opt/netapp-harvest/grafana/         
...
-rw-r--r-- 1 netapp-harvest netapp-harvest 6825 Jan 24 13:25 db_netapp-detail-nfs-connections.json
-rw-r--r-- 1 netapp-harvest netapp-harvest 7584 Jan 24 13:25 db_netapp-detail-snapmirror.json
-rw-r--r-- 1 netapp-harvest netapp-harvest 19602 Jan 24 13:25 db_netapp-detail-volume-capacity.json ...

You can either update the dashboards manually in the Grafana webgui (create -> import in main menu), or run the -import argument of netapp-manager, but you should be aware that this would overwrite anything that you might have changed in your dashboards after the install.

 

$opt/netapp-harvest.netapp-manager -import
[OK     ] Will import dashboards to [GRAFANA_SERVER]
[OK     ] Dashboard directory is [/opt/netapp-harvest/grafana]
[OK     ] Imported dashboard [db_netapp-detail-7-mode-lun.json] successfully
[OK     ] Imported dashboard [db_netapp-dashboard-cluster.json] successfully
...

If that's the case and you don't want to overwrite changes, you can temporarily remove the json's that you don't want to update from /opt/netapp-harvest/grafana/ then run the same import command.

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

As for the metrics not existing, have you activated the extension? I would check the logs of the extension (/opt/netapp-harvest/log/CLUSTER_netapp-harvest_snapmirror_replications.log) and if there are no messages, try to run the extension in foreground mode in verbose:

 

$ ./extension/snapmirror_replications.py -host <HOSTNAME> -user <USERNAME> -pass <PASSWORD> -v

or:

$ ./extension/snapmirror_replications.py -host <HOSTNAME> -auth_type ssl_cert -ssl_cert <SSL_CERT_FILENAME> -ssl_key <SSL_KEY_FILENAME> -v

 

Try the NEW Support Site!
NetApp Support Site
Forums