Data Infrastructure Management Software Discussions

Highlighted

NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

 

Dear Harvest users,

 

First of all, apologies for not responding to your questions lately, I was too busy but I'll try to get back to unanswered messages during the next week. Here is some good news: we released a Harvest updater to fix some issues in Harvest 1.6 and add some requested features. We don't go for an official release, since that would take a lot (more) time. The updates include:

 

  • Support for SSL authentication in Harvest Extensions
  • Fixing bug in the extension snapmirror_replications.py
  • New extension to collect capacity counters (without OCUM)
  • Caching resolved Graphite hostname (previously sending metrics to Graphite could add pressure on your DNS if caching was not configured in your server/network).

Here is how to run the updater:

  • Download the updater package here
  • Verify MD5 checksum:
$ md5sum harvest_updater_161.tar.gz 
> 1923977dee44366080ca19e724ad4650  harvest_updater_161.tar.gz
  • Unextract the package somewhere on your Harvest server, e.g.:
$ tar -xzvf harvest_updater_161.tar.gz -C /tmp/
  • Stop all harvest pollers
  • Run the updater:
$ cd harvest_updater_161/
$ ./harvest_updater
  • Restart Harvest

The updater adds three Grafana dashboards which you'll need to manually import in the Grafana webgui to use/update them:

/opt/netapp-harvest/grafana/db_netapp-detail-nfs-connections.json
/opt/netapp-harvest/grafana/db_netapp-detail-snapmirror.json
/opt/netapp-harvest/grafana/db_netapp-detail-volume-capacity.json

 

Reversing the update. Before the updater changes any files, it will create a backup in /opt/netapp-harvest/backup/harvest_updater_16100/, so if something goes wrong, you can reverse the update by:

$ ./harvest_updater --reverse

 

Second of all, many of you are asking about Harvest 2.0 and about replacing Graphite. We are well aware of the scalability issues of Graphite, and while we will continue supporting Graphite, our main backend in Harvest 2.0 will be (most likely) Prometheus. Unfortunately I can't give estimation of a release date, since at the moment we are trying to get more manpower behind this project, but I can tell you that Harvest 2.0 is our main focus at the moment.

 

Finally if by any chance you have written a Python module to send performance metrics to Prometheus and you want to contribute to an open-source project (Harvest 2.0 will be on Github!), please get in touch with me.

 

Cheers,

 

Vachagan

40 REPLIES 40
Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

I understand Harvest 2.0 is a WIP and adopting Prometheus as a backend is a very welcome news!

 

I do have two questions about Harvest 2.0:

 

1. Will Harvest and/or the Prometheus exporter be available as a docker image? This would help in deploying it in Kubernetes

2. Will there be flexibility in having Harvest 2.0's exporter expose metrics to another Prometheus instance?

 

Appreciate your focus on getting Harvest 2.0 happen.

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

Will you release an updated RPM for 1.6.1? This hacky patching approach is really bad when it comes to version management.

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

Hi Kumar,

 

1. Yes, that's on our schedule

2. Harvest 2.0 will architecture will allow to send each metric object to different (and multiple) DB instances. We consider also integrating with Thalos.

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

At some point we will do that, but I am not sure when exactly, because usually it takes a lot time to issue an official release.

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

Awesome!  Thanks for the update! 

 

I had a successful update but I'm getting Permission denied when I update the dashboards.  Any ideas or did I miss a step? 

 

[root@HOST harvest_updater_161]# /opt/netapp-harvest/grafana/db_netapp-detail-nfs-connections.json
-bash: /opt/netapp-harvest/grafana/db_netapp-detail-nfs-connections.json: Permission denied

[root@HOST harvest_updater_161]# /opt/netapp-harvest/grafana/db_netapp-detail-snapmirror.json
-bash: /opt/netapp-harvest/grafana/db_netapp-detail-snapmirror.json: Permission denied

 

I noticed for the snapmiorror metric, it's trying to pull from my graphite server this metric but it doesn't exist: netapp.perf.$Group.$Cluster.node.$Node.snapmirror.src.*   I have 10 CDOT clusters ranging from 9.3.P6 to 9.5 and that snapmirror metric doesn't exist.  

 

 

 

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0


@vachagan_gratian wrote:

Hi Kumar,

 

1. Yes, that's on our schedule

2. Harvest 2.0 will architecture will allow to send each metric object to different (and multiple) DB instances. We consider also integrating with Thalos.


Thank you Vachagan!

 

To clarify, did you mean Thanos rather than Thalos?

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

yes, I meant thanos!

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

Hi, it seems like you are trying to execute the json files, which have only read permissions (and write for owner):

 

$ ls -l /opt/netapp-harvest/grafana/         
...
-rw-r--r-- 1 netapp-harvest netapp-harvest 6825 Jan 24 13:25 db_netapp-detail-nfs-connections.json
-rw-r--r-- 1 netapp-harvest netapp-harvest 7584 Jan 24 13:25 db_netapp-detail-snapmirror.json
-rw-r--r-- 1 netapp-harvest netapp-harvest 19602 Jan 24 13:25 db_netapp-detail-volume-capacity.json ...

You can either update the dashboards manually in the Grafana webgui (create -> import in main menu), or run the -import argument of netapp-manager, but you should be aware that this would overwrite anything that you might have changed in your dashboards after the install.

 

$opt/netapp-harvest.netapp-manager -import
[OK     ] Will import dashboards to [GRAFANA_SERVER]
[OK     ] Dashboard directory is [/opt/netapp-harvest/grafana]
[OK     ] Imported dashboard [db_netapp-detail-7-mode-lun.json] successfully
[OK     ] Imported dashboard [db_netapp-dashboard-cluster.json] successfully
...

If that's the case and you don't want to overwrite changes, you can temporarily remove the json's that you don't want to update from /opt/netapp-harvest/grafana/ then run the same import command.

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

As for the metrics not existing, have you activated the extension? I would check the logs of the extension (/opt/netapp-harvest/log/CLUSTER_netapp-harvest_snapmirror_replications.log) and if there are no messages, try to run the extension in foreground mode in verbose:

 

$ ./extension/snapmirror_replications.py -host <HOSTNAME> -user <USERNAME> -pass <PASSWORD> -v

or:

$ ./extension/snapmirror_replications.py -host <HOSTNAME> -auth_type ssl_cert -ssl_cert <SSL_CERT_FILENAME> -ssl_key <SSL_KEY_FILENAME> -v

 

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

When running the test command I get the following error:


[2020-02-04 10:05:25,763] [ERROR] [connect_zapi] Failed to import NaServer: No module named NaServer
[2020-02-04 10:05:25,763] [ERROR] [connect_zapi] Make sure NetApp SDK python package is available in /opt/netapp-harvest/lib/. Exiting

 

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

For this extension you need to manually install the NMSDK Python library. Here's how:

  • download NMSDK from here,
  • unextract netapp-manageability-sdk-*.zip
  • copy  contents of /netapp-manageability-sdk-*/lib/python/NetApp/ to /opt/netapp-harvest/lib/python/
Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

That helped. Now I get this error:

 

[ERROR] [poll_snapmirrors] ZAPI request failed: Insufficient privileges: user 'netapp-harvest' does not have read access to this resource

 

 

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

you'll need to extend the user privileges, see explained here under 4.

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

I've played a bit with the idea about the volume capacity extension. I have to admit my love for python is not that deep, so I started by rewriting the new volume_capacity_counters.py in perl. Having done that, I added a little code to

add the vol_summary data as well to get the summary per SVM. And as things were so nice, I did the similar thing for

aggregates.

and a one-liner for netapp-worker to clean up the zombies:

--- a/netapp-worker
+++ b/netapp-worker
@@ -605,6 +605,9 @@ while (1)
# Check and rotate logfile if needed
check_and_rotate_logfile();

+ # clean up zombies
+ while ( waitpid ( -1, POSIX::WNOHANG ) >0 ) {};
+
my $sleep_time = ($counter_nextrun - time());
if ($sleep_time > 0)
{

 

the perl variant for volume_capacity_counters.pl will be attached, you'll probably have to adapt line 83/84 to

use the files from the SDK.

 

Highlighted

Re: NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

okay,  where to post files as extension to this ?

the terms of use tell me to not attach software files ...

I'd have extension/volume_capacity_counters.pl and extension/aggregate_capacity_counters.pl

I'm willing to freely share, but where to put them...

Try the NEW Knowledgebase!
NetApp KB Site
Forums