NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

[vachagan_gratian]

Dear Harvest users,

First of all, apologies for not responding to your questions lately, I was too busy but I'll try to get back to unanswered messages during the next week. Here is some good news: we released a Harvest updater to fix some issues in Harvest 1.6 and add some requested features. We don't go for an official release, since that would take a lot (more) time. The updates include:

• Support for SSL authentication in Harvest Extensions
• Fixing bug in the extension snapmirror_replications.py
• New extension to collect capacity counters (without OCUM)
• Caching resolved Graphite hostname (previously sending metrics to Graphite could add pressure on your DNS if caching was not configured in your server/network).

Here is how to run the updater:

• Download the updater package here
• Verify MD5 checksum:

$ md5sum harvest_updater_161.tar.gz 
> 1923977dee44366080ca19e724ad4650  harvest_updater_161.tar.gz

• Unextract the package somewhere on your Harvest server, e.g.:

$ tar -xzvf harvest_updater_161.tar.gz -C /tmp/

• Stop all harvest pollers
• Run the updater:

$ cd harvest_updater_161/
$ ./harvest_updater

• Restart Harvest

The updater adds three Grafana dashboards which you'll need to manually import in the Grafana webgui to use/update them:

/opt/netapp-harvest/grafana/db_netapp-detail-nfs-connections.json
/opt/netapp-harvest/grafana/db_netapp-detail-snapmirror.json
/opt/netapp-harvest/grafana/db_netapp-detail-volume-capacity.json

Reversing the update. Before the updater changes any files, it will create a backup in /opt/netapp-harvest/backup/harvest_updater_16100/, so if something goes wrong, you can reverse the update by:

$ ./harvest_updater --reverse

Second of all, many of you are asking about Harvest 2.0 and about replacing Graphite. We are well aware of the scalability issues of Graphite, and while we will continue supporting Graphite, our main backend in Harvest 2.0 will be (most likely) Prometheus. Unfortunately I can't give estimation of a release date, since at the moment we are trying to get more manpower behind this project, but I can tell you that Harvest 2.0 is our main focus at the moment.

Finally if by any chance you have written a Python module to send performance metrics to Prometheus and you want to contribute to an open-source project (Harvest 2.0 will be on Github!), please get in touch with me.

Cheers,

Vachagan

[KumarChalla]

I understand Harvest 2.0 is a WIP and adopting Prometheus as a backend is a very welcome news!

I do have two questions about Harvest 2.0:

1. Will Harvest and/or the Prometheus exporter be available as a docker image? This would help in deploying it in Kubernetes

2. Will there be flexibility in having Harvest 2.0's exporter expose metrics to another Prometheus instance?

Appreciate your focus on getting Harvest 2.0 happen.

[vachagan_gratian]

Hi Kumar,

1. Yes, that's on our schedule

2. Harvest 2.0 will architecture will allow to send each metric object to different (and multiple) DB instances. We consider also integrating with Thalos.

[KumarChalla]

---

@vachagan_gratian wrote:

Hi Kumar,

1. Yes, that's on our schedule

2. Harvest 2.0 will architecture will allow to send each metric object to different (and multiple) DB instances. We consider also integrating with Thalos.

---

Thank you Vachagan!

To clarify, did you mean Thanos rather than Thalos?

[vachagan_gratian]

yes, I meant thanos!

[moep]

Will you release an updated RPM for 1.6.1? This hacky patching approach is really bad when it comes to version management.

[vachagan_gratian]

At some point we will do that, but I am not sure when exactly, because usually it takes a lot time to issue an official release.

[Cbrown]

Awesome!  Thanks for the update! 

I had a successful update but I'm getting Permission denied when I update the dashboards.  Any ideas or did I miss a step? 

[root@HOST harvest_updater_161]# /opt/netapp-harvest/grafana/db_netapp-detail-nfs-connections.json
-bash: /opt/netapp-harvest/grafana/db_netapp-detail-nfs-connections.json: Permission denied

[root@HOST harvest_updater_161]# /opt/netapp-harvest/grafana/db_netapp-detail-snapmirror.json
-bash: /opt/netapp-harvest/grafana/db_netapp-detail-snapmirror.json: Permission denied

I noticed for the snapmiorror metric, it's trying to pull from my graphite server this metric but it doesn't exist: netapp.perf.$Group.$Cluster.node.$Node.snapmirror.src.*   I have 10 CDOT clusters ranging from 9.3.P6 to 9.5 and that snapmirror metric doesn't exist.  

[vachagan_gratian]

Hi, it seems like you are trying to execute the json files, which have only read permissions (and write for owner):

$ ls -l /opt/netapp-harvest/grafana/         
...
-rw-r--r-- 1 netapp-harvest netapp-harvest 6825 Jan 24 13:25 db_netapp-detail-nfs-connections.json
-rw-r--r-- 1 netapp-harvest netapp-harvest 7584 Jan 24 13:25 db_netapp-detail-snapmirror.json
-rw-r--r-- 1 netapp-harvest netapp-harvest 19602 Jan 24 13:25 db_netapp-detail-volume-capacity.json
...

You can either update the dashboards manually in the Grafana webgui (create -> import in main menu), or run the -import argument of netapp-manager, but you should be aware that this would overwrite anything that you might have changed in your dashboards after the install.

$opt/netapp-harvest.netapp-manager -import
[OK     ] Will import dashboards to [GRAFANA_SERVER]
[OK     ] Dashboard directory is [/opt/netapp-harvest/grafana]
[OK     ] Imported dashboard [db_netapp-detail-7-mode-lun.json] successfully
[OK     ] Imported dashboard [db_netapp-dashboard-cluster.json] successfully
...

If that's the case and you don't want to overwrite changes, you can temporarily remove the json's that you don't want to update from /opt/netapp-harvest/grafana/ then run the same import command.

[vachagan_gratian]

As for the metrics not existing, have you activated the extension? I would check the logs of the extension (/opt/netapp-harvest/log/CLUSTER_netapp-harvest_snapmirror_replications.log) and if there are no messages, try to run the extension in foreground mode in verbose:

$ ./extension/snapmirror_replications.py -host <HOSTNAME> -user <USERNAME> -pass <PASSWORD> -v

or:

$ ./extension/snapmirror_replications.py -host <HOSTNAME> -auth_type ssl_cert -ssl_cert <SSL_CERT_FILENAME> -ssl_key <SSL_KEY_FILENAME> -v

[israelmmi]

When running the test command I get the following error:

[2020-02-04 10:05:25,763] [ERROR] [connect_zapi] Failed to import NaServer: No module named NaServer
[2020-02-04 10:05:25,763] [ERROR] [connect_zapi] Make sure NetApp SDK python package is available in /opt/netapp-harvest/lib/. Exiting

[vachagan_gratian]

For this extension you need to manually install the NMSDK Python library. Here's how:

• download NMSDK from here,
• unextract netapp-manageability-sdk-*.zip
• copy  contents of /netapp-manageability-sdk-*/lib/python/NetApp/ to /opt/netapp-harvest/lib/python/

[israelmmi]

That helped. Now I get this error:

[ERROR] [poll_snapmirrors] ZAPI request failed: Insufficient privileges: user 'netapp-harvest' does not have read access to this resource

[vachagan_gratian]

you'll need to extend the user privileges, see explained here under 4.

[bugfinder]

I've played a bit with the idea about the volume capacity extension. I have to admit my love for python is not that deep, so I started by rewriting the new volume_capacity_counters.py in perl. Having done that, I added a little code to

add the vol_summary data as well to get the summary per SVM. And as things were so nice, I did the similar thing for

aggregates.

and a one-liner for netapp-worker to clean up the zombies:

--- a/netapp-worker
+++ b/netapp-worker
@@ -605,6 +605,9 @@ while (1)
# Check and rotate logfile if needed
check_and_rotate_logfile();

+ # clean up zombies
+ while ( waitpid ( -1, POSIX::WNOHANG ) >0 ) {};
+
my $sleep_time = ($counter_nextrun - time());
if ($sleep_time > 0)
{

the perl variant for volume_capacity_counters.pl will be attached, you'll probably have to adapt line 83/84 to

use the files from the SDK.

[bugfinder]

okay,  where to post files as extension to this ?

the terms of use tell me to not attach software files ...

I'd have extension/volume_capacity_counters.pl and extension/aggregate_capacity_counters.pl

I'm willing to freely share, but where to put them...

[bugfinder]

for the moment I have my extension scripts at https://users.suse.com/~ro/NetApp/index.html but this is just a temporary url.

[vachagan_gratian]

Hi bugfinder,

Sorry for such a late reply. I appreciate very much your willingness to share your work (and I share your sympathy for Perl!) Is it okay if I get back to you when we do our next release? We'll be happy to include your extension in our package (just make sure to add your name as author)!

[NEILHOLMES]

Hi,

I appreciate the v2.0 has been released, but it would be really useful if you could post a link for Harvest 1.6.1 updater again please. This would help be achieve my immediate goal before we look to migrate to v2.0 in the future.

[israelmmi]

Hi,

I fixed most of the things and when running the script with the "-v" switch I get:

Skipping send: no graphite root defined

Could that explain why I see no data? Where do I need to define my server?

I see in the files there are parameters:

 
REQUIRED:
_HARVEST_HOSTNAME - hostname/IP of ONTAP system
_HARVEST_AUTH_TYPE - either "password" (default) or "ssl_cert"
_HARVEST_CLUSTER - Name of ONTAP Cluster
_HARVEST_GRAPHITE_ROOT - prefix of metric path in Graphite
(e.g.: 'netapp.capacity.GROUP.CLUSTER'),
if prefix is "_", data will be collected
but not sent to Graphite.

Do I need to define them in each script? They don't inherit the info from the "old" harvest?

Also, do i need to schedule the scripts to be run separately or will they be run automatically by the "old" Harvest? I already ran the harvest script: 

./harvest_updater

[bugfinder]

basically I did run the plugins for testing with args like this:

/opt/netapp-harvest/extension/volume_capacity_counters.py -v -cluster mycluster -host myclusterhostname -user myuser -pass 'mypassword' -graphite_root 'netapp.capacity.mygroup.mycluster'

if called via the pre-post-exec plugin, these values are passed via the environment, but if you call the extension directly, you have to specify all these.