Active IQ Unified Manager Discussions

NetApp-Harvest 1.6.1 updater + some news about Harvest 2.0

vachagan_gratian
30,870 Views

 

Dear Harvest users,

 

First of all, apologies for not responding to your questions lately, I was too busy but I'll try to get back to unanswered messages during the next week. Here is some good news: we released a Harvest updater to fix some issues in Harvest 1.6 and add some requested features. We don't go for an official release, since that would take a lot (more) time. The updates include:

 

  • Support for SSL authentication in Harvest Extensions
  • Fixing bug in the extension snapmirror_replications.py
  • New extension to collect capacity counters (without OCUM)
  • Caching resolved Graphite hostname (previously sending metrics to Graphite could add pressure on your DNS if caching was not configured in your server/network).

Here is how to run the updater:

  • Download the updater package here
  • Verify MD5 checksum:
$ md5sum harvest_updater_161.tar.gz 
> 1923977dee44366080ca19e724ad4650  harvest_updater_161.tar.gz
  • Unextract the package somewhere on your Harvest server, e.g.:
$ tar -xzvf harvest_updater_161.tar.gz -C /tmp/
  • Stop all harvest pollers
  • Run the updater:
$ cd harvest_updater_161/
$ ./harvest_updater
  • Restart Harvest

The updater adds three Grafana dashboards which you'll need to manually import in the Grafana webgui to use/update them:

/opt/netapp-harvest/grafana/db_netapp-detail-nfs-connections.json
/opt/netapp-harvest/grafana/db_netapp-detail-snapmirror.json
/opt/netapp-harvest/grafana/db_netapp-detail-volume-capacity.json

 

Reversing the update. Before the updater changes any files, it will create a backup in /opt/netapp-harvest/backup/harvest_updater_16100/, so if something goes wrong, you can reverse the update by:

$ ./harvest_updater --reverse

 

Second of all, many of you are asking about Harvest 2.0 and about replacing Graphite. We are well aware of the scalability issues of Graphite, and while we will continue supporting Graphite, our main backend in Harvest 2.0 will be (most likely) Prometheus. Unfortunately I can't give estimation of a release date, since at the moment we are trying to get more manpower behind this project, but I can tell you that Harvest 2.0 is our main focus at the moment.

 

Finally if by any chance you have written a Python module to send performance metrics to Prometheus and you want to contribute to an open-source project (Harvest 2.0 will be on Github!), please get in touch with me.

 

Cheers,

 

Vachagan

52 REPLIES 52

KumarChalla
19,821 Views

I understand Harvest 2.0 is a WIP and adopting Prometheus as a backend is a very welcome news!

 

I do have two questions about Harvest 2.0:

 

1. Will Harvest and/or the Prometheus exporter be available as a docker image? This would help in deploying it in Kubernetes

2. Will there be flexibility in having Harvest 2.0's exporter expose metrics to another Prometheus instance?

 

Appreciate your focus on getting Harvest 2.0 happen.

vachagan_gratian
19,592 Views

Hi Kumar,

 

1. Yes, that's on our schedule

2. Harvest 2.0 will architecture will allow to send each metric object to different (and multiple) DB instances. We consider also integrating with Thalos.

KumarChalla
19,469 Views

@vachagan_gratian wrote:

Hi Kumar,

 

1. Yes, that's on our schedule

2. Harvest 2.0 will architecture will allow to send each metric object to different (and multiple) DB instances. We consider also integrating with Thalos.


Thank you Vachagan!

 

To clarify, did you mean Thanos rather than Thalos?

vachagan_gratian
19,432 Views

yes, I meant thanos!

moep
19,623 Views

Will you release an updated RPM for 1.6.1? This hacky patching approach is really bad when it comes to version management.

vachagan_gratian
19,589 Views

At some point we will do that, but I am not sure when exactly, because usually it takes a lot time to issue an official release.

Cbrown
19,738 Views

Awesome!  Thanks for the update! 

 

I had a successful update but I'm getting Permission denied when I update the dashboards.  Any ideas or did I miss a step? 

 

[root@HOST harvest_updater_161]# /opt/netapp-harvest/grafana/db_netapp-detail-nfs-connections.json
-bash: /opt/netapp-harvest/grafana/db_netapp-detail-nfs-connections.json: Permission denied

[root@HOST harvest_updater_161]# /opt/netapp-harvest/grafana/db_netapp-detail-snapmirror.json
-bash: /opt/netapp-harvest/grafana/db_netapp-detail-snapmirror.json: Permission denied

 

I noticed for the snapmiorror metric, it's trying to pull from my graphite server this metric but it doesn't exist: netapp.perf.$Group.$Cluster.node.$Node.snapmirror.src.*   I have 10 CDOT clusters ranging from 9.3.P6 to 9.5 and that snapmirror metric doesn't exist.  

 

 

 

vachagan_gratian
19,429 Views

Hi, it seems like you are trying to execute the json files, which have only read permissions (and write for owner):

 

$ ls -l /opt/netapp-harvest/grafana/         
...
-rw-r--r-- 1 netapp-harvest netapp-harvest 6825 Jan 24 13:25 db_netapp-detail-nfs-connections.json
-rw-r--r-- 1 netapp-harvest netapp-harvest 7584 Jan 24 13:25 db_netapp-detail-snapmirror.json
-rw-r--r-- 1 netapp-harvest netapp-harvest 19602 Jan 24 13:25 db_netapp-detail-volume-capacity.json ...

You can either update the dashboards manually in the Grafana webgui (create -> import in main menu), or run the -import argument of netapp-manager, but you should be aware that this would overwrite anything that you might have changed in your dashboards after the install.

 

$opt/netapp-harvest.netapp-manager -import
[OK     ] Will import dashboards to [GRAFANA_SERVER]
[OK     ] Dashboard directory is [/opt/netapp-harvest/grafana]
[OK     ] Imported dashboard [db_netapp-detail-7-mode-lun.json] successfully
[OK     ] Imported dashboard [db_netapp-dashboard-cluster.json] successfully
...

If that's the case and you don't want to overwrite changes, you can temporarily remove the json's that you don't want to update from /opt/netapp-harvest/grafana/ then run the same import command.

vachagan_gratian
19,603 Views

As for the metrics not existing, have you activated the extension? I would check the logs of the extension (/opt/netapp-harvest/log/CLUSTER_netapp-harvest_snapmirror_replications.log) and if there are no messages, try to run the extension in foreground mode in verbose:

 

$ ./extension/snapmirror_replications.py -host <HOSTNAME> -user <USERNAME> -pass <PASSWORD> -v

or:

$ ./extension/snapmirror_replications.py -host <HOSTNAME> -auth_type ssl_cert -ssl_cert <SSL_CERT_FILENAME> -ssl_key <SSL_KEY_FILENAME> -v

 

israelmmi
18,137 Views

When running the test command I get the following error:


[2020-02-04 10:05:25,763] [ERROR] [connect_zapi] Failed to import NaServer: No module named NaServer
[2020-02-04 10:05:25,763] [ERROR] [connect_zapi] Make sure NetApp SDK python package is available in /opt/netapp-harvest/lib/. Exiting

 

vachagan_gratian
18,124 Views

For this extension you need to manually install the NMSDK Python library. Here's how:

  • download NMSDK from here,
  • unextract netapp-manageability-sdk-*.zip
  • copy  contents of /netapp-manageability-sdk-*/lib/python/NetApp/ to /opt/netapp-harvest/lib/python/

israelmmi
18,122 Views

That helped. Now I get this error:

 

[ERROR] [poll_snapmirrors] ZAPI request failed: Insufficient privileges: user 'netapp-harvest' does not have read access to this resource

 

 

vachagan_gratian
18,115 Views

you'll need to extend the user privileges, see explained here under 4.

bugfinder
18,013 Views

I've played a bit with the idea about the volume capacity extension. I have to admit my love for python is not that deep, so I started by rewriting the new volume_capacity_counters.py in perl. Having done that, I added a little code to

add the vol_summary data as well to get the summary per SVM. And as things were so nice, I did the similar thing for

aggregates.

and a one-liner for netapp-worker to clean up the zombies:

--- a/netapp-worker
+++ b/netapp-worker
@@ -605,6 +605,9 @@ while (1)
# Check and rotate logfile if needed
check_and_rotate_logfile();

+ # clean up zombies
+ while ( waitpid ( -1, POSIX::WNOHANG ) >0 ) {};
+
my $sleep_time = ($counter_nextrun - time());
if ($sleep_time > 0)
{

 

the perl variant for volume_capacity_counters.pl will be attached, you'll probably have to adapt line 83/84 to

use the files from the SDK.

 

bugfinder
18,010 Views

okay,  where to post files as extension to this ?

the terms of use tell me to not attach software files ...

I'd have extension/volume_capacity_counters.pl and extension/aggregate_capacity_counters.pl

I'm willing to freely share, but where to put them...

bugfinder
13,560 Views

for the moment I have my extension scripts at https://users.suse.com/~ro/NetApp/index.html but this is just a temporary url.

vachagan_gratian
13,174 Views

Hi bugfinder,

 

Sorry for such a late reply. I appreciate very much your willingness to share your work (and I share your sympathy for Perl!) Is it okay if I get back to you when we do our next release? We'll be happy to include your extension in our package (just make sure to add your name as author)!

NEILHOLMES
9,415 Views

Hi,

I appreciate the v2.0 has been released, but it would be really useful if you could post a link for Harvest 1.6.1 updater again please. This would help be achieve my immediate goal before we look to migrate to v2.0 in the future.

israelmmi
13,792 Views

Hi,

 

I fixed most of the things and when running the script with the "-v" switch I get:

Skipping send: no graphite root defined

 

Could that explain why I see no data? Where do I need to define my server?

 

I see in the files there are parameters:

 
REQUIRED:
_HARVEST_HOSTNAME - hostname/IP of ONTAP system
_HARVEST_AUTH_TYPE - either "password" (default) or "ssl_cert"
_HARVEST_CLUSTER - Name of ONTAP Cluster
_HARVEST_GRAPHITE_ROOT - prefix of metric path in Graphite
(e.g.: 'netapp.capacity.GROUP.CLUSTER'),
if prefix is "_", data will be collected
but not sent to Graphite.

 

Do I need to define them in each script? They don't inherit the info from the "old" harvest?

 

Also, do i need to schedule the scripts to be run separately or will they be run automatically by the "old" Harvest? I already ran the harvest script: 

./harvest_updater

bugfinder
13,777 Views

basically I did run the plugins for testing with args like this:

/opt/netapp-harvest/extension/volume_capacity_counters.py -v -cluster mycluster -host myclusterhostname -user myuser -pass 'mypassword' -graphite_root 'netapp.capacity.mygroup.mycluster'

 

if called via the pre-post-exec plugin, these values are passed via the environment, but if you call the extension directly, you have to specify all these.

Public