Dear Harvest users,
First of all, apologies for not responding to your questions lately, I was too busy but I'll try to get back to unanswered messages during the next week. Here is some good news: we released a Harvest updater to fix some issues in Harvest 1.6 and add some requested features. We don't go for an official release, since that would take a lot (more) time. The updates include:
Here is how to run the updater:
$ md5sum harvest_updater_161.tar.gz > 1923977dee44366080ca19e724ad4650 harvest_updater_161.tar.gz
$ tar -xzvf harvest_updater_161.tar.gz -C /tmp/
$ cd harvest_updater_161/ $ ./harvest_updater
The updater adds three Grafana dashboards which you'll need to manually import in the Grafana webgui to use/update them:
/opt/netapp-harvest/grafana/db_netapp-detail-nfs-connections.json /opt/netapp-harvest/grafana/db_netapp-detail-snapmirror.json /opt/netapp-harvest/grafana/db_netapp-detail-volume-capacity.json
Reversing the update. Before the updater changes any files, it will create a backup in /opt/netapp-harvest/backup/harvest_updater_16100/, so if something goes wrong, you can reverse the update by:
$ ./harvest_updater --reverse
Second of all, many of you are asking about Harvest 2.0 and about replacing Graphite. We are well aware of the scalability issues of Graphite, and while we will continue supporting Graphite, our main backend in Harvest 2.0 will be (most likely) Prometheus. Unfortunately I can't give estimation of a release date, since at the moment we are trying to get more manpower behind this project, but I can tell you that Harvest 2.0 is our main focus at the moment.
Finally if by any chance you have written a Python module to send performance metrics to Prometheus and you want to contribute to an open-source project (Harvest 2.0 will be on Github!), please get in touch with me.
the basic structure of the setup for the harvest extensions is explained in docs/NetApp_Harvest_Extension_Manager_1.6.pdf
on page 5 of 9 it talks about the extensions.conf file (probably meant extension.conf, but whatever ...)
basically at the end of the main config netapp-harvest.conf, in the section for your cluster you can have a line
"template = default,extensions.conf"
then in template/extensions.conf you can have the structure as explained in the howto cat uses the pre-post-exec plugin
to call scripts from the "command_list" variable and thus collect additional data.
I have "auth_type = password" in configuration, but still getting:
[WARNING] [connect_zapi] Failed to connect to NaServer: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)
For the snapmirror extension.
It seems like SVM-DR SnapMirror relationships are not supported by the SnapMirror extension and only Volume SnapMirror is supported? I'm getting a lot of this in the log file (we're purely an SVM-DR shop):
[2020-03-12 16:49:27,914] [WARNING] [poll_snapmirrors] Destination relationship with no node or volume. Skipping [2020-03-12 16:49:27,914] [WARNING] [poll_snapmirrors] Source relationship with no volume. Skipping
Yes, that's exactly the cas.
The reason why we skip them now is that we want to summarize SnapMirrors relationships per node (to compare that against the active replications limit of each node). We can't associate an SVM(-DR)s to a Node, but we can sum up SnapMirrors under SVM as well.
It probably will require a 2-3 lines of additional code... So if you need that, let me know!
One more thought.
How to check if Harvest AutoSupport actually works or not?
My Harvest servers can only access the Internet via proxy so just wanted to make sure you guys will receive the useful info.
Appreciate all the great work being done on this project!
It took me a bit more than 2 lines
I blame Python for all my bugs ...but here it is. Here's how to update your extension:
$ md5sum snapmirror_svm_extended.tar.gz > c30592a26ee4b9093b29ee257729c54d snapmirror_svm_extended.tar.gz
$ tar -xzvf snapmirror_svm_extended.tar.gz
cp python_extension_methods_v12.py snapmirror_replications.py /opt/netapp-harvest/extension/
You should see new dashboards displaying SVM breakdown:
Thanks @vachagan_gratian !
I'm getting these in _snapmirror_replicatons.log log:
[2020-03-20 12:42:13,859] [ERROR] [find_missing_nodes] ZAPI request failed: either instances or instance-uuids must be given [2020-03-20 12:45:56,091] [WARNING] [timeout_handler] Extension timeout exceeded. Terminating process [2020-03-20 12:46:27,599] [ERROR] [find_missing_nodes] ZAPI request failed: either instances or instance-uuids must be given [2020-03-20 12:47:26,989] [ERROR] [find_missing_nodes] ZAPI request failed: either instances or instance-uuids must be given
Regarding the "python_extension_methods_v12.py" file:
If I do the copy like you suggested the existing file "python_extension_methods.py" won't get overwritten - not sure if that desired. I tried both ways, overwritting an existing file and copying as a new one, same result.
@bkamil , hmm, this is strange.. Which Ontap release do you use? Also, could you run the extension in verbose and share the full logs with me?
No need to overwrite the extension methods files. The old file will be still required by the other extensions (so you should have both, python_extension_methods.py and python_extension_methods_v12.py in /opt/netapp-harvest/extension/).
[ERROR] [get_snapmirrors] ZAPI request failed: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)
With Ontap 9.5P10, everything else works with harvest.
Seems the link expired again 😞
Not sure if it's related to snapmirror features, but our MetroCluster cannot be graphed by netapp-harvest 1.6.0
via the OCUM section, we get this in the log file :
[2020-07-10 10:57:46] [WARNING] [sysinfo] Discovered [SUMMERMCDC1] on OCUM server but unable to submit metrics because no matching conf section found; to collect this cluster please add a section. [2020-07-10 10:57:46] [WARNING] [sysinfo] Discovered [SUMMERMCDC2] on OCUM server but unable to submit metrics because no matching conf section found; to collect this cluster please add a section.
via the dedicated section in the harvest configuration file, I get :
[2020-07-10 10:57:46] [NORMAL ] [main] Metrics will be submitted with graphite_root [netapp.perf.***************]
But there is absolutely nothing in the graphs (no IOPS, no latency, etc...)
any ideas ?
And it's expired again.
I'll do the 1.6 update anyway (9.7 update broke our old 1.4 install) but either 1.6.1 needs to be rounded up into the normal download package or... 1.7?
(I know this is rough, since time is better spent getting 2.0 ready!)