It has been a while, here is finally some news about NetApp Harvest.
- We are planning Harvest 2.0 alpha-test release in around a month
- Simultaneously, we will do a (probably last) legacy release, Harvest 1.7, in August.
Things have been going a bit slower than we expected, but we made a lot of progress. As announced earlier, we've changed the whole architecture to meet two goals that are often regarded as disjoint in software engineering: high performance and high flexibility. Flexibility mainly means that it will be much easier to integrate Harvest to any system that you want to monitor (including but not limited to Ontap systems). It will be also relatively easier to feed metrics into new databases. Out of the box, we will support Prometheus. Graphite will be supported as well, but with some limitations (since Graphite only accepts numerical data).
Before we plan on the official release of Harvest 2.0, we want to do an alpha-test release for users who want to "play around" with the new software and give us feedback. Tentatively we want to do this release late July or early August (once we decide the final date, I'll post an update here).
If you are interested to get the alpha release, please send me a message with your name, email address and organization. But please note the following:
- this will be a strictly test release, not intended for production
- Harvest 2.0 requires Linux + Python3 and optionally pip to install dependencies,
- supports only CDOT systems, 7 mode will come later
- gives you approximately the same set of counters as Harvest 1.6 + extensions
- for capacity counters there is no need for OCUM/ActiveIQUM anymore
- collected counters can be sent to Prometheus or Graphite or both, but:
- there will be limited number of Grafana dashboards ready and only for Prometheus (but of course, you can create your dashboards if wished)
- Harvest 2.0 includes a self-monitoring collector, so you can see how much CPU/RAM it uses and compare to Harvest 1.6
- it will come with a developer's documentation for writing your own collectors and exporters.
If you want to test the new package, we appreciate that a lot, since feedback at this point can help us incredibly.
If you want to wait for the official (stable) release, just keep tuned. We will keep you updated as soon as we have a release date on the horizon.
This will probably be our last legacy release which we are planning to do in August. It will mainly include bug fixes and no new major features.
Lastly, again, I need to apologize for not responding to many messages here in communities. I will try to catch up, but if you don't get a response to your issue, please DM or tag me (probably again -- sorry).
yes, and I think we won't need to write a collector from scratch, we can probably just use existing tools (e.g. https://github.com/NetApp-StorageGRID/storagegrid-monitoring) as an extension or plugin.
I would like to see them on the graphs if possible, but a kind of log repository would be cool also. (Just to have everything in 1 place.)
But for example hardware fault events would be good on the performance graphs (iops drops cause the aggr is rebuilding, etc).
Yes, this should be possible (I just need to find the appropriate ZAPI for disk failures/replacement). Thanks for the idea, at this point, any ideas like this are very helpful!
Since not all metrics are (yet) available in REST, we will continue using ZAPI for some time (which offers around 200k metrics). But yeah, I'll be writing a REST collector once I am done with ZAPI!
This is exciting! Are you planning to have the Prometheus dashboards available on https://grafana.com/grafana/dashboards ? Also curious if there is a way to disable the Grafana instance that I assume will be in the OVA as I plan to ship the metrics to Grafana Cloud.
Regarding, the OVA / NaBox, this project is maintained by @yannb, but I assume you can disable Grafana after installation (which runs as a systemd service by default).
I'll ask my PM if we can upload the dashboards to Grafana, thanks for the idea!
Cool, thanks for your interest, just ping me a DM if you want to be on the list.
We have a bit of delay, since we changed a few things in the implementation (e.g. we don't use the NMDSK anymore). If you are on our list, we'll keep you updated!
Any news on harvest 2.0? We've been running 1.6 for a while now but have a substantial prometheus infrastructure so would like to move to 2.0?
we rolled out the alpha-release internally today, next week we'll roll it out publicly (for anyone who signed up for the alpha package). I will post some more news next week here.
based on the feedback we get from the alpha-testers, we will set a date for the official beta release (which should be sometime in September).