Data Infrastructure Management Software Discussions

Highlighted

Update on NetApp Harvest (2.0 & 1.7)

Dear all,

 

It has been a while, here is finally some news about NetApp Harvest.

 

TL;DR:

- We are planning Harvest 2.0 alpha-test release in around a month

- Simultaneously, we will do a (probably last) legacy release, Harvest 1.7, in August.

 

Harvest 2.0

Things have been going a bit slower than we expected, but we made a lot of progress. As announced earlier, we've changed the whole architecture to meet two goals that are often regarded as disjoint in software engineering: high performance and high flexibility. Flexibility mainly means that it will be much easier to integrate Harvest to any system that you want to monitor (including but not limited to Ontap systems). It will be also relatively easier to feed metrics into new databases. Out of the box, we will support Prometheus. Graphite will be supported as well, but with some limitations (since Graphite only accepts numerical data).

 

Screenshot from 2020-06-25 15-21-52_.png

 

Before we plan on the official release of Harvest 2.0, we want to do an alpha-test release for users who want to "play around" with the new software and give us feedback. Tentatively we want to do this release late July or early August (once we decide the final date, I'll post an update here).

 

If you are interested to get the alpha release, please send me a message with your name, email address and organization. But please note the following:

- this will be a strictly test release, not intended for production

- Harvest 2.0 requires Linux + Python3 and optionally pip to install dependencies,

- supports only CDOT systems, 7 mode will come later

- gives you approximately the same set of counters as Harvest 1.6 + extensions

- for capacity counters there is no need for OCUM/ActiveIQUM anymore

- collected counters can be sent to Prometheus or Graphite or both, but:

- there will be limited number of Grafana dashboards ready and only for Prometheus (but of course, you can create your dashboards if wished)

- Harvest 2.0 includes a self-monitoring collector, so you can see how much CPU/RAM it uses and compare to Harvest 1.6

- it will come with a developer's documentation for writing your own collectors and exporters.

 

If you want to test the new package, we appreciate that a lot, since feedback at this point can help us incredibly.

If you want to wait for the official (stable) release, just keep tuned. We will keep you updated as soon as we have a release date on the horizon.

 

Harvest 1.7

This will probably be our last legacy release which we are planning to do in August. It will mainly include bug fixes and no new major features.

 

Lastly, again, I need to apologize for not responding to many messages here in communities. I will try to catch up, but if you don't get a response to your issue, please DM or tag me (probably again -- sorry).

 

Cheers,

 

Vachagan

23 REPLIES 23
Highlighted

Re: Update on NetApp Harvest (2.0 & 1.7)

Is StorageGRID support coming later as well?

Highlighted

Re: Update on NetApp Harvest (2.0 & 1.7)

yes, and I think we won't need to write a collector from scratch, we can probably just use existing tools (e.g. https://github.com/NetApp-StorageGRID/storagegrid-monitoring) as an extension or plugin.

Highlighted

Re: Update on NetApp Harvest (2.0 & 1.7)

Will there be a builtin option to show events on graphs?

Highlighted

Re: Update on NetApp Harvest (2.0 & 1.7)

Will there be any way to use InfluxDB as the back-end?

Highlighted

Re: Update on NetApp Harvest (2.0 & 1.7)

Hi Simon,

We are working on an InfluxDB exporter as well!

Highlighted

Re: Update on NetApp Harvest (2.0 & 1.7)

Hi, do you have an example of the event you would like to see? And do you just want to see the event in a graph or get alerts?

Highlighted

Re: Update on NetApp Harvest (2.0 & 1.7)

Update: we have set the date for the alpha release on 27 July!

Highlighted

Re: Update on NetApp Harvest (2.0 & 1.7)

I would like to see them on the graphs if possible, but a kind of log repository would be cool also. (Just to have everything in 1 place.)

But for example hardware fault events would be good on the performance graphs (iops drops cause the aggr is rebuilding, etc).

Highlighted

Re: Update on NetApp Harvest (2.0 & 1.7)

Yes, this should be possible (I just need to find the appropriate ZAPI for disk failures/replacement). Thanks for the idea, at this point, any ideas like this are very helpful!

Highlighted

Re: Update on NetApp Harvest (2.0 & 1.7)

Death to ZAPI - I think you mean REST API 🙂

point your browser to your cluster URL and go to /docs/api/#/storage/disk_collection_get

 

Highlighted

Re: Update on NetApp Harvest (2.0 & 1.7)

Since not all metrics are (yet) available in REST, we will continue using ZAPI for some time (which offers around 200k metrics). But yeah, I'll be writing a REST collector once I am done with ZAPI!

Highlighted

Re: Update on NetApp Harvest (2.0 & 1.7)

This is exciting! Are you planning to have the Prometheus dashboards available on https://grafana.com/grafana/dashboards ? Also curious if there is a way to disable the Grafana instance that I assume will be in the OVA as I plan to ship the metrics to Grafana Cloud.

Highlighted

Re: Update on NetApp Harvest (2.0 & 1.7)

Hi,

 

Regarding, the OVA / NaBox, this project is maintained by @yannb, but I assume you can disable Grafana after installation (which runs as a systemd service by default).

 

I'll ask my PM if we can upload the dashboards to Grafana, thanks for the idea!

Highlighted

Re: Update on NetApp Harvest (2.0 & 1.7)

I just wanted to check in here, I'm definitely interested in Harvest 2.0 - is it available now to us eager folks 🙂

Highlighted

Re: Update on NetApp Harvest (2.0 & 1.7)

Hi,

Cool, thanks for your interest, just ping me a DM if you want to be on the list.

We have a bit of delay, since we changed a few things in the implementation (e.g. we don't use the NMDSK anymore). If you are on our list, we'll keep you updated!

 

Check out the KB!
Knowledge Base
All Community Forums
Public