Active IQ Unified Manager Discussions

Update on NetApp Harvest (2.0 & 1.7)

vachagan_gratian

Dear all,

 

It has been a while, here is finally some news about NetApp Harvest.

 

TL;DR:

- We are planning Harvest 2.0 alpha-test release in around a month

- Simultaneously, we will do a (probably last) legacy release, Harvest 1.7, in August.

 

Harvest 2.0

Things have been going a bit slower than we expected, but we made a lot of progress. As announced earlier, we've changed the whole architecture to meet two goals that are often regarded as disjoint in software engineering: high performance and high flexibility. Flexibility mainly means that it will be much easier to integrate Harvest to any system that you want to monitor (including but not limited to Ontap systems). It will be also relatively easier to feed metrics into new databases. Out of the box, we will support Prometheus. Graphite will be supported as well, but with some limitations (since Graphite only accepts numerical data).

 

Screenshot from 2020-06-25 15-21-52_.png

 

Before we plan on the official release of Harvest 2.0, we want to do an alpha-test release for users who want to "play around" with the new software and give us feedback. Tentatively we want to do this release late July or early August (once we decide the final date, I'll post an update here).

 

If you are interested to get the alpha release, please send me a message with your name, email address and organization. But please note the following:

- this will be a strictly test release, not intended for production

- Harvest 2.0 requires Linux + Python3 and optionally pip to install dependencies,

- supports only CDOT systems, 7 mode will come later

- gives you approximately the same set of counters as Harvest 1.6 + extensions

- for capacity counters there is no need for OCUM/ActiveIQUM anymore

- collected counters can be sent to Prometheus or Graphite or both, but:

- there will be limited number of Grafana dashboards ready and only for Prometheus (but of course, you can create your dashboards if wished)

- Harvest 2.0 includes a self-monitoring collector, so you can see how much CPU/RAM it uses and compare to Harvest 1.6

- it will come with a developer's documentation for writing your own collectors and exporters.

 

If you want to test the new package, we appreciate that a lot, since feedback at this point can help us incredibly.

If you want to wait for the official (stable) release, just keep tuned. We will keep you updated as soon as we have a release date on the horizon.

 

Harvest 1.7

This will probably be our last legacy release which we are planning to do in August. It will mainly include bug fixes and no new major features.

 

Lastly, again, I need to apologize for not responding to many messages here in communities. I will try to catch up, but if you don't get a response to your issue, please DM or tag me (probably again -- sorry).

 

Cheers,

 

Vachagan

36 REPLIES 36

Re: Update on NetApp Harvest (2.0 & 1.7)

EWILTS_SAS

Is StorageGRID support coming later as well?

Re: Update on NetApp Harvest (2.0 & 1.7)

vachagan_gratian

yes, and I think we won't need to write a collector from scratch, we can probably just use existing tools (e.g. https://github.com/NetApp-StorageGRID/storagegrid-monitoring) as an extension or plugin.

Re: Update on NetApp Harvest (2.0 & 1.7)

zsliptaii

Will there be a builtin option to show events on graphs?

Re: Update on NetApp Harvest (2.0 & 1.7)

SimonGordon

Will there be any way to use InfluxDB as the back-end?

Re: Update on NetApp Harvest (2.0 & 1.7)

vachagan_gratian

Hi Simon,

We are working on an InfluxDB exporter as well!

Re: Update on NetApp Harvest (2.0 & 1.7)

vachagan_gratian

Hi, do you have an example of the event you would like to see? And do you just want to see the event in a graph or get alerts?

Re: Update on NetApp Harvest (2.0 & 1.7)

vachagan_gratian

Update: we have set the date for the alpha release on 27 July!

Re: Update on NetApp Harvest (2.0 & 1.7)

zsliptaii

I would like to see them on the graphs if possible, but a kind of log repository would be cool also. (Just to have everything in 1 place.)

But for example hardware fault events would be good on the performance graphs (iops drops cause the aggr is rebuilding, etc).

Re: Update on NetApp Harvest (2.0 & 1.7)

vachagan_gratian

Yes, this should be possible (I just need to find the appropriate ZAPI for disk failures/replacement). Thanks for the idea, at this point, any ideas like this are very helpful!

Re: Update on NetApp Harvest (2.0 & 1.7)

EWILTS_SAS

Death to ZAPI - I think you mean REST API 🙂

point your browser to your cluster URL and go to /docs/api/#/storage/disk_collection_get

 

Re: Update on NetApp Harvest (2.0 & 1.7)

vachagan_gratian

Since not all metrics are (yet) available in REST, we will continue using ZAPI for some time (which offers around 200k metrics). But yeah, I'll be writing a REST collector once I am done with ZAPI!

Re: Update on NetApp Harvest (2.0 & 1.7)

genepuppet

This is exciting! Are you planning to have the Prometheus dashboards available on https://grafana.com/grafana/dashboards ? Also curious if there is a way to disable the Grafana instance that I assume will be in the OVA as I plan to ship the metrics to Grafana Cloud.

Re: Update on NetApp Harvest (2.0 & 1.7)

vachagan_gratian

Hi,

 

Regarding, the OVA / NaBox, this project is maintained by @yannb, but I assume you can disable Grafana after installation (which runs as a systemd service by default).

 

I'll ask my PM if we can upload the dashboards to Grafana, thanks for the idea!

Re: Update on NetApp Harvest (2.0 & 1.7)

eespin

I just wanted to check in here, I'm definitely interested in Harvest 2.0 - is it available now to us eager folks 🙂

Re: Update on NetApp Harvest (2.0 & 1.7)

vachagan_gratian

Hi,

Cool, thanks for your interest, just ping me a DM if you want to be on the list.

We have a bit of delay, since we changed a few things in the implementation (e.g. we don't use the NMDSK anymore). If you are on our list, we'll keep you updated!

 

Earn Rewards for Your Review!
GPI Review Banner
All Community Forums
Public