Active IQ Unified Manager Discussions

Update on NetApp Harvest (2.0 & 1.7)

vachagan_gratian
21,113 Views

Dear all,

 

It has been a while, here is finally some news about NetApp Harvest.

 

TL;DR:

- We are planning Harvest 2.0 alpha-test release in around a month

- Simultaneously, we will do a (probably last) legacy release, Harvest 1.7, in August.

 

Harvest 2.0

Things have been going a bit slower than we expected, but we made a lot of progress. As announced earlier, we've changed the whole architecture to meet two goals that are often regarded as disjoint in software engineering: high performance and high flexibility. Flexibility mainly means that it will be much easier to integrate Harvest to any system that you want to monitor (including but not limited to Ontap systems). It will be also relatively easier to feed metrics into new databases. Out of the box, we will support Prometheus. Graphite will be supported as well, but with some limitations (since Graphite only accepts numerical data).

 

Screenshot from 2020-06-25 15-21-52_.png

 

Before we plan on the official release of Harvest 2.0, we want to do an alpha-test release for users who want to "play around" with the new software and give us feedback. Tentatively we want to do this release late July or early August (once we decide the final date, I'll post an update here).

 

If you are interested to get the alpha release, please send me a message with your name, email address and organization. But please note the following:

- this will be a strictly test release, not intended for production

- Harvest 2.0 requires Linux + Python3 and optionally pip to install dependencies,

- supports only CDOT systems, 7 mode will come later

- gives you approximately the same set of counters as Harvest 1.6 + extensions

- for capacity counters there is no need for OCUM/ActiveIQUM anymore

- collected counters can be sent to Prometheus or Graphite or both, but:

- there will be limited number of Grafana dashboards ready and only for Prometheus (but of course, you can create your dashboards if wished)

- Harvest 2.0 includes a self-monitoring collector, so you can see how much CPU/RAM it uses and compare to Harvest 1.6

- it will come with a developer's documentation for writing your own collectors and exporters.

 

If you want to test the new package, we appreciate that a lot, since feedback at this point can help us incredibly.

If you want to wait for the official (stable) release, just keep tuned. We will keep you updated as soon as we have a release date on the horizon.

 

Harvest 1.7

This will probably be our last legacy release which we are planning to do in August. It will mainly include bug fixes and no new major features.

 

Lastly, again, I need to apologize for not responding to many messages here in communities. I will try to catch up, but if you don't get a response to your issue, please DM or tag me (probably again -- sorry).

 

Cheers,

 

Vachagan

36 REPLIES 36

EWILTS_SAS
16,914 Views

Is StorageGRID support coming later as well?

vachagan_gratian
16,824 Views

yes, and I think we won't need to write a collector from scratch, we can probably just use existing tools (e.g. https://github.com/NetApp-StorageGRID/storagegrid-monitoring) as an extension or plugin.

zsliptaii
16,707 Views

Will there be a builtin option to show events on graphs?

vachagan_gratian
16,543 Views

Hi, do you have an example of the event you would like to see? And do you just want to see the event in a graph or get alerts?

zsliptaii
16,542 Views

I would like to see them on the graphs if possible, but a kind of log repository would be cool also. (Just to have everything in 1 place.)

But for example hardware fault events would be good on the performance graphs (iops drops cause the aggr is rebuilding, etc).

vachagan_gratian
16,537 Views

Yes, this should be possible (I just need to find the appropriate ZAPI for disk failures/replacement). Thanks for the idea, at this point, any ideas like this are very helpful!

EWILTS_SAS
16,514 Views

Death to ZAPI - I think you mean REST API 🙂

point your browser to your cluster URL and go to /docs/api/#/storage/disk_collection_get

 

vachagan_gratian
16,504 Views

Since not all metrics are (yet) available in REST, we will continue using ZAPI for some time (which offers around 200k metrics). But yeah, I'll be writing a REST collector once I am done with ZAPI!

SimonGordon
16,606 Views

Will there be any way to use InfluxDB as the back-end?

vachagan_gratian
16,542 Views

Hi Simon,

We are working on an InfluxDB exporter as well!

vachagan_gratian
16,542 Views

Update: we have set the date for the alpha release on 27 July!

genepuppet
16,432 Views

This is exciting! Are you planning to have the Prometheus dashboards available on https://grafana.com/grafana/dashboards ? Also curious if there is a way to disable the Grafana instance that I assume will be in the OVA as I plan to ship the metrics to Grafana Cloud.

vachagan_gratian
16,376 Views

Hi,

 

Regarding, the OVA / NaBox, this project is maintained by @yannb, but I assume you can disable Grafana after installation (which runs as a systemd service by default).

 

I'll ask my PM if we can upload the dashboards to Grafana, thanks for the idea!

eespin
17,072 Views

I just wanted to check in here, I'm definitely interested in Harvest 2.0 - is it available now to us eager folks 🙂

vachagan_gratian
16,984 Views

Hi,

Cool, thanks for your interest, just ping me a DM if you want to be on the list.

We have a bit of delay, since we changed a few things in the implementation (e.g. we don't use the NMDSK anymore). If you are on our list, we'll keep you updated!

 

MikeSystems
11,842 Views

Any news on harvest 2.0?  We've been running 1.6 for a while now but have a substantial prometheus infrastructure so would like to move to 2.0?

 

Thanks,

 

Mike

vachagan_gratian
11,782 Views

Hi Mike,

 

we rolled out the alpha-release internally today, next week we'll roll it out publicly (for anyone who signed up for the alpha package). I will post some more news next week here.

 

based on the feedback we get from the alpha-testers, we will set a date for the official beta release (which should be sometime in September).

 

Cheers,

Vachagan

JayDev
10,255 Views

Hi Vachagan,

 

Is the beta version 2.0 is out? Really interested in that integration with prometheus.

 

 

evelez
11,475 Views

are you still looking for Alpha testers?

vachagan_gratian
11,459 Views

yes!

Public