Active IQ Unified Manager Discussions

NetApp Harvest 2.0 - Status update!

vachagan_gratian
10,800 Views

 

Dear all,

 

According to our initial plans, the Alpha package of Harvest 2.0 should have been rolled out long time ago, but we had to delay it several times. The good news is: at the moment we have passed all tests and fixed some final glitches and I'm waiting for my PM's approval to send it out.

 

Why the delay? We made changes in the architecture several times. Mostly because we want a highly flexible architecture that we don't need to change later again. But also because we had to do some hacking with some collectors and exporters (e.g. with Prometheus, which requires a completely different workflow compared to the other DBs). At the same time we want high performance so you will not need expensive hardware to run Harvest (we have successfully tested running Harvest 2.0 on a single-cpu Raspberry Pi with 500mb ram).


 Here are a few more details if you are interested:

 

  • Exporters: we have support for three DBs: Prometheus, InfluxDB and Graphite. The new ones allow us to do more with Harvest than before, since they accept not only numeric data, but also labels. Some screenshots of the new dashboards (all Prometheus-based):

    Screenshot from 2020-09-11 15-47-38_.pngScreenshot from 2020-09-11 15-48-14_.pngScreenshot from 2020-09-11 15-56-58_.png

 

  • Backends: there is an option to use different backends for internal data storage. Python arrays is the default now, but there is the option to use NumPy instead (gives high performance benefits if you are monitoring large clusters, but is not worth the overhead otherwise).

    This internal data storage ("Matrix") is actually what makes Harvest highly flexible, since it provides uniform APIs to the different components of Harvest. This means that you can write a plugin or exporter for Harvest, without worrying much about how the data is collected.

    Here is a small illustration to give you the idea. There is a collector called Psutil (based on the Python library of the same name), which collects metrics about running processes on the local system. I use it to monitor Harvest itself. But it can also collect things like network traffic and open sockets. I had the idea to visualize all the connections on my laptop. All I had to do is write a plugin with 20 lines of code that does IP lookups and geohashing. And voilà, this is what I get:

    Screenshot from 2020-07-05 14-53-57_.png

 

  • User Interface: we are working on an experimental tool for configuring and monitoring Harvest. This tool can pull the list of all available APIs, objects and counters from your cluster and let you choose which ones to collect. If you are interested, I present a preview of the prototype here:


    This won't be ready for the Alpha release (otherwise we have to delay it once again).

 

 

If you have ideas what we could do more with Harvest, let us know! Btw., you can still join the list of the Alpha-testers if you want. Just drop me a message with your email and company name.

 

And thanks for your patience!

 

Best regards,
Vachagan & Harvest Team

7 REPLIES 7

Drew_C
10,783 Views

@vachagan_gratian, thank you for sharing this with the Community. The "insider" updates on tools like Harvest is an excellent reason for users to participate.

 

Cheers,

-Drew

Community Manager \\ NetApp

johannes_mitterer
10,710 Views

@vachagan_gratian Thanks for the insight and also for sharing the video about the user interface.

It is definitely worth waiting for me, as it solves a lot of the pain points i currently have.

We have a lot of small scripts collecting single counters and write them into our graphite instance, the user interface would make them redundant and much easier to handle. I never really got used to the extension manager...

 

Will you also provide some Grafana dashboards with this harvest release? If yes for which DB do you plan this?

I assume all DB's is probably too much work. In my opinion one reason why Harvest 1 was so successful was that Chris Madden created a ton of very detailed dashboards which made it easy to have a fast success. After so many years we still use them and barely had to change anything.

 

Thanks for all your effort!!

Greg_Wilson
10,257 Views

I need to do some deploys for clients and need to use Influx over Graphite.

 

Any update when we can expect 2.0 ?

 

GW

vachagan_gratian
10,208 Views

Hi Greg, Alpha release is already rolled out (it is really a unstable release, a more stable Beta is planned later this month).

Regarding Influx: the Alpha includes an InfluxDB exporter, but it had some issues, we have rewritten it and will release with a patch release next week (I am waiting currently for some tests to be done).

 

If you want to get the package just ping me your email, name & company name!

Preston
7,940 Views

Hi,

 

Please help sharing the download link for Harvest 2.0 to Preston.Nguyen@ice.com .

 

Please help me with the below Harvest 2.0 questions:

 

  1. Is Harvest 2.0 compatible with NABox 3.0b (https://nabox.org/downloads/)?
  2. Is it required to configure AIQ/OCUM for aggregates, volumes, qtrees, flexgroup capacity?
  3. What is the recommended "NMSDK" version?
  4. Does Harvest 2.0 require fresh installation on NABox 3.0b (currently running Harvest 1.6) or I can perform Harvest upgrade from 1.6?
  5. Does Harvest 2.0 installation erase all existing Grafana dashboards & their historical data?

Thanks,

--Preston

408-206-7779

Marguin
7,195 Views

wondering if there is any news on 2.0?  looking at the harvest page i see only up to 1.6 and hoping to use harvest with prometheus.

daleh
9,535 Views

Great Update.  Thanks.  My partner is looking for a link to download.  Are you going to post this?  Did i miss something?

Public