Subscribe
Accepted Solution

Removing renamed node information from harvest/grafana

Hi all,

 

when a cluster node is renamed, the old entry is not wiped from the node list in the "Netapp Dashboard: Cluster"
How can we fix it?

Thanks



Re: Removing renamed node information from harvest/grafana

Hi @LORENZO_CONTI

 

 

Each metric in Graphite correlates to a <metricname>.wsp database file on the Graphite host filesystem, and the hierarchy is built as subdirectories that ultimately contain that file.  If you want to rename any piece of the metric name a simple filesystem directory and/or database file rename will do it.  In your case you probably have metrics files with the old node name (and lots of history), and recently created metrics files with the new names (and only recent history).  If you don't care about history you can simply remove those old directories/files.  If you do care about history you can use a Graphite whisper tool to merge the datapoints: whisper-fill.py  It gets a little harder since all the metrics are in two sets of subdirectories, but a small script, or even manual creation of the SRC DST paths with cmd to cut-n-paste in Excel, shouldn't be too hard.  Also, if you want to rename something in the future you could avoid this work by stopping Harvest, doing the object rename on the cluster, rename the directories on the Graphite host, and start Harvest again.  It's not ideal but unfortunately Graphite has no API to handle renames, so filesystem renames is the only option.

 

Cheers,
Chris Madden

Storage Architect, NetApp EMEA (and author of Harvest)

Blog: It all begins with data

 

If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO or both!

Re: Removing renamed node information from harvest/grafana

Hello Chris,

thanks for the help. 

When you say 

"Also, if you want to rename something in the future you could avoid this work by stopping Harvest, doing the object rename on the cluster, rename the directories on the Graphite host, and start Harvest again."

you mean we can simply stop the poller of the involved cluster, do the rename, and then start it again?

Cheers

Lorenzo

Re: Removing renamed node information from harvest/grafana

Hi @LORENZO_CONTI

 

Yes, exactly!  Harvest (by default) refreshes it's instance list every 1 hour.  So if you rename something (a lif, a vol, a node) it will be discovered within 1 hour and you'll get a new metric for the new name.  If you stop Harvest for the cluster, rename in the cluster, rename the files on the Graphite host, and start Harvest, then when it starts it discovers the new name and when it submits the metric Graphite will update the existing db files.

 

Cheers,
Chris Madden

Storage Architect, NetApp EMEA (and author of Harvest)

Blog: It all begins with data

 

If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO or both!