Active IQ Unified Manager Discussions

Grafana Dashboards are Blank for Detailed Dashboards

danielkeisling
7,422 Views

Hello,

 

I am using the Grafana 2.6.0 with Harvest 1.2.2 and the latest version of Graphite.  My issue is that clicking on dashboards under "Related Dashboards" are completely blank (no graphs or anything):

 

grafana.jpg

 

It seems that the "main" NetApp dashboards all work fine, just not the detailed Related ones (e.g., NetApp Detail: LUN, NetApp Detail: Volume, etc).  I've verified I have no zero byte or corrupt whisper files or any errors in the /opt/graphite/storage/log/carbon-cache/carbon-cache-a/console.log log.  I've also imported the latest Harvest dashboards that work for Grafana 2.6.  The only errors I see are the "not scheduled because it occurred in the past" errors in the Harvest logs.  I get about 1,000 errors per day for each NetApp cluster (2 clusters, 6 nodes per cluster).  Not sure if that has anything to do with it.

 

 

 

 

5 REPLIES 5

madden
7,374 Views

Hi @danielkeisling

 

Something in Grafana 2.6 broke the 'add current variable values' option of the links feature that underpins the 'related dashboards' choice on the dashboard.  I haven't tracked down the root cause yet to file a bug with the Grafana project but it seems to be related to not having the template items saved in the DB on the dashboard you are going to.  

 

From a tech perspective each dashboard is a json document, and in that document can be a list of template variables.  When you import the default dashboards using netapp-manager this list is blank and each time you load the dashboard it will do a search in Graphite and populate it on demand.  For some reason though if you request a dashboard and pass template variables (like the 'related dashboards' feature does) it causes this blank page which is very frustrating.

 

I discovered a workaround is just to load every dashboard once from the menu and click save.  This action saves your template items in the json document and thereafter 'related dashboards' works fine.  I hope to dig into the root cause soon using Chrome inspect soon to get it fixed in the Grafana project but for now you can use this workaround.

 

Cheers,
Chris Madden

Storage Architect, NetApp EMEA (and author of Harvest)

Blog: It all begins with data

 

If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO

 

danielkeisling
7,362 Views

Thanks for the reply.  I went through the main dashboards, loaded the data and hit Save.  The NetApp Detail: Volume dashboard now loads but has a red exclaimation point next to all graphs.  The response message is:

 

500 Internal Server Error

Internal Server Error
The server encountered an internal error or
misconfiguration and was unable to complete
your request.
Please contact the server administrator,
root@localhost and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.
More information about this error may be available
in the server error log.

Apache/2.2.15 (Red Hat) Server at localhost Port 81

 

 

Here are the request deails:

 

Request details

Url /api/datasources/proxy/2/render
Method POST
Content-Type application/x-www-form-urlencoded
Accept application/json, text/plain, */*
Request parameters

target averageSeries(highestAverage(netapp.perf.PPD.*.svm.*.vol.*.avg_latency, 5))
from -6h
until now
format json
maxDataPoints 100

 

Here's the errors in the Apache log:

 

[Fri Mar 11 09:42:14 2016] [error] [client 127.0.0.1] Premature end of script headers: graphite.wsgi, referer: http://<myservername>:3000/dashboard/db/netapp-detail-volume
[Fri Mar 11 09:42:14 2016] [error] [client 127.0.0.1] Premature end of script headers: graphite.wsgi, referer: http://<myservername>:3000/dashboard/db/netapp-detail-volume
[Fri Mar 11 09:42:14 2016] [error] [client 127.0.0.1] Premature end of script headers: graphite.wsgi, referer: http://<myservername>:3000/dashboard/db/netapp-detail-volume
[Fri Mar 11 09:42:14 2016] [error] [client 127.0.0.1] Premature end of script headers: graphite.wsgi, referer: http://<myservername>:3000/dashboard/db/netapp-detail-volume
[Fri Mar 11 09:42:14 2016] [error] [client 127.0.0.1] Premature end of script headers: graphite.wsgi, referer: http://<myservername>:3000/dashboard/db/netapp-detail-volume
[Fri Mar 11 09:42:14 2016] [error] [client 127.0.0.1] Premature end of script headers: graphite.wsgi, referer: http://<myservername>:3000/dashboard/db/netapp-detail-volume
[Fri Mar 11 09:42:15 2016] [error] /usr/lib/python2.6/site-packages/django/conf/__init__.py:75: DeprecationWarning: The ADMIN_MEDIA_PREFIX setting has been removed; use STATIC_URL instead.
[Fri Mar 11 09:42:15 2016] [error] "use STATIC_URL instead.", DeprecationWarning)
[Fri Mar 11 09:42:15 2016] [error] /usr/lib/python2.6/site-packages/django/conf/__init__.py:75: DeprecationWarning: The ADMIN_MEDIA_PREFIX setting has been removed; use STATIC_URL instead.
[Fri Mar 11 09:42:15 2016] [error] /usr/lib/python2.6/site-packages/django/conf/__init__.py:75: DeprecationWarning: The ADMIN_MEDIA_PREFIX setting has been removed; use STATIC_URL instead.
[Fri Mar 11 09:42:15 2016] [error] "use STATIC_URL instead.", DeprecationWarning)
[Fri Mar 11 09:42:15 2016] [error] "use STATIC_URL instead.", DeprecationWarning)

 

 

However, the Network Port and LUN detailed dashboards still have the same problem of showing a blank page.  More confusing, the NetApp Detail: Network LIF loaded just fine.  

madden
7,343 Views

Hi @danielkeisling

 

Argh!  I have seen that after saving you might need to reload it fully (F5 refresh), or load a different dashboard, and then come back, for it to load all the panels properly for the future.  If the problem persists, or you want to start over, you can always reload the default dashboards from the netapp-harvest.zip.  You can use the GUI or the netapp-manager -import for this.  Then go one by one through each dashboard using the normal Grafana menu to navigate to avoid any template variables being passed in the URL.  Maybe also make sure before you hit save you have selected some items in the template so that the graphs populate.  Again, I'm still trying to track down the root cause so sorry I can't be more specific.

 

Let me know how it goes!

 

Cheers,
Chris Madden

Storage Architect, NetApp EMEA (and author of Harvest)

Blog: It all begins with data

 

If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO

 

danielkeisling
7,329 Views

All the main dashboards populated fine, except for the 7-mode ones because I am not monitoring any 7-mode filers.  So I'm not sure what else I need to do.  I've navigated around quite extensively and most of the Related Dashboards are still blank pages.

 

If this was introduced in Grafana 2.6, would it just be easier to downgrade to 2.5 and wait for a fix?   

madden
7,320 Views

Hi @danielkeisling

 

Grafana 2.5 had an annoying issue that rows could be expanded by clicking on the title but not contracted.  So you could go back to 2.1.3 I guess, but then you loose the related dashboard feature altogether.  So pick your poison.  I will try to find another workaround or collect enough info on the behavior to log a bug for Grafana to fix. 

 

Cheers,
Chris

Public