Grafana / Harvest 'Request Error' - "unpack requires a string argument of length 12"

[csalitros]

I have a Harvest 1.3 / Grafana 4.1.2 server.

I have two ONTAP 9.1P3 Clusters (FAS9000) that are giving me a 'Request Error' on the 'Cluster Dashboard'  

One cluster errors on Highights / 'CPU Utlilization'

One cluster errors on Highligts / 'Disk Utilization'

This machine did run out of space at one time, I had have cleared all the zero sizes files, which resolved many other issues, but this one remains.

The response error I see on both of these is:

error: unpack requires a string argument of length 12

Research has lead me to something being currupted, but that doesn't feel right since I have other clusters that are working fine.

Has anyone run into this before?

Thanks,

--Craig

[madden · Accepted solution]

Hi @csalitros

This looks like a Graphite error related to corrupt db files.

Check this GitHub issue for more, including a one-liner that might identify and remove the corrupted files:

https://github.com/graphite-project/carbon/issues/99

Cheers,

Chris Madden

Solution Architect - 3rd Platform - Systems Engineering NetApp EMEA (and author of Harvest)

Blog: It all begins with data

If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO or both!

[csalitros]

Thanks for that tip... That was one GitHub thread I did not run into in my initial google search.

I initially did not see the errors in my console.log, but after doing a restart the log started populating again.   Looking at the time stamps on the affected files it appears to line up with the time we ran out of space.   I cleared all those files and everything returned to normal.

Thanks,

--Craig