Solved: NetApp Harvest - Can't start carbon-cache service

dhubert27 · ‎2016-08-19

We initially installed graphite on a 10 GB filesystem and it filled up. After shutting down services and expanding the filesystem to 120 GB, i'm unable to start the carbon-cache service. The httpd and grafana-server services start fine. Below is the error:

# service carbon-cache start
Starting carbon-cache:a...
Traceback (most recent call last):
  File "bin/carbon-cache.py", line 32, in <module>
    run_twistd_plugin(__file__)
  File "/opt/graphite/lib/carbon/util.py", line 94, in run_twistd_plugin
    config.parseOptions(twistd_options)
  File "/usr/lib64/python2.6/site-packages/twisted/application/app.py", line 669, in parseOptions
    usage.Options.parseOptions(self, options)
  File "/usr/lib64/python2.6/site-packages/twisted/python/usage.py", line 231, in parseOptions
    self.subOptions.parseOptions(rest)
  File "/usr/lib64/python2.6/site-packages/twisted/python/usage.py", line 241, in parseOptions
    self.postOptions()
  File "/opt/graphite/lib/carbon/conf.py", line 256, in postOptions
    self.handleAction()
  File "/opt/graphite/lib/carbon/conf.py", line 348, in handleAction
    pid = int(pf.read().strip())
ValueError: invalid literal for int() with base 10: ''
                                                           [FAILED]

I found another post with a solution to delete the following file and it would be re-generated

/opt/graphite/storage/whisper/netapp

but it made no difference. I'm guessing the db was corrupted once the filesystem filled up. Any ideas?

madden · ‎2016-08-22

Hi @dhubert27

This is not a harvest problem but rather a graphite/carbon problem.

The error seems to be:

 File "/opt/graphite/lib/carbon/conf.py", line 348, in handleAction
    pid = int(pf.read().strip())
ValueError: invalid literal for int() with base 10: ''
                                                           [FAILED]

Checking https://github.com/graphite-project/carbon/blob/master/lib/carbon/conf.py around line 348 I see:

        elif action == "start":
            if exists(pidfile):
                pf = open(pidfile, 'r')
                try:
                    pid = int(pf.read().strip())
                    pf.close()
                except IOError:
                    print "Could not read pidfile %s" % pidfile
                    raise SystemExit(1)

So it looks like the code is reading the pidfile, and the ValueError is raised when trying to convert an empty string into an integer, but this is not expected/handled by the code.

My guess is you started carbon at the time your filesystem was full which caused the file to be created but to have no contents. Delete the pidfile(s) (default location is in /opt/graphite/storage) and restart carbon-cache and see if that solves things.

Cheers,
Chris Madden

Storage Architect, NetApp EMEA (and author of Harvest)

Blog: It all begins with data

If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO or both!

View solution in original post

madden · ‎2016-08-22