2016-08-19 12:40 PM
We initially installed graphite on a 10 GB filesystem and it filled up. After shutting down services and expanding the filesystem to 120 GB, i'm unable to start the carbon-cache service. The httpd and grafana-server services start fine. Below is the error:
# service carbon-cache start Starting carbon-cache:a... Traceback (most recent call last): File "bin/carbon-cache.py", line 32, in <module> run_twistd_plugin(__file__) File "/opt/graphite/lib/carbon/util.py", line 94, in run_twistd_plugin config.parseOptions(twistd_options) File "/usr/lib64/python2.6/site-packages/twisted/application/app.py", line 669, in parseOptions usage.Options.parseOptions(self, options) File "/usr/lib64/python2.6/site-packages/twisted/python/usage.py", line 231, in parseOptions self.subOptions.parseOptions(rest) File "/usr/lib64/python2.6/site-packages/twisted/python/usage.py", line 241, in parseOptions self.postOptions() File "/opt/graphite/lib/carbon/conf.py", line 256, in postOptions self.handleAction() File "/opt/graphite/lib/carbon/conf.py", line 348, in handleAction pid = int(pf.read().strip()) ValueError: invalid literal for int() with base 10: '' [FAILED]
I found another post with a solution to delete the following file and it would be re-generated
but it made no difference. I'm guessing the db was corrupted once the filesystem filled up. Any ideas?
Solved! SEE THE SOLUTION
2016-08-22 01:03 AM - edited 2016-08-22 01:05 AM
This is not a harvest problem but rather a graphite/carbon problem.
The error seems to be:
File "/opt/graphite/lib/carbon/conf.py", line 348, in handleAction pid = int(pf.read().strip()) ValueError: invalid literal for int() with base 10: '' [FAILED]
Checking https://github.com/graphite-project/carbon/blob/master/lib/carbon/conf.py around line 348 I see:
elif action == "start": if exists(pidfile): pf = open(pidfile, 'r') try: pid = int(pf.read().strip()) pf.close() except IOError: print "Could not read pidfile %s" % pidfile raise SystemExit(1)
So it looks like the code is reading the pidfile, and the ValueError is raised when trying to convert an empty string into an integer, but this is not expected/handled by the code.
My guess is you started carbon at the time your filesystem was full which caused the file to be created but to have no contents. Delete the pidfile(s) (default location is in /opt/graphite/storage) and restart carbon-cache and see if that solves things.
Storage Architect, NetApp EMEA (and author of Harvest)
Blog: It all begins with data
If this post resolved your issue, please help others by selecting ACCEPT AS SOLUTION or adding a KUDO or both!