We initially installed graphite on a 10 GB filesystem and it filled up. After shutting down services and expanding the filesystem to 120 GB, i'm unable to start the carbon-cache service. The httpd and grafana-server services start fine. Below is the error:
# service carbon-cache start
Traceback (most recent call last):
File "bin/carbon-cache.py", line 32, in <module>
File "/opt/graphite/lib/carbon/util.py", line 94, in run_twistd_plugin
File "/usr/lib64/python2.6/site-packages/twisted/application/app.py", line 669, in parseOptions
File "/usr/lib64/python2.6/site-packages/twisted/python/usage.py", line 231, in parseOptions
File "/usr/lib64/python2.6/site-packages/twisted/python/usage.py", line 241, in parseOptions
File "/opt/graphite/lib/carbon/conf.py", line 256, in postOptions
File "/opt/graphite/lib/carbon/conf.py", line 348, in handleAction
pid = int(pf.read().strip())
ValueError: invalid literal for int() with base 10: ''
I found another post with a solution to delete the following file and it would be re-generated
but it made no difference. I'm guessing the db was corrupted once the filesystem filled up. Any ideas?
So it looks like the code is reading the pidfile, and the ValueError is raised when trying to convert an empty string into an integer, but this is not expected/handled by the code.
My guess is you started carbon at the time your filesystem was full which caused the file to be created but to have no contents. Delete the pidfile(s) (default location is in /opt/graphite/storage) and restart carbon-cache and see if that solves things.
Cheers, Chris Madden
Storage Architect, NetApp EMEA (and author of Harvest)