Hi Mike,
Glad you like it so far and I think I know the problem.
Graphite's storage-schemas.conf file controls the frequency and retention of stored metrics. That file can have many entries and each entry has a regex expression that is compared against the incoming metrics string. The file is processed in order and the first regex that matches will cause the metrics file to be created with those retentions. So having correct entries, in the correct order (especially not having a 'catch all' as the first one), is critical.
The file if you don't edit it looks like this:
[default_1min_for_1day]
pattern = .*
retentions = 60s:1d
As you can see, this is 1min samples for 1 day retention. Maybe you forgot to edit the file and add the strings from the Harvest install guide 1.2.2 section 7.1? Or, maybe you pasted them at the end of the file and not in front of the default catch-all entry?
If this is the case, fix up the file and then future metrics will be created with the correct settings which look like this:
[netapp_perf]
pattern = ^netapp(\.poller)?\.perf7?\.
retentions = 1m:35d,5m:100d,15m:395d,1h:5y
[netapp_capacity]
pattern = ^netapp(\.poller)?\.capacity\.
retentions = 15m:100d,1d:5y
[default_1min_for_1day]
pattern = .*
retentions = 60s:1d
For existing metrics you have two options: (A) delete them and they will be created again automatically or (B) resize the existing db files.
For (A) to delete the files (and lose your existing 24hrs of data) so they will be created again automatically use one of these (depending on your metrics storage location):
rm -rf /opt/graphite/storage/whisper/netapp
or
rm -rf /var/lib/graphite/whisper/netapp
For (B) to resize them and retain your 24hrs of data:
1. Make sure you have plenty of freespace on your filesystem because the files will get much bigger (from 17KB to a little less than 2MB each) with the correct retention.
2. Change directory to your stoarge location (depending on your metrics storage location):
cd /opt/graphite/storage/whisper/netapp
or
cd /var/lib/graphite/whisper/netapp
3. Update db files (whisper-resize or whisper-resize.py depending on your distribution, just run that command by itself to see which one you have):
find perf -name *.wsp -exec whisper-resize.py --xFilesFactor=0.0 --nobackup {} 1m:35d 5m:100d 15m:395d 1h:5y \;
find perf7 -name *.wsp -exec whisper-resize.py --xFilesFactor=0.0 --nobackup {} 1m:35d 5m:100d 15m:395d 1h:5y \;
find capacity -name *.wsp -exec whisper-resize.py --xFilesFactor=0.0 --nobackup {} 15m:100d 1d:5y \;
cd poller;
find perf -name *.wsp -exec whisper-resize.py --xFilesFactor=0.0 --nobackup {} 1m:35d 5m:100d 15m:395d 1h:5y \;
find perf7 -name *.wsp -exec whisper-resize.py --xFilesFactor=0.0 --nobackup {} 1m:35d 5m:100d 15m:395d 1h:5y \;
find capacity -name *.wsp -exec whisper-resize.py --xFilesFactor=0.0 --nobackup {} 15m:100d 1d:5y \;
or
find perf -name *.wsp -exec whisper-resize --xFilesFactor=0.0 --nobackup {} 1m:35d 5m:100d 15m:395d 1h:5y \;
find perf7 -name *.wsp -exec whisper-resize --xFilesFactor=0.0 --nobackup {} 1m:35d 5m:100d 15m:395d 1h:5y \;
find capacity -name *.wsp -exec whisper-resize --xFilesFactor=0.0 --nobackup {} 15m:100d 1d:5y \;
cd poller;
find perf -name *.wsp -exec whisper-resize --xFilesFactor=0.0 --nobackup {} 1m:35d 5m:100d 15m:395d 1h:5y \;
find perf7 -name *.wsp -exec whisper-resize --xFilesFactor=0.0 --nobackup {} 1m:35d 5m:100d 15m:395d 1h:5y \;
find capacity -name *.wsp -exec whisper-resize --xFilesFactor=0.0 --nobackup {} 15m:100d 1d:5y \;
4. Reset permissions so the files can be read/written by the carbon and webserver processes:
chown -R apache:apache *
or
chown -R _graphite:_graphite *
Hope this helps! You aren't the first one to have this problem so I will also update the install guide to be more clear to help others in the future.
Cheers,
Chris Madden
Storage Architect, NetApp EMEA (and author of Harvest)
Blog: It all begins with data
