Active IQ Unified Manager Discussions

netapp-harvest graphs not showing more than 1 day

mikecurrin
20,846 Views

Hi,

 

I have Graphite working pretty well except for a few things which I'm still trying to fix. this is something I've really been wanting to see from our systems for a while.

 

The main thing is that my graphs will only display data for max of 24 hour period even if I zoom out more (eg screenshot below is showing a 48hr period). I copied all the configs as per your instructions from the Quick Start guide but when I zoom out I don't see more data.

 

graphite-24hr.jpg

 

I'm not sure if the data is actually being collected and just not being displayed or if I don't have the data stored/collected.

 

Where could I start looking to get more of an idea?

 

Thanks,

Mike

1 ACCEPTED SOLUTION

madden
20,816 Views

Hi Mike,

 

Glad you like it so far and I think I know the problem.  

 

Graphite's storage-schemas.conf file controls the frequency and retention of stored metrics.  That file can have many entries and each entry has a regex expression that is compared against the incoming metrics string.  The file is processed in order and the first regex that matches will cause the metrics file to be created with those retentions.  So having correct entries, in the correct order (especially not having a 'catch all' as the first one), is critical.

 

The file if you don't edit it looks like this:

 

[default_1min_for_1day]
pattern = .*
retentions = 60s:1d

 

As you can see, this is 1min samples for 1 day retention.  Maybe you forgot to edit the file and add the strings from the Harvest install guide 1.2.2 section 7.1?  Or, maybe you pasted them at the end of the file and not in front of the default catch-all entry? 

 

 

If this is the case, fix up the file and then future metrics will be created with the correct settings which look like this:

 

 

[netapp_perf]
pattern = ^netapp(\.poller)?\.perf7?\.
retentions = 1m:35d,5m:100d,15m:395d,1h:5y

[netapp_capacity]
pattern = ^netapp(\.poller)?\.capacity\.
retentions = 15m:100d,1d:5y

[default_1min_for_1day]
pattern = .*
retentions = 60s:1d

 

 

For existing metrics you have two options:  (A) delete them and they will be created again automatically or (B) resize the existing db files.

 

For (A) to delete the files (and lose your existing 24hrs of data) so they will be created again automatically use one of these (depending on your metrics storage location):

rm -rf /opt/graphite/storage/whisper/netapp
or 
rm -rf /var/lib/graphite/whisper/netapp

 

For (B) to resize them and retain your 24hrs of data:

 

1. Make sure you have plenty of freespace on your filesystem because the files will get much bigger (from 17KB to a little less than 2MB each) with the correct retention.

2. Change directory to your stoarge location (depending on your metrics storage location):

 

cd /opt/graphite/storage/whisper/netapp
or
cd /var/lib/graphite/whisper/netapp

3. Update db files (whisper-resize or whisper-resize.py depending on your distribution, just run that command by itself to see which one you have):

 

find perf -name *.wsp -exec whisper-resize.py --xFilesFactor=0.0 --nobackup {} 1m:35d 5m:100d 15m:395d 1h:5y \;
find perf7 -name *.wsp -exec whisper-resize.py --xFilesFactor=0.0 --nobackup {} 1m:35d 5m:100d 15m:395d 1h:5y \;
find capacity -name *.wsp -exec whisper-resize.py --xFilesFactor=0.0 --nobackup {} 15m:100d 1d:5y \;
cd poller;
find perf -name *.wsp -exec whisper-resize.py --xFilesFactor=0.0 --nobackup {} 1m:35d 5m:100d 15m:395d 1h:5y \;
find perf7 -name *.wsp -exec whisper-resize.py --xFilesFactor=0.0 --nobackup {} 1m:35d 5m:100d 15m:395d 1h:5y \;
find capacity -name *.wsp -exec whisper-resize.py --xFilesFactor=0.0 --nobackup {} 15m:100d 1d:5y \;

or 

find perf -name *.wsp -exec whisper-resize --xFilesFactor=0.0 --nobackup {} 1m:35d 5m:100d 15m:395d 1h:5y \;
find perf7 -name *.wsp -exec whisper-resize --xFilesFactor=0.0 --nobackup {} 1m:35d 5m:100d 15m:395d 1h:5y \;
find capacity -name *.wsp -exec whisper-resize --xFilesFactor=0.0 --nobackup {} 15m:100d 1d:5y \;
cd poller;
find perf -name *.wsp -exec whisper-resize --xFilesFactor=0.0 --nobackup {} 1m:35d 5m:100d 15m:395d 1h:5y \;
find perf7 -name *.wsp -exec whisper-resize --xFilesFactor=0.0 --nobackup {} 1m:35d 5m:100d 15m:395d 1h:5y \;
find capacity -name *.wsp -exec whisper-resize --xFilesFactor=0.0 --nobackup {} 15m:100d 1d:5y \; 

4. Reset permissions so the files can be read/written by the carbon and webserver processes:

chown -R apache:apache *

or
chown -R _graphite:_graphite *

 

Hope this helps!  You aren't the first one to have this problem so I will also update the install guide to be more clear to help others in the future.

 

Cheers,
Chris Madden

Storage Architect, NetApp EMEA (and author of Harvest)

Blog: It all begins with data

View solution in original post

15 REPLIES 15
Public