Graphite db not getting any data even though access log shows the metrics

Novicer · ‎2020-06-30

I have setup everything according to the documentation "Quick Start: Installing Graphite and Grafana Christopher Madden, NetApp 7 September, 2015".

Harvest version 1.6. Graphite version latest available on GitHub.

I see that the access log shows the below information:

10.23.107.20 - - [30/Jun/2020:12:35:00 +0200] "netapp.perf.C-mode.netapp01.node.netapp01-01.cifs.cifs_op_count.GetAttr 11.9666666666667 1593513300\n" 400 226

10.23.107.20 - - [30/Jun/2020:12:35:01 +0200] "netapp.perf.C-mode.labnetapp01.node.labnetapp01-01.aggr.l01_01_aggr0_STD.plex0.rg0.cp_read_chain 1.53571428571429 1593513301\n" 400 226

10.23.107.20 - - [30/Jun/2020:12:35:08 +0200] "netapp.perf.C-mode.netapp03.svm.bn_backup.vol.bn_fsw.qos_ops 0.241379310344828 1593513306\n" 400 226

I cant see the graphite.db getting modified. It shows a modified date from one week ago. The grafana is connected to the graphite. It doesnt show any data either which confirms there is nothing coming into the DB.

I also checked the "metrics received" graph in graphite dashboard and it is empty too.

Error log show.

[Thu Jun 25 16:28:26.289427 2020] [:error] [pid 26814] warn('SECRET_KEY is set to an unsafe default. This should be set in local_settings.py for better security')

[Fri Jun 26 11:44:35.873521 2020] [:error] [pid 18431] mod_wsgi (pid=18431): Target WSGI script '/opt/graphite/conf/graphite.wsgi' cannot be loaded as Python module.

[Fri Jun 26 11:44:35.873566 2020] [:error] [pid 18431] mod_wsgi (pid=18431): Exception occurred processing WSGI script '/opt/graphite/conf/graphite.wsgi'.

[Fri Jun 26 11:44:35.873622 2020] [:error] [pid 18431] Traceback (most recent call last):

[Fri Jun 26 11:44:35.873650 2020] [:error] [pid 18431] File "/opt/graphite/conf/graphite.wsgi", line 25, in <module>

[Fri Jun 26 11:44:35.873739 2020] [:error] [pid 18431] import graphite.metrics.search

[Fri Jun 26 11:44:35.873757 2020] [:error] [pid 18431] File "/usr/lib/python2.7/site-packages/graphite/metrics/search.py", line 6, in <module>

[Fri Jun 26 11:44:35.873838 2020] [:error] [pid 18431] from graphite.storage import is_pattern, match_entries

[Fri Jun 26 11:44:35.873871 2020] [:error] [pid 18431] File "/usr/lib/python2.7/site-packages/graphite/storage.py", line 9, in <module>

[Fri Jun 26 11:44:35.874029 2020] [:error] [pid 18431] from graphite.remote_storage import RemoteStore

[Fri Jun 26 11:44:35.874048 2020] [:error] [pid 18431] File "/usr/lib/python2.7/site-packages/graphite/remote_storage.py", line 8, in <module>

[Fri Jun 26 11:44:35.874351 2020] [:error] [pid 18430] mod_wsgi (pid=18430): Target WSGI script '/opt/graphite/conf/graphite.wsgi' cannot be loaded as Python module.

[Fri Jun 26 11:44:35.874381 2020] [:error] [pid 18430] mod_wsgi (pid=18430): Exception occurred processing WSGI script '/opt/graphite/conf/graphite.wsgi'.

Please advice which is the best place to look at to find out the root cause of this issue.

@vachagan_gratian @yannb

Novicer · ‎2020-06-30

The graphite db is not getting updated. The metrics seem to be flowing in but the db is not getting updated. I don't understand what the error log is trying to tell. kindly help with a what should we look at to solve this issue.

[root@netappharvestlab-graphite webapp]# pwd

/opt/graphite/storage/log/webapp

[root@netappharvestlab-graphite webapp]# ll

total 31744

-rw-r--r-- 1 apache apache 15631568 Jun 30 14:37 access.log

-rw-r--r-- 1 apache apache 129654 Jun 26 11:44 error.log

[root@netappharvestlab-graphite storage]# ll

total 68

drwxr-xr-x 3 apache apache 23 Jun 15 16:00 ceres

-rw-rw-rw- 1 apache apache 69632 Jun 26 11:44 graphite.db

drwxr-xr-x 2 apache apache 6 Jun 15 16:00 lists

drwxr-xr-x 4 apache apache 37 Jun 15 16:01 log

drwxr-xr-x 3 apache apache 23 Jun 15 16:00 rrd

Acces log:

10.23.107.20 - - [30/Jun/2020:14:34:04 +0200] "netapp.capacity.C-mode.netapp02.svm.abc.vol.unb_root.afs_used 11.58984375 1593520440\n" 400 226

10.23.107.20 - - [30/Jun/2020:14:34:04 +0200] "netapp.perf.C-mode.netapp03.svm.netapp06-02.vol.vol0.other_latency 0.00604095159190437 1593520442\n" 400 226

10.23.107.20 - - [30/Jun/2020:14:34:09 +0200] "netapp.perf.C-mode.netapp02.svm.bac_backup.vol.bn_fsw.qos_ops 0.225806451612903 1593520447\n" 400 226

Error log:

[Thu Jun 25 16:28:26.280416 2020] [:error] [pid 26813] warn('SECRET_KEY is set to an unsafe default. This should be set in local_settings.py for better security')

[Thu Jun 25 16:28:26.289408 2020] [:error] [pid 26814] /usr/lib/python2.7/site-packages/graphite/settings.py:249: UserWarning: SECRET_KEY is set to an unsafe default. This should be set in local_settings.py for better security