Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Graphite db not getting any data even though access log shows the metrics
2020-06-30
03:45 AM
3,403 Views
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have setup everything according to the documentation "Quick Start: Installing Graphite and Grafana Christopher Madden, NetApp 7 September, 2015".
Harvest version 1.6. Graphite version latest available on GitHub.
I see that the access log shows the below information:
10.23.107.20 - - [30/Jun/2020:12:35:00 +0200] "netapp.perf.C-mode.netapp01.node.netapp01-01.cifs.cifs_op_count.GetAttr 11.9666666666667 1593513300\n" 400 226
10.23.107.20 - - [30/Jun/2020:12:35:01 +0200] "netapp.perf.C-mode.labnetapp01.node.labnetapp01-01.aggr.l01_01_aggr0_STD.plex0.rg0.cp_read_chain 1.53571428571429 1593513301\n" 400 226
10.23.107.20 - - [30/Jun/2020:12:35:08 +0200] "netapp.perf.C-mode.netapp03.svm.bn_backup.vol.bn_fsw.qos_ops 0.241379310344828 1593513306\n" 400 226
I cant see the graphite.db getting modified. It shows a modified date from one week ago. The grafana is connected to the graphite. It doesnt show any data either which confirms there is nothing coming into the DB.
I also checked the "metrics received" graph in graphite dashboard and it is empty too.
Error log show.
[Thu Jun 25 16:28:26.289427 2020] [:error] [pid 26814] warn('SECRET_KEY is set to an unsafe default. This should be set in local_settings.py for better security')
[Fri Jun 26 11:44:35.873521 2020] [:error] [pid 18431] mod_wsgi (pid=18431): Target WSGI script '/opt/graphite/conf/graphite.wsgi' cannot be loaded as Python module.
[Fri Jun 26 11:44:35.873566 2020] [:error] [pid 18431] mod_wsgi (pid=18431): Exception occurred processing WSGI script '/opt/graphite/conf/graphite.wsgi'.
[Fri Jun 26 11:44:35.873622 2020] [:error] [pid 18431] Traceback (most recent call last):
[Fri Jun 26 11:44:35.873650 2020] [:error] [pid 18431] File "/opt/graphite/conf/graphite.wsgi", line 25, in <module>
[Fri Jun 26 11:44:35.873739 2020] [:error] [pid 18431] import graphite.metrics.search
[Fri Jun 26 11:44:35.873757 2020] [:error] [pid 18431] File "/usr/lib/python2.7/site-packages/graphite/metrics/search.py", line 6, in <module>
[Fri Jun 26 11:44:35.873838 2020] [:error] [pid 18431] from graphite.storage import is_pattern, match_entries
[Fri Jun 26 11:44:35.873871 2020] [:error] [pid 18431] File "/usr/lib/python2.7/site-packages/graphite/storage.py", line 9, in <module>
[Fri Jun 26 11:44:35.874029 2020] [:error] [pid 18431] from graphite.remote_storage import RemoteStore
[Fri Jun 26 11:44:35.874048 2020] [:error] [pid 18431] File "/usr/lib/python2.7/site-packages/graphite/remote_storage.py", line 8, in <module>
[Fri Jun 26 11:44:35.874351 2020] [:error] [pid 18430] mod_wsgi (pid=18430): Target WSGI script '/opt/graphite/conf/graphite.wsgi' cannot be loaded as Python module.
[Fri Jun 26 11:44:35.874381 2020] [:error] [pid 18430] mod_wsgi (pid=18430): Exception occurred processing WSGI script '/opt/graphite/conf/graphite.wsgi'.
Please advice which is the best place to look at to find out the root cause of this issue.
1 REPLY 1
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The graphite db is not getting updated. The metrics seem to be flowing in but the db is not getting updated. I don't understand what the error log is trying to tell. kindly help with a what should we look at to solve this issue.
[root@netappharvestlab-graphite webapp]# pwd
/opt/graphite/storage/log/webapp
[root@netappharvestlab-graphite webapp]# ll
total 31744
-rw-r--r-- 1 apache apache 15631568 Jun 30 14:37 access.log
-rw-r--r-- 1 apache apache 129654 Jun 26 11:44 error.log
[root@netappharvestlab-graphite storage]# ll
total 68
drwxr-xr-x 3 apache apache 23 Jun 15 16:00 ceres
-rw-rw-rw- 1 apache apache 69632 Jun 26 11:44 graphite.db
drwxr-xr-x 2 apache apache 6 Jun 15 16:00 lists
drwxr-xr-x 4 apache apache 37 Jun 15 16:01 log
drwxr-xr-x 3 apache apache 23 Jun 15 16:00 rrd
Acces log:
10.23.107.20 - - [30/Jun/2020:14:34:04 +0200] "netapp.capacity.C-mode.netapp02.svm.abc.vol.unb_root.afs_used 11.58984375 1593520440\n" 400 226
10.23.107.20 - - [30/Jun/2020:14:34:04 +0200] "netapp.perf.C-mode.netapp03.svm.netapp06-02.vol.vol0.other_latency 0.00604095159190437 1593520442\n" 400 226
10.23.107.20 - - [30/Jun/2020:14:34:09 +0200] "netapp.perf.C-mode.netapp02.svm.bac_backup.vol.bn_fsw.qos_ops 0.225806451612903 1593520447\n" 400 226
Error log:
[Thu Jun 25 16:28:26.280416 2020] [:error] [pid 26813] warn('SECRET_KEY is set to an unsafe default. This should be set in local_settings.py for better security')
[Thu Jun 25 16:28:26.289408 2020] [:error] [pid 26814] /usr/lib/python2.7/site-packages/graphite/settings.py:249: UserWarning: SECRET_KEY is set to an unsafe default. This should be set in local_settings.py for better security
[Thu Jun 25 16:28:26.289427 2020] [:error] [pid 26814] warn('SECRET_KEY is set to an unsafe default. This should be set in local_settings.py for better security')
[Fri Jun 26 11:44:35.873521 2020] [:error] [pid 18431] mod_wsgi (pid=18431): Target WSGI script '/opt/graphite/conf/graphite.wsgi' cannot be loaded as Python module.
[Fri Jun 26 11:44:35.873566 2020] [:error] [pid 18431] mod_wsgi (pid=18431): Exception occurred processing WSGI script '/opt/graphite/conf/graphite.wsgi'.
[Fri Jun 26 11:44:35.873622 2020] [:error] [pid 18431] Traceback (most recent call last):
[Fri Jun 26 11:44:35.873650 2020] [:error] [pid 18431] File "/opt/graphite/conf/graphite.wsgi", line 25, in <module>
[Fri Jun 26 11:44:35.873739 2020] [:error] [pid 18431] import graphite.metrics.search
[Fri Jun 26 11:44:35.873757 2020] [:error] [pid 18431] File "/usr/lib/python2.7/site-packages/graphite/metrics/search.py", line 6, in <module>
[Fri Jun 26 11:44:35.873838 2020] [:error] [pid 18431] from graphite.storage import is_pattern, match_entries
[Fri Jun 26 11:44:35.873871 2020] [:error] [pid 18431] File "/usr/lib/python2.7/site-packages/graphite/storage.py", line 9, in <module>
[Fri Jun 26 11:44:35.874029 2020] [:error] [pid 18431] from graphite.remote_storage import RemoteStore
[Fri Jun 26 11:44:35.874048 2020] [:error] [pid 18431] File "/usr/lib/python2.7/site-packages/graphite/remote_storage.py", line 8, in <module>
[Fri Jun 26 11:44:35.874351 2020] [:error] [pid 18430] mod_wsgi (pid=18430): Target WSGI script '/opt/graphite/conf/graphite.wsgi' cannot be loaded as Python module.
[Fri Jun 26 11:44:35.874381 2020] [:error] [pid 18430] mod_wsgi (pid=18430): Exception occurred processing WSGI script '/opt/graphite/conf/graphite.wsgi'.