NetApp Harvest Graphite Carbon GAIResolver issue

[shashii82]

All, we recently upgraded our NABox OVA to beta version 5.2b and suddenly data collection stopped working. After some analysis we came to know that the GAIResolver module is missing. Any help would be appreciated! 

root@harvestbox01:/opt/graphite# systemctl status carbon-relay.service

* carbon-relay.service - graphite carbon relay

   Loaded: loaded (/etc/systemd/system/carbon-relay.service; enabled)

   Active: failed (Result: exit-code) since Wed 2018-09-19 13:42:13 CEST; 5min ago

  Process: 6003 ExecStart=/opt/graphite/bin/carbon-relay.py --pidfile=/opt/graphite/storage/carbon-relay-a.pid start (code=exited, status=1/FAILURE)

  Process: 6001 ExecStartPre=/bin/rm -f /opt/graphite/storage/carbon-relay-a.pid (code=exited, status=0/SUCCESS)

Sep 19 13:42:13  harvestbox01 carbon-relay.py[6003]: File "/opt/graphite/lib/carbon/service.py", line 183, in setupRelayProcessor

Sep 19 13:42:13 harvestbox01 carbon-relay.py[6003]: from carbon.client import CarbonClientManager

Sep 19 13:42:13 harvestbox01 carbon-relay.py[6003]: File "/opt/graphite/lib/carbon/client.py", line 15, in <module>

Sep 19 13:42:13 harvestbox01 carbon-relay.py[6003]: from carbon.resolver import setUpRandomResolver

Sep 19 13:42:13 harvestbox01 carbon-relay.py[6003]: File "/opt/graphite/lib/carbon/resolver.py", line 5, in <module>

Sep 19 13:42:13 harvestbox01 carbon-relay.py[6003]: from twisted.internet._resolver import GAIResolver

Sep 19 13:42:13 harvestbox01 carbon-relay.py[6003]: ImportError: No module named _resolver

Sep 19 13:42:13 harvestbox01 systemd[1]: carbon-relay.service: control process exited, code=exited status=1

Sep 19 13:42:13 harvestbox01 systemd[1]: Failed to start graphite carbon relay.

Sep 19 13:42:13 harvestbox01 systemd[1]: Unit carbon-relay.service entered failed state.

[vachagan_gratian]

Hi, you probably mean v 2.5b, right? Did you verify md5 after download?

@yannb, maybe you can look at this? I only worked with 2.3 until now.

[yannb]

That's indeed a known bug with a specific version of carbon.

For some reason it seems sometimes NAbox updates itself when it's connected to internet.

I would try re-applying the most recent 2.5b update available on https://nabox.org/downloads/

Le me know how it goes

[shashii82]

Thanks, we haven't get chance to do re-apply the update but looks like there is no alternative. 

[shashii82]

yes, previous beta version was indeed working correctly. do you think any other alternative or solution?

[yannb]

Did you re-apply the update? It's not breaking anything, just upload the tgz in the UI

[Raviere · Accepted solution]

connect with root account on nabox

edit

/opt/graphite/lib/carbon/client.py

comment line 15

"from carbon.resolver import setUpRandomResolver"

and add just after this code just below

try:
    from carbon.resolver import setUpRandomResolver
except ImportError:
    setUpRandomResolver = None

save

restart cache-relay

systemctl start graphite_carbon_relay.service

[shashii82]

I haven't tried this for almost a week now.

[shashii82]

@Raviere Great, thank you Raviere! it did worked for me like a charm. My grafana was not working for almost two weeks.

root@harvestbox01:~# vi /opt/graphite/lib/carbon/client.py

root@harvestbox01:/opt/graphite/lib/carbon# systemctl status carbon-relay.service
* carbon-relay.service - graphite carbon relay
   Loaded: loaded (/etc/systemd/system/carbon-relay.service; enabled)
   Active: failed (Result: exit-code) since Mon 2018-09-24 17:23:14 CEST; 1 weeks 3 days ago
  Process: 646 ExecStart=/opt/graphite/bin/carbon-relay.py --pidfile=/opt/graphite/storage/carbon-relay-a.pid start (code=exited, status=1/FAILURE)
  Process: 605 ExecStartPre=/bin/rm -f /opt/graphite/storage/carbon-relay-a.pid (code=exited, status=0/SUCCESS)

Sep 24 17:23:13 harvestbox01 carbon-relay.py[646]: File "/opt/graphite/lib/carbon/service.py", line 183, in setupRelayProcessor
Sep 24 17:23:13 harvestbox01 carbon-relay.py[646]: from carbon.client import CarbonClientManager
Sep 24 17:23:13 harvestbox01 carbon-relay.py[646]: File "/opt/graphite/lib/carbon/client.py", line 15, in <module>
Sep 24 17:23:13 harvestbox01 carbon-relay.py[646]: from carbon.resolver import setUpRandomResolver
Sep 24 17:23:13 harvestbox01 carbon-relay.py[646]: File "/opt/graphite/lib/carbon/resolver.py", line 5, in <module>
Sep 24 17:23:13 harvestbox01 carbon-relay.py[646]: from twisted.internet._resolver import GAIResolver
Sep 24 17:23:13 harvestbox01 carbon-relay.py[646]: ImportError: No module named _resolver
Sep 24 17:23:14 harvestbox01 systemd[1]: carbon-relay.service: control process exited, code=exited status=1
Sep 24 17:23:14 harvestbox01 systemd[1]: Failed to start graphite carbon relay.
Sep 24 17:23:14 harvestbox01 systemd[1]: Unit carbon-relay.service entered failed state.

root@harvestbox01:/opt/graphite/lib/carbon# systemctl start carbon-relay.service

root@harvestbox01:/opt/graphite/lib/carbon# systemctl status carbon-relay.service
* carbon-relay.service - graphite carbon relay
   Loaded: loaded (/etc/systemd/system/carbon-relay.service; enabled)
   Active: active (running) since Fri 2018-10-05 10:57:56 CEST; 3s ago
  Process: 18145 ExecStart=/opt/graphite/bin/carbon-relay.py --pidfile=/opt/graphite/storage/carbon-relay-a.pid start (code=exited, status=0/SUCCESS)
  Process: 18143 ExecStartPre=/bin/rm -f /opt/graphite/storage/carbon-relay-a.pid (code=exited, status=0/SUCCESS)
 Main PID: 18150 (carbon-relay.py)
   CGroup: /system.slice/carbon-relay.service
           `-18150 /usr/bin/python /opt/graphite/bin/carbon-relay.py --pidfile=/opt/graphite/storage/carbon-relay-a.pid start

Oct 05 10:57:55 harvestbox01 systemd[1]: Starting graphite carbon relay...
Oct 05 10:57:56 harvestbox01 carbon-relay.py[18145]: /usr/local/lib/python2.7/dist-packages/cffi/model.py:531: UserWarning: 'point_conversion_form_t' has no valu...gned int'
Oct 05 10:57:56 harvestbox01 carbon-relay.py[18145]: % self._get_c_name())
Oct 05 10:57:56 harvestbox01 systemd[1]: Started graphite carbon relay.
Hint: Some lines were ellipsized, use -l to show in full.

[shashii82]

Does this have any impact to the reporting? Please find below error while starting the carbon-relay service:

----------------------------------------------------------------------------------------

Oct 05 10:57:56 harvestbox01 carbon-relay.py[18145]: /usr/local/lib/python2.7/dist-packages/cffi/model.py:531: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'
Oct 05 10:57:56 harvestbox01 carbon-relay.py[18145]: % self._get_c_name())

----------------------------------------------------------------------------------------

root@harvestbox01:/opt/graphite/lib/carbon# systemctl status carbon-relay.service -l
* carbon-relay.service - graphite carbon relay
   Loaded: loaded (/etc/systemd/system/carbon-relay.service; enabled)
   Active: active (running) since Fri 2018-10-05 10:57:56 CEST; 24min ago
  Process: 18145 ExecStart=/opt/graphite/bin/carbon-relay.py --pidfile=/opt/graphite/storage/carbon-relay-a.pid start (code=exited, status=0/SUCCESS)
  Process: 18143 ExecStartPre=/bin/rm -f /opt/graphite/storage/carbon-relay-a.pid (code=exited, status=0/SUCCESS)
 Main PID: 18150 (carbon-relay.py)
   CGroup: /system.slice/carbon-relay.service
           `-18150 /usr/bin/python /opt/graphite/bin/carbon-relay.py --pidfile=/opt/graphite/storage/carbon-relay-a.pid start

Oct 05 10:57:55 harvestbox01 systemd[1]: Starting graphite carbon relay...
Oct 05 10:57:56 harvestbox01 carbon-relay.py[18145]: /usr/local/lib/python2.7/dist-packages/cffi/model.py:531: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'
Oct 05 10:57:56 harvestbox01 carbon-relay.py[18145]: % self._get_c_name())
Oct 05 10:57:56 harvestbox01 systemd[1]: Started graphite carbon relay.

[yannb]

@shashii82, were you running the beta version as well?

[shashii82]

@yannb Yes, we did updated it with the recent beta version and it stopped working. But with the help of Raviere I did modified the /opt/graphite/lib/carbon/client.py and it started. However, I'm getting the another error while starting the carbon-relay service i.e.

carbon-relay.py[18145]: /usr/local/lib/python2.7/dist-packages/cffi/model.py:531: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'

carbon-relay.py[18145]: % self._get_c_name())

I hope this won't affect the reporting or anything.