Hravest installation, Graphite / Carbon cache issue

[schmitz_peter]

Guys,

please help.

I needed a challenge, so I am trying to get Harvest running without using nabox.

* Installed a Debian server
* Installed Graphite / Grafana via Debian repos and Harvest 1.4.1 via zip as per "NetApp Harvest Installation and

Administration Guide 1.4" and "Quick Start: Installing Graphite and Grafana"

* Configured everything, started Apache, Carbon cache, Grafana and netapp-manager.

As "graphite-manage syncdb" doesn't work anymore I used "graphite-manage migrate --run-syncdb".

Everythig seems to work fine, polling is starting and I can login to the Grafana site, except the Graphite server is not available on port 2003, so Harvest can't send its data to it:

"Metrics NOT sent to Graphite. Cannot establish/maintain a connection to [http://localhost:2003/tcp]: failed without a specific library error"

Running

echo "test.metric 50 `date +%s`" | nc 127.0.0.1 2003

but it never comes back! The test.metric is created in "/var/lib/graphite/whisper/test", though!

The port is open, though:

# ss -an | grep 2003
tcp LISTEN 0 50 0.0.0.0:2003 0.0.0.0:*

The Graphite server is set to "http://localhost" in /opt/netapp-harvest/netapp-harvest.conf

I tried 127.0.0.1, localhost, FQDN, and short name by the way.

Connecting to port 81 the Graphite server serves "ERR_CONNECTION_REFUSED", but I guess this is an aftereffect.

Please tell me it's a simple Apache typo...

Thanks for any insights on this.

Peter

[vachagan_gratian]

Hi Peter,

Thanks for bringing this up. This is exactly the challenge that I am looking for right now 🙂 Although, with my experience I might not be able to help you immediately (and we might ask @yannb for input).

As a starting point: Did you check if everything is OK in your Carbon conf?

less /opt/graphite/conf/carbon.conf

What could be of interest are the variables LINE_RECEIVER_INTERFACE and LINE_RECEIVER_PORT.

[schmitz_peter]

Hi,

@vachagan_gratian wrote:

Hi Peter,

Thanks for bringing this up. This is exactly the challenge that I am looking for right now 🙂 Although, with my experience I might not be able to help you immediately (and we might ask @yannb for input).

As a starting point: Did you check if everything is OK in your Carbon conf?

less /opt/graphite/conf/carbon.conf

What could be of interest are the variables LINE_RECEIVER_INTERFACE and LINE_RECEIVER_PORT.

---

carbon.conf should be OK. I only changed ENABLE_LOGROTATION and MAX_CREATES_PER_MINUTE

Unfortunately, the documentation is lacking  a crucial part on page 8 of the quickstart guide: You need to add three "header set" directives to the apache config, just like so:

# egrep '^<Virtual|Header set' /etc/apache2/sites-enabled/apache2-graphite.conf
<VirtualHost *:81>
Header set Access-Control-Allow-Origin "*"
Header set Access-Control-Allow-Methods "GET, OPTIONS"
Header set Access-Control-Allow-Headers "origin, authorization, accept"

[...]

Unfortunately, I still get a CONNECTION REFUSED / BAD REQUEST on port 81.

Searching further

Peter

[yannb]

I think you made a mistake in the configuration for Harvest.

You put http://servername and this is incorrect, you should just put servername (or 127.0.0.1).

Port 2003 talking to carbon doesn't use HTTP by the way.

As far as I remember, nc should return to the terminal once the line has been sent, it seems a bit weird to me, but if the folder is created, then it's a good sign, it should work find once you fix Harvest configuration.

Port 81 is probably a different issue. Here is how I configure graphite in NAbox :

# Default configuration file for NAbox/Graphite

Listen 127.0.0.1:81

<IfModule !wsgi_module.c>
    LoadModule wsgi_module modules/mod_wsgi.so
</IfModule>

# Read http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGISocketPrefix
WSGISocketPrefix /var/run/apache2/wsgi

<VirtualHost 127.0.0.1:81>
        Header set Access-Control-Allow-Origin "*"
        Header set Access-Control-Allow-Methods "GET, OPTIONS"
        Header set Access-Control-Allow-Headers "origin, authorization, accept"

        ServerName graphite
        DocumentRoot "/opt/graphite/webapp"
        ErrorLog /opt/graphite/storage/log/webapp/error.log
        CustomLog /opt/graphite/storage/log/webapp/access.log common

        # I've found that an equal number of processes & threads tends
        # to show the best performance for Graphite (ymmv).
        WSGIDaemonProcess graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120
        WSGIProcessGroup graphite
        WSGIApplicationGroup %{GLOBAL}
        WSGIImportScript /opt/graphite/conf/graphite.wsgi process-group=graphite application-group=%{GLOBAL}

        # XXX You will need to create this file! There is a graphite.wsgi.example
        # file in this directory that you can safely use, just copy it to graphite.wgsi
        WSGIScriptAlias / /opt/graphite/conf/graphite.wsgi

        Alias /content/ /opt/graphite/webapp/content/
        Alias /graphite/static/ /opt/graphite/static/

        <Location "/content/">
                SetHandler None
        </Location>

        # XXX In order for the django admin site media to work you
        # must change @DJANGO_ROOT@ to be the path to your django
        # installation, which is probably something like:
        # /usr/lib/python2.6/site-packages/django
        Alias /media/ "@DJANGO_ROOT@/contrib/admin/media/"
        <Location "/media/">
                SetHandler None
        </Location>

        # The graphite.wsgi file has to be accessible by apache. It won't
        # be visible to clients because of the DocumentRoot though.
        <Directory /opt/graphite/>
                Require all granted
                Order deny,allow
                Allow from all
        </Directory>

</VirtualHost>
# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

[schmitz_peter]

---

Hi Yann,

thanks for hopping in.

@yannb wrote:

I think you made a mistake in the configuration for Harvest.

You put http://servername and this is incorrect, you should just put servername (or 127.0.0.1).

Port 2003 talking to carbon doesn't use HTTP by the way.

 You are right. There's an error in the config file. I corrected it to 127.0.0.1

As far as I remember, nc should return to the terminal once the line has been sent, it seems a bit weird to me, but if the folder is created, then it's a good sign, it should work find once you fix Harvest configuration.

Port 81 is probably a different issue. Here is how I configure graphite in NAbox :

[...]

Thanks for this. As far as I can see my configuration looks OK, so I am still at a loss here.

nc still comes back with 

# echo "blah.metric 50 `date +%s`" | nc 127.0.0.1 81
HTTP/1.1 400 Bad Request
Date: Thu, 04 Oct 2018 14:01:52 GMT
Server: Apache/2.4.34 (Debian)
Content-Length: 321
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
<hr>
<address>Apache/2.4.34 (Debian) Server at mysecretserver Port 81</address>
</body></html>

Maybe I'll come back to you and the WebEx session ;D

Cheers

Peter

ssds

[yannb]

You're talking to the wrong process.

metrics go to port 2003, not 81. 81 is just for querying the metrics and display graphite web interface.

It should be :

echo "blah.metric 50 `date +%s`" | nc 127.0.0.1 2003

[schmitz_peter]

---

@yannb wrote:

You're talking to the wrong process.

metrics go to port 2003, not 81. 81 is just for querying the metrics and display graphite web interface.

It should be :

echo "blah.metric 50 `date +%s`" | nc 127.0.0.1 2003

---

Yeah, either way, 2003 doesn't come back and conencting the browser to 81 gives back a "CONNECTION REFUSED".

Funny enough, the metrix seem to be sent to the db. I'm getting get pretty graphs in Grafana and all. I just can't connect to Graphite.

Oh well, I'm going home, now....

Thanks a lot for your input!

Peter

[yannb]

It's probably an apache server configuration issue.

My config file, for example, locks the listen directive 127.0.0.1

Listen 127.0.0.1:81

Because in effect, I have a rveerse proxy that redirects /graphite to this port internally.

If you share your complete configuration file I will probably be able to figure out why you get connection refused.

[schmitz_peter]

Hi Yann,

here's the vhost config:

# cat /etc/apache2/sites-enabled/apache2-graphite.conf
<VirtualHost 127.0.0.1:81>
  Header set Access-Control-Allow-Origin "*"
  Header set Access-Control-Allow-Methods "GET, OPTIONS"
  Header set Access-Control-Allow-Headers "origin, authorization, accept"

  WSGIDaemonProcess _graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120 user=_graphite group=_graphite
  WSGIProcessGroup _graphite
  WSGIImportScript /usr/share/graphite-web/graphite.wsgi process-group=_graphite application-group=%{GLOBAL}
  WSGIScriptAlias / /usr/share/graphite-web/graphite.wsgi

  Alias /static/ /usr/share/graphite-web/static/
  <Location "/static/">
    SetHandler None
  </Location>

  ErrorLog ${APACHE_LOG_DIR}/graphite-web_error.log

  # Possible values include: debug, info, notice, warn, error, crit,
  # alert, emerg.
  LogLevel warn

  CustomLog ${APACHE_LOG_DIR}/graphite-web_access.log combined

</VirtualHost>

...and as recommended by the manual I also changed the Apache ports.conf:

Listen 81

<IfModule ssl_module>
  Listen 443
</IfModule>

<IfModule mod_gnutls.c>
  Listen 443
</IfModule>

When I set "Listen 127.0.0.1:81" in front of the vhost definition I get a "port already in use" error (yes, I am no Apache pro....)

Cheers

Peter

[yannb]

Do you have your firewall opened for port 81?

Is there anything in graphite-web_error.log ?

[schmitz_peter]

Hi Yann,

I uninstalled iptables. I will talk to the network guys next.

graphite-web_error.log is empty.

telnet says:

# telnet localhost 81
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET /
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Frameset//EN" "http://www.w3.org/TR/html4/frameset.dtd">
<!-- Copyright 2008 Orbitz WorldWide

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. -->

<html>
<head>
<title>Graphite Browser</title>
</head>

<frameset rows="80,*" frameborder="1" border="1">
<frame src="/browserheader" name="Header" id='header' scrolling="no" noresize="true" />

<frame src="/composer?" name="content" id="composerFrame"/>

</frameset>
</html>

ss says:
tcp LISTEN 0 128 *:81 *:*

telnet from external says "Connection refused", though.

# nmap localhost
Starting Nmap 7.70 ( https://nmap.org ) at 2018-10-05 10:22 CEST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.0000080s latency).
Not shown: 993 closed ports
PORT STATE SERVICE
22/tcp open ssh
81/tcp open hosts2-ns
111/tcp open rpcbind
443/tcp open https
2003/tcp open finger
2004/tcp open mailbox
7002/tcp open afs3-prserver

# nmap <from external to server>
Starting Nmap 7.70 ( https://nmap.org ) at 2018-10-05 10:27 CEST
Nmap scan report for <blah>
Host is up (0.00062s latency).
Not shown: 998 closed ports
PORT STATE SERVICE
22/tcp open ssh
443/tcp open https

Nmap done: 1 IP address (1 host up) scanned in 0.06 seconds

There's no one using the graphite.db:

# fuser -v /var/lib/graphite/graphite.db
#

Peter

[schmitz_peter]

Yay!

After the network guys opened port 81 I now can connect via web browser to the Graphite server and I get the Graphite composer! (Insert rolling eyes smiley here)

What's still not working is the basic nc test:

# echo "test.metric 50 `date +%s`" | nc 127.0.0.1 2003

Still, we are getting closer.

At least, now I can imagine and appreciate your work on nabox, which I really like.

It is challenging and somewhat rewarding to get this beast running.

@vachagan_gratian You still with us? When this thing is running I will write up some documentation for setting it up on Debian. Maybe it's of use for you.

Or, you can simply use Yann's nabox. I guess version 2.5 is on the way.

Peter

Peter

[yannb · Accepted solution]

Actually it makes sense as you're only listening to 127.0.0.1.

Change 127.0.0.1:81 to *:81

[schmitz_peter]

---

@yannb wrote:

Actually it makes sense as you're only listening to 127.0.0.1.

Change 127.0.0.1:81 to *:81

---

Did that and everything seems to be working fine now.

Except this "nc" thing, but whispher files are created and I can see a lot of graphs in Grafana, except cluster throughput, but I will investigate later.

Maybe I can mark this as resolved.

Anyway, thanks a lot for your thoughts. I will definitively test nabox with the next version.

Kind regards

Peter

[schmitz_peter]

Epilogue:

* Don't install Grafana from the Debian sources. It's an old package from 2015 (2.6.0)

* After installing the latest package from https://packagecloud.io/grafana/stable everything looked find , *except* the "NetApp Detail: LUN" page. I get a "Templating init failed [object Object]" error. That worked well with the old Grafana version. Oh, well.

* The Harvest poller seems to die sometimes, have to investigate further.

Kind regards

Peter

[vachagan_gratian]

Hi Peter,

Sorry, I couldn't reply erlier. It would be great if you shared your documentation. I am planning to setup a webpage with tips for installing Harvest on different platforms, we could include your input there!

Vachagan

[schmitz_peter]

Hi @vachagan_gratian,

Sorry, I couldn't reply erlier. It would be great if you shared your documentation. I am planning to setup a webpage with tips for installing Harvest on different platforms, we could include your input there!

---

No problem. I have a document ready. Just tell me where to send it. It's Libreoffice's ODT format at the moment, but I could convert it to something else.

Cheers

Peter

[vachagan_gratian]

Great! ODT is fine! I'll DM you my email address.