Data Infrastructure Management Software Discussions

Data Infrastructure Management Software Discussions

NetApp Harvest Graphite issue

ChaitanUni
‎2019-09-29 10:13 AM

Hello All,

 

I am trying to setup Harvest/Grafana/Graphite in dockers (Kubernetes), i am currently havning issues with the setup, i did follow the setup guide,  but can't see any metrics on graphite.

Here is service for grphite on Kubernetes.

kind: Service
apiVersion: v1
metadata:
name: ocum-graphite
spec:
selector:
app: graphite
ports:
- name: graphiteport
port: 80
targetPort: 80
nodePort: 30080
- name: carbontextport
port: 2003
targetPort: 2003
nodePort: 30083
- name: carbonpickle
port: 2004
targetPort: 2004
nodePort: 30084
- name: carbonaggrplain
port: 2023
targetPort: 2023
nodePort: 30085
- name: carbonaggrpickle
port: 2024
targetPort: 2024
nodePort: 30086
- name: statsd
port: 8125
protocol: UDP
targetPort: 8125
nodePort: 30087
- name: statsdadmin
port: 8126
targetPort: 8126
nodePort: 30088

type: NodePort

 

Also i could see seom data on PV (Volumes)

 

aggregation-rules.conf carbon.amqp.conf dashboard.conf relay-rules.conf storage-aggregation.conf whitelist.conf
blacklist.conf carbon.conf graphTemplates.conf rewrite-rules.conf storage-schemas.conf

 

 

Some logs:

 

 

[2019-09-29 18:03:38] [NORMAL ] [main] Metrics will be submitted with graphite_root [netapp.perf.MH.NetAppProdVault]
[2019-09-29 18:03:38] [NORMAL ] [main] Using graphite_meta_metrics_root [netapp.poller.perf.MH.NetAppProdVault]
[2019-09-29 18:03:38] [NORMAL ] Creating output plugins
[2019-09-29 18:03:38] [NORMAL ] Created output plugins
[2019-09-29 18:03:38] [WARNING] [wafl_hya_per_aggr] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [nfsv4:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [cifs:vserver] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [DEBUG ] [workload_detail_volume] counter-list poller first poll at [2019-09-29 18:03:38]
[2019-09-29 18:03:38] [DEBUG ] [workload_detail_volume] data-list poller first poll at [2019-09-29 18:04:00]
[2019-09-29 18:03:38] [WARNING] [hostadapter] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [object_store_client_op] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [lun] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [DEBUG ] [workload_detail] counter-list poller first poll at [2019-09-29 18:03:38]
[2019-09-29 18:03:38] [DEBUG ] [workload_detail] data-list poller first poll at [2019-09-29 18:04:00]
[2019-09-29 18:03:38] [WARNING] [wafl_comp_aggr_vol_bin] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [nfsv3] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [processor] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [offbox_vscan_server] Object type does not exist in Data ONTAP release; skipping

[2019-09-29 18:10:00] [DEBUG ] [workload_detail] data-list poller next refresh at [2019-09-29 18:11:00]
[2019-09-29 18:10:00] [DEBUG ] [workload_detail_volume] plugin starting [cdot-workload]
[2019-09-29 18:10:00] [DEBUG ] [workload_detail_volume] data-list poller next refresh at [2019-09-29 18:11:00]
[2019-09-29 18:11:00] [DEBUG ] [workload_detail] plugin starting [cdot-workload]
[2019-09-29 18:11:00] [DEBUG ] [workload_detail] data-list poller next refresh at [2019-09-29 18:12:00]
[2019-09-29 18:11:00] [DEBUG ] [workload_detail_volume] plugin starting [cdot-workload]
[2019-09-29 18:11:00] [DEBUG ] [workload_detail_volume] data-list poller next refresh at [2019-09-29 18:12:00]
[2019-09-29 18:12:00] [DEBUG ] [workload_detail] plugin starting [cdot-workload]
[2019-09-29 18:12:00] [DEBUG ] [workload_detail] data-list poller next refresh at [2019-09-29 18:13:00]
[2019-09-29 18:12:00] [DEBUG ] [workload_detail_volume] plugin starting [cdot-workload]
[2019-09-29 18:12:00] [DEBUG ] [workload_detail_volume] data-list poller next refresh at [2019-09-29 18:13:00]

 

Any help on this much appreciated.

 

Thanks

Chaitan

Reply
9 Replies
0 Likes
9 REPLIES 9

Re: NetApp Harvest Graphite issue

yannb
NetApp
‎2019-09-30 05:41 AM

Reformatting for readability :-)

 

apiVersion: v1
kind: Service
metadata:
  name: ocum-graphite
spec:
  selector:
    app: graphite
  ports:
    - name: graphiteport
      port: 80
      targetPort: 80
      nodePort: 30080
    - name: carbontextport
      port: 2003
      targetPort: 2003
      nodePort: 30083
    - name: carbonpickle
      port: 2004
      targetPort: 2004
      nodePort: 30084
    - name: carbonaggrplain
      port: 2023
      targetPort: 2023
      nodePort: 30085
    - name: carbonaggrpickle
      port: 2024
      targetPort: 2024
      nodePort: 30086
    - name: statsd
      port: 8125
      protocol: UDP
      targetPort: 8125
      nodePort: 30087
    - name: statsdadmin
      port: 8126
      targetPort: 8126
      nodePort: 30088
      type: NodePort

Unfortunately I think we need more informations to troubleshoot this.

 

You declared everything as NodePort, than means you should have configured Harvest to send data on these ports, is that what you did ? How is Harvest running, as a container as well ? What is the kubernetes environment (on-prem or in the cloud)

 

Please describe your environment !

 

 

Reply
9 Replies
0 Likes

Re: NetApp Harvest Graphite issue

ChaitanUni
‎2019-09-30 05:51 AM

Hello 

Harvest ins installed on standalone RHEL 7, and we are running grafana (image : grafana)and grahpite  (Image:graphiteapp/graphite-statsd) as containers 

 

You declared everything as NodePort, than means you should have configured Harvest to send data on these ports,

Yes , i have done that:

graphite_server = 10.167.13.11
graphite_port =  30083

 

[root@snapp03 netapp-harvest]# /opt/netapp-harvest/netapp-manager -status
STATUS POLLER GROUP
############### #################### ##################
[RUNNING] NetAppProd MHA
[RUNNING] NetAppProdVa MHA

 

Here are some logs: poller

[2019-09-30 13:49:22] [NORMAL ] [main] Poller will use [password] authentication with username [netapp-harvest] and password [**********]
[2019-09-30 13:49:22] [DEBUG ] [sysinfo] Updating system-info cache
[2019-09-30 13:49:22] [WARNING] [connect] Setting HTTP/1.0 because reverse hostname resolution (IP -> hostname) fails. To enable HTTP/1.1 ensure reverse hostname resolution succeeds.
[2019-09-30 13:49:22] [DEBUG ] [sysinfo] DOT version is [NetApp Release 9.5P7] ([9] [5] [0])
[2019-09-30 13:49:22] [WARNING] [connect] Setting HTTP/1.0 because reverse hostname resolution (IP -> hostname) fails. To enable HTTP/1.1 ensure reverse hostname resolution succeeds.
[2019-09-30 13:49:23] [DEBUG ] [sysinfo] Cluster name is [NetAppProdVault]
[2019-09-30 13:49:23] [DEBUG ] [sysinfo] System info for system_name=[NetAppProdVault-01], model=[FAS2554], cpu=[4], memory=[18432]
[2019-09-30 13:49:23] [DEBUG ] [sysinfo] System info for system_name=[NetAppProdVault-02], model=[FAS2554], cpu=[4], memory=[18432]
[2019-09-30 13:49:23] [NORMAL ] [main] Collection of system info from [10.167.4.4] running [NetApp Release 9.5P7] successful.
[2019-09-30 13:49:23] [NORMAL ] [main] Found best-fit monitoring template (same generation and major release, minor same or less): [cdot-9.5.0.conf]
[2019-09-30 13:49:23] [NORMAL ] [main] Added and/or merged monitoring template [/opt/netapp-harvest/template/default/cdot-9.5.0.conf]
[2019-09-30 13:49:23] [NORMAL ] [main] Metrics will be submitted with graphite_root [netapp.perf.MH.NetAppProd]
[2019-09-30 13:49:23] [NORMAL ] [main] Using graphite_meta_metrics_root [netapp.poller.perf.MH.NetAppProdVt]
[2019-09-30 13:49:23] [NORMAL ] Creating output plugins

 

[2019-09-30 13:49:23] [WARNING] [offbox_vscan_server] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nfsv4_1:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [offbox_vscan] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [cifs:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [volume:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nfsv4] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nfsv3:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [lif] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [wafl_hya_sizer] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [workload] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [iscsi_lif] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [fcvi] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [token_manager] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [disk:constituent] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [workload_volume] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [fcp_lif] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [system:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [copy_manager] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [resource_headroom_aggr] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nic_common] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [volume] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [path] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [resource_headroom_cpu] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [wafl] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nfsv4_1] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [ext_cache_obj] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [NORMAL ] [main] Startup complete. Polling for new data every [60] seconds.

 

Thanks

Chaitan

Reply
9 Replies
0 Likes

Re: NetApp Harvest Graphite issue

yannb
NetApp
‎2019-09-30 06:10 AM

Are you able to connect to Graphite web interface on port 30080 ?

Reply
9 Replies
0 Likes

Re: NetApp Harvest Graphite issue

ChaitanUni
‎2019-09-30 06:52 AM

 

Hi,

 

Port: 30100

http://10.16.13.1:30100/

 

kubectl get all:

service/ocum-grafana NodePort 10.110.182.210 <none> 3000:30100/TCP 3d3h
service/ocum-graphite NodePort 10.104.200.238 <none> 80:30080/TCP,2003:30083/TCP,2004:30084/TCP,2023:30085/TCP,2024:30086/TCP,8125:30087/UDP,8126:30088/TCP 19h

 

Thanks

Reply
9 Replies
0 Likes

Re: NetApp Harvest Graphite issue

ChaitanUni
‎2019-09-30 07:00 AM

Sorry, yes i can access graphite:

http://hostname:30080/

Reply
9 Replies
0 Likes

Re: NetApp Harvest Graphite issue

yannb
NetApp
‎2019-09-30 07:27 AM

What do you see here ?

 

Screenshot 2019-09-30 at 16.25.53.png

Reply
9 Replies
0 Likes

Re: NetApp Harvest Graphite issue

ChaitanUni
‎2019-09-30 09:54 AM

I don't see NetApp there ;(

 

harvest1.PNGharvest2.PNG

Reply
9 Replies
0 Likes

Re: NetApp Harvest Graphite issue

yannb
NetApp
‎2019-10-01 12:04 PM

Can you share netapp-harvest.conf without the passwords?

What is 10.167.13.11 ?

Reply
9 Replies
0 Likes

Re: NetApp Harvest Graphite issue

ChaitanUni
‎2019-10-02 01:51 AM

afana_url = http://10.167.13.12:30100

#grafana_api_key = eyJrIjoiVXRYVjBBampPRTRDaGs0VFJrRW05WHhCNVIyVmowd2wiLCJuIjoibmV0YXBwLWhhcnZlc3QtMSIsImlkIjoxfQ==
#grafana_url = http://10.167.13.12:30100/api/dashboards/home


##
#### Default section to set defaults for any user created poller section
##
[default]
#====== Graphite server setup defaults ========================================
#graphite_enabled = 1
graphite_server = 10.167.13.11:30096        --- i have tried both option, added the port in extra line also.

@yannb wrote:

Can you share netapp-harvest.conf without the passwords?

What is 10.167.13.11 ?



#graphite_port = 30096
#graphite_proto = tcp
#normalized_xfer = mb_per_sec
#normalized_time = millisec
#graphite_root = default
#graphite_meta_metrics_root = default


#graphite_server = 10.167.13.11
#username = root
#password = root

# Harvest will send monthly statistics to AutoSupport, to disable set to "0"
send_autosupport_stats = 0

## If using ssl_cert (and not password auth)
## uncomment and populate next three lines
# auth_type = ssl_cert
# ssl_cert = INSERT_PEM_FILE_NAME_HERE
# ssl_key = INSERT_KEY_FILE_NAME_HERE

#====== NetApp OnCommand Insight output plugin setup defaults ==================
# To manage (create or update) OCI agent tokens and generate
# settings below, run script util/netapp-oci-setup
# oci_enabled = 1
# oci_url = https://INSERT_IP_OR_HOSTNAME_OF_OCI_SERVER_HERE
# oci_agent_token = INSERT_AGENT_TOKEN_FOR_OCI_SERVER_HERE
# oci_namespaces = volume:node,volume,system:node

##
#### Poller sections; Add one section for each cDOT cluster, 7-mode node, or OCUM server
#### If any keys are different from those in default duplicate them in the poller section to override.
##

[INSERT_CLUSTER_OR_CONTROLLER_NAME_HERE_EXACTLY_AS_SHOWN_FROM_CLI_PROMPT]
# hostname = INSERT_IP_ADDRESS_OR_HOSTNAME_OF_CONTROLLER_OR_CLUSTER_LIF_HERE
# group = INSERT_GROUP_IDENTIFIER_HERE
[NetAppProdVR]
host_type = Filer
hostname = 10.16.50.192
username = netapp-harvest
password = 123
group = MH


[NetAppProd]
host_type = Filer
hostname = 10.16.54.192
username = netapp-harvest
password = 123
group = MH

 

# [INSERT_OCUM_SERVER_NAME_HERE]
#[OCUM]
# hostname = 10.167.113.40
# group = nus
# password = 123
# host_type = OCUM
# data_update_freq = 900
# normalized_xfer = gb_per_sec

Reply
9 Replies
0 Likes
ASK A QUESTION
Try the new BETA Support Site!
Beta Support Site
Forums
Learn about the Community Get Started
Still have Questions Start a Discussion
Rules of the Community Read More
© 2020 NetApp
Let us know