NetApp Harvest Graphite issue

ChaitanUni · ‎2019-09-29

Hello All,

I am trying to setup Harvest/Grafana/Graphite in dockers (Kubernetes), i am currently havning issues with the setup, i did follow the setup guide, but can't see any metrics on graphite.

Here is service for grphite on Kubernetes.

kind: Service
apiVersion: v1
metadata:
name: ocum-graphite
spec:
selector:
app: graphite
ports:
- name: graphiteport
port: 80
targetPort: 80
nodePort: 30080
- name: carbontextport
port: 2003
targetPort: 2003
nodePort: 30083
- name: carbonpickle
port: 2004
targetPort: 2004
nodePort: 30084
- name: carbonaggrplain
port: 2023
targetPort: 2023
nodePort: 30085
- name: carbonaggrpickle
port: 2024
targetPort: 2024
nodePort: 30086
- name: statsd
port: 8125
protocol: UDP
targetPort: 8125
nodePort: 30087
- name: statsdadmin
port: 8126
targetPort: 8126
nodePort: 30088

type: NodePort

Also i could see seom data on PV (Volumes)

aggregation-rules.conf carbon.amqp.conf dashboard.conf relay-rules.conf storage-aggregation.conf whitelist.conf
blacklist.conf carbon.conf graphTemplates.conf rewrite-rules.conf storage-schemas.conf

Some logs:

[2019-09-29 18:03:38] [NORMAL ] [main] Metrics will be submitted with graphite_root [netapp.perf.MH.NetAppProdVault]
[2019-09-29 18:03:38] [NORMAL ] [main] Using graphite_meta_metrics_root [netapp.poller.perf.MH.NetAppProdVault]
[2019-09-29 18:03:38] [NORMAL ] Creating output plugins
[2019-09-29 18:03:38] [NORMAL ] Created output plugins
[2019-09-29 18:03:38] [WARNING] [wafl_hya_per_aggr] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [nfsv4:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [cifs:vserver] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [DEBUG ] [workload_detail_volume] counter-list poller first poll at [2019-09-29 18:03:38]
[2019-09-29 18:03:38] [DEBUG ] [workload_detail_volume] data-list poller first poll at [2019-09-29 18:04:00]
[2019-09-29 18:03:38] [WARNING] [hostadapter] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [object_store_client_op] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [lun] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [DEBUG ] [workload_detail] counter-list poller first poll at [2019-09-29 18:03:38]
[2019-09-29 18:03:38] [DEBUG ] [workload_detail] data-list poller first poll at [2019-09-29 18:04:00]
[2019-09-29 18:03:38] [WARNING] [wafl_comp_aggr_vol_bin] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [nfsv3] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [processor] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [offbox_vscan_server] Object type does not exist in Data ONTAP release; skipping

[2019-09-29 18:10:00] [DEBUG ] [workload_detail] data-list poller next refresh at [2019-09-29 18:11:00]
[2019-09-29 18:10:00] [DEBUG ] [workload_detail_volume] plugin starting [cdot-workload]
[2019-09-29 18:10:00] [DEBUG ] [workload_detail_volume] data-list poller next refresh at [2019-09-29 18:11:00]
[2019-09-29 18:11:00] [DEBUG ] [workload_detail] plugin starting [cdot-workload]
[2019-09-29 18:11:00] [DEBUG ] [workload_detail] data-list poller next refresh at [2019-09-29 18:12:00]
[2019-09-29 18:11:00] [DEBUG ] [workload_detail_volume] plugin starting [cdot-workload]
[2019-09-29 18:11:00] [DEBUG ] [workload_detail_volume] data-list poller next refresh at [2019-09-29 18:12:00]
[2019-09-29 18:12:00] [DEBUG ] [workload_detail] plugin starting [cdot-workload]
[2019-09-29 18:12:00] [DEBUG ] [workload_detail] data-list poller next refresh at [2019-09-29 18:13:00]
[2019-09-29 18:12:00] [DEBUG ] [workload_detail_volume] plugin starting [cdot-workload]
[2019-09-29 18:12:00] [DEBUG ] [workload_detail_volume] data-list poller next refresh at [2019-09-29 18:13:00]

Any help on this much appreciated.

Thanks

Chaitan

yannb · ‎2019-09-30

Reformatting for readability 🙂

apiVersion: v1
kind: Service
metadata:
  name: ocum-graphite
spec:
  selector:
    app: graphite
  ports:
    - name: graphiteport
      port: 80
      targetPort: 80
      nodePort: 30080
    - name: carbontextport
      port: 2003
      targetPort: 2003
      nodePort: 30083
    - name: carbonpickle
      port: 2004
      targetPort: 2004
      nodePort: 30084
    - name: carbonaggrplain
      port: 2023
      targetPort: 2023
      nodePort: 30085
    - name: carbonaggrpickle
      port: 2024
      targetPort: 2024
      nodePort: 30086
    - name: statsd
      port: 8125
      protocol: UDP
      targetPort: 8125
      nodePort: 30087
    - name: statsdadmin
      port: 8126
      targetPort: 8126
      nodePort: 30088
      type: NodePort

Unfortunately I think we need more informations to troubleshoot this.

You declared everything as NodePort, than means you should have configured Harvest to send data on these ports, is that what you did ? How is Harvest running, as a container as well ? What is the kubernetes environment (on-prem or in the cloud)

Please describe your environment !

ChaitanUni · ‎2019-09-30

Hello

Harvest ins installed on standalone RHEL 7, and we are running grafana (image : grafana)and grahpite (Image:graphiteapp/graphite-statsd) as containers

You declared everything as NodePort, than means you should have configured Harvest to send data on these ports,

Yes , i have done that:

graphite_server = 10.167.13.11
graphite_port = 30083

[root@snapp03 netapp-harvest]# /opt/netapp-harvest/netapp-manager -status
STATUS POLLER GROUP
############### #################### ##################
[RUNNING] NetAppProd MHA
[RUNNING] NetAppProdVa MHA

Here are some logs: poller

[2019-09-30 13:49:22] [NORMAL ] [main] Poller will use [password] authentication with username [netapp-harvest] and password [**********]
[2019-09-30 13:49:22] [DEBUG ] [sysinfo] Updating system-info cache
[2019-09-30 13:49:22] [WARNING] [connect] Setting HTTP/1.0 because reverse hostname resolution (IP -> hostname) fails. To enable HTTP/1.1 ensure reverse hostname resolution succeeds.
[2019-09-30 13:49:22] [DEBUG ] [sysinfo] DOT version is [NetApp Release 9.5P7] ([9] [5] [0])
[2019-09-30 13:49:22] [WARNING] [connect] Setting HTTP/1.0 because reverse hostname resolution (IP -> hostname) fails. To enable HTTP/1.1 ensure reverse hostname resolution succeeds.
[2019-09-30 13:49:23] [DEBUG ] [sysinfo] Cluster name is [NetAppProdVault]
[2019-09-30 13:49:23] [DEBUG ] [sysinfo] System info for system_name=[NetAppProdVault-01], model=[FAS2554], cpu=[4], memory=[18432]
[2019-09-30 13:49:23] [DEBUG ] [sysinfo] System info for system_name=[NetAppProdVault-02], model=[FAS2554], cpu=[4], memory=[18432]
[2019-09-30 13:49:23] [NORMAL ] [main] Collection of system info from [10.167.4.4] running [NetApp Release 9.5P7] successful.
[2019-09-30 13:49:23] [NORMAL ] [main] Found best-fit monitoring template (same generation and major release, minor same or less): [cdot-9.5.0.conf]
[2019-09-30 13:49:23] [NORMAL ] [main] Added and/or merged monitoring template [/opt/netapp-harvest/template/default/cdot-9.5.0.conf]
[2019-09-30 13:49:23] [NORMAL ] [main] Metrics will be submitted with graphite_root [netapp.perf.MH.NetAppProd]
[2019-09-30 13:49:23] [NORMAL ] [main] Using graphite_meta_metrics_root [netapp.poller.perf.MH.NetAppProdVt]
[2019-09-30 13:49:23] [NORMAL ] Creating output plugins

[2019-09-30 13:49:23] [WARNING] [offbox_vscan_server] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nfsv4_1:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [offbox_vscan] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [cifs:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [volume:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nfsv4] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nfsv3:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [lif] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [wafl_hya_sizer] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [workload] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [iscsi_lif] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [fcvi] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [token_manager] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [disk:constituent] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [workload_volume] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [fcp_lif] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [system:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [copy_manager] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [resource_headroom_aggr] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nic_common] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [volume] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [path] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [resource_headroom_cpu] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [wafl] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nfsv4_1] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [ext_cache_obj] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [NORMAL ] [main] Startup complete. Polling for new data every [60] seconds.

Thanks

Chaitan

yannb · ‎2019-09-30

Are you able to connect to Graphite web interface on port 30080 ?

ChaitanUni · ‎2019-09-30

Hi,

Port: 30100

http://10.16.13.1:30100/

kubectl get all:

service/ocum-grafana NodePort 10.110.182.210 <none> 3000:30100/TCP 3d3h
service/ocum-graphite NodePort 10.104.200.238 <none> 80:30080/TCP,2003:30083/TCP,2004:30084/TCP,2023:30085/TCP,2024:30086/TCP,8125:30087/UDP,8126:30088/TCP 19h

Thanks

ChaitanUni · ‎2019-09-30

Sorry, yes i can access graphite:

http://hostname:30080/

yannb · ‎2019-09-30

What do you see here ?

ChaitanUni · ‎2019-09-30

I don't see NetApp there ;(

yannb · ‎2019-10-01

Can you share netapp-harvest.conf without the passwords?

What is 10.167.13.11 ?

ChaitanUni · ‎2019-10-02

afana_url = http://10.167.13.12:30100

#grafana_api_key = eyJrIjoiVXRYVjBBampPRTRDaGs0VFJrRW05WHhCNVIyVmowd2wiLCJuIjoibmV0YXBwLWhhcnZlc3QtMSIsImlkIjoxfQ==
#grafana_url = http://10.167.13.12:30100/api/dashboards/home

##
#### Default section to set defaults for any user created poller section
##
[default]
#====== Graphite server setup defaults ========================================
#graphite_enabled = 1
graphite_server = 10.167.13.11:30096 --- i have tried both option, added the port in extra line also.

@yannb wrote:

Can you share netapp-harvest.conf without the passwords?

What is 10.167.13.11 ?

#graphite_port = 30096
#graphite_proto = tcp
#normalized_xfer = mb_per_sec
#normalized_time = millisec
#graphite_root = default
#graphite_meta_metrics_root = default

#graphite_server = 10.167.13.11
#username = root
#password = root

# Harvest will send monthly statistics to AutoSupport, to disable set to "0"
send_autosupport_stats = 0

## If using ssl_cert (and not password auth)
## uncomment and populate next three lines
# auth_type = ssl_cert
# ssl_cert = INSERT_PEM_FILE_NAME_HERE
# ssl_key = INSERT_KEY_FILE_NAME_HERE

#====== NetApp OnCommand Insight output plugin setup defaults ==================
# To manage (create or update) OCI agent tokens and generate
# settings below, run script util/netapp-oci-setup
# oci_enabled = 1
# oci_url = https://INSERT_IP_OR_HOSTNAME_OF_OCI_SERVER_HERE
# oci_agent_token = INSERT_AGENT_TOKEN_FOR_OCI_SERVER_HERE
# oci_namespaces = volume:node,volume,system:node

##
#### Poller sections; Add one section for each cDOT cluster, 7-mode node, or OCUM server
#### If any keys are different from those in default duplicate them in the poller section to override.
##

[INSERT_CLUSTER_OR_CONTROLLER_NAME_HERE_EXACTLY_AS_SHOWN_FROM_CLI_PROMPT]
# hostname = INSERT_IP_ADDRESS_OR_HOSTNAME_OF_CONTROLLER_OR_CLUSTER_LIF_HERE
# group = INSERT_GROUP_IDENTIFIER_HERE
[NetAppProdVR]
host_type = Filer
hostname = 10.16.50.192
username = netapp-harvest
password = 123
group = MH

[NetAppProd]
host_type = Filer
hostname = 10.16.54.192
username = netapp-harvest
password = 123
group = MH

# [INSERT_OCUM_SERVER_NAME_HERE]
#[OCUM]
# hostname = 10.167.113.40
# group = nus
# password = 123
# host_type = OCUM
# data_update_freq = 900
# normalized_xfer = gb_per_sec

NetApp Harvest Graphite issue

Re: NetApp Harvest Graphite issue

Re: NetApp Harvest Graphite issue

Re: NetApp Harvest Graphite issue

Re: NetApp Harvest Graphite issue

Re: NetApp Harvest Graphite issue

Re: NetApp Harvest Graphite issue

Re: NetApp Harvest Graphite issue

Re: NetApp Harvest Graphite issue

Re: NetApp Harvest Graphite issue