Active IQ Unified Manager Discussions

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Active IQ Unified Manager Discussions

Ask a Question

NetApp Harvest Graphite issue

ChaitanUni
‎2019-09-29 10:13 AM
7,097 Views

Hello All,

 

I am trying to setup Harvest/Grafana/Graphite in dockers (Kubernetes), i am currently havning issues with the setup, i did follow the setup guide,  but can't see any metrics on graphite.

Here is service for grphite on Kubernetes.

kind: Service
apiVersion: v1
metadata:
name: ocum-graphite
spec:
selector:
app: graphite
ports:
- name: graphiteport
port: 80
targetPort: 80
nodePort: 30080
- name: carbontextport
port: 2003
targetPort: 2003
nodePort: 30083
- name: carbonpickle
port: 2004
targetPort: 2004
nodePort: 30084
- name: carbonaggrplain
port: 2023
targetPort: 2023
nodePort: 30085
- name: carbonaggrpickle
port: 2024
targetPort: 2024
nodePort: 30086
- name: statsd
port: 8125
protocol: UDP
targetPort: 8125
nodePort: 30087
- name: statsdadmin
port: 8126
targetPort: 8126
nodePort: 30088

type: NodePort

 

Also i could see seom data on PV (Volumes)

 

aggregation-rules.conf carbon.amqp.conf dashboard.conf relay-rules.conf storage-aggregation.conf whitelist.conf
blacklist.conf carbon.conf graphTemplates.conf rewrite-rules.conf storage-schemas.conf

 

 

Some logs:

 

 

[2019-09-29 18:03:38] [NORMAL ] [main] Metrics will be submitted with graphite_root [netapp.perf.MH.NetAppProdVault]
[2019-09-29 18:03:38] [NORMAL ] [main] Using graphite_meta_metrics_root [netapp.poller.perf.MH.NetAppProdVault]
[2019-09-29 18:03:38] [NORMAL ] Creating output plugins
[2019-09-29 18:03:38] [NORMAL ] Created output plugins
[2019-09-29 18:03:38] [WARNING] [wafl_hya_per_aggr] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [nfsv4:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [cifs:vserver] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [DEBUG ] [workload_detail_volume] counter-list poller first poll at [2019-09-29 18:03:38]
[2019-09-29 18:03:38] [DEBUG ] [workload_detail_volume] data-list poller first poll at [2019-09-29 18:04:00]
[2019-09-29 18:03:38] [WARNING] [hostadapter] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [object_store_client_op] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [lun] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [DEBUG ] [workload_detail] counter-list poller first poll at [2019-09-29 18:03:38]
[2019-09-29 18:03:38] [DEBUG ] [workload_detail] data-list poller first poll at [2019-09-29 18:04:00]
[2019-09-29 18:03:38] [WARNING] [wafl_comp_aggr_vol_bin] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [nfsv3] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [processor] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [offbox_vscan_server] Object type does not exist in Data ONTAP release; skipping

[2019-09-29 18:10:00] [DEBUG ] [workload_detail] data-list poller next refresh at [2019-09-29 18:11:00]
[2019-09-29 18:10:00] [DEBUG ] [workload_detail_volume] plugin starting [cdot-workload]
[2019-09-29 18:10:00] [DEBUG ] [workload_detail_volume] data-list poller next refresh at [2019-09-29 18:11:00]
[2019-09-29 18:11:00] [DEBUG ] [workload_detail] plugin starting [cdot-workload]
[2019-09-29 18:11:00] [DEBUG ] [workload_detail] data-list poller next refresh at [2019-09-29 18:12:00]
[2019-09-29 18:11:00] [DEBUG ] [workload_detail_volume] plugin starting [cdot-workload]
[2019-09-29 18:11:00] [DEBUG ] [workload_detail_volume] data-list poller next refresh at [2019-09-29 18:12:00]
[2019-09-29 18:12:00] [DEBUG ] [workload_detail] plugin starting [cdot-workload]
[2019-09-29 18:12:00] [DEBUG ] [workload_detail] data-list poller next refresh at [2019-09-29 18:13:00]
[2019-09-29 18:12:00] [DEBUG ] [workload_detail_volume] plugin starting [cdot-workload]
[2019-09-29 18:12:00] [DEBUG ] [workload_detail_volume] data-list poller next refresh at [2019-09-29 18:13:00]

 

Any help on this much appreciated.

 

Thanks

Chaitan

0 Kudos
9 REPLIES 9

yannb
NetApp
‎2019-09-30 05:41 AM
7,000 Views

Reformatting for readability 🙂

 

apiVersion: v1
kind: Service
metadata:
  name: ocum-graphite
spec:
  selector:
    app: graphite
  ports:
    - name: graphiteport
      port: 80
      targetPort: 80
      nodePort: 30080
    - name: carbontextport
      port: 2003
      targetPort: 2003
      nodePort: 30083
    - name: carbonpickle
      port: 2004
      targetPort: 2004
      nodePort: 30084
    - name: carbonaggrplain
      port: 2023
      targetPort: 2023
      nodePort: 30085
    - name: carbonaggrpickle
      port: 2024
      targetPort: 2024
      nodePort: 30086
    - name: statsd
      port: 8125
      protocol: UDP
      targetPort: 8125
      nodePort: 30087
    - name: statsdadmin
      port: 8126
      targetPort: 8126
      nodePort: 30088
      type: NodePort

Unfortunately I think we need more informations to troubleshoot this.

 

You declared everything as NodePort, than means you should have configured Harvest to send data on these ports, is that what you did ? How is Harvest running, as a container as well ? What is the kubernetes environment (on-prem or in the cloud)

 

Please describe your environment !

 

 

0 Kudos

ChaitanUni
‎2019-09-30 05:51 AM
In response to yannb
6,998 Views

Hello 

Harvest ins installed on standalone RHEL 7, and we are running grafana (image : grafana)and grahpite  (Image:graphiteapp/graphite-statsd) as containers 

 

You declared everything as NodePort, than means you should have configured Harvest to send data on these ports,

Yes , i have done that:

graphite_server = 10.167.13.11
graphite_port =  30083

 

[root@snapp03 netapp-harvest]# /opt/netapp-harvest/netapp-manager -status
STATUS POLLER GROUP
############### #################### ##################
[RUNNING] NetAppProd MHA
[RUNNING] NetAppProdVa MHA

 

Here are some logs: poller

[2019-09-30 13:49:22] [NORMAL ] [main] Poller will use [password] authentication with username [netapp-harvest] and password [**********]
[2019-09-30 13:49:22] [DEBUG ] [sysinfo] Updating system-info cache
[2019-09-30 13:49:22] [WARNING] [connect] Setting HTTP/1.0 because reverse hostname resolution (IP -> hostname) fails. To enable HTTP/1.1 ensure reverse hostname resolution succeeds.
[2019-09-30 13:49:22] [DEBUG ] [sysinfo] DOT version is [NetApp Release 9.5P7] ([9] [5] [0])
[2019-09-30 13:49:22] [WARNING] [connect] Setting HTTP/1.0 because reverse hostname resolution (IP -> hostname) fails. To enable HTTP/1.1 ensure reverse hostname resolution succeeds.
[2019-09-30 13:49:23] [DEBUG ] [sysinfo] Cluster name is [NetAppProdVault]
[2019-09-30 13:49:23] [DEBUG ] [sysinfo] System info for system_name=[NetAppProdVault-01], model=[FAS2554], cpu=[4], memory=[18432]
[2019-09-30 13:49:23] [DEBUG ] [sysinfo] System info for system_name=[NetAppProdVault-02], model=[FAS2554], cpu=[4], memory=[18432]
[2019-09-30 13:49:23] [NORMAL ] [main] Collection of system info from [10.167.4.4] running [NetApp Release 9.5P7] successful.
[2019-09-30 13:49:23] [NORMAL ] [main] Found best-fit monitoring template (same generation and major release, minor same or less): [cdot-9.5.0.conf]
[2019-09-30 13:49:23] [NORMAL ] [main] Added and/or merged monitoring template [/opt/netapp-harvest/template/default/cdot-9.5.0.conf]
[2019-09-30 13:49:23] [NORMAL ] [main] Metrics will be submitted with graphite_root [netapp.perf.MH.NetAppProd]
[2019-09-30 13:49:23] [NORMAL ] [main] Using graphite_meta_metrics_root [netapp.poller.perf.MH.NetAppProdVt]
[2019-09-30 13:49:23] [NORMAL ] Creating output plugins

 

[2019-09-30 13:49:23] [WARNING] [offbox_vscan_server] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nfsv4_1:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [offbox_vscan] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [cifs:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [volume:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nfsv4] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nfsv3:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [lif] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [wafl_hya_sizer] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [workload] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [iscsi_lif] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [fcvi] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [token_manager] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [disk:constituent] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [workload_volume] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [fcp_lif] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [system:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [copy_manager] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [resource_headroom_aggr] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nic_common] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [volume] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [path] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [resource_headroom_cpu] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [wafl] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nfsv4_1] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [ext_cache_obj] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [NORMAL ] [main] Startup complete. Polling for new data every [60] seconds.

 

Thanks

Chaitan

0 Kudos

yannb
NetApp
‎2019-09-30 06:10 AM
In response to ChaitanUni
6,988 Views

Are you able to connect to Graphite web interface on port 30080 ?

0 Kudos

ChaitanUni
‎2019-09-30 06:52 AM
In response to yannb
6,986 Views

 

Hi,

 

Port: 30100

http://10.16.13.1:30100/

 

kubectl get all:

service/ocum-grafana NodePort 10.110.182.210 <none> 3000:30100/TCP 3d3h
service/ocum-graphite NodePort 10.104.200.238 <none> 80:30080/TCP,2003:30083/TCP,2004:30084/TCP,2023:30085/TCP,2024:30086/TCP,8125:30087/UDP,8126:30088/TCP 19h

 

Thanks

0 Kudos

ChaitanUni
‎2019-09-30 07:00 AM
In response to ChaitanUni
6,982 Views

Sorry, yes i can access graphite:

http://hostname:30080/

0 Kudos

yannb
NetApp
‎2019-09-30 07:27 AM
In response to ChaitanUni
6,962 Views

What do you see here ?

 

Screenshot 2019-09-30 at 16.25.53.png

0 Kudos

ChaitanUni
‎2019-09-30 09:54 AM
In response to yannb
6,927 Views

I don't see NetApp there ;(

 

harvest1.PNGharvest2.PNG

0 Kudos

yannb
NetApp
‎2019-10-01 12:04 PM
In response to ChaitanUni
6,751 Views

Can you share netapp-harvest.conf without the passwords?

What is 10.167.13.11 ?

0 Kudos

ChaitanUni
‎2019-10-02 01:51 AM
In response to yannb
6,679 Views

afana_url = http://10.167.13.12:30100

#grafana_api_key = eyJrIjoiVXRYVjBBampPRTRDaGs0VFJrRW05WHhCNVIyVmowd2wiLCJuIjoibmV0YXBwLWhhcnZlc3QtMSIsImlkIjoxfQ==
#grafana_url = http://10.167.13.12:30100/api/dashboards/home


##
#### Default section to set defaults for any user created poller section
##
[default]
#====== Graphite server setup defaults ========================================
#graphite_enabled = 1
graphite_server = 10.167.13.11:30096        --- i have tried both option, added the port in extra line also.

@yannb wrote:

Can you share netapp-harvest.conf without the passwords?

What is 10.167.13.11 ?



#graphite_port = 30096
#graphite_proto = tcp
#normalized_xfer = mb_per_sec
#normalized_time = millisec
#graphite_root = default
#graphite_meta_metrics_root = default


#graphite_server = 10.167.13.11
#username = root
#password = root

# Harvest will send monthly statistics to AutoSupport, to disable set to "0"
send_autosupport_stats = 0

## If using ssl_cert (and not password auth)
## uncomment and populate next three lines
# auth_type = ssl_cert
# ssl_cert = INSERT_PEM_FILE_NAME_HERE
# ssl_key = INSERT_KEY_FILE_NAME_HERE

#====== NetApp OnCommand Insight output plugin setup defaults ==================
# To manage (create or update) OCI agent tokens and generate
# settings below, run script util/netapp-oci-setup
# oci_enabled = 1
# oci_url = https://INSERT_IP_OR_HOSTNAME_OF_OCI_SERVER_HERE
# oci_agent_token = INSERT_AGENT_TOKEN_FOR_OCI_SERVER_HERE
# oci_namespaces = volume:node,volume,system:node

##
#### Poller sections; Add one section for each cDOT cluster, 7-mode node, or OCUM server
#### If any keys are different from those in default duplicate them in the poller section to override.
##

[INSERT_CLUSTER_OR_CONTROLLER_NAME_HERE_EXACTLY_AS_SHOWN_FROM_CLI_PROMPT]
# hostname = INSERT_IP_ADDRESS_OR_HOSTNAME_OF_CONTROLLER_OR_CLUSTER_LIF_HERE
# group = INSERT_GROUP_IDENTIFIER_HERE
[NetAppProdVR]
host_type = Filer
hostname = 10.16.50.192
username = netapp-harvest
password = 123
group = MH


[NetAppProd]
host_type = Filer
hostname = 10.16.54.192
username = netapp-harvest
password = 123
group = MH

 

# [INSERT_OCUM_SERVER_NAME_HERE]
#[OCUM]
# hostname = 10.167.113.40
# group = nus
# password = 123
# host_type = OCUM
# data_update_freq = 900
# normalized_xfer = gb_per_sec

0 Kudos
Announcements
All Community Forums
Public