Active IQ Unified Manager Discussions
Active IQ Unified Manager Discussions
Hello All,
I am trying to setup Harvest/Grafana/Graphite in dockers (Kubernetes), i am currently havning issues with the setup, i did follow the setup guide, but can't see any metrics on graphite.
Here is service for grphite on Kubernetes.
kind: Service
apiVersion: v1
metadata:
name: ocum-graphite
spec:
selector:
app: graphite
ports:
- name: graphiteport
port: 80
targetPort: 80
nodePort: 30080
- name: carbontextport
port: 2003
targetPort: 2003
nodePort: 30083
- name: carbonpickle
port: 2004
targetPort: 2004
nodePort: 30084
- name: carbonaggrplain
port: 2023
targetPort: 2023
nodePort: 30085
- name: carbonaggrpickle
port: 2024
targetPort: 2024
nodePort: 30086
- name: statsd
port: 8125
protocol: UDP
targetPort: 8125
nodePort: 30087
- name: statsdadmin
port: 8126
targetPort: 8126
nodePort: 30088
type: NodePort
Also i could see seom data on PV (Volumes)
aggregation-rules.conf carbon.amqp.conf dashboard.conf relay-rules.conf storage-aggregation.conf whitelist.conf
blacklist.conf carbon.conf graphTemplates.conf rewrite-rules.conf storage-schemas.conf
Some logs:
[2019-09-29 18:03:38] [NORMAL ] [main] Metrics will be submitted with graphite_root [netapp.perf.MH.NetAppProdVault]
[2019-09-29 18:03:38] [NORMAL ] [main] Using graphite_meta_metrics_root [netapp.poller.perf.MH.NetAppProdVault]
[2019-09-29 18:03:38] [NORMAL ] Creating output plugins
[2019-09-29 18:03:38] [NORMAL ] Created output plugins
[2019-09-29 18:03:38] [WARNING] [wafl_hya_per_aggr] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [nfsv4:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [cifs:vserver] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [DEBUG ] [workload_detail_volume] counter-list poller first poll at [2019-09-29 18:03:38]
[2019-09-29 18:03:38] [DEBUG ] [workload_detail_volume] data-list poller first poll at [2019-09-29 18:04:00]
[2019-09-29 18:03:38] [WARNING] [hostadapter] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [object_store_client_op] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [lun] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [DEBUG ] [workload_detail] counter-list poller first poll at [2019-09-29 18:03:38]
[2019-09-29 18:03:38] [DEBUG ] [workload_detail] data-list poller first poll at [2019-09-29 18:04:00]
[2019-09-29 18:03:38] [WARNING] [wafl_comp_aggr_vol_bin] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [nfsv3] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [processor] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:03:38] [WARNING] [offbox_vscan_server] Object type does not exist in Data ONTAP release; skipping
[2019-09-29 18:10:00] [DEBUG ] [workload_detail] data-list poller next refresh at [2019-09-29 18:11:00]
[2019-09-29 18:10:00] [DEBUG ] [workload_detail_volume] plugin starting [cdot-workload]
[2019-09-29 18:10:00] [DEBUG ] [workload_detail_volume] data-list poller next refresh at [2019-09-29 18:11:00]
[2019-09-29 18:11:00] [DEBUG ] [workload_detail] plugin starting [cdot-workload]
[2019-09-29 18:11:00] [DEBUG ] [workload_detail] data-list poller next refresh at [2019-09-29 18:12:00]
[2019-09-29 18:11:00] [DEBUG ] [workload_detail_volume] plugin starting [cdot-workload]
[2019-09-29 18:11:00] [DEBUG ] [workload_detail_volume] data-list poller next refresh at [2019-09-29 18:12:00]
[2019-09-29 18:12:00] [DEBUG ] [workload_detail] plugin starting [cdot-workload]
[2019-09-29 18:12:00] [DEBUG ] [workload_detail] data-list poller next refresh at [2019-09-29 18:13:00]
[2019-09-29 18:12:00] [DEBUG ] [workload_detail_volume] plugin starting [cdot-workload]
[2019-09-29 18:12:00] [DEBUG ] [workload_detail_volume] data-list poller next refresh at [2019-09-29 18:13:00]
Any help on this much appreciated.
Thanks
Chaitan
Reformatting for readability 🙂
apiVersion: v1 kind: Service metadata: name: ocum-graphite spec: selector: app: graphite ports: - name: graphiteport port: 80 targetPort: 80 nodePort: 30080 - name: carbontextport port: 2003 targetPort: 2003 nodePort: 30083 - name: carbonpickle port: 2004 targetPort: 2004 nodePort: 30084 - name: carbonaggrplain port: 2023 targetPort: 2023 nodePort: 30085 - name: carbonaggrpickle port: 2024 targetPort: 2024 nodePort: 30086 - name: statsd port: 8125 protocol: UDP targetPort: 8125 nodePort: 30087 - name: statsdadmin port: 8126 targetPort: 8126 nodePort: 30088 type: NodePort
Unfortunately I think we need more informations to troubleshoot this.
You declared everything as NodePort, than means you should have configured Harvest to send data on these ports, is that what you did ? How is Harvest running, as a container as well ? What is the kubernetes environment (on-prem or in the cloud)
Please describe your environment !
Hello
Harvest ins installed on standalone RHEL 7, and we are running grafana (image : grafana)and grahpite (Image:graphiteapp/graphite-statsd) as containers
You declared everything as NodePort, than means you should have configured Harvest to send data on these ports,
Yes , i have done that:
graphite_server = 10.167.13.11
graphite_port = 30083
[root@snapp03 netapp-harvest]# /opt/netapp-harvest/netapp-manager -status
STATUS POLLER GROUP
############### #################### ##################
[RUNNING] NetAppProd MHA
[RUNNING] NetAppProdVa MHA
Here are some logs: poller
[2019-09-30 13:49:22] [NORMAL ] [main] Poller will use [password] authentication with username [netapp-harvest] and password [**********]
[2019-09-30 13:49:22] [DEBUG ] [sysinfo] Updating system-info cache
[2019-09-30 13:49:22] [WARNING] [connect] Setting HTTP/1.0 because reverse hostname resolution (IP -> hostname) fails. To enable HTTP/1.1 ensure reverse hostname resolution succeeds.
[2019-09-30 13:49:22] [DEBUG ] [sysinfo] DOT version is [NetApp Release 9.5P7] ([9] [5] [0])
[2019-09-30 13:49:22] [WARNING] [connect] Setting HTTP/1.0 because reverse hostname resolution (IP -> hostname) fails. To enable HTTP/1.1 ensure reverse hostname resolution succeeds.
[2019-09-30 13:49:23] [DEBUG ] [sysinfo] Cluster name is [NetAppProdVault]
[2019-09-30 13:49:23] [DEBUG ] [sysinfo] System info for system_name=[NetAppProdVault-01], model=[FAS2554], cpu=[4], memory=[18432]
[2019-09-30 13:49:23] [DEBUG ] [sysinfo] System info for system_name=[NetAppProdVault-02], model=[FAS2554], cpu=[4], memory=[18432]
[2019-09-30 13:49:23] [NORMAL ] [main] Collection of system info from [10.167.4.4] running [NetApp Release 9.5P7] successful.
[2019-09-30 13:49:23] [NORMAL ] [main] Found best-fit monitoring template (same generation and major release, minor same or less): [cdot-9.5.0.conf]
[2019-09-30 13:49:23] [NORMAL ] [main] Added and/or merged monitoring template [/opt/netapp-harvest/template/default/cdot-9.5.0.conf]
[2019-09-30 13:49:23] [NORMAL ] [main] Metrics will be submitted with graphite_root [netapp.perf.MH.NetAppProd]
[2019-09-30 13:49:23] [NORMAL ] [main] Using graphite_meta_metrics_root [netapp.poller.perf.MH.NetAppProdVt]
[2019-09-30 13:49:23] [NORMAL ] Creating output plugins
[2019-09-30 13:49:23] [WARNING] [offbox_vscan_server] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nfsv4_1:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [offbox_vscan] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [cifs:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [volume:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nfsv4] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nfsv3:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [lif] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [wafl_hya_sizer] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [workload] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [iscsi_lif] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [fcvi] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [token_manager] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [disk:constituent] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [workload_volume] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [fcp_lif] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [system:node] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [copy_manager] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [resource_headroom_aggr] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nic_common] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [volume] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [path] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [resource_headroom_cpu] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [wafl] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [nfsv4_1] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [WARNING] [ext_cache_obj] Object type does not exist in Data ONTAP release; skipping
[2019-09-30 13:49:23] [NORMAL ] [main] Startup complete. Polling for new data every [60] seconds.
Thanks
Chaitan
Are you able to connect to Graphite web interface on port 30080 ?
Hi,
Port: 30100
kubectl get all:
service/ocum-grafana NodePort 10.110.182.210 <none> 3000:30100/TCP 3d3h
service/ocum-graphite NodePort 10.104.200.238 <none> 80:30080/TCP,2003:30083/TCP,2004:30084/TCP,2023:30085/TCP,2024:30086/TCP,8125:30087/UDP,8126:30088/TCP 19h
Thanks
Sorry, yes i can access graphite:
What do you see here ?
I don't see NetApp there ;(
Can you share netapp-harvest.conf without the passwords?
What is 10.167.13.11 ?
afana_url = http://10.167.13.12:30100
#grafana_api_key = eyJrIjoiVXRYVjBBampPRTRDaGs0VFJrRW05WHhCNVIyVmowd2wiLCJuIjoibmV0YXBwLWhhcnZlc3QtMSIsImlkIjoxfQ==
#grafana_url = http://10.167.13.12:30100/api/dashboards/home
##
#### Default section to set defaults for any user created poller section
##
[default]
#====== Graphite server setup defaults ========================================
#graphite_enabled = 1
graphite_server = 10.167.13.11:30096 --- i have tried both option, added the port in extra line also.
@yannb wrote:
Can you share netapp-harvest.conf without the passwords?
What is 10.167.13.11 ?
#graphite_port = 30096
#graphite_proto = tcp
#normalized_xfer = mb_per_sec
#normalized_time = millisec
#graphite_root = default
#graphite_meta_metrics_root = default
#graphite_server = 10.167.13.11
#username = root
#password = root
# Harvest will send monthly statistics to AutoSupport, to disable set to "0"
send_autosupport_stats = 0
## If using ssl_cert (and not password auth)
## uncomment and populate next three lines
# auth_type = ssl_cert
# ssl_cert = INSERT_PEM_FILE_NAME_HERE
# ssl_key = INSERT_KEY_FILE_NAME_HERE
#====== NetApp OnCommand Insight output plugin setup defaults ==================
# To manage (create or update) OCI agent tokens and generate
# settings below, run script util/netapp-oci-setup
# oci_enabled = 1
# oci_url = https://INSERT_IP_OR_HOSTNAME_OF_OCI_SERVER_HERE
# oci_agent_token = INSERT_AGENT_TOKEN_FOR_OCI_SERVER_HERE
# oci_namespaces = volume:node,volume,system:node
##
#### Poller sections; Add one section for each cDOT cluster, 7-mode node, or OCUM server
#### If any keys are different from those in default duplicate them in the poller section to override.
##
[INSERT_CLUSTER_OR_CONTROLLER_NAME_HERE_EXACTLY_AS_SHOWN_FROM_CLI_PROMPT]
# hostname = INSERT_IP_ADDRESS_OR_HOSTNAME_OF_CONTROLLER_OR_CLUSTER_LIF_HERE
# group = INSERT_GROUP_IDENTIFIER_HERE
[NetAppProdVR]
host_type = Filer
hostname = 10.16.50.192
username = netapp-harvest
password = 123
group = MH
[NetAppProd]
host_type = Filer
hostname = 10.16.54.192
username = netapp-harvest
password = 123
group = MH
# [INSERT_OCUM_SERVER_NAME_HERE]
#[OCUM]
# hostname = 10.167.113.40
# group = nus
# password = 123
# host_type = OCUM
# data_update_freq = 900
# normalized_xfer = gb_per_sec