Subscribe

OnComman Performance manager not getting perf data from one cluster

Hi,

 

I had to reinstall OCPM, and forget to remove clusters from it before

reinstall (2.0 to 6.x). Months ago. I have re-reinstalled OCPM (7.1P2)

over a month ago from scratch.

 

Now one cluster (2 node/4 controller FAS3220) shows fine, all

objects and perf data is fine, and perf history exists since reinstall.

nappi::> version
NetApp Release 9.1P2: Tue Feb 28 18:17:30 UTC 2017

 

The other cluster (1 node/2 controller FAS8020) shows objects,

but not any performance data.

pomppa::* version
NetApp Release 9.1P5: Sun Jun 11 02:26:58 UTC 2017

 

Pomppa was upgraded two weeksa ago from 9.1P2, but problem

existed before upgrade, since the time I forget to remove

pomppa from OCPM when upggrading it from 2.0 to 6.x

 

 

Bugs 1053008 or 1097636 do not apply, as

 

mysql> select datasourceId, name,managementIp from netapp_model.cluster where datasourceId = -1;
Empty set (0.00 sec)

mysql> select datasourceId, name,managementIp from netapp_model.cluster where datasourceId NOT IN (select id from acquisition.ds);
Empty set (0.02 sec)

 

mysql> select datasourceId, name,managementIp from netapp_model.cluster ;
+--------------+-------+----------------------------+
| datasourceId | name  | managementIp               |
+--------------+-------+----------------------------+
| 2            | nappi | nappi.local.it.helsinki.fi |
+--------------+-------+----------------------------+
1 row in set (0.00 sec)

 

Nappi is the working cluster, I removed the not working one.

 

And from cluster

 

pomppa::*> application-record show
(cluster application-record show)
Last Modified
Vserver Name Value Timestamp Modified From
--------------- --------------- ------------------ ------------- --------------
pomppa sysmgr.inactivitytimeout
180 7/9/2015 13:33:24
10.101.24.174

pomppa::*> Connection to pomppa.local.it.helsinki.fi closed.

 

  Added cluster back to OCPM:

 

pomppa::*> application-record show
(cluster application-record show)
Last Modified
Vserver Name Value Timestamp Modified From
--------------- --------------- ------------------ ------------- --------------
pomppa app-OnCommand Performance Manager-f1b9253e-9971-4b77-9308-bb1eeace4439
{"id":"f1b9253e-9971-4b77-9308-bb1eeace4439","url":"https://ocpm-1.local.it.helsinki.fi:443","version":"7.1P2","dateAdded":"2017-08-09 15:01:50.951"}
8/9/2017 15:01:50
10.101.27.2
pomppa sysmgr.inactivitytimeout
180 7/9/2015 13:33:24
10.101.24.174
2 entries were displayed.

 

And from mysql:

 

mysql> select datasourceId, name,managementIp from netapp_model.cluster ;
+--------------+--------+-----------------------------+
| datasourceId | name | managementIp |
+--------------+--------+-----------------------------+
| 2 | nappi | nappi.local.it.helsinki.fi |
| 5 | pomppa | pomppa.local.it.helsinki.fi |
+--------------+--------+-----------------------------+
2 rows in set (0.00 sec)

 

mysql> select * from acquisition.ds;
+----+------------+-------+-----------------------------+-----------------------+-----------------------+--------------------------------------------------------------------------------------------------+-----------+------+--------+--------+---------+
| id | ds_type_id | au_id | name | vendor | model | status | statusext | time | manual | active | version |
+----+------------+-------+-----------------------------+-----------------------+-----------------------+--------------------------------------------------------------------------------------------------+-----------+------+--------+--------+---------+
| 2 | 91 | 1 | nappi.local.it.helsinki.fi | NetApp OCI Essentials | NetApp OCI Essentials | Inventory success, Performance success, netappfoundation failed, netappstorageperformance failed | | 0 | 1 | 1 | 0 |
| 5 | 91 | 1 | pomppa.local.it.helsinki.fi | NetApp OCI Essentials | NetApp OCI Essentials | Inventory success, Performance acquiring, netappstorageperformance failed | | 0 | 1 | 1 | 0 |
+----+------------+-------+-----------------------------+-----------------------+-----------------------+--------------------------------------------------------------------------------------------------+-----------+------+--------+--------+---------+
2 rows in set (0.00 sec)

mysql> select now();
+---------------------+
| now() |
+---------------------+
| 2017-08-09 16:00:23 |
+---------------------+
1 row in set (0.08 sec)

 

Something prevents OCPM getting performance data from pomppa,

but allows inventory to succeed.

 

I did not pair OCPM and UCUM to demonstrate that problem

lies in OCPM / pomppa relation somehow.

 

I am happy to provide more information.

 

Re: OnComman Performance manager not getting perf data from one cluster

pomppa::*> application-record show

shows 2 entries. Is it possible that this cluster was still registered with the other OPM earlier? We do not allow the same cluster to be monitored by two OPMs.

Re: OnComman Performance manager not getting perf data from one cluster

The other entry is (i think) ssh-command shell timeout.

And it is on the working cluster too

(I added some linefeeds to clarify the output)

 

 

The working cluster

nappi::*> cluster application-record show
Last Modified
Vserver Name Value Timestamp Modified From
--------------- --------------- ------------------ ------------- --------------
nappi app-OnCommand Performance Manager-f1b9253e-9971-4b77-9308-bb1eeace4439
{

  "id":"f1b9253e-9971-4b77-9308-bb1eeace4439",

  "url":"https://ocpm-1.local.it.helsinki.fi:443",

  "version":"7.1P2",

  "dateAdded":

  "2017-08-03 14:09:09.417"

}
8/3/2017 14:09:09 10.101.27.2

 

nappi sysmgr.inactivitytimeout  180 7/9/2015 13:33:02 10.101.24.174
2 entries were displayed.

 

The not working cluster

pomppa::*> cluster application-record show
Last Modified
Vserver Name Value Timestamp Modified From
--------------- --------------- ------------------ ------------- --------------
pomppa app-OnCommand Performance Manager-f1b9253e-9971-4b77-9308-bb1eeace4439
{

  "id":"f1b9253e-9971-4b77-9308-bb1eeace4439",

  "url":"https://ocpm-1.local.it.helsinki.fi:443",

  "version":"7.1P2",

  "dateAdded":

  "2017-08-09 15:01:50.951"

}
8/9/2017 15:01:50 10.101.27.2

 

pomppa sysmgr.inactivitytimeout 180 7/9/2015 13:33:24 10.101.24.174
2 entries were displayed.

 

Hmmm-- CLI timeout is 360 minutes on both clusters, that is some other

timeout, propably the web interface (i.e. system manager) timeout. Yes,

our web-interface timeout is 180 minutes (as set in Administration in

right upper corner -> Settings).