ClusterMode OnCommand Qtree report empty

JTVNETAPP · ‎2013-03-18

I am just starting to use OnCommand, Standard edition of DataFabric Manager server, version 5.1.0.15008 (5.1) Cluster-Mode, on a Win2008R2 SP1 x64 system.

OnCommand is able to query/get information about the system - I am seeing volume changes and events reflected in OnCommand.

I am testing out reports, at this time Qtrees: Detailed Reports -> Storage capacity -> Capacity -> Qtrees.

Unfortunately the report is empty, "No records to display"; which I know to be false.

I try the same report in Operations Manager, same empty result, "There are no qtrees."

I have four clusters which have been successfully added to OnCommand. I have them setup as four different Groups under Global.

At this time I am testing against one of those clusters/Groups: Cluster02.

I having a hard time locating exactly what is needed for OnCommand to query the storage system (e.g. cluster login, using method X with these rights).

Am I missing a good manual somewhere - I have the "Operations_Manager_Administration_Guide" and "Installation_and_Setup_Guide" from the ocum51docs download, all other documents in that download say 7-Mode.

The default snmpv1 (planning on changing to v3) seems to be able to query for volumes - but not for qtrees?

Assuming snmpv1 was not able to view Qtrees I tried a cluster user login.

First create user, with http method:

sec login create -username dfm-user -application http -authmethod password -role readonly

Then set OnCommand to user that user:

Storage tab, Clusers, highlight Cluster02, and click edit. Under the Management Options section, I enter the dfm-user and password, set transport to HTTP. Login to http://ClusterIP as that user successfully (i.e. verified login works), and can list qtrees (left side, volume, qtree, manage, lists them all under their respective volume).

However, no qtrees in OnCommand (neither Console nor Operations Manager). Also - there was a "Host Login Failed" event with source Cluster02-00, which prompted me to:

try https, no qtrees in OnCommand (after changing transport to HTTPS). Test https://ClusterIP, can login. Note - we have self signed certs for https at this time.

try adding ontapi to applications for that user. no qtrees in OnCommand.

try adding ssh to applications for that user. verify ssh manually, ssh to cluster, vol qtre show... shows all the qtrees. no qtrees in OnCommand.

I was looking at the Operations Manager Diagnostics page, and see the list of Installed Plugins, lots of versions of "Storage System Config #.#.# (#.#.#) - storage systems and vFilers", but no other types of plugins. Is there a plugin needed to see Qtrees? Is there a ClusterMode plugin needed (anytime I see vFiler I have to wonder if that is ClusterMode aware).

I checked dfm host diag:

--------------------------------------

C:\>dfm host diag Cluster02-00

Diagnostics of Hosting Cluster - Cluster02-00

Network Connectivity

IP Address IP_ADDRESS

Network (discovery disabled)

DNS Aliases Failed

DNS Addresses Failed

SNMP Version in Use SNMPv1

SNMPv1 Passed (109 ms)

SNMP Community public

SNMP sysName Cluster02-00

SNMP sysObjectID .1.3.6.1.4.1.789.2.5 (Cluster)

SNMPv3 Failed: No SNMPv3 username specified.

SNMPv3 Auth Protocol

SNMPv3 Privacy Enabled No

SNMPv3 Username

ICMP Echo Passed (0 ms)

HTTP Passed (0 ms)

NDMP (login not set) Skipped

SSH Passed (1529 ms)

RLM Skipped (hostRLMAddress is empty)

XML (http port 443) HTTP POST - failed, status code = 0

Host Details

According to: DataFabric Manager server Host

Host Name Cluster02-00 Cluster02-00

Type Cluster Cluster

OS Version 8.1.2 Cluster-Mode 8.1.2 Cluster-Mode

Revisions 380,8.1.1,2.2.1 380,8.1.1,2.2.1

Monitoring Timestamps

Timestamp Name Status Interval Default Last Updated Status

Error if older than ...

ccTimestamp Normal 4 hours 4 hours

18 Mar 10:53

cfTimestamp Normal 5 minutes 5 minutes

18 Mar 14:48

clusterTimestamp Normal 15 minutes 15 minutes 18 Mar 14:42 Normal

18 Mar 14:38

cpuTimestamp Normal 5 minutes 5 minutes 18 Mar 14:49 Normal

18 Mar 14:48

dfTimestamp Normal 30 minutes 30 minutes 18 Mar 14:52 Normal

18 Mar 14:23

diskTimestamp Normal 4 hours 4 hours 18 Mar 14:23 Normal

18 Mar 10:53

envTimestamp Normal 5 minutes 5 minutes 18 Mar 14:51 Normal

18 Mar 14:48

fsTimestamp Normal 15 minutes 15 minutes 18 Mar 14:41 Normal

18 Mar 14:38

hostPingTimestamp Normal 1 minute 1 minute 18 Mar 14:53 Normal

18 Mar 14:52

ifTimestamp Normal 15 minutes 15 minutes 18 Mar 14:41 Normal

18 Mar 14:38

licenseTimestamp Normal 4 hours 4 hours 18 Mar 12:43 Normal

18 Mar 10:53

lunTimestamp Normal 30 minutes 30 minutes 18 Mar 14:38 Normal

18 Mar 14:23

opsTimestamp Normal 10 minutes 10 minutes 18 Mar 14:50 Normal

18 Mar 14:43

qtreeTimestamp Normal 8 hours 8 hours

18 Mar 06:53

rbacTimestamp Normal 1 day 1 day

17 Mar 14:53

userQuotaTimestamp Normal 1 day 1 day 18 Mar 13:48 Normal

17 Mar 14:53

sanhostTimestamp Normal 5 minutes 5 minutes

18 Mar 14:48

snapmirrorTimestamp Error 6 hours 30 minutes

18 Mar 08:53

snapshotTimestamp Normal 30 minutes 30 minutes 18 Mar 14:33 Normal

18 Mar 14:23

statusTimestamp Normal 10 minutes 10 minutes

18 Mar 14:43

sysInfoTimestamp Normal 1 hour 1 hour 18 Mar 13:56 Normal

18 Mar 13:53

svTimestamp Error Off 30 minutes Not updated ( snapvaultM

onInterval is set to Off )

svMonTimestamp Normal 8 hours 8 hours

18 Mar 06:53

xmlQtreeTimestamp Normal 8 hours 8 hours 18 Mar 13:48 Normal

18 Mar 06:53

vFilerTimestamp Error Off 1 hour Not updated ( vFilerMonI

nterval is set to Off )

vserverTimestamp Normal 1 hour 1 hour 18 Mar 14:50 Normal

18 Mar 13:53

Performance Advisor Checklist

perfAdvisorEnabled Passed

hostType Passed

hostRevision Passed

hostLogin Passed

perfAdvisorTransport Passed

--------------------------------------

Summarizing my questions:

- Am I missing a good manual?

- Should the snmpv1 login be able to query for qtrees? If not, what type of user is needed? Cluster? One for each vServer? more than read-only? what method(s)?

- If a cluster user is needed, and the appliation method is HTTPS - could the self-signed certs be an issue?

- Is the "XML (http port 443) HTTP POST - failed, status code = 0" in the host diag pertinent? (dfm-user is only read-only so I do not think this is a surprising error)

- Is there a plugin needed to see Qtrees? Is there a ClusterMode plugin needed (anytime I see vFiler I have to wonder if that is ClusterMode aware)

Just to make this post even longer here's another question: What is the difference between Operations Manager and OnCommand console? Is OpMgr just the older interface, which still has all the capabilities, but is/will be replaced by Console?

Thanks for any suggestions or questions answered.

-jeff

adaikkap · ‎2013-03-18

Hi Jeff,

I think you have already done and would recommend you to use the Admin Guide.

Pls find my responses inline.

- Am I missing a good manual?

Pls use the Admin Guide

- Should the snmpv1 login be able to query for qtrees? If not, what type of user is needed? Cluster? One for each vServer? more than read-only? what method(s)?

Yes, if either of snmpv1 or v3 is working that should be fine.

- If a cluster user is needed, and the application method is HTTPS - could the self-signed certs be an issue?

As long as the cluster admin user is used its sufficient.

- Is the "XML (http port 443) HTTP POST - failed, status code = 0" in the host diag pertinent? (dfm-user is only read-only so I do not think this is a surprising error)

Once credentials are set for a cluster esp the qtree information is got via xml. So xml failing is my hunch why qtrees aren't getting discovered.

- Is there a plugin needed to see Qtrees? Is there a ClusterMode plugin needed (anytime I see vFiler I have to wonder if that is ClusterMode aware)

No. Plugins are not required

Just to make this post even longer here's another question: What is the difference between Operations Manager and OnCommand console? Is OpMgr just the older interface, which still has all the capabilities, but is/will be replaced by Console?

Yes. OnCommand Console refers to the new UI and the default UI. Operations Manager Console refers to the old Web UI

Regards

adai

JTVNETAPP · ‎2013-03-19

Hi Adai,

Thanks for your response.

In regards to snmpv1, so if I can see volumes with just snmp configuration present and I can not see Qtrees, something which should be working is not.

Regarding the XML HTTP POST failed message - why would OnCommand need to do a POST to get the qtree information?

To simplify things, I've removed the cluster user configuration (the dfm-user I had mentioned), so OnCommand should only be able to try via snmpv1. Just for clarity in the logs I did a dfm service stop and then start.

I tried getting a listing of the qtrees via OnCommand Console (Reports->Detailed Reports->Inventory->Qtrees and Reports->Detailed Reports->Storage Capacity->Qtrees) and get the "No records to display." From OnCommand Operations Manager:8080/dfm/report/view/qtrees?limit=20, "There are no qtrees."

I've looked through

dfm.log

dfmserver.log

error.log

jetty<date_stamp>.log

DFMMonitor.log

The DFMMonitor.log is only one that seemed to have possibly pertinent data. There are entries such as:

Mar 19 14:04:51 [DFMMonitor: WARN]: [5364:0x1f40]: fsmon: apis login failed.

Mar 19 14:04:51 [DFMMonitor: WARN]: [5364:0x1f40]: apis login failed

Mar 19 14:04:51 [DFMMonitor: WARN]: [5364:0x1f40]: fsmon_get_vol_move_details: apis login failed

Mar 19 14:05:14 [DFMMonitor:ERROR]: [5364:0x1cc]: Failed to set server context for zapi calls on: Cluster02-00

Mar 19 14:06:00 [DFMMonitor:ERROR]: [5364:0x1f40]: Unable to walk on nodeStats table for cluster node Cluster02-00:Cluster02-02 for port category(111) counters.

Mar 19 14:06:00 [DFMMonitor:ERROR]: [5364:0x1f40]: Unable to walk on nodeStats table for cluster node Cluster02-00:Cluster02-01 for port category(111) counters.

Mar 19 14:06:01 [DFMMonitor:ERROR]: [5364:0x1f40]: Unable to walk on nodeStats table for cluster node Cluster02-00:Cluster02-02 for lif category(73) counters.

Mar 19 14:06:01 [DFMMonitor:ERROR]: [5364:0x1f40]: Unable to walk on nodeStats table for cluster node Cluster02-00:Cluster02-01 for lif category(73) counters.

However none of those mention Qtrees. The closest seem to be the SNMP looking "unable to walk" messages, but those are port and lif, not Qtree related. The other messages seem to revolvevd around the ONTAPI connection method and not SNMP.

I am at a loss as to why, if SNMPv1 should be able to view Qtrees, our installation of OnCommand using SNMP can not - but at the same time *can* see volume information.

thanks for any suggestions,

Jeff

kryan · ‎2013-03-19

Jeff,

the nodeStats table walk failure and resulting behavior leads to to believe you are hitting bug 395154 (ONTAP bug):

http://support.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=395154

You should open up a support case and refer to this bug. The workaround listed will most likely not help you - instead the metrics on both the ip addresses and routing groups might need to be adjusted.

Thanks,

Kevin

adaikkap · ‎2013-03-19

Hi Jeff,

DFM collects/Discover objects in controller via multiple protocols. We use a combination of SNMP,API, SSH,Http etc to discover objects. In ClusterMode we use API to discover qtree unlike volumes which are discovered via SNMP

Regards

adai

JTVNETAPP · ‎2013-03-20

Hi Adai,

Do you know if there is any documentation which describes what settings need to be set, what users need to be created, with what rights, using what protocols or login methods, to connect to what LIFs, to query different objects?

I have tried having the cluster admin vserver user be permitted to use http, ssh, and ontapi, and I can login as that user via http/https and ssh with the same credentials specified in Operations Manager, but Operations Manager does not list qtrees.

I do not want to grant any rights or access to DFM which it does not need. I need to have a good understanding of how DFM connects.

(This goes back to the comment I had made earlier... having a hard time locating exactly what is needed for OnCommand to query the storage system).

If this type of question is better served by going a support case route, no problem I can create a case.

Thanks for your help,

Jeff