ClusterMode OnCommand Qtree report empty

[JTVNETAPP]

I am just starting to use OnCommand, Standard edition of DataFabric Manager server, version 5.1.0.15008 (5.1) Cluster-Mode, on a Win2008R2 SP1 x64 system.

OnCommand is able to query/get information about the system - I am seeing volume changes and events reflected in OnCommand.

I am testing out reports, at this time Qtrees: Detailed Reports -> Storage capacity -> Capacity -> Qtrees.

Unfortunately the report is empty, "No records to display"; which I know to be false.

I try the same report in Operations Manager, same empty result, "There are no qtrees."

I have four clusters which have been successfully added to OnCommand.  I have them setup as four different Groups under Global.

At this time I am testing against one of those clusters/Groups: Cluster02.

I having a hard time locating exactly what is needed for OnCommand to query the storage system (e.g. cluster login, using method X with these rights).

Am I missing a good manual somewhere - I have the "Operations_Manager_Administration_Guide" and "Installation_and_Setup_Guide" from the ocum51docs download, all other documents in that download say 7-Mode.

The default snmpv1 (planning on changing to v3) seems to be able to query for volumes - but not for qtrees?

Assuming snmpv1 was not able to view Qtrees I tried a cluster user login. 

First create user, with http method:

sec login create -username dfm-user -application http -authmethod password -role readonly

Then set OnCommand to user that user:

Storage tab, Clusers, highlight Cluster02, and click edit.  Under the Management Options section, I enter the dfm-user and password, set transport to HTTP.  Login to http://ClusterIP as that user successfully (i.e. verified login works), and can list qtrees (left side, volume, qtree, manage, lists them all under their respective volume).

However, no qtrees in OnCommand (neither Console nor Operations Manager).   Also - there was a "Host Login Failed" event with source Cluster02-00, which prompted me to:

try https, no qtrees in OnCommand (after changing transport to HTTPS).  Test https://ClusterIP, can login.  Note - we have self signed certs for https at this time.

try adding ontapi to applications for that user.  no qtrees in OnCommand.

try adding ssh to applications for that user. verify ssh manually, ssh to cluster, vol qtre show... shows all the qtrees.  no qtrees in OnCommand.

I was looking at the Operations Manager Diagnostics page, and see the list of Installed Plugins, lots of versions of "Storage System Config #.#.# (#.#.#) - storage systems and vFilers", but no other types of plugins.  Is there a plugin needed to see Qtrees?  Is there a ClusterMode plugin needed (anytime I see vFiler I have to wonder if that is ClusterMode aware).

I checked dfm host diag:

--------------------------------------

C:\>dfm host diag Cluster02-00

Diagnostics of Hosting Cluster - Cluster02-00

Network Connectivity

IP Address             IP_ADDRESS

Network                (discovery disabled)

DNS Aliases            Failed

DNS Addresses          Failed

SNMP Version in Use    SNMPv1

SNMPv1                 Passed (109 ms)

SNMP Community         public

SNMP sysName           Cluster02-00

SNMP sysObjectID       .1.3.6.1.4.1.789.2.5 (Cluster)

SNMPv3                 Failed: No SNMPv3 username specified.

SNMPv3 Auth Protocol

SNMPv3 Privacy Enabled No

SNMPv3 Username

ICMP Echo              Passed (0 ms)

HTTP                   Passed (0 ms)

NDMP (login not set)   Skipped

SSH                    Passed (1529 ms)

RLM                    Skipped (hostRLMAddress is empty)

XML (http port 443)    HTTP POST - failed, status code = 0

Host Details

According to:   DataFabric Manager server       Host

Host Name       Cluster02-00                   Cluster02-00

Type            Cluster                        Cluster

OS Version      8.1.2 Cluster-Mode             8.1.2 Cluster-Mode

Revisions       380,8.1.1,2.2.1                380,8.1.1,2.2.1

Monitoring Timestamps

Timestamp Name       Status   Interval     Default      Last Updated     Status

  Error if older than ...

ccTimestamp          Normal   4 hours      4 hours

  18 Mar 10:53

cfTimestamp          Normal   5 minutes    5 minutes

  18 Mar 14:48

clusterTimestamp     Normal   15 minutes   15 minutes   18 Mar 14:42     Normal

  18 Mar 14:38

cpuTimestamp         Normal   5 minutes    5 minutes    18 Mar 14:49     Normal

  18 Mar 14:48

dfTimestamp          Normal   30 minutes   30 minutes   18 Mar 14:52     Normal

  18 Mar 14:23

diskTimestamp        Normal   4 hours      4 hours      18 Mar 14:23     Normal

  18 Mar 10:53

envTimestamp         Normal   5 minutes    5 minutes    18 Mar 14:51     Normal

  18 Mar 14:48

fsTimestamp          Normal   15 minutes   15 minutes   18 Mar 14:41     Normal

  18 Mar 14:38

hostPingTimestamp    Normal   1 minute     1 minute     18 Mar 14:53     Normal

  18 Mar 14:52

ifTimestamp          Normal   15 minutes   15 minutes   18 Mar 14:41     Normal

  18 Mar 14:38

licenseTimestamp     Normal   4 hours      4 hours      18 Mar 12:43     Normal

  18 Mar 10:53

lunTimestamp         Normal   30 minutes   30 minutes   18 Mar 14:38     Normal

  18 Mar 14:23

opsTimestamp         Normal   10 minutes   10 minutes   18 Mar 14:50     Normal

  18 Mar 14:43

qtreeTimestamp       Normal   8 hours      8 hours

  18 Mar 06:53

rbacTimestamp        Normal   1 day        1 day

  17 Mar 14:53

userQuotaTimestamp   Normal   1 day        1 day        18 Mar 13:48     Normal

  17 Mar 14:53

sanhostTimestamp     Normal   5 minutes    5 minutes

  18 Mar 14:48

snapmirrorTimestamp  Error    6 hours      30 minutes

  18 Mar 08:53

snapshotTimestamp    Normal   30 minutes   30 minutes   18 Mar 14:33     Normal

  18 Mar 14:23

statusTimestamp      Normal   10 minutes   10 minutes

  18 Mar 14:43

sysInfoTimestamp     Normal   1 hour       1 hour       18 Mar 13:56     Normal

  18 Mar 13:53

svTimestamp          Error    Off          30 minutes   Not updated ( snapvaultM

onInterval is set to Off )

svMonTimestamp       Normal   8 hours      8 hours

  18 Mar 06:53

xmlQtreeTimestamp    Normal   8 hours      8 hours      18 Mar 13:48     Normal

  18 Mar 06:53

vFilerTimestamp      Error    Off          1 hour       Not updated ( vFilerMonI

nterval is set to Off )

vserverTimestamp     Normal   1 hour       1 hour       18 Mar 14:50     Normal

  18 Mar 13:53

Performance Advisor Checklist

perfAdvisorEnabled     Passed

hostType               Passed

hostRevision           Passed

hostLogin              Passed

perfAdvisorTransport   Passed

--------------------------------------

Summarizing my questions:

- Am I missing a good manual?

- Should the snmpv1 login be able to query for qtrees? If not, what type of user is needed?  Cluster? One for each vServer? more than read-only? what method(s)?

- If a cluster user is needed, and the appliation method is HTTPS - could the self-signed certs be an issue?

- Is the "XML (http port 443) HTTP POST - failed, status code = 0" in the host diag pertinent? (dfm-user is only read-only so I do not think this is a surprising error)

- Is there a plugin needed to see Qtrees?  Is there a ClusterMode plugin needed (anytime I see vFiler I have to wonder if that is ClusterMode aware)

Just to make this post even longer here's another question: What is the difference between Operations Manager and OnCommand console?  Is OpMgr just the older interface, which still has all the capabilities, but is/will be replaced by Console?

Thanks for any suggestions or questions answered.

-jeff

[adaikkap]

Hi Jeff,

     I think you have already done and would recommend you to use the Admin Guide.

Pls find my responses inline.

- Am I missing a good manual?

Pls use the Admin Guide

- Should the snmpv1 login be able to query for qtrees? If not, what type of user is needed?  Cluster? One for each vServer? more than read-only? what method(s)?

Yes, if either of snmpv1 or v3 is working that should be fine.

- If a cluster user is needed, and the application method is HTTPS - could the self-signed certs be an issue?

As long as the cluster admin user is used its sufficient.

- Is the "XML (http port 443) HTTP POST - failed, status code = 0" in the host diag pertinent? (dfm-user is only read-only so I do not think this is a surprising error)

Once credentials are set for a cluster esp the qtree information is got via xml. So xml failing is my hunch why qtrees aren't getting discovered.

- Is there a plugin needed to see Qtrees?  Is there a ClusterMode plugin needed (anytime I see vFiler I have to wonder if that is ClusterMode aware)

No. Plugins are not required

Just to make this post even longer  here's another question: What is the difference between Operations Manager and OnCommand console?  Is OpMgr just the older interface, which still has all the capabilities, but is/will be replaced by Console?

Yes. OnCommand Console refers to the new UI and the default UI. Operations Manager Console refers to the old Web UI

Regards

adai

[JTVNETAPP]

Hi Adai,

Thanks for your response.

In regards to snmpv1, so if I can see volumes with just snmp configuration present and I can not see Qtrees, something which should be working is not.

Regarding the XML HTTP POST failed message - why would  OnCommand need to do a POST to get the qtree information?

To simplify things, I've removed the cluster user configuration (the dfm-user I had mentioned), so OnCommand should only be able to try via snmpv1.  Just for clarity in the logs I did a dfm service stop and then start.

I tried getting a listing of the qtrees via OnCommand Console (Reports->Detailed Reports->Inventory->Qtrees and Reports->Detailed Reports->Storage Capacity->Qtrees) and get the "No records to display."  From OnCommand Operations Manager:8080/dfm/report/view/qtrees?limit=20, "There are no qtrees."

I've looked through

dfm.log

dfmserver.log

error.log

jetty<date_stamp>.log

DFMMonitor.log

The DFMMonitor.log is only one that seemed to have possibly pertinent data.  There are entries such as:

Mar 19 14:04:51 [DFMMonitor: WARN]: [5364:0x1f40]: fsmon: apis login failed.

Mar 19 14:04:51 [DFMMonitor: WARN]: [5364:0x1f40]: apis login failed

Mar 19 14:04:51 [DFMMonitor: WARN]: [5364:0x1f40]: fsmon_get_vol_move_details: apis login failed

Mar 19 14:05:14 [DFMMonitor:ERROR]: [5364:0x1cc]: Failed to set server context for zapi calls on: Cluster02-00

Mar 19 14:06:00 [DFMMonitor:ERROR]: [5364:0x1f40]: Unable to walk on nodeStats table for cluster node Cluster02-00:Cluster02-02 for port category(111) counters.

Mar 19 14:06:00 [DFMMonitor:ERROR]: [5364:0x1f40]: Unable to walk on nodeStats table for cluster node Cluster02-00:Cluster02-01 for port category(111) counters.

Mar 19 14:06:01 [DFMMonitor:ERROR]: [5364:0x1f40]: Unable to walk on nodeStats table for cluster node Cluster02-00:Cluster02-02 for lif category(73) counters.

Mar 19 14:06:01 [DFMMonitor:ERROR]: [5364:0x1f40]: Unable to walk on nodeStats table for cluster node Cluster02-00:Cluster02-01 for lif category(73) counters.

However none of those mention Qtrees. The closest seem to be the SNMP looking "unable to walk" messages, but those are port and lif, not Qtree related.  The other messages seem to revolvevd around the ONTAPI connection method and not SNMP.

I am at a loss as to why, if SNMPv1 should be able to view Qtrees, our installation of OnCommand using SNMP can not - but at the same time *can* see volume information.

thanks for any suggestions,

Jeff

[kryan]

Jeff,

the nodeStats table walk failure and resulting behavior leads to to believe you are hitting bug 395154 (ONTAP bug):

http://support.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=395154

You should open up a support case and refer to this bug.  The workaround listed will most likely not help you - instead the metrics on both the ip addresses and routing groups might need to be adjusted.

Thanks,

Kevin

[adaikkap]

Hi Jeff,

     DFM collects/Discover objects in controller via multiple protocols. We use a combination of SNMP,API, SSH,Http etc to discover objects. In ClusterMode we use API to discover qtree unlike volumes which are discovered via SNMP

Regards

adai

[JTVNETAPP]

Hi Adai,

Do you know if there is any documentation which describes what settings need to be set, what users need to be created, with what rights, using what protocols or login methods, to connect to what LIFs, to query different objects?

I have tried having the cluster admin vserver user be permitted to use http, ssh, and ontapi, and I can login as that user via http/https and ssh with the same credentials specified in Operations Manager, but Operations Manager does not list qtrees.

I do not want to grant any rights or access to DFM which it does not need.  I need to have a good understanding of how DFM connects.

(This goes back to the comment I had made earlier... having a hard time locating exactly what is needed for OnCommand to query the storage system).

If this type of question is better served by going a support case route, no problem I can create a case.

Thanks for your help,

Jeff