OnCommand Data seems to be stale.

[markicardy]

After performing a number of vFiler migration we have a bunch of dupe vols on different aggr's in the OnCommand dB.  Is there a process to "flush" the dB off old data? In an effort to remove the data we deleted the vol from the webUI and hit refresh on the Filer (vfiler0) but the data on the vols doesn't get re-discovered.

We believe this is creating a few other issues we are seeing, logically we want this issue resolved before looking further at the others.

We have 2 clusters (4 heads) with 80 vFilers between them.  Stale and missing data is being seen across all Filers

Thanks

[reide]

markicardy,

Be aware that OnCommand uses a polling interval to learn of changes on the storage controllers.  Some of these intervals are 30 minutes to 1 hour. This means that it can take OnCommand quite a while to learn of new/deleted volumes in an aggregate.

One option is to force a refresh of the controller/vfiler.  Rather than waiting for a polling interval, this forces OnCommand to immediately query the controller or vfiler to see what has changed.  Once you issue the refresh, give OnCommand 2-3 minutes to complete its investigation and update the database.  See if that resolves the issue.  If not, then its likely some other issue.

[adaikkap]

The discovery of objects happen once every 15mins in OnCommand. But in order to delete a volume/qtree/aggr. OnCommand take 4 cycles or 1hour to mark it deleted. This done as a safety just in case snmp didn't respond in one of the monitoring cycle, we don't remove them from OC db.

Regards

adai

[jamey]

I've waited for OnCommand to discover the new aggregate and new volumes for over a week now and still no luck.  I have attempted to manually refresh but that doesn't help either.  Suggestions?

[kjag]

Hi,

Could you paste the output of "dfm host diag <Controller IP or Name>" of the controller where the new aggregates and volumes are created?

-KJag

[jamey]

dfm host diag

Network Connectivity

IP Address             10.64.8.12

Network                (discovery disabled)

DNS Aliases            opsna2.tcbna.net

DNS Addresses          xxxxxxxxx

SNMP Version in Use    SNMPv1

SNMPv1                 Failed: Timeout

SNMP Community         xxxxxxxxx

SNMPv3                 Failed: Unknown user name

SNMPv3 Auth Protocol   MD5

SNMPv3 Privacy Enabled No

SNMPv3 Username        oncalld120\s_netapp_om

ICMP Echo              Passed (16 ms)

HTTP                   Passed (0 ms)

NDMP Ping              Passed (port 10000, 0 ms)

NDMP Connect           Passed (1531 ms)

NDMP MD5 Passwd Check  Passed

RSH                    Skipped (rshBinary is empty in global option)

SSH                    Passed (1578 ms)

RLM                    Skipped (hostRLMAddress is empty)

XML (http port 80)     Passed (1016 ms)

Host Details

According to:   DataFabric Manager server       Host

Host Name       opsna2.tcbna.net

System ID       0151706941

Model           FAS3140

Type            Clustered Storage System

OS Version      8.0.2 7-Mode

Revisions       350,8.0.1,2.1.1

Monitoring Timestamps

Timestamp Name       Status   Interval     Default      Last Updated     Status   Error if older than ...

ccTimestamp          Normal   4 hours      4 hours                                28 Jan 06:25

cfTimestamp          Normal   5 minutes    5 minutes    28 Jan 10:20     Warning  28 Jan 10:20

clusterTimestamp     Normal   15 minutes   15 minutes                             28 Jan 10:10

cpuTimestamp         Normal   5 minutes    5 minutes    28 Jan 10:20     Warning  28 Jan 10:20

dfTimestamp          Normal   30 minutes   30 minutes                             28 Jan 09:55

diskTimestamp        Normal   4 hours      4 hours      28 Jan 10:15     Normal   28 Jan 06:25

envTimestamp         Normal   5 minutes    5 minutes    28 Jan 10:20     Normal   28 Jan 10:20

fsTimestamp          Normal   15 minutes   15 minutes   28 Jan 10:22     Normal   28 Jan 10:10

hostPingTimestamp    Error    2 minutes    1 minute     28 Jan 10:24     Normal   28 Jan 10:23

ifTimestamp          Normal   15 minutes   15 minutes   28 Jan 10:15     Normal   28 Jan 10:10

licenseTimestamp     Normal   4 hours      4 hours      28 Jan 10:16     Normal   28 Jan 06:25

lunTimestamp         Normal   30 minutes   30 minutes   28 Jan 10:14     Normal   28 Jan 09:55

opsTimestamp         Normal   10 minutes   10 minutes   28 Jan 10:15     Warning  28 Jan 10:15

qtreeTimestamp       Normal   8 hours      8 hours                                28 Jan 02:25

rbacTimestamp        Normal   1 day        1 day        28 Jan 10:16     Normal   27 Jan 10:25

userQuotaTimestamp   Normal   1 day        1 day        28 Jan 10:18     Normal   27 Jan 10:25

sanhostTimestamp     Normal   5 minutes    5 minutes                              28 Jan 10:20

snapmirrorTimestamp  Normal   30 minutes   30 minutes                             28 Jan 09:55

snapshotTimestamp    Normal   30 minutes   30 minutes                             28 Jan 09:55

statusTimestamp      Normal   10 minutes   10 minutes   28 Jan 10:15     Normal   28 Jan 10:15

sysInfoTimestamp     Normal   1 hour       1 hour       28 Jan 10:16     Normal   28 Jan 09:25

svTimestamp          Normal   30 minutes   30 minutes   28 Jan 10:15     Normal   28 Jan 09:55

svMonTimestamp       Normal   8 hours      8 hours                                28 Jan 02:25

xmlQtreeTimestamp    Normal   8 hours      8 hours      28 Jan 10:18     Normal   28 Jan 02:25

vFilerTimestamp      Normal   1 hour       1 hour                                 28 Jan 09:25

vserverTimestamp     Normal   1 hour       1 hour                                 28 Jan 09:25

Performance Advisor Checklist

perfAdvisorEnabled     Passed

hostType               Passed

hostRevision           Passed

hostLogin              Passed

perfAdvisorTransport   Passed

[kjag]

SNMP is timing out.

>> SNMP Version in Use    SNMPv1

>> SNMPv1                 Failed: Timeout

Can you check you have enabled snmp on the controller and snmp.access is given for your DFM.

-KJag

[jamey]

opsna2> options snmp

snmp.access                  legacy    

snmp.enable                  on        

[kjag]

This looks fine.

Also check for the SNMP community shown in the "dfm host diag" matches with the community configured in the filer.

If not you can you can add the community using "dfm snmp add" command.

-KJag

[jamey]

Fixed it!  The domain account that the SNMP v3 was using was locked out.  Unlocked the account and did a manual refresh and the new agg/volumes are now seen.  I appreciate the help KJag!

[markicardy · Accepted solution]

I fix for this the issue we had was to:

stop the dfm services,

start only the sql service,

remove the vFiler with stale data,

restart dfm services and add the vFiler back to the dfm dB.  

All vols can now be seen.  Thanks for you input folks!