ONTAP Discussions

qtreeTimestamp Error

skellner
3,782 Views

DFM Server of my customer isn't updating qtree information. All hosts show the qtreeTimestamp in Error. However the dfrm host diag shows no issues regarding access or authentication. Please find attached the dfm host diag of a host example. Thanks in advance for any hints.

j255030@v100spldv02:/> dfm host diag q100bpmc001

Network Connectivity

IP Address             17.247.34.129

Network                (discovery disabled)

DNS Aliases            Q100BPMC001.q100.intern

DNS Addresses          17.247.34.129

SNMP Version in Use    SNMPv3

SNMPv1                 Passed (195 ms)

SNMP Community         public

SNMP sysName           Q100BPMC001.q100.intern

SNMP sysObjectID       .1.3.6.1.4.1.789.2.3 (Clustered Filer)

SNMP productId         0118074118

SNMPv3                 Passed (8 ms)

SNMPv3 Auth Protocol   MD5

SNMPv3 Privacy Enabled No

SNMPv3 Username        omuser

SNMP sysName           Q100BPMC001.q100.intern

SNMP sysObjectID       .1.3.6.1.4.1.789.2.3 (Clustered Filer)

SNMP productId         0118074118

ICMP Echo              Passed

HTTP                   Passed (1 ms)

NDMP Ping              Passed (port 10000, 1 ms)

NDMP Connect           Error: Couldn't authenticate NDMP connection to host 17.247.34.129

RSH                    timed out

SSH                    Passed (10353 ms)

RLM                    Skipped (hostRLMAddress is empty)

XML (https port 443)   Passed (11 ms)

Host Details

According to:   DataFabric Manager server       Host

Host Name       Q100BPMC001.q100.intern        Q100BPMC001.q100.intern

System ID       0118074118                     0118074118

Model           FAS6080                        FAS6080

Type            Clustered Storage System       Clustered Storage System

OS Version      7.3.3P5                        7.3.3P5

Revisions       330,8.0,2.1                    330,8.0,2.1

Monitoring Timestamps

Timestamp Name       Status   Interval     Default      Last Updated     Status   Error if older than ...

ccTimestamp          Error    1 hour       4 hours      19 Mär 12:39    Normal   19 Mär 12:12

cfTimestamp          Error    3 minutes    5 minutes    19 Mär 13:11    Normal   19 Mär 13:09

clusterTimestamp     Normal   15 minutes   15 minutes                             19 Mär 12:57

cpuTimestamp         Normal   5 minutes    5 minutes    19 Mär 13:12    Normal   19 Mär 13:07

dfTimestamp          Normal   30 minutes   30 minutes   19 Mär 12:53    Normal   19 Mär 12:42

diskTimestamp        Normal   4 hours      4 hours      19 Mär 09:33    Normal   19 Mär 09:12

envTimestamp         Normal   5 minutes    5 minutes    19 Mär 13:08    Normal   19 Mär 13:07

fcTimestamp          Normal   5 minutes    5 minutes    19 Mär 13:10    Normal   19 Mär 13:07

fsTimestamp          Normal   15 minutes   15 minutes   19 Mär 13:07    Normal   19 Mär 12:57

hostPingTimestamp    Error    2 minutes    1 minute     19 Mär 13:11    Normal   19 Mär 13:10

ifTimestamp          Normal   15 minutes   15 minutes   19 Mär 12:59    Normal   19 Mär 12:57

licenseTimestamp     Error    1 day        4 hours      18 Mär 13:25    Normal   18 Mär 13:12

lunTimestamp         Error    1 hour       30 minutes   19 Mär 12:13    Normal   19 Mär 12:12

opsTimestamp         Normal   10 minutes   10 minutes   19 Mär 13:05    Normal   19 Mär 13:02

qtreeTimestamp       Error    1 day        8 hours                                18 Mär 13:12     <==============

rbacTimestamp        Normal   1 day        1 day        18 Mär 13:26    Normal   18 Mär 13:12

userQuotaTimestamp   Normal   1 day        1 day                                  18 Mär 13:12

sanhostTimestamp     Error    2 hours      5 minutes                              19 Mär 11:12

snapmirrorTimestamp  Error    2 hours      30 minutes   19 Mär 00:42    Error    19 Mär 11:12

snapshotTimestamp    Error    1 hour       30 minutes   19 Mär 13:01    Normal   19 Mär 12:12

statusTimestamp      Error    2 minutes    10 minutes   19 Mär 13:11    Normal   19 Mär 13:10

sysInfoTimestamp     Normal   1 hour       1 hour       19 Mär 12:33    Normal   19 Mär 12:12

svTimestamp          Error    Off          30 minutes   Not updated ( snapvaultMonInterval is set to Off )

svMonTimestamp       Normal   8 hours      8 hours                                19 Mär 05:12

xmlQtreeTimestamp    Error    1 day        8 hours                                18 Mär 13:12

vFilerTimestamp      Error    30 minutes   1 hour       19 Mär 12:52    Normal   19 Mär 12:42

vserverTimestamp     Normal   1 hour       1 hour                                 19 Mär 12:12

Performance Advisor Checklist

perfAdvisorEnabled     Passed

hostType               Passed

hostRevision           Passed

hostLogin              Passed

perfAdvisorTransport   Passed

j255030@v100spldv02:/> dfm about

Version                          5.0 (5.0)

Executable Type                  64-bit

Serial Number                    1-50-005803

Edition                          Standard edition of DataFabric Manager server

Administrator Name               j255030

Host Name                        v100spldv02

Host IP Address                  127.0.0.2

Host Full Name                   v100spldv02.v100.intern

Node limit                       250 (currently managing 69)

Operating System                 SUSE Linux Enterprise Server 11 (x86_64) 2.6.27.19-5-default x86_64

CPU Count                        8

System Memory                    19955 MB (load excluding cached memory: 40%)

Installation Directory           /opt/NTAPdfm

                                 415 GB free (59.4%)

Perf Data Directory              /om-perfdata

Data Export Directory            /opt/NTAPdfm/dataExport

Database Backup Directory        /opt/NTAPdfm/data

Reports Archival Directory       /opt/NTAPdfm/reports

Database Directory               /om-db

                                 96.4 GB free (76.5%)

Database Log Directory           /om-db

                                 96.4 GB free (76.5%)

Licensed Features                DataFabric Manager server: installed

                                 Storage Area Network Option: installed

                                 File SRM Option: installed

7 REPLIES 7

adaikkap
3,782 Views

Hi Stefan,

          Firstly I request you to change all you monitoring intervals back to default. Esp the following monitors ?

ccTimestamp          Error    1 hour       4 hours      19 Mär 12:39    Normal   19 Mär 12:12

cfTimestamp          Error    3 minutes    5 minutes    19 Mär 13:11    Normal   19 Mär 13:09

hostPingTimestampError2 minutes1 minute 19 Mär 13:11Normal   19 Mär 13:10
statusTimestamp  Error2 minutes10 minutes   19 Mär 13:11Normal   19 Mär 13:10
vFilerTimestamp  Error30 minutes   1 hour   19 Mär 12:52Normal   19 Mär 12:42

Regards

adai

skellner
3,782 Views

Hi Adai,

in the meantime we adjusted all monitoring intervals to default. Additionally we setup a new dfm Server. However, the problem nearly stays the same. The Qtree (XML) monitor gets only infrequently. This happens on the new and on the old dfm server. Is there a possibility to analyse why the qtree monitor doesn't get updated every time? Any kind of debug mode I can set in dfm so that I can check the log files for any reason?

Regards

Stefan

adaikkap
3,782 Views

Hi Stefan,

     Can I have the output of dfm diag once more ? Do you have any rough estimate of what would be your userquota/qtree quota numbers ? In the mean time I strongly suggest you to open a support case so that they could do a packet trace and other troubleshooting to solve the issue as they would have all needed logs and details. Which is little difficult to share over communities for all the know reasons.

Regards

adai

skellner
3,782 Views

Hi Adai,

attached the dfm diag.

Thanks

Stefan

dfm-h1:~ # dfm diag

Management Station

Version                          5.0 (5.0)

Executable Type                  64-bit

Serial Number                    1-50-005803

Edition                          Standard edition of DataFabric Manager server

Administrator Name               root

Host Name                        dfm-h1

Host IP Address                  17.247.33.201

Host Full Name                   v100spldv01.v100.intern

Node limit                       250 (currently managing 42)

Operating System                 SUSE Linux Enterprise Server 11 (x86_64) 2.6.32.12-0.7-default x86_64

CPU Count                        8

System Memory                    64560 MB (load excluding cached memory: 11%)

Installation Directory           /opt/NTAPdfm

                                 17.3 GB free (33.1%)

Perf Data Directory              /om-perfdata

Data Export Directory            /opt/NTAPdfm/dataExport

Database Backup Directory        /opt/NTAPdfm/data

Reports Archival Directory       /opt/NTAPdfm/reports

Database Directory               /om-db

                                 477 GB free (94.7%)

Licensed Features                DataFabric Manager server: installed

                                 Storage Area Network Option: installed

                                 File SRM Option: installed

Installed Plugins                Storage System Config 6.5.1 (6.5.1) - storage systems and vFilers

                                 Storage System Config 6.5.2 (6.5.2) - storage systems and vFilers

                                 Storage System Config 6.5.3 (6.5.3) - storage systems and vFilers

                                 Storage System Config 6.5.4 (6.5.4) - storage systems and vFilers

                                 Storage System Config 6.5.5 (6.5.5) - storage systems and vFilers

                                 Storage System Config 6.5.6 (6.5.6) - storage systems and vFilers

                                 Storage System Config 6.5.7 (6.5.7) - storage systems and vFilers

                                 Storage System Config 7.0 (7.0.0.1) - storage systems and vFilers

                                 Storage System Config 7.0.1 (7.0.1.1) - storage systems and vFilers

                                 Storage System Config 7.0.2 (7.0.2) - storage systems and vFilers

                                 Storage System Config 7.0.3 (7.0.3) - storage systems and vFilers

                                 Storage System Config 7.0.4 (7.0.4) - storage systems and vFilers

                                 Storage System Config 7.0.5 (7.0.5) - storage systems and vFilers

                                 Storage System Config 7.0.6 (7.0.6) - storage systems and vFilers

                                 Storage System Config 7.0.7 (7.0.7) - storage systems and vFilers

                                 Storage System Config 7.1 (7.1.0.1) - storage systems and vFilers

                                 Storage System Config 7.1.1 (7.1.1.1) - storage systems and vFilers

                                 Storage System Config 7.1.2 (7.1.2.1) - storage systems and vFilers

                                 Storage System Config 7.1.3 (7.1.3) - storage systems and vFilers

                                 Storage System Config 7.2 (7.2) - storage systems and vFilers

                                 Storage System Config 7.2.1 (7.2.1.1) - storage systems and vFilers

                                 Storage System Config 7.2.2 (7.2.2) - storage systems and vFilers

                                 Storage System Config 7.2.3 (7.2.3) - storage systems and vFilers

                                 Storage System Config 7.2.4 (7.2.4) - storage systems and vFilers

                                 Storage System Config 7.2.5 (7.2.5.1) - storage systems and vFilers

                                 Storage System Config 7.2.6 (7.2.6.1) - storage systems and vFilers

                                 Storage System Config 7.2.7 (7.2.7) - storage systems and vFilers

                                 Storage System Config 7.3 (7.3) - storage systems and vFilers

                                 Storage System Config 7.3.1 (7.3.1.1) - storage systems and vFilers

                                 Storage System Config 7.3.2 (7.3.2) - storage systems and vFilers

                                 Storage System Config 7.3.3 (7.3.3) - storage systems and vFilers

                                 Storage System Config 7.3.4 (7.3.4) - storage systems and vFilers

                                 Storage System Config 7.3.5 (7.3.5) - storage systems and vFilers

                                 Storage System Config 8.0 (8.0) - storage systems and vFilers

                                 Storage System Config 8.0.1 (8.0.1) - storage systems and vFilers

                                 Storage System Config 8.0.2 (8.0.2) - storage systems and vFilers

Global Status

Status Key               Last Updated Status Value

autosupport.installTime  11 Apr 12:47 2012-04-11 12:47:01.914262

autosupport.lastStatusSendTime 11 Apr 12:47 1335363937

autoThrottleMultiplier   27 Apr 06:47 100

dpSnapNameFormatVersion  11 Apr 12:47 1

events.purged.currentEvents 26 Apr 12:49 0

events.purged.dpJobProgressEvents 26 Apr 12:49 0

events.purged.dpJobs     26 Apr 12:49 0

events.purged.events     26 Apr 12:49 0

generationId             11 Apr 12:47 691

generationId.alonso      11 Apr 12:47 88

generationId.aquarius    11 Apr 12:47 1

generationId.aquarius-ms1 11 Apr 12:47 4

generationId.aquarius-ms2 11 Apr 12:47 25

generationId.aquarius-ms2new 11 Apr 12:47 18

generationId.aquarius-ms3 11 Apr 12:47 86

generationId.aquarius-ms4 11 Apr 12:47 11

generationId.aquarius-ms4-cba 11 Apr 12:47 3

generationId.asada       11 Apr 12:47 1

generationId.asada-config 11 Apr 12:47 1

generationId.baldi       11 Apr 12:47 51

generationId.beni        11 Apr 12:47 1

generationId.beni-ms3    11 Apr 12:47 5

generationId.carter      11 Apr 12:47 38

generationId.clark       11 Apr 12:47 42

generationId.daly-ms1    11 Apr 12:47 2

generationId.daly-ms2    11 Apr 12:47 4

generationId.daly-ms3    11 Apr 12:47 8

generationId.frederick   11 Apr 12:47 64

generationId.freya-reloaded 11 Apr 12:47 104

generationId.gayatri     11 Apr 12:47 21

generationId.guiredesign 11 Apr 12:47 1

generationId.hel         11 Apr 12:47 18

generationId.indra       11 Apr 12:47 30

generationId.isis        11 Apr 12:47 25

generationId.janak       11 Apr 12:47 5

generationId.janak-cloud 11 Apr 12:47 7

generationId.janneau     11 Apr 12:47 77

generationId.juno-cbsprint 11 Apr 12:47 4

generationId.juno-dc     11 Apr 12:47 1

generationId.juno-sprint1 11 Apr 12:47 1

generationId.juno-sprint3 11 Apr 12:47 1

generationId.juno-ssm    11 Apr 12:47 1

generationId.juno-ssm-sm-susp 11 Apr 12:47 2

generationId.juno-ssm-sprint4 11 Apr 12:47 4

generationId.kahlua      11 Apr 12:47 16

generationId.kakia       11 Apr 12:47 1

generationId.kakia-dc    11 Apr 12:47 3

generationId.kakia-sprint1 11 Apr 12:47 10

generationId.oldmonk     11 Apr 12:47 36

generationId.padataexport 11 Apr 12:47 4

generationId.pucker      11 Apr 12:47 7

groupCache.verifier.current 27 Apr 08:25 3422

groupChildCache.verifier.current 27 Apr 08:25 3422

history.rollup.aggrHistoryMonth 27 Apr 09:03 Timeslot 27 Apr 10:00: 62 records in 0.00644 seconds (9632 rps)

history.rollup.aggrHistoryQuarter 27 Apr 09:03 Timeslot 28 Apr 02:00: 62 records in 0.0181 seconds (3417 rps)

history.rollup.aggrHistoryWeek 27 Apr 09:03 Timeslot 27 Apr 10:00: 62 records in 0.00366 seconds (16954 rps)

history.rollup.aggrHistoryYear 27 Apr 09:03 Timeslot 30 Apr 02:00: 62 records in 0.00435 seconds (14237 rps)

history.rollup.cpuHistoryMonth 27 Apr 08:51 Timeslot 27 Apr 10:00: 324 records in 0.0104 seconds (31023 rps)

history.rollup.cpuHistoryQuarter 27 Apr 08:51 Timeslot 28 Apr 02:00: 324 records in 0.0081 seconds (40020 rps)

history.rollup.cpuHistoryWeek 27 Apr 08:51 Timeslot 27 Apr 10:00: 324 records in 0.0119 seconds (27314 rps)

history.rollup.cpuHistoryYear 27 Apr 08:51 Timeslot 30 Apr 02:00: 324 records in 0.008 seconds (40494 rps)

history.rollup.datatransferHistoryDay 27 Apr 09:03 Timeslot 27 Apr 09:15: 3 records in 0.989 seconds (3 rps)

history.rollup.datatransferHistoryMonth 27 Apr 09:03 Timeslot 27 Apr 10:00: 151 records in 0.008 seconds (18873 rps)

history.rollup.datatransferHistoryQuarter 27 Apr 09:03 Timeslot 28 Apr 02:00: 151 records in 0.0478 seconds (3156 rps)

history.rollup.datatransferHistoryWeek 27 Apr 09:03 Timeslot 27 Apr 10:00: 21 records in 0.00537 seconds (3912 rps)

history.rollup.datatransferHistoryYear 27 Apr 09:03 Timeslot 30 Apr 02:00: 151 records in 0.00761 seconds (19855 rps)

history.rollup.dfHistoryMonth 27 Apr 09:03 Timeslot 27 Apr 10:00: 2386 records in 0.277 seconds (8626 rps)

history.rollup.dfHistoryQuarter 27 Apr 09:03 Timeslot 28 Apr 02:00: 2386 records in 0.326 seconds (7312 rps)

history.rollup.dfHistoryWeek 27 Apr 09:03 Timeslot 27 Apr 10:00: 2386 records in 1.29 seconds (1856 rps)

history.rollup.dfHistoryYear 27 Apr 09:03 Timeslot 30 Apr 02:00: 2386 records in 0.293 seconds (8144 rps)

history.rollup.fcswHistoryMonth 27 Apr 08:51 Timeslot 27 Apr 10:00: 0 records in 0.0109 seconds (0 rps)

history.rollup.fcswHistoryQuarter 27 Apr 08:51 Timeslot 28 Apr 02:00: 0 records in 0.00175 seconds (0 rps)

history.rollup.fcswHistoryWeek 27 Apr 08:51 Timeslot 27 Apr 10:00: 0 records in 0.00129 seconds (0 rps)

history.rollup.fcswHistoryYear 27 Apr 08:51 Timeslot 30 Apr 02:00: 0 records in 0.0018 seconds (0 rps)

history.rollup.historyMonth 27 Apr 08:45 Timeslot 27 Apr 10:00: 1423 records in 0.537 seconds (2649 rps)

history.rollup.historyQuarter 27 Apr 08:45 Timeslot 28 Apr 02:00: 1423 records in 0.521 seconds (2732 rps)

history.rollup.historyWeek 27 Apr 08:45 Timeslot 27 Apr 10:00: 1349 records in 0.759 seconds (1778 rps)

history.rollup.historyYear 27 Apr 08:45 Timeslot 30 Apr 02:00: 1425 records in 0.11 seconds (12977 rps)

history.rollup.ifHistoryMonth 27 Apr 08:51 Timeslot 27 Apr 10:00: 335 records in 0.0107 seconds (31441 rps)

history.rollup.ifHistoryQuarter 27 Apr 08:51 Timeslot 28 Apr 02:00: 335 records in 0.00834 seconds (40182 rps)

history.rollup.ifHistoryWeek 27 Apr 08:51 Timeslot 27 Apr 10:00: 333 records in 0.0152 seconds (21970 rps)

history.rollup.ifHistoryYear 27 Apr 08:51 Timeslot 30 Apr 02:00: 335 records in 0.0104 seconds (32103 rps)

history.rollup.initiatorHistoryMonth 27 Apr 08:51 Timeslot 27 Apr 10:00: 0 records in 0.00166 seconds (0 rps)

history.rollup.initiatorHistoryQuarter 27 Apr 08:51 Timeslot 28 Apr 02:00: 0 records in 0.00134 seconds (0 rps)

history.rollup.initiatorHistoryWeek 27 Apr 08:51 Timeslot 27 Apr 10:00: 0 records in 0.00229 seconds (0 rps)

history.rollup.initiatorHistoryYear 27 Apr 08:51 Timeslot 30 Apr 02:00: 0 records in 0.00167 seconds (0 rps)

history.rollup.lifHistoryMonth 27 Apr 08:51 Timeslot 27 Apr 10:00: 0 records in 0.00163 seconds (0 rps)

history.rollup.lifHistoryQuarter 27 Apr 08:51 Timeslot 28 Apr 02:00: 0 records in 0.00212 seconds (0 rps)

history.rollup.lifHistoryWeek 27 Apr 08:51 Timeslot 27 Apr 10:00: 0 records in 0.00143 seconds (0 rps)

history.rollup.lifHistoryYear 27 Apr 08:51 Timeslot 30 Apr 02:00: 0 records in 0.00157 seconds (0 rps)

history.rollup.lunHistoryMonth 27 Apr 09:03 Timeslot 27 Apr 10:00: 0 records in 0.00188 seconds (0 rps)

history.rollup.lunHistoryQuarter 27 Apr 09:03 Timeslot 28 Apr 02:00: 0 records in 0.0015 seconds (0 rps)

history.rollup.lunHistoryWeek 27 Apr 09:03 Timeslot 27 Apr 10:00: 0 records in 0.00181 seconds (0 rps)

history.rollup.lunHistoryYear 27 Apr 09:03 Timeslot 30 Apr 02:00: 0 records in 0.00169 seconds (0 rps)

history.rollup.perfThreshViolationHistoryMonth 27 Apr 08:51 Timeslot 27 Apr 10:00: 0 records in 0.00145 seconds (0 rps)

history.rollup.perfThreshViolationHistoryQuarter 27 Apr 08:51 Timeslot 28 Apr 02:00: 0 records in 0.00149 seconds (0 rps)

history.rollup.perfThreshViolationHistoryWeek 27 Apr 08:51 Timeslot 27 Apr 10:00: 0 records in 0.00151 seconds (0 rps)

history.rollup.perfThreshViolationHistoryYear 27 Apr 08:51 Timeslot 30 Apr 02:00: 0 records in 0.00144 seconds (0 rps)

history.rollup.qtreeHistoryMonth 27 Apr 06:51 Timeslot 27 Apr 10:00: 208 records in 0.0197 seconds (10572 rps)

history.rollup.qtreeHistoryQuarter 27 Apr 06:51 Timeslot 28 Apr 02:00: 208 records in 0.0143 seconds (14560 rps)

history.rollup.qtreeHistoryWeek 27 Apr 06:51 Timeslot 27 Apr 08:00: 0 records in 0.00294 seconds (0 rps)

history.rollup.qtreeHistoryYear 27 Apr 06:51 Timeslot 30 Apr 02:00: 2458 records in 0.577 seconds (4261 rps)

history.rollup.quotaHistoryMonth 27 Apr 06:51 Timeslot 27 Apr 10:00: 258 records in 0.192 seconds (1346 rps)

history.rollup.quotaHistoryQuarter 27 Apr 06:51 Timeslot 28 Apr 02:00: 258 records in 0.0479 seconds (5383 rps)

history.rollup.quotaHistoryWeek 27 Apr 06:51 Timeslot 27 Apr 08:00: 0 records in 0.0123 seconds (0 rps)

history.rollup.quotaHistoryYear 27 Apr 06:51 Timeslot 30 Apr 02:00: 9052 records in 1.96 seconds (4609 rps)

history.rollup.snapmirrorHistoryMonth 27 Apr 09:03 Timeslot 27 Apr 10:00: 272 records in 0.0691 seconds (3937 rps)

history.rollup.snapmirrorHistoryQuarter 27 Apr 09:03 Timeslot 28 Apr 02:00: 272 records in 0.00891 seconds (30544 rps)

history.rollup.snapmirrorHistoryWeek 27 Apr 09:03 Timeslot 27 Apr 10:00: 272 records in 0.0248 seconds (10955 rps)

history.rollup.snapmirrorHistoryYear 27 Apr 09:03 Timeslot 30 Apr 02:00: 274 records in 0.00991 seconds (27643 rps)

history.rollup.vfilerHistoryMonth 27 Apr 08:27 Timeslot 27 Apr 10:00: 282 records in 0.0088 seconds (32045 rps)

history.rollup.vfilerHistoryQuarter 27 Apr 08:27 Timeslot 28 Apr 02:00: 282 records in 0.00668 seconds (42229 rps)

history.rollup.vfilerHistoryWeek 27 Apr 08:27 Timeslot 27 Apr 10:00: 166 records in 0.0144 seconds (11559 rps)

history.rollup.vfilerHistoryYear 27 Apr 08:27 Timeslot 30 Apr 02:00: 282 records in 0.0196 seconds (14371 rps)

history.rollup.volHistoryMonth 27 Apr 09:03 Timeslot 27 Apr 10:00: 1131 records in 0.229 seconds (4939 rps)

history.rollup.volHistoryQuarter 27 Apr 09:03 Timeslot 28 Apr 02:00: 1131 records in 0.0737 seconds (15337 rps)

history.rollup.volHistoryWeek 27 Apr 09:03 Timeslot 27 Apr 10:00: 1131 records in 0.218 seconds (5193 rps)

history.rollup.volHistoryYear 27 Apr 09:03 Timeslot 30 Apr 02:00: 1131 records in 0.13 seconds (8731 rps)

history.stats.aggrHistory 26 Apr 09:51 0.0858 seconds

history.stats.cpuHistory 27 Apr 07:23 0.218 seconds

history.stats.datatransferHistory 27 Apr 00:12 0.0189 seconds

history.stats.dfHistory  27 Apr 08:24 2.36 seconds

history.stats.fcswHistory 27 Apr 05:21 0.00363 seconds

history.stats.history    26 Apr 13:57 1.47 seconds

history.stats.ifHistory  27 Apr 04:18 0.377 seconds

history.stats.initiatorHistory 27 Apr 06:21 0.00244 seconds

history.stats.lifHistory 26 Apr 12:57 0.000898 seconds

history.stats.lunHistory 27 Apr 02:15 0.00109 seconds

history.stats.perfThreshViolationHistory 26 Apr 11:54 0.00436 seconds

history.stats.qtreeHistory 26 Apr 20:06 0.47 seconds

history.stats.quotaHistory 26 Apr 21:09 1.23 seconds

history.stats.snapmirrorHistory 26 Apr 23:11 0.161 seconds

history.stats.vfilerHistory 26 Apr 10:53 0.182 seconds

history.stats.volHistory 26 Apr 08:50 2.56 seconds

hostservice.internal_username 11 Apr 12:47 HSAdmin3128245486

isDfmVersionRegular      11 Apr 12:47 Yes

licensesMigrated         11 Apr 12:47 1

monitor.dbck.checkObjFlags 27 Apr 05:20 succeeded=31 failed=0 elapsed=0s

monitor.dbck.consolidateHistory 27 Apr 09:08 succeeded=1791 failed=0 elapsed=61s

monitor.dbck.deleteDupQuotaUsers 27 Apr 09:06 succeeded=655 failed=0 elapsed=0s

monitor.dbck.mgmtStationName 27 Apr 08:25 succeeded=178 failed=0 elapsed=0s

monitor.dbck.preCacheGroups 27 Apr 09:05 succeeded=656 failed=0 elapsed=0s

monitor.dbck.pruneUsageMetricHistory 26 Apr 16:20 succeeded=8 failed=0 elapsed=0s

monitor.dbck.rebuildHistoryStatistics 27 Apr 08:24 succeeded=178 failed=0 elapsed=3s

monitor.dbck.removeOrphans 27 Apr 08:59 succeeded=179 failed=0 elapsed=1s

monitor.dbck.verifyDbcache 27 Apr 08:25 succeeded=178 failed=0 elapsed=3s

monitor.discover.network 11 Apr 12:49 none

objectMap.verifier.current 27 Apr 08:25 57

perf.threshviolation.update 27 Apr 09:07

postTransformLogicSuccess 11 Apr 12:47 1

postupgrade.app_policies.create_samples 11 Apr 12:47 no

postupgrade.dp_policies.create_samples 11 Apr 12:47 no

postupgrade.prov_policies.create_samples 11 Apr 12:47 no

postupgrade.storage_services.create_samples 11 Apr 12:47 no

Global Options

Option Name              Option Value

autosupportMonInterval   00:02:00

agentMonInterval         Off

cfMonInterval            00:05:00

clusterMonInterval       00:15:00

cpuMonInterval           00:05:00

dfMonInterval            00:30:00

snapshotMonInterval      00:30:00

diskMonInterval          04:00:00

envMonInterval           00:05:00

fcMonInterval            00:05:00

fsMonInterval            00:15:00

ifMonInterval            00:15:00

licenseMonInterval       04:00:00

lunMonInterval           00:30:00

opsMonInterval           00:10:00

pingMonInterval          00:01:00

qtreeMonInterval         08:00:00

SANHostMonInterval       00:05:00

snapmirrorMonInterval    00:30:00

statusMonInterval        00:10:00

sysInfoMonInterval       01:00:00

userQuotaMonInterval     1:00:00:00

snapvaultMonInterval     Off

srmMonInterval           Off

shareMonInterval         01:00:00

ccMonInterval            04:00:00

vFilerMonInterval        01:00:00

vserverMonInterval       01:00:00

clusterMonInterval       00:15:00

backupDirMonInterval     08:00:00

ndmpMonInterval          01:00:00

dsConformanceMonInterval 01:00:00

dsDRMonInterval          00:30:00

dsProtectionMonInterval  00:30:00

respoolSpaceMonInterval  02:00:00

hostRBACMonInterval      24:00:00

dpReaperInterval         00:30:00

isHsAliveMonInterval     00:01:00

dsUsageMetricMonInterval 2:00:00

authUsePam               no

autosupportAdminContact  FILESERVICES-NAS@f-i.de

autosupportContent       complete

autosupportDestinationURL support.netapp.com/asupprod/post/1.0/postAsup

autosupportDestinationEmail autosupport@netapp.com

autosupportEnabled       Yes

autosupportProtocol      smtp

autosupportRetryCount    4

autosupportRetryDelay    00:15:00

autosupportIncludePerf   Yes

autosupportIncludeProv   Yes

autosupportIncludeVirtual Yes

autosupportIncludeAllDiagInfo No

preferredIPAddressType   IPv4

dpReBaselineMode         Confirm

dpReaperCleanupMode      Orphans

currentEventsCacheSize

discoverInterval         00:05:00

discoverTimeout          00:00:20

discoverEnabled          Yes

discoverNetworks         No

networkDiscoveryLimit    15

discoverAgents           No

discoverClusters         Yes

discoverVfilers          Yes

discoverSAN              No

discoverHosts            No

discoverHostInitEnabled  Yes

SMTPServerName           d100sskext05.d100.intern

SMTPServerBackup

SMTPServerPort           25

monMinFreePercent        10.0

monMinFreeBytes          10000.0

localHostName            dfm-h1.v100.intern

cpuTooBusyThreshold      110

cpuBusyThresholdInterval 01:00:00

qtreeFullThreshold       1000

qtreeFullThresholdInterval 00:00:00

qtreeNearlyFullThreshold 1000

qtreeGrowthEventMinChangePct 100

aggrFullThreshold        95

aggrFullThresholdInterval 00:00:00

aggrNearlyFullThreshold  90

aggrOvercommittedThreshold 1000

aggrNearlyOvercommittedThreshold 1000

aggrSnapshotFullThreshold 90

aggrSnapshotNearlyFullThreshold 80

volFullThreshold         99

volFullThresholdInterval 00:00:00

volNearlyFullThreshold   1000

volGrowthEventMinChangePct 100

volNearlyOvercommittedThreshold 1000

volOvercommittedThreshold 1000

nodesRemainingWarningThreshold 1

provMgrNodesRemainingWarningThreshold 1

protMgrNodesRemainingWarningThreshold 1

volSnapshotFullThreshold 110

volNoFirstSnapThreshold  90

volNearlyNoFirstSnapThreshold 80

volReserveDepletedThreshold 90

volReserveNearlyDepletedThreshold 80

volSnapshotCountThreshold 250

volSnapshotTooOldThreshold 168:00:00:00

licenseExpireWarningThreshold 5

guiRefreshInterval       off

growthRateSensitivity    2

monVerbose               30

monVerboseModule         engine|cpu

pingMonTimeout           180

pingMonRetryDelay        60

hostPingMethod           all

profileTTL               45:00:00:00

credCacheTTL             00:20:00

ldapEnabled              No

ldapBindDN

ldapBindPass

ldapBaseDN

ldapVersion              3

ldapUID                  UID

ldapGID

ldapUGID                 CN

ldapMember               uniqueMember

filerConfigSaveLocalChanges yes

chargebackIncrement      Monthly

chargebackRate           0.42

currencyFormat           $ #,###.##

chargebackDayOfMonth     1

userEmailDefaultDomain

userEnableAlerts         No

userFullThreshold        1000

userNearlyFullThreshold  1000

httpEnabled              No

httpPort                 0

httpsEnabled             Yes

httpsPort                8443

webUIPort                8123

hostLoginProtocol        ssh

hostAdminTransport       https

hostAdminPort            80

hostEnclosureDiscoveryEvents Disabled

agentHostTransport       https

agentHostPort            4092

agentHostLogin           guest

agentHostGuestPassword

agentHostAdminPassword

agentHostCIFSAccount

agentHostCIFSPassword

alertFrom                DFM

hbaportTooBusyThreshold  90

SANHostMonSnapshotLUNs   Yes

srmLargestFiles          20

srmLeastRecentlyModifiedFiles 20

srmLeastRecentlyAccessedFiles 20

srmRecentlyModifiedFiles 20

srmLargestDirs           100

perfAdvisorEnabled       Yes

perfAdvisorTransport     httpsOk

perfAdvisorPollInterval  00:05:00

perfAdvisorMaxMonitorThreads 32

perfArchiveDir           /om-perfdata/

perfAdvisorShowDiagCounters Yes

perfAdvisorShowAllViews  No

serverHTTPSEnabled       Yes

serverHTTPEnabled        No

serverHTTPPort           8088

serverHTTPSPort          8488

serverCertAuthEnabled    Yes

hsNotificationsPurgingInterval 86400

hsNotificationsMaxCount  100000

alarmScriptRunAs

ownerEmailFieldName      ownerEmail

snmpTrapListenerPort     162

snmpTrapRcvdMaxPerWindow 2500

snmpTrapRcvdWindowSize   00:01:00

eventsPurgeInterval      28:00:00:00

snmpTrapListenerEnabled  No

scriptPath

monSNMPTimeout           15

monSNMPRetries           20

groupTreeShowStatus      Enabled

pluginsDir

scriptDir

databaseBackupDir        /opt/NTAPdfm/data

databaseBackupDbengWaitTime 600

backupRetentionCount     2

auditLogForever          Yes

auditLogEnabled          Yes

serverAPILogExclude      host-service-discover|dfm-about

respoolFullThreshold     90

respoolNearlyFullThreshold 80

useHostsEquiv            No

apacheUserName

pmQSMBackupPreferred     Yes

dpDynamicSecondarySizing No

dpMaxFanInRatio          1

vFilerRootVolumeSizeMb   50

ndmpDataUseAllInterfaces 0

favoriteMaxReports       25

recentMaxReports         25

defReportLinesPerPage    30

maxReportLinesPerPage    1000

reportsArchiveDir

snapshotDiscoveryEventsEnabled No

dataExportDir

dfmDataExportEnabled     No

perfDataExportEnabled    No

perfAdvThreshViolationMonInterval 00:00:15:00

enableFQDNInFilerViewLinks Yes

volNearlyOverDeduplicatedThreshold 1000

volOverDeduplicatedThreshold 1000

aggrNearlyOverDeduplicatedThreshold 1000

aggrOverDeduplicatedThreshold 1000

dpRestoreTransfersPerHost 8

processHostPrimaryAddress warn

processOSSVPrimaryAddress warn

pmUseSDUCompatibleSnapshotNames No

rshBinary

perfExportDir

perfSampleRate1          00:00:1:00

perfSampleRate2          00:00:5:00

perfSampleRate3          00:00:15:00

perfSampleRate4          00:00:30:00

perfMaxObjectInstancesInBarChart 20

autoClientStatEnabled    No

clientStatCpuThreshold   80

clientStatTotalOpsRate   30

clientStatThresholdPeriod 300

clientStatNfsLatency     30

clientStatCifsLatency    30

clientStatMinTotalOpsRate 500

dpSnapNameFormat         %T_%R_%L_%H_%N_%A

dpSnapNameOption         global-format

dpSnapNameScriptPath

dpSnapNameScriptRunAs

dpPriVolNameOption       global-format

dpSecVolNameOption       global-format

dpPriVolNameFormat       %L

dpSecVolNameFormat       %L_%C_%S_%V

dpPriVolNameScriptPath

dpPriVolNameScriptRunAs

hostClockSkewedThreshold 60

hostClockNearlySkewedThreshold 30

dpSecVolNameScriptPath

dpSecVolNameScriptRunAs

dpSecQtreeNameFormat     %Q

dsUsageMetricCommentFields

dsUsageMetricIoInterval  1:00:00:00

dsUsageMetricSpaceInterval 1:00:00:00

reportDesignPath         /opt/NTAPdfm/reports/

keystorePassphraseFile

snapmirrorLagErrorThreshold 172800

snapmirrorLagWarningThreshold 129600

webUIMinHeapSizeMB       256

webUIMaxHeapSizeMB       1024

webUIMinPermGenSizeMB    128

webUIMaxPermGenSizeMB    512

Object Counts

Object Type                    Count

Administrator                  27

Aggregate                      62

App Policy                     2

Array LUNs                     0

Backup jobs started within last week 0

Clusters                       0

Configuration                  4

Datasets containing generic app objects 0

Datasets containing generic app objects and with storage service attached 0

Datasets with application policy attached 0

Datasets with storage service attached 0

Disks                          5420

DP Managed OSSV Rels           0

DP Managed SnapMirror Rels     0

DP Managed SnapVault Rels      0

DP Policy                      36

DP Schedule                    18

DP Throttle                    2

FCP Target                     26

Host                           324

interface                      1027

Mgmt Station                   1

Mirror                         272

Mount jobs started within last week 0

Network                        5

OSSV Hosts                     0

OSSV Rels                      0

Primary Storage Systems        0

Prov Policy                    16

Qtree                          59394

Qtrees in DP Managed QtreeSnapMirror Rel 0

Qtrees in DP Managed SnapVault Rel 0

Qtrees in QtreeSnapMirror Rel  0

Qtrees in SnapVault Rel        0

QuotaUser                      112774

Resource Group                 18

Resource Pool                  2

Restore jobs started within last week 0

Role                           35

Secondary Storage Systems      0

SnapMirror Rels                292

SnapVault Rels                 0

Storage Service                16

Unmount jobs started within last week 0

UserQuota                      116089

vFilers                        282

Virtual Servers                0

Volume                         1132

Volumes in DP Managed VolumeSnapMirror Rel 0

Volumes in VolumeSnapMirror Rel 538

Zapi Hosts                     42

Monitoring Timestamps

Timestamp Name       Status   Interval     Default      Last Updated     Status   Error if older than ...

ccTimestamp          Normal   4 hours      4 hours                                27 Apr 05:10

cfTimestamp          Normal   5 minutes    5 minutes    27 Apr 09:10     Normal   27 Apr 09:05

clusterTimestamp     Normal   15 minutes   15 minutes                             27 Apr 08:55

cpuTimestamp         Normal   5 minutes    5 minutes    27 Apr 09:10     Normal   27 Apr 09:05

dfTimestamp          Normal   30 minutes   30 minutes   27 Apr 09:10     Normal   27 Apr 08:40

diskTimestamp        Normal   4 hours      4 hours      27 Apr 09:10     Normal   27 Apr 05:10

envTimestamp         Normal   5 minutes    5 minutes    27 Apr 09:10     Normal   27 Apr 09:05

fcTimestamp          Normal   5 minutes    5 minutes    27 Apr 09:10     Normal   27 Apr 09:05

fsTimestamp          Normal   15 minutes   15 minutes   27 Apr 09:09     Normal   27 Apr 08:55

hostPingTimestamp    Normal   1 minute     1 minute     27 Apr 09:10     Normal   27 Apr 09:09

ifTimestamp          Normal   15 minutes   15 minutes   27 Apr 09:10     Normal   27 Apr 08:55

licenseTimestamp     Normal   4 hours      4 hours      27 Apr 09:04     Normal   27 Apr 05:10

lunTimestamp         Normal   30 minutes   30 minutes   27 Apr 09:10     Normal   27 Apr 08:40

opsTimestamp         Normal   10 minutes   10 minutes   27 Apr 09:10     Normal   27 Apr 09:00

qtreeTimestamp       Normal   8 hours      8 hours      11 Apr 13:02     Error    27 Apr 01:10

rbacTimestamp        Normal   1 day        1 day        27 Apr 09:10     Normal   26 Apr 09:10

userQuotaTimestamp   Normal   1 day        1 day        27 Apr 08:15     Normal   26 Apr 09:10

sanhostTimestamp     Normal   5 minutes    5 minutes                              27 Apr 09:05

snapmirrorTimestamp  Normal   30 minutes   30 minutes   27 Apr 09:09     Normal   27 Apr 08:40

snapshotTimestamp    Normal   30 minutes   30 minutes   27 Apr 09:10     Normal   27 Apr 08:40

statusTimestamp      Normal   10 minutes   10 minutes   27 Apr 09:10     Normal   27 Apr 09:00

sysInfoTimestamp     Normal   1 hour       1 hour       27 Apr 09:03     Normal   27 Apr 08:10

svTimestamp          Error    Off          30 minutes   Not updated ( snapvaultMonInterval is set to Off )

svMonTimestamp       Normal   8 hours      8 hours                                27 Apr 01:10

xmlQtreeTimestamp    Normal   8 hours      8 hours      27 Apr 08:15     Normal   27 Apr 01:10

vFilerTimestamp      Normal   1 hour       1 hour       27 Apr 09:09     Normal   27 Apr 08:10

vserverTimestamp     Normal   1 hour       1 hour                                 27 Apr 08:10

Database

monitordb.db                     728 MB

dbFileVersion                    10

ConnCount                        61 connections

MaxCacheSize                     33055224 KBytes

CurrentCacheSize                 719264 KBytes

PeakCacheSize                    719264 KBytes

PageSize                         8192 Bytes

Cluster Details

Object Type                    Count

Clusters                       0

Controllers                    0

Virtual Systems                0

Volumes (All)                  0

Striped Volumes                0

Aggregates (All)               0

Striped Aggregates             0

Logical Interfaces (All)       0

Interface Groups               0

Ports (All)                    0

Physical Disk Details

Total Capacity                 0 bytes

Total Used                     0 bytes

Logs

discovery      1.44 KB 11 Apr 12:49

dfmmonitor     2.36 MB 27 Apr 09:10

dfmeventd      17.6 KB 27 Apr 00:58

dfmserver      20.0 KB 26 Apr 23:10

dfmscheduler   354 bytes 17 Apr 15:40

dfmwatchdog    310 KB 27 Apr 09:06

dfm            43.1 KB 20 Apr 13:32

sybase         424 KB 27 Apr 09:08

pingmon        6.54 KB 20 Apr 23:35

audit          567 KB 27 Apr 08:46

vfmon          550 KB 27 Apr 09:05

Services

sql        Normal Started

webui      Normal Started

http       Normal Started

eventd     Normal Started

monitor    Normal Started

scheduler  Normal Started

server     Normal Started

watchdog   Normal Started

Time Since Confirmed Alive

Eventd     5 seconds

Monitor    19 seconds

Scheduler  24 seconds

Server     11 seconds

Watchdog   0 seconds

Events

There are no events that have warning severity or worse.

DP Job Information

Job State                        Count

Jobs Running                     0

Jobs Completed Total             0

Jobs Aborted Total               0

Jobs Aborting Total              0

Jobs Completed Today             0

Jobs Aborted Today               0

Jobs Aborting Today              0

Dataset Protection Status

Protection State                 Count

Protected                        0

Unprotected                      0

Event Counts

Table                            Count

Events                           578872

Current Events                   578282

Abnormal Events                  2523

Event Type Counts

Event Type                 Count

userquota.files.soft.limit 117081

userquota.kbytes.soft.limit 117081

userquota.kbytes           117081

userquota.files            117081

qtree.files                42426

qtree.kbytes               42426

df.snapshot.kbytes         1698

df.avail                   1577

df.kbytes                  1568

snap-status                1564

Version                          5.0 (5.0)

adaikkap
3,782 Views

Hi Stefan,

     As I suspected, you have a lot of userquota's hundred 1000 +. But you are already on OC 5.0 64 bit  platform with a really beefy server with 64GB RAM. I suggest you to open a case with NetApp Support to look into this issue.

Regards

adai

anton_oks
3,782 Views

Hi there.

We see exactly the same damned problem. I've an open case, but so far no usable answers from the support guys at NetApp

  1. Any feedback from this/your support case?
  2. Anybody out there who has any ideas?

Any help is highly welcome.

Thanks & regards

Anton

skellner
3,782 Views

Hi Anton,

no, sorry. No news on this.

Regards

Stefan

Public