qtreeTimestamp Error

skellner · ‎2012-03-19

DFM Server of my customer isn't updating qtree information. All hosts show the qtreeTimestamp in Error. However the dfrm host diag shows no issues regarding access or authentication. Please find attached the dfm host diag of a host example. Thanks in advance for any hints.

j255030@v100spldv02:/> dfm host diag q100bpmc001

Network Connectivity

IP Address 17.247.34.129

Network (discovery disabled)

DNS Aliases Q100BPMC001.q100.intern

DNS Addresses 17.247.34.129

SNMP Version in Use SNMPv3

SNMPv1 Passed (195 ms)

SNMP Community public

SNMP sysName Q100BPMC001.q100.intern

SNMP sysObjectID .1.3.6.1.4.1.789.2.3 (Clustered Filer)

SNMP productId 0118074118

SNMPv3 Passed (8 ms)

SNMPv3 Auth Protocol MD5

SNMPv3 Privacy Enabled No

SNMPv3 Username omuser

SNMP sysName Q100BPMC001.q100.intern

SNMP sysObjectID .1.3.6.1.4.1.789.2.3 (Clustered Filer)

SNMP productId 0118074118

ICMP Echo Passed

HTTP Passed (1 ms)

NDMP Ping Passed (port 10000, 1 ms)

NDMP Connect Error: Couldn't authenticate NDMP connection to host 17.247.34.129

RSH timed out

SSH Passed (10353 ms)

RLM Skipped (hostRLMAddress is empty)

XML (https port 443) Passed (11 ms)

Host Details

According to: DataFabric Manager server Host

Host Name Q100BPMC001.q100.intern Q100BPMC001.q100.intern

System ID 0118074118 0118074118

Model FAS6080 FAS6080

Type Clustered Storage System Clustered Storage System

OS Version 7.3.3P5 7.3.3P5

Revisions 330,8.0,2.1 330,8.0,2.1

Monitoring Timestamps

Timestamp Name Status Interval Default Last Updated Status Error if older than ...

ccTimestamp Error 1 hour 4 hours 19 MÃ¤r 12:39 Normal 19 MÃ¤r 12:12

cfTimestamp Error 3 minutes 5 minutes 19 MÃ¤r 13:11 Normal 19 MÃ¤r 13:09

clusterTimestamp Normal 15 minutes 15 minutes 19 MÃ¤r 12:57

cpuTimestamp Normal 5 minutes 5 minutes 19 MÃ¤r 13:12 Normal 19 MÃ¤r 13:07

dfTimestamp Normal 30 minutes 30 minutes 19 MÃ¤r 12:53 Normal 19 MÃ¤r 12:42

diskTimestamp Normal 4 hours 4 hours 19 MÃ¤r 09:33 Normal 19 MÃ¤r 09:12

envTimestamp Normal 5 minutes 5 minutes 19 MÃ¤r 13:08 Normal 19 MÃ¤r 13:07

fcTimestamp Normal 5 minutes 5 minutes 19 MÃ¤r 13:10 Normal 19 MÃ¤r 13:07

fsTimestamp Normal 15 minutes 15 minutes 19 MÃ¤r 13:07 Normal 19 MÃ¤r 12:57

hostPingTimestamp Error 2 minutes 1 minute 19 MÃ¤r 13:11 Normal 19 MÃ¤r 13:10

ifTimestamp Normal 15 minutes 15 minutes 19 MÃ¤r 12:59 Normal 19 MÃ¤r 12:57

licenseTimestamp Error 1 day 4 hours 18 MÃ¤r 13:25 Normal 18 MÃ¤r 13:12

lunTimestamp Error 1 hour 30 minutes 19 MÃ¤r 12:13 Normal 19 MÃ¤r 12:12

opsTimestamp Normal 10 minutes 10 minutes 19 MÃ¤r 13:05 Normal 19 MÃ¤r 13:02

qtreeTimestamp Error 1 day 8 hours 18 MÃ¤r 13:12 <==============

rbacTimestamp Normal 1 day 1 day 18 MÃ¤r 13:26 Normal 18 MÃ¤r 13:12

userQuotaTimestamp Normal 1 day 1 day 18 MÃ¤r 13:12

sanhostTimestamp Error 2 hours 5 minutes 19 MÃ¤r 11:12

snapmirrorTimestamp Error 2 hours 30 minutes 19 MÃ¤r 00:42 Error 19 MÃ¤r 11:12

snapshotTimestamp Error 1 hour 30 minutes 19 MÃ¤r 13:01 Normal 19 MÃ¤r 12:12

statusTimestamp Error 2 minutes 10 minutes 19 MÃ¤r 13:11 Normal 19 MÃ¤r 13:10

sysInfoTimestamp Normal 1 hour 1 hour 19 MÃ¤r 12:33 Normal 19 MÃ¤r 12:12

svTimestamp Error Off 30 minutes Not updated ( snapvaultMonInterval is set to Off )

svMonTimestamp Normal 8 hours 8 hours 19 MÃ¤r 05:12

xmlQtreeTimestamp Error 1 day 8 hours 18 MÃ¤r 13:12

vFilerTimestamp Error 30 minutes 1 hour 19 MÃ¤r 12:52 Normal 19 MÃ¤r 12:42

vserverTimestamp Normal 1 hour 1 hour 19 MÃ¤r 12:12

Performance Advisor Checklist

perfAdvisorEnabled Passed

hostType Passed

hostRevision Passed

hostLogin Passed

perfAdvisorTransport Passed

j255030@v100spldv02:/> dfm about

Version 5.0 (5.0)

Executable Type 64-bit

Serial Number 1-50-005803

Edition Standard edition of DataFabric Manager server

Administrator Name j255030

Host Name v100spldv02

Host IP Address 127.0.0.2

Host Full Name v100spldv02.v100.intern

Node limit 250 (currently managing 69)

Operating System SUSE Linux Enterprise Server 11 (x86_64) 2.6.27.19-5-default x86_64

CPU Count 8

System Memory 19955 MB (load excluding cached memory: 40%)

Installation Directory /opt/NTAPdfm

415 GB free (59.4%)

Perf Data Directory /om-perfdata

Data Export Directory /opt/NTAPdfm/dataExport

Database Backup Directory /opt/NTAPdfm/data

Reports Archival Directory /opt/NTAPdfm/reports

Database Directory /om-db

96.4 GB free (76.5%)

Database Log Directory /om-db

96.4 GB free (76.5%)

Licensed Features DataFabric Manager server: installed

Storage Area Network Option: installed

File SRM Option: installed

adaikkap · ‎2012-03-19

Hi Stefan,

Firstly I request you to change all you monitoring intervals back to default. Esp the following monitors ?

ccTimestamp Error 1 hour 4 hours 19 MÃ¤r 12:39 Normal 19 MÃ¤r 12:12

cfTimestamp Error 3 minutes 5 minutes 19 MÃ¤r 13:11 Normal 19 MÃ¤r 13:09

hostPingTimestamp

Error

2 minutes

1 minute

19 MÃ¤r 13:11

Normal 19 MÃ¤r 13:10

statusTimestamp

Error

2 minutes

10 minutes 19 MÃ¤r 13:11

Normal 19 MÃ¤r 13:10

vFilerTimestamp

Error

30 minutes 1 hour

19 MÃ¤r 12:52

Normal 19 MÃ¤r 12:42

Regards

adai

skellner · ‎2012-04-26

Hi Adai,

in the meantime we adjusted all monitoring intervals to default. Additionally we setup a new dfm Server. However, the problem nearly stays the same. The Qtree (XML) monitor gets only infrequently. This happens on the new and on the old dfm server. Is there a possibility to analyse why the qtree monitor doesn't get updated every time? Any kind of debug mode I can set in dfm so that I can check the log files for any reason?

Regards

Stefan

adaikkap · ‎2012-04-26

Hi Stefan,

Can I have the output of dfm diag once more ? Do you have any rough estimate of what would be your userquota/qtree quota numbers ? In the mean time I strongly suggest you to open a support case so that they could do a packet trace and other troubleshooting to solve the issue as they would have all needed logs and details. Which is little difficult to share over communities for all the know reasons.

Regards

adai

skellner · ‎2012-04-27

Hi Adai,

attached the dfm diag.

Thanks

Stefan

dfm-h1:~ # dfm diag

Management Station

Version 5.0 (5.0)

Executable Type 64-bit

Serial Number 1-50-005803

Edition Standard edition of DataFabric Manager server

Administrator Name root

Host Name dfm-h1

Host IP Address 17.247.33.201

Host Full Name v100spldv01.v100.intern

Node limit 250 (currently managing 42)

Operating System SUSE Linux Enterprise Server 11 (x86_64) 2.6.32.12-0.7-default x86_64

CPU Count 8

System Memory 64560 MB (load excluding cached memory: 11%)

Installation Directory /opt/NTAPdfm

17.3 GB free (33.1%)

Perf Data Directory /om-perfdata

Data Export Directory /opt/NTAPdfm/dataExport

Database Backup Directory /opt/NTAPdfm/data

Reports Archival Directory /opt/NTAPdfm/reports

Database Directory /om-db

477 GB free (94.7%)

Licensed Features DataFabric Manager server: installed

Storage Area Network Option: installed

File SRM Option: installed

Installed Plugins Storage System Config 6.5.1 (6.5.1) - storage systems and vFilers

Storage System Config 6.5.2 (6.5.2) - storage systems and vFilers

Storage System Config 6.5.3 (6.5.3) - storage systems and vFilers

Storage System Config 6.5.4 (6.5.4) - storage systems and vFilers

Storage System Config 6.5.5 (6.5.5) - storage systems and vFilers

Storage System Config 6.5.6 (6.5.6) - storage systems and vFilers

Storage System Config 6.5.7 (6.5.7) - storage systems and vFilers

Storage System Config 7.0 (7.0.0.1) - storage systems and vFilers

Storage System Config 7.0.1 (7.0.1.1) - storage systems and vFilers

Storage System Config 7.0.2 (7.0.2) - storage systems and vFilers

Storage System Config 7.0.3 (7.0.3) - storage systems and vFilers

Storage System Config 7.0.4 (7.0.4) - storage systems and vFilers

Storage System Config 7.0.5 (7.0.5) - storage systems and vFilers

Storage System Config 7.0.6 (7.0.6) - storage systems and vFilers

Storage System Config 7.0.7 (7.0.7) - storage systems and vFilers

Storage System Config 7.1 (7.1.0.1) - storage systems and vFilers

Storage System Config 7.1.1 (7.1.1.1) - storage systems and vFilers

Storage System Config 7.1.2 (7.1.2.1) - storage systems and vFilers

Storage System Config 7.1.3 (7.1.3) - storage systems and vFilers

Storage System Config 7.2 (7.2) - storage systems and vFilers

Storage System Config 7.2.1 (7.2.1.1) - storage systems and vFilers

Storage System Config 7.2.2 (7.2.2) - storage systems and vFilers

Storage System Config 7.2.3 (7.2.3) - storage systems and vFilers

Storage System Config 7.2.4 (7.2.4) - storage systems and vFilers

Storage System Config 7.2.5 (7.2.5.1) - storage systems and vFilers

Storage System Config 7.2.6 (7.2.6.1) - storage systems and vFilers

Storage System Config 7.2.7 (7.2.7) - storage systems and vFilers

Storage System Config 7.3 (7.3) - storage systems and vFilers

Storage System Config 7.3.1 (7.3.1.1) - storage systems and vFilers

Storage System Config 7.3.2 (7.3.2) - storage systems and vFilers

Storage System Config 7.3.3 (7.3.3) - storage systems and vFilers

Storage System Config 7.3.4 (7.3.4) - storage systems and vFilers

Storage System Config 7.3.5 (7.3.5) - storage systems and vFilers

Storage System Config 8.0 (8.0) - storage systems and vFilers

Storage System Config 8.0.1 (8.0.1) - storage systems and vFilers

Storage System Config 8.0.2 (8.0.2) - storage systems and vFilers

Global Status

Status Key Last Updated Status Value

autosupport.installTime 11 Apr 12:47 2012-04-11 12:47:01.914262

autosupport.lastStatusSendTime 11 Apr 12:47 1335363937

autoThrottleMultiplier 27 Apr 06:47 100

dpSnapNameFormatVersion 11 Apr 12:47 1

events.purged.currentEvents 26 Apr 12:49 0

events.purged.dpJobProgressEvents 26 Apr 12:49 0

events.purged.dpJobs 26 Apr 12:49 0

events.purged.events 26 Apr 12:49 0

generationId 11 Apr 12:47 691

generationId.alonso 11 Apr 12:47 88

generationId.aquarius 11 Apr 12:47 1

generationId.aquarius-ms1 11 Apr 12:47 4

generationId.aquarius-ms2 11 Apr 12:47 25

generationId.aquarius-ms2new 11 Apr 12:47 18

generationId.aquarius-ms3 11 Apr 12:47 86

generationId.aquarius-ms4 11 Apr 12:47 11

generationId.aquarius-ms4-cba 11 Apr 12:47 3

generationId.asada 11 Apr 12:47 1

generationId.asada-config 11 Apr 12:47 1

generationId.baldi 11 Apr 12:47 51

generationId.beni 11 Apr 12:47 1

generationId.beni-ms3 11 Apr 12:47 5

generationId.carter 11 Apr 12:47 38

generationId.clark 11 Apr 12:47 42

generationId.daly-ms1 11 Apr 12:47 2

generationId.daly-ms2 11 Apr 12:47 4

generationId.daly-ms3 11 Apr 12:47 8

generationId.frederick 11 Apr 12:47 64

generationId.freya-reloaded 11 Apr 12:47 104

generationId.gayatri 11 Apr 12:47 21

generationId.guiredesign 11 Apr 12:47 1

generationId.hel 11 Apr 12:47 18

generationId.indra 11 Apr 12:47 30

generationId.isis 11 Apr 12:47 25

generationId.janak 11 Apr 12:47 5

generationId.janak-cloud 11 Apr 12:47 7

generationId.janneau 11 Apr 12:47 77

generationId.juno-cbsprint 11 Apr 12:47 4

generationId.juno-dc 11 Apr 12:47 1

generationId.juno-sprint1 11 Apr 12:47 1

generationId.juno-sprint3 11 Apr 12:47 1

generationId.juno-ssm 11 Apr 12:47 1

generationId.juno-ssm-sm-susp 11 Apr 12:47 2

generationId.juno-ssm-sprint4 11 Apr 12:47 4

generationId.kahlua 11 Apr 12:47 16

generationId.kakia 11 Apr 12:47 1

generationId.kakia-dc 11 Apr 12:47 3

generationId.kakia-sprint1 11 Apr 12:47 10

generationId.oldmonk 11 Apr 12:47 36

generationId.padataexport 11 Apr 12:47 4

generationId.pucker 11 Apr 12:47 7

groupCache.verifier.current 27 Apr 08:25 3422

groupChildCache.verifier.current 27 Apr 08:25 3422

history.rollup.aggrHistoryMonth 27 Apr 09:03 Timeslot 27 Apr 10:00: 62 records in 0.00644 seconds (9632 rps)

history.rollup.aggrHistoryQuarter 27 Apr 09:03 Timeslot 28 Apr 02:00: 62 records in 0.0181 seconds (3417 rps)

history.rollup.aggrHistoryWeek 27 Apr 09:03 Timeslot 27 Apr 10:00: 62 records in 0.00366 seconds (16954 rps)

history.rollup.aggrHistoryYear 27 Apr 09:03 Timeslot 30 Apr 02:00: 62 records in 0.00435 seconds (14237 rps)

history.rollup.cpuHistoryMonth 27 Apr 08:51 Timeslot 27 Apr 10:00: 324 records in 0.0104 seconds (31023 rps)

history.rollup.cpuHistoryQuarter 27 Apr 08:51 Timeslot 28 Apr 02:00: 324 records in 0.0081 seconds (40020 rps)

history.rollup.cpuHistoryWeek 27 Apr 08:51 Timeslot 27 Apr 10:00: 324 records in 0.0119 seconds (27314 rps)

history.rollup.cpuHistoryYear 27 Apr 08:51 Timeslot 30 Apr 02:00: 324 records in 0.008 seconds (40494 rps)

history.rollup.datatransferHistoryDay 27 Apr 09:03 Timeslot 27 Apr 09:15: 3 records in 0.989 seconds (3 rps)

history.rollup.datatransferHistoryMonth 27 Apr 09:03 Timeslot 27 Apr 10:00: 151 records in 0.008 seconds (18873 rps)

history.rollup.datatransferHistoryQuarter 27 Apr 09:03 Timeslot 28 Apr 02:00: 151 records in 0.0478 seconds (3156 rps)

history.rollup.datatransferHistoryWeek 27 Apr 09:03 Timeslot 27 Apr 10:00: 21 records in 0.00537 seconds (3912 rps)

history.rollup.datatransferHistoryYear 27 Apr 09:03 Timeslot 30 Apr 02:00: 151 records in 0.00761 seconds (19855 rps)

history.rollup.dfHistoryMonth 27 Apr 09:03 Timeslot 27 Apr 10:00: 2386 records in 0.277 seconds (8626 rps)

history.rollup.dfHistoryQuarter 27 Apr 09:03 Timeslot 28 Apr 02:00: 2386 records in 0.326 seconds (7312 rps)

history.rollup.dfHistoryWeek 27 Apr 09:03 Timeslot 27 Apr 10:00: 2386 records in 1.29 seconds (1856 rps)

history.rollup.dfHistoryYear 27 Apr 09:03 Timeslot 30 Apr 02:00: 2386 records in 0.293 seconds (8144 rps)

history.rollup.fcswHistoryMonth 27 Apr 08:51 Timeslot 27 Apr 10:00: 0 records in 0.0109 seconds (0 rps)

history.rollup.fcswHistoryQuarter 27 Apr 08:51 Timeslot 28 Apr 02:00: 0 records in 0.00175 seconds (0 rps)

history.rollup.fcswHistoryWeek 27 Apr 08:51 Timeslot 27 Apr 10:00: 0 records in 0.00129 seconds (0 rps)

history.rollup.fcswHistoryYear 27 Apr 08:51 Timeslot 30 Apr 02:00: 0 records in 0.0018 seconds (0 rps)

history.rollup.historyMonth 27 Apr 08:45 Timeslot 27 Apr 10:00: 1423 records in 0.537 seconds (2649 rps)

history.rollup.historyQuarter 27 Apr 08:45 Timeslot 28 Apr 02:00: 1423 records in 0.521 seconds (2732 rps)

history.rollup.historyWeek 27 Apr 08:45 Timeslot 27 Apr 10:00: 1349 records in 0.759 seconds (1778 rps)

history.rollup.historyYear 27 Apr 08:45 Timeslot 30 Apr 02:00: 1425 records in 0.11 seconds (12977 rps)

history.rollup.ifHistoryMonth 27 Apr 08:51 Timeslot 27 Apr 10:00: 335 records in 0.0107 seconds (31441 rps)

history.rollup.ifHistoryQuarter 27 Apr 08:51 Timeslot 28 Apr 02:00: 335 records in 0.00834 seconds (40182 rps)

history.rollup.ifHistoryWeek 27 Apr 08:51 Timeslot 27 Apr 10:00: 333 records in 0.0152 seconds (21970 rps)

history.rollup.ifHistoryYear 27 Apr 08:51 Timeslot 30 Apr 02:00: 335 records in 0.0104 seconds (32103 rps)

history.rollup.initiatorHistoryMonth 27 Apr 08:51 Timeslot 27 Apr 10:00: 0 records in 0.00166 seconds (0 rps)

history.rollup.initiatorHistoryQuarter 27 Apr 08:51 Timeslot 28 Apr 02:00: 0 records in 0.00134 seconds (0 rps)

history.rollup.initiatorHistoryWeek 27 Apr 08:51 Timeslot 27 Apr 10:00: 0 records in 0.00229 seconds (0 rps)

history.rollup.initiatorHistoryYear 27 Apr 08:51 Timeslot 30 Apr 02:00: 0 records in 0.00167 seconds (0 rps)

history.rollup.lifHistoryMonth 27 Apr 08:51 Timeslot 27 Apr 10:00: 0 records in 0.00163 seconds (0 rps)

history.rollup.lifHistoryQuarter 27 Apr 08:51 Timeslot 28 Apr 02:00: 0 records in 0.00212 seconds (0 rps)

history.rollup.lifHistoryWeek 27 Apr 08:51 Timeslot 27 Apr 10:00: 0 records in 0.00143 seconds (0 rps)

history.rollup.lifHistoryYear 27 Apr 08:51 Timeslot 30 Apr 02:00: 0 records in 0.00157 seconds (0 rps)

history.rollup.lunHistoryMonth 27 Apr 09:03 Timeslot 27 Apr 10:00: 0 records in 0.00188 seconds (0 rps)

history.rollup.lunHistoryQuarter 27 Apr 09:03 Timeslot 28 Apr 02:00: 0 records in 0.0015 seconds (0 rps)

history.rollup.lunHistoryWeek 27 Apr 09:03 Timeslot 27 Apr 10:00: 0 records in 0.00181 seconds (0 rps)

history.rollup.lunHistoryYear 27 Apr 09:03 Timeslot 30 Apr 02:00: 0 records in 0.00169 seconds (0 rps)

history.rollup.perfThreshViolationHistoryMonth 27 Apr 08:51 Timeslot 27 Apr 10:00: 0 records in 0.00145 seconds (0 rps)

history.rollup.perfThreshViolationHistoryQuarter 27 Apr 08:51 Timeslot 28 Apr 02:00: 0 records in 0.00149 seconds (0 rps)

history.rollup.perfThreshViolationHistoryWeek 27 Apr 08:51 Timeslot 27 Apr 10:00: 0 records in 0.00151 seconds (0 rps)

history.rollup.perfThreshViolationHistoryYear 27 Apr 08:51 Timeslot 30 Apr 02:00: 0 records in 0.00144 seconds (0 rps)

history.rollup.qtreeHistoryMonth 27 Apr 06:51 Timeslot 27 Apr 10:00: 208 records in 0.0197 seconds (10572 rps)

history.rollup.qtreeHistoryQuarter 27 Apr 06:51 Timeslot 28 Apr 02:00: 208 records in 0.0143 seconds (14560 rps)

history.rollup.qtreeHistoryWeek 27 Apr 06:51 Timeslot 27 Apr 08:00: 0 records in 0.00294 seconds (0 rps)

history.rollup.qtreeHistoryYear 27 Apr 06:51 Timeslot 30 Apr 02:00: 2458 records in 0.577 seconds (4261 rps)

history.rollup.quotaHistoryMonth 27 Apr 06:51 Timeslot 27 Apr 10:00: 258 records in 0.192 seconds (1346 rps)

history.rollup.quotaHistoryQuarter 27 Apr 06:51 Timeslot 28 Apr 02:00: 258 records in 0.0479 seconds (5383 rps)

history.rollup.quotaHistoryWeek 27 Apr 06:51 Timeslot 27 Apr 08:00: 0 records in 0.0123 seconds (0 rps)

history.rollup.quotaHistoryYear 27 Apr 06:51 Timeslot 30 Apr 02:00: 9052 records in 1.96 seconds (4609 rps)

history.rollup.snapmirrorHistoryMonth 27 Apr 09:03 Timeslot 27 Apr 10:00: 272 records in 0.0691 seconds (3937 rps)

history.rollup.snapmirrorHistoryQuarter 27 Apr 09:03 Timeslot 28 Apr 02:00: 272 records in 0.00891 seconds (30544 rps)

history.rollup.snapmirrorHistoryWeek 27 Apr 09:03 Timeslot 27 Apr 10:00: 272 records in 0.0248 seconds (10955 rps)

history.rollup.snapmirrorHistoryYear 27 Apr 09:03 Timeslot 30 Apr 02:00: 274 records in 0.00991 seconds (27643 rps)

history.rollup.vfilerHistoryMonth 27 Apr 08:27 Timeslot 27 Apr 10:00: 282 records in 0.0088 seconds (32045 rps)

history.rollup.vfilerHistoryQuarter 27 Apr 08:27 Timeslot 28 Apr 02:00: 282 records in 0.00668 seconds (42229 rps)

history.rollup.vfilerHistoryWeek 27 Apr 08:27 Timeslot 27 Apr 10:00: 166 records in 0.0144 seconds (11559 rps)

history.rollup.vfilerHistoryYear 27 Apr 08:27 Timeslot 30 Apr 02:00: 282 records in 0.0196 seconds (14371 rps)

history.rollup.volHistoryMonth 27 Apr 09:03 Timeslot 27 Apr 10:00: 1131 records in 0.229 seconds (4939 rps)

history.rollup.volHistoryQuarter 27 Apr 09:03 Timeslot 28 Apr 02:00: 1131 records in 0.0737 seconds (15337 rps)

history.rollup.volHistoryWeek 27 Apr 09:03 Timeslot 27 Apr 10:00: 1131 records in 0.218 seconds (5193 rps)

history.rollup.volHistoryYear 27 Apr 09:03 Timeslot 30 Apr 02:00: 1131 records in 0.13 seconds (8731 rps)

history.stats.aggrHistory 26 Apr 09:51 0.0858 seconds

history.stats.cpuHistory 27 Apr 07:23 0.218 seconds

history.stats.datatransferHistory 27 Apr 00:12 0.0189 seconds

history.stats.dfHistory 27 Apr 08:24 2.36 seconds

history.stats.fcswHistory 27 Apr 05:21 0.00363 seconds

history.stats.history 26 Apr 13:57 1.47 seconds

history.stats.ifHistory 27 Apr 04:18 0.377 seconds

history.stats.initiatorHistory 27 Apr 06:21 0.00244 seconds

history.stats.lifHistory 26 Apr 12:57 0.000898 seconds

history.stats.lunHistory 27 Apr 02:15 0.00109 seconds

history.stats.perfThreshViolationHistory 26 Apr 11:54 0.00436 seconds

history.stats.qtreeHistory 26 Apr 20:06 0.47 seconds

history.stats.quotaHistory 26 Apr 21:09 1.23 seconds

history.stats.snapmirrorHistory 26 Apr 23:11 0.161 seconds

history.stats.vfilerHistory 26 Apr 10:53 0.182 seconds

history.stats.volHistory 26 Apr 08:50 2.56 seconds

hostservice.internal_username 11 Apr 12:47 HSAdmin3128245486

isDfmVersionRegular 11 Apr 12:47 Yes

licensesMigrated 11 Apr 12:47 1

monitor.dbck.checkObjFlags 27 Apr 05:20 succeeded=31 failed=0 elapsed=0s

monitor.dbck.consolidateHistory 27 Apr 09:08 succeeded=1791 failed=0 elapsed=61s

monitor.dbck.deleteDupQuotaUsers 27 Apr 09:06 succeeded=655 failed=0 elapsed=0s

monitor.dbck.mgmtStationName 27 Apr 08:25 succeeded=178 failed=0 elapsed=0s

monitor.dbck.preCacheGroups 27 Apr 09:05 succeeded=656 failed=0 elapsed=0s

monitor.dbck.pruneUsageMetricHistory 26 Apr 16:20 succeeded=8 failed=0 elapsed=0s

monitor.dbck.rebuildHistoryStatistics 27 Apr 08:24 succeeded=178 failed=0 elapsed=3s

monitor.dbck.removeOrphans 27 Apr 08:59 succeeded=179 failed=0 elapsed=1s

monitor.dbck.verifyDbcache 27 Apr 08:25 succeeded=178 failed=0 elapsed=3s

monitor.discover.network 11 Apr 12:49 none

objectMap.verifier.current 27 Apr 08:25 57

perf.threshviolation.update 27 Apr 09:07

postTransformLogicSuccess 11 Apr 12:47 1

postupgrade.app_policies.create_samples 11 Apr 12:47 no

postupgrade.dp_policies.create_samples 11 Apr 12:47 no

postupgrade.prov_policies.create_samples 11 Apr 12:47 no

postupgrade.storage_services.create_samples 11 Apr 12:47 no

Global Options

Option Name Option Value

autosupportMonInterval 00:02:00

agentMonInterval Off

cfMonInterval 00:05:00

clusterMonInterval 00:15:00

cpuMonInterval 00:05:00

dfMonInterval 00:30:00

snapshotMonInterval 00:30:00

diskMonInterval 04:00:00

envMonInterval 00:05:00

fcMonInterval 00:05:00

fsMonInterval 00:15:00

ifMonInterval 00:15:00

licenseMonInterval 04:00:00

lunMonInterval 00:30:00

opsMonInterval 00:10:00

pingMonInterval 00:01:00

qtreeMonInterval 08:00:00

SANHostMonInterval 00:05:00

snapmirrorMonInterval 00:30:00

statusMonInterval 00:10:00

sysInfoMonInterval 01:00:00

userQuotaMonInterval 1:00:00:00

snapvaultMonInterval Off

srmMonInterval Off

shareMonInterval 01:00:00

ccMonInterval 04:00:00

vFilerMonInterval 01:00:00

vserverMonInterval 01:00:00

clusterMonInterval 00:15:00

backupDirMonInterval 08:00:00

ndmpMonInterval 01:00:00

dsConformanceMonInterval 01:00:00

dsDRMonInterval 00:30:00

dsProtectionMonInterval 00:30:00

respoolSpaceMonInterval 02:00:00

hostRBACMonInterval 24:00:00

dpReaperInterval 00:30:00

isHsAliveMonInterval 00:01:00

dsUsageMetricMonInterval 2:00:00

authUsePam no

autosupportAdminContact FILESERVICES-NAS@f-i.de

autosupportContent complete

autosupportDestinationURL support.netapp.com/asupprod/post/1.0/postAsup

autosupportDestinationEmail autosupport@netapp.com

autosupportEnabled Yes

autosupportProtocol smtp

autosupportRetryCount 4

autosupportRetryDelay 00:15:00

autosupportIncludePerf Yes

autosupportIncludeProv Yes

autosupportIncludeVirtual Yes

autosupportIncludeAllDiagInfo No

preferredIPAddressType IPv4

dpReBaselineMode Confirm

dpReaperCleanupMode Orphans

currentEventsCacheSize

discoverInterval 00:05:00

discoverTimeout 00:00:20

discoverEnabled Yes

discoverNetworks No

networkDiscoveryLimit 15

discoverAgents No

discoverClusters Yes

discoverVfilers Yes

discoverSAN No

discoverHosts No

discoverHostInitEnabled Yes

SMTPServerName d100sskext05.d100.intern

SMTPServerBackup

SMTPServerPort 25

monMinFreePercent 10.0

monMinFreeBytes 10000.0

localHostName dfm-h1.v100.intern

cpuTooBusyThreshold 110

cpuBusyThresholdInterval 01:00:00

qtreeFullThreshold 1000

qtreeFullThresholdInterval 00:00:00

qtreeNearlyFullThreshold 1000

qtreeGrowthEventMinChangePct 100

aggrFullThreshold 95

aggrFullThresholdInterval 00:00:00

aggrNearlyFullThreshold 90

aggrOvercommittedThreshold 1000

aggrNearlyOvercommittedThreshold 1000

aggrSnapshotFullThreshold 90

aggrSnapshotNearlyFullThreshold 80

volFullThreshold 99

volFullThresholdInterval 00:00:00

volNearlyFullThreshold 1000

volGrowthEventMinChangePct 100

volNearlyOvercommittedThreshold 1000

volOvercommittedThreshold 1000

nodesRemainingWarningThreshold 1

provMgrNodesRemainingWarningThreshold 1

protMgrNodesRemainingWarningThreshold 1

volSnapshotFullThreshold 110

volNoFirstSnapThreshold 90

volNearlyNoFirstSnapThreshold 80

volReserveDepletedThreshold 90

volReserveNearlyDepletedThreshold 80

volSnapshotCountThreshold 250

volSnapshotTooOldThreshold 168:00:00:00

licenseExpireWarningThreshold 5

guiRefreshInterval off

growthRateSensitivity 2

monVerbose 30

monVerboseModule engine|cpu

pingMonTimeout 180

pingMonRetryDelay 60

hostPingMethod all

profileTTL 45:00:00:00

credCacheTTL 00:20:00

ldapEnabled No

ldapBindDN

ldapBindPass

ldapBaseDN

ldapVersion 3

ldapUID UID

ldapGID

ldapUGID CN

ldapMember uniqueMember

filerConfigSaveLocalChanges yes

chargebackIncrement Monthly

chargebackRate 0.42

currencyFormat $ #,###.##

chargebackDayOfMonth 1

userEmailDefaultDomain

userEnableAlerts No

userFullThreshold 1000

userNearlyFullThreshold 1000

httpEnabled No

httpPort 0

httpsEnabled Yes

httpsPort 8443

webUIPort 8123

hostLoginProtocol ssh

hostAdminTransport https

hostAdminPort 80

hostEnclosureDiscoveryEvents Disabled

agentHostTransport https

agentHostPort 4092

agentHostLogin guest

agentHostGuestPassword

agentHostAdminPassword

agentHostCIFSAccount

agentHostCIFSPassword

alertFrom DFM

hbaportTooBusyThreshold 90

SANHostMonSnapshotLUNs Yes

srmLargestFiles 20

srmLeastRecentlyModifiedFiles 20

srmLeastRecentlyAccessedFiles 20

srmRecentlyModifiedFiles 20

srmLargestDirs 100

perfAdvisorEnabled Yes

perfAdvisorTransport httpsOk

perfAdvisorPollInterval 00:05:00

perfAdvisorMaxMonitorThreads 32

perfArchiveDir /om-perfdata/

perfAdvisorShowDiagCounters Yes

perfAdvisorShowAllViews No

serverHTTPSEnabled Yes

serverHTTPEnabled No

serverHTTPPort 8088

serverHTTPSPort 8488

serverCertAuthEnabled Yes

hsNotificationsPurgingInterval 86400

hsNotificationsMaxCount 100000

alarmScriptRunAs

ownerEmailFieldName ownerEmail

snmpTrapListenerPort 162

snmpTrapRcvdMaxPerWindow 2500

snmpTrapRcvdWindowSize 00:01:00

eventsPurgeInterval 28:00:00:00

snmpTrapListenerEnabled No

scriptPath

monSNMPTimeout 15

monSNMPRetries 20

groupTreeShowStatus Enabled

pluginsDir

scriptDir

databaseBackupDir /opt/NTAPdfm/data

databaseBackupDbengWaitTime 600

backupRetentionCount 2

auditLogForever Yes

auditLogEnabled Yes

serverAPILogExclude host-service-discover|dfm-about

respoolFullThreshold 90

respoolNearlyFullThreshold 80

useHostsEquiv No

apacheUserName

pmQSMBackupPreferred Yes

dpDynamicSecondarySizing No

dpMaxFanInRatio 1

vFilerRootVolumeSizeMb 50

ndmpDataUseAllInterfaces 0

favoriteMaxReports 25

recentMaxReports 25

defReportLinesPerPage 30

maxReportLinesPerPage 1000

reportsArchiveDir

snapshotDiscoveryEventsEnabled No

dataExportDir

dfmDataExportEnabled No

perfDataExportEnabled No

perfAdvThreshViolationMonInterval 00:00:15:00

enableFQDNInFilerViewLinks Yes

volNearlyOverDeduplicatedThreshold 1000

volOverDeduplicatedThreshold 1000

aggrNearlyOverDeduplicatedThreshold 1000

aggrOverDeduplicatedThreshold 1000

dpRestoreTransfersPerHost 8

processHostPrimaryAddress warn

processOSSVPrimaryAddress warn

pmUseSDUCompatibleSnapshotNames No

rshBinary

perfExportDir

perfSampleRate1 00:00:1:00

perfSampleRate2 00:00:5:00

perfSampleRate3 00:00:15:00

perfSampleRate4 00:00:30:00

perfMaxObjectInstancesInBarChart 20

autoClientStatEnabled No

clientStatCpuThreshold 80

clientStatTotalOpsRate 30

clientStatThresholdPeriod 300

clientStatNfsLatency 30

clientStatCifsLatency 30

clientStatMinTotalOpsRate 500

dpSnapNameFormat %T_%R_%L_%H_%N_%A

dpSnapNameOption global-format

dpSnapNameScriptPath

dpSnapNameScriptRunAs

dpPriVolNameOption global-format

dpSecVolNameOption global-format

dpPriVolNameFormat %L

dpSecVolNameFormat %L_%C_%S_%V

dpPriVolNameScriptPath

dpPriVolNameScriptRunAs

hostClockSkewedThreshold 60

hostClockNearlySkewedThreshold 30

dpSecVolNameScriptPath

dpSecVolNameScriptRunAs

dpSecQtreeNameFormat %Q

dsUsageMetricCommentFields

dsUsageMetricIoInterval 1:00:00:00

dsUsageMetricSpaceInterval 1:00:00:00

reportDesignPath /opt/NTAPdfm/reports/

keystorePassphraseFile

snapmirrorLagErrorThreshold 172800

snapmirrorLagWarningThreshold 129600

webUIMinHeapSizeMB 256

webUIMaxHeapSizeMB 1024

webUIMinPermGenSizeMB 128

webUIMaxPermGenSizeMB 512

Object Counts

Object Type Count

Administrator 27

Aggregate 62

App Policy 2

Array LUNs 0

Backup jobs started within last week 0

Clusters 0

Configuration 4

Datasets containing generic app objects 0

Datasets containing generic app objects and with storage service attached 0

Datasets with application policy attached 0

Datasets with storage service attached 0

Disks 5420

DP Managed OSSV Rels 0

DP Managed SnapMirror Rels 0

DP Managed SnapVault Rels 0

DP Policy 36

DP Schedule 18

DP Throttle 2

FCP Target 26

Host 324

interface 1027

Mgmt Station 1

Mirror 272

Mount jobs started within last week 0

Network 5

OSSV Hosts 0

OSSV Rels 0

Primary Storage Systems 0

Prov Policy 16

Qtree 59394

Qtrees in DP Managed QtreeSnapMirror Rel 0

Qtrees in DP Managed SnapVault Rel 0

Qtrees in QtreeSnapMirror Rel 0

Qtrees in SnapVault Rel 0

QuotaUser 112774

Resource Group 18

Resource Pool 2

Restore jobs started within last week 0

Role 35

Secondary Storage Systems 0

SnapMirror Rels 292

SnapVault Rels 0

Storage Service 16

Unmount jobs started within last week 0

UserQuota 116089

vFilers 282

Virtual Servers 0

Volume 1132

Volumes in DP Managed VolumeSnapMirror Rel 0

Volumes in VolumeSnapMirror Rel 538

Zapi Hosts 42

Monitoring Timestamps

Timestamp Name Status Interval Default Last Updated Status Error if older than ...

ccTimestamp Normal 4 hours 4 hours 27 Apr 05:10

cfTimestamp Normal 5 minutes 5 minutes 27 Apr 09:10 Normal 27 Apr 09:05

clusterTimestamp Normal 15 minutes 15 minutes 27 Apr 08:55

cpuTimestamp Normal 5 minutes 5 minutes 27 Apr 09:10 Normal 27 Apr 09:05

dfTimestamp Normal 30 minutes 30 minutes 27 Apr 09:10 Normal 27 Apr 08:40

diskTimestamp Normal 4 hours 4 hours 27 Apr 09:10 Normal 27 Apr 05:10

envTimestamp Normal 5 minutes 5 minutes 27 Apr 09:10 Normal 27 Apr 09:05

fcTimestamp Normal 5 minutes 5 minutes 27 Apr 09:10 Normal 27 Apr 09:05

fsTimestamp Normal 15 minutes 15 minutes 27 Apr 09:09 Normal 27 Apr 08:55

hostPingTimestamp Normal 1 minute 1 minute 27 Apr 09:10 Normal 27 Apr 09:09

ifTimestamp Normal 15 minutes 15 minutes 27 Apr 09:10 Normal 27 Apr 08:55

licenseTimestamp Normal 4 hours 4 hours 27 Apr 09:04 Normal 27 Apr 05:10

lunTimestamp Normal 30 minutes 30 minutes 27 Apr 09:10 Normal 27 Apr 08:40

opsTimestamp Normal 10 minutes 10 minutes 27 Apr 09:10 Normal 27 Apr 09:00

qtreeTimestamp Normal 8 hours 8 hours 11 Apr 13:02 Error 27 Apr 01:10

rbacTimestamp Normal 1 day 1 day 27 Apr 09:10 Normal 26 Apr 09:10

userQuotaTimestamp Normal 1 day 1 day 27 Apr 08:15 Normal 26 Apr 09:10

sanhostTimestamp Normal 5 minutes 5 minutes 27 Apr 09:05

snapmirrorTimestamp Normal 30 minutes 30 minutes 27 Apr 09:09 Normal 27 Apr 08:40

snapshotTimestamp Normal 30 minutes 30 minutes 27 Apr 09:10 Normal 27 Apr 08:40

statusTimestamp Normal 10 minutes 10 minutes 27 Apr 09:10 Normal 27 Apr 09:00

sysInfoTimestamp Normal 1 hour 1 hour 27 Apr 09:03 Normal 27 Apr 08:10

svTimestamp Error Off 30 minutes Not updated ( snapvaultMonInterval is set to Off )

svMonTimestamp Normal 8 hours 8 hours 27 Apr 01:10

xmlQtreeTimestamp Normal 8 hours 8 hours 27 Apr 08:15 Normal 27 Apr 01:10

vFilerTimestamp Normal 1 hour 1 hour 27 Apr 09:09 Normal 27 Apr 08:10

vserverTimestamp Normal 1 hour 1 hour 27 Apr 08:10

Database

monitordb.db 728 MB

dbFileVersion 10

ConnCount 61 connections

MaxCacheSize 33055224 KBytes

CurrentCacheSize 719264 KBytes

PeakCacheSize 719264 KBytes

PageSize 8192 Bytes

Cluster Details

Object Type Count

Clusters 0

Controllers 0

Virtual Systems 0

Volumes (All) 0

Striped Volumes 0

Aggregates (All) 0

Striped Aggregates 0

Logical Interfaces (All) 0

Interface Groups 0

Ports (All) 0

Physical Disk Details

Total Capacity 0 bytes

Total Used 0 bytes

Logs

discovery 1.44 KB 11 Apr 12:49

dfmmonitor 2.36 MB 27 Apr 09:10

dfmeventd 17.6 KB 27 Apr 00:58

dfmserver 20.0 KB 26 Apr 23:10

dfmscheduler 354 bytes 17 Apr 15:40

dfmwatchdog 310 KB 27 Apr 09:06

dfm 43.1 KB 20 Apr 13:32

sybase 424 KB 27 Apr 09:08

pingmon 6.54 KB 20 Apr 23:35

audit 567 KB 27 Apr 08:46

vfmon 550 KB 27 Apr 09:05

Services

sql Normal Started

webui Normal Started

http Normal Started

eventd Normal Started

monitor Normal Started

scheduler Normal Started

server Normal Started

watchdog Normal Started

Time Since Confirmed Alive

Eventd 5 seconds

Monitor 19 seconds

Scheduler 24 seconds

Server 11 seconds

Watchdog 0 seconds

Events

There are no events that have warning severity or worse.

DP Job Information

Job State Count

Jobs Running 0

Jobs Completed Total 0

Jobs Aborted Total 0

Jobs Aborting Total 0

Jobs Completed Today 0

Jobs Aborted Today 0

Jobs Aborting Today 0

Dataset Protection Status

Protection State Count

Protected 0

Unprotected 0

Event Counts

Table Count

Events 578872

Current Events 578282

Abnormal Events 2523

Event Type Counts

Event Type Count

userquota.files.soft.limit 117081

userquota.kbytes.soft.limit 117081

userquota.kbytes 117081

userquota.files 117081

qtree.files 42426

qtree.kbytes 42426

df.snapshot.kbytes 1698

df.avail 1577

df.kbytes 1568

snap-status 1564

Version 5.0 (5.0)

adaikkap · ‎2012-04-29

Hi Stefan,

As I suspected, you have a lot of userquota's hundred 1000 +. But you are already on OC 5.0 64 bit platform with a really beefy server with 64GB RAM. I suggest you to open a case with NetApp Support to look into this issue.

Regards

adai

anton_oks · ‎2012-07-18

Hi there.

We see exactly the same damned problem. I've an open case, but so far no usable answers from the support guys at NetApp

Any feedback from this/your support case?
Anybody out there who has any ideas?

Any help is highly welcome.

Thanks & regards

Anton

skellner · ‎2012-08-07

Hi Anton,

no, sorry. No news on this.

Regards

Stefan