Re: dfm interface slow

BrendonHiggins · ‎2009-09-29

I have added 4 new filers to my estate and ops manager has become very slow. There are only 15 filers in total and the DFM database monitordb.db is now 370 Mb in size on a high spec, single task server with 4Gb of ram. This server is only used for Ops and performance manager and both apps are running standard monitoring and are not killing the box poling every second, etc.

Any ideas why the interface just sticks at "loading"? Please see image.

We have also move some volumes between aggregates by using snapmirror and rename.

Thanks

Bren

RichardSopp · ‎2009-09-29

Which version of DFM are you using?

BrendonHiggins · ‎2009-09-30

3.7 - Sorry should have put it in the post.

dhank · ‎2009-09-30

Can you provide the NMC logs for further debugging ? If you are able to reproduce this issue, set the log level as ALL (default is ERROR) in the log4j.properties in the NMC home directory and send them.

BrendonHiggins · ‎2009-11-03

Sorry I forgot about this issue but have started using DFM again and the problems are worse than ever.

The E:\Program Files\NetApp\DataFabric\DFM\data\monitordb.db is 395 Mb yet the dbsrv9.exe process only shows 353 Mb in windows task manager.

The windows service "DFM Sybase ASA" - DataFabric Manager database server starts the database with the switch below.

"E:\Program Files\NetApp\DataFabric\Sybase\ASA\win32\dbsrv9.exe" -hvDFMSybase

Has anyone played with the cache options to speed up the databse?

http://www.ianywhere.com/developer/product_manuals/sqlanywhere/0902/en/html/dbdaen9/00000146.htm

Thanks

Bren

pascalduk · ‎2009-11-03

Maybe you already checked it, but is a virusscanner perhaps active?

Running on linux here: 28 filers, 1.8 GB database and 125 GB of PA data. All running smooth.

BrendonHiggins · ‎2009-11-04

AV does not scan the database folder.

Thanks for the idea however.

Bren

BrendonHiggins · ‎2010-02-09

Going to deploy v4 of Operations manager. Many changes to improve performance in this release.

Bren

martin_fisher · ‎2010-11-26

Just wndering how anyone got on with v4 of DFM and if it has improved performance.

was v4 installed on a Linux OS rather than windows.

Im having performance issues with W2K3 server running DFM v3.x monitoring approx 10 appliances, mostly it appears DB/SQL related.

Thanks.

Martin

pbhanu · ‎2010-11-26

Hi Martin,

Can you paste the output of 'dfm about' in the DFM version that you are currently using.

Looks like in current version, there is not enough free space in the installation directory.

A sample output :

[root@lnx186-143 script]# dfm about
CPU Count                        1
System Memory                    2026 MB (load excluding cached memory: 69%)
Installation Directory           /opt/NTAPdfm
                                 2.17 GB free (17.1%)

Regards,

Bhanu

martin_fisher · ‎2010-11-29

Hi Bhanu

As requested:

About DataFabric® Manager server
Version	3.6.0.4808 (3.6)
Serial Number

Administrator Name	none
Host Name
Host IP Address
Host Full Name
Node Limit	10
Operating System	Microsoft Windows 2003 Service Pack 2 (Build 3790)
CPU Count	2
System Memory	3839 MB (load: 26%)
Installation Directory	E:/Program Files/DFM 8.07 GB free (20.2%)
Licensed Features	Operations Manager: installed Business Continuance Option: installed File SRM Option: installed Protection Manager: your dataprote

adaikkap · ‎2010-11-26

DFM version 3.8.1 and version 4.0 has lot of scalability and performance improvements.

Can you paste a dfm diag output of yours to see how big is your environment ?

Atleast the objects count part of the dfm diag output.

Regards

adai

martin_fisher · ‎2010-11-29

Hi Adai

Bit of a big post, but most details

autosupport.installTime	05 Oct 2007	2007-10-05 11:49:41.352000
autosupport.lastStatusSendTime	05 Oct 2007	1290697983
autoThrottleMultiplier	29 Nov 15:25	100
generationId	05 Oct 2007	641
generationId.alonso	05 Oct 2007	88
generationId.freya-reloaded	05 Oct 2007	104
generationId.gayatri	05 Oct 2007	21
generationId.janneau	05 Oct 2007	77
generationId.kahlua	05 Oct 2007	16
history.rollup.aggrHistoryMonth	29 Nov 15:35	Timeslot 29 Nov 16:00: 28 records in 0 seconds (100000 rps)
history.rollup.aggrHistoryQuarter	29 Nov 15:35	Timeslot 30 Nov 00:00: 28 records in 0 seconds (100000 rps)
history.rollup.aggrHistoryWeek	29 Nov 15:35	Timeslot 29 Nov 16:00: 19 records in 0 seconds (100000 rps)
history.rollup.aggrHistoryYear	29 Nov 15:35	Timeslot 01 Dec 00:00: 28 records in 0 seconds (100000 rps)
history.rollup.cpuHistoryMonth	29 Nov 15:29	Timeslot 29 Nov 16:00: 14 records in 0 seconds (100000 rps)
history.rollup.cpuHistoryQuarter	29 Nov 15:29	Timeslot 30 Nov 00:00: 14 records in 0 seconds (100000 rps)
history.rollup.cpuHistoryWeek	29 Nov 15:29	Timeslot 29 Nov 16:00: 14 records in 0 seconds (100000 rps)
history.rollup.cpuHistoryYear	29 Nov 15:29	Timeslot 01 Dec 00:00: 14 records in 0 seconds (100000 rps)
history.rollup.dfHistoryMonth	29 Nov 15:35	Timeslot 29 Nov 16:00: 968 records in 0.109 seconds (8881 rps)
history.rollup.dfHistoryQuarter	29 Nov 15:35	Timeslot 30 Nov 00:00: 986 records in 0.078 seconds (12641 rps)
history.rollup.dfHistoryWeek	29 Nov 15:35	Timeslot 29 Nov 16:00: 706 records in 0.078 seconds (9051 rps)
history.rollup.dfHistoryYear	29 Nov 15:35	Timeslot 01 Dec 00:00: 1014 records in 0.172 seconds (5895 rps)
history.rollup.fcswHistoryMonth	29 Nov 15:29	Timeslot 29 Nov 16:00: 0 records in 0 seconds (100000 rps)
history.rollup.fcswHistoryQuarter	29 Nov 15:29	Timeslot 30 Nov 00:00: 0 records in 0 seconds (100000 rps)
history.rollup.fcswHistoryWeek	29 Nov 15:29	Timeslot 29 Nov 16:00: 0 records in 0 seconds (100000 rps)
history.rollup.fcswHistoryYear	29 Nov 15:29	Timeslot 01 Dec 00:00: 0 records in 0 seconds (100000 rps)
history.rollup.historyMonth	29 Nov 15:23	Timeslot 29 Nov 16:00: 402 records in 0.047 seconds (8553 rps)
history.rollup.historyQuarter	29 Nov 15:23	Timeslot 30 Nov 00:00: 405 records in 0.063 seconds (6429 rps)
history.rollup.historyWeek	29 Nov 15:23	Timeslot 29 Nov 16:00: 285 records in 0 seconds (100000 rps)
history.rollup.historyYear	29 Nov 15:23	Timeslot 01 Dec 00:00: 408 records in 0.078 seconds (5231 rps)
history.rollup.ifHistoryMonth	29 Nov 15:29	Timeslot 29 Nov 16:00: 84 records in 0.031 seconds (2710 rps)
history.rollup.ifHistoryQuarter	29 Nov 15:29	Timeslot 30 Nov 00:00: 84 records in 0 seconds (100000 rps)
history.rollup.ifHistoryWeek	29 Nov 15:29	Timeslot 29 Nov 16:00: 84 records in 0 seconds (100000 rps)
history.rollup.ifHistoryYear	29 Nov 15:29	Timeslot 01 Dec 00:00: 84 records in 0 seconds (100000 rps)
history.rollup.initiatorHistoryMonth	29 Nov 15:29	Timeslot 29 Nov 16:00: 0 records in 0 seconds (100000 rps)
history.rollup.initiatorHistoryQuarter	29 Nov 15:29	Timeslot 30 Nov 00:00: 0 records in 0 seconds (100000 rps)
history.rollup.initiatorHistoryWeek	29 Nov 15:29	Timeslot 29 Nov 16:00: 0 records in 0 seconds (100000 rps)
history.rollup.initiatorHistoryYear	29 Nov 15:29	Timeslot 01 Dec 00:00: 0 records in 0 seconds (100000 rps)
history.rollup.lunHistoryMonth	29 Nov 15:35	Timeslot 29 Nov 16:00: 343 records in 0 seconds (100000 rps)
history.rollup.lunHistoryQuarter	29 Nov 15:35	Timeslot 30 Nov 00:00: 347 records in 0.031 seconds (11193 rps)
history.rollup.lunHistoryWeek	29 Nov 15:35	Timeslot 29 Nov 16:00: 343 records in 0.015 seconds (22867 rps)
history.rollup.lunHistoryYear	29 Nov 15:35	Timeslot 01 Dec 00:00: 349 records in 0 seconds (100000 rps)
history.rollup.qtreeHistoryMonth	29 Nov 15:05	Timeslot 29 Nov 16:00: 0 records in 0 seconds (100000 rps)
history.rollup.qtreeHistoryQuarter	29 Nov 15:05	Timeslot 30 Nov 00:00: 0 records in 0 seconds (100000 rps)
history.rollup.qtreeHistoryWeek	29 Nov 15:05	Timeslot 29 Nov 16:00: 0 records in 0 seconds (100000 rps)
history.rollup.qtreeHistoryYear	29 Nov 15:05	Timeslot 01 Dec 00:00: 0 records in 0 seconds (100000 rps)
history.rollup.quotaHistoryMonth	29 Nov 15:05	Timeslot 29 Nov 16:00: 673 records in 0.187 seconds (3599 rps)
history.rollup.quotaHistoryQuarter	29 Nov 15:05	Timeslot 30 Nov 00:00: 673 records in 0.156 seconds (4314 rps)
history.rollup.quotaHistoryWeek	29 Nov 15:05	Timeslot 29 Nov 16:00: 0 records in 0 seconds (100000 rps)
history.rollup.quotaHistoryYear	29 Nov 15:05	Timeslot 01 Dec 00:00: 1522 records in 0.047 seconds (32383 rps)
history.rollup.snapmirrorHistoryMonth	29 Nov 15:35	Timeslot 29 Nov 16:00: 109 records in 0 seconds (100000 rps)
history.rollup.snapmirrorHistoryQuarter	29 Nov 15:35	Timeslot 30 Nov 00:00: 109 records in 0 seconds (100000 rps)
history.rollup.snapmirrorHistoryWeek	29 Nov 15:35	Timeslot 29 Nov 16:00: 109 records in 0 seconds (100000 rps)
history.rollup.snapmirrorHistoryYear	29 Nov 15:35	Timeslot 01 Dec 00:00: 109 records in 0.031 seconds (3516 rps)
history.rollup.vfilerHistoryMonth	29 Nov 15:11	Timeslot 29 Nov 16:00: 5 records in 0 seconds (100000 rps)
history.rollup.vfilerHistoryQuarter	29 Nov 15:11	Timeslot 30 Nov 00:00: 5 records in 0 seconds (100000 rps)
history.rollup.vfilerHistoryWeek	29 Nov 15:11	Timeslot 29 Nov 16:00: 5 records in 0.047 seconds (106 rps)
history.rollup.vfilerHistoryYear	29 Nov 15:11	Timeslot 01 Dec 00:00: 5 records in 0 seconds (100000 rps)
history.rollup.volHistoryMonth	29 Nov 15:35	Timeslot 29 Nov 16:00: 332 records in 0.015 seconds (22133 rps)
history.rollup.volHistoryQuarter	29 Nov 15:35	Timeslot 30 Nov 00:00: 341 records in 0 seconds (100000 rps)
history.rollup.volHistoryWeek	29 Nov 15:35	Timeslot 29 Nov 16:00: 219 records in 0.031 seconds (7065 rps)
history.rollup.volHistoryYear	29 Nov 15:35	Timeslot 01 Dec 00:00: 352 records in 0.016 seconds (22000 rps)
history.stats.aggrHistory	29 Nov 06:16	0.265 seconds
history.stats.cpuHistory	29 Nov 03:13	0.333 seconds
history.stats.dfHistory	29 Nov 04:15	4.7 seconds
history.stats.fcswHistory	29 Nov 01:10	0 seconds
history.stats.history	28 Nov 19:02	1.74 seconds
history.stats.ifHistory	29 Nov 00:09	0.688 seconds
history.stats.initiatorHistory	29 Nov 02:12	0 seconds
history.stats.lunHistory	28 Nov 23:07	1.66 seconds
history.stats.qtreeHistory	28 Nov 20:03	0 seconds
history.stats.quotaHistory	28 Nov 21:05	11.3 seconds
history.stats.snapmirrorHistory	28 Nov 22:06	0.844 seconds
history.stats.vfilerHistory	29 Nov 07:18	0.156 seconds
history.stats.volHistory	29 Nov 05:16	2.62 seconds
monitor.dbck.checkObjFlags	29 Nov 12:43	succeeded=4 failed=0 elapsed=0s
monitor.dbck.consolidateHistory	29 Nov 15:46	succeeded=205 failed=0 elapsed=0s
monitor.dbck.deleteDupQuotaUsers	29 Nov 15:37	succeeded=76 failed=0 elapsed=0s
monitor.dbck.mgmtStationName	29 Nov 15:31	succeeded=21 failed=0 elapsed=0s
monitor.dbck.preCacheGroups	29 Nov 15:45	succeeded=77 failed=0 elapsed=0s
monitor.dbck.rebuildHistoryStatistics	29 Nov 15:30	succeeded=21 failed=0 elapsed=0s
monitor.dbck.removeOrphans	29 Nov 15:09	succeeded=21 failed=0 elapsed=83s
monitor.dbck.verifyDbcache	29 Nov 15:33	succeeded=21 failed=0 elapsed=0s
monitor.discover.network	26 Oct 2007	none
objectMap.verifier.current	29 Nov 15:33	71
postTransformLogicSuccess	05 Oct 2007	1
postupgrade.dp_policies.create_samples	05 Oct 2007	no

Global Options

Option Name	Option Value
agentHostCIFSAccount
agentHostLogin	guest
agentHostPort
agentHostTransport	http
agentMonInterval	00:02:00
aggrFullThreshold	85
aggrFullThresholdInterval	00:00:00
aggrNearlyFullThreshold	75
aggrNearlyOvercommittedThreshold	190
aggrOvercommittedThreshold	200
aggrSnapshotFullThreshold	90
aggrSnapshotNearlyFullThreshold	80
alertFrom	OperationsManager
apacheUserName
auditLogEnabled	Yes
auditLogForever	No
authUsePam	no
autosupportAdminContact
autosupportContent	complete
autosupportDestinationEmail	autosupport@netapp.com
autosupportDestinationURL	support.netapp.com/asupprod/post/1.0/postAsup
autosupportEnabled	Yes
autosupportMonInterval	00:02:00
autosupportProtocol	https
autosupportRetryCount	3
autosupportRetryDelay	00:15:00
backupDirMonInterval	08:00:00
backupRetentionCount	1
cacheAdminPort	3132
cacheAdminUser	admin
cacheClearLocalConfig	no
cacheMonInterval	00:05:00
cacheSecureAdminPort	3443
cacheUseDfmHostname	no
ccMonInterval	04:00:00
cfMonInterval	00:05:00
chargebackDayOfMonth	1
chargebackIncrement	Daily
chargebackRate	8.54
cmsAclEnable	No
cmsDir
cpuBusyThresholdInterval	00:15:00
cpuMonInterval	00:05:00
cpuTooBusyThreshold	95
credCacheTTL	00:20:00
currencyFormat	£ #,###.##
currentEventsCacheSize
databaseBackupDbengWaitTime	600
databaseBackupDir	E:\Program Files\DFM\data
dfMonInterval	00:30:00
discoverAgents	No
discoverAppInitEnabled	Yes
discoverAutoSnmp	No
discoverCacheAdminPort	3132
discoverEnabled	Yes
discoverHosts	No
discoverHttpEnabled	No
discoverInterval	00:15:00
discoverNetworks	No
discoverSAN	Yes
discoverTimeout	00:00:05
discoverVfilers	Yes
diskMonInterval	04:00:00
dsConformanceMonInterval	01:00:00
dsProtectionMonInterval	00:15:00
envMonInterval	00:05:00
eventsPurgeInterval	00:00:00
favoriteMaxReports	25
fcMonInterval	00:05:00
filerConfigSaveLocalChanges	yes
fsMonInterval	00:15:00
groupTreeShowStatus	Enabled
growthRateSensitivity	2
guiRefreshInterval	00:02:00
hbaportTooBusyThreshold	90
hostAdminTransport	http
hostEnclosureDiscoveryEvents	DisabledForCluster
hostLoginProtocol	rsh
hostPingMethod	echo_snmp
hostRBACMonInterval	1:00:00:00
httpEnabled	Yes
httpPort	8080
httpsEnabled	No
httpsPort	8081
ifMonInterval	00:01:00
ldapBaseDN
ldapBindDN
ldapBindPass
ldapEnabled	No
ldapGid
ldapMember	uniqueMember
ldapUgid	CN
ldapUid	UID
ldapVersion	3
licenseExpireWarningThreshold	5
licenseMonInterval	04:00:00
localHostName
lunMonInterval	00:30:00
monMinFreeBytes	10000.0
monMinFreePercent	10.0
monSNMPRetries	4
monSNMPTimeout	5
ncfileDir
ndmpDataUseAllInterfaces	0
ndmpMonInterval	00:30:00
networkDiscoveryLimit	15
nodesRemainingWarningThreshold	1
opsMonInterval	00:10:00
ownerEmailFieldName	ownerEmail
perfAdvisorEnabled	Yes
perfAdvisorMaxMonitorThreads	32
perfAdvisorPollInterval	00:05:00
perfAdvisorShowAllViews	No
perfAdvisorShowDiagCounters	No
perfAdvisorTransport	httpOnly
perfArchiveDir	E:/Program Files/DFM/perfdata
pingMonInterval	00:01:00
pingMonRetryDelay	3
pingMonTimeout	3
pluginsDir
pmQSMBackupPreferred	Yes
profileTTL	45:00:00:00
qtreeFullThreshold	90
qtreeFullThresholdInterval	00:00:00
qtreeGrowthEventMinChangePct	1
qtreeMonInterval	08:00:00
qtreeNearlyFullThreshold	80
recentMaxReports	25
reportsArchiveDir	F:\Archive
respoolFullThreshold	90
respoolNearlyFullThreshold	80
respoolSpaceMonInterval	1:00:00
SANHostMonInterval	00:05:00
SANHostMonSnapshotLUNs	Yes
sanManagerHostname
sanManagerLogin
scriptDir
scriptPath
serverHTTPEnabled	Yes
serverHTTPPort	8088
serverHTTPSEnabled	Yes
serverHTTPSPort	8488
shareMonInterval	01:00:00
SMTPServerBackup
SMTPServerName
SMTPServerPort	25
snapmirrorMonInterval	00:30:00
snapshotMonInterval	00:30:00
snapvaultMonInterval	00:30:00
snmpTrapListenerEnabled	Yes
snmpTrapListenerPort	162
snmpTrapRcvdMaxPerWindow	250
snmpTrapRcvdWindowSize	00:05:00
softwareDir
srmLargestDirs	100
srmLargestFiles	20
srmLeastRecentlyAccessedFiles	20
srmLeastRecentlyModifiedFiles	20
srmMonInterval	00:10:00
srmRecentlyModifiedFiles	20
statusMonInterval	00:10:00
sysInfoMonInterval	01:00:00
useHostsEquiv	No
userEmailDefaultDomain
userEnableAlerts	No
userFullThreshold	90
userNearlyFullThreshold	80
userQuotaMonInterval	1:00:00:00
vFilerMonInterval	01:00:00
volFullThreshold	90
volFullThresholdInterval	00:00:00
volGrowthEventMinChangePct	20
volNearlyFullThreshold	85
volNearlyNoFirstSnapThreshold	80
volNearlyOvercommittedThreshold	95
volNoFirstSnapThreshold	90
volOvercommittedThreshold	100
volReserveDepletedThreshold	90
volReserveNearlyDepletedThreshold	80
volSnapshotCountThreshold	250
volSnapshotFullThreshold	90
volSnapshotTooOldThreshold	42:00:00:00

Object Counts

Object Type	Count
Administrator	6
Aggregate	56
DP Policy	20
DP Schedule	18
DP Throttle	2
FCP Target	12
Host	15
Initiator Group	93
Interface	108
Lun Path	450
Mgmt Station	1
Mirror	109
Network	7
OSSV Hosts	0
Primary Filers	0
Qtree	2
QuotaUser	2614
report schedule	11
Resource Group	44
Role	22
schedule	6
Secondary Filers	0
SnapMirror Rels	194
SnapVault Rels	0
UserQuota	1522
vFilers	5
Volume	894
Zapi Hosts	6

Monitoring Timestamps

Timestamp Name	Interval	Last Updated
cacheTimestamp	5 minutes		29 Nov 15:42
ccTimestamp	4 hours		29 Nov 11:47
cfTimestamp	5 minutes	29 Nov 15:46	29 Nov 15:42
cpuTimestamp	5 minutes	29 Nov 15:47	29 Nov 15:42
dfTimestamp	30 minutes	29 Nov 15:45	29 Nov 15:17
diskTimestamp	4 hours	29 Nov 15:42	29 Nov 11:47
envTimestamp	5 minutes	29 Nov 15:47	29 Nov 15:42
fcTimestamp	5 minutes	29 Nov 15:46	29 Nov 15:42
fsTimestamp	15 minutes	29 Nov 15:46	29 Nov 15:32
hostPingTimestamp	1 minute	29 Nov 15:47	29 Nov 15:46
ifTimestamp	1 minute	29 Nov 15:47	29 Nov 15:46
licenseTimestamp	4 hours	29 Nov 15:37	29 Nov 11:47
lunTimestamp	30 minutes	29 Nov 15:30	29 Nov 15:17
opsTimestamp	10 minutes	29 Nov 15:46	29 Nov 15:37
qtreeTimestamp	8 hours	29 Nov 11:38	29 Nov 07:47
rbacTimestamp	1 day	29 Nov 11:36	28 Nov 15:47
userQuotaTimestamp	1 day	29 Nov 12:06	28 Nov 15:47
sanhostTimestamp	5 minutes		29 Nov 15:42
snapmirrorTimestamp	30 minutes	29 Nov 15:46	29 Nov 15:17
snapshotTimestamp	30 minutes	29 Nov 15:46	29 Nov 15:17
statusTimestamp	10 minutes	29 Nov 15:46	29 Nov 15:37
sysInfoTimestamp	1 hour	29 Nov 15:41	29 Nov 14:47
svTimestamp	30 minutes		29 Nov 15:17
svMonTimestamp	8 hours		29 Nov 07:47
xmlQtreeTimestamp	8 hours	29 Nov 13:42	29 Nov 07:47
vFilerTimestamp	1 hour	29 Nov 15:42	29 Nov 14:47

Database

monitordb.db	2.73 GB
dbFileVersion	9
ConnCount	25 connections
MaxCacheSize	523264 KBytes
CurrentCacheSize	523264 KBytes
PeakCacheSize	523264 KBytes
PageSize	8192 Bytes

Logs

discovery	140 KB	26 Oct 2007
DFMMonitor	2.40 MB	29 Nov 15:45
DFMEvent	2.78 MB	29 Nov 10:33
DFMServer	1.08 MB	29 Nov 11:19
DFMScheduler	222 KB	30 Oct 14:27
DFMWatchDog	1.16 MB	29 Nov 15:47
dfm	297 KB	29 Nov 15:40
sybase	5.00 MB	29 Nov 15:13
pingmon	205 KB	28 Nov 21:00
audit	318 KB	29 Nov 15:47
vfmon	46.4 KB	22 Aug 13:27

Services

sql	Started
http	Started
eventd	Started
monitor	Started
scheduler	Started
server	Started
watchdog	Started

Time Since Confirmed Alive

Eventd	5 seconds
Monitor	2 seconds
Scheduler	12 seconds
Server	51 seconds
Watchdog	1 second

Events

DP Job Information

Job State	Count
Jobs Running	0
Jobs Completed Total	0
Jobs Aborted Total	0
Jobs Aborting Total	0
Jobs Completed Today	0
Jobs Aborted Today	0
Jobs Aborting Today	0

Data Set Protection Status

Protection State	Count
Protected	0
Unprotected	0

Version Numbers

Program	Version
Version	3.6.0.4808 (3.6)