OnCommand Storage Management Software Discussions

Re: Poor performance - DFM database server

Hello guys,

We have a huge DFM environment with the same problems as you. The only way that we found to speed up was to split the DFM servers. But now it become again slower on 2 of them DFM for NAS and DFM for SAN, we found that it is really related to the number of datasets with protection policy. We also tried to Purge the DB with the tool embedded but we have seen no improvement.

We tried to increase CPU, RAM... but nothing help. We have the idea to put this VMs on SSD Drive; anybody try this ?

Now we have this configuration:

DFM for NAS

On Command Core 5.1

VM on Linux, 6vCPU, 12GB RAM, database on RDM FC

218 Datasets

14 Filers

DFM for SAP

On Command Core 5.1

VM on Linux, 6vCPU, 12GB RAM, database on RDM FC

220 Datasets

23 Filers

DFM for SAN

On Command Core 5.1

VM on Linux, 6vCPU, 12GB RAM, database on RDM FC

222 Datasets

12 Filers

DFM For Performance Advisor

On Command Core 5.1

VM on Linux, 6vCPU, 12GB RAM, database on RDM FC

22 Filers monitored

Regards

Jerome

Re: Poor performance - DFM database server

Hi Adai,

Its been a while since we upgraded to 5.0.1. A few weeks back I upgraded to 5.0.2 and just a couple of days back I upgraded to 5.0.2P1. Still very slow especially when backup jobs are running.

I'll try to increase RAM to 16GB but I doubt that will make a difference as the RAM utilization is barely 3GB right now.

Re: Poor performance - DFM database server

That's funny Jerome, I was thinking of putting the DFM server on SSD as well but don't have any available right now.

Jerome, are you noticing the dbsrv10.exe process constantly using around 25% cpu when your environment slows down? I myself had to split us the DFM servers as a result of this slowness and turned off performance advisor as well, but nothing has helped so far.

Re: Poor performance - DFM database server

Hello,

When it's slow we use 400% to 500% of CPU means the DB use 4 to 5 vcpu (6 vcpu in total)

PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND

  27182 root      16   0 6664m 4.7g  15m S 370.8 40.0  65899:10 dbsrv11

27222 root      15   0  245m  26m  10m S  0.3  0.2  36:37.45 dfmeventd

27226 root      15   0  159m 6608 5848 S  0.3  0.1  21:19.48 dfmwatchdog

jerome

Re: Poor performance - DFM database server

We foud also that a lot of things are not purged form the DB and I thinks it's not helping performance. Look at the event count number it's horrific

Exemple on our DFM NAS (CMD: dfm diag)

DP Job Information

Job State                        Count

Jobs Running                     44

Jobs Completed Total             124949

Jobs Aborted Total               0

Jobs Aborting Total              0

Jobs Completed Today             6184

Jobs Aborted Today               0

Jobs Aborting Today              0

Dataset Protection Status

Protection State                 Count

Protected                        216

Unprotected                      2

Event Counts

Table                            Count

Events                           15694712

Current Events                   15641683

Abnormal Events                  12152877

Event Type Counts

Event Type                 Count

volume.growthrate          11211599

sm.update                  3177912

data-protection-job.status 892396

snap-status                29322

df.avail                   29253

volume-clone.discovered    27466

df.kbytes                  25057

df.snapshot.kbytes         24847

df.inodes                  24845

volume.first-snap          24843

Version                          5.1 (5.1)

Re: Poor performance - DFM database server

Set RAM to 16GB, still seeing same issues.

It seems the sybase database is the bottleneck. I've tried setting the dbcachesize parameter higher before which didn't help much. All other functions appear to be fine but everything slows down as soon as any datasets come into play.

Any ideas gents?

Re: Poor performance - DFM database server

I have upgraded to 5.1 in hopes of sybase 11 improving performance but it seems to still be the same. As I mentioned, I believe the bottleneck is the database itself. Based on the following article, my guess is that only one vcpu gets pegged waiting for a sql query to finish and as a result, any other function takes a long time to execute (when the vcpu gets pegged, I can tell from task manager that the other processes are not using much cpu resources).

http://www.symantec.com/business/support/index?page=content&id=TECH190207

Does anyone have experience with using dbisqlc to monitor what's going on with the sybase database. Not really sure what credentials to use and what connection parameters should be set to access the database. Any help would be greatly appreciated.

Regards,

Usman

Re: Poor performance - DFM database server

Also, I am seeing in the dfmserver.log that dfm is still trying to connect to vfiler's that were deleted a while ago. How would I go about stopping it?

Regards,

Usman

Re: Poor performance - DFM database server

Hi Jerome,

     In your case the problem seem to be mainly due to DP jobs per day and its history. Looks like your jobs purge is set to 21 days. I would recommend you to bring this down to 1 week.

With respect to the events purge, 5.2 will address the same. I suggest you to lower your events purge to 1month or so.

Also putting a SSD will definitely help.

Regards

adai

Re: Poor performance - DFM database server

Hi Usman,

     The symantec link that you point to doesnt have much relevance here. But  can you run by this VMware KB to see you are not affected by this ?

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1005362

Also the time it takes for edit of the dataset is not necessarily due to server but also few other things. How many relationship per dataset do you have ?

Regards

adai

Forums