OC 5.0 server resource utilization

[stjerna]

Hi,

I recently helped one of my customer to upgrade from Operations Manager 4.0.2 to OnCommand 5.0D2.

Customer has now started to notice (cannot swear it was there before) high CPU utilization on the virtual server it's hosted on.

He has added vCPUs and more memory but it's still high on CPU.

He says:

I have upgraded the system with 6 GB RAM and 6 CPU.but still its showing 100% cpu usage. most of the resource are used by dbsrv10.exe and dfm process. can we limit it .

I  had already tried to limit the dbcachesize but no help with us.could you please help me out . 

Task Manager says:

So any idea what's going on? What do I tell my customer?

Thanks,

Erik Stjerna

[amirm]

Erik,

Could you please provide following info:

- What is the OS (the guest OS) on which OC server is running?

- What is the hypervisor its running on:ESX /HyperV/Xen…? If its ESX does the VM have latest VMware tools installed on guest?

- Was it working fine before? (Any changes made before this problem)

- Also would like to have the following log files for further analysis:

  dfmmonitor.log

  dfmserver.log

  (These files are available at the DFM installation directory)

- The output of dfm diag:

  dfm diag > <text_file>

Thanks,

-Amir

[stjerna]

Hi Amir,

W2k3 on ESX 4.1. Think it was actually NOT the latest VMware tools, will have the customer check on that and upgrade if need be.

I have a slight suspicion there was a utilization increase with 5.0. Never saw any real issues before the upgrade.

ASUP containing dfmmonitor and dfmserver logs is available via NTSTP, serial number 150009695.

Just had a look myself on the dfmserver.log and it seems that on 23 March, the same day I performed the 4.0 to 5.0 upgrade, the logs is filling with:

Mar 23 13:30:54 [dfmserver:ERROR]: [2960:0x568]: Password decryption failed - keyIndex delimiter not found

Mar 23 13:30:54 [dfmserver:ERROR]: [2960:0x568]: Password decryption failed - keyIndex delimiter not found

Mar 23 13:30:54 [dfmserver:ERROR]: [2960:0x568]: Password decryption failed - keyIndex delimiter not found

Mar 23 13:30:54 [dfmserver:ERROR]: [2960:0x568]: Password decryption failed - keyIndex delimiter not found

Mar 23 13:30:54 [dfmserver:ERROR]: [2960:0x568]: Password decryption failed - keyIndex delimiter not found

Maybe this is what is hogging the system, but if so, what is it and how do I fix it 🙂

Best Regards,

Erik

[amirm]

Hi Erik,

It seems there was some issue with password encryption at the time of upgrade. Could you please try resetting LDAP/AD bind password with the help of Admin?

Also its good practice to have latest VMware tools installed.

Thanks,

-Amir

[stjerna]

AFAIK there's no LDAP per se in use, authentication is done locally on windows -> AD.

I no longer have remote access to this customer so I'll have the admin check for any LDAP settings in Operations Manager GUI, or that "dfm ldap list" returns nothing because there's not supposed to be any.

Thanks,

Erik

[amirm]

Yep forgot to mention ldap/AD. With AD can you ask the admin to reset the credentials for the user with which the OC was installed and is being used.

Thanks,

-Amir

[adaikkap]

Hi Eric,

     As per the public report this happens due to ldap and upgrade to OC 5.0. BTW is the windows dfm server configured for ldap ?

http://support.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=533814

Regards

adai

[adaikkap]

Hi Erik,

     Since you said you have 6CPU, can you check if you are hitting the vmware problem ?

Determining if multiple virtual CPUs are causing performance issues

Regards

adai

[stjerna]

So the issue is finally resolved.

Turned out to be ESX server resource starvation.

CPU cycles are now reserved to the DFM VM in vSphere.

[adaikkap]

Can you tell us some pattern what you observed and what tools where used to determine the same so that it helps others too.

Regards

adai

[stjerna · Accepted solution]

So this is from the Tech Support case that was opened, notes from the TSE regarding changes done on the settings for this VM in vSphere Client:

In VM "Edit Settings" for the DFM VM (Windows 32bit)

Hardware was already set on 4GB and 4 CPU before changes but memory and cpu in Resources tab was not set

In Hardware tab we changed 4 CPUs to 2 CPUs

CPU Custom: 1000 mhz

Memory Reservations was changed to 4GB, 3GB, and 1GB. in 4 and 3 GB, the VM would not start

Config 1

Hardware tab

1 CPU and 4GB

Resources tab

CPU in Custom 1000 mhz, memory at 1GB host took about 15 minutes to come up, Windows not responsive when OnCommand started

Config 2

Hardware tab

2 CPUs and 4GB

Resource tab

CPU in Normal 4000 mhz, memory at 2GB host took about 3 minutes to come up, OnCommand Console comes up with no problem

Basic DFM functions work normally

Left the settings as Config 2 above.

Customer will consider moving to OnCommand 64bit with 64bit Windows 2008.