Subscribe
Accepted Solution

OC 5.0 server resource utilization

Hi,

I recently helped one of my customer to upgrade from Operations Manager 4.0.2 to OnCommand 5.0D2.

Customer has now started to notice (cannot swear it was there before) high CPU utilization on the virtual server it's hosted on.

He has added vCPUs and more memory but it's still high on CPU.

He says:

I have upgraded the system with 6 GB RAM and 6 CPU.but still its showing 100% cpu usage. most of the resource are used by dbsrv10.exe and dfm process. can we limit it .

I  had already tried to limit the dbcachesize but no help with us.could you please help me out . 

Task Manager says:

So any idea what's going on? What do I tell my customer?

Thanks,

Erik Stjerna

Re: OC 5.0 server resource utilization

Erik,

Could you please provide following info:

- What is the OS (the guest OS) on which OC server is running?

- What is the hypervisor its running on:ESX /HyperV/Xen…? If its ESX does the VM have latest VMware tools installed on guest?

- Was it working fine before? (Any changes made before this problem)

- Also would like to have the following log files for further analysis:

  dfmmonitor.log

  dfmserver.log

  (These files are available at the DFM installation directory)

- The output of dfm diag:

  dfm diag > <text_file>

Thanks,

-Amir

Re: OC 5.0 server resource utilization

Hi Amir,

W2k3 on ESX 4.1. Think it was actually NOT the latest VMware tools, will have the customer check on that and upgrade if need be.

I have a slight suspicion there was a utilization increase with 5.0. Never saw any real issues before the upgrade.

ASUP containing dfmmonitor and dfmserver logs is available via NTSTP, serial number 150009695.

Just had a look myself on the dfmserver.log and it seems that on 23 March, the same day I performed the 4.0 to 5.0 upgrade, the logs is filling with:

Mar 23 13:30:54 [dfmserver:ERROR]: [2960:0x568]: Password decryption failed - keyIndex delimiter not found

Mar 23 13:30:54 [dfmserver:ERROR]: [2960:0x568]: Password decryption failed - keyIndex delimiter not found

Mar 23 13:30:54 [dfmserver:ERROR]: [2960:0x568]: Password decryption failed - keyIndex delimiter not found

Mar 23 13:30:54 [dfmserver:ERROR]: [2960:0x568]: Password decryption failed - keyIndex delimiter not found

Mar 23 13:30:54 [dfmserver:ERROR]: [2960:0x568]: Password decryption failed - keyIndex delimiter not found

Maybe this is what is hogging the system, but if so, what is it and how do I fix it :-)

Best Regards,

Erik

Re: OC 5.0 server resource utilization

Hi Erik,

It seems there was some issue with password encryption at the time of upgrade. Could you please try resetting LDAP/AD bind password with the help of Admin?

Also its good practice to have latest VMware tools installed.

Thanks,

-Amir

Re: OC 5.0 server resource utilization

AFAIK there's no LDAP per se in use, authentication is done locally on windows -> AD.

I no longer have remote access to this customer so I'll have the admin check for any LDAP settings in Operations Manager GUI, or that "dfm ldap list" returns nothing because there's not supposed to be any.

Thanks,

Erik

Re: OC 5.0 server resource utilization

Hi Eric,

     As per the public report this happens due to ldap and upgrade to OC 5.0. BTW is the windows dfm server configured for ldap ?

http://support.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=533814

Regards

adai

Re: OC 5.0 server resource utilization

Yep forgot to mention ldap/AD. With AD can you ask the admin to reset the credentials for the user with which the OC was installed and is being used.

Thanks,

-Amir

Re: OC 5.0 server resource utilization

Hi Erik,

     Since you said you have 6CPU, can you check if you are hitting the vmware problem ?

Determining if multiple virtual CPUs are causing performance issues

Regards

adai

Re: OC 5.0 server resource utilization

So the issue is finally resolved.

Turned out to be ESX server resource starvation.

CPU cycles are now reserved to the DFM VM in vSphere.

Re: OC 5.0 server resource utilization

Can you tell us some pattern what you observed and what tools where used to determine the same so that it helps others too.

Regards

adai