DFM so s...l...o...w....when working with alarms

[BEN_HOLKO]

In the latest version of OnCommand (5.1) when creating or editing alarms it is taking over 1 minute after pressing "finish"/"save" etc to create or alter an alarm, It does this at CLI or GUI, and within performance advisor the problem occurs as well.

I have had a quick look at the logs but cant see anything obvious.

I have 5 different OnCommands installed, and I have seen this before on another instance. In the last 6 months I have built 3 instances and it behaved like this on two of them, while one was always fast to respond.

Any idea where to start looking?

[BEN_HOLKO]

Further to this, even going to into perf advisor and selecting "setup" -> "Alarms" it takes over a minute for the defined alarms to appear in the list....

[sharmadevender]

Have you met installation requirement as per Netapp Recommendations for Oncommand? Also  you can check for Performance Advisor data size, it may respond slow, if it's bigger in size.

[BEN_HOLKO]

this is a new install with a blank database, a VM, 4GB RAM and 2vCPU, same as all my other installs.

[adaikkap]

Hi Ben,

             To really understand whats going on, can you give us an output of dfm diag and upload to same to understand how this server is being used ?

Also can you check if multiple virutal cpus are causing any perofrmance issues?

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1005362

You said, 5 DFM server are they all talking to the same set of controllers ? If so you may have to tone down some of the monitoring intervals.

Regards

adai

[BEN_HOLKO]

I have tried dropping the VM back to 1 vCPU and it made no difference to the problem.

My 5 DFM machines are monitoring 5 different arrays in completely separate infrastructures. I have attached dfm diag output.

[kryan]

Hello Ben,

There are few areas to investigate when noticing slowness in the NMC.  Since this is a new installation (not a restored db backup) we will rule out large perf data files as a cause of the slowness.

1) Are you observing this slowness in any other area of the NMC?  Any portions of the Web UI or OnCommand Console?

2) Where is the NMC installed in regards to the UM server it is communicating to?  Is it installed on the UM server?  The same subnet?  Across a WAN?

3) Are the CPU and memory resources for the UM VM reserved and not limited in any fashion? 

Thanks,

Kevin

[BEN_HOLKO]

1) not really, just when working with alarms. In the management console if I click "Set Up" and "Alarms" it takes a good 1-2 minutes just for the alarms to show up as a list...

2) locally on the server

3) no limits, no cpu or memory constraints or problems that I can see.

[adaikkap]

Hi Ben,

          I just took a look at the dfm diag, I see no issue

1. Its a brand new/fresh setup
2. Only 2 controllers
3. Default Monitoring Intervals
4. None of the object counts are even near any of the limits
5. No protection only monitoring and performance advisor
6. No db issues just 20MB

Pls see if there are any performance problems with VMware.

Also what you have in our Server resource( RAM) is the minimum requirement which should be fine for 2 controllers monitoring.But if possible try increasing it to 6 or 8GB and let us know if that alleviates the problem.

Regards

adai

[kryan]

I just tested creating an alarm using the NMC against a UM 5.1 7-Mode VM with 4 hosts and default monitors (no special VM reservations) and the alarm created in seconds.

Kevin

[BEN_HOLKO]

The VM itself seems fine, has plenty of RAM available. All of my 5 DFMs are the same spec - as I say I have seen this behaviour before on another install, was never able to correct it there either. But also I built another DFM a few weeks ago, same process, same spec, and it is running fine with no problems.

[BEN_HOLKO]

ok I found something....

each of my created alarms has an SNMP trap destination. I installed wireshark and sniffed what was going on while waiting for the alarm list to load, and I see the DFM box sending a NBNS request to the trap destination IP.

The DFM server is sending a NetBIOS NameQuery to the trap IP, the query is all zeros; *<00><00><00><00><00><00><00><00><00><00><00><00><00><00><00>"

This traffic is all blocked by downstream firewalls.

any ideas?

[adaikkap]

Hi Ben,

     That looks like a possible culprit. Can you disable or remove the traps and see if you encounter the same delay or slowness ?

Regards

adai

[BEN_HOLKO]

Well rather oddly, the system is now performing fine. Alarm screen refresh is now taking about 3 seconds. I packet captured again and it seems the NBS packets have stopped. No changes have been made, weird! Will continue to monitor.