Subscribe

Identifying the cause of High CPU on 1 Proc

I am running sysstat -m and I am seeing one processor that always seems to be running at about 85% to 95%.    I would love to identify the cause.

What are people using to identify the process that would be causing that?

Russ Bass

Univita Health

4x3160s

Re: Identifying the cause of High CPU on 1 Proc

One of the NetApp gurus told me that some processes are not as well threaded as others which causes the asymmetric CPU utilization.  I believe it is possible to really see what is going on by doing a "priv set advanced" and then using the ps command but I haven't really tested that.  I've also anecdotally been told that multi-threading is better on 7.3.2 and later.

Re: Identifying the cause of High CPU on 1 Proc

Hi,

Use this with care

priv set diag

sysstat -M 1

Now you can see what's going on on your filer. You will see several items (called domains). Probably the "Kahuna" domain is hogging your CPU ...

The Kahuna domain contains "all the rest" that is not mentioned seperateley (wafl task, snapmirrir, deduplication and other system tasks are part of the kahuna domain - we still have ontap 7.2.x)

As of Ontap 7.3, some tasks are seperated from the kahuna domain. I don't know which tasks.

See for yourself and use with care.

If you don't understand what's going on on your filer, open a support case and the will ask a perfstat. Netapp support can see what's going on

Greetings,

Boeckx Kris

Pidpa

Re: Identifying the cause of High CPU on 1 Proc

If you have Operations Manager setup, you can also use Performance Advisor inside the NetApp Management Console (NMC) to see something of a graphical breakdown of what sysstat gives you.

Re: Identifying the cause of High CPU on 1 Proc

Hi Russ and welcome to Communities!

What ONTAP version are you running? As already said in this thread: 7.3.2 onwards is dealing with multithreading better than previous versions, so if you are on anything prior to 7.3.2, ONTAP upgrade may solve the issue.

Regards,

Radek

Re: Identifying the cause of High CPU on 1 Proc

We are running 7.3.2. It always seems to be the 4th proc that is the highest.

The ps cmd doesn't show anything consuming significant amounts of CPU except the idle_thread* process.

Russ Bass

Univita Health

4x3160s

v7.3.2

Re: Identifying the cause of High CPU on 1 Proc

Boeckx response does show Kahuna consuming 50-60% CPU, which corresponds to CPU3.

Re: Identifying the cause of High CPU on 1 Proc

did you run

priv set diag

sysstat -m 1

can you post some of it here?

Eric

Re: Identifying the cause of High CPU on 1 Proc

Hi Eric,

You have to use a capital "M" like "sysstat -M 1"

A "sysstat -m 1" will show you:

ANY  AVG  CPU0 CPU1 CPU2 CPU3

A "sysstat -M 1" will show you: (for ontap version prior to 7.3.2)

ANY1+ ANY2+ ANY3+ ANY4+  AVG CPU0 CPU1 CPU2 CPU3 Network Storage Raid Target Kahuna WAFL_Ex(Kahu) Cifs Exempt Intr Host Ops/s   CP

Greets,

Kris

Re: Identifying the cause of High CPU on 1 Proc

Hi,

These are the commands I use to troubleshoot performance issues:

always start with "priv set diag"

sysstat -M -i 5

--> already explained

sysstat -x 5

--> shows you the different I/O ("Disk util" is an importend one and also "CPty" see http://now.netapp.com/NOW/knowledge/docs/ontap/rel707/html/ontap/cmdref/man1/na_sysstat.1.html)

lun stats -i 5

--> shows you the read / write / latency's of luns

stats show lun

--> shows you detailed info of every lun (you will want to capture this in an output file)

stats show volume

--> same as lun but now for the volumes (you will want to capture this in an output file)

reallocate status

--> shows if any reallocation jobs are running (walf scan status shows you even more info)

If this is not enough you can get some info with statit

a "statit -b" will start the data collection (wait a few minutes)

a "statit -e" will stop the collection and will give you the result. (you will want this to capture in an output file)

If you want to capture the output in a file, connect via PUTTY to the filer. In PUTTY, you can specify an capture file.

Greetings,

Kris Boeckx