Subscribe

CPU getting killed on v3240 w/ONTAP 8.01 7-Mode

We recently installed a new v3240.  I have created all of 4 2.5 TB LUNs connected to two physical Win 2008 R2 hosts.  There is hardly any I/O going on as you can see from the output of sysstat-x below.  Dedupe is not configured for any of the volumes, snapmirror is not running, and compression is turned off.  We only use FCP--no NFS nor iSCSI.  WTF is killing my CPU here?  The chassis fans are blowing at full bore.  Thanks.

sysstat -x -c 10 1 output:

CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
100%      0      0      0      75       1      1      32     24       0      0     1    100%    0%  -     3%      24     51      0     201    344       0      0
100%      0      0      0      38       1      1       0      8       0      0     1    100%    0%  -     1%       0     38      0     259     50       0      0
100%      0      0      0      41       1      1       0      0       0      0     1      -     0%  -     0%       0     41      0     128      1       0      0
100%      0      0      0      28       1      1      24     24       0      0     1    100%    0%  -     4%       0     28      0    1793     33       0      0
100%      0      0      0      31       1      1      48    296       0      0     1    100%   12%  Tf   12%       0     31      0     163     83       0      0
100%      0      0      0     115       1      1     276    252       0      0     1    100%   15%  :    10%       0    115      0     624    452       0      0
100%      0      0      0      46       1      0      24     28       0      0     1    100%    0%  -    13%       5     41      0     237     66       0      0
100%      0      0      0       6       1      0       0      4       0      0     1    100%    0%  -     2%       0      6      0      17      0       0      0
100%      0      0      0      93       1      1       0      0       0      0     1      -     0%  -     0%       0     93      0    1460     27       0      0
100%      0      0      0      13       1      0      24     24       0      0     1    100%    0%  -     4%       0     13      0      36     17       0      0

sysstat -m -c 10 1 output:

ANY  AVG  CPU0 CPU1 CPU2 CPU3
10%  60%   85%  74%  76%   5%
  9%  60%   85%  74%  76%   5%
14%  61%   85%  75%  77%   9%
12%  61%   85%  74%  77%   7%
10%  60%   85%  74%  76%   6%
15%  61%   85%  74%  76%   9%
11%  60%   85%  74%  77%   6%
12%  60%   85%  74%  77%   6%
15%  61%   85%  74%  77%   8%
14%  61%   85%  74%  77%   7%

Re: CPU getting killed on v3240 w/ONTAP 8.01 7-Mode

Try:

priv set advanced

then run:

ps

This might point you to the culprit.

If this continues, I'd create a call with NetApp support.

Regards,

Niek

Re: CPU getting killed on v3240 w/ONTAP 8.01 7-Mode

https://now.netapp.com/NOW/download/software/ontap/8.0.1P3/

There are several CPU related bugs fixed, you might want to give P3 a shot.

Re: CPU getting killed on v3240 w/ONTAP 8.01 7-Mode

We're indeed running P3.  I should have been more clear in my original post.

Re: CPU getting killed on v3240 w/ONTAP 8.01 7-Mode

vmsjaak13 wrote:

Try:

priv set advanced

then run:

ps

This might point you to the culprit.

If this continues, I'd create a call with NetApp support.

Regards,

Niek

I ran ps and am now more confused than ever.  If I'm reading the output correctly--which is in the attached text file--it shows that idle threads are using all the CPU time.  I know idle threads should be using the CPU when the CPUs are idle, but they most certainly are not idle right now.  I'm going to open a support ticket regardless.

sysstat -c 5 -x 1

CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
99%      0      0      0      60       0      1       8     32       0      0    26    100%    0%  -     5%       0     60      0     250    181       0      0
100%      0      0      0      79       1      0       0      0       0      0    26    100%    0%  -     0%       0     79      0     294    382       0      0
100%      0      0      0      31       0      1      16      0       0      0    26    100%    0%  -     0%       0     31      0      34    156       0      0
100%      0      0      0      64       9     11       8     24       0      0    26    100%    0%  -     3%       3     61      0     115    255       0      0
100%      0      0      0     108       0      1       8      0       0      0    26    100%    0%  -     0%       0    108      0     181    648       0      0

Re: CPU getting killed on v3240 w/ONTAP 8.01 7-Mode

Hi,

If you have a chance to generate an NMI-panic via the "RLM" (can't remember what the new name on the 32xx series is at the moment) then you can get a coredump to send along with your case and then this should get cleared up in a much more concrete way by engineering.

It will "crash" the filer "on purpose" to get a stateful coredump of what is going on.  If you have a cluster, it will failover and you could, in theory, do this in a lightly loaded production environment if the host timeouts are set correctly.  YMMV  (your mileage may vary  ;-)  )

Good Luck.

Re: CPU getting killed on v3240 w/ONTAP 8.01 7-Mode

We're seeing this same behavior on 8.0.2. Have you found any more info on this?

Re: CPU getting killed on v3240 w/ONTAP 8.01 7-Mode

I have the same issue on Ontap 8.1RC3. I will try to make a cluster failover tonight. I hope the restart will fix the problem.

Re: CPU getting killed on v3240 w/ONTAP 8.01 7-Mode

it doesn't help. same behavior after restart. one cpu is consequently at 100%. no sis, no compression, nothing...

Re: CPU getting killed on v3240 w/ONTAP 8.01 7-Mode

Just looking at CPU is not a reliable indicator of performance. We will use CPUs opportunistically for all kinds of things.

You have to tie CPU to I/O. Your examples show very little I/O. Almost none.

This reminds me of another thread where someone thought their 6280 was slow because of the CPU.

They were able to add 3x the workload before noticing an increase in latency - all the time, CPU was high...

Add some real load to the system and see how it works please.