CPU getting killed on v3240 w/ONTAP 8.01 7-Mode

[ogdenclinic]

We recently installed a new v3240.  I have created all of 4 2.5 TB LUNs connected to two physical Win 2008 R2 hosts.  There is hardly any I/O going on as you can see from the output of sysstat-x below.  Dedupe is not configured for any of the volumes, snapmirror is not running, and compression is turned off.  We only use FCP--no NFS nor iSCSI.  WTF is killing my CPU here?  The chassis fans are blowing at full bore.  Thanks.

sysstat -x -c 10 1 output:

CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
100%      0      0      0      75       1      1      32     24       0      0     1    100%    0%  -     3%      24     51      0     201    344       0      0
100%      0      0      0      38       1      1       0      8       0      0     1    100%    0%  -     1%       0     38      0     259     50       0      0
100%      0      0      0      41       1      1       0      0       0      0     1      -     0%  -     0%       0     41      0     128      1       0      0
100%      0      0      0      28       1      1      24     24       0      0     1    100%    0%  -     4%       0     28      0    1793     33       0      0
100%      0      0      0      31       1      1      48    296       0      0     1    100%   12%  Tf   12%       0     31      0     163     83       0      0
100%      0      0      0     115       1      1     276    252       0      0     1    100%   15%  :    10%       0    115      0     624    452       0      0
100%      0      0      0      46       1      0      24     28       0      0     1    100%    0%  -    13%       5     41      0     237     66       0      0
100%      0      0      0       6       1      0       0      4       0      0     1    100%    0%  -     2%       0      6      0      17      0       0      0
100%      0      0      0      93       1      1       0      0       0      0     1      -     0%  -     0%       0     93      0    1460     27       0      0
100%      0      0      0      13       1      0      24     24       0      0     1    100%    0%  -     4%       0     13      0      36     17       0      0

sysstat -m -c 10 1 output:

ANY  AVG  CPU0 CPU1 CPU2 CPU3
 10%  60%   85%  74%  76%   5%
  9%  60%   85%  74%  76%   5%
 14%  61%   85%  75%  77%   9%
 12%  61%   85%  74%  77%   7%
 10%  60%   85%  74%  76%   6%
 15%  61%   85%  74%  76%   9%
 11%  60%   85%  74%  77%   6%
 12%  60%   85%  74%  77%   6%
 15%  61%   85%  74%  77%   8%
 14%  61%   85%  74%  77%   7%

[c_beseler]

I recommend you refer Netapp Support to our Case #: 2003000599. We hit the BUG 568758(but no ossv or snapvault used in environment). And if your case is handled at the moment by first level support, you should insist to come to the second level support. We hung three weeks at the first level support and nothing happened. That was really frustrating.

[colin_graham]

We had a similar thing last year with our 6210. (801p2)  there was minimal load on the filer, yet the CPU was showing being pinned at 99% due to the silly way the filers show their CPU stats it was only 99% on one core out of 8.. performance was unaffected.

It turned out to be related to something in the kernel "beneath" ONTAP that was taking up cpu time.

Dropping into the nodeshell, a ps -auxw output showed a process "/usr/bin/env_mgr -l"  was using 99% cpu time.

We did get a bugID from it (515581), but it was one of those that dissapeared when the filer was rebooted.

[alexander_fassbender]

Hello everybody,

we have the same problem with four or five NetApp systems from 2040 till 3270.

The systems are all SnapVault / SnapMirror targets and the cpu load of the system in "idle" (system does nothing) state ist at 100 %.

Also the console is slow when want to have a snapvault status etc.

But only with ONTAP 8.1. We didn't use or test 8.0.1 releases, only 8.0.2 and with this release we had no problems.

But what I could see is, that when you have "normal" load on the system, eg snapmirror update or snapvault transfers, then the load gone down of the system to a normal load. Then also the console was faster. When the transfers were finished, the cpu load was also at 100 %.

Does anyone know news about this?

[c_beseler]

I could really recommend to open a support case and refer to our Case #: 2003000599 or the official BUG 568758. Netapp believes that this problem has only a few customers worldwide. But if I read your story...

[christin]

Hi Alexander,

Right now BUG 568758 does not have a public report. Please open a case with NetApp Support so that we may investigate your issue in detail.

Regards,

Christine Carcallas