100% CPU utilization on A30

Question

When looking at a system node run -command sysstat -m I get half of the cpus as pegged at 100%:

ntap01::> system node run -node ntap01-01 -command sysstat -mANY  AVG  CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 CPU24 CPU25 CPU26 CPU27 CPU28 CPU29 CPU30 CPU31

99%  28%   27% 100%  13% 100%  12% 100%  25% 100%  14% 100%   15%  100%   22%  100%   13%  100%   14%  100%   18%  100%   21%  100%   11%  100%   12%  100%   76%  100%   76%  100%   76%  100%

99%  25%   23% 100%  10% 100%   9% 100%  18% 100%  11% 100%   12%  100%   18%  100%   12%  100%   11%  100%   15%  100%   16%  100%   11%  100%   12%  100%   73%  100%   72%  100%   73%  100%

97%  25%   23% 100%  10% 100%  11% 100%  16% 100%  10% 100%   11%  100%   18%  100%   12%  100%   10%  100%   17%  100%   19%  100%   12%  100%   11%  100%   72%  100%   72%  100%   72%  100%

99%  26%   23% 100%  12% 100%  13% 100%  20% 100%  11% 100%   13%  100%   19%  100%   12%  100%   12%  100%   17%  100%   18%  100%   20%  100%   11%  100%   74%  100%   73%  100%   74%  100%

97%  25%   24% 100%  11% 100%  10% 100%  19% 100%  10% 100%   11%  100%   18%  100%   12%  100%   11%  100%   14%  100%   22%  100%   10%  100%   11%  100%   73%  100%   72%  100%   73%  100%

99%  24%   19% 100%  10% 100%  10% 100%  16% 100%  11% 100%   12%  100%   16%  100%   11%  100%   11%  100%   13%  100%   17%  100%   11%  100%   11%  100%   71%  100%   71%  100%   70%  100%

99%  33%   35% 100%  19% 100%  19% 100%  26% 100%  19% 100%   20%  100%   28%  100%   19%  100%   18%  100%   27%  100%   28%  100%   17%  100%   18%  100%   78%  100%   79%  100%   79%  100%

This is an A30 running 9.17.1P2.

AI on Google says that this is due to each dual core CPU being divided into a control plane and a data plane: one core for data, the other core for administrative and background tasks. The data plane/core is constantly polling and hence looks like it is "hot" and at 100%. This makes sense, but I can't find any official NetApp documentation on this. Can anyone verify this? Thanks.

JonathanGaudette · Answer

I need to make another correction. The AFF A30 does not have two physical CPUs. It has a single 16-core Xeon but has Intel Hyperthreading enabled, so it will report as though it has 32 cores.

Hence the 32 cores listed in "sysstat".

Apologies for any confusion.

JonathanGaudette · Answer

Newer ONTAP versions heavily utilize as much additional CPU and RAM resources as possible for the myriad of background tasks required of it. Much of these resources are consumed by our "storage efficiency" features, which can run constantly in the background compressing blocks, deduplicating data, snapshotting, etc. WAFL also spends a lot of its time performing all the various background duties to clean up all unused blocks, mapping the filesystem, etc. This is true even for Snapmirror destination systems that have to replicate the storage efficiency and layouts of the source systems.

Instead of looking at individual cores (which ONTAP's kernel scheduler handles automatically), what do you see for a simple "systat -1"? Are any of the nodes running at above 50% utilization in total on a consistent basis?

::> run -node <node> sysstat -1

If not, that's your true total utilization number and it is well within expected operating parameters.

Looking at individual cores is missing the overall picture. The old idiom, "missing the forest for the trees" holds true here.

Disk_Jockey · Answer

Jonathan,

Thanks for the response. I should have been a little more clear on my observations. We have 3 of these A30s  and all of them are exhibiting the same behavior. And all of the pegged CPUs are the odd numbered ones. This is not organic. And the workloads are very, very minimal, mostly snapmirror destinations updated hourly. Also, there are only 16 cores per head, but sysstat is showing 32 per head. Also, it is obvious that the AVG calculation is not factoring in the odd 100% CPUs, so this begs the question: what are those odd "cpus" and why are they pegged 24/7?

I have been working with NetApp since the 800 series and I have never seen this behavior. When a head is running hot, pegged CPUs will fluctuate at least on some iterations. But to have all odd numbers NEVER fluctuate from 100% is not normal workload behavior. I'm not saying there is a problem with the system. What I'm suspicious of is that this is some sort of design issue. And it would be nice to know what that is when the customer is asking about it.

Thanks!