topic Re: Filer high CPU cause vcs service group goes down in ONTAP Discussions

topic Re: Filer high CPU cause vcs service group goes down in ONTAP Discussions https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30459#M7202 <HTML><HEAD></HEAD><BODY><P>It doesn't look that loaded down to me - just the normal noises in here.</P><P></P><P>I'd leave the options open - it may nat have been the filer that caused the crash.</P><P></P><P>Performance is an end to end thing with complex environs in this day and age.</P><P></P><P>How about some 'sysstat -x 1' output ?</P><P></P><P><BR />I hope this response has been helpful to you.</P><P></P><P>At your service,</P><P></P><P><BR />Eugene E. Kashpureff<BR /><A class="jive-link-email-small" href="mailto:ekashp@kashpureff.org" target="_blank">ekashp@kashpureff.org</A><BR />Fastlane NetApp Instructor and Independent Consultant<BR /><A class="jive-link-external-small" href="http://www.fastlaneus.com/" target="_blank">http://www.fastlaneus.com/</A><SPAN> </SPAN><A class="jive-link-external-small" href="http://www.linkedin.com/in/eugenekashpureff" target="_blank">http://www.linkedin.com/in/eugenekashpureff</A></P><P></P><P>(P.S. I appreciate points for helpful or correct answers.)</P></BODY></HTML> Fri, 12 Nov 2010 04:09:56 GMT ekashpureff 2010-11-12T04:09:56Z Filer high CPU cause vcs service group goes down https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30449#M7195 <HTML><HEAD></HEAD><BODY><P>Hi All,</P><P>In my company, we are using vcs cluster running oracle and the storage is on the Netapp filer FAS3020 version 7.2.4P7. </P><P>There is one time some service group on VCS went down. and at the same time the filer having high CPU usage. </P><P>I didn't capture the CPU usage at that time, But this is what i got after that:-</P><P></P><DIV id="_mcePaste">ANY1+ ANY2+  AVG CPU0 CPU1 Network Storage Raid Target Kahuna Cifs Exempt Intr Host Ops/s   CP</DIV><DIV id="_mcePaste">  87%   39%  64%  85%  47%     37%      9%  17%     0%    50%   6%     5%   4%   0% 11757 100%</DIV><DIV id="_mcePaste">  74%   21%  48%  70%  26%     27%      8%  15%     0%    33%   4%     4%   4%   0%  7841 100%</DIV><DIV id="_mcePaste">  82%   23%  53%  76%  30%     30%      9%  12%     0%    46%   4%     2%   4%   0%  8752  43%</DIV><DIV id="_mcePaste">  83%   22%  53%  79%  28%     35%      7%   7%     0%    49%   4%     0%   4%   0% 10205   0%</DIV><DIV id="_mcePaste">  85%   18%  52%  80%  25%     34%      6%   6%     0%    50%   4%     0%   4%   0% 11074   0%</DIV><DIV id="_mcePaste">  79%   17%  49%  76%  22%     35%      6%   6%     0%    42%   3%     0%   4%   0% 10815   0%</DIV><DIV id="_mcePaste">  83%   20%  53%  80%  24%     39%      8%   8%     0%    42%   4%     0%   4%   0% 11123   0%</DIV><DIV id="_mcePaste">  64%   10%  38%  62%  15%     26%      6%   6%     0%    30%   3%     0%   4%   0%  7414   0%</DIV><DIV id="_mcePaste">  50%    5%  28%  48%   9%     20%      4%   4%     0%    21%   3%     0%   4%   0%  8062   0%</DIV><DIV id="_mcePaste">  76%   19%  49%  70%  26%     31%      6%   6%     0%    46%   4%     0%   4%   0%  8673   0%</DIV><DIV id="_mcePaste">  96%   52%  74%  82%  67%     45%      7%   7%     0%    79%   8%     0%   4%   0% 10407   0%</DIV><DIV id="_mcePaste">--</DIV><DIV id="_mcePaste">Summary Statistics (   11 samples  1.0 secs/sample)</DIV><DIV id="_mcePaste">ANY1+ ANY2+  AVG CPU0 CPU1 Network Storage Raid Target Kahuna Cifs Exempt Intr Host Ops/s   CP</DIV><DIV id="_mcePaste">Min</DIV><DIV id="_mcePaste"> 28%   6357    880      0    5024 27894    23300      0       0     0       3</DIV><DIV id="_mcePaste">Avg</DIV><DIV id="_mcePaste"> 51%   8480   1167      0   10135 40876    30658   3335       0     0       3</DIV><DIV id="_mcePaste">Max</DIV><DIV id="_mcePaste"> 80%  10209   1780      0   13475 59917    39357  16955       0     0       3</DIV><DIV> </DIV><P></P><P>What puzzle me, There is no clear error in /etc/messages on the filer shows there is problem with the filer. Only the snapshot delete.<BR />The filer some time having CPU spike, since there is no issue, we just ignore it.Please help to investigate this.</P></BODY></HTML> Thu, 05 Jun 2025 07:05:38 GMT https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30449#M7195 khairulanuar 2025-06-05T07:05:38Z Re: Filer high CPU cause vcs service group goes down https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30453#M7197 <HTML><HEAD></HEAD><BODY><P>Anyone have any ideas? In the VCS log shows nothing abnormal. My question maybe, what can cause CPU high?</P><P>Some possibilities :-</P><P>1. Overload on the NFS --> how to check and confirm there is overload at that time?</P></BODY></HTML> Fri, 12 Nov 2010 03:59:26 GMT https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30453#M7197 khairulanuar 2010-11-12T03:59:26Z Re: Filer high CPU cause vcs service group goes down https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30459#M7202 <HTML><HEAD></HEAD><BODY><P>It doesn't look that loaded down to me - just the normal noises in here.</P><P></P><P>I'd leave the options open - it may nat have been the filer that caused the crash.</P><P></P><P>Performance is an end to end thing with complex environs in this day and age.</P><P></P><P>How about some 'sysstat -x 1' output ?</P><P></P><P><BR />I hope this response has been helpful to you.</P><P></P><P>At your service,</P><P></P><P><BR />Eugene E. Kashpureff<BR /><A class="jive-link-email-small" href="mailto:ekashp@kashpureff.org" target="_blank">ekashp@kashpureff.org</A><BR />Fastlane NetApp Instructor and Independent Consultant<BR /><A class="jive-link-external-small" href="http://www.fastlaneus.com/" target="_blank">http://www.fastlaneus.com/</A><SPAN> </SPAN><A class="jive-link-external-small" href="http://www.linkedin.com/in/eugenekashpureff" target="_blank">http://www.linkedin.com/in/eugenekashpureff</A></P><P></P><P>(P.S. I appreciate points for helpful or correct answers.)</P></BODY></HTML> Fri, 12 Nov 2010 04:09:56 GMT https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30459#M7202 ekashpureff 2010-11-12T04:09:56Z Re: Filer high CPU cause vcs service group goes down https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30464#M7205 <HTML><HEAD></HEAD><BODY><P>Currently i receive this error from the console. </P><P>======================</P><P></P><DIV id="_mcePaste">Object action: admin.monitor.getvar</DIV><DIV id="_mcePaste">Object action: admin.util.ONTAP</DIV><DIV id="_mcePaste">Object action: admin.monitor.getvar</DIV><DIV id="_mcePaste">Object action: admin.util.ONTAP</DIV><DIV id="_mcePaste">Object action: admin.monitor.getvar</DIV><DIV id="_mcePaste">Object action: admin.util.ONTAP</DIV><DIV id="_mcePaste">Object action: admin.monitor.getvar</DIV><DIV id="_mcePaste">Object action: admin.util.ONTAP</DIV><DIV id="_mcePaste">Object action: admin.monitor.getvar</DIV><DIV id="_mcePaste">Object action: admin.util.ONTAP</DIV><DIV id="_mcePaste">Object action: admin.monitor.getvar</DIV><DIV id="_mcePaste">Object action: admin.util.ONTAP</DIV><DIV> </DIV><P></P><P>No change been done. and the filer having slowness right now. Can anyone tell what it is, tried to google found nothing useful.</P></BODY></HTML> Fri, 12 Nov 2010 08:49:48 GMT https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30464#M7205 khairulanuar 2010-11-12T08:49:48Z Re: Filer high CPU cause vcs service group goes down https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30470#M7208 <HTML><HEAD></HEAD><BODY><P>This is the output of "ps" command. After sort it out base on CPU.</P><P></P><P></P><DIV id="_mcePaste">Process statistics over 232077.452 seconds...</DIV><DIV id="_mcePaste">   ID State Domain %CPU StackUsed %StackUsed Name</DIV><DIV> </DIV><P></P><P>    2 RR    i       60%       368         8% idle_thread1</P><P>  601 RR    k       41%      9640        29% wafl_hipri</P><P>    1 RR    i       23%       368         8% idle_thread0</P><P>  251 RR    n       13%      3552        10% nfsadmin</P><P>  184 BR    r       12%      4608        28% raidio_thread</P><P>   76 RR    n        7%      3768        11% 10/100/1000-VI/e0a</P><P>   72 RR    n        6%      3544        10% 10/100/1000-V/e1a</P><P>   63 BR    s        5%      2672        32% ispfc_main</P><P>   73 BR    n        5%      3316        10% 10/100/1000-V/e1b</P><DIV> </DIV><DIV> </DIV><DIV><DIV>ANY1+ ANY2+  AVG CPU0 CPU1 Network Storage Raid Target Kahuna Cifs Exempt Intr Host Ops/s   CP</DIV><DIV> 100%   96%  98% 100%  98%     52%     18%  32%     0%    74%   9%     8%   3%   0% 13375 100%</DIV><DIV>  99%   70%  85%  93%  76%     50%     16%  17%     0%    73%   9%     0%   4%   0% 11684  18%</DIV><DIV>  99%   66%  83%  94%  68%     55%     12%  12%     0%    73%   9%     0%   4%   0% 12354   0%</DIV><DIV> 100%   86%  93%  98%  91%     60%     15%  15%     0%    84%   9%     0%   3%   0% 12815   0%</DIV><DIV> 100%   82%  91%  96%  85%     58%     15%  15%     0%    81%   8%     0%   3%   0% 11404   0%</DIV><DIV> 100%   92%  96%  83%  77%     56%     15%  21%     0%    87%   6%     4%   3%   0% 10451  27%</DIV><DIV> 100%  100% 100% 100% 100%     45%     15%  36%     0%    85%   4%    11%   3%   0% 10235 100%</DIV><DIV> 100%  100% 100% 100% 100%     50%     16%  31%     0%    87%   5%     9%   3%   0% 11924 100%</DIV><DIV> 100%   98%  99% 100% 100%     49%     17%  34%     0%    78%   7%    10%   3%   0% 10813 100%</DIV><DIV> 100%   94%  97% 100%  97%     53%     18%  29%     0%    77%   8%     7%   3%   0% 12485 100%</DIV><DIV> 100%   85%  92%  97%  88%     58%     15%  15%     0%    85%   9%     0%   3%   0% 14064  15%</DIV><DIV> 100%   81%  91%  95%  87%     54%     14%  14%     0%    86%   9%     0%   4%   0% 12504   0%</DIV><DIV> 100%   74%  87%  97%  78%     54%     14%  14%     0%    77%  10%     0%   4%   0% 13205   0%</DIV><DIV>  96%   44%  70%  93%  48%     48%     14%  14%     0%    53%   7%     0%   5%   0% 14745   0%</DIV><DIV>  98%   60%  79%  95%  64%     56%     16%  16%     0%    60%   6%     0%   4%   0% 12839   0%</DIV><DIV>  98%   57%  78%  95%  61%     56%     15%  15%     0%    58%   7%     0%   5%   0% 13254   0%</DIV><DIV> 100%   83%  92%  88%  93%     36%     15%  34%     0%    82%   4%    10%   3%   0%  7039  95%</DIV><DIV>--</DIV><DIV>Summary Statistics (   36 samples  1.0 secs/sample)</DIV><DIV>Min</DIV><DIV> 70%   6076    787      0    5666 45209    28548      0       0     0      30s</DIV><DIV>Avg</DIV><DIV> 92%  10636   1702      0   11872 64211    52321  12303       0     0      34s</DIV><DIV>Max</DIV><DIV> 99%  15020   2476      0   18905 81916    72986  41458       0     0      51s</DIV><DIV> </DIV></DIV><P></P></BODY></HTML> Fri, 12 Nov 2010 09:12:45 GMT https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30470#M7208 khairulanuar 2010-11-12T09:12:45Z Re: Filer high CPU cause vcs service group goes down https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30475#M7210 <HTML><HEAD></HEAD><BODY><P>Next time it happens drop into priv set advanced and run a statit -b wait a little bit (30 seconds or so) and run a statit -e  post the results.</P><P>Is this NFS connected? if so and nfsstat would help.</P><P></P><P></P><P>On a side note 10k op/s would have to be getting close to the max a FAS3020 could do? (dont know havent really worked with them much)</P></BODY></HTML> Sat, 13 Nov 2010 09:30:49 GMT https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30475#M7210 shane_bradley 2010-11-13T09:30:49Z Re: Filer high CPU cause vcs service group goes down https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30480#M7212 <HTML><HEAD></HEAD><BODY><P>shane</P><P>Right now the CPU is running normal. on the statit command , what particular event should i focus to?</P><P>You mention 10k op/s would have to be getting close to the max a FAS3020, How can i calculate what is the max it can go.</P><P>Sorry for newbie question.</P><P>Thanks </P></BODY></HTML> Mon, 15 Nov 2010 03:04:20 GMT https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30480#M7212 khairulanuar 2010-11-15T03:04:20Z Re: Filer high CPU cause vcs service group goes down https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30485#M7215 <HTML><HEAD></HEAD><BODY><P>Hey</P><P></P><P>Never apologise for being a newbie, everyone was a newbie once <SPAN __jive_emoticon_name="happy" __jive_macro_name="emoticon" class="jive_macro jive_emote" height="16" src="https://community.netapp.com/4.0.8/images/emoticons/happy.gif" width="16"></SPAN>  This is IT face it everyone becomes a newbie all over again every 3-5 years <SPAN __jive_emoticon_name="silly" __jive_macro_name="emoticon" class="jive_macro jive_macro_emoticon" src="https://community.netapp.com/4.0.8/images/emoticons/silly.gif"></SPAN></P><P></P><P></P><P>The statit gathers a pile of useful stats for troubleshooting performance issues, its not quite as hardcore as a perfstat but it has alot of good info for working out where a performance bottleneck maybe.</P><P></P><P>Run it when you hit the next high CPU event.</P><P></P><P>There are a few questions to ask too, how often do they happen? do they happen regularly?  the last performance issue i saw was on a FAS2040 turns out it was due to a DBA running a database dump into the snapinfo lun at a certain time every day. </P><P></P><P>The op/s comment was more a guess based on what i've seen FAS3050 and 6080's do.  How many "op's" a box can do is dependant on a lot of things the least of which is what size.</P></BODY></HTML> Mon, 15 Nov 2010 05:34:22 GMT https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30485#M7215 shane_bradley 2010-11-15T05:34:22Z Re: Filer high CPU cause vcs service group goes down https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30490#M7218 <HTML><HEAD></HEAD><BODY><P>Thanks for the advice. </P><P>From DFM the CPU spikes few times last month and this month, but not at the same timing.</P><P>Yes, the filer has oracle run on VCS in it, i might need to check on oracle after this.</P><P>Just another question, is there any possibilities too big size of volume may cause the slowness and latency?</P><P>Just throwing a wild guess cause i found one bigger size of volume was there.</P><P>#df -g</P><DIV id="_mcePaste">/vol/WDB/          1170GB     1084GB       85GB      93%  /vol/TIPITWDB/</DIV><DIV id="_mcePaste">/vol/WDB/.snapshot      130GB       49GB       80GB      38%  /vol/TIPITWDB/.snapshot</DIV><DIV> </DIV><P></P><P>thanks for the feedback.</P></BODY></HTML> Tue, 16 Nov 2010 01:41:06 GMT https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30490#M7218 khairulanuar 2010-11-16T01:41:06Z Re: Filer high CPU cause vcs service group goes down https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30495#M7221 <HTML><HEAD></HEAD><BODY><P>You should really post a "sysstat -x 1" output during the peaks. besides that, better multitasking performance is achieved with ontap 7.3.x.</P></BODY></HTML> Tue, 16 Nov 2010 08:08:52 GMT https://community.netapp.com/t5/ONTAP-Discussions/Filer-high-CPU-cause-vcs-service-group-goes-down/m-p/30495#M7221 thomas_glodde 2010-11-16T08:08:52Z