Solved: CPU High Usage On my filer FAS2040 A-A

NAYABRASOOL · ‎2012-12-04

Hi Experts,

We are getting CPU usage high alerts on our filers very often , don't know the reason why can anybody help me to trace this out.

Thanks,

Nayab

radek_kubka · ‎2012-12-17

Hi Nayab,

Some of the links are within partner area, so I believe your NOW account is end-user one, hence you cannot access it.

You can find reallocate manual here though on p. 480:

https://library.netapp.com/ecm/ecm_get_file/ECMP1120818

When you run reallocate measure you should get an output at the end specifying the level of fragmentation. If you didn't notice that, the previous check is saved and may be viewed by running reallocate status, rather than running reallocate measure once again.

As per the doc quoted above:

"The threshold when a LUN, file or volume is considered unoptimized enough that a reallocation should be performed is given as a number from 3 (moderately optimized) to 10 (very unoptimized)."

regards,

Radek

View solution in original post

thomas_glodde · ‎2012-12-05

your filer generates performance stats in /vol/vol0/etc/stats/archive.

you can analyze these with CMPG from tech.netapp.com and see what causes the high cpu.

besides that, high i/o (backup) causes high cpu or deduplication, there might just be serveral regular scheduled processes which will cause these alerts.

NAYABRASOOL · ‎2012-12-05

It says my company doesnt have the Technet Access What do i need for this case open a case to Netapp , As i think this is nt technical related

radek_kubka · ‎2012-12-05

If you roughly know when these CPU spikes occur, try running "sysstat -x -i 2" from CLI (for few minutes) & post the output here.

NAYABRASOOL · ‎2012-12-06

Sure i will run this command next time wen i see the CPU high usage and paste it here

NAYABRASOOL · ‎2012-12-10

Hi Radek,

Please find the output of sysstat ran while the CPU high usage. Please examine and come back with your findings.

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2012.12.11 04:35:52 =~=~=~=~=~=~=~=~=~=~=~=

sysstat -x -i 2

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s

in out read write read write age hit time ty util in out in out

94% 0 0 0 833 29487 163131 169393 49187 0 0 4s 79% 98% Fv 67% 0 833 0 0 27400 154742

95% 0 0 0 851 32369 174080 181847 29641 0 0 21s 81% 55% Ff 46% 0 851 0 0 29985 167789

90% 0 0 0 790 27104 175319 175024 43761 0 0 4s 81% 53% Ff 60% 0 790 0 0 24924 166186

84% 0 0 0 846 26189 185390 190786 10048 0 0 20s 74% 19% Fn 57% 0 846 0 0 23994 175403

94% 0 0 0 819 34633 169097 176722 46230 0 0 21s 79% 96% Fn 74% 0 819 0 0 32189 166085

91% 0 0 0 827 35876 160273 167275 57606 0 0 4s 80% 72% Fs 58% 0 827 0 0 33600 153043

87% 0 0 0 728 36142 156081 155181 37494 0 0 4s 80% 45% Fs 57% 0 728 0 0 33816 148504

92% 0 0 0 742 33521 155506 164628 34876 0 0 4s 77% 60% Fs 77% 0 742 0 0 31043 149979

88% 0 0 0 802 36463 159385 168933 35658 0 0 23s 77% 69% Ff 69% 0 802 0 0 34153 149001

94% 0 0 0 850 39564 160872 173439 48747 0 0 24s 78% 88% Ff 58% 0 850 0 0 36953 153869

89% 0 0 0 813 33763 153363 163810 51213 0 0 24s 95% 71% F 58% 0 813 0 0 31484 147822

90% 0 0 0 737 31241 153209 178475 38918 0 0 26s 74% 51% F 74% 0 737 0 0 29001 143568

92% 0 0 0 832 32811 156456 189095 33544 0 0 4s 72% 55% Ff 71% 0 832 0 0 30544 151044

90% 0 0 0 919 29704 155161 191770 33898 0 0 4s 69% 56% Ff 90% 0 919 0 0 27592 146584

88% 0 0 0 889 28365 166607 192448 19928 0 0 24s 72% 49% Fs 81% 0 889 0 0 26306 159095

86% 0 0 0 889 24806 180436 192052 30809 0 0 24s 78% 60% : 66% 0 889 0 0 22770 173101

89% 0 0 0 889 26213 177119 199654 25448 0 0 24s 75% 68% Ff 62% 0 889 0 0 24048 168977

88% 0 0 0 767 27308 163938 176072 42767 0 0 4s 79% 80% Ff 53% 0 767 0 0 25247 161691

75% 0 0 0 837 22196 193263 182537 8279 0 0 3s 76% 19% : 33% 0 837 0 0 20180 184292

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s

in out read write read write age hit time ty util in out in out

87% 0 0 0 800 25309 184739 182144 37285 0 0 3s 79% 98% Ff 44% 0 800 0 0 23023 174258

85% 0 0 0 921 16065 189620 200236 26556 0 0 2s 72% 65% Ff 53% 0 921 0 0 14234 179540

82% 0 0 0 881 18340 188290 203049 18173 0 0 29s 78% 62% : 63% 0 881 0 0 16473 180674

84% 0 0 0 801 23956 183288 180842 23364 0 0 30s 78% 59% Ff 48% 0 801 0 0 21922 174959

90% 0 0 0 821 31642 164519 177063 39262 0 0 29s 76% 90% Ff 50% 0 821 0 0 29309 158726

90% 0 0 0 743 35326 157521 151368 56048 0 0 2s 80% 84% Ff 55% 0 743 0 0 33035 148262

89% 0 0 0 840 34210 178018 176172 30776 0 0 2s 78% 52% Ff 42% 0 840 0 0 31712 170302

92% 0 0 0 858 37348 159177 176004 50516 0 0 28s 77% 88% Ff 51% 0 858 0 0 34858 154288

88% 0 0 0 828 28140 178836 184174 28408 0 0 2s 78% 50% Ff 62% 0 828 0 0 25925 169245

91% 0 0 0 883 31674 161892 188590 36074 0 0 29s 71% 64% Fs 73% 0 883 0 0 29460 155008

92% 0 0 0 849 29564 163945 190967 28286 0 0 30s 74% 60% Fs 82% 0 849 0 0 27434 155345

89% 0 0 0 924 20700 176983 206250 30056 0 0 28s 69% 65% : 88% 0 924 0 0 18899 167445

81% 0 0 0 900 15719 160016 199039 13944 0 0 2s 66% 28% Ff 85% 0 900 0 0 14271 152456

83% 0 0 0 925 17613 168362 200204 23970 0 0 28s 68% 99% : 88% 0 925 0 0 16022 161162

83% 0 0 0 907 15710 173101 200745 15195 0 0 27s 68% 21% Ff 83% 0 907 0 0 14147 167292

87% 0 0 0 935 25650 173438 202735 22536 0 0 26s 70% 86% : 88% 0 935 0 0 23661 163762

89% 0 0 0 896 24912 170308 194223 26021 0 0 26s 72% 77% Ff 81% 0 896 0 0 22965 164970

87% 0 0 0 879 18926 178244 199172 22980 0 0 24s 74% 66% Ff 85% 0 879 0 0 17254 170451

88% 0 0 0 956 23377 176897 207166 26955 0 0 2s 70% 100% :v 84% 0 956 0 0 21571 169426

86% 0 0 0 871 18881 184656 199318 22249 0 0 21s 75% 67% Ff 68% 0 871 0 0 16880 177667

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s

in out read write read write age hit time ty util in out in out

81% 0 0 0 854 14997 195666 200320 16440 0 0 21s 77% 58% : 67% 0 854 0 0 13193 188974

85% 0 0 0 867 14996 183728 196143 16131 0 0 22s 77% 38% Ff 66% 0 867 0 0 13236 176138

82% 0 0 0 791 19025 192277 181796 22885 0 0 21s 80% 82% : 61% 0 791 0 0 17188 180339

82% 0 0 0 827 16781 173162 196941 12429 0 0 21s 75% 36% Fs 59% 0 827 0 0 15148 168916

87% 0 0 0 871 21585 186578 186979 30439 0 0 4s 78% 93% : 61% 0 871 0 0 19583 179288

88% 0 0 0 807 16823 181026 187694 24950 0 0 4s 76% 71% Ff 78% 0 807 0 0 15006 169878

84% 0 0 0 943 23454 179213 200503 17040 0 0 21s 71% 57% : 81% 0 943 0 0 21532 173574

86% 0 0 0 825 17159 173218 188563 32599 0 0 21s 76% 93% Ff 79% 0 825 0 0 15496 164470

86% 0 0 0 881 25994 171880 189612 22633 0 0 21s 73% 64% Ff 74% 0 881 0 0 23970 162964

86% 0 0 0 960 23653 175455 203080 24710 0 0 21s 71% 74% : 89% 0 960 0 0 21676 169204

90% 0 0 0 936 24107 168213 200169 34476 0 0 21s 69% 92% Ff 80% 0 936 0 0 22198 161771

87% 0 0 0 781 27269 172103 168688 41529 0 0 21s 76% 75% Ff 61% 0 781 0 0 25207 163722

84% 0 0 0 888 21901 184556 197913 5357 0 0 21s 73% 20% Fs 68% 0 888 0 0 19899 175918

85% 0 0 0 845 25202 175205 191620 33163 0 0 4s 74% 100% :v 68% 0 845 0 0 23207 166193

86% 0 0 0 900 20851 183182 197532 25796 0 0 4s 73% 64% Ff 61% 0 900 0 0 18956 177661

79% 0 0 0 861 19892 195889 196256 12024 0 0 20s 76% 45% : 48% 0 861 0 0 17982 189402

85% 0 0 0 847 24883 179135 180048 34832 0 0 21s 77% 96% Ff 44% 0 847 0 0 22810 171582

88% 0 0 0 767 20801 173437 177765 42040 0 0 21s 79% 82% Ff 58% 0 767 0 0 18923 165477

84% 0 0 0 837 25288 188784 183249 12716 0 0 4s 78% 21% Fn 45% 0 837 0 0 23089 179012

88% 0 0 0 771 32656 175075 173646 32197 0 0 4s 78% 65% Fn 47% 0 771 0 0 30298 167672

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s

in out read write read write age hit time ty util in out in out

84% 0 0 0 798 19982 179820 183013 41545 0 0 22s 76% 64% : 55% 0 798 0 0 18144 171061

89% 0 0 0 981 30300 171567 195885 22465 0 0 22s 72% 52% Ff 75% 0 981 0 0 28174 163917

90% 0 0 0 818 27002 164417 175687 50167 0 0 22s 76% 94% Ff 64% 0 818 0 0 24938 162364

75% 0 0 0 822 25021 188326 173765 3229 0 0 23s 78% 11% Fn 49% 0 822 0 0 22919 175682

90% 0 0 0 768 24742 172555 178900 38144 0 0 4s 77% 100% :f 60% 0 768 0 0 22687 164579

85% 0 0 0 751 30570 164012 162120 32831 0 0 3s 79% 82% Ff 50% 0 751 0 0 28327 160337

87% 0 0 0 833 23559 173984 180072 48204 0 0 3s 76% 63% Ff 63% 0 833 0 0 21520 162849

80% 0 0 0 893 21172 189017 196848 5765 0 0 2s 78% 12% : 61% 0 893 0 0 19159 179434

91% 0 0 0 840 26210 169094 195946 33677 0 0 30s 72% 89% Ff 75% 0 840 0 0 24131 165641

90% 0 0 0 818 25326 179536 180284 29691 0 0 2s 80% 79% Ff 69% 0 818 0 0 23271 168908

85% 0 0 0 884 20194 183964 192285 12581 0 0 30s 73% 63% Fn 87% 0 884 0 0 18333 175926

90% 0 0 0 862 22770 172605 180398 38574 0 0 30s 73% 100% :f 81% 0 862 0 0 20850 165757

92% 0 0 0 814 39067 165311 165196 41429 0 0 31s 81% 78% F 72% 0 814 0 0 36499 159260

91% 0 0 0 820 33661 176880 171316 37991 0 0 32s 81% 59% F 77% 0 820 0 0 31215 164836

87% 0 0 0 827 15323 181656 190416 37961 0 0 2s 77% 63% F 81% 0 827 0 0 13564 176306

83% 0 0 0 921 18241 202275 197463 16 0 0 32s 75% 0% - 79% 0 921 0 0 16257 192366

86% 0 0 0 875 19226 184037 195319 19421 0 0 31s 76% 62% Ff 77% 0 875 0 0 17355 177111

81% 0 0 0 829 17461 192725 193249 18238 0 0 31s 76% 75% : 63% 0 829 0 0 15621 183670

87% 0 0 0 818 19408 177697 184636 25913 0 0 31s 79% 65% Ff 71% 0 818 0 0 17489 173005

82% 0 0 0 880 17456 194789 193656 13438 0 0 31s 76% 61% : 66% 0 880 0 0 15526 186439

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s

in out read write read write age hit time ty util in out in out

83% 0 0 0 767 13197 181597 176992 25083 0 0 31s 79% 61% Ff 73% 0 767 0 0 11689 168407

86% 0 0 0 918 19129 182517 201161 17219 0 0 30s 74% 66% : 82% 0 918 0 0 17335 179779

84% 0 0 0 826 15252 183049 196582 13543 0 0 28s 73% 39% Fs 72% 0 826 0 0 13616 174730

87% 0 0 0 872 23593 184976 193461 29237 0 0 27s 76% 92% : 74% 0 872 0 0 21495 175298

87% 0 0 0 877 24340 177082 191906 32517 0 0 26s 73% 83% Ff 59% 0 877 0 0 22243 171684

83% 0 0 0 807 22150 187380 183418 20102 0 0 25s 76% 53% Ff 41% 0 807 0 0 20130 176834

85% 0 0 0 915 27308 176052 187532 24440 0 0 24s 76% 80% Fn 54% 0 915 0 0 25264 167940

88% 0 0 0 754 17994 167813 184229 37596 0 0 23s 78% 33% : 71% 0 754 0 0 16275 163808

84% 0 0 0 836 21592 183280 188976 16192 0 0 23s 78% 39% Fs 56% 0 836 0 0 19580 173128

87% 0 0 0 905 26434 181813 196398 27346 0 0 22s 76% 96% : 62% 0 905 0 0 24266 174635

82% 0 0 0 799 14414 186416 188784 30540 0 0 4s 78% 91% Ff 55% 0 799 0 0 12726 177814

79% 0 0 0 898 16763 193423 204986 7247 0 0 4s 77% 24% : 45% 0 898 0 0 14886 185852

86% 0 0 0 805 16273 186486 191531 19591 0 0 22s 78% 74% Ff 59% 0 805 0 0 14489 179463

82% 0 0 0 879 21642 191245 191344 21061 0 0 22s 77% 75% : 53% 0 879 0 0 19590 182097

84% 0 0 0 819 13437 185834 189193 30322 0 0 22s 79% 87% Ff 57% 0 819 0 0 11767 178732

80% 0 0 0 814 18796 199616 190956 8763 0 0 4s 80% 28% : 46% 0 814 0 0 16921 190973

84% 0 0 0 795 18644 167421 179924 28211 0 0 24s 78% 81% Ff 68% 0 795 0 0 16824 161397

81% 0 0 0 807 20091 193973 193045 12042 0 0 24s 81% 44% : 64% 0 807 0 0 18100 184072

87% 0 0 0 823 21819 181742 188895 28941 0 0 24s 79% 86% Ff 72% 0 823 0 0 19814 172772

89% 0 0 0 831 25890 175304 178655 37471 0 0 25s 80% 81% Ff 73% 0 831 0 0 23814 168065

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s

in out read write read write age hit time ty util in out in out

82% 0 0 0 823 18925 193704 191309 13010 0 0 25s 77% 56% : 69% 0 823 0 0 17011 184149

88% 0 0 0 870 30881 170619 173407 31817 0 0 25s 80% 81% Ff 70% 0 870 0 0 28689 163628

85% 0 0 0 749 21284 170047 173757 40782 0 0 26s 79% 80% Ff 71% 0 749 0 0 19427 162471

85% 0 0 0 859 23649 186026 189312 12500 0 0 4s 81% 35% Fs 69% 0 859 0 0 21597 180306

84% 0 0 0 842 22648 192688 184712 28976 0 0 25s 79% 100% :v 61% 0 842 0 0 20534 183636

87% 0 0 0 802 24264 182101 182038 28218 0 0 26s 81% 68% Ff 74% 0 802 0 0 22192 173628

84% 0 0 0 901 20371 192771 190167 11068 0 0 4s 80% 48% Fn 80% 0 901 0 0 18414 186027

92% 0 0 0 764 18884 176920 183443 38519 0 0 27s 80% 98% : 88% 0 764 0 0 17071 170033

84% 0 0 0 825 18148 196403 193556 4922 0 0 27s 81% 15% Fs 73% 0 825 0 0 16220 187327

88% 0 0 0 815 19177 187406 187996 30802 0 0 27s 82% 100% :f 70% 0 815 0 0 17295 180523

89% 0 0 0 767 26055 183221 176940 26357 0 0 3s 81% 78% Ff 68% 0 767 0 0 23860 171445

88% 0 0 0 851 20917 184462 188274 23756 0 0 2s 79% 84% : 75% 0 851 0 0 18964 178240

86% 0 0 0 766 21210 182773 174552 41666 0 0 2s 80% 68% F 61% 0 766 0 0 19289 174251

88% 0 0 0 845 26713 183023 187870 25276 0 0 34s 81% 46% Ff 67% 0 845 0 0 24558 176519

90% 0 0 0 811 29641 178850 175486 31514 0 0 2s 82% 82% Ff 70% 0 811 0 0 27357 168555

88% 0 0 0 858 22592 184809 191006 19398 0 0 34s 77% 43% : 85% 0 858 0 0 20433 178754

89% 0 0 0 846 27752 175803 174536 36638 0 0 34s 82% 83% Ff 67% 0 846 0 0 25508 166298

86% 0 0 0 805 16477 186967 191742 18500 0 0 2s 79% 61% Ff 76% 0 805 0 0 14599 178758

85% 0 0 0 819 23469 190576 188440 21521 0 0 35s 80% 60% : 65% 0 819 0 0 21311 183667

89% 0 0 0 764 17247 176901 186320 36641 0 0 35s 81% 74% Ff 67% 0 764 0 0 15381 170283

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s

in out read write read write age hit time ty util in out in out

81% 0 0 0 849 23373 198727 195154 1305 0 0 35s 80% 12% : 50% 0 849 0 0 21166 189826

SVUSFAS01> priv set diag

Warning: These diagnostic commands are for use by NetApp

personnel only.

SVUSFAS01*>

SVUSFAS01*> sysstat -M 1

ANY1+ ANY2+ AVG CPU0 CPU1 Network Storage Raid Target Kahuna WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP

93% 77% 86% 88% 83% 43% 29% 17% 9% 64% 0% 0% 0% 3% 6% 0% 925 26%

93% 78% 86% 89% 83% 45% 28% 18% 10% 59% 0% 0% 0% 6% 7% 0% 812 100%

91% 74% 83% 88% 79% 45% 29% 17% 10% 54% 0% 0% 0% 6% 7% 0% 796 100%

92% 78% 86% 89% 84% 42% 29% 18% 9% 64% 0% 0% 0% 4% 7% 0% 824 36%

97% 88% 93% 94% 93% 42% 32% 21% 9% 71% 0% 0% 0% 6% 6% 0% 759 100%

88% 71% 81% 85% 76% 43% 29% 17% 9% 51% 0% 0% 0% 4% 7% 0% 852 77%

89% 72% 81% 86% 76% 44% 32% 16% 9% 54% 0% 0% 0% 0% 8% 0% 829 0%

87% 68% 79% 83% 74% 43% 29% 15% 9% 54% 0% 0% 0% 0% 8% 0% 901 4%

93% 80% 87% 90% 85% 38% 31% 22% 8% 62% 0% 0% 0% 7% 6% 0% 800 100%

91% 75% 84% 87% 81% 41% 31% 20% 9% 56% 0% 0% 0% 5% 7% 0% 780 100%

89% 72% 81% 85% 77% 40% 30% 18% 9% 55% 0% 0% 0% 4% 7% 0% 824 77%

86% 67% 78% 83% 73% 43% 31% 16% 9% 50% 0% 0% 0% 0% 7% 0% 815 0%

87% 67% 78% 82% 73% 43% 31% 16% 9% 50% 0% 0% 0% 0% 7% 0% 826 0%

86% 67% 78% 82% 73% 42% 31% 16% 9% 50% 0% 0% 0% 0% 7% 0% 832 0%

83% 61% 73% 79% 68% 41% 28% 14% 8% 48% 0% 0% 0% 0% 7% 0% 739 0%

86% 68% 78% 82% 74% 41% 31% 17% 9% 51% 0% 0% 0% 0% 7% 0% 852 0%

91% 75% 84% 87% 81% 40% 31% 18% 9% 61% 0% 0% 0% 2% 7% 0% 856 19%

94% 82% 89% 92% 86% 40% 32% 23% 9% 58% 0% 0% 0% 9% 7% 0% 705 100%

90% 73% 82% 87% 77% 45% 28% 17% 9% 52% 0% 0% 0% 5% 7% 0% 794 100%

ANY1+ ANY2+ AVG CPU0 CPU1 Network Storage Raid Target Kahuna WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP

92% 77% 85% 89% 82% 43% 34% 18% 9% 57% 0% 0% 0% 0% 8% 0% 916 12%

96% 86% 92% 94% 90% 41% 30% 22% 9% 67% 0% 0% 0% 9% 6% 0% 853 75%

91% 76% 84% 88% 80% 44% 32% 19% 9% 54% 0% 0% 0% 5% 7% 0% 813 100%

89% 72% 81% 85% 77% 44% 31% 15% 9% 56% 0% 0% 0% 0% 7% 0% 846 13%

88% 71% 80% 85% 75% 44% 31% 16% 9% 53% 0% 0% 0% 0% 7% 0% 841 0%

94% 82% 89% 91% 86% 41% 30% 20% 9% 65% 0% 0% 0% 6% 6% 0% 757 91%

93% 79% 87% 91% 83% 45% 32% 20% 10% 54% 0% 0% 0% 6% 7% 0% 838 100%

89% 71% 81% 87% 76% 46% 28% 16% 9% 53% 0% 0% 0% 3% 8% 0% 792 74%

89% 73% 82% 85% 79% 41% 28% 16% 8% 62% 0% 0% 0% 2% 7% 0% 785 27%

90% 75% 83% 88% 79% 39% 33% 20% 9% 53% 0% 0% 0% 5% 7% 0% 921 100%

91% 78% 85% 89% 82% 44% 31% 19% 10% 55% 0% 0% 0% 6% 7% 0% 798 100%

90% 75% 84% 88% 80% 45% 32% 17% 10% 55% 0% 0% 0% 2% 7% 0% 853 51%

94% 80% 87% 89% 86% 41% 30% 20% 9% 63% 0% 0% 0% 5% 7% 0% 785 51%

93% 80% 87% 91% 84% 44% 32% 21% 9% 56% 0% 0% 0% 5% 7% 0% 795 100%

90% 73% 83% 87% 79% 45% 30% 18% 9% 53% 0% 0% 0% 3% 7% 0% 826 75%

88% 70% 80% 85% 76% 46% 30% 15% 10% 53% 0% 0% 0% 0% 7% 0% 847 0%

91% 74% 84% 87% 81% 41% 27% 18% 9% 60% 0% 0% 0% 6% 6% 0% 748 57%

90% 74% 83% 87% 79% 43% 30% 18% 10% 53% 0% 0% 0% 5% 7% 0% 851 100%

89% 71% 81% 86% 77% 44% 29% 16% 9% 53% 0% 0% 0% 4% 7% 0% 894 74%

88% 71% 81% 84% 77% 43% 28% 15% 9% 59% 0% 0% 0% 1% 7% 0% 806 14%

ANY1+ ANY2+ AVG CPU0 CPU1 Network Storage Raid Target Kahuna WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP

95% 85% 91% 92% 89% 42% 32% 22% 9% 63% 0% 0% 0% 8% 6% 0% 873 100%

90% 73% 82% 86% 78% 43% 31% 18% 9% 52% 0% 0% 0% 5% 7% 0% 786 97%

90% 74% 83% 86% 80% 42% 33% 17% 9% 58% 0% 0% 0% 0% 7% 0% 1056 0%

96% 87% 92% 92% 92% 36% 32% 21% 8% 75% 0% 0% 0% 6% 6% 0% 779 86%

93% 48% 71% 91% 50% 34% 26% 16% 7% 48% 0% 0% 0% 5% 5% 0% 825 100%

92% 76% 85% 89% 80% 42% 34% 19% 9% 55% 0% 0% 0% 3% 7% 0% 935 70%

90% 73% 82% 86% 79% 43% 33% 16% 10% 56% 0% 0% 0% 0% 7% 0% 968 0%

92% 77% 85% 87% 83% 39% 32% 19% 8% 63% 0% 0% 0% 2% 7% 0% 883 35%

92% 77% 85% 88% 83% 38% 35% 21% 9% 55% 0% 0% 0% 5% 7% 0% 852 100%

84% 65% 76% 80% 71% 37% 31% 19% 8% 44% 0% 0% 0% 5% 7% 0% 837 100%

89% 71% 81% 85% 76% 42% 32% 17% 9% 52% 0% 0% 0% 2% 7% 0% 849 45%

89% 72% 82% 86% 78% 44% 32% 16% 10% 54% 0% 0% 0% 0% 7% 0% 1008 0%

87% 69% 79% 83% 74% 42% 31% 16% 9% 53% 0% 0% 0% 0% 7% 0% 859 0%

92% 79% 86% 89% 83% 39% 35% 19% 8% 61% 0% 0% 0% 2% 7% 0% 836 17%

93% 81% 88% 90% 86% 37% 33% 23% 8% 60% 0% 0% 0% 7% 7% 0% 687 100%

92% 78% 86% 88% 83% 39% 33% 20% 8% 59% 0% 0% 0% 5% 8% 0% 735 100%

91% 77% 85% 88% 82% 41% 37% 20% 9% 54% 0% 0% 0% 0% 8% 0% 811 91%

89% 74% 82% 87% 78% 43% 33% 18% 9% 54% 0% 0% 0% 0% 8% 0% 801 0%

92% 77% 85% 89% 82% 42% 35% 19% 9% 58% 0% 0% 0% 0% 8% 0% 867 0%

94% 82% 89% 90% 88% 35% 33% 21% 8% 69% 0% 0% 0% 5% 7% 0% 766 43%

ANY1+ ANY2+ AVG CPU0 CPU1 Network Storage Raid Target Kahuna WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP

90% 74% 83% 86% 80% 39% 33% 22% 8% 51% 0% 0% 0% 5% 8% 0% 751 100%

95% 83% 89% 89% 89% 38% 32% 19% 9% 69% 0% 0% 0% 4% 6% 0% 864 96%

87% 66% 78% 81% 74% 40% 31% 16% 9% 52% 0% 0% 0% 0% 7% 0% 893 0%

87% 68% 78% 82% 75% 40% 33% 17% 9% 51% 0% 0% 0% 0% 7% 0% 880 0%

81% 58% 71% 76% 66% 40% 27% 14% 8% 46% 0% 0% 0% 0% 7% 0% 750 0%

93% 79% 87% 88% 85% 37% 34% 20% 8% 64% 0% 0% 0% 4% 6% 0% 944 63%

92% 77% 85% 88% 81% 41% 34% 22% 9% 52% 0% 0% 0% 5% 7% 0% 823 100%

88% 70% 80% 83% 76% 40% 31% 19% 9% 50% 0% 0% 0% 5% 7% 0% 806 100%

88% 70% 80% 84% 75% 40% 32% 18% 9% 51% 0% 0% 0% 2% 7% 0% 874 53%

88% 73% 82% 84% 79% 40% 34% 18% 9% 54% 0% 0% 0% 0% 8% 0% 819 0%

89% 74% 83% 85% 80% 37% 36% 20% 9% 56% 0% 0% 0% 0% 8% 0% 877 0%

90% 74% 83% 86% 80% 40% 35% 19% 9% 54% 0% 0% 0% 0% 8% 0% 903 0%

93% 79% 87% 88% 85% 35% 33% 22% 8% 62% 0% 0% 0% 6% 7% 0% 807 85%

93% 80% 87% 88% 87% 35% 35% 23% 8% 61% 0% 0% 0% 5% 7% 0% 843 100%

94% 82% 89% 91% 87% 40% 36% 21% 9% 60% 0% 0% 0% 3% 8% 0% 870 96%

87% 70% 80% 82% 78% 36% 34% 19% 8% 53% 0% 0% 0% 0% 8% 0% 892 0%

85% 66% 77% 79% 74% 35% 33% 18% 8% 51% 0% 0% 0% 0% 8% 0% 776 0%

88% 71% 80% 83% 78% 37% 35% 19% 8% 53% 0% 0% 0% 0% 8% 0% 821 0%

96% 87% 92% 93% 91% 34% 36% 28% 8% 63% 0% 0% 0% 9% 7% 0% 766 70%

94% 82% 89% 89% 89% 41% 36% 22% 9% 58% 0% 0% 0% 5% 8% 0% 837 100%

ANY1+ ANY2+ AVG CPU0 CPU1 Network Storage Raid Target Kahuna WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP

91% 77% 85% 87% 82% 40% 36% 20% 9% 58% 0% 0% 0% 0% 8% 0% 935 43%

88% 71% 80% 83% 78% 37% 34% 19% 8% 54% 0% 0% 0% 0% 8% 0% 891 0%

84% 62% 74% 75% 73% 32% 32% 18% 7% 50% 0% 0% 0% 0% 8% 0% 911 0%

90% 73% 83% 83% 83% 30% 32% 23% 7% 60% 0% 0% 0% 7% 7% 0% 678 81%

87% 68% 79% 79% 79% 33% 34% 23% 8% 48% 0% 0% 0% 5% 8% 0% 840 100%

89% 73% 82% 82% 82% 36% 36% 22% 8% 52% 0% 0% 0% 2% 8% 0% 831 100%

85% 66% 77% 78% 75% 32% 35% 20% 7% 51% 0% 0% 0% 0% 8% 0% 764 70%

82% 61% 73% 73% 73% 35% 32% 18% 7% 46% 0% 0% 0% 0% 8% 0% 811 0%

95% 84% 90% 91% 89% 36% 36% 23% 8% 65% 0% 0% 0% 5% 7% 0% 747 52%

94% 82% 88% 88% 88% 39% 38% 24% 9% 54% 0% 0% 0% 5% 8% 0% 799 100%

91% 76% 85% 87% 82% 38% 36% 23% 9% 52% 0% 0% 0% 5% 8% 0% 858 100%

88% 72% 81% 84% 78% 39% 32% 19% 9% 55% 0% 0% 0% 0% 8% 0% 844 79%

91% 75% 84% 87% 81% 42% 33% 18% 9% 57% 0% 0% 0% 0% 8% 0% 826 0%

96% 85% 91% 92% 91% 38% 36% 21% 8% 67% 0% 0% 0% 5% 7% 0% 717 72%

98% 89% 94% 94% 94% 37% 45% 23% 8% 62% 0% 0% 0% 5% 7% 0% 859 100%

99% 86% 93% 90% 95% 41% 41% 20% 9% 63% 0% 0% 0% 4% 7% 0% 795 100%

98% 87% 93% 93% 94% 40% 45% 20% 9% 65% 0% 0% 0% 0% 8% 0% 806 20%

93% 79% 87% 87% 87% 44% 36% 19% 9% 59% 0% 0% 0% 0% 8% 0% 899 0%

98% 92% 96% 95% 96% 36% 33% 25% 8% 75% 0% 0% 0% 9% 7% 0% 703 92%

93% 80% 87% 90% 84% 41% 34% 21% 9% 57% 0% 0% 0% 5% 7% 0% 781 100%

ANY1+ ANY2+ AVG CPU0 CPU1 Network Storage Raid Target Kahuna WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP

89% 72% 82% 85% 78% 41% 33% 18% 9% 54% 0% 0% 0% 0% 8% 0% 864 7%

85% 65% 76% 80% 72% 39% 31% 17% 8% 48% 0% 0% 0% 0% 8% 0% 830 0%

88% 71% 80% 83% 77% 38% 34% 18% 9% 53% 0% 0% 0% 0% 7% 0% 856 0%

84% 62% 74% 78% 71% 38% 30% 16% 8% 48% 0% 0% 0% 0% 7% 0% 924 0%

90% 73% 82% 85% 80% 40% 34% 18% 9% 57% 0% 0% 0% 0% 7% 0% 931 7%

94% 84% 90% 91% 88% 37% 33% 23% 9% 63% 0% 0% 0% 8% 6% 0% 829 100%

SVUSFAS01*>

SVUSFAS01*> priv set admin

SVUSFAS01> priv set admin

SVUSFAS01> sysstat -us 1

CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk

ops/s in out read write read write age hit time ty util

86% 867 19107 186295 195078 19297 0 0 19s 72% 39% F 57%

83% 827 17729 187506 198704 29556 0 0 20s 76% 100% : 60%

83% 823 23177 190773 185460 27432 0 0 20s 77% 100% : 63%

82% 947 15740 192689 216359 591 0 0 20s 72% 14% : 87%

89% 867 18941 178821 189363 34008 0 0 20s 74% 78% F 87%

89% 938 24991 182085 202228 26052 0 0 20s 70% 100% : 81%

85% 1008 18508 181986 217811 16187 0 0 4s 69% 63% : 82%

87% 885 17294 171457 203789 17312 0 0 19s 70% 36% F 80%

82% 913 12935 194753 191369 28428 0 0 4s 73% 100% : 86%

88% 1020 23285 181893 198912 29312 0 0 19s 70% 100% : 86%

82% 1060 15628 198290 206322 545 0 0 19s 69% 18% : 82%

90% 933 19624 161673 206713 16213 0 0 19s 70% 36% F 94%

80% 952 20927 171951 179936 24754 0 0 19s 73% 100% : 81%

86% 1057 23251 166049 212930 28551 0 0 19s 74% 100% : 80%

81% 1032 13071 179975 209598 9878 0 0 18s 73% 54% : 92%

88% 838 14085 167546 192108 26696 0 0 18s 73% 52% F 94%

81% 832 12097 181810 202371 29150 0 0 18s 72% 100% : 74%

81% 850 20434 190375 183796 21594 0 0 18s 76% 79% : 66%

81% 1065 17122 189364 218952 0 0 0 4s 71% 0% - 68%

CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk

ops/s in out read write read write age hit time ty util

89% 915 17420 170598 189429 30906 0 0 19s 79% 45% F 74%

86% 743 12210 185872 177484 27840 0 0 4s 80% 100% : 83%

84% 826 21870 190831 188218 19011 0 0 4s 77% 66% : 71%

81% 853 15935 195777 200663 20 0 0 20s 73% 0% - 76%

84% 801 20914 190389 183839 9843 0 0 20s 79% 20% F 66%

89% 767 9843 184961 194929 29674 0 0 20s 80% 100% : 75%

83% 905 23716 196546 177070 26659 0 0 20s 81% 100% : 65%

80% 903 15126 195730 195572 16832 0 0 21s 79% 65% : 63%

80% 936 14738 197238 218238 0 0 0 20s 72% 0% - 66%

89% 890 17288 172216 186941 42026 0 0 4s 80% 81% F 78%

90% 875 12066 172453 198095 29388 0 0 3s 77% 100% : 89%

76% 825 23177 168075 183616 10352 0 0 3s 76% 47% : 71%

78% 927 13133 190172 210309 32 0 0 2s 75% 0% - 76%

83% 763 14392 192691 178810 25214 0 0 2s 81% 59% F 53%

88% 845 20078 188403 194720 30064 0 0 2s 77% 100% : 82%

79% 885 15971 183346 200821 26713 0 0 27s 78% 100% : 63%

80% 891 11787 195154 207612 6295 0 0 27s 78% 28% : 76%

76% 797 14314 203478 187311 8 0 0 28s 81% 0% - 56%

89% 745 18547 175438 170941 50551 0 0 28s 76% 90% F 65%

82% 884 24638 188029 175356 25788 0 0 28s 79% 100% : 58%

CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk

ops/s in out read write read write age hit time ty util

77% 881 10282 209941 211940 12 0 0 28s 81% 0% - 57%

78% 848 17287 201761 188587 24 0 0 29s 79% 0% - 63%

82% 694 8100 188552 171278 39754 0 0 2s 79% 74% F 62%

77% 815 8982 193877 197471 28876 0 0 29s 76% 100% : 60%

78% 831 11545 202374 204072 8562 0 0 30s 81% 46% : 55%

77% 902 12679 205419 207445 0 0 0 30s 78% 0% - 49%

73% 832 10918 203135 190773 0 0 0 2s 79% 0% - 39%

76% 809 10310 206160 202258 32 0 0 31s 79% 0% - 55%

75% 845 12663 210855 200724 0 0 0 31s 78% 0% - 53%

88% 789 13517 171193 187466 46397 0 0 31s 79% 97% F 62%

82% 804 13837 194625 184071 29055 0 0 31s 76% 100% : 87%

80% 796 17486 193438 178841 1003 0 0 31s 80% 29% : 93%

69% 776 7947 183766 172525 0 0 0 32s 70% 0% - 94%

75% 813 11955 174718 176877 24 0 0 31s 70% 0% - 100%

71% 683 7952 140887 157392 15452 0 0 32s 65% 44% F 100%

83% 860 19955 158896 165064 32268 0 0 2s 67% 100% : 100%

79% 782 18617 156726 156703 22837 0 0 32s 66% 100% : 100%

73% 763 16225 161897 169972 6864 0 0 31s 63% 97% : 97%

83% 807 19062 163780 173912 15436 0 0 31s 74% 46% F 95%

80% 769 11882 169147 181373 29162 0 0 30s 69% 100% : 89%

CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk

ops/s in out read write read write age hit time ty util

88% 1016 19624 175977 198867 28020 0 0 30s 71% 100% : 93%

76% 912 16421 182875 171581 4273 0 0 30s 76% 36% : 90%

89% 741 21258 172594 172569 26352 0 0 29s 73% 47% F 93%

88% 722 16612 177127 172812 36108 0 0 29s 71% 100% : 96%

89% 906 23189 183421 195178 13389 0 0 27s 76% 100% : 97%

83% 914 21610 191940 178963 99 0 0 26s 75% 32% : 96%

85% 773 14935 151645 166409 42090 0 0 27s 62% 89% F 98%

83% 870 16569 167205 187221 30014 0 0 26s 67% 100% : 98%

76% 757 14235 154981 173888 12624 0 0 26s 64% 100% : 100%

82% 931 17116 178098 207821 131 0 0 24s 73% 33% : 96%

91% 959 14677 173279 206699 22331 0 0 23s 65% 55% F 95%

--

Summary Statistics ( 70 samples 1 secs/sample)

CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk

ops/s in out read write read write age hit time ty util

Min

69% 683 7947 140887 156703 0 0 0 2s 62% 0% * 39%

Avg

82% 861 16322 182756 190607 18342 0 0 20s 74% 59% * 78%

Max

91% 1065 24991 210855 218952 50551 0 0 32s 81% 100% * 100%

SVUSFAS01>

radek_kubka · ‎2012-12-12

Hi,

I'd say your filer is just 'normally' busy - i.e. it is trying to cope with a lot of iSCSI traffic, mostly reads, but some writes at the same time as well.

NVRAM is almost constantly full, hence a lot of Ff-type consistency points, where NVRAM is flushed to disk.

No wonder CPU utilisation is high.

Regards,

Radek

NAYABRASOOL · ‎2012-12-12

What is F-Type CP's ??? and also could you please tell how to overcome this High Usage??

radek_kubka · ‎2012-12-12

F means NVRAM full, the second (lowercase) f means flushing NVRAM content to disk.

To lower CPU usage you need to hammer your filler less!

NAYABRASOOL · ‎2012-12-12

Hmmm Good One

So no other go other than this like changing some parameters will help me to overcome this ??

radek_kubka · ‎2012-12-12

Well, not much really.

As there are many reads from LUNs, I'd probably run reallocate measure on the volumes containing LUNs to see whether fragmentation is playing any role.

NAYABRASOOL · ‎2012-12-12

How i will be able to run the reallocation and how to check the fragmentation ??

Is the command

FAS01> reallocate start <vol-path> will do , will it have any impact if i run this command ???

radek_kubka · ‎2012-12-13

I actually provided the command to check the fragmentation:

reallocate measure <vol_path>

Before running actual reallocation (reallocate start) gave a good read about different caveats & parameters, e.g.:

https://communities.netapp.com/message/20969#20969

https://communities.netapp.com/message/84099#84099

https://fieldportal.netapp.com/home.aspx?oparams=68936

NAYABRASOOL · ‎2012-12-14

Hi Radek,

I was able to open only the first link rest of all showing as UNAUTH access

Thanks,

Nayab

NAYABRASOOL · ‎2012-12-17

Hi Radek,

I have run the reallocation But how i can see if the fragmentation is the cause for the CPU high usage will it give me some message else how i can trace it out.

Thanks,

Nayab

radek_kubka · ‎2012-12-17

Hi Nayab,

Some of the links are within partner area, so I believe your NOW account is end-user one, hence you cannot access it.

You can find reallocate manual here though on p. 480:

https://library.netapp.com/ecm/ecm_get_file/ECMP1120818

When you run reallocate measure you should get an output at the end specifying the level of fragmentation. If you didn't notice that, the previous check is saved and may be viewed by running reallocate status, rather than running reallocate measure once again.

As per the doc quoted above:

"The threshold when a LUN, file or volume is considered unoptimized enough that a reallocation should be performed is given as a number from 3 (moderately optimized) to 10 (very unoptimized)."

regards,

Radek

madden · ‎2012-12-17

Hi,

I agree with Radek; this system is simply near bottleneck.

Take a look at this snippet:

Summary Statistics ( 70 samples 1 secs/sample)

CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk

ops/s in out read write read write age hit time ty util

Min

69% 683 7947 140887 156703 0 0 0 2s 62% 0% * 39%

Avg

82% 861 16322 182756 190607 18342 0 0 20s 74% 59% * 78%

Max

91% 1065 24991 210855 218952 50551 0 0 32s 81% 100% * 100%

From the Average row:

82% CPU: Quite high. Checking the sysstat -M output it looks balanced around processing functions

183 MB/s reads + 16 MB/s writes = 199 MB/s network xfer: You are using at least 2 x GbE links. You might hit link bottleneck if traffic is not balanced.

861 IOPs: take the 199MB/s / 861 IOPs = 231 KB / op. So you are doing large IOs.

78% disk busy: Above 50% and you can already start to see some variability in latency. At 78% average the disks are quite busy; they are processing at least one xfer 78% of the time so likely some IOs are also waiting to be processed at times. If IOs are waiting at the disk layer these will eventually make IOs wait on the host side.

Disk IO efficiency: Disk read is 190MB/s, network out is 182 MB/s which is roughly balanced. Disk write is 18MB/s, network in is 16MB/s which is roughly balanced; the extra little bit is parity. You have no evidence of freespace fragmentation.

Reallocate is useful if you have freespace fragmentation (you don't) or if you have sequential read after random write (you seem to be doing large block IO so I doubt it). Even then, it is has a bigger impact on reducing disk busy than cpu busy. Checking your disk IO efficiency however I'd say it looks pretty good; network in/out matches disk in/out.

So across the main system aspects (CPU, Disk IO, Network) on each you are nearing bottleneck. The solution is to stay as-is and be satisfied with latency you have, reduce workload (or move to the other head if it is less busy) and see latency improve, or buy a more powerful controller.

Hope that helps!

Christopher Madden

Storage Architect, NetApp EMEA

NAYABRASOOL · ‎2012-12-17

Hi Radek,

Thanks a lot for making me to understand things better hope will run the reallocate and will see for the optimization value.

Madden,

Thank you so much for your crystal clear explanation about the busy stats from the output above. We are looking forward to buy a High end filer and migrate all our stuff to the new one .

And also I am confused whose response i need to mark as CORRECT ANSWER so that it will be helpful for others .

Thanks,

Nayab

radek_kubka · ‎2012-12-17

Fragmentation / reallocation is a bit of a side-tracking really, as it is fairly obvious your system is under heavy load - so I suggest you mark one of the answers mentioning this as correct.

(I actually think when people search communities, they tend to read a whole thread anyway, rather than looking only at the "correct" answer without its context)