ONTAP Discussions
- Home
- :
- ONTAP, AFF, and FAS
- :
- ONTAP Discussions
- :
- Identify what process causes high cpu utilization
ONTAP Discussions
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I am facing high CPU utilization issue with multiple NetApp filers running on 7- Mode, i tried to identify the cause of high CPU utilization but its not getting clear to me.
Below are the sample out put of sysstat -x 1, sysstat -m 1 and sysstat -M 1, and statit capture and try to identify what process is causing high CPU utilization.
sysstat -x 1
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
96% 0 0 0 2569 55126 34452 167972 152950 0 0 7s 92% 99% Bn 96% 0 0 2569 0 0 52513 32921
95% 0 0 0 4858 110991 67370 219428 174513 0 0 9s 88% 100% :s 99% 6 0 4852 0 0 105656 64185
79% 0 0 0 3285 47090 73832 190141 93772 0 0 4s 86% 100% :f 75% 0 0 3285 0 0 44455 72380
91% 0 0 0 1840 24627 29797 210990 293491 0 0 4s 94% 99% Bs 83% 0 0 1840 0 0 23294 29005
99% 0 0 0 3424 147063 17753 159145 260516 0 0 6s 91% 98% Hn 93% 0 0 3424 0 0 140604 14158
99% 0 0 0 3610 95470 17616 167211 188118 0 0 6s 92% 100% :s 89% 0 0 3610 0 0 91313 15192
89% 0 0 0 4446 98371 58813 177714 118398 0 0 8s 92% 100% :v 70% 5 0 4441 0 0 93805 56098
98% 0 0 0 2045 67803 16601 163976 367372 0 0 8s 93% 99% Bs 91% 0 0 2045 0 0 65370 14781
96% 0 0 0 4858 131156 21542 132989 133849 0 0 3s 89% 98% Hn 79% 0 0 4858 0 0 124643 18160
99% 0 0 0 4121 90312 32774 190865 156729 0 0 5s 92% 100% :n 100% 0 0 4121 0 0 86247 30353
83% 0 0 0 3821 93764 20924 128495 137691 0 0 5s 89% 100% :s 84% 0 0 3821 0 0 89439 18351
75% 0 0 0 1755 27946 12668 164329 208325 0 0 7s 81% 99% Bn 62% 5 0 1750 0 0 26564 11839
97% 0 0 0 4205 119386 7600 144720 327178 0 0 7s 93% 100% :f 85% 0 0 4205 0 0 114179 4545
99% 0 0 0 3279 83809 4598 172910 225482 0 0 7s 90% 99% Hs 99% 0 0 3279 0 0 80071 2515
97% 0 0 0 7285 100670 9977 115014 160170 0 0 7s 88% 100% :f 74% 0 0 7285 0 0 96324 7142
95% 0 0 0 2496 83731 4836 143294 155091 0 0 7s 87% 99% Bn 100% 39 0 2457 0 0 80372 2835
88% 0 0 0 2489 102816 7312 106326 146807 0 0 12s 89% 100% :s 95% 318 0 2171 0 0 98129 4764
53% 0 0 0 1652 16819 6639 48480 46076 0 0 12s 86% 100% :s 50% 0 0 1652 0 0 16013 6114
89% 0 0 0 1976 56528 4249 125429 220625 0 0 12s 88% 99% Bn 77% 0 0 1976 0 0 54126 2844
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP
sysstat -m 1
sysstat -m -c 50
ANY AVG CPU0 CPU1 CPU2 CPU3
99% 92% 92% 92% 94% 88%
100% 90% 89% 88% 92% 91%
93% 77% 75% 75% 81% 76%
98% 87% 86% 86% 89% 86%
98% 86% 85% 85% 89% 86%
98% 92% 91% 91% 95% 89%
99% 86% 82% 88% 87% 86%
sysstat -M 1 in diag mode
ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Raid_Ex Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt SSAN_Ex Intr Host Ops/s CP
100% 100% 99% 89% 97% 98% 98% 99% 92% 31% 0% 0% 25% 0% 64% 12% 10% 158%( 88%) 14% 30% 0% 39% 0% 3% 1% 4030 100%
100% 96% 90% 77% 91% 87% 90% 92% 95% 21% 0% 0% 25% 0% 59% 8% 12% 167%( 88%) 10% 20% 0% 36% 0% 3% 1% 2697 99%
100% 99% 96% 84% 95% 96% 96% 96% 91% 40% 0% 0% 18% 0% 52% 11% 14% 150%( 83%) 11% 37% 0% 40% 0% 3% 2% 3636 100%
100% 100% 97% 84% 95% 97% 98% 97% 89% 16% 0% 0% 22% 0% 68% 6% 9% 138%( 86%) 22% 30% 0% 53% 0% 3% 15% 1655 99%
100% 95% 83% 59% 84% 85% 83% 85% 83% 22% 0% 0% 18% 0% 39% 12% 15% 133%( 75%) 6% 69% 0% 18% 0% 3% 1% 3713 99%
100% 97% 89% 74% 90% 90% 90% 91% 90% 14% 0% 0% 25% 1% 54% 10% 10% 163%( 88%) 12% 40% 0% 29% 0% 3% 1% 2291 100%
100% 94% 85% 64% 86% 84% 84% 85% 91% 26% 0% 0% 19% 0% 36% 18% 21% 132%( 78%) 5% 70% 0% 14% 0% 3% 1% 4512 100%
100% 97% 90% 70% 89% 90% 90% 90% 86% 16% 0% 0% 23% 0% 56% 13% 13% 130%( 80%) 13% 55% 0% 33% 0% 3% 2% 3064 99%
100% 97% 85% 59% 85% 89% 90% 85% 77% 24% 0% 0% 15% 0% 38% 17% 21% 92%( 61%) 7% 95% 0% 20% 0% 3% 9% 4432 100%
100% 98% 93% 76% 92% 94% 93% 92% 88% 30% 0% 0% 19% 0% 45% 15% 18% 130%( 76%) 12% 63% 0% 28% 0% 3% 2% 4263 84%
100% 97% 91% 73% 90% 92% 90% 92% 89% 24% 0% 0% 21% 0% 51% 11% 14% 141%( 80%) 13% 50% 0% 31% 0% 3% 1% 3591 99%
100% 98% 94% 79% 93% 93% 93% 94% 90% 30% 0% 0% 23% 0% 51% 13% 15% 143%( 82%) 8% 50% 0% 32% 0% 3% 1% 4161 100%
100% 99% 97% 86% 96% 98% 97% 97% 91% 33% 0% 0% 24% 1% 56% 14% 13% 140%( 83%) 16% 43% 0% 36% 0% 3% 3% 4441 99%
100% 98% 95% 86% 95% 94% 95% 95% 95% 31% 0% 0% 26% 0% 56% 13% 16% 158%( 83%) 13% 29% 0% 31% 0% 3% 1% 4880 99%
100% 98% 97% 89% 96% 96% 96% 97% 96% 30% 0% 0% 29% 0% 59% 12% 15% 170%( 85%) 12% 23% 0% 28% 0% 3% 1% 4341 99%
Statit output -
ID: 2014386083 Memory: 6114 MB
NetApp Release 8.2P6 7-Mode: Fri Jan 17 22:23:50 PST 2014
<1O>
Start time: Fri Dec 23 07:24:07 GMT 2016
CPU Statistics
19.444790 time (seconds) 100 %
69.620104 system time 358 %
0.653313 rupt time 3 % (491410 rupts x 1 usec/rupt)
68.966791 non-rupt system time 355 %
8.159056 idle time 42 %
19.378947 time in CP 100 % 100 %
0.651628 rupt time in CP 3 % (490122 rupts x 1 usec/rupt)
Multiprocessor Statistics (per second)
cpu0 cpu1 cpu2 cpu3 total
sk switches 11072.32 13190.17 15529.20 5615.69 45407.38
hard switches 11010.66 13146.04 15502.40 5323.02 44982.13
domain switches 327.85 286.61 265.37 901.53 1781.35
CP rupts 4931.66 9591.57 5725.90 4956.70 25205.83
nonCP rupts 12.65 24.27 16.66 12.65 66.24
IPI rupts 0.00 0.00 0.00 0.00 0.00
grab kahuna 0.00 0.00 0.00 0.00 0.00
grab kahuna usec 0.00 0.00 0.00 0.00 0.00
suspend domain 0.00 0.00 0.00 0.00 0.00
suspend domain usec 0.00 0.00 0.00 0.00 0.00
CP rupt usec 11355.18 10717.83 7170.97 4267.62 33511.70
nonCP rupt usec 33.22 24.99 19.75 8.59 86.66
idle 121405.43 105709.19 77763.35 114723.17 419601.19
kahuna 0.00 0.00 0.00 93915.49 93915.49
storage 0.00 0.00 276214.71 0.00 276214.71
exempt 145447.29 132218.40 86338.76 60.79 364065.34
raid 1716.35 1295.15 722.87 0.00 3734.42
raid_exempt 237375.20 213198.86 180858.42 0.00 631432.48
target 17350.87 96006.13 12860.26 0.00 126217.31
dnscache 0.00 0.00 0.00 0.00 0.00
cifs 92.83 75.19 94.73 0.00 262.80
wafl_exempt 324797.44 287419.00 250991.14 787024.18 1650231.76
wafl_xcleaner 46336.63 52061.25 31884.79 0.00 130282.66
sm_exempt 28.90 37.54 27.21 0.00 93.70
cluster 0.00 0.00 0.00 0.00 0.00
protocol 443.25 450.76 298.38 0.00 1192.45
nwk_exclusive 3788.52 2920.68 2340.42 0.00 9049.72
nwk_exempt 81534.08 89270.29 64611.45 0.00 235415.91
nwk_legacy 1975.34 1872.12 1464.87 0.00 5312.37
hostOS 6319.12 6722.21 6337.58 0.00 19378.97
ssan_exempt 0.00 0.00 0.00 0.00 0.00
FreeBSD CPU state Statistics (per second)
user 0.10 0.05 0.00 0.00 0.15
nice 0.00 0.00 0.00 0.00 0.00
sys 115.97 118.03 122.29 118.54 474.83
intr 0.05 0.10 0.10 0.00 0.26
idle 16.92 14.86 10.59 14.45 56.83
nonrt-pf-cnt 0.00 0.00 0.00 0.00 0.00
nonrt-pf-usec 0.00 0.00 0.00 0.00 0.00
rt-pf-cnt 0.00 0.00 0.00 0.00 0.00
rt-pf-usec 0.00 0.00 0.00 0.00 0.00
kern-pf-cnt 0.00 0.00 0.00 0.00 0.00
kern-pf-usec 0.00 0.00 0.00 0.00 0.00
19.251686 seconds with one or more CPUs active ( 99%)
18.366020 seconds with 2 or more CPUs active ( 94%)
17.215831 seconds with 3 or more CPUs active ( 89%)
0.885665 seconds with one CPU active ( 5%)
1.150189 seconds with 2 CPUs active ( 6%)
2.500874 seconds with 3 CPUs active ( 13%)
14.714956 seconds with all CPUs active ( 76%)
Domain Utilization of Shared Domains (per second)
0.00 idle 862962.57 kahuna
0.00 storage 0.00 exempt
521829.55 raid 0.00 raid_exempt
0.00 target 0.00 dnscache
0.00 cifs 0.00 wafl_exempt
0.00 wafl_xcleaner 0.00 sm_exempt
0.00 cluster 0.00 protocol
225824.76 nwk_exclusive 0.00 nwk_exempt
0.00 nwk_legacy 0.00 hostOS
0.00 ssan_exempt
switch domain to domain (per second)
0.00 idle 571.36 kahuna
74.88 storage 86.40 exempt
19.23 raid 0.00 raid_exempt
0.26 target 0.00 dnscache
1.03 cifs 373.72 wafl_exempt
0.00 wafl_xcleaner 0.00 sm_exempt
0.00 cluster 0.31 protocol
12.50 nwk_exclusive 296.58 nwk_exempt
345.08 nwk_legacy 0.00 hostOS
0.00 ssan_exempt
Exempt Domain Suspension Stats (per second)
Miscellaneous Statistics (per second)
44982.13 hard context switches 0.00 NFS operations
0.00 CIFS operations 0.00 HTTP operations
81696.95 network KB received 35315.53 network KB transmitted
187898.56 disk KB read 202124.07 disk KB written
78651.25 NVRAM KB written 27675.07 nolog KB written
8542.08 WAFL bufs given to clients 0.00 checksum cache hits ( 0%)
0.00 no checksum - partial buffer 0.00 FCP operations
3913.54 iSCSI operations
WAFL Statistics (per second)
165.60 name cache hits ( 82%) 36.10 name cache misses ( 18%)
306702.62 buf hash hits ( 67%) 149447.44 buf hash misses ( 33%)
10920.35 inode cache hits ( 100%) 0.00 inode cache misses ( 0%)
63701.85 buf cache hits ( 89%) 7497.12 buf cache misses ( 11%)
4732.48 blocks read 16781.20 blocks read-ahead
4832.09 chains read-ahead 199.54 dummy reads
3626.68 blocks speculative read-ahead 31543.00 blocks written
153.77 stripes written 0.00 blocks page flipped
0.00 blocks over-written 0.00 wafl_timer generated CP
0.00 snapshot generated CP 0.00 wafl_avail_bufs generated CP
0.15 dirty_blk_cnt generated CP 0.10 full NV-log generated CP
0.15 back-to-back CP 0.00 flush generated CP
0.00 sync generated CP 0.00 deferred back-to-back CP
0.00 low mbufs generated CP 0.00 low datavecs generated CP
0.00 nvlog replay takeover time limit CP 34380.93 non-restart messages
2321.65 IOWAIT suspends 434.10 next nvlog nearly full msecs
804.48 dirty buffer susp msecs 0.00 nvlog full susp msecs
998538 buffers
RAID Statistics (per second)
13156.79 xors 0.00 long dispatches [0]
0.00 long consumed [0] 0.00 long consumed hipri [0]
0.00 long low priority [0] 0.00 long high priority [0]
99.36 long monitor tics [0] 0.00 long monitor clears [0]
0.00 long dispatches [1] 0.00 long consumed [1]
0.00 long consumed hipri [1] 0.00 long low priority [1]
99.36 long high priority [1] 99.36 long monitor tics [1]
0.00 long monitor clears [1] 18 max batch
164.83 blocked mode xor 129.39 timed mode xor
2.73 fast adjustments 0.57 slow adjustments
0 avg batch start 0 avg stripe/msec
0.00 checksum dispatches 0.00 checksum consumed
157.06 tetrises written 0.00 master tetrises
0.00 slave tetrises 8037.32 stripes written
5203.35 partial stripes 2833.97 full stripes
33879.87 blocks written 19607.51 blocks read
1676.70 1 blocks per stripe size 3 1791.89 2 blocks per stripe size 3
2184.44 3 blocks per stripe size 3 83.67 1 blocks per stripe size 13
76.78 2 blocks per stripe size 13 92.57 3 blocks per stripe size 13
83.88 4 blocks per stripe size 13 115.61 5 blocks per stripe size 13
122.09 6 blocks per stripe size 13 139.93 7 blocks per stripe size 13
159.89 8 blocks per stripe size 13 154.95 9 blocks per stripe size 13
170.28 10 blocks per stripe size 13 228.70 11 blocks per stripe size 13
286.30 12 blocks per stripe size 13 639.04 13 blocks per stripe size 13
2.57 1 blocks per stripe size 14 2.83 2 blocks per stripe size 14
1.44 3 blocks per stripe size 14 0.98 4 blocks per stripe size 14
0.51 5 blocks per stripe size 14 0.87 6 blocks per stripe size 14
0.87 7 blocks per stripe size 14 0.87 8 blocks per stripe size 14
0.62 9 blocks per stripe size 14 1.29 10 blocks per stripe size 14
1.70 11 blocks per stripe size 14 3.29 12 blocks per stripe size 14
2.26 13 blocks per stripe size 14 10.49 14 blocks per stripe size 14
Network Interface Statistics (per second)
iface side bytes packets multicasts errors collisions pkt drops
e0a recv 309.70 4.73 0.00 0.00 0.00
xmit 33.22 0.31 0.05 0.00 0.00
e0b recv 793.12 8.07 0.00 0.00 0.00
xmit 59.04 0.41 0.05 0.00 0.00
e1a recv 3309129.33 1102.76 0.15 0.00 0.00
xmit 2426425.59 1209.06 0.00 0.00 0.00
e1b recv 3223077.90 1099.57 0.15 0.00 0.00
xmit 2350960.64 1156.71 0.00 0.00 0.00
e2a recv 38777662.40 27336.11 1.18 0.00 0.00
xmit 15955724.49 14593.42 0.00 0.00 0.00
e2b recv 38345798.95 27027.55 1.90 0.00 0.00
xmit 15429151.25 14434.15 0.00 0.00 0.00
c0a recv 381.08 12.55 2.06 0.00 0.00
xmit 363.49 1.95 1.95 0.00 0.00
c0b recv 381.08 1.95 2.06 0.00 0.00
xmit 363.49 1.95 1.95 0.00 0.00
e0M recv 92.57 1.54 1.54 0.00 0.00
xmit 0.00 0.00 0.00 0.00 0.00
e0P recv 80.69 0.46 0.15 0.00 0.00
xmit 36.41 0.31 0.00 0.00 0.00
IFGRP_LAN recv 1116.96 12.91 11.73 0.00 0.00
xmit 87.22 0.67 0.10 0.00 0.00
Disk Statistics (per second)
ut% is the percent of time the disk was busy.
xfers is the number of data-transfer commands issued per second.
xfers = ureads + writes + cpreads + greads + gwrites
chain is the average number of 4K blocks per command.
usecs is the average disk round-trip time per 4K block.
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs
/Aggr1_SATA3T_FP/plex0/rg0:
5c.12.8L1 14 25.97 0.00 .... . 11.77 40.54 223 14.19 26.47 221 0.00 .... . 0.00 .... .
3b.11.8L1 15 26.12 0.00 .... . 12.08 39.55 264 14.04 26.97 238 0.00 .... . 0.00 .... .
5c.12.9L1 77 127.57 97.34 2.83 7430 16.76 20.27 863 13.47 6.51 2362 0.00 .... . 0.00 .... .
3b.11.9L1 76 125.57 102.74 2.86 7193 12.39 26.01 812 10.44 9.61 1479 0.00 .... . 0.00 .... .
5c.12.10L1 75 125.15 100.94 2.50 7907 12.49 25.68 1044 11.72 10.13 1292 0.00 .... . 0.00 .... .
3b.11.10L1 76 128.19 101.66 2.63 8569 14.81 22.44 1023 11.72 10.03 1405 0.00 .... . 0.00 .... .
5c.12.11L1 74 126.03 100.11 2.57 6939 13.78 23.25 853 12.13 9.55 1092 0.00 .... . 0.00 .... .
3b.11.11L1 77 128.81 102.99 2.77 7770 14.71 22.75 988 11.11 10.38 1186 0.00 .... . 0.00 .... .
5c.12.12L1 78 128.75 101.91 2.68 8168 14.96 20.99 954 11.88 9.93 1265 0.00 .... . 0.00 .... .
3b.11.12L1 77 132.76 108.60 2.58 7311 13.37 24.64 852 10.80 11.74 1351 0.00 .... . 0.00 .... .
5c.12.13L1 76 124.85 99.91 2.56 8780 13.06 27.61 954 11.88 9.10 1359 0.00 .... . 0.00 .... .
3b.11.13L1 73 126.90 100.11 2.73 7748 14.40 24.33 772 12.39 8.34 1318 0.00 .... . 0.00 .... .
5c.12.14L1 73 121.14 95.64 2.76 8399 13.93 24.37 962 11.57 8.08 1707 0.00 .... . 0.00 .... .
3b.11.14L1 72 120.99 94.97 2.71 6709 14.09 23.53 1017 11.93 8.22 1612 0.00 .... . 0.00 .... .
5c.12.15L1 75 129.88 102.79 2.61 8023 14.35 23.10 896 12.75 9.49 1457 0.00 .... . 0.00 .... .
/Aggr1_SATA3T_FP/plex0/rg1:
3b.11.15L1 13 21.90 0.00 .... . 9.15 49.29 224 12.75 26.54 231 0.00 .... . 0.00 .... .
5c.12.16L1 13 22.01 0.00 .... . 9.26 48.76 210 12.75 26.54 224 0.00 .... . 0.00 .... .
3b.11.16L1 72 120.58 97.85 2.60 7245 12.44 26.86 760 10.28 10.48 1121 0.00 .... . 0.00 .... .
5c.12.17L1 74 118.93 96.87 2.86 7616 12.44 26.00 845 9.62 10.59 1208 0.00 .... . 0.00 .... .
3b.11.17L1 72 118.68 96.82 2.74 7185 11.72 27.21 763 10.13 10.07 1429 0.00 .... . 0.00 .... .
5c.12.18L1 74 121.66 98.88 2.69 7428 12.80 24.19 831 9.98 10.84 1193 0.00 .... . 0.00 .... .
3b.11.18L1 72 116.62 96.36 2.97 5842 11.62 27.73 736 8.64 11.32 1120 0.00 .... . 0.00 .... .
5c.12.19L1 70 113.89 92.91 2.71 7781 12.08 26.20 790 8.90 10.41 1416 0.00 .... . 0.00 .... .
3b.11.19L1 72 116.77 96.26 2.91 8033 11.16 28.06 814 9.36 8.41 1830 0.00 .... . 0.00 .... .
5c.12.20L1 73 118.62 97.34 2.73 6981 11.67 25.65 804 9.62 10.26 1279 0.00 .... . 0.00 .... .
3b.11.20L1 74 121.71 102.22 2.85 7607 10.90 29.59 817 8.59 12.59 1044 0.00 .... . 0.00 .... .
5c.12.21L1 75 121.09 101.14 2.63 7662 11.52 29.55 813 8.43 10.06 1579 0.00 .... . 0.00 .... .
3b.11.21L1 76 119.55 100.37 3.13 6643 10.64 30.10 740 8.54 10.92 1326 0.00 .... . 0.00 .... .
5c.12.22L1 73 114.77 94.10 2.77 7080 11.67 26.58 729 9.00 12.43 1110 0.00 .... . 0.00 .... .
3b.11.22L1 75 118.01 96.93 2.42 7885 11.72 26.61 760 9.36 10.25 1125 0.00 .... . 0.00 .... .
/Aggr1_SATA3T_FP/plex0/rg2:
5c.12.23L1 14 22.98 0.00 .... . 9.67 49.59 249 13.32 26.05 271 0.00 .... . 0.00 .... .
3b.11.23L1 14 23.04 0.00 .... . 9.77 49.08 229 13.27 26.38 250 0.00 .... . 0.00 .... .
5c.12.1L2 76 124.43 102.48 2.65 8441 12.55 28.27 950 9.41 9.63 1646 0.00 .... . 0.00 .... .
3b.11.1L2 76 125.00 103.20 2.51 8328 12.39 28.20 785 9.41 10.21 1287 0.00 .... . 0.00 .... .
5c.12.2L2 77 123.10 101.71 2.97 6477 12.19 29.04 793 9.20 11.33 1060 0.00 .... . 0.00 .... .
3b.11.2L2 74 119.81 98.42 2.74 8025 12.49 29.49 851 8.90 10.77 1297 0.00 .... . 0.00 .... .
5c.12.3L2 74 119.19 95.02 2.62 7376 13.78 25.29 788 10.39 10.19 1212 0.00 .... . 0.00 .... .
3b.11.3L2 73 119.40 97.34 2.48 7677 12.60 27.12 798 9.46 10.59 1193 0.00 .... . 0.00 .... .
5c.12.4L2 75 121.35 99.70 2.74 6796 11.83 29.44 754 9.82 9.75 1480 0.00 .... . 0.00 .... .
3b.11.4L2 76 126.08 103.92 2.67 7461 13.21 27.09 846 8.95 11.41 1101 0.00 .... . 0.00 .... .
5c.12.5L2 75 120.68 98.36 2.64 7517 13.16 28.04 827 9.15 10.63 1339 0.00 .... . 0.00 .... .
3b.11.5L2 75 120.12 98.78 2.75 7045 11.93 30.57 848 9.41 11.25 1023 0.00 .... . 0.00 .... .
5c.12.6L2 75 120.58 96.57 2.29 9169 13.57 24.19 982 10.44 10.58 1452 0.00 .... . 0.00 .... .
3b.11.6L2 76 122.74 100.99 2.85 7564 12.24 30.64 836 9.51 7.86 1739 0.00 .... . 0.00 .... .
5c.12.7L2 73 112.20 92.55 2.55 7464 11.11 32.34 902 8.54 10.27 1474 0.00 .... . 0.00 .... .
/Aggr1_SATA3T_FP/plex0/rg3:
3b.11.7L2 14 23.40 0.00 .... . 9.51 50.12 243 13.88 25.37 221 0.00 .... . 0.00 .... .
5c.12.9L2 14 23.40 0.00 .... . 9.62 49.61 240 13.78 25.48 215 0.00 .... . 0.00 .... .
3b.11.9L2 71 113.94 90.55 2.84 6064 13.21 27.06 639 10.18 8.12 1365 0.00 .... . 0.00 .... .
5c.12.10L2 74 118.78 94.87 2.62 7255 13.63 25.56 822 10.28 7.91 1810 0.00 .... . 0.00 .... .
3b.11.10L2 73 124.49 98.11 2.71 7207 14.86 23.12 716 11.52 8.71 1310 0.00 .... . 0.00 .... .
5c.12.11L2 72 113.89 90.70 2.79 6770 12.85 26.74 774 10.34 7.59 1714 0.00 .... . 0.00 .... .
3b.11.11L2 73 117.80 95.13 2.75 7133 12.55 26.23 827 10.13 9.61 1340 0.00 .... . 0.00 .... .
5c.12.12L2 72 120.68 93.17 2.69 6649 15.37 22.36 799 12.13 9.56 1152 0.00 .... . 0.00 .... .
3b.11.12L2 75 120.22 94.25 2.84 6997 15.12 24.13 791 10.85 9.14 1694 0.00 .... . 0.00 .... .
5c.12.13L2 75 125.87 99.34 2.73 7279 15.84 22.95 816 10.70 8.63 1401 0.00 .... . 0.00 .... .
3b.11.13L2 68 109.78 84.79 3.11 7043 14.71 22.58 742 10.28 9.55 1188 0.00 .... . 0.00 .... .
5c.12.14L2 74 118.37 94.41 2.79 7403 14.60 23.97 816 9.36 9.12 1613 0.00 .... . 0.00 .... .
3b.11.14L2 75 125.93 99.14 2.59 7628 15.68 22.40 870 11.11 9.82 1393 0.00 .... . 0.00 .... .
5c.12.15L2 73 112.45 88.39 2.73 6870 13.78 24.78 859 10.28 9.19 1482 0.00 .... . 0.00 .... .
3b.11.16L2 76 126.29 102.22 2.74 8802 13.57 26.33 813 10.49 8.88 1696 0.00 .... . 0.00 .... .
/Aggr1_SATA3T_FP/plex0/rg4:
5c.12.17L2 14 21.24 0.00 .... . 9.41 50.84 234 11.83 30.77 226 0.00 .... . 0.00 .... .
3b.11.17L2 14 21.44 0.00 .... . 9.51 50.31 241 11.93 30.59 223 0.00 .... . 0.00 .... .
5c.12.18L2 72 120.17 99.50 2.78 6103 11.00 29.16 645 9.67 11.04 971 0.00 .... . 0.00 .... .
3b.11.18L2 72 120.73 101.09 2.53 6831 11.06 29.50 759 8.59 12.83 934 0.00 .... . 0.00 .... .
5c.12.19L2 72 119.86 99.70 2.83 5722 10.80 31.77 726 9.36 11.81 1095 0.00 .... . 0.00 .... .
3b.11.19L2 72 113.64 95.02 2.54 7330 10.34 31.59 706 8.28 12.35 1021 0.00 .... . 0.00 .... .
5c.12.20L2 74 118.06 99.55 2.54 7949 10.18 34.63 764 8.33 11.60 1344 0.00 .... . 0.00 .... .
3b.11.20L2 76 125.67 104.69 2.79 7218 11.57 29.04 778 9.41 12.87 1142 0.00 .... . 0.00 .... .
5c.12.21L2 75 120.53 100.78 2.70 7411 11.31 29.86 802 8.43 12.04 1142 0.00 .... . 0.00 .... .
3b.11.21L2 73 119.34 100.32 2.79 5728 10.64 32.20 674 8.38 12.20 918 0.00 .... . 0.00 .... .
5c.12.22L2 72 125.15 105.46 2.71 7004 10.90 29.75 640 8.79 12.43 903 0.00 .... . 0.00 .... .
3b.11.22L2 72 113.02 93.99 2.81 6758 10.34 34.85 630 8.69 11.70 1141 0.00 .... . 0.00 .... .
5c.12.0L2 74 122.02 101.50 2.59 6933 11.67 28.49 723 8.84 10.31 1561 0.00 .... . 0.00 .... .
3b.11.8L2 69 112.25 93.27 2.73 5918 10.59 31.81 644 8.38 14.35 944 0.00 .... . 0.00 .... .
5c.12.16L2 75 116.00 96.98 2.61 7461 10.28 32.69 779 8.74 12.24 1072 0.00 .... . 0.00 .... .
/Aggr1_SATA3T_FP/plex0/rg5:
5b.21.6 37 1356.75 0.00 .... . 481.13 11.75 89 875.62 3.98 199 0.00 .... . 0.00 .... .
3c.21.7 32 1357.31 0.00 .... . 483.13 11.71 75 874.18 3.98 152 0.00 .... . 0.00 .... .
5b.21.8 79 3378.86 2154.77 1.34 2009 632.97 6.23 141 591.11 1.96 485 0.00 .... . 0.00 .... .
3c.21.9 80 3381.84 2165.52 1.36 1732 627.78 6.26 104 588.54 2.02 393 0.00 .... . 0.00 .... .
5b.21.10 79 3382.30 2169.22 1.35 2014 632.15 6.23 142 580.93 1.96 478 0.00 .... . 0.00 .... .
/Aggr0_SATA3T/plex0/rg0:
3b.11.0L1 6 6.84 0.31 1.00 16167 4.11 8.14 733 2.42 11.23 722 0.00 .... . 0.00 .... .
5c.12.0L1 9 7.71 0.31 1.00 15500 5.04 7.01 834 2.37 9.96 1511 0.00 .... . 0.00 .... .
3b.11.1L1 7 12.13 5.14 1.29 6953 3.14 7.38 969 3.86 2.12 1943 0.00 .... . 0.00 .... .
5c.12.1L1 4 7.04 4.42 1.22 5295 1.23 16.38 578 1.39 6.67 422 0.00 .... . 0.00 .... .
3b.11.2L1 4 6.84 4.42 1.16 6740 1.34 16.50 615 1.08 7.19 517 0.00 .... . 0.00 .... .
5c.12.2L1 4 7.40 4.78 1.17 6294 1.34 16.65 490 1.29 5.36 694 0.00 .... . 0.00 .... .
3b.11.3L1 5 8.43 5.55 1.16 5552 1.23 16.54 562 1.65 5.22 1024 0.00 .... . 0.00 .... .
5c.12.3L1 4 7.61 4.83 1.19 6080 1.23 17.33 550 1.54 4.20 1349 0.00 .... . 0.00 .... .
3b.11.4L1 4 6.89 4.27 1.11 6967 1.23 17.21 547 1.39 4.19 1097 0.00 .... . 0.00 .... .
5c.12.4L1 5 9.05 6.63 1.14 6789 1.34 16.35 569 1.08 3.95 1181 0.00 .... . 0.00 .... .
3b.11.5L1 4 7.35 4.78 1.14 5415 1.34 16.27 515 1.23 5.83 843 0.00 .... . 0.00 .... .
5c.12.5L1 4 5.91 3.14 1.23 7427 1.39 15.37 472 1.39 7.52 1000 0.00 .... . 0.00 .... .
3b.11.6L1 4 6.43 4.01 1.21 6383 1.34 16.38 502 1.08 2.14 889 0.00 .... . 0.00 .... .
5c.12.6L1 4 6.63 3.81 1.31 6216 1.34 16.12 525 1.49 3.69 841 0.00 .... . 0.00 .... .
3b.11.7L1 4 6.53 3.81 1.22 5933 1.23 17.42 577 1.49 4.10 1067 0.00 .... . 0.00 .... .
5c.12.7L1 4 8.38 5.66 1.30 4804 1.34 16.31 580 1.39 5.63 664 0.00 .... . 0.00 .... .
Aggregate statistics:
Minimum 4 5.91 0.00 1.23 1.08 0.00 0.00
Mean 55 219.41 134.77 39.70 44.94 0.00 0.00
Maximum 80 3382.30 2169.22 632.97 875.62 0.00 0.00
Spares and other disks:
3c.21.3 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3c.21.1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3c.21.11 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5b.21.4 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5b.21.0 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5b.21.2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.0L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.1L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.0L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.1L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.2L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.2L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.3L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.3L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.4L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.5L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.4L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.5L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.6L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.6L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.8L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.7L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.8L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.7L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.9L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.9L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.10L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.11L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.10L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.11L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.12L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.13L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.12L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.13L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.14L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.15L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.14L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.16L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.15L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5c.12.8L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.16L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.17L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.17L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.18L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.19L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.18L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.19L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.20L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.20L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.21L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.22L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.21L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.22L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.23L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5d.02.23L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.0L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.1L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.0L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.1L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
5c.12.23L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3b.11.0L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.2L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.3L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.2L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.3L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.6L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.7L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.6L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.4L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.4L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.7L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.5L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.5L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.8L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.10L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.8L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.9L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.9L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.10L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.11L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.11L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3b.11.15L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.12L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.13L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.12L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.13L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.14L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.15L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.14L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.15L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.16L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.17L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.16L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.17L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3b.11.23L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.18L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.19L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.18L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.19L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.20L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.20L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.21L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.22L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.21L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.22L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.23L1 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3a.01.23L2 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
3c.21.5 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
FCP Statistics (per second)
0.00 FCP Bytes recv 0.00 FCP Bytes sent
0.00 FCP ops
iSCSI Statistics (per second)
79838395.37 iSCSI Bytes recv 34573619.36 iSCSI Bytes xmit
3912.98 iSCSI ops
Interrupt Statistics (per second)
10.13 int_1 1532.18 int_2
1687.29 int_3 3641.59 int_4
2907.05 int_5 2721.81 int_6
1840.08 int_7 2871.51 int_8
5.19 Gigabit Ethernet (IRQ 12) 8.43 Gigabit Ethernet (IRQ 13)
1.54 int_14 0.72 int_15
0.00 RTC 0.00 IPI
996.93 Msec Clock 18224.47 total
'
We are facing this on multiple filers, and on one of our filer we resolved this with a strange solution which was by changing the SNMP string from public to some private string.
Hope for some solution on the basis of provided output and thanks in advance.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi
sorry for not getting too deep but just something attract my eye here with the overall behivior.
Do you have known LUN aligment issue? ( priv set advanced;lun show -v;priv set) or very funny Guest file system ?
the sripes sizes, chain, and overall network in/out vs disk op do not consists.
is it such high all time of the day? specific time ? is it correlate to the use of the system (IOPS)?
another thing that was easy to spot - the small 3 disks Raid group /Aggr1_SATA3T_FP/rg5 is what can really impact your clients and likelly the cause of the back to back (B). this should be you main concern on that node. not the CPU.
Gidi
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
How full are the aggregates? If they are over 80%, or even close to that the amount of time doing CP reads will eat up a lot of io, and the processors actually spend time finding free blocks. We have seen this when the aggregates get full and/or very busy.
All of the raid groups seem equally busy, even the one with just 5 disks (3 data disks I assume) around 80%.
What is the hardware (FAS Controller) behind this other than SATA drives? Its only 4 cpus, which used to be a lot in the world of NetApp, but now is pretty paltry.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Concur on checking how full the aggregates are. The disk utilization is significant and there are a ton of regular Back-to-back CPs indicated. The read/write block ratio is 35/65 per the statit output and while I didn't do the math there are a lot of partial blocks. That all points to signs of relatively full aggregates with a lot of CPU overhead dedicated to finding space to write new data coming in.
There is the aggregate that appears to have very little IO - does that have space available for use as well or is it just non-busy data? If space is available balancing workload between the aggregates would be in order. Granted in 7-mode moving a volume is a non-trivial exercise, but if that aggregate has space it would make sense to balance the volumes between the aggregates just to relieve the pressure induced by disk actions to the controllers. Even for a small system balancing aggregates and watching usage can make a significant difference. Just went through that for a customer on a pair of 8020's that had two aggregates each, but on each one one was 90% and higher used while the other was not used at all. Cut almost 30% off the average CPU load by balancing the aggregates - no other change.
Hope this helps.
Bob Greenwald
Senior Systems Engineer | cStor
NCIE SAN Clustered, Data Protection
Kudos and accepted solutions are always appreciated.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks guys for your reply on this.
I want to clear the few queries asked in trail conversation.
1. Lun allgnment - i have checked and did not found any misalligned LUN.
2. CPU Utilization schenario- Approx all the time, it is near to 90% and when snapmirror transfer starts it went near to 98 or 100% and which is visible that "sm_exempt" went high during that.
3. Aggregate space utilization on this filer - This filer contains 2 aggregate and both are near to 90%.
Aggregate total used avail capacity
Aggr1_SATA3T_FP 156TB 140TB 15TB 90%
Aggr1_SATA3T_FP/.snapshot 0TB 0TB 0TB 0%
Aggr0_SATA3T 33TB 28TB 5252GB 85%
Aggr0_SATA3T/.snapshot 0TB 0TB 0TB 0%
4. Ontap version - NetApp Release 8.2P6 7-Mode
5. Most of time we found that WAFL_EXEMPT domain is highly utilizaed and which represent some wafl backend tasks.
6. From the DFM, CPU utilization seems dependent on ISCSI OPS find the attched screenshot.
7. Here one aggregate is hybrid aggrate.
Aggr State Status Options
Aggr1_SATA3T_FP online raid_dp, aggr nosnap=off, raidtype=raid_dp, raidsize=15,
64-bit cache_raid_group_size=23, raid_lost_write=on,
hybrid ignore_inconsistent=off, snapmirrored=off,
rlw_on resyncsnaptime=60, fs_size_fixed=off,
lost_write_protect=on, ha_policy=cfo,
hybrid_enabled=on, percent_snapshot_space=0%,
free_space_realloc=off, raid_cv=on
Aggr0_SATA3T online raid_dp, aggr root, diskroot, nosnap=off, raidtype=raid_dp,
64-bit raidsize=16, raid_lost_write=on,
rlw_on ignore_inconsistent=off, snapmirrored=off,
resyncsnaptime=60, fs_size_fixed=off,
lost_write_protect=on, ha_policy=cfo,
hybrid_enabled=off, percent_snapshot_space=0%,
free_space_realloc=off, raid_cv=on
Regards.
Upadhayay.1122
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
One thing which can help us is the scrubbing status.
I have read on some articales that it can impact the CPU utilization, below is the output of aggrgate scrub status -v.
aggr scrub: status of /Aggr1_SATA3T_FP/plex0/rg0 :
Current scrub is 78% complete (suspended).
Last full scrub completed: Sun Jun 19 06:50:41 BST 2016
aggr scrub: status of /Aggr1_SATA3T_FP/plex0/rg1 :
Current scrub is 12% complete (suspended).
Last full scrub completed: Sun Jul 10 05:17:36 BST 2016
aggr scrub: status of /Aggr1_SATA3T_FP/plex0/rg2 :
Scrub is not active.
Last full scrub completed: Sun Nov 13 05:30:35 GMT 2016
aggr scrub: status of /Aggr1_SATA3T_FP/plex0/rg3 :
Current scrub is 14% complete (suspended).
Last full scrub completed: Sun Jun 26 02:22:47 BST 2016
aggr scrub: status of /Aggr1_SATA3T_FP/plex0/rg4 :
Scrub is not active.
Last full scrub completed: Sun Oct 30 01:37:44 BST 2016
aggr scrub: status of /Aggr1_SATA3T_FP/plex0/rg5 :
Scrub is not active.
Last full scrub completed: Sun Oct 30 01:48:55 GMT 2016
aggr scrub: status of /Aggr0_SATA3T/plex0/rg0 :
Current scrub is 19% complete (suspended).
Last full scrub completed: Sun Oct 23 05:43:39 BST 2016
And i missed to provide the filer model in trail conversation.
This is NetApp FAS 3240 filer.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Sadly there are several issues.
1. No aggr should ever be more than 80% full or it will experience performance problems. Wafl is not like a standard block based array, it is a file system. That file system needs space to do maintenance and make changes, and the rule is 20% free. The more full an aggr is, the more read time is required to perform a write. The file system has to find free space to perform the write, and this also takes more cpu time.
2. If this is a snapmirror destination, there is a process called deswizzling that occurs when a transfer is complete. It reorders the blocks, which causes even more disk contention.
3. Scrubs need to run. They find bad blocks, and without them the risk of data loss increases. Don't turm these off.
4. This filer uses sata drives, which are very slow for random operations, making all of this worse.
There isnt much you can other than freeing up disk space or adding more drives. You a probably at the point of jyst getting new filers as these are several generations old. You could look at running reallocates on the non-destination volumes, but at this point I dont think you would gain anything.
