statit question, strange numbers on disk util %

peter_lehmann · ‎2011-07-06

hi community

I did run statit on a customer storage controller (FAS6070) and got some strange numbers I cannot interpret.

The disks on port 0e are showing 24% util and the disks on port 0a show 65%... Why this?

statit=

disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs

/aggrfc04/plex0/rg0:

0e.96 0 0.84 0.14 1.00 5951 0.39 5.60 776 0.31 4.62 1077 0.00 .... . 0.00 .... .

0a.96 4 0.92 0.14 1.00 110146 0.53 4.63 6795 0.24 3.73 16455 0.00 .... . 0.00 .... .

0e.97 24 66.35 65.92 10.36 923 0.26 5.15 701 0.16 4.22 1814 0.00 .... . 0.00 .... .

0a.97 65 66.03 65.71 10.42 4471 0.14 8.63 8414 0.18 3.25 19976 0.00 .... . 0.00 .... .

0e.98 24 66.23 65.93 10.38 921 0.14 8.92 843 0.16 4.06 1780 0.00 .... . 0.00 .... .

0a.98 65 65.53 65.20 10.42 4504 0.15 7.98 8960 0.19 2.71 23190 0.00 .... . 0.00 .... .

0e.99 24 66.09 65.81 10.35 928 0.12 9.55 870 0.16 3.97 2076 0.00 .... . 0.00 .... .

0a.99 65 65.46 65.14 10.34 4542 0.15 7.85 9635 0.16 3.14 19633 0.00 .... . 0.00 .... .

0e.100 24 66.83 66.52 10.28 934 0.16 7.41 1194 0.15 3.46 1877 0.00 .... . 0.00 .... .

0a.100 65 65.55 65.24 10.40 4526 0.17 7.24 10364 0.14 3.20 21361 0.00 .... . 0.00 .... .

0e.101 24 66.67 66.39 10.34 940 0.14 8.37 986 0.14 3.13 2128 0.00 .... . 0.00 .... .

0a.101 65 66.26 65.97 10.34 4573 0.15 8.18 9091 0.14 2.96 22449 0.00 .... . 0.00 .... .

0e.102 24 65.77 65.46 10.27 948 0.16 7.56 1187 0.14 3.26 2559 0.00 .... . 0.00 .... .

0a.102 65 65.77 65.46 10.47 4510 0.17 7.33 10163 0.14 2.84 23123 0.00 .... . 0.00 .... .

0e.103 24 65.62 65.29 10.39 926 0.16 7.39 1147 0.17 3.14 1923 0.00 .... . 0.00 .... .

0a.103 65 67.60 67.28 10.38 4509 0.16 7.09 10530 0.16 3.28 19736 0.00 .... . 0.00 .... .

0e.104 23 65.64 65.30 10.37 923 0.17 7.16 1299 0.17 2.95 2311 0.00 .... . 0.00 .... .

0a.104 65 66.21 65.88 10.32 4595 0.17 7.41 10037 0.16 2.79 23408 0.00 .... . 0.00 .... .

0e.105 24 65.34 65.00 10.37 923 0.17 7.28 1263 0.17 3.01 2207 0.00 .... . 0.00 .... .

0a.105 65 65.76 65.43 10.40 4531 0.15 7.86 9616 0.18 2.89 22359 0.00 .... . 0.00 .... .

0e.106 24 66.96 66.60 10.29 948 0.17 7.36 1352 0.19 2.91 2573 0.00 .... . 0.00 .... .

0a.106 66 67.27 66.90 10.40 4510 0.18 6.81 10829 0.20 2.76 23589 0.00 .... . 0.00 .... .

0e.107 24 66.05 65.68 10.42 914 0.17 7.26 1305 0.19 2.75 2545 0.00 .... . 0.00 .... .

0a.107 65 66.04 65.69 10.47 4485 0.16 7.70 9623 0.19 2.65 24086 0.00 .... . 0.00 .... .

0e.108 24 66.77 66.41 10.36 945 0.16 7.39 1364 0.20 3.02 2873 0.00 .... . 0.00 .... .

0a.108 65 65.72 65.38 10.43 4503 0.14 9.12 8173 0.20 3.30 20259 0.00 .... . 0.00 .... .

0e.109 24 65.62 65.25 10.45 937 0.17 7.26 1303 0.21 3.30 2420 0.00 .... . 0.00 .... .

Aggregate aggrfc04 (online, raid_dp) (block checksums)

Plex /aggrfc04/plex0 (online, normal, active)

RAID group /aggrfc04/plex0/rg0 (normal)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

dparity 0e.96 0e 6 0 FC:A - FCAL 15000 418000/856064000 420156/860480768

parity 0a.96 0a 6 0 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0e.97 0e 6 1 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0a.97 0a 6 1 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0e.98 0e 6 2 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0a.98 0a 6 2 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0e.99 0e 6 3 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0a.99 0a 6 3 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0e.100 0e 6 4 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0a.100 0a 6 4 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0e.101 0e 6 5 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0a.101 0a 6 5 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0e.102 0e 6 6 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0a.102 0a 6 6 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0e.103 0e 6 7 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0a.103 0a 6 7 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0e.104 0e 6 8 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0a.104 0a 6 8 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0e.105 0e 6 9 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0a.105 0a 6 9 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0e.106 0e 6 10 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0a.106 0a 6 10 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0e.107 0e 6 11 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0a.107 0a 6 11 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0e.108 0e 6 12 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0a.108 0a 6 12 FC:A - FCAL 15000 418000/856064000 420156/860480768

data 0e.109 0e 6 13 FC:A - FCAL 15000 418000/856064000 420156/860480768

sPonsford · ‎2011-07-06

I have seen hot disks being caused by firmware bugs etc but this is consistent depending on which port.

Have you considered somehow different adapter speeds have been introduced... certainly wouldnt be best practices but maybe someone has used a 2gb SFP or something on the a loop and 4gb on the alternate..

peter_lehmann · ‎2011-07-06

Hi and thanks, I've already checked this and both ports 2gb...

And there seems to be newest firmware on them (NA01).

sPonsford · ‎2011-07-06

Were the disks added to on of the ports? ie its not actually mpha and they have added a new stack of disk to 0e?

I just noticed the the columns are way out for what looks to be greads and cpreads.. and this alternates along with the utilisation.

You may have to do a wafl reallocate? or is the RAID group just levelling after adding a heap of disk which would make sense for it to be so uniform.

sPonsford · ‎2011-07-06

Just to add Greads - reads generated by RAID for reconstruct and other guaranteed-bandwidth operations so this may align to disks being added???

A statit lined up would help to say exactly what operations are out of step as its a little hard to match them up in your statit output...

peter_lehmann · ‎2011-07-06

statit lined up....

like this?

disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecsgreads--chain-usecs gwrites-chain-usecs

/aggrfc04/plex0/rg0:

0e.96 0 0.84 0.14 1.00 5951 0.39 5.60 776 0.31 4.621077 0.00 .... . 0.00 .... .

0a.96 4 0.92 0.14 1.00 110146 0.53 4.63 6795 0.24 3.7316455 0.00 .... . 0.00 .... .

0e.97 24 66.35 65.92 10.36 923 0.26 5.15 701 0.16 4.221814 0.00 .... . 0.00 .... .

0a.97 65 66.03 65.71 10.42 4471 0.14 8.63 8414 0.18 3.2519976 0.00 .... . 0.00 .... .

0e.98 24 66.23 65.93 10.38 921 0.14 8.92 843 0.16 4.061780 0.00 .... . 0.00 .... .

0a.98 65 65.53 65.20 10.42 4504 0.15 7.98 8960 0.19 2.7123190 0.00 .... . 0.00 .... .

0e.99 24 66.09 65.81 10.35 928 0.12 9.55 870 0.16 3.972076 0.00 .... . 0.00 .... .

0a.99 65 65.46 65.14 10.34 4542 0.15 7.85 9635 0.16 3.1419633 0.00 .... . 0.00 .... .

0e.100 24 66.83 66.52 10.28 934 0.16 7.41 1194 0.15 3.461877 0.00 .... . 0.00 .... .

0a.100 65 65.55 65.24 10.40 4526 0.17 7.24 10364 0.14 3.2021361 0.00 .... . 0.00 .... .

0e.101 24 66.67 66.39 10.34 940 0.14 8.37 986 0.14 3.132128 0.00 .... . 0.00 .... .

0a.101 65 66.26 65.97 10.34 4573 0.15 8.18 9091 0.14 2.9622449 0.00 .... . 0.00 .... .

0e.102 24 65.77 65.46 10.27 948 0.16 7.56 1187 0.14 3.262559 0.00 .... . 0.00 .... .

0a.102 65 65.77 65.46 10.47 4510 0.17 7.33 10163 0.14 2.8423123 0.00 .... . 0.00 .... .

0e.103 24 65.62 65.29 10.39 926 0.16 7.39 1147 0.17 3.141923 0.00 .... . 0.00 .... .

0a.103 65 67.60 67.28 10.38 4509 0.16 7.09 10530 0.16 3.2819736 0.00 .... . 0.00 .... .

0e.104 23 65.64 65.30 10.37 923 0.17 7.16 1299 0.17 2.952311 0.00 .... . 0.00 .... .

0a.104 65 66.21 65.88 10.32 4595 0.17 7.41 10037 0.16 2.7923408 0.00 .... . 0.00 .... .

0e.105 24 65.34 65.00 10.37 923 0.17 7.28 1263 0.17 3.012207 0.00 .... . 0.00 .... .

0a.105 65 65.76 65.43 10.40 4531 0.15 7.86 9616 0.18 2.8922359 0.00 .... . 0.00 .... .

0e.106 24 66.96 66.60 10.29 948 0.17 7.36 1352 0.19 2.912573 0.00 .... . 0.00 .... .

0a.106 66 67.27 66.90 10.40 4510 0.18 6.81 10829 0.20 2.7623589 0.00 .... . 0.00 .... .

0e.107 24 66.05 65.68 10.42 914 0.17 7.26 1305 0.19 2.752545 0.00 .... . 0.00 .... .

0a.107 65 66.04 65.69 10.47 4485 0.16 7.70 9623 0.19 2.6524086 0.00 .... . 0.00 .... .

0e.108 24 66.77 66.41 10.36 945 0.16 7.39 1364 0.20 3.022873 0.00 .... . 0.00 .... .

0a.108 65 65.72 65.38 10.43 4503 0.14 9.12 8173 0.20 3.300259 0.00 .... . 0.00 .... .

0e.109 24 65.62 65.25 10.45 937 0.17 7.26 1303 0.21 3.302420 0.00 .... . 0.00 .... .

eric_barlier · ‎2011-07-06

Hi Peter,

Are you actually seeing any problems? Disks on path X being used more than disks on path Y might be perfectily normal, it could be by design. Have you checked which is the primary path on the controllers? My guess is that A is primary path. If A is primary path then fine. The other thing you need to look out for are LUNs with multipathing issues. I dont even know if you ve got LUNs here but

from experience I know that AIX LUNs and Vmware needs to have pathing setup properly.

are you seeing any of these : https://login.netapp.com/sso/jsp/authnoam.jsp?doneURL=%2Fuser%2Floginsso&authnMech=oracle%3Afed%3Aauthentication%3Apassword-protected&refID=id-T6yjo3b...

In short, if you run this on the LUNs (assuming you have any)

lun stats -o

You should look for partner OPS and partner bytes at the end of the output line. There should be none, or hardly none.

I would recommend against doing any WAFL reallocates unless tech support advises it will help.

Hope this helps,

Eric

peter_lehmann · ‎2011-07-06

Hi Eric

Yes it could be "normal", but in this case we are having performance issues on this system and I'm trying to find out which components actively participate in this issue. Anything out of the ordinary is suspect until cleared

No LUNs part of this, thanks anyway.

Peter

sPonsford · ‎2011-07-06

Eric, agree with you there.. i was not looking at the fact the disk numbers were the same just alternating between adapters..

In terms of an issue i would still question why the USECS are so long for the disks showign 65%, usecs is the average disk round-trip time per 4K block so i qould be wondering why.

You have xfers the same as well as chain lenght but the usecs are way out... i would have to confirm if this is a normal output by looking at one of my systems...

peter_lehmann · ‎2011-07-06

here is another one, looking "normal":

/aggrfc01/plex0/rg0:

0a.16 14 3.12 0.50 1.13 144920 2.24 10.05 3240 0.38 6.55 10330 0.00 .... . 0.00 .... .

0a.32 14 3.26 0.50 1.13 162227 2.40 9.51 3191 0.36 5.14 13021 0.00 .... . 0.00 .... .

0a.48 72 54.55 53.00 8.12 6772 1.09 17.11 4209 0.46 8.56 7420 0.00 .... . 0.00 .... .

0a.64 71 54.44 52.94 8.20 6606 0.99 18.36 4493 0.52 7.95 8452 0.00 .... . 0.00 .... .

0a.80 71 54.63 53.20 8.17 6541 0.96 18.75 4474 0.47 8.33 7902 0.00 .... . 0.00 .... .

0a.17 72 53.82 52.43 8.18 6767 0.95 19.36 4318 0.44 9.19 7430 0.00 .... . 0.00 .... .

0a.33 72 54.91 53.49 8.21 6669 0.92 18.96 4234 0.49 9.54 7194 0.00 .... . 0.00 .... .

0a.49 71 54.91 53.44 8.15 6613 0.96 18.91 4361 0.51 7.96 8541 0.00 .... . 0.00 .... .

0a.65 71 54.37 52.97 8.10 6795 0.93 18.94 4511 0.47 9.00 7329 0.00 .... . 0.00 .... .

0a.81 71 55.10 53.66 8.16 6668 0.96 18.66 4474 0.48 9.34 7036 0.00 .... . 0.00 .... .

0a.18 72 54.32 52.93 8.18 6778 0.93 19.06 4423 0.45 11.01 6386 0.00 .... . 0.00 .... .

0a.34 72 55.07 53.63 8.18 6730 0.95 18.74 4469 0.49 8.85 7478 0.00 .... . 0.00 .... .

0a.50 71 53.74 52.31 8.30 6545 0.97 19.28 4378 0.47 8.32 8203 0.00 .... . 0.00 .... .

0a.66 72 54.12 52.79 8.11 6686 0.90 20.23 4192 0.43 9.73 7031 0.00 .... . 0.00 .... .

0a.82 71 54.95 53.49 8.22 6557 0.97 18.36 4732 0.49 8.90 7469 0.00 .... . 0.00 .... .

0a.19 72 54.07 52.64 8.28 6660 0.96 18.64 4606 0.47 9.51 7374 0.00 .... . 0.00 .... .

0a.35 71 54.62 53.15 8.13 6724 0.97 18.64 4668 0.50 7.97 8605 0.00 .... . 0.00 .... .

peter_lehmann · ‎2011-07-06

And for better readability the original one with colored lines (0a and 0e).

Seeing the usecs numbers are 20times higher!