ONTAP Hardware

Understanding disk paths

iUser
7,874 Views

Hi, guys!

 We have 4-node fas8200 metrocluster, ontap 9.6p5. I'm trying understand how to interpret the output of the "storage disk show" command. I did the following:

cluster_a::> storage disk show -primary-port B -secondary-port B
                     Usable           Disk    Container   Container   
Disk                   Size Shelf Bay Type    Type        Name      Owner
---------------- ---------- ----- --- ------- ----------- --------- --------
3.10.1               1.63TB    10   1 SAS     aggregate   aggr1_a   cluster_a-01
3.10.2               1.63TB    10   2 SAS     aggregate   aggr1_a   cluster_a-01
3.10.4               1.63TB    10   4 SAS     aggregate   aggr0_cluster_a_01 
                                                                    cluster_a-01
3.10.5               1.63TB    10   5 SAS     aggregate   aggr1_a   cluster_a-01
3.10.8               1.63TB    10   8 SAS     aggregate   aggr1_a   cluster_a-01
3.10.10              1.63TB    10  10 SAS     aggregate   aggr1_a   cluster_a-01
3.10.11              1.63TB    10  11 SAS     aggregate   aggr1_a   cluster_a-01
3.10.15              1.63TB    10  15 SAS     aggregate   aggr1_a   cluster_a-01
3.10.16              1.63TB    10  16 SAS     aggregate   aggr1_a   cluster_a-01
3.10.18              1.63TB    10  18 SAS     aggregate   aggr1_a   cluster_a-01
3.10.20              1.63TB    10  20 SAS     aggregate   aggr1_a   cluster_a-01
3.10.22              1.63TB    10  22 SAS     aggregate   aggr1_a   cluster_a-01
3.10.23              1.63TB    10  23 SAS     spare       Pool1     cluster_a-01
3.11.0               1.63TB    11   0 SAS     aggregate   aggr0_cluster_a_02 
                                                                    cluster_a-02
3.11.2               1.63TB    11   2 SAS     aggregate   aggr0_cluster_a_02 
                                                                    cluster_a-02
3.11.3               1.63TB    11   3 SAS     aggregate   aggr2_a   cluster_a-02
3.11.7               1.63TB    11   7 SAS     aggregate   aggr2_a   cluster_a-02
3.11.8               1.63TB    11   8 SAS     aggregate   aggr2_a   cluster_a-02
3.11.9               1.63TB    11   9 SAS     aggregate   aggr2_a   cluster_a-02
3.11.11              1.63TB    11  11 SAS     aggregate   aggr2_a   cluster_a-02
3.11.12              1.63TB    11  12 SAS     aggregate   aggr2_a   cluster_a-02
3.11.18              1.63TB    11  18 SAS     aggregate   aggr2_a   cluster_a-02
3.11.20              1.63TB    11  20 SAS     aggregate   aggr2_a   cluster_a-02
3.11.21              1.63TB    11  21 SAS     aggregate   aggr2_a   cluster_a-02
3.12.0                    -    12   0 SAS     remote      -         cluster_b-01
3.12.1                    -    12   1 SAS     remote      -         cluster_b-01
3.12.2                    -    12   2 SAS     remote      -         cluster_b-01
3.12.8                    -    12   8 SAS     remote      -         cluster_b-01
3.12.12                   -    12  12 SAS     remote      -         cluster_b-01
3.12.14                   -    12  14 SAS     remote      -         cluster_b-01
3.12.18                   -    12  18 SAS     remote      -         cluster_b-01
3.12.20                   -    12  20 SAS     remote      -         cluster_b-01
3.12.22                   -    12  22 SAS     remote      -         cluster_b-01
3.13.3                    -    13   3 SAS     remote      -         cluster_b-02
3.13.6                    -    13   6 SAS     remote      -         cluster_b-02
3.13.7                    -    13   7 SAS     remote      -         cluster_b-02
3.13.8                    -    13   8 SAS     remote      -         cluster_b-02
3.13.10                   -    13  10 SAS     remote      -         cluster_b-02
3.13.11                   -    13  11 SAS     remote      -         cluster_b-02
3.13.12                   -    13  12 SAS     remote      -         cluster_b-02
3.13.14                   -    13  14 SAS     remote      -         cluster_b-02
3.13.15                   -    13  15 SAS     remote      -         cluster_b-02
3.13.16                   -    13  16 SAS     remote      -         cluster_b-02
3.13.17                   -    13  17 SAS     remote      -         cluster_b-02
3.13.18                   -    13  18 SAS     remote      -         cluster_b-02
3.13.19                   -    13  19 SAS     remote      -         cluster_b-02
3.13.20                   -    13  20 SAS     remote      -         cluster_b-02
3.13.21                   -    13  21 SAS     remote      -         cluster_b-02
3.13.23                   -    13  23 SAS     remote      -         cluster_b-02
                                                                               
                     Usable           Disk    Container   Container
Disk                   Size Shelf Bay Type    Type        Name      Owner
---------------- ---------- ----- --- ------- ----------- --------- --------
4.14.0                    -    14   0 SSD     remote      -         cluster_b-02
4.14.1                    -    14   1 SSD     remote      -         cluster_b-02
4.14.2                    -    14   2 SSD     remote      -         cluster_b-02
4.14.4                    -    14   4 SSD     remote      -         cluster_b-02
4.14.6                    -    14   6 SSD     remote      -         cluster_b-02
4.14.9                    -    14   9 SSD     remote      -         cluster_b-02
4.14.16                   -    14  16 SSD     remote      -         cluster_b-02
4.14.19                   -    14  19 SSD     remote      -         cluster_b-02
4.14.20                   -    14  20 SSD     remote      -         cluster_b-02
4.14.22                   -    14  22 SSD     remote      -         cluster_b-02
4.14.23                   -    14  23 SSD     remote      -         cluster_b-02
60 entries were displayed.

cluster_a::> storage disk show -primary-port A -secondary-port A
                     Usable           Disk    Container   Container   
Disk                   Size Shelf Bay Type    Type        Name      Owner
---------------- ---------- ----- --- ------- ----------- --------- --------
1.4.0                     -     4   0 SSD     unassigned  -         -
1.4.2                     -     4   2 SSD     unassigned  -         -
1.4.3                     -     4   3 SSD     unassigned  -         -
1.4.4                     -     4   4 SSD     unassigned  -         -
1.4.5                     -     4   5 SSD     unassigned  -         -
1.4.6                     -     4   6 SSD     unassigned  -         -
1.4.7                     -     4   7 SSD     unassigned  -         -
1.4.9                     -     4   9 SSD     unassigned  -         -
1.4.10                    -     4  10 SSD     unassigned  -         -
1.4.11                    -     4  11 SSD     unassigned  -         -
1.4.12                    -     4  12 SSD     unassigned  -         -
1.4.14                    -     4  14 SSD     unassigned  -         -
1.4.15                    -     4  15 SSD     unassigned  -         -
1.4.18                    -     4  18 SSD     unassigned  -         -
1.4.19                    -     4  19 SSD     unassigned  -         -
1.4.23                    -     4  23 SSD     unassigned  -         -
2.0.1                1.63TB     0   1 SAS     aggregate   aggr0_cluster_a_01 
                                                                    cluster_a-01
2.0.2                1.63TB     0   2 SAS     aggregate   aggr0_cluster_a_01 
                                                                    cluster_a-01
2.0.3                1.63TB     0   3 SAS     aggregate   aggr1_a   cluster_a-01
2.0.7                1.63TB     0   7 SAS     aggregate   aggr1_a   cluster_a-01
2.0.8                1.63TB     0   8 SAS     aggregate   aggr1_a   cluster_a-01
2.0.13               1.63TB     0  13 SAS     aggregate   aggr1_a   cluster_a-01
2.0.14               1.63TB     0  14 SAS     aggregate   aggr1_a   cluster_a-01
2.0.18               1.63TB     0  18 SAS     aggregate   aggr1_a   cluster_a-01
2.0.19               1.63TB     0  19 SAS     aggregate   aggr1_a   cluster_a-01
2.0.20               1.63TB     0  20 SAS     aggregate   aggr1_a   cluster_a-01
2.0.21               1.63TB     0  21 SAS     aggregate   aggr1_a   cluster_a-01
2.0.23               1.63TB     0  23 SAS     spare       Pool0     cluster_a-01
2.1.0                1.63TB     1   0 SAS     aggregate   aggr0_cluster_a_02 
                                                                    cluster_a-02
2.1.1                1.63TB     1   1 SAS     aggregate   aggr0_cluster_a_02 
                                                                    cluster_a-02
2.1.6                1.63TB     1   6 SAS     aggregate   aggr2_a   cluster_a-02
2.1.7                1.63TB     1   7 SAS     aggregate   aggr2_a   cluster_a-02
2.1.9                1.63TB     1   9 SAS     aggregate   aggr2_a   cluster_a-02
2.1.11               1.63TB     1  11 SAS     aggregate   aggr2_a   cluster_a-02
2.1.12               1.63TB     1  12 SAS     aggregate   aggr2_a   cluster_a-02
2.1.14               1.63TB     1  14 SAS     aggregate   aggr2_a   cluster_a-02
2.1.20               1.63TB     1  20 SAS     aggregate   aggr2_a   cluster_a-02
2.1.21               1.63TB     1  21 SAS     aggregate   aggr2_a   cluster_a-02
2.1.22               1.63TB     1  22 SAS     aggregate   aggr2_a   cluster_a-02
2.1.23               1.63TB     1  23 SAS     spare       Pool0     cluster_a-02
2.2.0                     -     2   0 SAS     remote      -         cluster_b-01
2.2.2                     -     2   2 SAS     remote      -         cluster_b-01
2.2.3                     -     2   3 SAS     remote      -         cluster_b-01
2.2.4                     -     2   4 SAS     remote      -         cluster_b-01
2.2.6                     -     2   6 SAS     remote      -         cluster_b-01
2.2.8                     -     2   8 SAS     remote      -         cluster_b-01
2.2.10                    -     2  10 SAS     remote      -         cluster_b-01
2.2.12                    -     2  12 SAS     remote      -         cluster_b-01
                                                                               
                     Usable           Disk    Container   Container
Disk                   Size Shelf Bay Type    Type        Name      Owner
---------------- ---------- ----- --- ------- ----------- --------- --------
2.2.13                    -     2  13 SAS     remote      -         cluster_b-01
2.2.15                    -     2  15 SAS     remote      -         cluster_b-01
2.2.16                    -     2  16 SAS     remote      -         cluster_b-01
2.2.18                    -     2  18 SAS     remote      -         cluster_b-01
2.2.19                    -     2  19 SAS     remote      -         cluster_b-01
2.2.21                    -     2  21 SAS     remote      -         cluster_b-01
2.2.22                    -     2  22 SAS     remote      -         cluster_b-01
2.2.23                    -     2  23 SAS     remote      -         cluster_b-01
2.3.21                    -     3  21 SAS     remote      -         cluster_b-02
57 entries were displayed.

So, as you can see, according this output, i have 60 disks with both B paths as primary and secondary and 57 disks with both A paths as primary and secondary. Does that mean that the system is not redundant? I thought that all the disks should have A and B paths as primary and secondary, but if the "metrocluster check show" shows that everything is ok, my assumption is wrong. So, could you give me any comments about the output of the "storage disk show"?

16 REPLIES 16

kahuna
7,793 Views

can you provide the output of this:

 

::> storage disk show -fields secondary-port,primary-port,initiator,nodelist

iUser
7,783 Views

Here:

 

cluster_a::> storage disk show -fields secondary-port,primary-port,initiator,nodelist
disk    initiator               nodelist                                                                                                primary-port secondary-port 
------- ----------------------- ------------------------------------------------------------------------------------------------------- ------------ -------------- 
1.4.0   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.1   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
1.4.2   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.3   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.4   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.5   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.6   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.7   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.8   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.9   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.10  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.11  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.12  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.13  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
1.4.14  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.15  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.16  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.17  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.18  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.19  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.20  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
1.4.21  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.22  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
1.4.23  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.0.0   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            A              
2.0.1   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            A              
2.0.2   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            A              
2.0.3   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            A              
2.0.4   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            A              
2.0.5   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            A              
2.0.6   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            A              
2.0.7   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            A              
2.0.8   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            A              
2.0.9   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            A              
2.0.10  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            A              
2.0.11  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            A              
2.0.12  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            A              
2.0.13  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            A              
2.0.14  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            A              
2.0.15  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            A              
2.0.16  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            A              
2.0.17  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            A              
2.0.18  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            A              
2.0.19  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            A              
2.0.20  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            A              
2.0.21  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            A              
2.0.22  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            A              
2.0.23  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            A              
2.1.0   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.1.1   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.1.2   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.1.3   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.1.4   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.1.5   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.1.6   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.1.7   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.1.8   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.1.9   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.1.10  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.1.11  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.1.12  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.1.13  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.1.14  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.1.15  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.1.16  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.1.17  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.1.18  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.1.19  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.1.20  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.1.21  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.1.22  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.1.23  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.0   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.1   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.2   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.3   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.4   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.5   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.2.6   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.7   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.2.8   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.9   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.2.10  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.11  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.2.12  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.13  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.14  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.2.15  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.16  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.17  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.2.18  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.19  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.20  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.2.21  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.22  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.2.23  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.3.0   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.1   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.2   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.3   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.4   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.5   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.6   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.7   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.8   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.9   1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.10  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.11  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.12  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.13  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.14  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.15  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.16  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.17  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.18  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.19  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.20  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.21  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            A              
2.3.22  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
2.3.23  1c,1d,1a,1b,1c,1d,1a,1b cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            A              
3.10.0  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            B              
3.10.1  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            B              
3.10.2  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            B              
3.10.3  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            B              
3.10.4  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            B              
3.10.5  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            B              
3.10.6  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            B              
3.10.7  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            B              
3.10.8  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            B              
3.10.9  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            B              
3.10.10 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            B              
3.10.11 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            B              
3.10.12 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            B              
3.10.13 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            B              
3.10.14 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            B              
3.10.15 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            B              
3.10.16 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            B              
3.10.17 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            B              
3.10.18 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            B              
3.10.19 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            B              
3.10.20 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            B              
3.10.21 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 A            B              
3.10.22 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            B              
3.10.23 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02 B            B              
3.11.0  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.11.1  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.11.2  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.11.3  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.11.4  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.11.5  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.11.6  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.11.7  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.11.8  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.11.9  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.11.10 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.11.11 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.11.12 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.11.13 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.11.14 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.11.15 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.11.16 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.11.17 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.11.18 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.11.19 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.11.20 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.11.21 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.11.22 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.11.23 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.12.0  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.12.1  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.12.2  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.12.3  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.12.4  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.12.5  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.12.6  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.12.7  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.12.8  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.12.9  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.12.10 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.12.11 1a,1b,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.12.12 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.12.13 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.12.14 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.12.15 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.12.16 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.12.17 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.12.18 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.12.19 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.12.20 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.12.21 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.12.22 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.12.23 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.13.0  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.13.1  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.13.2  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.13.3  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.13.4  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.13.5  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.13.6  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.13.7  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.13.8  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.13.9  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.13.10 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.13.11 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.13.12 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.13.13 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.13.14 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.13.15 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.13.16 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.13.17 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.13.18 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.13.19 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.13.20 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.13.21 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
3.13.22 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
3.13.23 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
4.14.0  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
4.14.1  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
4.14.2  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
4.14.3  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
4.14.4  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
4.14.5  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
4.14.6  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
4.14.7  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
4.14.8  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
4.14.9  1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
4.14.10 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
4.14.11 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
4.14.12 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
4.14.13 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
4.14.14 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
4.14.15 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
4.14.16 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
4.14.17 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
4.14.18 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
4.14.19 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
4.14.20 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
4.14.21 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 A            B              
4.14.22 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
4.14.23 1b,1a,1d,1c,1b,1a,1d,1c cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-02,cluster_a-01,cluster_a-01,cluster_a-01,cluster_a-01 B            B              
240 entries were displayed.

cluster_a::> 

Ontapforrum
7,724 Views

Something don't look alright in the output you have shared (Just from regular HA perspective). Probably, metrocluster check show command doesn't even meant to check the disk paths and therefore says ok.

 

Using one example:

For Shelf 4, only BAYS:1,13,20 shows dual-paths [Pri:A, Sec:B], rest all the Bays are showing Prim & Sec as 'A'. Could you ask someone to take a physical inspection of shelf, and ensure the cabling is done correctly ?

 

Could you simply run this command and share the output.
::> storage disk show -port

 

Thanks!

iUser
7,686 Views

Hi!

I agree with you that something is wrong here, but i just don't understand, what. Let me explain: there're two stacks, on the each side, one with four shelfs with hdd drives, second one with single ssd shelf. I checked the cabling and it's done correctly, just exactly as in the metrocluster installation documentation (there's  an example with two stacks - just as here).

I've already tried to find some information from the "storage disk show -port", but it looks abcolutely illogical for me. In the output, as you can see, the ssds, which are located in one shelf (well, I hope, that they are, according netapp shelf partnumbers 🙂 ), some have A-A paths, some B-A *(disks 1.4.0-1.4.23) and the same is for the second shelf with ssd drives: some have A-B and some have B-B (4.14.0-4.14.23). Just in theory, how this could happen?

 

Here's the output you asked:

 

cluster_a::> storage disk show -port
Primary         Port Secondary       Port Type   Shelf Bay            
--------------- ---- --------------- ---- ------ ----- ---
1.4.0           A    cluster_a-01:netapp-sansw-1b:1-9.126L501 A SSD 4 0
1.4.1           A    cluster_a-01:netapp-sansw-1a:1-9.126L502 A SSD 4 1
1.4.2           B    cluster_a-01:netapp-sansw-1b:1-9.126L503 A SSD 4 2
1.4.3           B    cluster_a-01:netapp-sansw-1b:1-9.126L504 A SSD 4 3
1.4.4           B    cluster_a-01:netapp-sansw-1b:1-9.126L505 A SSD 4 4
1.4.5           B    cluster_a-01:netapp-sansw-1b:1-9.126L506 A SSD 4 5
1.4.6           B    cluster_a-01:netapp-sansw-1b:1-9.126L507 A SSD 4 6
1.4.7           B    cluster_a-01:netapp-sansw-1b:1-9.126L508 A SSD 4 7
1.4.8           A    cluster_a-01:netapp-sansw-1b:1-9.126L509 A SSD 4 8
1.4.9           A    cluster_a-01:netapp-sansw-1a:1-9.126L510 A SSD 4 9
1.4.10          A    cluster_a-01:netapp-sansw-1a:1-9.126L511 A SSD 4 10
1.4.11          B    cluster_a-01:netapp-sansw-1b:1-9.126L512 A SSD 4 11
1.4.12          B    cluster_a-01:netapp-sansw-1b:1-9.126L513 A SSD 4 12
1.4.13          A    cluster_a-01:netapp-sansw-1a:1-9.126L514 A SSD 4 13
1.4.14          A    cluster_a-01:netapp-sansw-1a:1-9.126L515 A SSD 4 14
1.4.15          A    cluster_a-01:netapp-sansw-1a:1-9.126L516 A SSD 4 15
1.4.16          B    cluster_a-01:netapp-sansw-1b:1-9.126L517 A SSD 4 16
1.4.17          A    cluster_a-01:netapp-sansw-1a:1-9.126L518 A SSD 4 17
1.4.18          B    cluster_a-01:netapp-sansw-1b:1-9.126L519 A SSD 4 18
1.4.19          A    cluster_a-01:netapp-sansw-1a:1-9.126L520 A SSD 4 19
1.4.20          A    cluster_a-01:netapp-sansw-1a:1-9.126L521 A SSD 4 20
1.4.21          A    cluster_a-01:netapp-sansw-1a:1-9.126L522 A SSD 4 21
1.4.22          A    cluster_a-01:netapp-sansw-1a:1-9.126L523 A SSD 4 22
1.4.23          B    cluster_a-01:netapp-sansw-1b:1-9.126L524 A SSD 4 23
2.0.0           A    cluster_a-01:netapp-sansw-1b:1-9.126L1 A SAS 0 0
2.0.1           B    cluster_a-01:netapp-sansw-1b:1-9.126L2 A SAS 0 1
2.0.2           A    cluster_a-01:netapp-sansw-1a:1-9.126L3 A SAS 0 2
2.0.3           B    cluster_a-01:netapp-sansw-1b:1-9.126L4 A SAS 0 3
2.0.4           A    cluster_a-01:netapp-sansw-1a:1-9.126L5 A SAS 0 4
2.0.5           B    cluster_a-01:netapp-sansw-1b:1-9.126L6 A SAS 0 5
2.0.6           B    cluster_a-01:netapp-sansw-1b:1-9.126L7 A SAS 0 6
2.0.7           B    cluster_a-01:netapp-sansw-1b:1-9.126L8 A SAS 0 7
2.0.8           A    cluster_a-01:netapp-sansw-1a:1-9.126L9 A SAS 0 8
2.0.9           A    cluster_a-01:netapp-sansw-1b:1-9.126L10 A SAS 0 9
2.0.10          B    cluster_a-01:netapp-sansw-1b:1-9.126L11 A SAS 0 10
2.0.11          A    cluster_a-01:netapp-sansw-1b:1-9.126L12 A SAS 0 11
2.0.12          A    cluster_a-01:netapp-sansw-1b:1-9.126L13 A SAS 0 12
2.0.13          B    cluster_a-01:netapp-sansw-1b:1-9.126L14 A SAS 0 13
2.0.14          A    cluster_a-01:netapp-sansw-1b:1-9.126L15 A SAS 0 14
2.0.15          A    cluster_a-01:netapp-sansw-1a:1-9.126L16 A SAS 0 15
2.0.16          A    cluster_a-01:netapp-sansw-1b:1-9.126L17 A SAS 0 16
2.0.17          B    cluster_a-01:netapp-sansw-1b:1-9.126L18 A SAS 0 17
2.0.18          B    cluster_a-01:netapp-sansw-1b:1-9.126L19 A SAS 0 18
2.0.19          B    cluster_a-01:netapp-sansw-1b:1-9.126L20 A SAS 0 19
2.0.20          B    cluster_a-01:netapp-sansw-1b:1-9.126L21 A SAS 0 20
2.0.21          A    cluster_a-01:netapp-sansw-1a:1-9.126L22 A SAS 0 21
2.0.22          A    cluster_a-01:netapp-sansw-1a:1-9.126L23 A SAS 0 22
2.0.23          B    cluster_a-01:netapp-sansw-1b:1-9.126L24 A SAS 0 23
2.1.0           B    cluster_a-02:netapp-sansw-1b:1-9.126L27 A SAS 1 0
2.1.1           B    cluster_a-02:netapp-sansw-1b:1-9.126L28 A SAS 1 1
2.1.2           B    cluster_a-02:netapp-sansw-1b:1-9.126L29 A SAS 1 2
2.1.3           A    cluster_a-02:netapp-sansw-1b:1-9.126L30 A SAS 1 3
2.1.4           B    cluster_a-02:netapp-sansw-1b:1-9.126L31 A SAS 1 4
2.1.5           A    cluster_a-02:netapp-sansw-1a:1-9.126L32 A SAS 1 5
2.1.6           A    cluster_a-02:netapp-sansw-1b:1-9.126L33 A SAS 1 6
2.1.7           B    cluster_a-02:netapp-sansw-1b:1-9.126L34 A SAS 1 7
2.1.8           B    cluster_a-02:netapp-sansw-1b:1-9.126L35 A SAS 1 8
2.1.9           A    cluster_a-02:netapp-sansw-1b:1-9.126L36 A SAS 1 9
2.1.10          B    cluster_a-02:netapp-sansw-1b:1-9.126L37 A SAS 1 10
2.1.11          A    cluster_a-02:netapp-sansw-1a:1-9.126L38 A SAS 1 11
2.1.12          B    cluster_a-02:netapp-sansw-1b:1-9.126L39 A SAS 1 12
2.1.13          A    cluster_a-02:netapp-sansw-1b:1-9.126L40 A SAS 1 13
2.1.14          A    cluster_a-02:netapp-sansw-1a:1-9.126L41 A SAS 1 14
2.1.15          A    cluster_a-02:netapp-sansw-1b:1-9.126L42 A SAS 1 15
2.1.16          B    cluster_a-02:netapp-sansw-1b:1-9.126L43 A SAS 1 16
2.1.17          B    cluster_a-02:netapp-sansw-1b:1-9.126L44 A SAS 1 17
2.1.18          A    cluster_a-02:netapp-sansw-1b:1-9.126L45 A SAS 1 18
2.1.19          A    cluster_a-02:netapp-sansw-1a:1-9.126L46 A SAS 1 19
2.1.20          A    cluster_a-02:netapp-sansw-1a:1-9.126L47 A SAS 1 20
2.1.21          B    cluster_a-02:netapp-sansw-1b:1-9.126L48 A SAS 1 21
2.1.22          B    cluster_a-02:netapp-sansw-1b:1-9.126L49 A SAS 1 22
2.1.23          A    cluster_a-02:netapp-sansw-1b:1-9.126L50 A SAS 1 23
2.2.0           A    cluster_a-01:netapp-sansw-1a:1-9.126L53 A SAS 2 0
2.2.1           B    cluster_a-01:netapp-sansw-1b:1-9.126L54 A SAS 2 1
2.2.2           A    cluster_a-01:netapp-sansw-1a:1-9.126L55 A SAS 2 2
2.2.3           A    cluster_a-01:netapp-sansw-1a:1-9.126L56 A SAS 2 3
2.2.4           A    cluster_a-01:netapp-sansw-1a:1-9.126L57 A SAS 2 4
2.2.5           B    cluster_a-01:netapp-sansw-1b:1-9.126L58 A SAS 2 5
2.2.6           B    cluster_a-01:netapp-sansw-1b:1-9.126L59 A SAS 2 6
2.2.7           B    cluster_a-01:netapp-sansw-1b:1-9.126L60 A SAS 2 7
2.2.8           B    cluster_a-01:netapp-sansw-1b:1-9.126L61 A SAS 2 8
2.2.9           B    cluster_a-01:netapp-sansw-1b:1-9.126L62 A SAS 2 9
2.2.10          A    cluster_a-01:netapp-sansw-1a:1-9.126L63 A SAS 2 10
2.2.11          B    cluster_a-01:netapp-sansw-1b:1-9.126L64 A SAS 2 11
2.2.12          A    cluster_a-01:netapp-sansw-1a:1-9.126L65 A SAS 2 12
2.2.13          A    cluster_a-01:netapp-sansw-1a:1-9.126L66 A SAS 2 13
2.2.14          B    cluster_a-01:netapp-sansw-1b:1-9.126L67 A SAS 2 14
2.2.15          A    cluster_a-01:netapp-sansw-1b:1-9.126L68 A SAS 2 15
2.2.16          A    cluster_a-01:netapp-sansw-1a:1-9.126L69 A SAS 2 16
2.2.17          B    cluster_a-01:netapp-sansw-1b:1-9.126L70 A SAS 2 17
2.2.18          A    cluster_a-01:netapp-sansw-1b:1-9.126L71 A SAS 2 18
2.2.19          A    cluster_a-01:netapp-sansw-1b:1-9.126L72 A SAS 2 19
2.2.20          B    cluster_a-01:netapp-sansw-1b:1-9.126L73 A SAS 2 20
2.2.21          A    cluster_a-01:netapp-sansw-1b:1-9.126L74 A SAS 2 21
2.2.22          A    cluster_a-01:netapp-sansw-1b:1-9.126L75 A SAS 2 22
2.2.23          A    cluster_a-01:netapp-sansw-1a:1-9.126L76 A SAS 2 23
2.3.0           B    cluster_a-01:netapp-sansw-1b:1-9.126L79 A SAS 3 0
2.3.1           B    cluster_a-01:netapp-sansw-1b:1-9.126L80 A SAS 3 1
2.3.2           B    cluster_a-01:netapp-sansw-1b:1-9.126L81 A SAS 3 2
2.3.3           B    cluster_a-01:netapp-sansw-1b:1-9.126L82 A SAS 3 3
2.3.4           B    cluster_a-01:netapp-sansw-1b:1-9.126L83 A SAS 3 4
2.3.5           B    cluster_a-01:netapp-sansw-1b:1-9.126L84 A SAS 3 5
2.3.6           A    cluster_a-01:netapp-sansw-1b:1-9.126L85 A SAS 3 6
2.3.7           B    cluster_a-01:netapp-sansw-1b:1-9.126L86 A SAS 3 7
2.3.8           B    cluster_a-01:netapp-sansw-1b:1-9.126L87 A SAS 3 8
2.3.9           B    cluster_a-01:netapp-sansw-1b:1-9.126L88 A SAS 3 9
2.3.10          B    cluster_a-01:netapp-sansw-1b:1-9.126L89 A SAS 3 10
2.3.11          B    cluster_a-01:netapp-sansw-1b:1-9.126L90 A SAS 3 11
2.3.12          A    cluster_a-01:netapp-sansw-1b:1-9.126L91 A SAS 3 12
2.3.13          A    cluster_a-01:netapp-sansw-1a:1-9.126L92 A SAS 3 13
2.3.14          B    cluster_a-01:netapp-sansw-1b:1-9.126L93 A SAS 3 14
2.3.15          A    cluster_a-01:netapp-sansw-1a:1-9.126L94 A SAS 3 15
2.3.16          B    cluster_a-01:netapp-sansw-1b:1-9.126L95 A SAS 3 16
2.3.17          B    cluster_a-01:netapp-sansw-1b:1-9.126L96 A SAS 3 17
2.3.18          B    cluster_a-01:netapp-sansw-1b:1-9.126L97 A SAS 3 18
2.3.19          B    cluster_a-01:netapp-sansw-1b:1-9.126L98 A SAS 3 19
2.3.20          B    cluster_a-01:netapp-sansw-1b:1-9.126L99 A SAS 3 20
2.3.21          A    cluster_a-01:netapp-sansw-1b:1-9.126L100 A SAS 3 21
2.3.22          B    cluster_a-01:netapp-sansw-1b:1-9.126L101 A SAS 3 22
2.3.23          B    cluster_a-01:netapp-sansw-1b:1-9.126L102 A SAS 3 23
3.10.0          B    cluster_a-01:netapp-sansw-2b:1-10.126L79 B SAS 10 0
3.10.1          B    cluster_a-01:netapp-sansw-2a:1-10.126L80 B SAS 10 1
3.10.2          A    cluster_a-01:netapp-sansw-2a:1-10.126L81 B SAS 10 2
3.10.3          A    cluster_a-01:netapp-sansw-2a:1-10.126L82 B SAS 10 3
3.10.4          B    cluster_a-01:netapp-sansw-2a:1-10.126L83 B SAS 10 4
3.10.5          B    cluster_a-01:netapp-sansw-2b:1-10.126L84 B SAS 10 5
3.10.6          A    cluster_a-01:netapp-sansw-2a:1-10.126L85 B SAS 10 6
3.10.7          B    cluster_a-01:netapp-sansw-2a:1-10.126L86 B SAS 10 7
3.10.8          A    cluster_a-01:netapp-sansw-2a:1-10.126L87 B SAS 10 8
3.10.9          A    cluster_a-01:netapp-sansw-2a:1-10.126L88 B SAS 10 9
3.10.10         B    cluster_a-01:netapp-sansw-2b:1-10.126L89 B SAS 10 10
3.10.11         A    cluster_a-01:netapp-sansw-2a:1-10.126L90 B SAS 10 11
3.10.12         A    cluster_a-01:netapp-sansw-2a:1-10.126L91 B SAS 10 12
3.10.13         A    cluster_a-01:netapp-sansw-2a:1-10.126L92 B SAS 10 13
3.10.14         A    cluster_a-01:netapp-sansw-2a:1-10.126L93 B SAS 10 14
3.10.15         B    cluster_a-01:netapp-sansw-2b:1-10.126L94 B SAS 10 15
3.10.16         B    cluster_a-01:netapp-sansw-2a:1-10.126L95 B SAS 10 16
3.10.17         A    cluster_a-01:netapp-sansw-2a:1-10.126L96 B SAS 10 17
3.10.18         A    cluster_a-01:netapp-sansw-2a:1-10.126L97 B SAS 10 18
3.10.19         A    cluster_a-01:netapp-sansw-2a:1-10.126L98 B SAS 10 19
3.10.20         A    cluster_a-01:netapp-sansw-2a:1-10.126L99 B SAS 10 20
3.10.21         A    cluster_a-01:netapp-sansw-2a:1-10.126L100 B SAS 10 21
3.10.22         A    cluster_a-01:netapp-sansw-2a:1-10.126L101 B SAS 10 22
3.10.23         B    cluster_a-01:netapp-sansw-2b:1-10.126L102 B SAS 10 23
3.11.0          A    cluster_a-02:netapp-sansw-2a:1-10.126L53 B SAS 11 0
3.11.1          A    cluster_a-02:netapp-sansw-2a:1-10.126L54 B SAS 11 1
3.11.2          B    cluster_a-02:netapp-sansw-2b:1-10.126L55 B SAS 11 2
3.11.3          A    cluster_a-02:netapp-sansw-2a:1-10.126L56 B SAS 11 3
3.11.4          A    cluster_a-02:netapp-sansw-2a:1-10.126L57 B SAS 11 4
3.11.5          B    cluster_a-02:netapp-sansw-2b:1-10.126L58 B SAS 11 5
3.11.6          B    cluster_a-02:netapp-sansw-2b:1-10.126L59 B SAS 11 6
3.11.7          A    cluster_a-02:netapp-sansw-2a:1-10.126L60 B SAS 11 7
3.11.8          B    cluster_a-02:netapp-sansw-2a:1-10.126L61 B SAS 11 8
3.11.9          A    cluster_a-02:netapp-sansw-2a:1-10.126L62 B SAS 11 9
3.11.10         B    cluster_a-02:netapp-sansw-2a:1-10.126L63 B SAS 11 10
3.11.11         B    cluster_a-02:netapp-sansw-2a:1-10.126L64 B SAS 11 11
3.11.12         B    cluster_a-02:netapp-sansw-2a:1-10.126L65 B SAS 11 12
3.11.13         B    cluster_a-02:netapp-sansw-2b:1-10.126L66 B SAS 11 13
3.11.14         B    cluster_a-02:netapp-sansw-2a:1-10.126L67 B SAS 11 14
3.11.15         A    cluster_a-02:netapp-sansw-2a:1-10.126L68 B SAS 11 15
3.11.16         A    cluster_a-02:netapp-sansw-2a:1-10.126L69 B SAS 11 16
3.11.17         A    cluster_a-02:netapp-sansw-2a:1-10.126L70 B SAS 11 17
3.11.18         B    cluster_a-02:netapp-sansw-2b:1-10.126L71 B SAS 11 18
3.11.19         B    cluster_a-02:netapp-sansw-2b:1-10.126L72 B SAS 11 19
3.11.20         A    cluster_a-02:netapp-sansw-2a:1-10.126L73 B SAS 11 20
3.11.21         B    cluster_a-02:netapp-sansw-2a:1-10.126L74 B SAS 11 21
3.11.22         B    cluster_a-02:netapp-sansw-2a:1-10.126L75 B SAS 11 22
3.11.23         B    cluster_a-02:netapp-sansw-2a:1-10.126L76 B SAS 11 23
3.12.0          B    cluster_a-01:netapp-sansw-2b:1-10.126L27 B SAS 12 0
3.12.1          B    cluster_a-01:netapp-sansw-2a:1-10.126L28 B SAS 12 1
3.12.2          B    cluster_a-01:netapp-sansw-2a:1-10.126L29 B SAS 12 2
3.12.3          A    cluster_a-01:netapp-sansw-2a:1-10.126L30 B SAS 12 3
3.12.4          A    cluster_a-01:netapp-sansw-2a:1-10.126L31 B SAS 12 4
3.12.5          A    cluster_a-01:netapp-sansw-2a:1-10.126L32 B SAS 12 5
3.12.6          A    cluster_a-01:netapp-sansw-2a:1-10.126L33 B SAS 12 6
3.12.7          A    cluster_a-01:netapp-sansw-2a:1-10.126L34 B SAS 12 7
3.12.8          B    cluster_a-01:netapp-sansw-2b:1-10.126L35 B SAS 12 8
3.12.9          A    cluster_a-01:netapp-sansw-2a:1-10.126L36 B SAS 12 9
3.12.10         A    cluster_a-01:netapp-sansw-2a:1-10.126L37 B SAS 12 10
3.12.11         B    cluster_a-01:netapp-sansw-2b:1-10.126L38 B SAS 12 11
3.12.12         B    cluster_a-01:netapp-sansw-2a:1-10.126L39 B SAS 12 12
3.12.13         B    cluster_a-01:netapp-sansw-2b:1-10.126L40 B SAS 12 13
3.12.14         B    cluster_a-01:netapp-sansw-2b:1-10.126L41 B SAS 12 14
3.12.15         A    cluster_a-01:netapp-sansw-2a:1-10.126L42 B SAS 12 15
3.12.16         A    cluster_a-01:netapp-sansw-2a:1-10.126L43 B SAS 12 16
3.12.17         B    cluster_a-01:netapp-sansw-2a:1-10.126L44 B SAS 12 17
3.12.18         B    cluster_a-01:netapp-sansw-2b:1-10.126L45 B SAS 12 18
3.12.19         A    cluster_a-01:netapp-sansw-2a:1-10.126L46 B SAS 12 19
3.12.20         B    cluster_a-01:netapp-sansw-2a:1-10.126L47 B SAS 12 20
3.12.21         A    cluster_a-01:netapp-sansw-2a:1-10.126L48 B SAS 12 21
3.12.22         B    cluster_a-01:netapp-sansw-2a:1-10.126L49 B SAS 12 22
3.12.23         A    cluster_a-01:netapp-sansw-2a:1-10.126L50 B SAS 12 23
3.13.0          A    cluster_a-01:netapp-sansw-2a:1-10.126L1 B SAS 13 0
3.13.1          A    cluster_a-01:netapp-sansw-2a:1-10.126L2 B SAS 13 1
3.13.2          A    cluster_a-01:netapp-sansw-2a:1-10.126L3 B SAS 13 2
3.13.3          B    cluster_a-01:netapp-sansw-2b:1-10.126L4 B SAS 13 3
3.13.4          A    cluster_a-01:netapp-sansw-2a:1-10.126L5 B SAS 13 4
3.13.5          A    cluster_a-01:netapp-sansw-2a:1-10.126L6 B SAS 13 5
3.13.6          B    cluster_a-01:netapp-sansw-2a:1-10.126L7 B SAS 13 6
3.13.7          B    cluster_a-01:netapp-sansw-2a:1-10.126L8 B SAS 13 7
3.13.8          B    cluster_a-01:netapp-sansw-2a:1-10.126L9 B SAS 13 8
3.13.9          A    cluster_a-01:netapp-sansw-2a:1-10.126L10 B SAS 13 9
3.13.10         B    cluster_a-01:netapp-sansw-2b:1-10.126L11 B SAS 13 10
3.13.11         B    cluster_a-01:netapp-sansw-2b:1-10.126L12 B SAS 13 11
3.13.12         B    cluster_a-01:netapp-sansw-2a:1-10.126L13 B SAS 13 12
3.13.13         B    cluster_a-01:netapp-sansw-2a:1-10.126L14 B SAS 13 13
3.13.14         B    cluster_a-01:netapp-sansw-2a:1-10.126L15 B SAS 13 14
3.13.15         B    cluster_a-01:netapp-sansw-2b:1-10.126L16 B SAS 13 15
3.13.16         B    cluster_a-01:netapp-sansw-2b:1-10.126L17 B SAS 13 16
3.13.17         B    cluster_a-01:netapp-sansw-2b:1-10.126L18 B SAS 13 17
3.13.18         B    cluster_a-01:netapp-sansw-2b:1-10.126L19 B SAS 13 18
3.13.19         B    cluster_a-01:netapp-sansw-2a:1-10.126L20 B SAS 13 19
3.13.20         B    cluster_a-01:netapp-sansw-2a:1-10.126L21 B SAS 13 20
3.13.21         B    cluster_a-01:netapp-sansw-2b:1-10.126L22 B SAS 13 21
3.13.22         A    cluster_a-01:netapp-sansw-2a:1-10.126L23 B SAS 13 22
3.13.23         B    cluster_a-01:netapp-sansw-2b:1-10.126L24 B SAS 13 23
4.14.0          A    cluster_a-01:netapp-sansw-2a:1-10.126L501 B SSD 14 0
4.14.1          A    cluster_a-01:netapp-sansw-2a:1-10.126L502 B SSD 14 1
4.14.2          A    cluster_a-01:netapp-sansw-2a:1-10.126L503 B SSD 14 2
4.14.3          B    cluster_a-01:netapp-sansw-2b:1-10.126L504 B SSD 14 3
4.14.4          A    cluster_a-01:netapp-sansw-2a:1-10.126L505 B SSD 14 4
4.14.5          A    cluster_a-01:netapp-sansw-2a:1-10.126L506 B SSD 14 5
4.14.6          A    cluster_a-01:netapp-sansw-2a:1-10.126L507 B SSD 14 6
4.14.7          A    cluster_a-01:netapp-sansw-2a:1-10.126L508 B SSD 14 7
4.14.8          B    cluster_a-01:netapp-sansw-2a:1-10.126L509 B SSD 14 8
4.14.9          B    cluster_a-01:netapp-sansw-2b:1-10.126L510 B SSD 14 9
4.14.10         A    cluster_a-01:netapp-sansw-2a:1-10.126L511 B SSD 14 10
4.14.11         A    cluster_a-01:netapp-sansw-2a:1-10.126L512 B SSD 14 11
4.14.12         A    cluster_a-01:netapp-sansw-2a:1-10.126L513 B SSD 14 12
4.14.13         A    cluster_a-01:netapp-sansw-2a:1-10.126L514 B SSD 14 13
4.14.14         A    cluster_a-01:netapp-sansw-2a:1-10.126L515 B SSD 14 14
4.14.15         A    cluster_a-01:netapp-sansw-2a:1-10.126L516 B SSD 14 15
4.14.16         A    cluster_a-01:netapp-sansw-2a:1-10.126L517 B SSD 14 16
4.14.17         A    cluster_a-01:netapp-sansw-2a:1-10.126L518 B SSD 14 17
4.14.18         B    cluster_a-01:netapp-sansw-2a:1-10.126L519 B SSD 14 18
4.14.19         B    cluster_a-01:netapp-sansw-2a:1-10.126L520 B SSD 14 19
4.14.20         A    cluster_a-01:netapp-sansw-2a:1-10.126L521 B SSD 14 20
4.14.21         B    cluster_a-01:netapp-sansw-2a:1-10.126L522 B SSD 14 21
4.14.22         A    cluster_a-01:netapp-sansw-2a:1-10.126L523 B SSD 14 22
4.14.23         B    cluster_a-01:netapp-sansw-2a:1-10.126L524 B SSD 14 23
240 entries were displayed.

 

And the shelf output, which is also strange (the shelf naming):

 

cluster_a::> storage shelf show 
                                                                      
                                                       Module Operational
       Shelf Name Shelf ID Serial Number   Model       Type   Status
----------------- -------- --------------- ----------- ------ -----------
              1.4        4 SHFHU1852XXXXXX DS224-12    IOM12  Normal
              2.0        0 SHFHU1849XXXXXX DS224-12    IOM12  Normal
              2.1        1 SHFHU1850XXXXXX DS224-12    IOM12  Normal
              2.2        2 SHFHU1850XXXXXX DS224-12    IOM12  Normal
              2.3        3 SHFHU1848XXXXXX DS224-12    IOM12  Normal
             3.10       10 SHFHU1848XXXXXX DS224-12    IOM12  Normal
             3.11       11 SHFHU1848XXXXXX DS224-12    IOM12  Normal
             3.12       12 SHFHU1848XXXXXX DS224-12    IOM12  Normal
             3.13       13 SHFHU1848XXXXXX DS224-12    IOM12  Normal
             4.14       14 SHFHU1852XXXXXX DS224-12    IOM12  Normal
10 entries were displayed.

cluster_a::> 

and the shelf ports

 

cluster_a::> storage shelf port show
                                                                      
Shelf ID Module State        Internal?
----- -- ------ ------------ ---------
1.4
       0 A      connected    false
       1 A      disconnected false
       2 A      disconnected false
       3 A      disconnected false
       4 B      disconnected false
       5 B      disconnected false
       6 B      connected    false
       7 B      disconnected false
2.0
       0 A      connected    false
       1 A      disconnected false
       2 A      connected    false
       3 A      disconnected false
       4 B      disconnected false
       5 B      disconnected false
       6 B      connected    false
       7 B      disconnected false
2.1
       0 A      connected    false
       1 A      disconnected false
       2 A      connected    false
       3 A      disconnected false
       4 B      connected    false
       5 B      disconnected false
       6 B      connected    false
       7 B      disconnected false
2.2
       0 A      connected    false
       1 A      disconnected false
       2 A      connected    false
       3 A      disconnected false
       4 B      connected    false
       5 B      disconnected false
       6 B      connected    false
       7 B      disconnected false
2.3
       0 A      connected    false
       1 A      disconnected false
       2 A      disconnected false
       3 A      disconnected false
       4 B      connected    false
       5 B      disconnected false
       6 B      connected    false
       7 B      disconnected false
3.10
       0 A      connected    false
       1 A      disconnected false
       2 A      connected    false
       3 A      disconnected false
       4 B      disconnected false
       5 B      disconnected false
       6 B      connected    false
       7 B      disconnected false
3.11
       0 A      connected    false
       1 A      disconnected false
       2 A      connected    false
       3 A      disconnected false
       4 B      connected    false
       5 B      disconnected false
       6 B      connected    false
       7 B      disconnected false
3.12
       0 A      connected    false
       1 A      disconnected false
       2 A      connected    false
       3 A      disconnected false
       4 B      connected    false
       5 B      disconnected false
       6 B      connected    false
       7 B      disconnected false
3.13
       0 A      connected    false
       1 A      disconnected false
       2 A      disconnected false
       3 A      disconnected false
       4 B      connected    false
       5 B      disconnected false
       6 B      connected    false
       7 B      disconnected false
4.14
       0 A      connected    false
       1 A      disconnected false
       2 A      disconnected false
       3 A      disconnected false
       4 B      disconnected false
       5 B      disconnected false
       6 B      connected    false
       7 B      disconnected false
80 entries were displayed.

cluster_a::> 

 

The question is, how exactly should this output look like, if it's cabled as it is shown in the guide? I suspect only one that could happen: after I've checked the physical cabling, there was a test during which one SAS cable was plugged out from FC-SAS bridge and then plugged in again. After that I haven't checked anything, so i guess one cable could be plugged in a wrong port. I've checked this via CLI, but all the bridges show just the same - that everything is ok:

 

cluster_a::> storage bridge run-cli -name ATTO_2000001086a9d800 -command SASPortList
[Job 581] Job is queued: Executing ATTO CLI command "SASPortList" on bridge "ATTO_200000XXXXXXXXX0"..                                          
;Connector      PHY     Link            Speed   SAS Address
;=============================================================
Device  A       1       Up              12Gb    5001086000XXXXX0
Device  A       2       Up              12Gb    5001086000XXXXX0
Device  A       3       Up              12Gb    5001086000XXXXX0
Device  A       4       Up              12Gb    5001086000XXXXX0
Device  B       1       Up              12Gb    5001086000XXXXX4
Device  B       2       Up              12Gb    5001086000XXXXX4
Device  B       3       Up              12Gb    5001086000XXXXX4
Device  B       4       Up              12Gb    5001086000XXXXX4
Device  C       1       Disabled        12Gb    5001086000XXXXX8
Device  C       2       Disabled        12Gb    5001086000XXXXX8
Device  C       3       Disabled        12Gb    5001086000XXXXX8
Device  C       4       Disabled        12Gb    5001086000XXXXX8
Device  D       1       Disabled        12Gb    5001086000XXXXXc
Device  D       2       Disabled        12Gb    5001086000XXXXXc
Device  D       3       Disabled        12Gb    5001086000XXXXXc
Device  D       4       Disabled        12Gb    5001086000XXXXXc

kahuna
7,662 Views

"some have A-B and some have B-B (4.14.0-4.14.23). Just in theory, how this could happen?"

 

It is not bad cabling. It could be either IOM or ATTO bridge that skip some of the disks. The fact that it is showing this anomaly in multiple shelves tells me that it is not an IOM issue

 

Can you provide the output of 'sysconfig -a' and 'sysconfig -r' from both nodes? (node shell)

iUser
7,655 Views

Could you please check your PM?

iUser
7,654 Views

Wops, couldn't send the PM, because the length of output (500-600kb of text)... There's a lot of serial numbers there, i'm not sure that it's a good idea to post it here 😞

iUser
7,644 Views

Looks like the situation is not good, so i opened a support case. I hope that helps... Thanks for your answers!

Ontapforrum
7,605 Views

Will be keen to know what is the cause here, Please do let us know later.

 

As far as the storage shelf output is concerend, it looks fine. You mentioned you have 2 stacks on each side, so based on the shelf naming I am assuming it is in this order.

 


Node1:
Stack 1: shelf 4 [1.4]
Stack 2: Shelf 0,1,2,3 [2.0,2.1,2.2,2.3]

 

Node2:
Stack 3: Shelf 10,11,12,13 [3.10,3.11,3.12,3.13]
Stack 4: Shelf 14 [4.14]

iUser
7,059 Views

As soon as i'll solve this case, i'll write here the detailed report. According the shelves: physically, they're placed identically on both sites: four hdd-only shelves from the top, then ssd shelf.

Ontapforrum
7,031 Views

Found a defect (Could be your case or may be not but let us know)

 

Limited number of SAS drives programming defect resulted in the reporting of port "A" on both primary and secondary ports.
https://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=472649

 

To verify:

::> node run -node NODE-01
NODE-01> storage show disk -p
PRIMARY PORT SECONDARY PORT SHELF BAY
-------- ---- --------- ---- ---------
1d.00.4 A 1b.00.4 A 0 4

Do you see PORT A for PRI & SEC for some disks.

 

Workaround suggested in the BUG : After confirming there are no multipath cabling problems, replace the drive to resolve this issue.

 

Hopefully you will have more information from NetApp support.

iUser
6,992 Views

I've updated the firmware on the bridges, but i've found a very strange thing - either different commands show me different outputs, or I just don't understand what they show me :). Here's two outputs, according the first one, everything is ok, and all the disks have two paths - A and B, but according the another one - some disks have two identical paths - A-A or A-B:

 

cluster_a::> node run -node cluster_a-01                        
Type 'exit' or 'Ctrl-D' to return to the CLI
cluster_a-01> storage show disk -p
PRIMARY                      PORT  SECONDARY                    PORT SHELF BAY
---------------------------- ----  ---------------------------- ---- ---------
netapp-sansw-1a:1-9.126L1     A    netapp-sansw-1a:1-10.126L79   B     0    0 
netapp-sansw-1a:1-9.126L2     A    netapp-sansw-1a:1-10.126L80   B     0    1 
netapp-sansw-1b:1-9.126L3     A    netapp-sansw-1a:1-10.126L81   B     0    2 
netapp-sansw-1b:1-10.126L82   B    netapp-sansw-1b:1-9.126L4     A     0    3 
netapp-sansw-1a:1-10.126L83   B    netapp-sansw-1b:1-9.126L5     A     0    4 
netapp-sansw-1a:1-10.126L84   B    netapp-sansw-1b:1-9.126L6     A     0    5 
netapp-sansw-1b:1-10.126L85   B    netapp-sansw-1b:1-9.126L7     A     0    6 
netapp-sansw-1a:1-10.126L86   B    netapp-sansw-1b:1-9.126L8     A     0    7 
netapp-sansw-1b:1-9.126L9     A    netapp-sansw-1a:1-10.126L87   B     0    8 
netapp-sansw-1a:1-9.126L10    A    netapp-sansw-1a:1-10.126L88   B     0    9 
netapp-sansw-1b:1-9.126L11    A    netapp-sansw-1a:1-10.126L89   B     0   10 
netapp-sansw-1a:1-10.126L90   B    netapp-sansw-1b:1-9.126L12    A     0   11 
netapp-sansw-1a:1-10.126L91   B    netapp-sansw-1b:1-9.126L13    A     0   12 
netapp-sansw-1b:1-9.126L14    A    netapp-sansw-1a:1-10.126L92   B     0   13 
netapp-sansw-1b:1-9.126L15    A    netapp-sansw-1a:1-10.126L93   B     0   14 
netapp-sansw-1a:1-10.126L94   B    netapp-sansw-1b:1-9.126L16    A     0   15 
netapp-sansw-1a:1-9.126L17    A    netapp-sansw-1a:1-10.126L95   B     0   16 
netapp-sansw-1a:1-9.126L18    A    netapp-sansw-1a:1-10.126L96   B     0   17 
netapp-sansw-1b:1-10.126L97   B    netapp-sansw-1b:1-9.126L19    A     0   18 
netapp-sansw-1b:1-10.126L98   B    netapp-sansw-1b:1-9.126L20    A     0   19 
netapp-sansw-1b:1-10.126L99   B    netapp-sansw-1b:1-9.126L21    A     0   20 
netapp-sansw-1b:1-9.126L22    A    netapp-sansw-1a:1-10.126L100  B     0   21 
netapp-sansw-1a:1-10.126L101  B    netapp-sansw-1b:1-9.126L23    A     0   22 
netapp-sansw-1a:1-9.126L24    A    netapp-sansw-1a:1-10.126L102  B     0   23 
netapp-sansw-1a:1-10.126L53   B    netapp-sansw-1b:1-9.126L27    A     1    0 
netapp-sansw-1a:1-10.126L54   B    netapp-sansw-1b:1-9.126L28    A     1    1 
netapp-sansw-1a:1-10.126L55   B    netapp-sansw-1b:1-9.126L29    A     1    2 
netapp-sansw-1b:1-9.126L30    A    netapp-sansw-1a:1-10.126L56   B     1    3 
netapp-sansw-1a:1-9.126L31    A    netapp-sansw-1a:1-10.126L57   B     1    4 
netapp-sansw-1b:1-9.126L32    A    netapp-sansw-1a:1-10.126L58   B     1    5 
netapp-sansw-1a:1-10.126L59   B    netapp-sansw-1b:1-9.126L33    A     1    6 
netapp-sansw-1a:1-10.126L60   B    netapp-sansw-1b:1-9.126L34    A     1    7 
netapp-sansw-1a:1-10.126L61   B    netapp-sansw-1b:1-9.126L35    A     1    8 
netapp-sansw-1a:1-9.126L36    A    netapp-sansw-1a:1-10.126L62   B     1    9 
netapp-sansw-1b:1-9.126L37    A    netapp-sansw-1a:1-10.126L63   B     1   10 
netapp-sansw-1a:1-10.126L64   B    netapp-sansw-1b:1-9.126L38    A     1   11 
netapp-sansw-1a:1-9.126L39    A    netapp-sansw-1a:1-10.126L65   B     1   12 
netapp-sansw-1a:1-10.126L66   B    netapp-sansw-1b:1-9.126L40    A     1   13 
netapp-sansw-1b:1-10.126L67   B    netapp-sansw-1b:1-9.126L41    A     1   14 
netapp-sansw-1a:1-10.126L68   B    netapp-sansw-1b:1-9.126L42    A     1   15 
netapp-sansw-1b:1-10.126L69   B    netapp-sansw-1b:1-9.126L43    A     1   16 
netapp-sansw-1a:1-10.126L70   B    netapp-sansw-1b:1-9.126L44    A     1   17 
netapp-sansw-1b:1-10.126L71   B    netapp-sansw-1b:1-9.126L45    A     1   18 
netapp-sansw-1b:1-10.126L72   B    netapp-sansw-1b:1-9.126L46    A     1   19 
netapp-sansw-1a:1-10.126L73   B    netapp-sansw-1b:1-9.126L47    A     1   20 
netapp-sansw-1b:1-10.126L74   B    netapp-sansw-1b:1-9.126L48    A     1   21 
netapp-sansw-1a:1-10.126L75   B    netapp-sansw-1b:1-9.126L49    A     1   22 
netapp-sansw-1b:1-10.126L76   B    netapp-sansw-1b:1-9.126L50    A     1   23 
netapp-sansw-1b:1-9.126L53    A    netapp-sansw-1a:1-10.126L27   B     2    0 
netapp-sansw-1a:1-9.126L54    A    netapp-sansw-1a:1-10.126L28   B     2    1 
netapp-sansw-1b:1-9.126L55    A    netapp-sansw-1a:1-10.126L29   B     2    2 
netapp-sansw-1b:1-9.126L56    A    netapp-sansw-1a:1-10.126L30   B     2    3 
netapp-sansw-1a:1-9.126L57    A    netapp-sansw-1a:1-10.126L31   B     2    4 
netapp-sansw-1a:1-10.126L32   B    netapp-sansw-1b:1-9.126L58    A     2    5 
netapp-sansw-1b:1-9.126L59    A    netapp-sansw-1a:1-10.126L33   B     2    6 
netapp-sansw-1a:1-9.126L60    A    netapp-sansw-1a:1-10.126L34   B     2    7 
netapp-sansw-1a:1-10.126L35   B    netapp-sansw-1b:1-9.126L61    A     2    8 
netapp-sansw-1b:1-9.126L62    A    netapp-sansw-1a:1-10.126L36   B     2    9 
netapp-sansw-1a:1-9.126L63    A    netapp-sansw-1a:1-10.126L37   B     2   10 
netapp-sansw-1a:1-10.126L38   B    netapp-sansw-1b:1-9.126L64    A     2   11 
netapp-sansw-1b:1-9.126L65    A    netapp-sansw-1a:1-10.126L39   B     2   12 
netapp-sansw-1a:1-9.126L66    A    netapp-sansw-1a:1-10.126L40   B     2   13 
netapp-sansw-1a:1-9.126L67    A    netapp-sansw-1a:1-10.126L41   B     2   14 
netapp-sansw-1a:1-10.126L42   B    netapp-sansw-1b:1-9.126L68    A     2   15 
netapp-sansw-1b:1-9.126L69    A    netapp-sansw-1a:1-10.126L43   B     2   16 
netapp-sansw-1a:1-9.126L70    A    netapp-sansw-1a:1-10.126L44   B     2   17 
netapp-sansw-1b:1-10.126L45   B    netapp-sansw-1b:1-9.126L71    A     2   18 
netapp-sansw-1b:1-10.126L46   B    netapp-sansw-1b:1-9.126L72    A     2   19 
netapp-sansw-1b:1-10.126L47   B    netapp-sansw-1b:1-9.126L73    A     2   20 
netapp-sansw-1b:1-10.126L48   B    netapp-sansw-1b:1-9.126L74    A     2   21 
netapp-sansw-1b:1-10.126L49   B    netapp-sansw-1b:1-9.126L75    A     2   22 
netapp-sansw-1b:1-10.126L50   B    netapp-sansw-1b:1-9.126L76    A     2   23 
netapp-sansw-1b:1-9.126L79    A    netapp-sansw-1a:1-10.126L1    B     3    0 
netapp-sansw-1a:1-9.126L80    A    netapp-sansw-1a:1-10.126L2    B     3    1 
netapp-sansw-1a:1-10.126L3    B    netapp-sansw-1b:1-9.126L81    A     3    2 
netapp-sansw-1a:1-9.126L82    A    netapp-sansw-1a:1-10.126L4    B     3    3 
netapp-sansw-1b:1-9.126L83    A    netapp-sansw-1a:1-10.126L5    B     3    4 
netapp-sansw-1a:1-10.126L6    B    netapp-sansw-1b:1-9.126L84    A     3    5 
netapp-sansw-1b:1-9.126L85    A    netapp-sansw-1a:1-10.126L7    B     3    6 
netapp-sansw-1a:1-9.126L86    A    netapp-sansw-1a:1-10.126L8    B     3    7 
netapp-sansw-1b:1-9.126L87    A    netapp-sansw-1a:1-10.126L9    B     3    8 
netapp-sansw-1b:1-9.126L88    A    netapp-sansw-1a:1-10.126L10   B     3    9 
netapp-sansw-1a:1-9.126L89    A    netapp-sansw-1a:1-10.126L11   B     3   10 
netapp-sansw-1a:1-9.126L90    A    netapp-sansw-1a:1-10.126L12   B     3   11 
netapp-sansw-1a:1-9.126L91    A    netapp-sansw-1a:1-10.126L13   B     3   12 
netapp-sansw-1a:1-9.126L92    A    netapp-sansw-1a:1-10.126L14   B     3   13 
netapp-sansw-1b:1-9.126L93    A    netapp-sansw-1a:1-10.126L15   B     3   14 
netapp-sansw-1b:1-9.126L94    A    netapp-sansw-1a:1-10.126L16   B     3   15 
netapp-sansw-1b:1-9.126L95    A    netapp-sansw-1a:1-10.126L17   B     3   16 
netapp-sansw-1a:1-9.126L96    A    netapp-sansw-1a:1-10.126L18   B     3   17 
netapp-sansw-1b:1-9.126L97    A    netapp-sansw-1a:1-10.126L19   B     3   18 
netapp-sansw-1a:1-9.126L98    A    netapp-sansw-1a:1-10.126L20   B     3   19 
netapp-sansw-1a:1-9.126L99    A    netapp-sansw-1a:1-10.126L21   B     3   20 
netapp-sansw-1b:1-9.126L100   A    netapp-sansw-1a:1-10.126L22   B     3   21 
netapp-sansw-1b:1-9.126L101   A    netapp-sansw-1a:1-10.126L23   B     3   22 
netapp-sansw-1b:1-9.126L102   A    netapp-sansw-1a:1-10.126L24   B     3   23 
netapp-sansw-1b:1-10.126L501  B    netapp-sansw-1b:1-9.126L501   A     4    0 
netapp-sansw-1a:1-9.126L502   A    netapp-sansw-1a:1-10.126L502  B     4    1 
netapp-sansw-1b:1-10.126L503  B    netapp-sansw-1b:1-9.126L503   A     4    2 
netapp-sansw-1b:1-10.126L504  B    netapp-sansw-1b:1-9.126L504   A     4    3 
netapp-sansw-1b:1-9.126L505   A    netapp-sansw-1a:1-10.126L505  B     4    4 
netapp-sansw-1a:1-9.126L506   A    netapp-sansw-1a:1-10.126L506  B     4    5 
netapp-sansw-1b:1-10.126L507  B    netapp-sansw-1b:1-9.126L507   A     4    6 
netapp-sansw-1b:1-10.126L508  B    netapp-sansw-1b:1-9.126L508   A     4    7 
netapp-sansw-1b:1-10.126L509  B    netapp-sansw-1b:1-9.126L509   A     4    8 
netapp-sansw-1b:1-9.126L510   A    netapp-sansw-1a:1-10.126L510  B     4    9 
netapp-sansw-1b:1-10.126L511  B    netapp-sansw-1b:1-9.126L511   A     4   10 
netapp-sansw-1b:1-10.126L512  B    netapp-sansw-1b:1-9.126L512   A     4   11 
netapp-sansw-1a:1-9.126L513   A    netapp-sansw-1a:1-10.126L513  B     4   12 
netapp-sansw-1b:1-10.126L514  B    netapp-sansw-1b:1-9.126L514   A     4   13 
netapp-sansw-1b:1-10.126L515  B    netapp-sansw-1b:1-9.126L515   A     4   14 
netapp-sansw-1a:1-9.126L516   A    netapp-sansw-1a:1-10.126L516  B     4   15 
netapp-sansw-1b:1-10.126L517  B    netapp-sansw-1b:1-9.126L517   A     4   16 
netapp-sansw-1b:1-10.126L518  B    netapp-sansw-1b:1-9.126L518   A     4   17 
netapp-sansw-1a:1-10.126L519  B    netapp-sansw-1b:1-9.126L519   A     4   18 
netapp-sansw-1b:1-10.126L520  B    netapp-sansw-1b:1-9.126L520   A     4   19 
netapp-sansw-1b:1-10.126L521  B    netapp-sansw-1b:1-9.126L521   A     4   20 
netapp-sansw-1a:1-10.126L522  B    netapp-sansw-1b:1-9.126L522   A     4   21 
netapp-sansw-1b:1-10.126L523  B    netapp-sansw-1b:1-9.126L523   A     4   22 
netapp-sansw-1b:1-10.126L524  B    netapp-sansw-1b:1-9.126L524   A     4   23 
netapp-sansw-2a:1-10.126L79   B    netapp-sansw-2b:1-9.126L1     A    10    0 
netapp-sansw-2a:1-10.126L80   B    netapp-sansw-2b:1-9.126L2     A    10    1 
netapp-sansw-2b:1-9.126L3     A    netapp-sansw-2b:1-10.126L81   B    10    2 
netapp-sansw-2a:1-9.126L4     A    netapp-sansw-2b:1-10.126L82   B    10    3 
netapp-sansw-2b:1-10.126L83   B    netapp-sansw-2b:1-9.126L5     A    10    4 
netapp-sansw-2b:1-10.126L84   B    netapp-sansw-2b:1-9.126L6     A    10    5 
netapp-sansw-2a:1-9.126L7     A    netapp-sansw-2b:1-10.126L85   B    10    6 
netapp-sansw-2b:1-10.126L86   B    netapp-sansw-2b:1-9.126L8     A    10    7 
netapp-sansw-2a:1-9.126L9     A    netapp-sansw-2b:1-10.126L87   B    10    8 
netapp-sansw-2b:1-9.126L10    A    netapp-sansw-2b:1-10.126L88   B    10    9 
netapp-sansw-2a:1-9.126L11    A    netapp-sansw-2b:1-10.126L89   B    10   10 
netapp-sansw-2b:1-9.126L12    A    netapp-sansw-2b:1-10.126L90   B    10   11 
netapp-sansw-2b:1-9.126L13    A    netapp-sansw-2b:1-10.126L91   B    10   12 
netapp-sansw-2a:1-9.126L14    A    netapp-sansw-2b:1-10.126L92   B    10   13 
netapp-sansw-2b:1-9.126L15    A    netapp-sansw-2b:1-10.126L93   B    10   14 
netapp-sansw-2b:1-9.126L16    A    netapp-sansw-2b:1-10.126L94   B    10   15 
netapp-sansw-2b:1-10.126L95   B    netapp-sansw-2b:1-9.126L17    A    10   16 
netapp-sansw-2b:1-10.126L96   B    netapp-sansw-2b:1-9.126L18    A    10   17 
netapp-sansw-2a:1-9.126L19    A    netapp-sansw-2b:1-10.126L97   B    10   18 
netapp-sansw-2a:1-10.126L98   B    netapp-sansw-2b:1-9.126L20    A    10   19 
netapp-sansw-2b:1-9.126L21    A    netapp-sansw-2b:1-10.126L99   B    10   20 
netapp-sansw-2a:1-10.126L100  B    netapp-sansw-2b:1-9.126L22    A    10   21 
netapp-sansw-2b:1-9.126L23    A    netapp-sansw-2b:1-10.126L101  B    10   22 
netapp-sansw-2a:1-10.126L102  B    netapp-sansw-2b:1-9.126L24    A    10   23 
netapp-sansw-2a:1-10.126L53   B    netapp-sansw-2b:1-9.126L27    A    11    0 
netapp-sansw-2a:1-10.126L54   B    netapp-sansw-2b:1-9.126L28    A    11    1 
netapp-sansw-2a:1-10.126L55   B    netapp-sansw-2b:1-9.126L29    A    11    2 
netapp-sansw-2b:1-9.126L30    A    netapp-sansw-2b:1-10.126L56   B    11    3 
netapp-sansw-2b:1-10.126L57   B    netapp-sansw-2b:1-9.126L31    A    11    4 
netapp-sansw-2a:1-10.126L58   B    netapp-sansw-2b:1-9.126L32    A    11    5 
netapp-sansw-2b:1-9.126L33    A    netapp-sansw-2b:1-10.126L59   B    11    6 
netapp-sansw-2b:1-10.126L60   B    netapp-sansw-2b:1-9.126L34    A    11    7 
netapp-sansw-2b:1-9.126L35    A    netapp-sansw-2b:1-10.126L61   B    11    8 
netapp-sansw-2b:1-9.126L36    A    netapp-sansw-2b:1-10.126L62   B    11    9 
netapp-sansw-2b:1-9.126L37    A    netapp-sansw-2b:1-10.126L63   B    11   10 
netapp-sansw-2b:1-9.126L38    A    netapp-sansw-2b:1-10.126L64   B    11   11 
netapp-sansw-2b:1-10.126L65   B    netapp-sansw-2b:1-9.126L39    A    11   12 
netapp-sansw-2b:1-9.126L40    A    netapp-sansw-2b:1-10.126L66   B    11   13 
netapp-sansw-2b:1-9.126L41    A    netapp-sansw-2b:1-10.126L67   B    11   14 
netapp-sansw-2b:1-9.126L42    A    netapp-sansw-2b:1-10.126L68   B    11   15 
netapp-sansw-2b:1-9.126L43    A    netapp-sansw-2b:1-10.126L69   B    11   16 
netapp-sansw-2b:1-10.126L70   B    netapp-sansw-2b:1-9.126L44    A    11   17 
netapp-sansw-2b:1-9.126L45    A    netapp-sansw-2b:1-10.126L71   B    11   18 
netapp-sansw-2a:1-9.126L46    A    netapp-sansw-2b:1-10.126L72   B    11   19 
netapp-sansw-2b:1-9.126L47    A    netapp-sansw-2b:1-10.126L73   B    11   20 
netapp-sansw-2b:1-9.126L48    A    netapp-sansw-2b:1-10.126L74   B    11   21 
netapp-sansw-2b:1-9.126L49    A    netapp-sansw-2b:1-10.126L75   B    11   22 
netapp-sansw-2a:1-9.126L50    A    netapp-sansw-2b:1-10.126L76   B    11   23 
netapp-sansw-2b:1-10.126L27   B    netapp-sansw-2b:1-9.126L53    A    12    0 
netapp-sansw-2b:1-9.126L54    A    netapp-sansw-2b:1-10.126L28   B    12    1 
netapp-sansw-2b:1-9.126L55    A    netapp-sansw-2b:1-10.126L29   B    12    2 
netapp-sansw-2b:1-10.126L30   B    netapp-sansw-2b:1-9.126L56    A    12    3 
netapp-sansw-2a:1-9.126L57    A    netapp-sansw-2b:1-10.126L31   B    12    4 
netapp-sansw-2b:1-9.126L58    A    netapp-sansw-2b:1-10.126L32   B    12    5 
netapp-sansw-2a:1-10.126L33   B    netapp-sansw-2b:1-9.126L59    A    12    6 
netapp-sansw-2a:1-9.126L60    A    netapp-sansw-2b:1-10.126L34   B    12    7 
netapp-sansw-2b:1-10.126L35   B    netapp-sansw-2b:1-9.126L61    A    12    8 
netapp-sansw-2b:1-9.126L62    A    netapp-sansw-2b:1-10.126L36   B    12    9 
netapp-sansw-2b:1-9.126L63    A    netapp-sansw-2b:1-10.126L37   B    12   10 
netapp-sansw-2a:1-10.126L38   B    netapp-sansw-2b:1-9.126L64    A    12   11 
netapp-sansw-2a:1-10.126L39   B    netapp-sansw-2b:1-9.126L65    A    12   12 
netapp-sansw-2a:1-10.126L40   B    netapp-sansw-2b:1-9.126L66    A    12   13 
netapp-sansw-2b:1-10.126L41   B    netapp-sansw-2b:1-9.126L67    A    12   14 
netapp-sansw-2b:1-10.126L42   B    netapp-sansw-2b:1-9.126L68    A    12   15 
netapp-sansw-2a:1-9.126L69    A    netapp-sansw-2b:1-10.126L43   B    12   16 
netapp-sansw-2b:1-10.126L44   B    netapp-sansw-2b:1-9.126L70    A    12   17 
netapp-sansw-2a:1-10.126L45   B    netapp-sansw-2b:1-9.126L71    A    12   18 
netapp-sansw-2a:1-10.126L46   B    netapp-sansw-2b:1-9.126L72    A    12   19 
netapp-sansw-2a:1-10.126L47   B    netapp-sansw-2b:1-9.126L73    A    12   20 
netapp-sansw-2b:1-9.126L74    A    netapp-sansw-2b:1-10.126L48   B    12   21 
netapp-sansw-2a:1-10.126L49   B    netapp-sansw-2b:1-9.126L75    A    12   22 
netapp-sansw-2b:1-10.126L50   B    netapp-sansw-2b:1-9.126L76    A    12   23 
netapp-sansw-2b:1-9.126L79    A    netapp-sansw-2b:1-10.126L1    B    13    0 
netapp-sansw-2b:1-9.126L80    A    netapp-sansw-2b:1-10.126L2    B    13    1 
netapp-sansw-2b:1-9.126L81    A    netapp-sansw-2b:1-10.126L3    B    13    2 
netapp-sansw-2b:1-10.126L4    B    netapp-sansw-2b:1-9.126L82    A    13    3 
netapp-sansw-2b:1-10.126L5    B    netapp-sansw-2b:1-9.126L83    A    13    4 
netapp-sansw-2b:1-9.126L84    A    netapp-sansw-2b:1-10.126L6    B    13    5 
netapp-sansw-2b:1-9.126L85    A    netapp-sansw-2b:1-10.126L7    B    13    6 
netapp-sansw-2b:1-9.126L86    A    netapp-sansw-2b:1-10.126L8    B    13    7 
netapp-sansw-2a:1-9.126L87    A    netapp-sansw-2b:1-10.126L9    B    13    8 
netapp-sansw-2b:1-9.126L88    A    netapp-sansw-2b:1-10.126L10   B    13    9 
netapp-sansw-2a:1-9.126L89    A    netapp-sansw-2b:1-10.126L11   B    13   10 
netapp-sansw-2b:1-9.126L90    A    netapp-sansw-2b:1-10.126L12   B    13   11 
netapp-sansw-2b:1-9.126L91    A    netapp-sansw-2b:1-10.126L13   B    13   12 
netapp-sansw-2b:1-9.126L92    A    netapp-sansw-2b:1-10.126L14   B    13   13 
netapp-sansw-2a:1-9.126L93    A    netapp-sansw-2b:1-10.126L15   B    13   14 
netapp-sansw-2b:1-9.126L94    A    netapp-sansw-2b:1-10.126L16   B    13   15 
netapp-sansw-2b:1-9.126L95    A    netapp-sansw-2b:1-10.126L17   B    13   16 
netapp-sansw-2b:1-9.126L96    A    netapp-sansw-2b:1-10.126L18   B    13   17 
netapp-sansw-2a:1-9.126L97    A    netapp-sansw-2b:1-10.126L19   B    13   18 
netapp-sansw-2b:1-9.126L98    A    netapp-sansw-2b:1-10.126L20   B    13   19 
netapp-sansw-2a:1-9.126L99    A    netapp-sansw-2b:1-10.126L21   B    13   20 
netapp-sansw-2a:1-9.126L100   A    netapp-sansw-2b:1-10.126L22   B    13   21 
netapp-sansw-2a:1-10.126L23   B    netapp-sansw-2b:1-9.126L101   A    13   22 
netapp-sansw-2b:1-9.126L102   A    netapp-sansw-2b:1-10.126L24   B    13   23 
netapp-sansw-2b:1-9.126L501   A    netapp-sansw-2b:1-10.126L501  B    14    0 
netapp-sansw-2b:1-9.126L502   A    netapp-sansw-2b:1-10.126L502  B    14    1 
netapp-sansw-2a:1-10.126L503  B    netapp-sansw-2b:1-9.126L503   A    14    2 
netapp-sansw-2b:1-10.126L504  B    netapp-sansw-2b:1-9.126L504   A    14    3 
netapp-sansw-2b:1-10.126L505  B    netapp-sansw-2b:1-9.126L505   A    14    4 
netapp-sansw-2b:1-9.126L506   A    netapp-sansw-2b:1-10.126L506  B    14    5 
netapp-sansw-2b:1-9.126L507   A    netapp-sansw-2b:1-10.126L507  B    14    6 
netapp-sansw-2b:1-9.126L508   A    netapp-sansw-2b:1-10.126L508  B    14    7 
netapp-sansw-2b:1-9.126L509   A    netapp-sansw-2b:1-10.126L509  B    14    8 
netapp-sansw-2b:1-9.126L510   A    netapp-sansw-2b:1-10.126L510  B    14    9 
netapp-sansw-2a:1-10.126L511  B    netapp-sansw-2b:1-9.126L511   A    14   10 
netapp-sansw-2a:1-10.126L512  B    netapp-sansw-2b:1-9.126L512   A    14   11 
netapp-sansw-2b:1-10.126L513  B    netapp-sansw-2b:1-9.126L513   A    14   12 
netapp-sansw-2b:1-10.126L514  B    netapp-sansw-2b:1-9.126L514   A    14   13 
netapp-sansw-2b:1-9.126L515   A    netapp-sansw-2b:1-10.126L515  B    14   14 
netapp-sansw-2b:1-9.126L516   A    netapp-sansw-2b:1-10.126L516  B    14   15 
netapp-sansw-2b:1-9.126L517   A    netapp-sansw-2b:1-10.126L517  B    14   16 
netapp-sansw-2b:1-9.126L518   A    netapp-sansw-2b:1-10.126L518  B    14   17 
netapp-sansw-2b:1-9.126L519   A    netapp-sansw-2b:1-10.126L519  B    14   18 
netapp-sansw-2b:1-9.126L520   A    netapp-sansw-2b:1-10.126L520  B    14   19 
netapp-sansw-2a:1-10.126L521  B    netapp-sansw-2b:1-9.126L521   A    14   20 
netapp-sansw-2a:1-9.126L522   A    netapp-sansw-2b:1-10.126L522  B    14   21 
netapp-sansw-2b:1-9.126L523   A    netapp-sansw-2b:1-10.126L523  B    14   22 
netapp-sansw-2a:1-9.126L524   A    netapp-sansw-2b:1-10.126L524  B    14   23 
cluster_a-01> exit
logout
cluster_a::>  storage disk show -fields primary-port,secondary-port
disk    primary-port secondary-port                                  
------- ------------ --------------
1.4.0   A            A             
1.4.1   B            A             
1.4.2   B            A             
1.4.3   B            A             
1.4.4   B            A             
1.4.5   A            A             
1.4.6   A            A             
1.4.7   B            A             
1.4.8   B            A             
1.4.9   A            A             
1.4.10  A            A              
1.4.11  B            A             
1.4.12  A            A             
1.4.13  A            A             
1.4.14  B            A             
1.4.15  B            A             
1.4.16  B            A             
1.4.17  A            A              
1.4.18  B            A             
1.4.19  A            A             
1.4.20  B            A             
1.4.21  B            A             
1.4.22  A            A             
1.4.23  A            A             
2.0.0   A            A              
2.0.1   A            A             
2.0.2   A            A             
2.0.3   B            A             
2.0.4   B            A             
2.0.5   B            A             
2.0.6   B            A             
2.0.7   B            A              
2.0.8   A            A             
2.0.9   A            A             
2.0.10  A            A             
2.0.11  B            A             
2.0.12  B            A             
2.0.13  A            A             
2.0.14  A            A             
2.0.15  B            A             
2.0.16  A            A             
2.0.17  A            A             
2.0.18  B            A             
2.0.19  B            A              
2.0.20  B            A             
2.0.21  A            A             
2.0.22  B            A             
2.0.23  A            A             
2.1.0   B            A             
2.1.1   B            A             
2.1.2   B            A              
2.1.3   B            A             
2.1.4   B            A             
2.1.5   A            A             
2.1.6   B            A             
2.1.7   B            A             
2.1.8   A            A             
2.1.9   B            A              
2.1.10  A            A             
2.1.11  A            A             
2.1.12  B            A             
2.1.13  B            A             
2.1.14  A            A             
2.1.15  A            A             
2.1.16  A            A             
2.1.17  B            A             
2.1.18  B            A             
2.1.19  B            A             
2.1.20  A            A             
2.1.21  A            A              
2.1.22  A            A             
2.1.23  A            A             
2.2.0   A            A             
2.2.1   A            A             
2.2.2   A            A             
2.2.3   B            A             
                                                                               
disk    primary-port secondary-port
------- ------------ --------------
2.2.4   A            A             
2.2.5   A            A             
2.2.6   A            A             
2.2.7   B            A             
2.2.8   A            A             
2.2.9   A            A             
2.2.10  A            A             
2.2.11  B            A             
2.2.12  A            A             
2.2.13  A            A             
2.2.14  A            A             
2.2.15  A            A             
2.2.16  B            A             
2.2.17  A            A             
2.2.18  A            A             
2.2.19  B            A             
2.2.20  B            A             
2.2.21  A            A             
2.2.22  B            A             
2.2.23  B            A             
2.3.0   B            A             
2.3.1   A            A             
2.3.2   A            A              
2.3.3   A            A             
2.3.4   A            A             
2.3.5   B            A             
2.3.6   A            A             
2.3.7   A            A             
2.3.8   A            A             
2.3.9   A            A              
2.3.10  A            A             
2.3.11  A            A             
2.3.12  A            A             
2.3.13  A            A             
2.3.14  A            A             
2.3.15  A            A             
2.3.16  A            A              
2.3.17  A            A             
2.3.18  A            A             
2.3.19  A            A             
2.3.20  A            A             
2.3.21  A            A             
2.3.22  A            A             
2.3.23  A            A             
3.10.0  B            B             
3.10.1  B            B             
3.10.2  A            B             
3.10.3  A            B             
3.10.4  B            B              
3.10.5  B            B             
3.10.6  A            B             
3.10.7  B            B             
3.10.8  A            B             
3.10.9  A            B             
3.10.10 A            B             
3.10.11 A            B              
3.10.12 A            B             
3.10.13 A            B             
3.10.14 A            B             
3.10.15 A            B             
3.10.16 B            B             
3.10.17 B            B             
3.10.18 A            B              
3.10.19 B            B             
3.10.20 A            B             
3.10.21 B            B             
3.10.22 A            B             
3.10.23 B            B             
3.11.0  B            B             
3.11.1  B            B              
3.11.2  B            B             
3.11.3  A            B             
3.11.4  B            B             
3.11.5  B            B             
3.11.6  A            B             
                                                                               
disk    primary-port secondary-port
------- ------------ --------------
3.11.7  A            B             
3.11.8  A            B             
3.11.9  A            B              
3.11.10 A            B             
3.11.11 A            B             
3.11.12 A            B             
3.11.13 A            B             
3.11.14 A            B             
3.11.15 A            B             
3.11.16 A            B              
3.11.17 A            B             
3.11.18 A            B             
3.11.19 A            B             
3.11.20 B            B             
3.11.21 B            B             
3.11.22 B            B             
3.11.23 A            B             
3.12.0  B            B             
3.12.1  B            B             
3.12.2  B            B             
3.12.3  B            B             
3.12.4  B            B             
3.12.5  B            B             
3.12.6  B            B             
3.12.7  B            B             
3.12.8  B            B             
3.12.9  B            B             
3.12.10 A            B             
3.12.11 A            B              
3.12.12 A            B             
3.12.13 A            B             
3.12.14 A            B             
3.12.15 B            B             
3.12.16 B            B             
3.12.17 B            B             
3.12.18 B            B              
3.12.19 A            B             
3.12.20 B            B             
3.12.21 B            B             
3.12.22 B            B             
3.12.23 A            B             
3.13.0  A            B             
3.13.1  A            B              
3.13.2  A            B             
3.13.3  B            B             
3.13.4  B            B             
3.13.5  B            B             
3.13.6  B            B             
3.13.7  B            B             
3.13.8  B            B              
3.13.9  B            B             
3.13.10 B            B             
3.13.11 B            B             
3.13.12 B            B             
3.13.13 B            B             
3.13.14 B            B             
3.13.15 B            B             
3.13.16 B            B             
3.13.17 B            B             
3.13.18 B            B             
3.13.19 B            B             
3.13.20 B            B              
3.13.21 B            B             
3.13.22 A            B             
3.13.23 B            B             
4.14.0  A            B             
4.14.1  B            B             
4.14.2  B            B             
4.14.3  A            B              
4.14.4  A            B             
4.14.5  A            B             
4.14.6  A            B             
4.14.7  A            B             
4.14.8  A            B             
4.14.9  B            B             
                                                                               
disk    primary-port secondary-port
------- ------------ --------------
4.14.10 A            B             
4.14.11 B            B             
4.14.12 A            B             
4.14.13 B            B             
4.14.14 A            B             
4.14.15 B            B             
4.14.16 A            B             
4.14.17 A            B             
4.14.18 A            B              
4.14.19 A            B             
4.14.20 A            B             
4.14.21 B            B             
4.14.22 A            B             
4.14.23 B            B             
240 entries were displayed.
 
cluster_a::> 

 

Ontapforrum
6,986 Views

Interesting, as I see.

 

Cluster level command is reporting incorrectly
cluster_a::> storage disk show -fields primary-port,secondary-port
Shelf 4, Bay:5 & 6 are showing both as A.
1.4.5 A A
1.4.6 A A


Node level command is reporting correctly (or is it)
cluster_a-01> storage show disk -p
Shelf 4, Bay:5 & 6 are showing both showing dual-paths
netapp-sansw-1a:1-9.126L506 A netapp-sansw-1a:1-10.126L506 B 4 5
netapp-sansw-1b:1-10.126L507 B netapp-sansw-1b:1-9.126L507 A 4 6

 

Something I just noticed:

In the node-level command output: Normally on my system (Non-Metro) I see disks in more simple way such as:
PRIMARY PORT SECONDARY PORT SHELF BAY
0d.04.5 B 0a.04.5 A 4 5
0d.04.6 B 0a.04.6 A 4 6

 

On your system it is bit different, but I pick the exact separate adapter, then it shows same path.

PRIMARY PORT SECONDARY PORT SHELF BAY
netapp-sansw-1a:1-9.126L506   A  netapp-sansw-1b:1-9.126L507  A  46

netapp-sansw-1a:1-10.126L506 B netapp-sansw-1b:1-10.126L507 B 45

 


Yes, it can get confusing, therefore it is worth verifying to this BUG:472649

Could you verify if this is the same SAS Model: X412_S15K7560A15

😆 storage disk show -disk 1.4.5


If it's not the same Model, then let NetApp investigate.

iUser
6,984 Views

Looks like it's not the drive, from the affected by the bug, so i'm waiting for the support's answer:

 

cluster_a::> storage disk show -disk 1.4.5
Disk: 1.4.5
Container Type: unassigned
Owner/Home: - / -
DR Home: -
Stack ID/Shelf/Bay: 1 / 4 / 5
LUN: 506
Array: ATTO_FibreBridge7500N_1
Vendor: NETAPP
Model: X371_S1643960ATE
Serial Number: S3SENF0KC01188
UID: 5002538B:48C05440:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000
BPS: 520
Physical Size: 894.3GB
Position: present
Checksum Compatibility: block
Aggregate: -
Plex: -
Paths:
LUN Initiator Side Target Side Link
Controller Initiator ID Switch Port Switch Port Acc Use Target Port TPGN Speed I/O KB/s IOPS
------------------ --------- ----- -------------------- -------------------- --- --- ----------------------- ------ ------- ------------ ------------
cluster_a-02 1c 506 netapp-sansw-1a:1-8 netapp-sansw-1a:1-10 AO RDY 2100001086a9c800 87 16 Gb/S 0 0
cluster_a-02 1d 506 netapp-sansw-1b:1-8 netapp-sansw-1b:1-10 AO RDY 2200001086a9c800 88 16 Gb/S 0 0
cluster_a-02 1b 506 netapp-sansw-1b:1-7 netapp-sansw-1b:1-9 AO INU 2200001086a9d700 67 16 Gb/S 0 0
cluster_a-02 1a 506 netapp-sansw-1a:1-7 netapp-sansw-1a:1-9 AO RDY 2100001086a9d700 66 16 Gb/S 0 0
cluster_a-01 1c 506 netapp-sansw-1a:1-4 netapp-sansw-1a:1-10 AO RDY 2100001086a9c800 87 16 Gb/S 0 0
cluster_a-01 1d 506 netapp-sansw-1b:1-4 netapp-sansw-1b:1-10 AO RDY 2200001086a9c800 88 16 Gb/S 0 0
cluster_a-01 1b 506 netapp-sansw-1b:1-3 netapp-sansw-1b:1-9 AO RDY 2200001086a9d700 67 16 Gb/S 0 0
cluster_a-01 1a 506 netapp-sansw-1a:1-3 netapp-sansw-1a:1-9 AO INU 2100001086a9d700 66 16 Gb/S 0 0

Errors:
-

cluster_a::>

iUser
6,879 Views

I've got the answer from support that the command "storage disk show" doesn’t check the multipathing in the MetroCluster. So, looks like if sysconfig shows us that we have quad-path for the metrocluster - it should be enough.

Ontapforrum
4,479 Views

Thanks for the update. Interesting!!

 
Hope it helps another customer who discovers the same issue. I guess NetApp kb stating the same is desirable.
Public