Spare disks with root partitionned disks

[lucbussieres]

I experoience with netapp up to version 8.2 and now I have just been pushed a preinstalled 8.3.1P2 cluster and I'm validating the setup.

There is something strange that I found there is no physical spare for the SAS drive only one define for FSAS:

tdc1-nasca::*> disk show -fields pool,type,container-type,position,aggregate
disk    aggregate           container-type pool  position type
------- ------------------- -------------- ----- -------- ----
1.0.0   -                   shared         Pool0 shared   SSD
1.0.1   -                   shared         Pool0 shared   SSD
1.0.2   -                   shared         Pool0 shared   SSD
1.0.3   -                   shared         Pool0 shared   SSD
1.0.4   -                   shared         Pool0 shared   SAS
1.0.5   -                   shared         Pool0 shared   SAS
1.0.6   -                   shared         Pool0 shared   SAS
1.0.7   -                   shared         Pool0 shared   SAS
1.0.8   -                   shared         Pool0 shared   SAS
1.0.9   -                   shared         Pool0 shared   SAS
1.0.10  -                   shared         Pool0 shared   SAS
1.0.11  -                   shared         Pool0 shared   SAS
1.0.12  -                   shared         Pool0 shared   SAS
1.0.13  -                   shared         Pool0 shared   SAS
1.0.14  -                   shared         Pool0 shared   SAS
1.0.15  -                   shared         Pool0 shared   SAS
1.0.16  -                   shared         Pool0 shared   SAS
1.0.17  -                   shared         Pool0 shared   SAS
1.0.18  -                   shared         Pool0 shared   SAS
1.0.19  -                   shared         Pool0 shared   SAS
1.0.20  -                   shared         Pool0 shared   SAS
1.0.21  -                   shared         Pool0 shared   SAS
1.0.22  -                   shared         Pool0 shared   SAS
1.0.23  -                   shared         Pool0 shared   SAS
1.10.0  sata1_tdc1_nasca_02 aggregate      Pool0 dparity  FSAS
1.10.1  sata1_tdc1_nasca_02 aggregate      Pool0 parity   FSAS
1.10.2  sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.3  sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.4  sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.5  sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.6  sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.7  sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.8  sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.9  sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.10 sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.11 sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.12 sata1_tdc1_nasca_02 aggregate      Pool0 dparity  FSAS
1.10.13 sata1_tdc1_nasca_02 aggregate      Pool0 parity   FSAS
1.10.14 sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.15 sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.16 sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.17 sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.18 sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.19 sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.20 sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.21 sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.22 sata1_tdc1_nasca_02 aggregate      Pool0 data     FSAS
1.10.23 -                   spare          Pool0 present  FSAS

The sas drive have been configure for root-partitionning and this is new for me. I'm not sure if it is a normal setup without the spare disk. Running the show-spare-disks there seems to be something that could look like a spare in a partitionned setup:

tdc1-nasca::*> storage aggregate show-spare-disks

Original Owner: tdc1-nasca-01
 Pool0
  Partitioned Spares
                                                            Local    Local
                                                             Data     Root Physical
 Disk                        Type     RPM Checksum         Usable   Usable     Size Status
 --------------------------- ----- ------ -------------- -------- -------- -------- --------
 1.0.23                      SAS    10000 block                0B  61.58GB  838.4GB zeroed

Original Owner: tdc1-nasca-02
 Pool0
  Spare Pool

                                                           Usable Physical
 Disk                        Type     RPM Checksum           Size     Size Status
 --------------------------- ----- ------ -------------- -------- -------- --------
 1.10.23                     FSAS    7200 block            3.63TB   3.64TB zeroed

Original Owner: tdc1-nasca-02
 Pool0
  Partitioned Spares
                                                            Local    Local
                                                             Data     Root Physical
 Disk                        Type     RPM Checksum         Usable   Usable     Size Status
 --------------------------- ----- ------ -------------- -------- -------- -------- --------
 1.0.22                      SAS    10000 block                0B  61.58GB  838.4GB zeroed
3 entries were displayed.

tdc1-nasca::*>

Is this a correct setup for spare drives?

Regards

Luc

[aborzenkov]

They are spares for root partitions on both controllers. It does not look like you have spares for data partitions. Do you use them at all? What aggregates do you have?

[lucbussieres]

Thanks for your clarification.

There is currently 4 different aggragate 2 are root aggragates the ones starting with ar_ and the 2 others are data aggregates :

tdc1-nasca::> aggr show

Aggregate Size Available Used% State #Vols Nodes RAID Status
--------- -------- --------- ----- ------- ------ ---------------- ------------
ar_tdc1_nasca_01
    368.4GB 17.86GB 95% online 1 tdc1-nasca-01 raid_dp,

                                               normal
ar_tdc1_nasca_02
    368.4GB 17.86GB 95% online 1 tdc1-nasca-02 raid_dp,

                                               normal
sas0_tdc1_nasca_01
    10.22TB 8.82TB 14% online 4 tdc1-nasca-01 mixed_raid_

                                              type,
                                              hybrid,
                                              normal
sata1_tdc1_nasca_02
    62.10TB 39.18TB 37% online 7 tdc1-nasca-02 mixed_raid_
                                               type,
                                               hybrid,
                                               normal

according to you answer I guess I should delete the sas0... aggregates and recreate it without the 1.0.23 disk so I can have a spare disk for data also..

[aborzenkov]

Could you paste output of "storage aggregate show-status".

[lucbussieres]

tdc1-nasca::*> storage aggregate show-status

Owner Node: tdc1-nasca-01
 Aggregate: ar_tdc1_nasca_01 (online, raid_dp) (block checksums)
  Plex: /ar_tdc1_nasca_01/plex0 (online, normal, active, pool0)
   RAID Group /ar_tdc1_nasca_01/plex0/rg0 (normal, block checksums)
                                                              Usable Physical
     Position Disk                        Pool Type     RPM     Size     Size Status
     -------- --------------------------- ---- ----- ------ -------- -------- ----------
     shared   1.0.5                        0   SAS    10000  61.58GB  838.4GB (normal)
     shared   1.0.7                        0   SAS    10000  61.58GB  838.4GB (normal)
     shared   1.0.9                        0   SAS    10000  61.58GB  838.4GB (normal)
     shared   1.0.11                       0   SAS    10000  61.58GB  838.4GB (normal)
     shared   1.0.13                       0   SAS    10000  61.58GB  838.4GB (normal)
     shared   1.0.15                       0   SAS    10000  61.58GB  838.4GB (normal)
     shared   1.0.17                       0   SAS    10000  61.58GB  838.4GB (normal)
     shared   1.0.19                       0   SAS    10000  61.58GB  838.4GB (normal)
     shared   1.0.21                       0   SAS    10000  61.58GB  838.4GB (normal)

Owner Node: tdc1-nasca-02
 Aggregate: ar_tdc1_nasca_02 (online, raid_dp) (block checksums)
  Plex: /ar_tdc1_nasca_02/plex0 (online, normal, active, pool0)
   RAID Group /ar_tdc1_nasca_02/plex0/rg0 (normal, block checksums)
                                                              Usable Physical
     Position Disk                        Pool Type     RPM     Size     Size Status
     -------- --------------------------- ---- ----- ------ -------- -------- ----------
     shared   1.0.4                        0   SAS    10000  61.58GB  838.4GB (normal)
     shared   1.0.6                        0   SAS    10000  61.58GB  838.4GB (normal)
     shared   1.0.8                        0   SAS    10000  61.58GB  838.4GB (normal)
     shared   1.0.10                       0   SAS    10000  61.58GB  838.4GB (normal)
     shared   1.0.12                       0   SAS    10000  61.58GB  838.4GB (normal)
     shared   1.0.14                       0   SAS    10000  61.58GB  838.4GB (normal)
     shared   1.0.16                       0   SAS    10000  61.58GB  838.4GB (normal)
     shared   1.0.18                       0   SAS    10000  61.58GB  838.4GB (normal)
     shared   1.0.20                       0   SAS    10000  61.58GB  838.4GB (normal)

Owner Node: tdc1-nasca-01
 Aggregate: sas0_tdc1_nasca_01 (online, mixed_raid_type, hybrid) (block checksums)
  Plex: /sas0_tdc1_nasca_01/plex0 (online, normal, active, pool0)
   RAID Group /sas0_tdc1_nasca_01/plex0/rg0 (normal, block checksums, raid_dp)
                                                              Usable Physical
     Position Disk                        Pool Type     RPM     Size     Size Status
     -------- --------------------------- ---- ----- ------ -------- -------- ----------
     shared   1.0.4                        0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.6                        0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.8                        0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.16                       0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.18                       0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.20                       0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.22                       0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.23                       0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.11                       0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.13                       0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.15                       0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.17                       0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.19                       0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.21                       0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.7                        0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.5                        0   SAS    10000  775.3GB  838.4GB (normal)

   RAID Group /sas0_tdc1_nasca_01/plex0/rg1 (normal, block checksums, raid_dp)
                                                              Usable Physical
     Position Disk                        Pool Type     RPM     Size     Size Status
     -------- --------------------------- ---- ----- ------ -------- -------- ----------
     shared   1.0.10                       0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.12                       0   SAS    10000  775.3GB  838.4GB (normal)
     shared   1.0.14                       0   SAS    10000  775.3GB  838.4GB (normal)

   RAID Groups /sas0_tdc1_nasca_01/plex0/rg2 (normal, block checksums, raid4) (Storage Pool: SP_HA1)
               /sas0_tdc1_nasca_01/plex0/rg3 (normal, block checksums, raid4) (Storage Pool: SP_HA1)
                                                              Usable Physical
     Position Disk                        Pool Type     RPM     Size     Size Status
     -------- --------------------------- ---- ----- ------ -------- -------- ----------
     shared   1.0.3                        0   SSD        -  93.01GB  186.3GB (normal)
     shared   1.0.1                        0   SSD        -  93.01GB  186.3GB (normal)
     shared   1.0.2                        0   SSD        -  93.01GB  186.3GB (normal)
     shared   1.0.0                        0   SSD        -  93.01GB  186.3GB (normal)

Owner Node: tdc1-nasca-02
 Aggregate: sata1_tdc1_nasca_02 (online, mixed_raid_type, hybrid) (block checksums)
  Plex: /sata1_tdc1_nasca_02/plex0 (online, normal, active, pool0)
   RAID Group /sata1_tdc1_nasca_02/plex0/rg0 (normal, block checksums, raid_dp)
                                                              Usable Physical
     Position Disk                        Pool Type     RPM     Size     Size Status
     -------- --------------------------- ---- ----- ------ -------- -------- ----------
     dparity  1.10.0                       0   FSAS    7200   3.63TB   3.64TB (normal)
     parity   1.10.1                       0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.2                       0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.3                       0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.4                       0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.5                       0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.6                       0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.7                       0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.8                       0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.9                       0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.10                      0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.11                      0   FSAS    7200   3.63TB   3.64TB (normal)

   RAID Group /sata1_tdc1_nasca_02/plex0/rg1 (normal, block checksums, raid_dp)
                                                              Usable Physical
     Position Disk                        Pool Type     RPM     Size     Size Status
     -------- --------------------------- ---- ----- ------ -------- -------- ----------
     dparity  1.10.12                      0   FSAS    7200   3.63TB   3.64TB (normal)
     parity   1.10.13                      0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.14                      0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.15                      0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.16                      0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.17                      0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.18                      0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.19                      0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.20                      0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.21                      0   FSAS    7200   3.63TB   3.64TB (normal)
     data     1.10.22                      0   FSAS    7200   3.63TB   3.64TB (normal)

   RAID Groups /sata1_tdc1_nasca_02/plex0/rg2 (normal, block checksums, raid4) (Storage Pool: SP_HA1)
               /sata1_tdc1_nasca_02/plex0/rg3 (normal, block checksums, raid4) (Storage Pool: SP_HA1)
                                                              Usable Physical
     Position Disk                        Pool Type     RPM     Size     Size Status
     -------- --------------------------- ---- ----- ------ -------- -------- ----------
     shared   1.0.3                        0   SSD        -  93.01GB  186.3GB (normal)
     shared   1.0.1                        0   SSD        -  93.01GB  186.3GB (normal)
     shared   1.0.2                        0   SSD        -  93.01GB  186.3GB (normal)
     shared   1.0.0                        0   SSD        -  93.01GB  186.3GB (normal)
68 entries were displayed.

tdc1-nasca::*>

[aborzenkov]

According to this output you have one free data partition on 1.0.9. Could you show "storage disk show -partition-ownership".

[lucbussieres]

tdc1-nasca::*> storage disk show -partition-ownership
                                 Root                      Data        Container     Container
Disk     Aggregate Root Owner    Owner ID    Data Owner    Owner ID    Owner         Owner ID
-------- --------- ------------- ----------- ------------- ----------- ------------- -----------
1.0.0    -         -             -           -             -           tdc1-nasca-02 537025903
1.0.1    -         -             -           -             -           tdc1-nasca-01 537025871
1.0.2    -         -             -           -             -           tdc1-nasca-02 537025903
1.0.3    -         -             -           -             -           tdc1-nasca-01 537025871
1.0.4    -         tdc1-nasca-02 537025903   tdc1-nasca-01 537025871   tdc1-nasca-02 537025903
1.0.5    -         tdc1-nasca-01 537025871   tdc1-nasca-01 537025871   tdc1-nasca-01 537025871
1.0.6    -         tdc1-nasca-02 537025903   tdc1-nasca-01 537025871   tdc1-nasca-02 537025903
1.0.7    -         tdc1-nasca-01 537025871   tdc1-nasca-01 537025871   tdc1-nasca-01 537025871
1.0.8    -         tdc1-nasca-02 537025903   tdc1-nasca-01 537025871   tdc1-nasca-02 537025903
1.0.9    -         tdc1-nasca-01 537025871   -             -           tdc1-nasca-01 537025871
1.0.10   -         tdc1-nasca-02 537025903   tdc1-nasca-01 537025871   tdc1-nasca-02 537025903
1.0.11   -         tdc1-nasca-01 537025871   tdc1-nasca-01 537025871   tdc1-nasca-01 537025871
1.0.12   -         tdc1-nasca-02 537025903   tdc1-nasca-01 537025871   tdc1-nasca-02 537025903
1.0.13   -         tdc1-nasca-01 537025871   tdc1-nasca-01 537025871   tdc1-nasca-01 537025871
1.0.14   -         tdc1-nasca-02 537025903   tdc1-nasca-01 537025871   tdc1-nasca-02 537025903
1.0.15   -         tdc1-nasca-01 537025871   tdc1-nasca-01 537025871   tdc1-nasca-01 537025871
1.0.16   -         tdc1-nasca-02 537025903   tdc1-nasca-01 537025871   tdc1-nasca-02 537025903
1.0.17   -         tdc1-nasca-01 537025871   tdc1-nasca-01 537025871   tdc1-nasca-01 537025871
1.0.18   -         tdc1-nasca-02 537025903   tdc1-nasca-01 537025871   tdc1-nasca-02 537025903
1.0.19   -         tdc1-nasca-01 537025871   tdc1-nasca-01 537025871   tdc1-nasca-01 537025871
1.0.20   -         tdc1-nasca-02 537025903   tdc1-nasca-01 537025871   tdc1-nasca-02 537025903
1.0.21   -         tdc1-nasca-01 537025871   tdc1-nasca-01 537025871   tdc1-nasca-01 537025871
1.0.22   -         tdc1-nasca-02 537025903   tdc1-nasca-01 537025871   tdc1-nasca-02 537025903
1.0.23   -         tdc1-nasca-01 537025871   tdc1-nasca-01 537025871   tdc1-nasca-01 537025871
1.10.0   sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.1   sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.2   sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.3   sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.4   sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903

                                 Root                      Data        Container     Container
Disk     Aggregate Root Owner    Owner ID    Data Owner    Owner ID    Owner         Owner ID
-------- --------- ------------- ----------- ------------- ----------- ------------- -----------
1.10.5   sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.6   sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.7   sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.8   sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.9   sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.10  sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.11  sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.12  sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.13  sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.14  sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.15  sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.16  sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.17  sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.18  sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.19  sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.20  sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903

                                 Root                      Data        Container     Container
Disk     Aggregate Root Owner    Owner ID    Data Owner    Owner ID    Owner         Owner ID
-------- --------- ------------- ----------- ------------- ----------- ------------- -----------
1.10.21  sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.22  sata1_tdc1_nasca_02
                   -             -           -             -           tdc1-nasca-02 537025903
1.10.23  -         -             -           -             -           tdc1-nasca-02 537025903
48 entries were displayed.

[aborzenkov · Accepted solution]

---

lucbussieres wrote:

tdc1-nasca::*> storage disk show -partition-ownership
                                 Root                      Data        Container     Container
Disk     Aggregate Root Owner    Owner ID    Data Owner    Owner ID    Owner         Owner ID
-------- --------- ------------- ----------- ------------- ----------- ------------- -----------
1.0.9    -         tdc1-nasca-01 537025871   -             -           tdc1-nasca-01 537025871

---

YOu need to assign ownership of data partition to correct node, I assume tdc1-nasca-01: storage disk assign -disk disk_name -owner owner_name -data true

[lucbussieres]

Thanks now if I'm OK I have a spare disk for data also :

tdc1-nasca::*> storage aggregate show-spare-disks

Original Owner: tdc1-nasca-01
 Pool0
  Partitioned Spares
                                                            Local    Local
                                                             Data     Root Physical
 Disk                        Type     RPM Checksum         Usable   Usable     Size Status
 --------------------------- ----- ------ -------------- -------- -------- -------- --------
 1.0.9                       SAS    10000 block           775.3GB       0B  838.4GB zeroed
 1.0.23                      SAS    10000 block                0B  61.58GB  838.4GB zeroed

Original Owner: tdc1-nasca-02
 Pool0
  Spare Pool

                                                           Usable Physical
 Disk                        Type     RPM Checksum           Size     Size Status
 --------------------------- ----- ------ -------------- -------- -------- --------
 1.10.23                     FSAS    7200 block            3.63TB   3.64TB zeroed

Original Owner: tdc1-nasca-02
 Pool0
  Partitioned Spares
                                                            Local    Local
                                                             Data     Root Physical
 Disk                        Type     RPM Checksum         Usable   Usable     Size Status
 --------------------------- ----- ------ -------------- -------- -------- -------- --------
 1.0.22                      SAS    10000 block                0B  61.58GB  838.4GB zeroed
4 entries were displayed.