Broken disk replaced, now failed disk status

[Maurice-FreeOnes]

Hi,

We had a broken disk in our cluster (1.0.5), so I replaced it for a new disk (old disk X343, new disk X426 model). No big deal normally, but this time it's different. After assigning the disk to node 1 it shows status FAILED in the aggregate show:

aggr show-status -aggregate ch_n1_aggr002_FPP

Owner Node:xxxxxxx-001
Aggregate: ch_n1_aggr002_FPP (online, mixed_raid_type, degraded, hybrid) (block checksums)
Plex: /ch_n1_aggr002_FPP/plex0 (online, normal, active, pool0)
RAID Group /ch_n1_aggr002_FPP/plex0/rg0 (degraded, block checksums, raid_dp)
Usable Physical
Position Disk Pool Type RPM Size Size Status
-------- --------------------------- ---- ----- ------ -------- -------- ----------
shared 1.0.4 0 SAS 10000 1.61TB 1.64TB (normal)
shared FAILED - - - 1.61TB 0B (failed)
shared 1.0.6 0 SAS 10000 1.61TB 1.64TB (normal)
shared 1.0.7 0 SAS 10000 1.61TB 1.64TB (normal)
shared 1.0.8 0 SAS 10000 1.61TB 1.64TB (normal)
shared 1.0.9 0 SAS 10000 1.61TB 1.64TB (normal)
shared 1.0.10 0 SAS 10000 1.61TB 1.64TB (normal)
shared 1.0.11 0 SAS 10000 1.61TB 1.64TB (normal)
shared 1.0.12 0 SAS 10000 1.61TB 1.64TB (normal)
shared 1.0.23 0 SAS 10000 1.61TB 1.64TB (normal)
shared 1.0.14 0 SAS 10000 1.61TB 1.64TB (normal)
shared 1.0.13 0 SAS 10000 1.61TB 1.64TB (normal)
shared 1.0.16 0 SAS 10000 1.61TB 1.64TB (normal)
shared 1.0.17 0 SAS 10000 1.61TB 1.64TB (normal)
shared 1.0.18 0 SAS 10000 1.61TB 1.64TB (normal)
shared 1.0.19 0 SAS 10000 1.61TB 1.64TB (normal)
shared 1.0.15 0 SAS 10000 1.61TB 1.64TB (normal)
shared 1.0.21 0 SAS 10000 1.61TB 1.64TB (normal)
shared 1.0.20 0 SAS 10000 1.61TB 1.64TB (normal)

RAID Group /ch_n1_aggr002_FPP/plex0/rg1 (normal, block checksums, raid4) (Storage Pool: ch_n1_n3_sp_gen_001)
Usable Physical
Position Disk Pool Type RPM Size Size Status
-------- --------------------------- ---- ----- ------ -------- -------- ----------
shared 1.0.3 0 SSD - 894.2GB 3.49TB (normal)
shared 1.0.0 0 SSD - 894.2GB 3.49TB (normal)
shared 1.0.2 0 SSD - 894.2GB 3.49TB (normal)
22 entries were displayed.

xxxxxxx-001::> disk show -disk 1.0.5
Disk: 1.0.5
Container Type: shared
Owner/Home: ch-strt-gen-001 / ch-strt-gen-001
DR Home: -
Stack ID/Shelf/Bay: 1 / 0 / 5
LUN: 0
Array: N/A
Vendor: NETAPP
Model: X426_HCBFE1T8A10
Serial Number: 08G6H96Z
UID: 5000CCA0:2C0BCE98:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000
BPS: 520
Physical Size: 1.64TB
Position: shared
Checksum Compatibility: block
Aggregate: -
Plex: -
Paths:
LUN Initiator Side Target Side Link
Controller Initiator ID Switch Port Switch Port Acc Use Target Port TPGN Speed I/O KB/s IOPS
------------------ --------- ----- -------------------- -------------------- --- --- ----------------------- ------ ------- ------------ ------------
ch-strt-gen-001 0a 0 N/A N/A AO RDY 5000cca02c0bce9a 34 6 Gb/S 0 0
ch-strt-gen-001 0b 0 N/A N/A AO INU 5000cca02c0bce99 3 12 Gb/S 3 0
ch-strt-gen-003 0a 0 N/A N/A AO RDY 5000cca02c0bce99 3 6 Gb/S 0 0
ch-strt-gen-003 0b 0 N/A N/A AO INU 5000cca02c0bce9a 34 12 Gb/S 1 0

Errors:
-

xxxxxxx-001::> set -privilege advanced

Warning: These advanced commands are potentially dangerous; use them only when directed to do so by NetApp personnel.
Do you want to continue? {y|n}: y

xxxxxxx-001::*> storage disk unfail -disk 1.0.5

Warning: Failed disk "1.0.5" might have aggregate labels and file system data present. In that case, this command will attempt to bring this disk back into the aggregate with which this disk had formerly been associated and preserve
file system data. Are you sure you want to continue with disk unfail? {y|n}: y

Error: command failed: Failed to unfail the disk. Reason: Disk is not currently failed.

I already tried another disk, but same problem.

Is this because it's a different model or something else?

thank for your time

Maurice

[AlexDawson]

Try to manually fail the disk first, to put it in the failed disk registry, then try to unfail it. I note you don't have a support contract on this system - so I assume these drives have come from an unknown source. It is likely they have been sanitized before sale, which leaves them in a "broken" state. Also ensure the DQP lists the new type of disks.

Hope this helps!

[Maurice-FreeOnes]

Hi Alex, thank you for your help.

I tried what you did, but this didn't solve my problem.

Maurice

[JonathanGaudette · Accepted solution]

You have to assign the disk (or partitions in this case) for it to work. Right now, only the disk itself is assigned, not the partitions (same for 02.23):

ISK         OWNER                     POOL   SERIAL NUMBER                     HOME                     DR HOME                CHKSUM
---------    -------------             -----  -------------                     -------------            -------------          --------
0b.00.1      ch-strt-gen-001(538016980)    Pool0  S3SGNY0M700121        ch-strt-gen-001(538016980)                          Block	PRkey=538016980   UpCnt=    3 duid=5002538B:0970A62, Attachment: Local
0b.05.0      ch-strt-gen-003(538016690)    Pool0  S40TNY0K400661        ch-strt-gen-003(538016690)                          Block	PRkey=538016690   UpCnt=    3 duid=5002538B:0843C51, Attachment: Local
0b.05.8      ch-strt-gen-003(538016690)    Pool0  S40TNY0K400680        ch-strt-gen-003(538016690)                          Block	PRkey=538016690   UpCnt=    3 duid=5002538B:0843C64, Attachment: Local
0b.05.1      ch-strt-gen-003(538016690)    Pool0  S40TNY0K400668        ch-strt-gen-003(538016690)                          Block	PRkey=538016690   UpCnt=    3 duid=5002538B:0843C58, Attachment: Local
0b.05.5      ch-strt-gen-003(538016690)    Pool0  S40TNY0K400672        ch-strt-gen-003(538016690)                          Block	PRkey=538016690   UpCnt=    3 duid=5002538B:0843C5C, Attachment: Local
0b.05.6      ch-strt-gen-003(538016690)    Pool0  S40TNY0K400653        ch-strt-gen-003(538016690)                          Block	PRkey=538016690   UpCnt=    3 duid=5002538B:0843C49, Attachment: Local
0b.05.10     ch-strt-gen-003(538016690)    Pool0  S40TNY0K400654        ch-strt-gen-003(538016690)                          Block	PRkey=538016690   UpCnt=    3 duid=5002538B:0843C4A, Attachment: Local
0b.05.2      ch-strt-gen-003(538016690)    Pool0  S40TNY0K400864        ch-strt-gen-003(538016690)                          Block	PRkey=538016690   UpCnt=    3 duid=5002538B:0843FEA, Attachment: Local
0b.05.9      ch-strt-gen-003(538016690)    Pool0  S40TNY0K400685        ch-strt-gen-003(538016690)                          Block	PRkey=538016690   UpCnt=    3 duid=5002538B:0843C69, Attachment: Local
0b.05.4      ch-strt-gen-003(538016690)    Pool0  S40TNY0K400626        ch-strt-gen-003(538016690)                          Block	PRkey=538016690   UpCnt=    3 duid=5002538B:0843C2E, Attachment: Local
0b.05.11     ch-strt-gen-003(538016690)    Pool0  S40TNY0K500202        ch-strt-gen-003(538016690)                          Block	PRkey=538016690   UpCnt=    1 duid=5002538B:08508D1, Attachment: Local
0b.05.3      ch-strt-gen-003(538016690)    Pool0  S40TNY0K400636        ch-strt-gen-003(538016690)                          Block	PRkey=538016690   UpCnt=    3 duid=5002538B:0843C38, Attachment: Local
0b.05.7      ch-strt-gen-003(538016690)    Pool0  S40TNY0K400839        ch-strt-gen-003(538016690)                          Block	PRkey=538016690   UpCnt=    3 duid=5002538B:0843FD1, Attachment: Local
0a.02.23     ch-strt-gen-001(538016980)    Pool0  08GTMDWA              ch-strt-gen-001(538016980)                          Block	PRkey=        0   UpCnt=    1 duid=5000CCA0:2C2CC728, Attachment: Local
0a.02.23P1   Not Owned                  NONE   08GTMDWANP001        Block	PRkey=        0   UpCnt=    0 duid=6000CCA0:2C2CC728:500A0981:00000001, Attachment: Local
0a.02.23P2   Not Owned                  NONE   08GTMDWANP002        Block	PRkey=        0   UpCnt=    0 duid=6000CCA0:2C2CC728:500A0981:00000002, Attachment: Local
0b.00.5      ch-strt-gen-001(538016980)    Pool0  08G6H96Z              ch-strt-gen-001(538016980)                          Block	PRkey=        0   UpCnt=    1 duid=5000CCA0:2C0BCE98, Attachment: Local
0b.00.5P1    Not Owned                  NONE   08G6H96ZNP001        Block	PRkey=        0   UpCnt=    0 duid=6000CCA0:2C0BCE98:500A0981:00000001, Attachment: Local
0b.00.5P2    Not Owned                  NONE   08G6H96ZNP002        Block	PRkey=        0   UpCnt=    0 duid=6000CCA0:2C0BCE98:500A0981:00000002, Attachment: Local0b.00.0      ch-strt-gen-003(538016690)    Pool0  S2JVNCAH301219        ch-strt-gen-003(538016690)                          Block	PRkey=        0   UpCnt=    3 duid=5002538A:7630AAD, Attachment: Local
0b.00.0P1    ch-strt-gen-001(538016980)    Pool0  S2JVNCAH301219NP001   ch-strt-gen-001(538016980)                          Block	PRkey=        0   UpCnt=   22 duid=6002538A:7630AAD0:500A0981:00000001, Attachment: Local

[Maurice-FreeOnes]

Hi Jonathan, this looks promising. thank you for looking into it. I am going to try this soon and will report back to you hear.

[Maurice-FreeOnes]

Wow I feel stupid for not seeing this before 😞

Thanks a lot for your help.

For everybody who is reading this, this is the solution:

storage disk assign -disk 1.0.5 -owner ch-strt-gen-001 -root true -force true

storage disk assign -disk 1.0.5 -owner ch-strt-gen-001 -data true -force true

RESULT:

 shared 1.0.5 0 SAS 10000 1.61TB 1.64TB (reconstruction 0% completed)