Solved: E5700 volume degraded after disk change

jtejada · ‎2023-10-10

Hi all,

After drive replacement on a E5700 system, reconstruction finished fine but volume remains "degraded". As you can see all drives are in "Optimal" state. How to recover volume to optimal state?

DETAILS

Name: vg1

Status: Degraded
Capacity: 85.496 TB
Current owner: Controller in bay A

Quality of Service (QoS) Attributes

RAID level: 6
Drive media type: Hard Disk Drive
Drive interface type: Serial Attached SCSI (SAS)
Shelf loss protection: No
Drawer Loss Protection: Yes
Secure-Capable: No
Secure-Enabled No
Data Assurance (DA) capable: Yes
DA enabled volume present: Yes

Total Volumes: 1
Thick volumes: 1
Repository volumes: 0
Free Capacity: 0.000 MB

Associated drives - present (in piece order)
Total drives present: 10

Shelf Drawer Bay
0 1 3
99 2 6
0 3 9
99 4 1
0 5 4
99 1 7
0 2 10
99 3 2
0 4 5
99 5 8

SHELF, DRAWER, BAY STATUS CAPACITY MEDIA TYPE INTERFACE TYPE CURRENT DATA RATE PRODUCT ID FIRMWARE VERSION CAPABILITIES
0, 1, 3 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
99, 2, 6 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
0, 3, 9 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
99, 4, 1 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
0, 5, 4 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
99, 1, 7 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
0, 2, 10 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
99, 3, 2 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
0, 4, 5 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
99, 5, 8 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA

Regards,

ahmadm · ‎2023-10-23

Yes, you would essentially reconstruct the drive. To do that, you can run this command in the controller shell:

> startDriveReconstruction_MT 0x10025

View solution in original post

ahmadm · ‎2023-10-10

Assuming reconstruction has completed for the drive, the volume should go to optimal. However, the fact they are not suggests there could be another failed piece in the RAID group. The failure could be logical, so while the drive is optimal, the logical piece on said drive is failed.

I am not sure what level of access you have to the system, but there are some shell commands we could capture to find out which logical piece is failed.

jtejada · ‎2023-10-11

Please could you let me know these commands and I will verify.

ahmadm · ‎2023-10-16

If you are familiar with the controller shell, the commands are:

vdmShowOosPieces
evfShowVol 0
vdmShowVGInfo

jtejada · ‎2023-10-20

Here the outputs, I see OOS piece failed.

-> vdmShowOosPieces

*************************** PieceManager OOS Pieces **********************

RAIDVolume:0x1(0x1660b4850), 1 pieces OOS

Ord:0x0004 Addr:0x00000001656cc3c0 state:FAILED (Active)

0x0 - Piece Failed Timestamp: 1970 Jan 01 00:00:00

**************************************************************************

value = 76 = 0x4c = 'L'

-> evfShowVol 0

Volume 0x0(RAIDVolume)

6 VolumeListener(s)

0xc000(MediaScanAgent)

N3vdm11VolumeGroupE

N3vdm17RAIDVolumeManagerE

N3spm18SymbolEventManagerE

N3evf10CmdHandlerE

N3evf22VolumeUserLabelManagerE

0 Children

Volume 0x0 Attributes:

Volume Type: 8+2 RAID 6

User Label: d0

WWN: 6d039ea0000201a8000001a05dd3fc69

Address: 0x0000000165e0f598

Devnum: 0x10000000

Capacity: 183601463296 blocks

BlockSize: 512 bytes

LargeIoSize: 4096 blocks

MaxBkgdIoSize: 4096 blocks

Has Extents: false

Ownership: This controller

PreferredPath: This controller

Transferring: false

StopIOInProgress: false

Suspended: 0

Quiesce Count: 0

Quiesce Both Ctl: 0 (Local: 0/Alt: 0)

State: RV_OPTIMAL

Unreadable Sectors:Not Present

App Tag: 0x0000

App Tag Owned: false

Protection Type: 1

PI Reporting: enabled

PI Mirroring: enabled

isSecure: false

isSecureCapable: false

isFIPS: false

isExopActive: true

isPersistent: true

isTCGSSupported: true

skipped URS: 0

NonCriticalTransferAllowed: true

Permissions: CONFIG=Y CPYsrc=Y CPYTGT=Y

DRIVESACCESSIBLE=Y FORMAT=Y FORMATTING=N

IMPORT=Y MAP=Y MIRPRI=Y

MIRSEC=Y MISSING=N MOVE_RELATED=Y

NON_RED=Y OFFLINE=N READ=Y

READY=Y RECONFG=Y REP_NOT_ON_PREF=Y

SNAP=Y TRANSFER=Y WRITE=Y

ROLLBACK=Y MIRACTV=Y NEWPIT=Y

DVE=Y DSS=Y CONCAT=N

FRC=Y PITGROUP=Y ASYNC_MIRPRI=Y

ASYNC_MIRSEC=Y DSP=Y CPC=Y

THIN_MANEXP=Y THIN_GROWPARAM=Y XCPY=Y

4KDSS=Y VAULT=Y RESTORE=Y

VAULT_GRP=Y ALBTRANSFER=Y PRIORITYXFER=Y

Activity Count: 1

QueueDepth: 0

EUI: 000001a05dd3fc69d039ea00000201a8

IAF Aborted: 0

Segment Size: 256 blocks

Stripe Size: 2048 blocks

Drive Recovery Mode: RECOVERY_INVALID

PreConfigedPI: 0

isVolumeCopyActive: False

isVolumeCopyInsertPI: False

DSS PreAllocation: Enabled

AbsoluteMaxDSSSegSize: 4096

Pre-Read Redundancy Check: Disabled

Retry on Mismatch: Enabled

Repair Writes: Disabled

Flash Read Cache: Disabled

Atomic Write Support: Disabled

Volume Usage: STANDARD

Initialization: Completed

Offline Type: 0

NoDemandFlush: False

Read Ahead: 1

Fast Write: 1

Sync Age: 2500

Cache Flush Modifier: 8

Min Warn Flush Modifier: 8

Cache Granularity: 64

Ignore FUA: Host type(s) - 0 1 2 5 6 7 8 9 10 15 17 22 23 24 25 26 27 28 29

Media Scan Settings: Enabled with Parity Check, but no Repair.

Period = 30 day(s). Current rate = 88542 blocks/sec.

Background Task: MEDIA SCAN at block 0x0211a246800 with 77% complete

Background Delay: 0.1 seconds per 4096 blocks

*** Volume Group Info ***

WWN : 6d039ea0000201a8000001945dd3fc4a

RAID Level : RAID 6

Boundary : 142172514304 0x211a246800

Persisted : 0 0x0

VG Label : vg0

Address : 0x000000024669b6b0

Drive Count: 10

Secure : False

PI Capable : True - 2

Media Type : HDD

Volumes on this Group:

Volume Count: 1

0x00000 [O]

*** Pieces ***

Offset: 0x0 Length: 0x557f00000 (22950182912 dec)

Count : 10 Data Count: 8

Piece Devnum Address Tray/Slot State

0 0x00010100 0x00000001656ff850 00,01 PieceOptimalState

1 0x0001000f 0x0000000165796640 99,16 PieceOptimalState

2 0x0001011e 0x00000001656e8f58 00,31 PieceOptimalState

3 0x0001002d 0x00000001656e8e48 99,46 PieceOptimalState

4 0x00010131 0x0000000165786280 00,50 PieceOptimalState

5 0x00010004 0x000000016575b9b8 99,05 PieceOptimalState

6 0x00010113 0x0000000165747760 00,20 PieceOptimalState

7 0x00010022 0x00000001657475f0 99,35 PieceOptimalState

8 0x00010126 0x00000001656e0ba0 00,39 PieceOptimalState

9 0x00010035 0x000000016571a330 99,54 PieceOptimalState

StripesPerCachePoolExtent: 16

BlocksPerCachePoolExtent : 32768

*** CACHE DEVICE 0 ***

Sector Size Secs/CacheBlock Cache Blk Size Clean Minimum

512 bytes 64 32768 bytes 0000208000

Dirty Cache Blks Dirty Cache High per Pool

5129 0:12771 1:15775 2:12761

Hash Data Blks Hash Parity Blks

150394 0

RD_CACHE WR_CACHE WR_CACHE_MIR FWT

ENABLED ENABLED ENABLED DISABLED

CWOB WT_BU_BATT_ON READ_CACHE WRITE_CACHE WRITE_CACHE_MIR

DISABLED NO ACTIVE ACTIVE ACTIVE

HIGH_PRIORITY_DEV: NO

Background sync is in progress : NO

CacheBlockMask: ffffffffffffffff

Cache Statistics:

current old current old

cache read read parity parity disk

VOL reads hits hits hits hits reads

---- --------- --------- --------- --------- --------- ---------

0 23089135 3983591 243 0 47308 20092317

read read

check check insert full partial disk

VOL reqs hits reqs overwrite overwrite writes

---- --------- --------- --------- --------- --------- ---------

0 4014500 2033274 965518 3290 187790 0

Cache Pool 0 Cache Pool 1 Cache Pool 2

------------------- ------------------- -------------------

WB blks % Dirty WB blks % Dirty WB blks % Dirty WB blks % Dirty

VOL current Limit high Limit high Limit high Limit

---- --------- --------- --------- --------- --------- --------- --------- ---------

0 5129 2 12771 6 15775 7 12761 6

+---------- cache blocks evicted after ---------+ evicted blocks

VOL fetch hit prefetch pftch hit write total in-use

---- --------- --------- --------- --------- --------- --------- ---------

0 18137518 70172 23524605 29676370 11431433 82840098 150394

prefetch prefetch pct

VOL hit blocks miss blocks hits

---- ------------------- ------------------- ----

0 1896349824 1505574720 55

value = 139928058009584 = 0x7f438a3117f0

-> vdmShowVGInfo

Total Volume Groups: ............. 12

Seq:1 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg0 / VGWwn:6d039ea0000201a8000001945dd3fc4a / RsvdSpace: 5 (GB)

(Active) Drive:0x000000024661e240 devnum:0x00010100 seqNum:1 Tray/Slot:00/01 State:Acc/GrA/Opt

(Active) Drive:0x000000024661e530 devnum:0x0001000f seqNum:2 Tray/Slot:99/16 State:Acc/GrA/Opt

(Active) Drive:0x000000024661e820 devnum:0x0001011e seqNum:3 Tray/Slot:00/31 State:Acc/GrA/Opt

(Active) Drive:0x000000024661eb10 devnum:0x0001002d seqNum:4 Tray/Slot:99/46 State:Acc/GrA/Opt

(Active) Drive:0x00000002466352e0 devnum:0x00010131 seqNum:5 Tray/Slot:00/50 State:Acc/GrA/Opt

(Active) Drive:0x000000024661f0f0 devnum:0x00010004 seqNum:6 Tray/Slot:99/05 State:Acc/GrA/Opt

(Active) Drive:0x000000024661ee00 devnum:0x00010113 seqNum:7 Tray/Slot:00/20 State:Acc/GrA/Opt

(Active) Drive:0x000000024661f6d0 devnum:0x00010022 seqNum:8 Tray/Slot:99/35 State:Acc/GrA/Opt

(Active) Drive:0x000000024661f9c0 devnum:0x00010126 seqNum:9 Tray/Slot:00/39 State:Acc/GrA/Opt

(Active) Drive:0x000000024661fcb0 devnum:0x00010035 seqNum:10 Tray/Slot:99/54 State:Acc/GrA/Opt

Volumes: 0x00000 [O]

Seq:2 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg1 / VGWwn:6d039ea0000201a8000001955dd3fc4c / RsvdSpace: 5 (GB)

(Active) Drive:0x000000024661ffa0 devnum:0x00010103 seqNum:1 Tray/Slot:00/04 State:Acc/GrA/Opt

(Active) Drive:0x0000000246620290 devnum:0x00010012 seqNum:2 Tray/Slot:99/19 State:Acc/GrA/Opt

(Active) Drive:0x0000000246620580 devnum:0x00010121 seqNum:3 Tray/Slot:00/34 State:Acc/GrA/Opt

(Active) Drive:0x0000000246a3f2d0 devnum:0x00010025 seqNum:4 Tray/Slot:99/38 State:Acc/GrA/Opt

(Active) Drive:0x0000000246620b60 devnum:0x00010134 seqNum:5 Tray/Slot:00/53 State:Acc/GrA/Opt

(Active) Drive:0x000000024683dc48 devnum:0x00010007 seqNum:6 Tray/Slot:99/08 State:Acc/GrA/Opt

(Active) Drive:0x00000002496193c8 devnum:0x00010116 seqNum:7 Tray/Slot:00/23 State:Acc/GrA/Opt

(Active) Drive:0x0000000246621430 devnum:0x0001001a seqNum:8 Tray/Slot:99/27 State:Acc/GrA/Opt

(Active) Drive:0x0000000246621720 devnum:0x00010129 seqNum:9 Tray/Slot:00/42 State:Acc/GrA/Opt

(Active) Drive:0x0000000246621a10 devnum:0x00010038 seqNum:10 Tray/Slot:99/57 State:Acc/GrA/Opt

Volumes: 0x00001 [D]

Seq:3 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:3 / PI Capable:T - 2 / Label:vg2 / VGWwn:6d039ea0000201a8000001965dd3fc4d / RsvdSpace: 5 (GB)

(Active) Drive:0x0000000246621d00 devnum:0x00010106 seqNum:1 Tray/Slot:00/07 State:Acc/GrA/Opt

(Active) Drive:0x0000000246621ff0 devnum:0x00010015 seqNum:2 Tray/Slot:99/22 State:Acc/GrA/Opt

(Active) Drive:0x00000002466222e0 devnum:0x00010119 seqNum:3 Tray/Slot:00/26 State:Acc/GrA/Opt

(Active) Drive:0x00000002466225d0 devnum:0x00010028 seqNum:4 Tray/Slot:99/41 State:Acc/GrA/Opt

(Active) Drive:0x00000002466228c0 devnum:0x00010137 seqNum:5 Tray/Slot:00/56 State:Acc/GrA/Opt

(Active) Drive:0x000000024661f3e0 devnum:0x0001000a seqNum:6 Tray/Slot:99/11 State:Acc/GrA/Opt

(Active) Drive:0x0000000246631820 devnum:0x0001010e seqNum:7 Tray/Slot:00/15 State:Acc/GrA/Opt

(Active) Drive:0x0000000246623190 devnum:0x0001001d seqNum:8 Tray/Slot:99/30 State:Acc/GrA/Opt

(Active) Drive:0x0000000246622ea0 devnum:0x0001012c seqNum:9 Tray/Slot:00/45 State:Acc/GrA/Opt

(Active) Drive:0x0000000246623770 devnum:0x0001003b seqNum:10 Tray/Slot:99/60 State:Acc/GrA/Opt

Volumes: 0x00002 [O]

Seq:4 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:3 / PI Capable:T - 2 / Label:vg3 / VGWwn:6d039ea0000201a8000001975dd3fc4f / RsvdSpace: 5 (GB)

(Active) Drive:0x0000000246623a60 devnum:0x00010109 seqNum:1 Tray/Slot:00/10 State:Acc/GrA/Opt

(Active) Drive:0x00000002466280e0 devnum:0x0001000d seqNum:2 Tray/Slot:99/14 State:Acc/GrA/Opt

(Active) Drive:0x0000000246624040 devnum:0x0001011c seqNum:3 Tray/Slot:00/29 State:Acc/GrA/Opt

(Active) Drive:0x0000000246624330 devnum:0x0001002b seqNum:4 Tray/Slot:99/44 State:Acc/GrA/Opt

(Active) Drive:0x0000000246624620 devnum:0x0001013a seqNum:5 Tray/Slot:00/59 State:Acc/GrA/Opt

(Active) Drive:0x0000000246624910 devnum:0x00010002 seqNum:6 Tray/Slot:99/03 State:Acc/GrA/Opt

(Active) Drive:0x0000000246624c00 devnum:0x00010111 seqNum:7 Tray/Slot:00/18 State:Acc/GrA/Opt

(Active) Drive:0x0000000246624ef0 devnum:0x00010020 seqNum:8 Tray/Slot:99/33 State:Acc/GrA/Opt

(Active) Drive:0x00000002466251e0 devnum:0x0001012f seqNum:9 Tray/Slot:00/48 State:Acc/GrA/Opt

(Active) Drive:0x00000002466254d0 devnum:0x00010030 seqNum:10 Tray/Slot:99/49 State:Acc/GrA/Opt

Volumes: 0x00003 [O]

Seq:5 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg4 / VGWwn:6d039ea0000201a8000001985dd3fc50 / RsvdSpace: 5 (GB)

(Active) Drive:0x00000002466257c0 devnum:0x00010101 seqNum:1 Tray/Slot:00/02 State:Acc/GrA/Opt

(Active) Drive:0x0000000246625ab0 devnum:0x00010010 seqNum:2 Tray/Slot:99/17 State:Acc/GrA/Opt

(Active) Drive:0x0000000246625da0 devnum:0x0001011f seqNum:3 Tray/Slot:00/32 State:Acc/GrA/Opt

(Active) Drive:0x0000000246626090 devnum:0x0001002e seqNum:4 Tray/Slot:99/47 State:Acc/GrA/Opt

(Active) Drive:0x0000000246626380 devnum:0x00010132 seqNum:5 Tray/Slot:00/51 State:Acc/GrA/Opt

(Active) Drive:0x0000000246626670 devnum:0x00010005 seqNum:6 Tray/Slot:99/06 State:Acc/GrA/Opt

(Active) Drive:0x0000000246626960 devnum:0x00010114 seqNum:7 Tray/Slot:00/21 State:Acc/GrA/Opt

(Active) Drive:0x0000000246626c50 devnum:0x00010023 seqNum:8 Tray/Slot:99/36 State:Acc/GrA/Opt

(Active) Drive:0x0000000246626f40 devnum:0x00010124 seqNum:9 Tray/Slot:00/37 State:Acc/GrA/Opt

(Active) Drive:0x000000024663a230 devnum:0x00010033 seqNum:10 Tray/Slot:99/52 State:Acc/GrA/Opt

Volumes: 0x00004 [O]

Seq:6 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg5 / VGWwn:6d039ea0000201a8000001995dd3fc52 / RsvdSpace: 5 (GB)

(Active) Drive:0x0000000246627520 devnum:0x00010104 seqNum:1 Tray/Slot:00/05 State:Acc/GrA/Opt

(Active) Drive:0x0000000246627810 devnum:0x00010013 seqNum:2 Tray/Slot:99/20 State:Acc/GrA/Opt

(Active) Drive:0x0000000246627b00 devnum:0x00010122 seqNum:3 Tray/Slot:00/35 State:Acc/GrA/Opt

(Active) Drive:0x0000000246627df0 devnum:0x00010026 seqNum:4 Tray/Slot:99/39 State:Acc/GrA/Opt

(Active) Drive:0x0000000246632cb0 devnum:0x00010135 seqNum:5 Tray/Slot:00/54 State:Acc/GrA/Opt

(Active) Drive:0x00000002466283d0 devnum:0x00010008 seqNum:6 Tray/Slot:99/09 State:Acc/GrA/Opt

(Active) Drive:0x00000002466286c0 devnum:0x00010117 seqNum:7 Tray/Slot:00/24 State:Acc/GrA/Opt

(Active) Drive:0x00000002466289b0 devnum:0x00010018 seqNum:8 Tray/Slot:99/25 State:Acc/GrA/Opt

(Active) Drive:0x0000000246628ca0 devnum:0x00010127 seqNum:9 Tray/Slot:00/40 State:Acc/GrA/Opt

(Active) Drive:0x0000000246628f90 devnum:0x00010036 seqNum:10 Tray/Slot:99/55 State:Acc/GrA/Opt

Volumes: 0x00005 [O]

Seq:7 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg6 / VGWwn:6d039ea0000201a80000019a5dd3fc53 / RsvdSpace: 5 (GB)

(Active) Drive:0x000000024662f4e0 devnum:0x00010107 seqNum:1 Tray/Slot:00/08 State:Acc/GrA/Opt

(Active) Drive:0x000000024662f7d0 devnum:0x00010016 seqNum:2 Tray/Slot:99/23 State:Acc/GrA/Opt

(Active) Drive:0x000000024662fac0 devnum:0x0001011a seqNum:3 Tray/Slot:00/27 State:Acc/GrA/Opt

(Active) Drive:0x000000024662fdb0 devnum:0x00010029 seqNum:4 Tray/Slot:99/42 State:Acc/GrA/Opt

(Active) Drive:0x00000002466300a0 devnum:0x00010138 seqNum:5 Tray/Slot:00/57 State:Acc/GrA/Opt

(Active) Drive:0x0000000246630390 devnum:0x0001000b seqNum:6 Tray/Slot:99/12 State:Acc/GrA/Opt

(Active) Drive:0x0000000246630680 devnum:0x0001010c seqNum:7 Tray/Slot:00/13 State:Acc/GrA/Opt

(Active) Drive:0x0000000246630970 devnum:0x0001001b seqNum:8 Tray/Slot:99/28 State:Acc/GrA/Opt

(Active) Drive:0x0000000246630c60 devnum:0x0001012a seqNum:9 Tray/Slot:00/43 State:Acc/GrA/Opt

(Active) Drive:0x0000000246630f50 devnum:0x00010039 seqNum:10 Tray/Slot:99/58 State:Acc/GrA/Opt

Volumes: 0x00006 [O]

Seq:8 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg7 / VGWwn:6d039ea0000201a80000019b5dd3fc55 / RsvdSpace: 5 (GB)

(Active) Drive:0x0000000246631240 devnum:0x0001010a seqNum:1 Tray/Slot:00/11 State:Acc/GrA/Opt

(Active) Drive:0x0000000246631530 devnum:0x0001000e seqNum:2 Tray/Slot:99/15 State:Acc/GrA/Opt

(Active) Drive:0x0000000246639380 devnum:0x0001011d seqNum:3 Tray/Slot:00/30 State:Acc/GrA/Opt

(Active) Drive:0x00000002492e4468 devnum:0x0001002c seqNum:4 Tray/Slot:99/45 State:Acc/GrA/Opt

(Active) Drive:0x0000000246631e00 devnum:0x0001013b seqNum:5 Tray/Slot:00/60 State:Acc/GrA/Opt

(Active) Drive:0x00000002466320f0 devnum:0x00010000 seqNum:6 Tray/Slot:99/01 State:Acc/GrA/Opt

(Active) Drive:0x00000002466323e0 devnum:0x0001010f seqNum:7 Tray/Slot:00/16 State:Acc/GrA/Opt

(Active) Drive:0x00000002466326d0 devnum:0x0001001e seqNum:8 Tray/Slot:99/31 State:Acc/GrA/Opt

(Active) Drive:0x00000002466329c0 devnum:0x0001012d seqNum:9 Tray/Slot:00/46 State:Acc/GrA/Opt

(Active) Drive:0x0000000246639960 devnum:0x00010031 seqNum:10 Tray/Slot:99/50 State:Acc/GrA/Opt

Volumes: 0x00007 [O]

Seq:9 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg8 / VGWwn:6d039ea0000201a80000019c5dd3fc56 / RsvdSpace: 5 (GB)

(Active) Drive:0x0000000246632fa0 devnum:0x00010102 seqNum:1 Tray/Slot:00/03 State:Acc/GrA/Opt

(Active) Drive:0x0000000246633290 devnum:0x00010011 seqNum:2 Tray/Slot:99/18 State:Acc/GrA/Opt

(Active) Drive:0x0000000246633580 devnum:0x00010120 seqNum:3 Tray/Slot:00/33 State:Acc/GrA/Opt

(Active) Drive:0x0000000246633870 devnum:0x0001002f seqNum:4 Tray/Slot:99/48 State:Acc/GrA/Opt

(Active) Drive:0x0000000246633b60 devnum:0x00010130 seqNum:5 Tray/Slot:00/49 State:Acc/GrA/Opt

(Active) Drive:0x0000000246633e50 devnum:0x00010003 seqNum:6 Tray/Slot:99/04 State:Acc/GrA/Opt

(Active) Drive:0x0000000246634140 devnum:0x00010112 seqNum:7 Tray/Slot:00/19 State:Acc/GrA/Opt

(Active) Drive:0x0000000246634430 devnum:0x00010021 seqNum:8 Tray/Slot:99/34 State:Acc/GrA/Opt

(Active) Drive:0x0000000246634720 devnum:0x00010125 seqNum:9 Tray/Slot:00/38 State:Acc/GrA/Opt

(Active) Drive:0x0000000246634a10 devnum:0x00010034 seqNum:10 Tray/Slot:99/53 State:Acc/GrA/Opt

Volumes: 0x00008 [O]

Seq:10 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg9 / VGWwn:6d039ea0000201a80000019d5dd3fc57 / RsvdSpace: 5 (GB)

(Active) Drive:0x0000000246634d00 devnum:0x00010105 seqNum:1 Tray/Slot:00/06 State:Acc/GrA/Opt

(Active) Drive:0x0000000246627230 devnum:0x00010014 seqNum:2 Tray/Slot:99/21 State:Acc/GrA/Opt

(Active) Drive:0x0000000246623d50 devnum:0x00010123 seqNum:3 Tray/Slot:00/36 State:Acc/GrA/Opt

(Active) Drive:0x00000002466355d0 devnum:0x00010024 seqNum:4 Tray/Slot:99/37 State:Acc/GrA/Opt

(Active) Drive:0x00000002466358c0 devnum:0x00010133 seqNum:5 Tray/Slot:00/52 State:Acc/GrA/Opt

(Active) Drive:0x0000000246635bb0 devnum:0x00010006 seqNum:6 Tray/Slot:99/07 State:Acc/GrA/Opt

(Active) Drive:0x0000000246635ea0 devnum:0x00010115 seqNum:7 Tray/Slot:00/22 State:Acc/GrA/Opt

(Active) Drive:0x0000000246636190 devnum:0x00010019 seqNum:8 Tray/Slot:99/26 State:Acc/GrA/Opt

(Active) Drive:0x0000000246636480 devnum:0x00010128 seqNum:9 Tray/Slot:00/41 State:Acc/GrA/Opt

(Active) Drive:0x0000000246636770 devnum:0x00010037 seqNum:10 Tray/Slot:99/56 State:Acc/GrA/Opt

Volumes: 0x00009 [O]

Seq:11 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg10 / VGWwn:6d039ea0000201a80000019e5dd3fc59 / RsvdSpace: 5 (GB)

(Active) Drive:0x0000000246636a60 devnum:0x00010108 seqNum:1 Tray/Slot:00/09 State:Acc/GrA/Opt

(Active) Drive:0x0000000246636d50 devnum:0x00010017 seqNum:2 Tray/Slot:99/24 State:Acc/GrA/Opt

(Active) Drive:0x0000000246637040 devnum:0x00010118 seqNum:3 Tray/Slot:00/25 State:Acc/GrA/Opt

(Active) Drive:0x0000000246637330 devnum:0x00010027 seqNum:4 Tray/Slot:99/40 State:Acc/GrA/Opt

(Active) Drive:0x0000000246637620 devnum:0x00010136 seqNum:5 Tray/Slot:00/55 State:Acc/GrA/Opt

(Active) Drive:0x0000000246637910 devnum:0x00010009 seqNum:6 Tray/Slot:99/10 State:Acc/GrA/Opt

(Active) Drive:0x0000000246637c00 devnum:0x0001010d seqNum:7 Tray/Slot:00/14 State:Acc/GrA/Opt

(Active) Drive:0x0000000246637ef0 devnum:0x0001001c seqNum:8 Tray/Slot:99/29 State:Acc/GrA/Opt

(Active) Drive:0x00000002466381e0 devnum:0x0001012b seqNum:9 Tray/Slot:00/44 State:Acc/GrA/Opt

(Active) Drive:0x00000002466384d0 devnum:0x0001003a seqNum:10 Tray/Slot:99/59 State:Acc/GrA/Opt

Volumes: 0x0000a [O]

Seq:12 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:3 / PI Capable:T - 2 / Label:vg11 / VGWwn:6d039ea0000201a80000019f5dd3fc5a / RsvdSpace: 5 (GB)

(Active) Drive:0x0000000246622bb0 devnum:0x0001010b seqNum:1 Tray/Slot:00/12 State:Acc/GrA/Opt

(Active) Drive:0x00000002466387c0 devnum:0x0001000c seqNum:2 Tray/Slot:99/13 State:Acc/GrA/Opt

(Active) Drive:0x0000000246638ab0 devnum:0x0001011b seqNum:3 Tray/Slot:00/28 State:Acc/GrA/Opt

(Active) Drive:0x0000000246638da0 devnum:0x0001002a seqNum:4 Tray/Slot:99/43 State:Acc/GrA/Opt

(Active) Drive:0x0000000246639090 devnum:0x00010139 seqNum:5 Tray/Slot:00/58 State:Acc/GrA/Opt

(Active) Drive:0x0000000246634ff0 devnum:0x00010001 seqNum:6 Tray/Slot:99/02 State:Acc/GrA/Opt

(Active) Drive:0x0000000246639670 devnum:0x00010110 seqNum:7 Tray/Slot:00/17 State:Acc/GrA/Opt

(Active) Drive:0x0000000246623480 devnum:0x0001001f seqNum:8 Tray/Slot:99/32 State:Acc/GrA/Opt

(Active) Drive:0x0000000246639c50 devnum:0x0001012e seqNum:9 Tray/Slot:00/47 State:Acc/GrA/Opt

(Active) Drive:0x0000000246639f40 devnum:0x00010032 seqNum:10 Tray/Slot:99/51 State:Acc/GrA/Opt

Volumes: 0x0000b [O]

value = 139927976262160 = 0x7f438551ba10

ahmadm · ‎2023-10-20

It appears that drive 99,4,1 is the one with the remaining logical piece failure. If you can show me the output of "evfShowVol 1" command, I can confirm 100%.

If that is the case, the safest solution would be to fail the drive then reconstruct it in the UI.

jtejada · ‎2023-10-23

Here the output. Yes, drive 99,4,1 was the drive replaced due to Impending failure status. How to fail the drive? Directly extract it? Or maybe directly making reconstruct in the UI should solve it.

-> evfShowVol 1

Volume 0x1(RAIDVolume)

6 VolumeListener(s)

0xc001(MediaScanAgent)

N3vdm11VolumeGroupE

N3vdm17RAIDVolumeManagerE

N3spm18SymbolEventManagerE

N3evf10CmdHandlerE

N3evf22VolumeUserLabelManagerE

0 Children

Volume 0x1 Attributes:

Volume Type: 8+2 RAID 6

User Label: d1

WWN: 6d039ea0000201a8000001a35dd3fc77

Address: 0x00000001660b4850

Devnum: 0x10000001

Capacity: 183601463296 blocks

BlockSize: 512 bytes

LargeIoSize: 4096 blocks

MaxBkgdIoSize: 4096 blocks

Has Extents: false

Ownership: This controller

PreferredPath: This controller

Transferring: false

StopIOInProgress: false

Suspended: 0

Quiesce Count: 0

Quiesce Both Ctl: 0 (Local: 0/Alt: 0)

State: RV_DEGRADED

Unreadable Sectors:Not Present

App Tag: 0x0000

App Tag Owned: false

Protection Type: 1

PI Reporting: enabled

PI Mirroring: enabled

isSecure: false

isSecureCapable: false

isFIPS: false

isExopActive: true

isPersistent: true

isTCGSSupported: true

skipped URS: 0

NonCriticalTransferAllowed: true

Permissions: CONFIG=Y CPYsrc=Y CPYTGT=Y

DRIVESACCESSIBLE=Y FORMAT=Y FORMATTING=N

IMPORT=Y MAP=Y MIRPRI=Y

MIRSEC=Y MISSING=N MOVE_RELATED=Y

NON_RED=Y OFFLINE=N READ=Y

READY=Y RECONFG=Y REP_NOT_ON_PREF=Y

SNAP=Y TRANSFER=Y WRITE=Y

ROLLBACK=Y MIRACTV=Y NEWPIT=Y

DVE=Y DSS=Y CONCAT=N

FRC=Y PITGROUP=Y ASYNC_MIRPRI=Y

ASYNC_MIRSEC=Y DSP=Y CPC=Y

THIN_MANEXP=Y THIN_GROWPARAM=Y XCPY=Y

4KDSS=Y VAULT=Y RESTORE=Y

VAULT_GRP=Y ALBTRANSFER=Y PRIORITYXFER=Y

Activity Count: 1

QueueDepth: 0

EUI: 000001a35dd3fc77d039ea00000201a8

IAF Aborted: 0

Segment Size: 256 blocks

Stripe Size: 2048 blocks

Drive Recovery Mode: RECOVERY_INVALID

PreConfigedPI: 0

isVolumeCopyActive: False

isVolumeCopyInsertPI: False

DSS PreAllocation: Enabled

AbsoluteMaxDSSSegSize: 4096

Pre-Read Redundancy Check: Disabled

Retry on Mismatch: Enabled

Repair Writes: Disabled

Flash Read Cache: Disabled

Atomic Write Support: Disabled

Volume Usage: STANDARD

Initialization: Completed

Offline Type: 0

NoDemandFlush: False

Read Ahead: 1

Fast Write: 1

Sync Age: 2500

Cache Flush Modifier: 8

Min Warn Flush Modifier: 8

Cache Granularity: 64

Ignore FUA: Host type(s) - 0 1 2 5 6 7 8 9 10 15 17 22 23 24 25 26 27 28 29

Media Scan Settings: Enabled with Parity Check, but no Repair.

Period = 30 day(s). Current rate = 88542 blocks/sec.

Background Task: MEDIA SCAN at block 0x027d7ded800 with 93% complete

Background Delay: 0.1 seconds per 4096 blocks

*** Volume Group Info ***

WWN : 6d039ea0000201a8000001955dd3fc4c

RAID Level : RAID 6

Boundary : 171125430272 0x27d7ded800

Persisted : 0 0x0

VG Label : vg1

Address : 0x000000024669de00

Drive Count: 10

Secure : False

PI Capable : True - 2

Media Type : HDD

Volumes on this Group:

Volume Count: 1

0x00001 [D]

*** Pieces ***

Offset: 0x0 Length: 0x557f00000 (22950182912 dec)

Count : 10 Data Count: 8

Piece Devnum Address Tray/Slot State

0 0x00010103 0x000000016575fa40 00,04 PieceOptimalState

1 0x00010012 0x0000000165732d20 99,19 PieceOptimalState

2 0x00010121 0x0000000165732c10 00,34 PieceOptimalState

3 0x00010025 0x00000001656cc3c0 99,38 FAILED

4 0x00010134 0x00000001656f5408 00,53 PieceOptimalState

5 0x00010007 0x00000001656f52f8 99,08 PieceOptimalState

6 0x00010116 0x000000016576bca0 00,23 PieceOptimalState

7 0x0001001a 0x00000001657a2978 99,27 PieceOptimalState

8 0x00010129 0x000000016579e910 00,42 PieceOptimalState

9 0x00010038 0x00000001656ff960 99,57 PieceOptimalState

StripesPerCachePoolExtent: 16

BlocksPerCachePoolExtent : 32768

*** CACHE DEVICE 1 ***

Sector Size Secs/CacheBlock Cache Blk Size Clean Minimum

512 bytes 64 32768 bytes 0000208000

Dirty Cache Blks Dirty Cache High per Pool

3896 0:18924 1:16870 2:15379

Hash Data Blks Hash Parity Blks

152029 0

RD_CACHE WR_CACHE WR_CACHE_MIR FWT

ENABLED ENABLED ENABLED DISABLED

CWOB WT_BU_BATT_ON READ_CACHE WRITE_CACHE WRITE_CACHE_MIR

DISABLED NO ACTIVE ACTIVE ACTIVE

HIGH_PRIORITY_DEV: NO

Background sync is in progress : NO

CacheBlockMask: ffffffffffffffff

Cache Statistics:

current old current old

cache read read parity parity disk

VOL reads hits hits hits hits reads

---- --------- --------- --------- --------- --------- ---------

1 8813418 3365511 945332 0 88559 5705963

read read

check check insert full partial disk

VOL reqs hits reqs overwrite overwrite writes

---- --------- --------- --------- --------- --------- ---------

1 2462860 979939 361100 7364 114205 0

Cache Pool 0 Cache Pool 1 Cache Pool 2

------------------- ------------------- -------------------

WB blks % Dirty WB blks % Dirty WB blks % Dirty WB blks % Dirty

VOL current Limit high Limit high Limit high Limit

---- --------- --------- --------- --------- --------- --------- --------- ---------

1 3896 1 18924 9 16870 8 15379 7

+---------- cache blocks evicted after ---------+ evicted blocks

VOL fetch hit prefetch pftch hit write total in-use

---- --------- --------- --------- --------- --------- --------- ---------

1 2839391 2105770 1010947 5949256 4849966 16755330 151999

prefetch prefetch pct

VOL hit blocks miss blocks hits

---- ------------------- ------------------- ----

1 106899520 64700608 62

value = 139928058009584 = 0x7f438a3117f0

ahmadm · ‎2023-10-23

Yes, you would essentially reconstruct the drive. To do that, you can run this command in the controller shell:

> startDriveReconstruction_MT 0x10025

E5700 volume degraded after disk change

Introducing GenAI Search on NSS