E5700 volume degraded after disk change

[jtejada]

Hi all,

After drive replacement on a E5700 system, reconstruction finished fine but volume remains "degraded".  As you can see all drives are in "Optimal" state. How to recover volume to optimal state?

DETAILS

Name: vg1

Status: Degraded
Capacity: 85.496 TB
Current owner: Controller in bay A

Quality of Service (QoS) Attributes

RAID level: 6
Drive media type: Hard Disk Drive
Drive interface type: Serial Attached SCSI (SAS)
Shelf loss protection: No
Drawer Loss Protection: Yes
Secure-Capable: No
Secure-Enabled No
Data Assurance (DA) capable: Yes
DA enabled volume present: Yes

Total Volumes: 1
Thick volumes: 1
Repository volumes: 0
Free Capacity: 0.000 MB

Associated drives - present (in piece order)
Total drives present: 10

Shelf Drawer Bay
0 1 3
99 2 6
0 3 9
99 4 1
0 5 4
99 1 7
0 2 10
99 3 2
0 4 5
99 5 8

SHELF, DRAWER, BAY STATUS CAPACITY MEDIA TYPE INTERFACE TYPE CURRENT DATA RATE PRODUCT ID FIRMWARE VERSION CAPABILITIES
0, 1, 3 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
99, 2, 6 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
0, 3, 9 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
99, 4, 1 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
0, 5, 4 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
99, 1, 7 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
0, 2, 10 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
99, 3, 2 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
0, 4, 5 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA
99, 5, 8 Optimal 10.692 TB Hard Disk Drive SAS 12 Gbps HUH721212AL5204 NE00 DA

Regards,

[ahmadm]

Assuming reconstruction has completed for the drive, the volume should go to optimal. However, the fact they are not suggests there could be another failed piece in the RAID group. The failure could be logical, so while the drive is optimal, the logical piece on said drive is failed.

I am not sure what level of access you have to the system, but there are some shell commands we could capture to find out which logical piece is failed.

[jtejada]

Please could you let me know these commands and I will verify.

[ahmadm]

If you are familiar with the controller shell, the commands are:

• vdmShowOosPieces
• evfShowVol 0
• vdmShowVGInfo

[jtejada]

Here the outputs, I see OOS piece failed.

-> vdmShowOosPieces

*************************** PieceManager OOS Pieces **********************

RAIDVolume:0x1(0x1660b4850), 1 pieces OOS

  Ord:0x0004 Addr:0x00000001656cc3c0 state:FAILED (Active)

  0x0 - Piece Failed Timestamp: 1970 Jan 01  00:00:00

**************************************************************************

value = 76 = 0x4c = 'L'

-> evfShowVol 0

Volume 0x0(RAIDVolume)

 6 VolumeListener(s)

     0xc000(MediaScanAgent)

     N3vdm11VolumeGroupE

     N3vdm17RAIDVolumeManagerE

     N3spm18SymbolEventManagerE

     N3evf10CmdHandlerE

     N3evf22VolumeUserLabelManagerE

 0 Children

Volume 0x0 Attributes:

     Volume Type:       8+2 RAID 6

     User Label:        d0

     WWN:               6d039ea0000201a8000001a05dd3fc69

     Address:           0x0000000165e0f598

     Devnum:            0x10000000

     Capacity:          183601463296 blocks

     BlockSize:         512 bytes

     LargeIoSize:       4096 blocks

     MaxBkgdIoSize:     4096 blocks

     Has Extents:       false

     Ownership:         This controller

     PreferredPath:     This controller

     Transferring:      false

     StopIOInProgress:  false

     Suspended:         0

     Quiesce Count:     0

     Quiesce Both Ctl:  0 (Local: 0/Alt: 0)

     State:             RV_OPTIMAL

     Unreadable Sectors:Not Present

     App Tag:           0x0000

     App Tag Owned:     false

     Protection Type:   1

     PI Reporting:      enabled

     PI Mirroring:      enabled

     isSecure:          false

     isSecureCapable:   false

     isFIPS:            false

     isExopActive:      true

     isPersistent:      true

     isTCGSSupported:   true

     skipped URS:       0

     NonCriticalTransferAllowed: true

     Permissions:       CONFIG=Y            CPYsrc=Y            CPYTGT=Y

                        DRIVESACCESSIBLE=Y  FORMAT=Y            FORMATTING=N

                        IMPORT=Y            MAP=Y               MIRPRI=Y

                        MIRSEC=Y            MISSING=N           MOVE_RELATED=Y

                        NON_RED=Y           OFFLINE=N           READ=Y

                        READY=Y             RECONFG=Y           REP_NOT_ON_PREF=Y

                        SNAP=Y              TRANSFER=Y          WRITE=Y

                        ROLLBACK=Y          MIRACTV=Y           NEWPIT=Y

                        DVE=Y               DSS=Y               CONCAT=N

                        FRC=Y               PITGROUP=Y          ASYNC_MIRPRI=Y

                        ASYNC_MIRSEC=Y      DSP=Y               CPC=Y

                        THIN_MANEXP=Y       THIN_GROWPARAM=Y    XCPY=Y

                        4KDSS=Y             VAULT=Y             RESTORE=Y

                        VAULT_GRP=Y         ALBTRANSFER=Y       PRIORITYXFER=Y

     Activity Count:    1

     QueueDepth:        0

     EUI:               000001a05dd3fc69d039ea00000201a8

     IAF Aborted:               0

     Segment Size:              256 blocks

     Stripe Size:               2048 blocks

     Drive Recovery Mode:       RECOVERY_INVALID

     PreConfigedPI:             0

     isVolumeCopyActive:        False

     isVolumeCopyInsertPI:      False

     DSS PreAllocation:         Enabled

     AbsoluteMaxDSSSegSize:     4096

     Pre-Read Redundancy Check: Disabled

     Retry on Mismatch:         Enabled

     Repair Writes:             Disabled

     Flash Read Cache:          Disabled

     Atomic Write Support:      Disabled

     Volume Usage:              STANDARD

     Initialization:            Completed

     Offline Type:              0

     NoDemandFlush:             False

     Read Ahead:                1

     Fast Write:                1

     Sync Age:                  2500

     Cache Flush Modifier:      8

     Min Warn Flush Modifier:   8

     Cache Granularity:         64

     Ignore FUA:                Host type(s) - 0 1 2 5 6 7 8 9 10 15 17 22 23 24 25 26 27 28 29

     Media Scan Settings: Enabled with Parity Check, but no Repair.

          Period = 30 day(s). Current rate = 88542 blocks/sec.

     Background Task:    MEDIA SCAN at block 0x0211a246800 with 77% complete

     Background Delay:   0.1 seconds per 4096 blocks

*** Volume Group Info ***

    WWN        : 6d039ea0000201a8000001945dd3fc4a

    RAID Level : RAID 6

    Boundary   : 142172514304 0x211a246800

    Persisted  : 0  0x0

    VG Label   : vg0

    Address    : 0x000000024669b6b0

    Drive Count: 10

    Secure     : False

    PI Capable : True - 2

    Media Type : HDD

     Volumes on this Group:

Volume Count:                      1

0x00000 [O]

*** Pieces ***

    Offset: 0x0                    Length:     0x557f00000 (22950182912 dec)

    Count : 10                     Data Count: 8

    Piece     Devnum        Address  Tray/Slot   State

        0  0x00010100    0x00000001656ff850    00,01     PieceOptimalState

        1  0x0001000f    0x0000000165796640    99,16     PieceOptimalState

        2  0x0001011e    0x00000001656e8f58    00,31     PieceOptimalState

        3  0x0001002d    0x00000001656e8e48    99,46     PieceOptimalState

        4  0x00010131    0x0000000165786280    00,50     PieceOptimalState

        5  0x00010004    0x000000016575b9b8    99,05     PieceOptimalState

        6  0x00010113    0x0000000165747760    00,20     PieceOptimalState

        7  0x00010022    0x00000001657475f0    99,35     PieceOptimalState

        8  0x00010126    0x00000001656e0ba0    00,39     PieceOptimalState

        9  0x00010035    0x000000016571a330    99,54     PieceOptimalState

StripesPerCachePoolExtent:            16

BlocksPerCachePoolExtent :         32768

*** CACHE DEVICE 0 ***

     Sector Size     Secs/CacheBlock          Cache Blk Size    Clean Minimum

       512 bytes                  64             32768 bytes 0000208000

Dirty Cache Blks   Dirty Cache High per Pool

            5129   0:12771 1:15775 2:12761

  Hash Data Blks    Hash Parity Blks

          150394                   0

       RD_CACHE       WR_CACHE   WR_CACHE_MIR            FWT

        ENABLED        ENABLED        ENABLED       DISABLED

           CWOB  WT_BU_BATT_ON     READ_CACHE    WRITE_CACHE   WRITE_CACHE_MIR

       DISABLED             NO         ACTIVE         ACTIVE            ACTIVE

HIGH_PRIORITY_DEV: NO

Background sync is in progress : NO

CacheBlockMask: ffffffffffffffff

Cache Statistics:

                 current       old   current       old

         cache      read      read    parity    parity     disk

 VOL     reads      hits      hits      hits      hits     reads

---- --------- --------- --------- --------- --------- ---------

   0  23089135   3983591       243         0     47308  20092317

          read      read

         check     check    insert      full   partial      disk

 VOL      reqs      hits      reqs overwrite overwrite    writes

---- --------- --------- --------- --------- --------- ---------

   0   4014500   2033274    965518      3290    187790         0

                            Cache Pool  0       Cache Pool  1       Cache Pool  2  

                         ------------------- ------------------- -------------------

       WB blks   % Dirty   WB blks   % Dirty   WB blks   % Dirty   WB blks   % Dirty

 VOL   current     Limit      high     Limit      high     Limit      high     Limit

---- --------- --------- --------- --------- --------- --------- --------- ---------

   0      5129         2     12771         6     15775         7     12761         6

     +---------- cache blocks evicted after ---------+   evicted    blocks

 VOL     fetch       hit  prefetch pftch hit     write     total    in-use

---- --------- --------- --------- --------- --------- --------- ---------

   0  18137518     70172  23524605 29676370  11431433  82840098    150394

                prefetch            prefetch  pct

 VOL          hit blocks         miss blocks hits

---- ------------------- ------------------- ----

   0          1896349824          1505574720   55

value = 139928058009584 = 0x7f438a3117f0

-> vdmShowVGInfo

Total Volume Groups: ............. 12

Seq:1 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg0 / VGWwn:6d039ea0000201a8000001945dd3fc4a / RsvdSpace:            5 (GB)

(Active) Drive:0x000000024661e240 devnum:0x00010100 seqNum:1 Tray/Slot:00/01  State:Acc/GrA/Opt

(Active) Drive:0x000000024661e530 devnum:0x0001000f seqNum:2 Tray/Slot:99/16  State:Acc/GrA/Opt

(Active) Drive:0x000000024661e820 devnum:0x0001011e seqNum:3 Tray/Slot:00/31  State:Acc/GrA/Opt

(Active) Drive:0x000000024661eb10 devnum:0x0001002d seqNum:4 Tray/Slot:99/46  State:Acc/GrA/Opt

(Active) Drive:0x00000002466352e0 devnum:0x00010131 seqNum:5 Tray/Slot:00/50  State:Acc/GrA/Opt

(Active) Drive:0x000000024661f0f0 devnum:0x00010004 seqNum:6 Tray/Slot:99/05  State:Acc/GrA/Opt

(Active) Drive:0x000000024661ee00 devnum:0x00010113 seqNum:7 Tray/Slot:00/20  State:Acc/GrA/Opt

(Active) Drive:0x000000024661f6d0 devnum:0x00010022 seqNum:8 Tray/Slot:99/35  State:Acc/GrA/Opt

(Active) Drive:0x000000024661f9c0 devnum:0x00010126 seqNum:9 Tray/Slot:00/39  State:Acc/GrA/Opt

(Active) Drive:0x000000024661fcb0 devnum:0x00010035 seqNum:10 Tray/Slot:99/54  State:Acc/GrA/Opt

Volumes: 0x00000 [O]

Seq:2 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg1 / VGWwn:6d039ea0000201a8000001955dd3fc4c / RsvdSpace:            5 (GB)

(Active) Drive:0x000000024661ffa0 devnum:0x00010103 seqNum:1 Tray/Slot:00/04  State:Acc/GrA/Opt

(Active) Drive:0x0000000246620290 devnum:0x00010012 seqNum:2 Tray/Slot:99/19  State:Acc/GrA/Opt

(Active) Drive:0x0000000246620580 devnum:0x00010121 seqNum:3 Tray/Slot:00/34  State:Acc/GrA/Opt

(Active) Drive:0x0000000246a3f2d0 devnum:0x00010025 seqNum:4 Tray/Slot:99/38  State:Acc/GrA/Opt

(Active) Drive:0x0000000246620b60 devnum:0x00010134 seqNum:5 Tray/Slot:00/53  State:Acc/GrA/Opt

(Active) Drive:0x000000024683dc48 devnum:0x00010007 seqNum:6 Tray/Slot:99/08  State:Acc/GrA/Opt

(Active) Drive:0x00000002496193c8 devnum:0x00010116 seqNum:7 Tray/Slot:00/23  State:Acc/GrA/Opt

(Active) Drive:0x0000000246621430 devnum:0x0001001a seqNum:8 Tray/Slot:99/27  State:Acc/GrA/Opt

(Active) Drive:0x0000000246621720 devnum:0x00010129 seqNum:9 Tray/Slot:00/42  State:Acc/GrA/Opt

(Active) Drive:0x0000000246621a10 devnum:0x00010038 seqNum:10 Tray/Slot:99/57  State:Acc/GrA/Opt

Volumes: 0x00001 [D]

Seq:3 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:3 / PI Capable:T - 2 / Label:vg2 / VGWwn:6d039ea0000201a8000001965dd3fc4d / RsvdSpace:            5 (GB)

(Active) Drive:0x0000000246621d00 devnum:0x00010106 seqNum:1 Tray/Slot:00/07  State:Acc/GrA/Opt

(Active) Drive:0x0000000246621ff0 devnum:0x00010015 seqNum:2 Tray/Slot:99/22  State:Acc/GrA/Opt

(Active) Drive:0x00000002466222e0 devnum:0x00010119 seqNum:3 Tray/Slot:00/26  State:Acc/GrA/Opt

(Active) Drive:0x00000002466225d0 devnum:0x00010028 seqNum:4 Tray/Slot:99/41  State:Acc/GrA/Opt

(Active) Drive:0x00000002466228c0 devnum:0x00010137 seqNum:5 Tray/Slot:00/56  State:Acc/GrA/Opt

(Active) Drive:0x000000024661f3e0 devnum:0x0001000a seqNum:6 Tray/Slot:99/11  State:Acc/GrA/Opt

(Active) Drive:0x0000000246631820 devnum:0x0001010e seqNum:7 Tray/Slot:00/15  State:Acc/GrA/Opt

(Active) Drive:0x0000000246623190 devnum:0x0001001d seqNum:8 Tray/Slot:99/30  State:Acc/GrA/Opt

(Active) Drive:0x0000000246622ea0 devnum:0x0001012c seqNum:9 Tray/Slot:00/45  State:Acc/GrA/Opt

(Active) Drive:0x0000000246623770 devnum:0x0001003b seqNum:10 Tray/Slot:99/60  State:Acc/GrA/Opt

Volumes: 0x00002 [O]

Seq:4 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:3 / PI Capable:T - 2 / Label:vg3 / VGWwn:6d039ea0000201a8000001975dd3fc4f / RsvdSpace:            5 (GB)

(Active) Drive:0x0000000246623a60 devnum:0x00010109 seqNum:1 Tray/Slot:00/10  State:Acc/GrA/Opt

(Active) Drive:0x00000002466280e0 devnum:0x0001000d seqNum:2 Tray/Slot:99/14  State:Acc/GrA/Opt

(Active) Drive:0x0000000246624040 devnum:0x0001011c seqNum:3 Tray/Slot:00/29  State:Acc/GrA/Opt

(Active) Drive:0x0000000246624330 devnum:0x0001002b seqNum:4 Tray/Slot:99/44  State:Acc/GrA/Opt

(Active) Drive:0x0000000246624620 devnum:0x0001013a seqNum:5 Tray/Slot:00/59  State:Acc/GrA/Opt

(Active) Drive:0x0000000246624910 devnum:0x00010002 seqNum:6 Tray/Slot:99/03  State:Acc/GrA/Opt

(Active) Drive:0x0000000246624c00 devnum:0x00010111 seqNum:7 Tray/Slot:00/18  State:Acc/GrA/Opt

(Active) Drive:0x0000000246624ef0 devnum:0x00010020 seqNum:8 Tray/Slot:99/33  State:Acc/GrA/Opt

(Active) Drive:0x00000002466251e0 devnum:0x0001012f seqNum:9 Tray/Slot:00/48  State:Acc/GrA/Opt

(Active) Drive:0x00000002466254d0 devnum:0x00010030 seqNum:10 Tray/Slot:99/49  State:Acc/GrA/Opt

Volumes: 0x00003 [O]

Seq:5 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg4 / VGWwn:6d039ea0000201a8000001985dd3fc50 / RsvdSpace:            5 (GB)

(Active) Drive:0x00000002466257c0 devnum:0x00010101 seqNum:1 Tray/Slot:00/02  State:Acc/GrA/Opt

(Active) Drive:0x0000000246625ab0 devnum:0x00010010 seqNum:2 Tray/Slot:99/17  State:Acc/GrA/Opt

(Active) Drive:0x0000000246625da0 devnum:0x0001011f seqNum:3 Tray/Slot:00/32  State:Acc/GrA/Opt

(Active) Drive:0x0000000246626090 devnum:0x0001002e seqNum:4 Tray/Slot:99/47  State:Acc/GrA/Opt

(Active) Drive:0x0000000246626380 devnum:0x00010132 seqNum:5 Tray/Slot:00/51  State:Acc/GrA/Opt

(Active) Drive:0x0000000246626670 devnum:0x00010005 seqNum:6 Tray/Slot:99/06  State:Acc/GrA/Opt

(Active) Drive:0x0000000246626960 devnum:0x00010114 seqNum:7 Tray/Slot:00/21  State:Acc/GrA/Opt

(Active) Drive:0x0000000246626c50 devnum:0x00010023 seqNum:8 Tray/Slot:99/36  State:Acc/GrA/Opt

(Active) Drive:0x0000000246626f40 devnum:0x00010124 seqNum:9 Tray/Slot:00/37  State:Acc/GrA/Opt

(Active) Drive:0x000000024663a230 devnum:0x00010033 seqNum:10 Tray/Slot:99/52  State:Acc/GrA/Opt

Volumes: 0x00004 [O]

Seq:6 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg5 / VGWwn:6d039ea0000201a8000001995dd3fc52 / RsvdSpace:            5 (GB)

(Active) Drive:0x0000000246627520 devnum:0x00010104 seqNum:1 Tray/Slot:00/05  State:Acc/GrA/Opt

(Active) Drive:0x0000000246627810 devnum:0x00010013 seqNum:2 Tray/Slot:99/20  State:Acc/GrA/Opt

(Active) Drive:0x0000000246627b00 devnum:0x00010122 seqNum:3 Tray/Slot:00/35  State:Acc/GrA/Opt

(Active) Drive:0x0000000246627df0 devnum:0x00010026 seqNum:4 Tray/Slot:99/39  State:Acc/GrA/Opt

(Active) Drive:0x0000000246632cb0 devnum:0x00010135 seqNum:5 Tray/Slot:00/54  State:Acc/GrA/Opt

(Active) Drive:0x00000002466283d0 devnum:0x00010008 seqNum:6 Tray/Slot:99/09  State:Acc/GrA/Opt

(Active) Drive:0x00000002466286c0 devnum:0x00010117 seqNum:7 Tray/Slot:00/24  State:Acc/GrA/Opt

(Active) Drive:0x00000002466289b0 devnum:0x00010018 seqNum:8 Tray/Slot:99/25  State:Acc/GrA/Opt

(Active) Drive:0x0000000246628ca0 devnum:0x00010127 seqNum:9 Tray/Slot:00/40  State:Acc/GrA/Opt

(Active) Drive:0x0000000246628f90 devnum:0x00010036 seqNum:10 Tray/Slot:99/55  State:Acc/GrA/Opt

Volumes: 0x00005 [O]

Seq:7 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg6 / VGWwn:6d039ea0000201a80000019a5dd3fc53 / RsvdSpace:            5 (GB)

(Active) Drive:0x000000024662f4e0 devnum:0x00010107 seqNum:1 Tray/Slot:00/08  State:Acc/GrA/Opt

(Active) Drive:0x000000024662f7d0 devnum:0x00010016 seqNum:2 Tray/Slot:99/23  State:Acc/GrA/Opt

(Active) Drive:0x000000024662fac0 devnum:0x0001011a seqNum:3 Tray/Slot:00/27  State:Acc/GrA/Opt

(Active) Drive:0x000000024662fdb0 devnum:0x00010029 seqNum:4 Tray/Slot:99/42  State:Acc/GrA/Opt

(Active) Drive:0x00000002466300a0 devnum:0x00010138 seqNum:5 Tray/Slot:00/57  State:Acc/GrA/Opt

(Active) Drive:0x0000000246630390 devnum:0x0001000b seqNum:6 Tray/Slot:99/12  State:Acc/GrA/Opt

(Active) Drive:0x0000000246630680 devnum:0x0001010c seqNum:7 Tray/Slot:00/13  State:Acc/GrA/Opt

(Active) Drive:0x0000000246630970 devnum:0x0001001b seqNum:8 Tray/Slot:99/28  State:Acc/GrA/Opt

(Active) Drive:0x0000000246630c60 devnum:0x0001012a seqNum:9 Tray/Slot:00/43  State:Acc/GrA/Opt

(Active) Drive:0x0000000246630f50 devnum:0x00010039 seqNum:10 Tray/Slot:99/58  State:Acc/GrA/Opt

Volumes: 0x00006 [O]

Seq:8 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg7 / VGWwn:6d039ea0000201a80000019b5dd3fc55 / RsvdSpace:            5 (GB)

(Active) Drive:0x0000000246631240 devnum:0x0001010a seqNum:1 Tray/Slot:00/11  State:Acc/GrA/Opt

(Active) Drive:0x0000000246631530 devnum:0x0001000e seqNum:2 Tray/Slot:99/15  State:Acc/GrA/Opt

(Active) Drive:0x0000000246639380 devnum:0x0001011d seqNum:3 Tray/Slot:00/30  State:Acc/GrA/Opt

(Active) Drive:0x00000002492e4468 devnum:0x0001002c seqNum:4 Tray/Slot:99/45  State:Acc/GrA/Opt

(Active) Drive:0x0000000246631e00 devnum:0x0001013b seqNum:5 Tray/Slot:00/60  State:Acc/GrA/Opt

(Active) Drive:0x00000002466320f0 devnum:0x00010000 seqNum:6 Tray/Slot:99/01  State:Acc/GrA/Opt

(Active) Drive:0x00000002466323e0 devnum:0x0001010f seqNum:7 Tray/Slot:00/16  State:Acc/GrA/Opt

(Active) Drive:0x00000002466326d0 devnum:0x0001001e seqNum:8 Tray/Slot:99/31  State:Acc/GrA/Opt

(Active) Drive:0x00000002466329c0 devnum:0x0001012d seqNum:9 Tray/Slot:00/46  State:Acc/GrA/Opt

(Active) Drive:0x0000000246639960 devnum:0x00010031 seqNum:10 Tray/Slot:99/50  State:Acc/GrA/Opt

Volumes: 0x00007 [O]

Seq:9 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg8 / VGWwn:6d039ea0000201a80000019c5dd3fc56 / RsvdSpace:            5 (GB)

(Active) Drive:0x0000000246632fa0 devnum:0x00010102 seqNum:1 Tray/Slot:00/03  State:Acc/GrA/Opt

(Active) Drive:0x0000000246633290 devnum:0x00010011 seqNum:2 Tray/Slot:99/18  State:Acc/GrA/Opt

(Active) Drive:0x0000000246633580 devnum:0x00010120 seqNum:3 Tray/Slot:00/33  State:Acc/GrA/Opt

(Active) Drive:0x0000000246633870 devnum:0x0001002f seqNum:4 Tray/Slot:99/48  State:Acc/GrA/Opt

(Active) Drive:0x0000000246633b60 devnum:0x00010130 seqNum:5 Tray/Slot:00/49  State:Acc/GrA/Opt

(Active) Drive:0x0000000246633e50 devnum:0x00010003 seqNum:6 Tray/Slot:99/04  State:Acc/GrA/Opt

(Active) Drive:0x0000000246634140 devnum:0x00010112 seqNum:7 Tray/Slot:00/19  State:Acc/GrA/Opt

(Active) Drive:0x0000000246634430 devnum:0x00010021 seqNum:8 Tray/Slot:99/34  State:Acc/GrA/Opt

(Active) Drive:0x0000000246634720 devnum:0x00010125 seqNum:9 Tray/Slot:00/38  State:Acc/GrA/Opt

(Active) Drive:0x0000000246634a10 devnum:0x00010034 seqNum:10 Tray/Slot:99/53  State:Acc/GrA/Opt

Volumes: 0x00008 [O]

Seq:10 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg9 / VGWwn:6d039ea0000201a80000019d5dd3fc57 / RsvdSpace:            5 (GB)

(Active) Drive:0x0000000246634d00 devnum:0x00010105 seqNum:1 Tray/Slot:00/06  State:Acc/GrA/Opt

(Active) Drive:0x0000000246627230 devnum:0x00010014 seqNum:2 Tray/Slot:99/21  State:Acc/GrA/Opt

(Active) Drive:0x0000000246623d50 devnum:0x00010123 seqNum:3 Tray/Slot:00/36  State:Acc/GrA/Opt

(Active) Drive:0x00000002466355d0 devnum:0x00010024 seqNum:4 Tray/Slot:99/37  State:Acc/GrA/Opt

(Active) Drive:0x00000002466358c0 devnum:0x00010133 seqNum:5 Tray/Slot:00/52  State:Acc/GrA/Opt

(Active) Drive:0x0000000246635bb0 devnum:0x00010006 seqNum:6 Tray/Slot:99/07  State:Acc/GrA/Opt

(Active) Drive:0x0000000246635ea0 devnum:0x00010115 seqNum:7 Tray/Slot:00/22  State:Acc/GrA/Opt

(Active) Drive:0x0000000246636190 devnum:0x00010019 seqNum:8 Tray/Slot:99/26  State:Acc/GrA/Opt

(Active) Drive:0x0000000246636480 devnum:0x00010128 seqNum:9 Tray/Slot:00/41  State:Acc/GrA/Opt

(Active) Drive:0x0000000246636770 devnum:0x00010037 seqNum:10 Tray/Slot:99/56  State:Acc/GrA/Opt

Volumes: 0x00009 [O]

Seq:11 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:2 / PI Capable:T - 2 / Label:vg10 / VGWwn:6d039ea0000201a80000019e5dd3fc59 / RsvdSpace:            5 (GB)

(Active) Drive:0x0000000246636a60 devnum:0x00010108 seqNum:1 Tray/Slot:00/09  State:Acc/GrA/Opt

(Active) Drive:0x0000000246636d50 devnum:0x00010017 seqNum:2 Tray/Slot:99/24  State:Acc/GrA/Opt

(Active) Drive:0x0000000246637040 devnum:0x00010118 seqNum:3 Tray/Slot:00/25  State:Acc/GrA/Opt

(Active) Drive:0x0000000246637330 devnum:0x00010027 seqNum:4 Tray/Slot:99/40  State:Acc/GrA/Opt

(Active) Drive:0x0000000246637620 devnum:0x00010136 seqNum:5 Tray/Slot:00/55  State:Acc/GrA/Opt

(Active) Drive:0x0000000246637910 devnum:0x00010009 seqNum:6 Tray/Slot:99/10  State:Acc/GrA/Opt

(Active) Drive:0x0000000246637c00 devnum:0x0001010d seqNum:7 Tray/Slot:00/14  State:Acc/GrA/Opt

(Active) Drive:0x0000000246637ef0 devnum:0x0001001c seqNum:8 Tray/Slot:99/29  State:Acc/GrA/Opt

(Active) Drive:0x00000002466381e0 devnum:0x0001012b seqNum:9 Tray/Slot:00/44  State:Acc/GrA/Opt

(Active) Drive:0x00000002466384d0 devnum:0x0001003a seqNum:10 Tray/Slot:99/59  State:Acc/GrA/Opt

Volumes: 0x0000a [O]

Seq:12 / RAID 6 / VGCompleteState / TLP:F / DLP:T / SSM:T / ActDrv:10 / InActDrv:0 / VolCnt:1 / Secure:No

BlockSize:512 / LPE:3 / DBF:3 / PI Capable:T - 2 / Label:vg11 / VGWwn:6d039ea0000201a80000019f5dd3fc5a / RsvdSpace:            5 (GB)

(Active) Drive:0x0000000246622bb0 devnum:0x0001010b seqNum:1 Tray/Slot:00/12  State:Acc/GrA/Opt

(Active) Drive:0x00000002466387c0 devnum:0x0001000c seqNum:2 Tray/Slot:99/13  State:Acc/GrA/Opt

(Active) Drive:0x0000000246638ab0 devnum:0x0001011b seqNum:3 Tray/Slot:00/28  State:Acc/GrA/Opt

(Active) Drive:0x0000000246638da0 devnum:0x0001002a seqNum:4 Tray/Slot:99/43  State:Acc/GrA/Opt

(Active) Drive:0x0000000246639090 devnum:0x00010139 seqNum:5 Tray/Slot:00/58  State:Acc/GrA/Opt

(Active) Drive:0x0000000246634ff0 devnum:0x00010001 seqNum:6 Tray/Slot:99/02  State:Acc/GrA/Opt

(Active) Drive:0x0000000246639670 devnum:0x00010110 seqNum:7 Tray/Slot:00/17  State:Acc/GrA/Opt

(Active) Drive:0x0000000246623480 devnum:0x0001001f seqNum:8 Tray/Slot:99/32  State:Acc/GrA/Opt

(Active) Drive:0x0000000246639c50 devnum:0x0001012e seqNum:9 Tray/Slot:00/47  State:Acc/GrA/Opt

(Active) Drive:0x0000000246639f40 devnum:0x00010032 seqNum:10 Tray/Slot:99/51  State:Acc/GrA/Opt

Volumes: 0x0000b [O]

value = 139927976262160 = 0x7f438551ba10

[ahmadm]

It appears that drive 99,4,1 is the one with the remaining logical piece failure. If you can show me the output of "evfShowVol 1" command, I can confirm 100%.

If that is the case, the safest solution would be to fail the drive then reconstruct it in the UI.

[jtejada]

Here the output. Yes, drive 99,4,1 was the drive replaced due to Impending failure status. How to fail the drive? Directly extract it? Or maybe directly making reconstruct in the UI should solve it.

-> evfShowVol 1

Volume 0x1(RAIDVolume)

 6 VolumeListener(s)

     0xc001(MediaScanAgent)

     N3vdm11VolumeGroupE

     N3vdm17RAIDVolumeManagerE

     N3spm18SymbolEventManagerE

     N3evf10CmdHandlerE

     N3evf22VolumeUserLabelManagerE

 0 Children

Volume 0x1 Attributes:

     Volume Type:       8+2 RAID 6

     User Label:        d1

     WWN:               6d039ea0000201a8000001a35dd3fc77

     Address:           0x00000001660b4850

     Devnum:            0x10000001

     Capacity:          183601463296 blocks

     BlockSize:         512 bytes

     LargeIoSize:       4096 blocks

     MaxBkgdIoSize:     4096 blocks

     Has Extents:       false

     Ownership:         This controller

     PreferredPath:     This controller

     Transferring:      false

     StopIOInProgress:  false

     Suspended:         0

     Quiesce Count:     0

     Quiesce Both Ctl:  0 (Local: 0/Alt: 0)

     State:             RV_DEGRADED

     Unreadable Sectors:Not Present

     App Tag:           0x0000

     App Tag Owned:     false

     Protection Type:   1

     PI Reporting:      enabled

     PI Mirroring:      enabled

     isSecure:          false

     isSecureCapable:   false

     isFIPS:            false

     isExopActive:      true

     isPersistent:      true

     isTCGSSupported:   true

     skipped URS:       0

     NonCriticalTransferAllowed: true

     Permissions:       CONFIG=Y            CPYsrc=Y            CPYTGT=Y

                        DRIVESACCESSIBLE=Y  FORMAT=Y            FORMATTING=N

                        IMPORT=Y            MAP=Y               MIRPRI=Y

                        MIRSEC=Y            MISSING=N           MOVE_RELATED=Y

                        NON_RED=Y           OFFLINE=N           READ=Y

                        READY=Y             RECONFG=Y           REP_NOT_ON_PREF=Y

                        SNAP=Y              TRANSFER=Y          WRITE=Y

                        ROLLBACK=Y          MIRACTV=Y           NEWPIT=Y

                        DVE=Y               DSS=Y               CONCAT=N

                        FRC=Y               PITGROUP=Y          ASYNC_MIRPRI=Y

                        ASYNC_MIRSEC=Y      DSP=Y               CPC=Y

                        THIN_MANEXP=Y       THIN_GROWPARAM=Y    XCPY=Y

                        4KDSS=Y             VAULT=Y             RESTORE=Y

                        VAULT_GRP=Y         ALBTRANSFER=Y       PRIORITYXFER=Y

     Activity Count:    1

     QueueDepth:        0

     EUI:               000001a35dd3fc77d039ea00000201a8

     IAF Aborted:               0

     Segment Size:              256 blocks

     Stripe Size:               2048 blocks

     Drive Recovery Mode:       RECOVERY_INVALID

     PreConfigedPI:             0

     isVolumeCopyActive:        False

     isVolumeCopyInsertPI:      False

     DSS PreAllocation:         Enabled

     AbsoluteMaxDSSSegSize:     4096

     Pre-Read Redundancy Check: Disabled

     Retry on Mismatch:         Enabled

     Repair Writes:             Disabled

     Flash Read Cache:          Disabled

     Atomic Write Support:      Disabled

     Volume Usage:              STANDARD

     Initialization:            Completed

     Offline Type:              0

     NoDemandFlush:             False

     Read Ahead:                1

     Fast Write:                1

     Sync Age:                  2500

     Cache Flush Modifier:      8

     Min Warn Flush Modifier:   8

     Cache Granularity:         64

     Ignore FUA:                Host type(s) - 0 1 2 5 6 7 8 9 10 15 17 22 23 24 25 26 27 28 29

     Media Scan Settings: Enabled with Parity Check, but no Repair.

          Period = 30 day(s). Current rate = 88542 blocks/sec.

     Background Task:    MEDIA SCAN at block 0x027d7ded800 with 93% complete

     Background Delay:   0.1 seconds per 4096 blocks

*** Volume Group Info ***

    WWN        : 6d039ea0000201a8000001955dd3fc4c

    RAID Level : RAID 6

    Boundary   : 171125430272 0x27d7ded800

    Persisted  : 0  0x0

    VG Label   : vg1

    Address    : 0x000000024669de00

    Drive Count: 10

    Secure     : False

    PI Capable : True - 2

    Media Type : HDD

     Volumes on this Group:

Volume Count:                      1

0x00001 [D]

*** Pieces ***

    Offset: 0x0                    Length:     0x557f00000 (22950182912 dec)

    Count : 10                     Data Count: 8

    Piece     Devnum        Address  Tray/Slot   State

        0  0x00010103    0x000000016575fa40    00,04     PieceOptimalState

        1  0x00010012    0x0000000165732d20    99,19     PieceOptimalState

        2  0x00010121    0x0000000165732c10    00,34     PieceOptimalState

        3  0x00010025    0x00000001656cc3c0    99,38     FAILED

        4  0x00010134    0x00000001656f5408    00,53     PieceOptimalState

        5  0x00010007    0x00000001656f52f8    99,08     PieceOptimalState

        6  0x00010116    0x000000016576bca0    00,23     PieceOptimalState

        7  0x0001001a    0x00000001657a2978    99,27     PieceOptimalState

        8  0x00010129    0x000000016579e910    00,42     PieceOptimalState

        9  0x00010038    0x00000001656ff960    99,57     PieceOptimalState

StripesPerCachePoolExtent:            16

BlocksPerCachePoolExtent :         32768

*** CACHE DEVICE 1 ***

     Sector Size     Secs/CacheBlock          Cache Blk Size    Clean Minimum

       512 bytes                  64             32768 bytes 0000208000

Dirty Cache Blks   Dirty Cache High per Pool

            3896   0:18924 1:16870 2:15379

  Hash Data Blks    Hash Parity Blks

          152029                   0

       RD_CACHE       WR_CACHE   WR_CACHE_MIR            FWT

        ENABLED        ENABLED        ENABLED       DISABLED

           CWOB  WT_BU_BATT_ON     READ_CACHE    WRITE_CACHE   WRITE_CACHE_MIR

       DISABLED             NO         ACTIVE         ACTIVE            ACTIVE

HIGH_PRIORITY_DEV: NO

Background sync is in progress : NO

CacheBlockMask: ffffffffffffffff

Cache Statistics:

                 current       old   current       old

         cache      read      read    parity    parity     disk

 VOL     reads      hits      hits      hits      hits     reads

---- --------- --------- --------- --------- --------- ---------

   1   8813418   3365511    945332         0     88559   5705963

          read      read

         check     check    insert      full   partial      disk

 VOL      reqs      hits      reqs overwrite overwrite    writes

---- --------- --------- --------- --------- --------- ---------

   1   2462860    979939    361100      7364    114205         0

                            Cache Pool  0       Cache Pool  1       Cache Pool  2  

                         ------------------- ------------------- -------------------

       WB blks   % Dirty   WB blks   % Dirty   WB blks   % Dirty   WB blks   % Dirty

 VOL   current     Limit      high     Limit      high     Limit      high     Limit

---- --------- --------- --------- --------- --------- --------- --------- ---------

   1      3896         1     18924         9     16870         8     15379         7

     +---------- cache blocks evicted after ---------+   evicted    blocks

 VOL     fetch       hit  prefetch pftch hit     write     total    in-use

---- --------- --------- --------- --------- --------- --------- ---------

   1   2839391   2105770   1010947  5949256   4849966  16755330    151999

                prefetch            prefetch  pct

 VOL          hit blocks         miss blocks hits

---- ------------------- ------------------- ----

   1           106899520            64700608   62

value = 139928058009584 = 0x7f438a3117f0

[ahmadm · Accepted solution]

Yes, you would essentially reconstruct the drive. To do that, you can run this command in the controller shell:

> startDriveReconstruction_MT 0x10025