Solved: Not all disks available after zero and initialisation

PooBear85 · ‎2019-08-02

I need some help with a filer, it is running 9.4 and has 24x1.2TB drives.

It was already deployed but has to be wiped and redeployed elsewhere.

Ive console onto both controllers at the same time, rebooted each node, broken the boot process and selected option 4.

The process completes and asks for the management interface to be configured, which I’ve done for both. Completed the cluster configuration. However when I try to create an aggregate I can only assign 7.5TB, there are no other spare drives.

I cam see all 24 drives on the filer, can someone tell me what I’ve missed or how I can resolve this please?

system node run -node Filer01-01 -command sysconfig -r

Aggregate Filer01_01_root (online, raid_dp) (block checksums)

Plex /Filer01_01_root/plex0 (online, normal, active, pool0)

RAID group /Filer01_01_root/plex0/rg0 (normal, block checksums)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

dparity 0a.00.1P2 0a 0 1 SA:B 0 SAS 10000 55176/113000448 55184/113016832

parity 0a.00.3P2 0a 0 3 SA:B 0 SAS 10000 55176/113000448 55184/113016832

data 0a.00.5P2 0a 0 5 SA:B 0 SAS 10000 55176/113000448 55184/113016832

data 0a.00.7P2 0a 0 7 SA:B 0 SAS 10000 55176/113000448 55184/113016832

data 0a.00.9P2 0a 0 9 SA:B 0 SAS 10000 55176/113000448 55184/113016832

data 0a.00.11P2 0a 0 11 SA:B 0 SAS 10000 55176/113000448 55184/113016832

data 0a.00.13P2 0a 0 13 SA:B 0 SAS 10000 55176/113000448 55184/113016832

data 0a.00.15P2 0a 0 15 SA:B 0 SAS 10000 55176/113000448 55184/113016832

data 0a.00.17P2 0a 0 17 SA:B 0 SAS 10000 55176/113000448 55184/113016832

data 0a.00.19P2 0a 0 19 SA:B 0 SAS 10000 55176/113000448 55184/113016832

Pool1 spare disks (empty)

Pool0 spare disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

Spare disks for block checksum

spare 0a.00.21P2 0a 0 21 SA:B 0 SAS 10000 55176/113000448 55184/113016832

spare 0a.00.23P2 0a 0 23 SA:B 0 SAS 10000 55176/113000448 55184/113016832

spare 0a.00.1P1 0a 0 1 SA:B 0 SAS 10000 1087139/2226462208 1087147/2226478592 (not zeroed)

spare 0a.00.3P1 0a 0 3 SA:B 0 SAS 10000 1087139/2226462208 1087147/2226478592 (not zeroed)

spare 0a.00.5P1 0a 0 5 SA:B 0 SAS 10000 1087139/2226462208 1087147/2226478592 (not zeroed)

spare 0a.00.7P1 0a 0 7 SA:B 0 SAS 10000 1087139/2226462208 1087147/2226478592 (not zeroed)

spare 0a.00.9P1 0a 0 9 SA:B 0 SAS 10000 1087139/2226462208 1087147/2226478592 (not zeroed)

spare 0a.00.11P1 0a 0 11 SA:B 0 SAS 10000 1087139/2226462208 1087147/2226478592 (not zeroed)

spare 0a.00.13P1 0a 0 13 SA:B 0 SAS 10000 1087139/2226462208 1087147/2226478592 (not zeroed)

spare 0a.00.15P1 0a 0 15 SA:B 0 SAS 10000 1087139/2226462208 1087147/2226478592 (not zeroed)

spare 0a.00.17P1 0a 0 17 SA:B 0 SAS 10000 1087139/2226462208 1087147/2226478592 (not zeroed)

spare 0a.00.19P1 0a 0 19 SA:B 0 SAS 10000 1087139/2226462208 1087147/2226478592 (not zeroed)

spare 0a.00.21P1 0a 0 21 SA:B 0 SAS 10000 1087139/2226462208 1087147/2226478592

spare 0a.00.23P1 0a 0 23 SA:B 0 SAS 10000 1087139/2226462208 1087147/2226478592

Partner disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

partner 0b.00.4P2 0b 0 4 SA:A 0 SAS 10000 0/0 55184/113016832

partner 0b.00.2P2 0b 0 2 SA:A 0 SAS 10000 0/0 55184/113016832

partner 0b.00.20P2 0b 0 20 SA:A 0 SAS 10000 0/0 55184/113016832

partner 0b.00.10P2 0b 0 10 SA:A 0 SAS 10000 0/0 55184/113016832

partner 0b.00.20P1 0b 0 20 SA:A 0 SAS 10000 0/0 1087147/2226478592

partner 0b.00.16P2 0b 0 16 SA:A 0 SAS 10000 0/0 55184/113016832

partner 0b.00.0P2 0b 0 0 SA:A 0 SAS 10000 0/0 55184/113016832

partner 0b.00.18P2 0b 0 18 SA:A 0 SAS 10000 0/0 55184/113016832

partner 0b.00.14P2 0b 0 14 SA:A 0 SAS 10000 0/0 55184/113016832

partner 0b.00.12P2 0b 0 12 SA:A 0 SAS 10000 0/0 55184/113016832

partner 0b.00.8P2 0b 0 8 SA:A 0 SAS 10000 0/0 55184/113016832

partner 0b.00.22P1 0b 0 22 SA:A 0 SAS 10000 0/0 1087147/2226478592

partner 0b.00.22P2 0b 0 22 SA:A 0 SAS 10000 0/0 55184/113016832

partner 0b.00.6P2 0b 0 6 SA:A 0 SAS 10000 0/0 55184/113016832

TMACMD · ‎2019-08-02

Do this which will completely wipe all data from the system

1. Boot both controllers to maintenance mode

2. Use the disk commands to work until all disks are unowned

3. Halt both controllers

4. set-defaults

5. saveenv

6. Boot both controllers to the boot menu with > boot_ontap menu

7. At the boot menu choose option 9 on both controllers and wait for prompts on both.

8. Choose 9a on one controller and wait for it to finish.

9. Choose option 9a on the other controller and wait for it to finish. You should see 24 drives now listed as unpartitioned and unowned.

10 if you want to use advanced disk partitioning then choose option 9b on one controller or if you want to use whole disks (I would not with only 24) choose option 9bc. In either case wait for the controller to reboot and start zeroing. You will see a few ...... on the screen.

11. After the first controller starts the zeroing process choose the same option on the second controller.

12. When both are done you should have access to all disks.

Note that ONTAP 9.4 introduced fast zeroing. This means it only takes a few seconds to zero the disks in any case..

You will need to reaquire the license keys.

You should be able to now configure ONTAP

View solution in original post

SpindleNinja · ‎2019-08-02

what's the output of:

"storage disk show -partition-ownership "

and sysconfig without the -r

system node run -node * -command sysconfig

PooBear85 · ‎2019-08-02

storage disk show -partition-ownership

Disk Partition Home Owner Home ID Owner ID

-------- --------- ----------------- ----------------- ----------- -----------

Info: This cluster has partitioned disks. To get a complete list of spare disk capacity use "storage aggregate

show-spare-disks".

1.0.0 Container Filer01-02 Filer01-02 537127838 537127838

Root Filer01-02 Filer01-02 537127838 537127838

Data Filer01-02 537127838 537127495

1.0.1 Container Filer01-01 Filer01-01 537127495 537127495

Root Filer01-01 Filer01-01 537127495 537127495

Data Filer01-01 Filer01-01 537127495 537127495

1.0.2 Container Filer01-02 Filer01-02 537127838 537127838

Root Filer01-02 Filer01-02 537127838 537127838

Data Filer01-02 537127838 537127495

1.0.3 Container Filer01-01 Filer01-01 537127495 537127495

Root Filer01-01 Filer01-01 537127495 537127495

Data Filer01-01 Filer01-01 537127495 537127495

1.0.4 Container Filer01-02 Filer01-02 537127838 537127838

Root Filer01-02 Filer01-02 537127838 537127838

Data Filer01-02 537127838 537127495

1.0.5 Container Filer01-01 Filer01-01 537127495 537127495

Root Filer01-01 Filer01-01 537127495 537127495

Data Filer01-01 Filer01-01 537127495 537127495

1.0.6 Container Filer01-02 Filer01-02 537127838 537127838

Root Filer01-02 Filer01-02 537127838 537127838

Data Filer01-02 537127838 537127495

1.0.7 Container Filer01-01 Filer01-01 537127495 537127495

Root Filer01-01 Filer01-01 537127495 537127495

Data Filer01-01 Filer01-01 537127495 537127495

Disk Partition Home Owner Home ID Owner ID

-------- --------- ----------------- ----------------- ----------- -----------

1.0.8 Container Filer01-02 Filer01-02 537127838 537127838

Root Filer01-02 Filer01-02 537127838 537127838

Data Filer01-02 537127838 537127495

1.0.9 Container Filer01-01 Filer01-01 537127495 537127495

Root Filer01-01 Filer01-01 537127495 537127495

Data Filer01-01 Filer01-01 537127495 537127495

1.0.10 Container Filer01-02 Filer01-02 537127838 537127838

Root Filer01-02 Filer01-02 537127838 537127838

Data Filer01-02 537127838 537127495

1.0.11 Container Filer01-01 Filer01-01 537127495 537127495

Root Filer01-01 Filer01-01 537127495 537127495

Data Filer01-01 Filer01-01 537127495 537127495

1.0.12 Container Filer01-02 Filer01-02 537127838 537127838

Root Filer01-02 Filer01-02 537127838 537127838

Data Filer01-02 537127838 537127495

1.0.13 Container Filer01-01 Filer01-01 537127495 537127495

Root Filer01-01 Filer01-01 537127495 537127495

Data Filer01-01 Filer01-01 537127495 537127495

1.0.14 Container Filer01-02 Filer01-02 537127838 537127838

Root Filer01-02 Filer01-02 537127838 537127838

Data Filer01-02 537127838 537127495

1.0.15 Container Filer01-01 Filer01-01 537127495 537127495

Root Filer01-01 Filer01-01 537127495 537127495

Data Filer01-01 Filer01-01 537127495 537127495

1.0.16 Container Filer01-02 Filer01-02 537127838 537127838

Root Filer01-02 Filer01-02 537127838 537127838

Data Filer01-02 537127838 537127495

Disk Partition Home Owner Home ID Owner ID

-------- --------- ----------------- ----------------- ----------- -----------

1.0.17 Container Filer01-01 Filer01-01 537127495 537127495

Root Filer01-01 Filer01-01 537127495 537127495

Data Filer01-01 Filer01-01 537127495 537127495

1.0.18 Container Filer01-02 Filer01-02 537127838 537127838

Root Filer01-02 Filer01-02 537127838 537127838

Data Filer01-02 537127838 537127495

1.0.19 Container Filer01-01 Filer01-01 537127495 537127495

Root Filer01-01 Filer01-01 537127495 537127495

Data Filer01-01 Filer01-01 537127495 537127495

1.0.20 Container Filer01-02 Filer01-02 537127838 537127838

Root Filer01-02 Filer01-02 537127838 537127838

Data Filer01-02 Filer01-02 537127838 537127838

1.0.21 Container Filer01-01 Filer01-01 537127495 537127495

Root Filer01-01 Filer01-01 537127495 537127495

Data Filer01-01 Filer01-01 537127495 537127495

1.0.22 Container Filer01-02 Filer01-02 537127838 537127838

Root Filer01-02 Filer01-02 537127838 537127838

Data Filer01-02 Filer01-02 537127838 537127838

1.0.23 Container Filer01-01 Filer01-01 537127495 537127495

Root Filer01-01 Filer01-01 537127495 537127495

Data Filer01-01 Filer01-01 537127495 537127495

24 entries were displayed.

system node run -node * -command sysconfig

2 entries were acted on.

Node: Filer01-01

NetApp Release 9.4P1: Fri Jul 20 23:30:57 EDT 2018

System ID: #### (Filer01-01); partner ID: #### (Filer01-02)

System Serial Number: #### (Filer01-01)

System Rev: B1

System Storage Configuration: Multi-Path HA

System ACP Connectivity: Inband Active

All-Flash Optimized: false

Backplane Part Number: DS22412

Backplane Rev:

Backplane Serial Number: #####

slot 0: System Board

Model Name: FAS2650

Processors: 6

Processor type: Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz

Memory Size: 32768 MB

Memory Attributes: Hoisting

Rank Interleaving

Channel Interleaving

Controller: A

Service Processor Status: Online

slot 0: Internal 10/100/1000 Ethernet Switch

Wrench: auto-1000t-fd-up

Device Type: 88E6176

slot 0: Gigabit Ethernet I210

e0M MAC Address: 00:a0:98:d5:1b:e8 (auto-1000t-fd-up)

slot 0: 10 Gigabit Ethernet Controller IX5-SFP+

e0a MAC Address: 00:a0:98:d5:1b:da (auto-10g_twinax-fd-up)

e0b MAC Address: 00:a0:98:d5:1b:db (auto-10g_twinax-fd-up)

slot 0: 1G/10G Ethernet Controller CNA EP 8324

(Multiple Dual-port, QLogic CNA 8324(8362) rev. 2)

e0c MAC Address: 00:a0:98:d5:1b:dc (auto-unknown-fd-down)

e0d MAC Address: 00:a0:98:d5:1b:dd (auto-1000t-fd-up)

e0e MAC Address: 02:a0:98:d5:1b:e8 (auto-1000t-fd-up)

e0f MAC Address: 00:a0:98:d5:1b:df (auto-unknown-fd-down)

slot 0: PLX DMA Engine 2 (PLX87d0)

State: Active

slot 0: Interconnect HBA: Generic OFED Provider

Port Name: ic0a

GID: fe80:0000:0000:0000:0000:0000:0000:0104

slot 0: NTB Interconnect (PLX87b0)

Max HW Data Rate: PCIe Gen 3 x 8

HW Data Rate: PCIe Gen 3 x 8

SW Data Rate: PCIe Gen 3 x 8

Logical Link: Up

Port State: Enabled

slot 0: SAS Host Adapter 0a

24 Disks: 27416.4GB

1 shelf with IOM12E

slot 0: SAS Host Adapter 0b

24 Disks: 27416.4GB

1 shelf with IOM12E

slot 0: Fibre Channel Target Host Adapter 0c

slot 0: Fibre Channel Target Host Adapter 0d

slot 0: Fibre Channel Target Host Adapter 0e

slot 0: Fibre Channel Target Host Adapter 0f

slot 0: Intel BDW SATA Controller u0a (0x92311000)

boot0 SV9MST6D120GLM41NP FW1122 114473MB 512B/sect (SPG1750076Z)

slot 0: Intel USB EHCI Adapter u0b (0x92312000)

slot 1: Flash Cache NVMe

Serial Number: #######

Part Number: 119-00328

Hardware Revision: A0

Firmware Version: NA01

Model Name: X3310A

Capacity: 512 GB

Vendor Name: Samsung

State: ok

Node: Filer01-02

NetApp Release 9.4P1: Fri Jul 20 23:30:57 EDT 2018

System ID: #### (Filer01-02); partner ID: #### (Filer01-01)

System Serial Number: #### (Filer01-02)

System Rev: B1

System Storage Configuration: Multi-Path HA

System ACP Connectivity: Inband Active

All-Flash Optimized: false

Backplane Part Number: DS22412

Backplane Rev:

Backplane Serial Number: ####

slot 0: System Board

Model Name: FAS2650

Processors: 6

Processor type: Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz

Memory Size: 32768 MB

Memory Attributes: Hoisting

Rank Interleaving

Channel Interleaving

Controller: B

Service Processor Status: Online

slot 0: Internal 10/100/1000 Ethernet Switch

Wrench: auto-1000t-fd-up

Device Type: 88E6176

slot 0: Gigabit Ethernet I210

e0M MAC Address: 00:a0:98:d5:36:7a (auto-1000t-fd-up)

slot 0: 10 Gigabit Ethernet Controller IX5-SFP+

e0a MAC Address: 00:a0:98:d5:36:6c (auto-10g_twinax-fd-up)

e0b MAC Address: 00:a0:98:d5:36:6d (auto-10g_twinax-fd-up)

slot 0: 1G/10G Ethernet Controller CNA EP 8324

(Multiple Dual-port, QLogic CNA 8324(8362) rev. 2)

e0c MAC Address: 00:a0:98:d5:36:6e (auto-1000t-fd-up)

e0d MAC Address: 00:a0:98:d5:36:6f (auto-1000t-fd-up)

e0e MAC Address: 02:a0:98:d5:36:7a (auto-1000t-fd-up)

e0f MAC Address: 00:a0:98:d5:36:71 (auto-unknown-fd-down)

slot 0: PLX DMA Engine 2 (PLX87d0)

State: Active

slot 0: Interconnect HBA: Generic OFED Provider

Port Name: ic0a

GID: fe80:0000:0000:0000:0000:0000:0000:0105

slot 0: NTB Interconnect (PLX87b0)

Max HW Data Rate: PCIe Gen 3 x 8

HW Data Rate: PCIe Gen 3 x 8

SW Data Rate: PCIe Gen 3 x 8

Logical Link: Up

Port State: Enabled

slot 0: SAS Host Adapter 0a

24 Disks: 27416.4GB

1 shelf with IOM12E

slot 0: SAS Host Adapter 0b

24 Disks: 27416.4GB

1 shelf with IOM12E

slot 0: Fibre Channel Target Host Adapter 0c

slot 0: Fibre Channel Target Host Adapter 0d

slot 0: Fibre Channel Target Host Adapter 0e

slot 0: Fibre Channel Target Host Adapter 0f

slot 0: Intel BDW SATA Controller u0a (0x92311000)

boot0 SV9MST6D120GLM41NP FW1122 114473MB 512B/sect (SPG1750079L)

slot 0: Intel USB EHCI Adapter u0b (0x92312000)

slot 1: Flash Cache NVMe

Serial Number: ####

Part Number: 119-00328

Hardware Revision: A0

Firmware Version: NA01

Model Name: X3310A

Capacity: 512 GB

Vendor Name: Samsung

State: ok

SpindleNinja · ‎2019-08-02

Looks like they're all there, but some (looks like 02's) are not owned. (Also, you might want to zero your spares)

How about these:

storage disk show -container-type unassigned

storage aggregate show-spare-disks

TMACMD · ‎2019-08-02

Do this which will completely wipe all data from the system

1. Boot both controllers to maintenance mode

2. Use the disk commands to work until all disks are unowned

3. Halt both controllers

4. set-defaults

5. saveenv

6. Boot both controllers to the boot menu with > boot_ontap menu

7. At the boot menu choose option 9 on both controllers and wait for prompts on both.

8. Choose 9a on one controller and wait for it to finish.

9. Choose option 9a on the other controller and wait for it to finish. You should see 24 drives now listed as unpartitioned and unowned.

10 if you want to use advanced disk partitioning then choose option 9b on one controller or if you want to use whole disks (I would not with only 24) choose option 9bc. In either case wait for the controller to reboot and start zeroing. You will see a few ...... on the screen.

11. After the first controller starts the zeroing process choose the same option on the second controller.

12. When both are done you should have access to all disks.

Note that ONTAP 9.4 introduced fast zeroing. This means it only takes a few seconds to zero the disks in any case..

You will need to reaquire the license keys.

You should be able to now configure ONTAP

TMACMD · ‎2019-08-02

Ok. After a wipe the controllers should each own 12 drives.

You should see 12 P1 (data) 53GiB partitions for each Controller and 12 P2 (root) 1016GiB partitions for each Controller

Losing 2 for parity and one for spare leaves 9 data partitions for each Controller

Take away wafl overhead and you may be around 7-8TB PORT CONTROLLER.