LOADER-A> set-defaults LOADER-A> saveenv LOADER-A> boot_ontap menu Loading X86_64/freebsd/image2/kernel:0x200000/15719336 0x10fdba8/13881768 Entry at 0xffffffff802dc4b0 Loading X86_64/freebsd/image2/platform.ko:0x1e3b000/4076864 0x221e540/586840 Starting program at 0xffffffff802dc4b0 NetApp Data ONTAP 9.5P5 IPsec: Initialized Security Association Processing. Copyright (C) 1992-2019 NetApp. All rights reserved. ******************************* * * * Press Ctrl-C for Boot Menu. * * * ******************************* cryptomod_fips: Executing Crypto FIPS Self Tests. cryptomod_fips: Crypto FIPS self-test: 'CPU COMPATIBILITY' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 ECB, AES-256 ECB' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CBC, AES-256 CBC' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 GCM, AES-256 GCM' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CCM' passed. cryptomod_fips: Crypto FIPS self-test: 'CTR_DRBG' passed. cryptomod_fips: Crypto FIPS self-test: 'SHA1, SHA256, SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'HMAC-SHA1, HMAC-SHA256, HMAC-SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'PBKDF2' passed. cryptomod_fips: Crypto FIPS self-test: 'Self-integrity' passed. Tue Jul 2 15:05:11 2019 [nv2flash.restage.progress:NOTICE]: ReStage is not needed because the flash has no data. Starting DHCP client on e0M Jul 02 15:06:31 Power outage protection flash de-staging: 49 cycles Please choose one of the following: (1) Normal Boot. (2) Boot without /etc/rc. (3) Change password. (4) Clean configuration and initialize all disks. (5) Maintenance mode boot. (6) Update flash from backup config. (7) Install new software first. (8) Reboot node. (9) Configure Advanced Drive Partitioning. Selection (1-9)? 5 Jul 02 15:06:50 [cluster01-01:cf.ic.sbb:notice]: HA interconnect: SBB Compatibility Event. Compatible partner node found. The interconnect device has been enabled. Jul 02 15:07:03 [cluster01-01:fal_nvme.partition.status:notice]: Partition 4cf809c00304d with capacity 894 GiB status: online. cryptomod_fips: Executing Crypto FIPS Self Tests. cryptomod_fips: Crypto FIPS self-test: 'CPU COMPATIBILITY' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 ECB, AES-256 ECB' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CBC, AES-256 CBC' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 GCM, AES-256 GCM' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CCM' passed. cryptomod_fips: Crypto FIPS self-test: 'CTR_DRBG' passed. cryptomod_fips: Crypto FIPS self-test: 'SHA1, SHA256, SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'HMAC-SHA1, HMAC-SHA256, HMAC-SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'PBKDF2' passed. cryptomod_fips: Crypto FIPS self-test: 'Self-integrity' passed. 5 You have selected the maintenance boot option: the system has booted in maintenance mode allowing the following operations to be performed: ? acorn acpadmin aggr azcs_w_compr_revert_disk clib cna_flash coredump_save coredump_show disk disk_latency disk_list disk_mung disk_shelf diskcopy disktest dumpblock environment fcadmin fcstat fctest fru_led ha-config halt help ifconfig key_manager led_off led_on nicadmin nv8 psmadmin raid_config sasadmin sasstat scsi sesdiag sldiag storage stsb sysconfig systemshell ucadmin version vmdisk_object_store vmservices vol vol_db vsa xortest Type "help " for more details. In a High Availablity configuration, you MUST ensure that the partner node is (and remains) down, or that takeover is manually disabled on the partner node, because High Availability software is not started or fully enabled in Maintenance mode. FAILURE TO DO SO CAN RESULT IN YOUR FILESYSTEMS BEING DESTROYED NOTE: It is okay to use 'show/status' sub-commands such as 'disk show or aggr status' in Maintenance mode while the partner is up Continue with boot? yes yes mkdir: /mroot/etc/.usercores: No such file or directory Could not create usercore directory. Error creating usercores directory web services cannot be started in 'maintenance' mode. Tue Jul 2 15:07:59 UTC 2019 *> Jul 02 15:07:59 [cluster01-01:kern.syslog.msg:notice]: root logged in from SP *> sysconfig -av NetApp Release 9.5P5: Fri Jun 14 11:35:45 EDT 2019 System ID: 0538048277 (cluster01-01) System Serial Number: 941924000013 (cluster01-01) System Rev: E0 System Storage Configuration: Multi-Path System ACP Connectivity: Inband Active All-Flash Optimized: false Capacity Optimized: false Backplane Part Number: DS21212 Backplane Rev: Backplane Serial Number: 031919003878 slot 0: System Board 1.5 GHz (System Board XXII E0) Model Name: FAS2720 Part Number: 111-04039 Revision: E0 Serial Number: 021920015191 BIOS version: 11.4 Loader version: 6.0.5 Processors: 12 Processor type: Intel(R) Xeon(R) CPU D-1557 @ 1.50GHz Memory Size: 32768 MB Memory Attributes: Hoisting Rank Interleaving Channel Interleaving Mezzanine Card: Part: 111-04040 Version: A1 Serial: 021909007931 NVMEM Size: 4094 MB of Main Memory Used Controller: A PCIE Switch 1: id 1 546 1.04 TPM FW Version: 0005.003E Service Processor Status: Unknown IPMI: unknown PKT: unknown slot 0: Internal 10/100/1000 Ethernet Switch Status: Unknown slot 0: Gigabit Ethernet I210 e0M MAC Address: d0:39:ea:03:0d:e5 (auto-1000t-fd-up) Device Type: 1533 Firmware Version: 3.25-7.5 0x80000674 slot 0: 10 Gigabit Ethernet Controller IX5-SFP+ e0a MAC Address: d0:39:ea:03:0d:d7 (auto-unknown-fd-down) e0b MAC Address: d0:39:ea:03:0d:d8 (auto-unknown-fd-down) Device Type: X550EM Firmware Version: 1.13-7.1 slot 0: 1/10 Gigabit Ethernet Controller IX4-T e0e MAC Address: d0:39:ea:03:0d:db (auto-unknown-fd-down) e0f MAC Address: d0:39:ea:03:0d:dc (auto-unknown-fd-down) e0c MAC Address: d0:39:ea:03:0d:d9 (auto-unknown-fd-down) e0d MAC Address: d0:39:ea:03:0d:da (auto-unknown-fd-down) Device Type: X550 Firmware Version: 1.93-144.3 slot 0: PLX DMA Engine 2 (PLX87d0) State: Inactive slot 0: Interconnect HBA: Generic OFED Provider Port Name: ic0a GID: fe80:0000:0000:0000:0000:0000:0000:0104 Base LID: 0x104 Active MTU: 8192 slot 0: NTB Interconnect (PLX87b0) HW Data Rate: PCIe Gen 1 x 0 SW Data Rate: PCIe Gen 1 x 0 Logical Link: Down Port State: Enabled slot 0: SAS Host Adapter 0a (PMC-Sierra PM8070 rev. C, SAS, ) Firmware rev: 03.08.09.00 Base WWN: 5:00a098:00b4ceb:b0 Phy State: [0] Enabled, 12.0 Gb/s [1] Enabled, 12.0 Gb/s [2] Enabled, 12.0 Gb/s [3] Enabled, 12.0 Gb/s Mini-SAS HD Vendor: Molex Inc. Mini-SAS HD Part Number: 112-00437+A0 Mini-SAS HD Type: Passive Copper (unequalized) 2m ID:00 Mini-SAS HD Serial Number: 905631438 ID Vendor Model FW Size 00.0 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.1 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.2 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.3 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.4 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.5 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.6 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.7 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.8 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.9 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.10: NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.11: NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.0 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.1 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.2 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.3 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.4 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.5 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.6 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.7 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.8 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.9 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.10: NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.11: NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) Shelf 0: DS212-12 Firmware rev. IOM12E A: 0120 IOM12E B: 0120 Shelf 1: DS212-12 Firmware rev. IOM12 A: 0230 IOM12 B: 0230 slot 0: SAS Host Adapter 0b (PMC-Sierra PM8070 rev. C, SAS, ) Firmware rev: 03.08.09.00 Base WWN: 5:00a098:00b4ceb:b4 Phy State: [4] Enabled, 12.0 Gb/s [5] Enabled, 12.0 Gb/s [6] Enabled, 12.0 Gb/s [7] Enabled, 12.0 Gb/s ID Vendor Model FW Size 00.0 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.1 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.2 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.3 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.4 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.5 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.6 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.7 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.8 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.9 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.10: NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 00.11: NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.0 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.1 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.2 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.3 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.4 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.5 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.6 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.7 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.8 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.9 : NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.10: NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) 01.11: NETAPP X380_STATE10TA07 NA00 9324.2GB (19134414848 520B/sect) Shelf 0: DS212-12 Firmware rev. IOM12E A: 0120 IOM12E B: 0120 Shelf 1: DS212-12 Firmware rev. IOM12 A: 0230 IOM12 B: 0230 slot 0: Intel BDW SATA Controller u0a (0x91f11000) boot0 SV9MST6D120GLM41NP FW1122 114473MB 512B/sect (SPG191801FQ) slot 0: Intel USB EHCI Adapter u0b (0x91f12000) slot 1: Flash Cache NVMe Serial Number: ZEP00APP Part Number: 119-00329 Hardware Revision: A0 Firmware Version: NA03 Model Name: X3311A Capacity: 1024 GB State: ok *> *> 031919003878 disk show -a Local System ID: 538048277 DISK OWNER POOL SERIAL NUMBER HOME DR HOME ------------ ------------- ----- ------------- ------------- ------------- NOTE: Currently 24 disks are unowned. Use 'disk show -n' for additional information. *> disk show -a n Local System ID: 538048277 DISK OWNER POOL SERIAL NUMBER HOME DR HOME ------------ ------------- ----- ------------- ------------- ------------- 0b.00.1 Not Owned NONE ZA2ALABW 0b.00.0 Not Owned NONE ZA2AL9ZL 0b.00.3 Not Owned NONE ZA2ALA49 0b.00.5 Not Owned NONE ZA2AL9JR 0a.01.11 Not Owned NONE ZA2B10ZA 0b.00.4 Not Owned NONE ZA2ALAAW 0b.00.8 Not Owned NONE ZA2AL9YD 0b.00.11 Not Owned NONE ZA2AL9FS 0b.00.6 Not Owned NONE ZA2ALAC2 0b.00.2 Not Owned NONE ZA2ALAA5 0a.01.9 Not Owned NONE ZA2B4433 0b.00.10 Not Owned NONE ZA2AL9SK 0a.01.0 Not Owned NONE ZA2B45GC 0a.01.1 Not Owned NONE ZA2B459V 0a.01.4 Not Owned NONE ZA2B46SQ 0a.01.8 Not Owned NONE ZA2B0ZYH 0b.00.9 Not Owned NONE ZA2AL9Y3 0a.01.6 Not Owned NONE ZA2B46KH 0a.01.5 Not Owned NONE ZA2B10B7 0b.00.7 Not Owned NONE ZA2ALA5T 0a.01.10 Not Owned NONE ZA2B1019 0a.01.3 Not Owned NONE ZA2B11XP 0a.01.7 Not Owned NONE ZA2B11V0 0a.01.2 Not Owned NONE ZA2B10H5 *> halt Terminated . Uptime: 4m50s System halting... BIOS Version: 11.4 Portions Copyright (C) 2014-2018 NetApp, Inc. All Rights Reserved. Initializing System Memory ... Loading Device Drivers ... Configuring Devices ... CPU = 1 Processor(s) Detected. Intel(R) Xeon(R) CPU D-1557 @ 1.50GHz (CPU 0) CPUID: 0x00050664. Cores per Processor = 12 32768 MB System RAM Installed. SATA (AHCI) Device: SV9MST6D120GLM41NP Boot Loader version 6.0.5 Copyright (C) 2000-2003 Broadcom Corporation. Portions Copyright (C) 2002-2018 NetApp, Inc. All Rights Reserved. LOADER-A> boot_ontap menu Loading X86_64/freebsd/image2/kernel:0x200000/15719336 0x10fdba8/13881768 Entry at 0xffffffff802dc4b0 Loading X86_64/freebsd/image2/platform.ko:0x1e3b000/4076864 0x221e540/586840 Starting program at 0xffffffff802dc4b0 NetApp Data ONTAP 9.5P5 IPsec: Initialized Security Association Processing. Copyright (C) 1992-2019 NetApp. All rights reserved. ******************************* * * * Press Ctrl-C for Boot Menu. * * * ******************************* cryptomod_fips: Executing Crypto FIPS Self Tests. cryptomod_fips: Crypto FIPS self-test: 'CPU COMPATIBILITY' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 ECB, AES-256 ECB' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CBC, AES-256 CBC' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 GCM, AES-256 GCM' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CCM' passed. cryptomod_fips: Crypto FIPS self-test: 'CTR_DRBG' passed. cryptomod_fips: Crypto FIPS self-test: 'SHA1, SHA256, SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'HMAC-SHA1, HMAC-SHA256, HMAC-SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'PBKDF2' passed. cryptomod_fips: Crypto FIPS self-test: 'Self-integrity' passed. Tue Jul 2 15:10:31 2019 [nv2flash.restage.progress:NOTICE]: ReStage is not needed because the flash has no data. Starting DHCP client on e0M Jul 02 15:11:51 Power outage protection flash de-staging: 49 cycles Please choose one of the following: (1) Normal Boot. (2) Boot without /etc/rc. (3) Change password. (4) Clean configuration and initialize all disks. (5) Maintenance mode boot. (6) Update flash from backup config. (7) Install new software first. (8) Reboot node. (9) Configure Advanced Drive Partitioning. Selection (1-9)? 9 ***OS2SP configured successfully***Jul 02 15:12:28 [cluster01-01:cf.ic.sbb:notice]: HA interconnect: SBB Compatibility Event. Compatible partner node found. The interconnect device has been enabled. Jul 02 15:12:41 [cluster01-01:fal_nvme.partition.status:notice]: Partition 4cf809c00304d with capacity 894 GiB status: online. cryptomod_fips: Executing Crypto FIPS Self Tests. cryptomod_fips: Crypto FIPS self-test: 'CPU COMPATIBILITY' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 ECB, AES-256 ECB' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CBC, AES-256 CBC' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 GCM, AES-256 GCM' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CCM' passed. cryptomod_fips: Crypto FIPS self-test: 'CTR_DRBG' passed. cryptomod_fips: Crypto FIPS self-test: 'SHA1, SHA256, SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'HMAC-SHA1, HMAC-SHA256, HMAC-SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'PBKDF2' passed. cryptomod_fips: Crypto FIPS self-test: 'Self-integrity' passed. ************************************************* * Advanced Drive Partitioning Boot Menu Options * ************************************************* (9a) Unpartition all disks and remove their ownership information. (9b) Clean configuration and initialize node with partitioned disks. (9c) Clean configuration and initialize node with whole disks. (9d) Reboot the node. (9e) Return to main boot menu. Selection (9a-9e)?: 9a 9a ########## WARNING ########## This is a disruptive operation and will result in the loss of all filesystem data. Before proceeding further, make sure that: 1) This option (9a) has been executed or will be executed on the HA partner node (and DR/DR-AUX partner nodes if applicable), prior to reinitializing any system in the HA-pair (or MCC setup). 2) The HA partner node (and DR/DR-AUX partner nodes if applicable) is currently waiting at the boot menu. Do you still want to continue (yes/no)? yes yes Reassigning locally attached disks to local node... Changing ownership of locally attached disk 0b.00.4 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0b.00.1 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0a.01.9 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0b.00.2 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0b.00.3 from 4294967295 to local site 538048277 Jul 02 15:13:53 [cluster01-01:diskown.RescanMessageFailed:error]: Could not send rescan message to cluster01-01. Changing ownership of locally attached disk 0a.01.8 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0b.00.6 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0a.01.0 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0a.01.1 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0b.00.10 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0b.00.5 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0b.00.7 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0b.00.11 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0a.01.4 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0a.01.11 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0a.01.5 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0b.00.9 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0b.00.0 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0b.00.8 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0a.01.6 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0a.01.10 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0a.01.7 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0a.01.3 from 4294967295 to local site 538048277 Changing ownership of locally attached disk 0a.01.2 from 4294967295 to local site 538048277 Cannot do remote rescan. Use 'run local disk show' on the console of cluster01-01 for it to scan the newly assigned disks Changed ownership of 24 out of 24 disks, status: Succeeded Destroying existing aggregates.. Boot destroy aggregates: Disk 0b.00.4 forced to be a SPARE disk Boot destroy aggregates: Disk 0b.00.1 forced to be a SPARE disk Boot destroy aggregates: Disk 0a.01.9 forced to be a SPARE disk Boot destroy aggregates: Disk 0b.00.2 forced to be a SPARE disk Boot destroy aggregates: Disk 0b.00.3 forced to be a SPARE disk Boot destroy aggregates: Disk 0a.01.8 forced to be a SPARE disk Boot destroy aggregates: Disk 0b.00.6 forced to be a SPARE disk Boot destroy aggregates: Disk 0a.01.0 forced to be a SPARE disk Boot destroy aggregates: Disk 0a.01.1 forced to be a SPARE disk Boot destroy aggregates: Disk 0b.00.10 forced to be a SPARE disk Boot destroy aggregates: Disk 0b.00.5 forced to be a SPARE disk Boot destroy aggregates: Disk 0b.00.7 forced to be a SPARE disk Boot destroy aggregates: Disk 0b.00.11 forced to be a SPARE disk Boot destroy aggregates: Disk 0a.01.4 forced to be a SPARE disk Boot destroy aggregates: Disk 0a.01.11 forced to be a SPARE disk Boot destroy aggregates: Disk 0a.01.5 forced to be a SPARE disk Boot destroy aggregates: Disk 0b.00.9 forced to be a SPARE disk Boot destroy aggregates: Disk 0b.00.0 forced to be a SPARE disk Boot destroy aggregates: Disk 0b.00.8 forced to be a SPARE disk Boot destroy aggregates: Disk 0a.01.6 forced to be a SPARE disk Boot destroy aggregates: Disk 0a.01.10 forced to be a SPARE disk Boot destroy aggregates: Disk 0a.01.7 forced to be a SPARE disk Boot destroy aggregates: Disk 0a.01.3 forced to be a SPARE disk Boot destroy aggregates: Disk 0a.01.2 forced to be a SPARE disk Unpartitioning disks... No disk found to be eligible for unpartitioning. Removing disk ownership... Disk ownership successfully removed. bootarg.bootmenu.selection is || ************************************************* * Advanced Drive Partitioning Boot Menu Options * ************************************************* (9a) Unpartition all disks and remove their ownership information. (9b) Clean configuration and initialize node with partitioned disks. (9c) Clean configuration and initialize node with whole disks. (9d) Reboot the node. (9e) Return to main boot menu. Selection (9a-9e)?: 9b 9b ########## WARNING ########## All configuration data will be deleted and the node will be initialized with partitioned disks. Existing disk partitions must be removed from all disks (9a) attached to this node and its HA partner (and DR/DR-AUX partner nodes if applicable). The HA partner (and DR/DR-AUX partner nodes if applicable) must be waiting at the boot menu or already initialized with partitioned disks (9b). Do you still want to continue (yes/no)? yes yes AdpInit: This system will now reboot to perform wipeclean. bootarg.bootmenu.selection is |wipeconfig| Terminated . Skipped backing up /var file system to boot device. Uptime: 5m32s System rebooting... BIOS Version: 11.4 Portions Copyright (C) 2014-2018 NetApp, Inc. All Rights Reserved. Initializing System Memory ... Loading Device Drivers ... Configuring Devices ... CPU = 1 Processor(s) Detected. Intel(R) Xeon(R) CPU D-1557 @ 1.50GHz (CPU 0) CPUID: 0x00050664. Cores per Processor = 12 32768 MB System RAM Installed. SATA (AHCI) Device: SV9MST6D120GLM41NP Boot Loader version 6.0.5 Copyright (C) 2000-2003 Broadcom Corporation. Portions Copyright (C) 2002-2018 NetApp, Inc. All Rights Reserved. Starting AUTOBOOT press Ctrl-C to abort... Loading X86_64/freebsd/image2/kernel:0x200000/15719336 0x10fdba8/13881768 Entry at 0xffffffff802dc4b0 Loading X86_64/freebsd/image2/platform.ko:0x1e3b000/4076864 0x221e540/586840 Starting program at 0xffffffff802dc4b0 NetApp Data ONTAP 9.5P5 IPsec: Initialized Security Association Processing. Copyright (C) 1992-2019 NetApp. All rights reserved. ******************************* * * * Press Ctrl-C for Boot Menu. * * * ******************************* cryptomod_fips: Executing Crypto FIPS Self Tests. cryptomod_fips: Crypto FIPS self-test: 'CPU COMPATIBILITY' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 ECB, AES-256 ECB' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CBC, AES-256 CBC' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 GCM, AES-256 GCM' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CCM' passed. cryptomod_fips: Crypto FIPS self-test: 'CTR_DRBG' passed. cryptomod_fips: Crypto FIPS self-test: 'SHA1, SHA256, SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'HMAC-SHA1, HMAC-SHA256, HMAC-SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'PBKDF2' passed. cryptomod_fips: Crypto FIPS self-test: 'Self-integrity' passed. Tue Jul 2 15:16:34 2019 [nv2flash.restage.progress:NOTICE]: ReStage is not needed because the flash has no data. Wipe filer procedure requested. Starting DHCP client on e0M Jul 02 15:17:54 Power outage protection flash de-staging: 49 cycles ***OS2SP configured successfully***Jul 02 15:18:03 [localhost:cf.ic.sbb:notice]: HA interconnect: SBB Compatibility Event. Compatible partner node found. The interconnect device has been enabled. Jul 02 15:18:16 [localhost:fal_nvme.partition.status:notice]: Partition 4cf809c00304d with capacity 894 GiB status: online. cryptomod_fips: Executing Crypto FIPS Self Tests. cryptomod_fips: Crypto FIPS self-test: 'CPU COMPATIBILITY' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 ECB, AES-256 ECB' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CBC, AES-256 CBC' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 GCM, AES-256 GCM' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CCM' passed. cryptomod_fips: Crypto FIPS self-test: 'CTR_DRBG' passed. cryptomod_fips: Crypto FIPS self-test: 'SHA1, SHA256, SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'HMAC-SHA1, HMAC-SHA256, HMAC-SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'PBKDF2' passed. cryptomod_fips: Crypto FIPS self-test: 'Self-integrity' passed. 9b AdpInit: Root will be created with 6 disks with configuration as (2d+3p+1s) using disks of type (FSAS). bootarg.bootmenu.selection is |4a| AdpInit: System will now perform initialization using option 4a BOOTMGR: The system has 0 disks assigned whereas it needs 6 to boot, will try to assign the required number. sanown_assign_X_disks: assign disks from my unowned local site pool0 loop Jul 02 15:18:29 [localhost:diskown.hlfShlf.assignStatus:notice]: Half shelf based automatic disk assignment is "disabled". Jul 02 15:18:29 [localhost:diskown.mccip.assignStatus:notice]: MCC quarter shelf-based automatic disk assignment is "disabled". Jul 02 15:18:29 [localhost:diskown.drawer.assignStatus:notice]: Drawer-based automatic disk assignment is "disabled". sanown_assign_disk_helper: Assigned disk 0b.00.9 Cannot do remote rescan. Use 'run local disk show' on the console of ?? for it to scan the newly assigned disks Jul 02 15:18:30 [localhost:diskown.RescanMessageFailed:error]: Could not send rescan message to ??. Jul 02 15:18:30 [localhost:config.noPartnerDisks:error]: No disks were detected for the partner; this node cannot perform takeover correctly. Jul 02 15:18:30 [localhost:callhome.dsk.config:error]: Call home for DISK CONFIGURATION ERROR sanown_assign_disk_helper: Assigned disk 0b.00.3 sanown_assign_disk_helper: Assigned disk 0b.00.7 sanown_assign_disk_helper: Assigned disk 0b.00.11 sanown_assign_disk_helper: Assigned disk 0b.00.5 sanown_assign_disk_helper: Assigned disk 0b.00.1 BOOTMGR: already_assigned=0, min_to_boot=6, num_assigned=6 .Jul 02 15:18:31 [localhost:raid.disk.fast.zero.done:notice]: Disk 0b.00.3 Shelf 0 Bay 3 [NETAPP X380_STATE10TA07 NA00] S/N [ZA2ALA49] UID [5000C500:A7EA0F6F:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000] : disk zeroing complete (0x5d1b75c710eca31c). Jul 02 15:18:32 [localhost:raid.disk.fast.zero.done:notice]: Disk 0b.00.7 Shelf 0 Bay 7 [NETAPP X380_STATE10TA07 NA00] S/N [ZA2ALA5T] UID [5000C500:A7EA098B:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000] : disk zeroing complete (0x5d1b75c81fc88bf2). Jul 02 15:18:32 [localhost:raid.disk.fast.zero.done:notice]: Disk 0b.00.1 Shelf 0 Bay 1 [NETAPP X380_STATE10TA07 NA00] S/N [ZA2ALABW] UID [5000C500:A7E9FE47:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000] : disk zeroing complete (0x5d1b75c8275bcd2b). Jul 02 15:18:32 [localhost:raid.disk.fast.zero.done:notice]: Disk 0b.00.5 Shelf 0 Bay 5 [NETAPP X380_STATE10TA07 NA00] S/N [ZA2AL9JR] UID [5000C500:A7EA2433:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000] : disk zeroing complete (0x5d1b75c87bfad63c). Jul 02 15:18:32 [localhost:raid.disk.fast.zero.done:notice]: Disk 0b.00.11 Shelf 0 Bay 11 [NETAPP X380_STATE10TA07 NA00] S/N [ZA2AL9FS] UID [5000C500:A7EA2B67:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000] : disk zeroing complete (0x5d1b75c811073cbb). Jul 02 15:18:32 [localhost:raid.disk.fast.zero.done:notice]: Disk 0b.00.9 Shelf 0 Bay 9 [NETAPP X380_STATE10TA07 NA00] S/N [ZA2AL9Y3] UID [5000C500:A7EA2FF7:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000] : disk zeroing complete (0x5d1b75c8722421b8). Jul 02 15:18:33 [localhost:raid.autoPart.start:notice]: System has started auto-partitioning 6 disks. Jul 02 15:18:34 [localhost:raid.partition.disk:notice]: Disk partition successful on Disk 0b.00.1 Shelf 0 Bay 1 [NETAPP X380_STATE10TA07 NA00] S/N [ZA2ALABW] UID [5000C500:A7E9FE47:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000], partitions created 2, partition sizes specified 1, partition spec summary [2]=24517760. .Jul 02 15:18:37 [localhost:raid.partition.disk:notice]: Disk partition successful on Disk 0b.00.3 Shelf 0 Bay 3 [NETAPP X380_STATE10TA07 NA00] S/N [ZA2ALA49] UID [5000C500:A7EA0F6F:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000], partitions created 2, partition sizes specified 1, partition spec summary [2]=24517760. Jul 02 15:18:39 [localhost:raid.partition.disk:notice]: Disk partition successful on Disk 0b.00.5 Shelf 0 Bay 5 [NETAPP X380_STATE10TA07 NA00] S/N [ZA2AL9JR] UID [5000C500:A7EA2433:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000], partitions created 2, partition sizes specified 1, partition spec summary [2]=24517760. Jul 02 15:18:41 [localhost:raid.partition.disk:notice]: Disk partition successful on Disk 0b.00.7 Shelf 0 Bay 7 [NETAPP X380_STATE10TA07 NA00] S/N [ZA2ALA5T] UID [5000C500:A7EA098B:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000], partitions created 2, partition sizes specified 1, partition spec summary [2]=24517760. .Jul 02 15:18:43 [localhost:raid.partition.disk:notice]: Disk partition successful on Disk 0b.00.9 Shelf 0 Bay 9 [NETAPP X380_STATE10TA07 NA00] S/N [ZA2AL9Y3] UID [5000C500:A7EA2FF7:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000], partitions created 2, partition sizes specified 1, partition spec summary [2]=24517760. Jul 02 15:18:45 [localhost:raid.partition.disk:notice]: Disk partition successful on Disk 0b.00.11 Shelf 0 Bay 11 [NETAPP X380_STATE10TA07 NA00] S/N [ZA2AL9FS] UID [5000C500:A7EA2B67:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000], partitions created 2, partition sizes specified 1, partition spec summary [2]=24517760. Jul 02 15:18:46 [localhost:raid.autoPart.done:notice]: Successfully auto-partitioned 6 of 6 disks. Unable to create root aggregate: 5 disks specified, but at least 7 disks are required for raid_tec Root aggregate creation failed. Aborting Initialization. Rebooting... (press ctrl-c during boot to reattempt initialization) Uptime: 2m20s System rebooting... BIOS Version: 11.4 Portions Copyright (C) 2014-2018 NetApp, Inc. All Rights Reserved. Initializing System Memory ... Loading Device Drivers ... Configuring Devices ... CPU = 1 Processor(s) Detected. Intel(R) Xeon(R) CPU D-1557 @ 1.50GHz (CPU 0) CPUID: 0x00050664. Cores per Processor = 12 32768 MB System RAM Installed. SATA (AHCI) Device: SV9MST6D120GLM41NP Boot Loader version 6.0.5 Copyright (C) 2000-2003 Broadcom Corporation. Portions Copyright (C) 2002-2018 NetApp, Inc. All Rights Reserved. Starting AUTOBOOT press Ctrl-C to abort... Loading X86_64/freebsd/image2/kernel:0x200000/15719336 0x10fdba8/13881768 Entry at 0xffffffff802dc4b0 Loading X86_64/freebsd/image2/platform.ko:0x1e3b000/4076864 0x221e540/586840 Starting program at 0xffffffff802dc4b0 NetApp Data ONTAP 9.5P5 IPsec: Initialized Security Association Processing. Copyright (C) 1992-2019 NetApp. All rights reserved. ******************************* * * * Press Ctrl-C for Boot Menu. * * * ******************************* cryptomod_fips: Executing Crypto FIPS Self Tests. cryptomod_fips: Crypto FIPS self-test: 'CPU COMPATIBILITY' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 ECB, AES-256 ECB' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CBC, AES-256 CBC' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 GCM, AES-256 GCM' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CCM' passed. cryptomod_fips: Crypto FIPS self-test: 'CTR_DRBG' passed. cryptomod_fips: Crypto FIPS self-test: 'SHA1, SHA256, SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'HMAC-SHA1, HMAC-SHA256, HMAC-SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'PBKDF2' passed. cryptomod_fips: Crypto FIPS self-test: 'Self-integrity' passed. Tue Jul 2 15:19:20 2019 [nv2flash.restage.progress:NOTICE]: ReStage is not needed because the flash has no data. Attempting to use existing varfs on /dev/nvrd1 Starting DHCP client on e0M Jul 02 15:20:39 Power outage protection flash de-staging: 49 cycles ***OS2SP configured successfully***Jul 02 15:21:28 [localhost:cf.ic.sbb:notice]: HA interconnect: SBB Compatibility Event. Compatible partner node found. The interconnect device has been enabled. Jul 02 15:21:41 [localhost:fal_nvme.partition.status:notice]: Partition 4cf809c00304d with capacity 894 GiB status: online. Jul 02 15:21:48 [localhost:config.noPartnerDisks:error]: No disks were detected for the partner; this node cannot perform takeover correctly. Jul 02 15:21:48 [localhost:callhome.dsk.config:error]: Call home for DISK CONFIGURATION ERROR cryptomod_fips: Executing Crypto FIPS Self Tests. cryptomod_fips: Crypto FIPS self-test: 'CPU COMPATIBILITY' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 ECB, AES-256 ECB' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CBC, AES-256 CBC' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 GCM, AES-256 GCM' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CCM' passed. cryptomod_fips: Crypto FIPS self-test: 'CTR_DRBG' passed. cryptomod_fips: Crypto FIPS self-test: 'SHA1, SHA256, SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'HMAC-SHA1, HMAC-SHA256, HMAC-SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'PBKDF2' passed. cryptomod_fips: Crypto FIPS self-test: 'Self-integrity' passed. Jul 02 15:21:50 [localhost:raid.assim.tree.noRootVol:error]: No usable root volume was found! disk:L#: rtID/plx/rgid/rgdn nplx/ngrp/ndsk/tot gen/time shtdwn vnm WARNING: Neither label appeared to be valid, the labels may be corrupt. 0b.00.11P1: :Invalid Label! CRC 0x0, mgc 0x0, lbl V0, sz 0 , flags 0x0 SPARE WARNING: Neither label appeared to be valid, the labels may be corrupt. 0b.00.11P2: :Invalid Label! CRC 0x9d94b571, mgc 0x0, lbl V0, sz 0 , flags 0x0 SPARE WARNING: Neither label appeared to be valid, the labels may be corrupt. 0b.00.1P1: :Invalid Label! CRC 0x0, mgc 0x0, lbl V0, sz 0 , flags 0x0 SPARE WARNING: Neither label appeared to be valid, the labels may be corrupt. 0b.00.1P2: :Invalid Label! CRC 0x9d94b571, mgc 0x0, lbl V0, sz 0 , flags 0x0 SPARE WARNING: Neither label appeared to be valid, the labels may be corrupt. 0b.00.3P1: :Invalid Label! CRC 0x0, mgc 0x0, lbl V0, sz 0 , flags 0x0 SPARE WARNING: Neither label appeared to be valid, the labels may be corrupt. 0b.00.5P1: :Invalid Label! CRC 0x0, mgc 0x0, lbl V0, sz 0 , flags 0x0 SPARE WARNING: Neither label appeared to be valid, the labels may be corrupt. 0b.00.7P1: :Invalid Label! CRC 0x0, mgc 0x0, lbl V0, sz 0 , flags 0x0 SPARE WARNING: Neither label appeared to be valid, the labels may be corrupt. 0b.00.9P1: :Invalid Label! CRC 0x0, mgc 0x0, lbl V0, sz 0 , flags 0x0 SPARE WARNING: Neither label appeared to be valid, the labels may be corrupt. 0b.00.3P2: :Invalid Label! CRC 0x9d94b571, mgc 0x0, lbl V0, sz 0 , flags 0x0 SPARE WARNING: Neither label appeared to be valid, the labels may be corrupt. 0b.00.9P2: :Invalid Label! CRC 0x9d94b571, mgc 0x0, lbl V0, sz 0 , flags 0x0 PANIC : raid: Unable to find root aggregate. Reason: Unknown. (DS=18, DL=18, DA=18, BDTOC=0, BDLBL=0, BLMAG=24 BLCRC=24, BLVER=0, BLSZ=0, BLTOC=0, BLOBJ=0) version: 9.5P5: Fri Jun 14 11:35:45 EDT 2019 conf : x86_64.optimize.nodar cpuid = 0 Uptime: 2m35s PANIC: raid: Unable to find root aggregate. Reason: Unknown. (DS=18, DL=18, DA=18, BDTOC=0, BDLBL=0, BLMAG=24 BLCRC=24, BLVER=0, BLSZ=0, BLTOC=0, BLOBJ=0) in SK process rc on release 9.5P5 (C) on Tue Jul 2 15:21:51 GMT 2019 version: 9.5P5: Fri Jun 14 11:35:45 EDT 2019 compile flags: x86_64.optimize.nodar Writing panic info to HA mailbox disks. HA: current time (in sk_msecs) 23154 (in sk_cycles) 486004438372883 DUMPCORE: START Dumping to boot device. coredump: Using block 8388578 instead of block 8388577 as the sparecore abort block coredump: processing segment 0, at relative offset 8192 . coredump: completed segment 0 (0ms) coredump: processing segment 1, at relative offset 8192 . coredump: completed segment 1 (0ms) coredump: processing segment 2, at relative offset 8192 .. coredump: completed segment 2 (3697ms) coredump: processing segment 3, at relative offset 349880320 . coredump: completed segment 3 (11ms) coredump: processing segment 4, at relative offset 350199808 . coredump: completed segment 4 (945ms) coredump: processing segment 5, at relative offset 435838976 . coredump: completed segment 5 (0ms) coredump: processing segment 6, at relative offset 435838976 ....... coredump: completed segment 6 (21248ms) coredump: processing segment 8, at relative offset 2378575872 ............ coredump: completed segment 8 (10004ms) coredump: processing segment 9, at relative offset 2587623424 . coredump: completed segment 9 (1ms) coredump: processing segment 7, at relative offset 2587639808 ... coredump: completed segment 7 (1848ms) coredump: processing segment 10, at relative offset 2600558592 .... coredump: completed segment 10 (58307ms) DUMPCORE: END -- coredump written coredump: compression ms 24675 compressions 12622 bytes in 34107015168 bytes out 3612244550 ratio 944% comp BW 1318MB/s coredump: write ms 8587 write count 945 write bytes 3612352512 write BW 401MB/s coredump: total ms 96395 chunks of all zeroes 5421 System halting... BIOS Version: 11.4 Portions Copyright (C) 2014-2018 NetApp, Inc. All Rights Reserved. Initializing System Memory ... Loading Device Drivers ... Configuring Devices ... CPU = 1 Processor(s) Detected. Intel(R) Xeon(R) CPU D-1557 @ 1.50GHz (CPU 0) CPUID: 0x00050664. Cores per Processor = 12 32768 MB System RAM Installed. SATA (AHCI) Device: SV9MST6D120GLM41NP Boot Loader version 6.0.5 Copyright (C) 2000-2003 Broadcom Corporation. Portions Copyright (C) 2002-2018 NetApp, Inc. All Rights Reserved. LOADER-A>