Solved: Re: Call home for DISK CONFIGURATION ERROR

Divine · ‎2020-10-19

Hello Guys,

We just set up a FAS2720. We attached some disk shelves from an older system to the controller after unowning them, but we realized that there were 4 disks that still pointed to the old controller. So we performed a disk zeroing on the first controller; that was successful, but on the second controller the process is stuck in a loop of rebooting. It cannot find any disk, not even the ones on the FAS2720 controller. This is what we have been getting over and over:

Oct 19 16:25:29 Power outage protection flash de-staging: 46 cycles
***OS2SP configured successfully***Oct 19 16:26:19 [localhost:cf.ic.sbb:notice]: HA interconnect: SBB Compatibility Event. Compatible partner node found. The interconnect device has been enabled.
Oct 19 16:27:20 [localhost:fal_nvme.partition.status:notice]: Partition 0-1 with capacity 894 GiB status: online.
Oct 19 16:27:53 [localhost:config.noPartnerDisks:error]: No disks were detected for the partner; this node cannot perform takeover correctly.
Oct 19 16:27:53 [localhost:callhome.dsk.config:error]: Call home for DISK CONFIGURATION ERROR
cryptomod_fips: Executing Crypto FIPS Self Tests.
cryptomod_fips: Crypto FIPS self-test: 'CPU COMPATIBILITY' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 ECB, AES-256 ECB' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 CBC, AES-256 CBC' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 GCM, AES-256 GCM' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 CCM' passed.
cryptomod_fips: Crypto FIPS self-test: 'CTR_DRBG' passed.
cryptomod_fips: Crypto FIPS self-test: 'SHA1, SHA256, SHA512' passed.
cryptomod_fips: Crypto FIPS self-test: 'HMAC-SHA1, HMAC-SHA256, HMAC-SHA512' passed.
cryptomod_fips: Crypto FIPS self-test: 'PBKDF2' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-XTS 128, AES-XTS 256' passed.
cryptomod_fips: Crypto FIPS self-test: 'Self-integrity' passed.
Oct 19 16:27:55 [localhost:raid.autoPart.disabled:notice]: Auto-partitioning is disabled on this system: the system needs a minimum of 8 usable internal hard disks.
WARNING: 0 disks found!

Storage Adapters found:
0 Fibre Channel Storage Adapters found!
2 SAS Adapters found!
0 Parallel SCSI Storage Adapters found!
0 ATA Adapters found!

Select option 4 from the boot menu to choose disks for the root volume and spare core space.
Target Adapters found:
4 Fibre Channel Target Adapters found!
0 iSCSI Target Adapters found!

WARNING: there do not appear to be any disks attached to the system.
No root volume found.
Rebooting... (press ctrl-c during boot to break reboot loop)
Uptime: 3m23s
System rebooting...
BIOS Version: 11.5
Portions Copyright (C) 2014-2018 NetApp, Inc. All Rights Reserved.

Initializing System Memory ...
Loading Device Drivers ...
Configuring Devices ...

CPU = 1 Processor(s) Detected.
Intel(R) Xeon(R) CPU D-1557 @ 1.50GHz (CPU 0)
CPUID: 0x00050664. Cores per Processor = 12
32768 MB System RAM Installed.
SATA (AHCI) Device: SV9MST6D120GLM41NP

SpindleNinja · ‎2020-10-20

What are the external shelves and drives? What's internal?

I have a feeling that it's doing something screwy with RAID TEC and the internal shelves when it tries to create the roots.

it's been talked about before - https://community.netapp.com/t5/FAS-and-V-Series-Storage-Systems-Discussions/FAS-2720-new-aggregate-not-create/td-p/145463

Try maybe doing the re-init without the attached shelves?

Halt

detatch external shelves.

Park Controller 2 at LOADER>
Boot Controller 1 in to the boot menu and do opt 9
Run opt 9a
Run opt 9b

Wait for the ….. to finish on Controller 1
Boot Controller2 in to the boot menu
Run opt 9a
Run opt 9b
Wait for Controller 2 to finish.
Create Cluster

View solution in original post

SpindleNinja · ‎2020-10-19

halt both sides. and since this is new try option 9a/9b and see where that gets you.

Divine · ‎2020-10-19

Okay. We will try that tomorrow and update you. Thanks.

SpindleNinja · ‎2020-10-19

no problem. opt 9 is kinda like opt4 but with more options.

Divine · ‎2020-10-20

It did not work. this is what we're seeing.

Starting AUTOBOOT press Ctrl-C to abort...
Loading X86_64/freebsd/image1/kernel:0x200000/1056103 0x302000/10666816 0xf2e340/3748248 0x12c14d8/13689912 0x200200/1016 Entry at 0xffffffff80302000
Loading X86_64/freebsd/image1/platform.ko:0x1fd0000/4294256 0x23e8670/591912
Starting program at 0xffffffff80302000
NetApp Data ONTAP 9.6P9
mlx5en: Mellanox Ethernet driver 3.5.0 (November 2018)
mlx5ib: Mellanox Connect-IB Infiniband driver 3.5.0 (November 2018)
ena_rss_init_default_deferred() [TID:100000]: No devclass ena

Copyright (C) 1992-2020 NetApp.
All rights reserved.
*******************************
* *
* Press Ctrl-C for Boot Menu. *
* *
*******************************
cryptomod_fips: Executing Crypto FIPS Self Tests.
cryptomod_fips: Crypto FIPS self-test: 'CPU COMPATIBILITY' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 ECB, AES-256 ECB' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 CBC, AES-256 CBC' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 GCM, AES-256 GCM' passed.
cryptomod_fips: Crypto FIPS self-test: 'CTR_DRBG' passed.
cryptomod_fips: Crypto FIPS self-test: 'SHA1, SHA256, SHA512' passed.
cryptomod_fips: Crypto FIPS self-test: 'HMAC-SHA1, HMAC-SHA256, HMAC-SHA512' passed.
cryptomod_fips: Crypto FIPS self-test: 'PBKDF2' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-XTS 128, AES-XTS 256' passed.
cryptomod_fips: Crypto FIPS self-test: 'Self-integrity' passed.
Tue Oct 20 13:21:34 2020 [nv2flash.restage.progress:NOTICE]: ReStage is not needed because the flash has no data.
Starting DHCP client on e0M

Oct 20 13:22:55 Power outage protection flash de-staging: 45 cycles
***OS2SP configured successfully***Oct 20 13:23:45 [localhost:cf.ic.sbb:notice]: HA interconnect: SBB Compatibility Event. Compatible partner node found. The interconnect device has been enabled.
Oct 20 13:24:46 [localhost:fal_nvme.partition.status:notice]: Partition 0-1 with capacity 894 GiB status: online.
Oct 20 13:25:20 [localhost:config.noPartnerDisks:error]: No disks were detected for the partner; this node cannot perform takeover correctly.
Oct 20 13:25:20 [localhost:callhome.dsk.config:error]: Call home for DISK CONFIGURATION ERROR
cryptomod_fips: Executing Crypto FIPS Self Tests.
cryptomod_fips: Crypto FIPS self-test: 'CPU COMPATIBILITY' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 ECB, AES-256 ECB' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 CBC, AES-256 CBC' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 GCM, AES-256 GCM' passed.
cryptomod_fips: Crypto FIPS self-test: 'CTR_DRBG' passed.
cryptomod_fips: Crypto FIPS self-test: 'SHA1, SHA256, SHA512' passed.
cryptomod_fips: Crypto FIPS self-test: 'HMAC-SHA1, HMAC-SHA256, HMAC-SHA512' passed.
cryptomod_fips: Crypto FIPS self-test: 'PBKDF2' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-XTS 128, AES-XTS 256' passed.
cryptomod_fips: Crypto FIPS self-test: 'Self-integrity' passed.
Oct 20 13:25:22 [localhost:raid.autoPart.disabled:notice]: Auto-partitioning is disabled on this system: the system needs a minimum of 8 usable internal hard disks.
WARNING: 0 disks found!

Storage Adapters found:
0 Fibre Channel Storage Adapters found!
2 SAS Adapters found!

0 Parallel SCSI Storage Adapters found!
0 ATA Adapters found!

Select option 4 from the boot menu to choose disks for the root volume and spare core space.
Target Adapters found:
4 Fibre Channel Target Adapters found!
0 iSCSI Target Adapters found!

WARNING: there do not appear to be any disks attached to the system.
No root volume found.
Rebooting... (press ctrl-c during boot to break reboot loop)
Uptime: 3m54s
System rebooting...
BIOS Version: 11.5
Portions Copyright (C) 2014-2018 NetApp, Inc. All Rights Reserved.

Initializing System Memory ...
Loading Device Drivers ...
Configuring Devices ...

CPU = 1 Processor(s) Detected.
Intel(R) Xeon(R) CPU D-1557 @ 1.50GHz (CPU 0)
CPUID: 0x00050664. Cores per Processor = 12
32768 MB System RAM Installed.
SATA (AHCI) Device: SV9MST6D120GLM41NP

SpindleNinja · ‎2020-10-20

What are the external shelves and drives? What's internal?

I have a feeling that it's doing something screwy with RAID TEC and the internal shelves when it tries to create the roots.

it's been talked about before - https://community.netapp.com/t5/FAS-and-V-Series-Storage-Systems-Discussions/FAS-2720-new-aggregate-not-create/td-p/145463

Try maybe doing the re-init without the attached shelves?

Halt

detatch external shelves.

Park Controller 2 at LOADER>
Boot Controller 1 in to the boot menu and do opt 9
Run opt 9a
Run opt 9b

Wait for the ….. to finish on Controller 1
Boot Controller2 in to the boot menu
Run opt 9a
Run opt 9b
Wait for Controller 2 to finish.
Create Cluster

Divine · ‎2020-10-21

We will try this option and feedback. thank you.

SpindleNinja · ‎2020-10-21

no problem. This is the BURT I was thinking about. https://mysupport.netapp.com/site/bugs-online/product/ONTAP/BURT/948840

You should be patched, but.. worth trying the bootarg.raid.allow-raid-tec work around.

if that still doesn't work, open a ticket at this point IMHO.

Divine · ‎2020-10-21

this option worked. Both controllers are up and can see the attached disks. thank you.🙏🙏🙏

Call home for DISK CONFIGURATION ERROR

Get ready to power on