How to enable FIPS on clean reinstall AFF300

[marcinmf]

Hello, 

I have second hand AFF300 with 2 shelves 48 drives all all self encrypting SSDs.

The previous ontap was 9.11.1 I uploaded 9.15.1P7 and trying to install from scratch.

There is no data to be saves and I do have license keys to the controllers.

When going to special boot option 5 i see error message

[NODENAME-01:disk.encryptNoSupport:ALERT]: Detected FIPS-certified encrypting drive 0d.02.0, but FIPS drives are not supported on this node. 48 of 48 disks checked are FIPS-certified.

Obviously the FIPS are not supported on the node. How do I enable this?

There need to be some argument to set so when reboot the disk will be available.

Right now disks are marked as failed:

[NODENAME-01:diskown.errorReadingOwnership:notice]: error 3 ( disk failed) while reading ownership on disk 0a.01.22 (S/N 9620AXXXXXXX)

Any Ideas?

[TMACMD]

a couple things here

 first FIPS mode won’t help. To enable

 set advanced

 security config modify -interface ssl -FIPS true

it sounds like you grabbed the wrong ONTAP version

 on the download page the first is for encryption enabled ONTAP. The second is for non-export countries that are not allowed to have encryption. 

i think you downloaded the wrong one my friend. 
try downloading again but the correct version. You may have to use the cli to push it. Have not tried in a long time (from non-enc version to enc version)

 the next issue: were the drives previously encrypted? If they were and you reinitialized the system did you check if the drives had the encryption key removed?

you really won’t be able to do much until you get the correct code anyway

 and if the drives are encrypted and have not been “opened” (meaning the drive is in encrypted mode) you are going to likely need to wipe again anyway

 easiest would be to do an option 9a on both nodes, then on one node go into maintenance mode and then sanitize the SSDs there. They can all be done at the same time and it’s pretty instant

[marcinmf]

Thank you for quick reply.

I can not really do set advanced because there is no OS installed.

I have access to Loader > and boot_ontap menu

Option 9a does not really do anything since getting message that there are no disks available to the controller. 

I will try to reverse to 9.11, this is what the system came with anyway.

The drives were previously encrypted. The system can not read them, getting error and fails them.

iskown.errorReadingOwnership:notice]: error 3 ( disk failed) while reading ownership

I am ok to erase all data from them anyway.

[marcinmf]

When booting to maintenance more i get this message. Does this mean that this version is FIPS enabled?

cryptomod_fips: Cryptomod FIPS version: Cryptomod FIPS 3.0
cryptomod_fips: Executing Crypto FIPS Self Tests.
cryptomod_fips: Crypto FIPS self-test: 'CPU COMPATIBILITY' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 ECB, AES-256 ECB' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 CBC, AES-256 CBC' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 GCM, AES-256 GCM' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 CCM' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128, AES-256 CMAC' passed.
cryptomod_fips: Crypto FIPS self-test: 'CTR_DRBG' passed.
cryptomod_fips: Crypto FIPS self-test: 'KDF' passed.
cryptomod_fips: Crypto FIPS self-test: 'SHA1, SHA256, SHA512' passed.
cryptomod_fips: Crypto FIPS self-test: 'SHA3-256' passed.
cryptomod_fips: Crypto FIPS self-test: 'HMAC-SHA1, HMAC-SHA256, HMAC-SHA512' passed.
cryptomod_fips: Crypto FIPS self-test: 'PBKDF2' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-XTS 128, AES-XTS 256' passed.
cryptomod_fips: Crypto FIPS self-test: 'Self-integrity' passed.
Feb 04 00:01:58 [CONTROLLER-01:raid.autoPart.disabled:ALERT]: Disk auto-partitioning is disabled on this system: the system needs a minimum of 8 usable internal hard disks.
Feb 04 00:01:58 [CONTROLLER-01:callhome.raid.adp.disabled:notice]: Disk auto-partitioning is disabled on this system: ADP DISABLED.

[TMACMD]

Try this

 boot both systems to the maintenance menu

 then choose option 7

 then it will ask for an interface, say e0M. When asked to reboot say n then choose option 7 again

 define the ip and then specify a location to grab the correct code from. let the node reboot. 

let us know

[TMACMD]

Sounds like you will need to sanitize the disks. When I get back to my laptop I can send help. But it sounds like before the clearing , the encryption key was not removed. You’re stuck until the disks are cleared

[andris]

To check you have the right ONTAP image that supports encryption..
https://kb.netapp.com/on-prem/ontap/DM/Encryption/Encryption-KBs/How_to_determine_if_the_running_ONTAP_version_supports_NetApp_Volume_Encryption_NVE

If the FIPS/NSE drives were not set back to "open"/factory MSID 0x0 before decommissioning, they are probably still locked with authentication keys. Check out these articles:

How to return SED to factory-configured settings after FIPS authentication key is lost
How to identify the PSID on a FIPs capable drive

[andris]

But your 1st step is to set the environment variable for FIPS/NSE drives.
See: All disk failed after reinstall ONTAP

[TMACMD]

@andris Those will not help in hist case. I am digging out what he needs. He needs to boot to maintenance mode and sanitize there. I do noth think you can do those commands in the KB without the cluster being active. give me a few minutes

[andris]

I don't believe Sanitize is even possible if you don't have the AK.
The 1st KB link has the command while in Maintenance mode.

If this is a new setup or a repurposed system with no data on it, then the process is to boot (1) node to the maintenance mode then run the command from there:

*> disk encrypt revert_original <psid> <disk>

[TMACMD]

That dis not work when I tried it. need to sanitize. see below

[marcinmf]

There is some progress, i was able to set in loader.

setenv bootarg.storageencryption.support true

I also went back to 9.11.1 since using X365A  drives  in DS224C. i think those drives do not go pass 9.11.1

At least i see the drives in maintenance more. Still need to sanitize them and clear encryption.

[TMACMD]

Those drives are EOA(31-Dec-2017) / EOS (31-Jan-2023). Latest supported ONTAP 9.12.1P16 (or current)

[TMACMD · Accepted solution]

at the loader on both:

setenv bootarg.storageencryption.support true

printenv bootarg.storageencryption.support -. make sure this is true!

see these:

https://kb.netapp.com/on-prem/ontap/Ontap_OS/OS-KBs/How_long_does_disk_encrypt_sanitize_all_take

https://kb.netapp.com/on-prem/ontap/Ontap_OS/OS-KBs/disk_encrypt_show_hangs_after_sanitize

Get both nodes to loader.

BOOT ONE NODE ONLY into maintenance mode

run

disk encrypt sanitize -all -> pay attention to messages! this should run nearly instantly.

If you try to run "disk encrypt show" you node will hang....DO NOT DO IT

reboot the node into maintenance mode and then run

disk encrypt show

All disks should be unlocked. Now, halt the node. and do this on both nodes

• loader: set-defaults
• loader: saveenv
• loader: printenv bootarg.storageencryption.support
  • If false or undefined, set to true
  • setenv bootarg.storageencryption.support true
• loader: boot_ontap menu
• At the boot menu -> 9
  • On one node only: 9a
    • Affirm prompts and let it run. wait until prompt returns
  • On second node only: 9a
    • Affirm prompts and let it run. wait until prompt returns
  • On one node only: 9b
    • Affirm prompts and let it run. wait until ONTAP license appears
  • On second node: 9b
    • Affirm prompts and let it run. 
    • When the second node gets to the license, setup the cluster

[marcinmf]

I went with disk encrypt sanitize -all  and it is running for 16 hours so far.  48 drives 1.6tb each drive.

It is still running, is this normal?

*> disk sanitize status
ERROR: Failed to recognize disks: No disks to read.
Feb 05 14:46:15 [localhost:raid.assim.tree.noRootVol:error]: No usable root volume was found!
. Still continuing...

Martin

[TMACMD]

Please reboot into 9.15! I think there were issues with 9.11 and sanitize

[TMACMD]

FYI, I have done this recently. It works.

Boot into maint, run the command, reboot

[TMACMD]

And it looks like you should top out at the latest ONTAP 9.12.1P version. for a full re-init:

• from Loader
  • set-defaults
  • setenv bootarg.storageencryption.support true
  • saveenv
  • ifconfig e0M -addr=192.168.100.10 -mask=255.255.255.0 -gw=192.168.100.1
  • netboot http://9121P16_q_image.tgz
• This will automatically go to the special boot menu. When there, choose
  • option 7 (install new software)
  • Specify the same URL (the e0M will maintain the IP you gave it)
  • When prompted about recovery, say no!
  • when prompted to reboot, say y
• Repeat on other node (or run simultaneously with a different IP)
• Both will automatically reboot to the Menu at which point, you will need to do the 9/9a/9b process!

[marcinmf]

 9.12.1P is based on the drive type? 

X365_TPM3V NA04 1.6tb SSD 

the AFF300 ends at 9.16.

Would you say do not even try 9.15 ?

[marcinmf]

I had 2 problems.

1st. setenv bootarg.storageencryption.support was set to false.

Had to set it to true on both nodes.

2nd. Ontap os version. Because of the type of drives 9.12 was the last supported version.

After changing settings and os version everything else was standard installation.

Thank you @TMACMD  and @andris for valuable comments.

Martin.