topic Re: How to enable FIPS on clean reinstall AFF300 in AFF

How to enable FIPS on clean reinstall AFF300

marcinmf — Mon, 03 Feb 2025 23:21:05 GMT

Hello,

I have second hand AFF300 with 2 shelves 48 drives all all self encrypting SSDs.

The previous ontap was 9.11.1 I uploaded 9.15.1P7 and trying to install from scratch.

There is no data to be saves and I do have license keys to the controllers.

When going to special boot option 5 i see error message

[NODENAME-01:disk.encryptNoSupport:ALERT]: Detected FIPS-certified encrypting drive 0d.02.0, but FIPS drives are not supported on this node. 48 of 48 disks checked are FIPS-certified.

Obviously the FIPS are not supported on the node. How do I enable this?

There need to be some argument to set so when reboot the disk will be available.

Right now disks are marked as failed:

[NODENAME-01:diskown.errorReadingOwnership:notice]: error 3 ( disk failed) while reading ownership on disk 0a.01.22 (S/N 9620AXXXXXXX)

Any Ideas?

Re: How to enable FIPS on clean reinstall AFF300

TMACMD — Mon, 03 Feb 2025 23:40:59 GMT

a couple things here

first FIPS mode won’t help. To enable

set advanced

security config modify -interface ssl -FIPS true

it sounds like you grabbed the wrong ONTAP version

on the download page the first is for encryption enabled ONTAP. The second is for non-export countries that are not allowed to have encryption.

i think you downloaded the wrong one my friend.
try downloading again but the correct version. You may have to use the cli to push it. Have not tried in a long time (from non-enc version to enc version)

the next issue: were the drives previously encrypted? If they were and you reinitialized the system did you check if the drives had the encryption key removed?

you really won’t be able to do much until you get the correct code anyway

and if the drives are encrypted and have not been “opened” (meaning the drive is in encrypted mode) you are going to likely need to wipe again anyway

easiest would be to do an option 9a on both nodes, then on one node go into maintenance mode and then sanitize the SSDs there. They can all be done at the same time and it’s pretty instant

Re: How to enable FIPS on clean reinstall AFF300

marcinmf — Tue, 04 Feb 2025 00:10:14 GMT

Thank you for quick reply.

I can not really do set advanced because there is no OS installed.

I have access to Loader > and boot_ontap menu

Option 9a does not really do anything since getting message that there are no disks available to the controller.

I will try to reverse to 9.11, this is what the system came with anyway.

The drives were previously encrypted. The system can not read them, getting error and fails them.

iskown.errorReadingOwnership:notice]: error 3 ( disk failed) while reading ownership

I am ok to erase all data from them anyway.

Re: How to enable FIPS on clean reinstall AFF300

marcinmf — Tue, 04 Feb 2025 00:19:12 GMT

When booting to maintenance more i get this message. Does this mean that this version is FIPS enabled?

cryptomod_fips: Cryptomod FIPS version: Cryptomod FIPS 3.0
cryptomod_fips: Executing Crypto FIPS Self Tests.
cryptomod_fips: Crypto FIPS self-test: 'CPU COMPATIBILITY' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 ECB, AES-256 ECB' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 CBC, AES-256 CBC' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 GCM, AES-256 GCM' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 CCM' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128, AES-256 CMAC' passed.
cryptomod_fips: Crypto FIPS self-test: 'CTR_DRBG' passed.
cryptomod_fips: Crypto FIPS self-test: 'KDF' passed.
cryptomod_fips: Crypto FIPS self-test: 'SHA1, SHA256, SHA512' passed.
cryptomod_fips: Crypto FIPS self-test: 'SHA3-256' passed.
cryptomod_fips: Crypto FIPS self-test: 'HMAC-SHA1, HMAC-SHA256, HMAC-SHA512' passed.
cryptomod_fips: Crypto FIPS self-test: 'PBKDF2' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-XTS 128, AES-XTS 256' passed.
cryptomod_fips: Crypto FIPS self-test: 'Self-integrity' passed.
Feb 04 00:01:58 [CONTROLLER-01:raid.autoPart.disabled:ALERT]: Disk auto-partitioning is disabled on this system: the system needs a minimum of 8 usable internal hard disks.
Feb 04 00:01:58 [CONTROLLER-01:callhome.raid.adp.disabled:notice]: Disk auto-partitioning is disabled on this system: ADP DISABLED.

Re: How to enable FIPS on clean reinstall AFF300

TMACMD — Tue, 04 Feb 2025 00:19:16 GMT

Try this

boot both systems to the maintenance menu

then choose option 7

then it will ask for an interface, say e0M. When asked to reboot say n then choose option 7 again

define the ip and then specify a location to grab the correct code from. let the node reboot.

let us know

Re: How to enable FIPS on clean reinstall AFF300

TMACMD — Tue, 04 Feb 2025 00:29:55 GMT

Sounds like you will need to sanitize the disks. When I get back to my laptop I can send help. But it sounds like before the clearing , the encryption key was not removed. You’re stuck until the disks are cleared

Re: How to enable FIPS on clean reinstall AFF300

andris — Tue, 04 Feb 2025 17:58:09 GMT

To check you have the right ONTAP image that supports encryption..
https://kb.netapp.com/on-prem/ontap/DM/Encryption/Encryption-KBs/How_to_determine_if_the_running_ONTAP_version_supports_NetApp_Volume_Encryption_NVE

If the FIPS/NSE drives were not set back to "open"/factory MSID 0x0 before decommissioning, they are probably still locked with authentication keys. Check out these articles:

How to return SED to factory-configured settings after FIPS authentication key is lost
How to identify the PSID on a FIPs capable drive

Re: How to enable FIPS on clean reinstall AFF300

andris — Tue, 04 Feb 2025 18:00:37 GMT

But your 1st step is to set the environment variable for FIPS/NSE drives.
See: All disk failed after reinstall ONTAP

Re: How to enable FIPS on clean reinstall AFF300

TMACMD — Tue, 04 Feb 2025 18:01:26 GMT

@andris Those will not help in hist case. I am digging out what he needs. He needs to boot to maintenance mode and sanitize there. I do noth think you can do those commands in the KB without the cluster being active. give me a few minutes

Re: How to enable FIPS on clean reinstall AFF300

andris — Tue, 04 Feb 2025 18:07:32 GMT

I don't believe Sanitize is even possible if you don't have the AK.
The 1st KB link has the command while in Maintenance mode.

If this is a new setup or a repurposed system with no data on it, then the process is to boot (1) node to the maintenance mode then run the command from there:

*> disk encrypt revert_original <psid> <disk>

Re: How to enable FIPS on clean reinstall AFF300

marcinmf — Tue, 04 Feb 2025 18:09:03 GMT

There is some progress, i was able to set in loader.

setenv bootarg.storageencryption.support true

I also went back to 9.11.1 since using X365A drives in DS224C. i think those drives do not go pass 9.11.1

At least i see the drives in maintenance more. Still need to sanitize them and clear encryption.

Re: How to enable FIPS on clean reinstall AFF300

TMACMD — Tue, 04 Feb 2025 18:11:50 GMT

at the loader on both:

setenv bootarg.storageencryption.support true

printenv bootarg.storageencryption.support -. make sure this is true!

see these:

https://kb.netapp.com/on-prem/ontap/Ontap_OS/OS-KBs/How_long_does_disk_encrypt_sanitize_all_take

https://kb.netapp.com/on-prem/ontap/Ontap_OS/OS-KBs/disk_encrypt_show_hangs_after_sanitize

Get both nodes to loader.

BOOT ONE NODE ONLY into maintenance mode

run

disk encrypt sanitize -all -> pay attention to messages! this should run nearly instantly.

If you try to run "disk encrypt show" you node will hang....DO NOT DO IT

reboot the node into maintenance mode and then run

disk encrypt show

All disks should be unlocked. Now, halt the node. and do this on both nodes

loader: set-defaults
loader: saveenv
loader: printenv bootarg.storageencryption.support
- If false or undefined, set to true
- setenv bootarg.storageencryption.support true
loader: boot_ontap menu
At the boot menu -> 9
- On one node only: 9a
  - Affirm prompts and let it run. wait until prompt returns
- On second node only: 9a
  - Affirm prompts and let it run. wait until prompt returns
- On one node only: 9b
  - Affirm prompts and let it run. wait until ONTAP license appears
- On second node: 9b
  - Affirm prompts and let it run.
  - When the second node gets to the license, setup the cluster

Re: How to enable FIPS on clean reinstall AFF300

TMACMD — Tue, 04 Feb 2025 18:12:44 GMT

Please reboot into 9.15! I think there were issues with 9.11 and sanitize

Re: How to enable FIPS on clean reinstall AFF300

TMACMD — Tue, 04 Feb 2025 18:14:01 GMT

FYI, I have done this recently. It works.

Boot into maint, run the command, reboot

Re: How to enable FIPS on clean reinstall AFF300

TMACMD — Tue, 04 Feb 2025 18:14:28 GMT

That dis not work when I tried it. need to sanitize. see below

Re: How to enable FIPS on clean reinstall AFF300

TMACMD — Tue, 04 Feb 2025 18:20:58 GMT

And it looks like you should top out at the latest ONTAP 9.12.1P version. for a full re-init:

from Loader
- set-defaults
- setenv bootarg.storageencryption.support true
- saveenv
- ifconfig e0M -addr=192.168.100.10 -mask=255.255.255.0 -gw=192.168.100.1
- netboot http://9121P16_q_image.tgz
This will automatically go to the special boot menu. When there, choose
- option 7 (install new software)
- Specify the same URL (the e0M will maintain the IP you gave it)
- When prompted about recovery, say no!
- when prompted to reboot, say y
Repeat on other node (or run simultaneously with a different IP)
Both will automatically reboot to the Menu at which point, you will need to do the 9/9a/9b process!

Re: How to enable FIPS on clean reinstall AFF300

TMACMD — Tue, 04 Feb 2025 18:22:22 GMT

Those drives are EOA(31-Dec-2017) / EOS (31-Jan-2023). Latest supported ONTAP 9.12.1P16 (or current)

Re: How to enable FIPS on clean reinstall AFF300

marcinmf — Tue, 04 Feb 2025 18:24:36 GMT

9.12.1P is based on the drive type?

X365_TPM3V NA04 1.6tb SSD

the AFF300 ends at 9.16.

Would you say do not even try 9.15 ?

Re: How to enable FIPS on clean reinstall AFF300

marcinmf — Wed, 05 Feb 2025 15:01:39 GMT

I went with disk encrypt sanitize -all and it is running for 16 hours so far. 48 drives 1.6tb each drive.

It is still running, is this normal?

*> disk sanitize status
ERROR: Failed to recognize disks: No disks to read.
Feb 05 14:46:15 [localhost:raid.assim.tree.noRootVol:error]: No usable root volume was found!
. Still continuing...

Martin

Re: How to enable FIPS on clean reinstall AFF300

marcinmf — Fri, 07 Feb 2025 00:51:44 GMT

I had 2 problems.

1st. setenv bootarg.storageencryption.support was set to false.

Had to set it to true on both nodes.

2nd. Ontap os version. Because of the type of drives 9.12 was the last supported version.

After changing settings and os version everything else was standard installation.

Thank you @TMACMD and @andris for valuable comments.

Martin.