Re: Reverting SED/ NSE disk back to original state.

GermanHermon · ‎2019-10-02

Hello everyone,

I have some x315A drives that im trying to use. I know they're encrypted and was trying to figure out how to return them to their original state. So far I am not having any luck. This is not the orginal system they were in. I no longer have the FAS system with the Keys. Does this mean these drives are completely uses less ? If so let me know so I could stop wasting my time trying to figure this out. Any explaination will be helpful.

The disk were autoassigned using some weired format. I tried unfailing them = no luck. I tried reassigning them = no luck.

TestCluster::storage*> encryption disk revert-to-original-state -disk TestCluster-01:0b.01.3 -psid 37FUU7QLD00000000LVCX6V53WE8YW1Z

Info: Starting revert-to-original-state on 1 disk on node TestCluster-01.
      View the status of the operation by using the "storage encryption disk show-status" command.
Error: Could not send revert-to-original-state request to node TestCluster-01. Reason: resource limit exceeded

TestCluster::storage disk*> show
                     Usable           Disk    Container   Container
Disk                   Size Shelf Bay Type    Type        Name      Owner
---------------- ---------- ----- --- ------- ----------- --------- --------
1.0.0               546.9GB     0   0 SAS     shared      aggr0_TestCluster_02
                                                                    TestCluster-02
1.0.1               546.9GB     0   1 SAS     shared      aggr0_TestCluster_01
                                                                    TestCluster-01
1.0.2               546.9GB     0   2 SAS     shared      aggr0_TestCluster_02
                                                                    TestCluster-02
1.0.3               546.9GB     0   3 SAS     shared      aggr0_TestCluster_01
                                                                    TestCluster-01
1.0.4               546.9GB     0   4 SAS     shared      aggr0_TestCluster_02
                                                                    TestCluster-02
1.0.5               546.9GB     0   5 SAS     shared      aggr0_TestCluster_01
                                                                    TestCluster-01
1.0.6               546.9GB     0   6 SAS     shared      aggr0_TestCluster_02
                                                                    TestCluster-02
1.0.7               546.9GB     0   7 SAS     shared      aggr0_TestCluster_01
                                                                    TestCluster-01
1.0.8               546.9GB     0   8 SAS     shared      aggr0_TestCluster_02
                                                                    TestCluster-02
1.0.9               546.9GB     0   9 SAS     shared      aggr0_TestCluster_01
                                                                    TestCluster-01
1.0.10              546.9GB     0  10 SAS     shared      -         TestCluster-02
1.0.11              546.9GB     0  11 SAS     shared      -         TestCluster-01
TestCluster-01:0b.01.5    -     1   5 FSAS    broken      -         -
TestCluster-02:0b.01.1    -     1   1 FSAS    broken      -         -
TestCluster-02:0b.01.2    -     1   2 FSAS    broken      -         -
TestCluster-02:0b.01.3    -     1   3 FSAS    broken      -         -
TestCluster-02:0b.01.4    -     1   4 FSAS    broken      -         -
TestCluster-02:0b.01.6    -     1   6 FSAS    broken      -         -
TestCluster-02:0b.01.7    -     1   7 FSAS    broken      -         -
TestCluster-02:0b.01.8    -     1   8 FSAS    broken      -         -
TestCluster-02:0b.01.12   -     1  12 FSAS    broken      -         -
21 entries were displayed.

TestCluster::*> security config show
          Cluster                                              Cluster Security
Interface FIPS Mode  Supported Protocols Supported Ciphers     Config Ready
--------- ---------- ------------------- --------------------- ----------------
SSL       true       TLSv1.2, TLSv1.1    ALL:!LOW:!aNULL:!EXP: yes
                                         !eNULL:!3DES:!kDH:
                                         !kECDH

TestCluster::storage disk*> unfail TestCluster-01:0b.01.5

Warning: Failed disk "TestCluster-01:0b.01.5" may have aggregate labels and file system data present. In that case, this command will attempt to bring this disk back into
         the aggregate with which this disk had formerly been associated and preserve file system data. Are you sure you want to continue with disk unfail?
          {y|n}: y

Error: command failed: Failed to unfail the disk. Reason: Disk 0b.01.5 does not exist.

TestCluster::storage disk*>
TestCluster::storage disk*>
TestCluster::storage disk*> unfail 0b.01.5

Error: command failed: Disk "0b.01.5" does not exist.

TestCluster::storage disk*>

TestCluster-01> sysconfig -v
        NetApp Release 9.6P1: Thu Jul 18 22:31:23 EDT 2019
        System ID: 123025260 (TestCluster-01); partner ID: 123455246 (TestCluster-02)
        System Serial Number: 71234300021 (TestCluster-01)
        System Rev: D1
        System Storage Configuration: Multi-Path HA
        System ACP Connectivity: Additional Connectivity
        All-Flash Optimized: false
        Capacity Optimized: false
        Backplane Part Number: DS212
        Backplane Rev:
        Backplane Serial Number: 4591503044
        slot 0: System Board 1.7 GHz (System Board XIX D1)
                Model Name:         FAS2520
                Part Number:        111-01323
                Revision:           D1
                Serial Number:      031541000089
                BIOS version:       8.3.0
                Loader version:     4.3
                Processors:         4
                Processor type:     Intel(R) Xeon(R) CPU           C3528  @ 1.73GHz
                Memory Size:        18432 MB
                Memory Attributes:  Hoisting
                                    Normal ECC
                NVMEM Size:         1280 MB of Main Memory Used
                Controller:         A
        Service Processor           Status: Online
                Firmware Version:   2.9
                Mgmt MAC Address:   00:A0:98:8B:A9:39
                Ethernet Link:      up, 100Mb, full duplex, auto-neg complete
                Using DHCP:         no
        IPv4 configuration:
                IP Address:         3.112.169.151
                Netmask:            255.255.255.128
                Gateway:            3.112.169.129
        IPv6 configuration:         Disabled
        [Service Processor cached network information determined at Wed Oct  2 22:57:02 GMT 2019]
        slot 0: Internal 10/100/1000 Ethernet Switch
                Wrench:             auto-100tx-fd-up
                Locked-wrench:      auto-100tx-fd-up
                Device Type:        88E6176
        slot 0: Gigabit Ethernet Controller 82580
                e0a MAC Address:    00:a0:98:8b:a9:35 (auto-100tx-fd-up)
                e0b MAC Address:    00:a0:98:8b:a9:36 (auto-unknown-fd-down)
                e0M MAC Address:    00:a0:98:8b:a9:37 (auto-1000t-fd-up)
                e0P MAC Address:    00:a0:98:8b:a9:38 (auto-1000t-fd-up)
                Device Type:        150E
                Firmware Version:   3.22 0x80000101
        slot 0: 1/10 Gigabit Ethernet Controller IX2-T
                e0c MAC Address:    00:a0:98:8b:a9:3b (auto-unknown-fd-down)
                e0d MAC Address:    00:a0:98:8b:a9:3c (auto-10g_t-fd-up)
                e0e MAC Address:    00:a0:98:8b:a9:3d (auto-unknown-fd-down)
                e0f MAC Address:    00:a0:98:8b:a9:3e (auto-10g_t-fd-up)
                Device Type:        X540
                Firmware Version:   4.4-0
        slot 0: Interconnect HBA:   Mellanox IB MT25204
                Port Name:          ib0a
                GUID:               0x100000a0988ba935
                Base LID:           0x4
                Remote LID:         0x5
                Firmware rev:       1.0.800
                Hardware rev:       160
                Command rev:        1
                Interconnect Port:  1x
        slot 0: SAS Host Adapter 0a (PMC-Sierra PM8001 rev. C, SAS, <UP>)
                Firmware rev:       01.12.09.00
                Base WWN:           5:00a098:00659db:b0
                Phy State:          [0] Enabled, 6.0 Gb/s
                                    [1] Enabled, 6.0 Gb/s
                                    [2] Enabled, 6.0 Gb/s
                                    [3] Enabled, 6.0 Gb/s
                 ID     Vendor   Model            FW    Size
                00.0 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.1 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.2 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.3 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.4 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.5 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.6 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.7 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.8 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.9 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.10: NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.11: NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                01.0 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.1 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.2 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.3 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.4 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.5 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.6 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.7 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.8 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.12: NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                Shelf 0: DS2126  Firmware rev. IOM6E A: 0172  IOM6E B: 0172
                Shelf ?: unsupported DS4243, IOM3
        slot 0: SAS Host Adapter 0b (PMC-Sierra PM8001 rev. C, SAS, <UP>)
                Firmware rev:       01.12.09.00
                Base WWN:           5:00a098:00659db:b4
                Phy State:          [4] Enabled, 3.0 Gb/s
                                    [5] Enabled, 3.0 Gb/s
                                    [6] Enabled, 3.0 Gb/s
                                    [7] Enabled, 3.0 Gb/s
                QSFP Vendor:        Molex Inc.
                QSFP Part Number:   112-00177+A0
                QSFP Type:          Passive Copper 2m ID:01
                QSFP Serial Number: 326020990
                 ID     Vendor   Model            FW    Size
                00.0 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.1 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.2 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.3 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.4 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.5 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.6 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.7 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.8 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.9 : NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.10: NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                00.11: NETAPP   X487_SLTNG600A10 NA02 560.0GB (1147307696 520B/sect)
                01.0 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.1 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.2 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.3 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.4 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.5 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.6 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.7 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.8 : NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                01.12: NETAPP   X315_SMEGA04TA07 NA00 3807.8GB (7814037168 520B/sect) (Failed)
                Shelf 0: DS2126  Firmware rev. IOM6E A: 0172  IOM6E B: 0172
                Shelf ?: unsupported DS4243, IOM3
        slot 0: Intel USB EHCI Adapter u0a (0xdf301000)
                boot0   Micron Technology 0x655, class 0/0, rev 2.00/11.10, addr 2 1936MB 512B/sect (devproto=0x00)

SpindleNinja · ‎2019-10-02

It's more than just keys. X487_SLTNG600A10 (your SAS drives) are not encrypted disks, you can't mix non-encrypted and enctyped in the same HA pair.

This doens't look to good either: Shelf ?: unsupported DS4243, IOM3

GermanHermon · ‎2019-10-02

ah okay. So if I had all encrypted drives this will resolve my issue? I also swapped out the DS4243 for a DS4246.

SpindleNinja · ‎2019-10-02

Converting to and from encrypted disks is a disruptive process, there's a boot flag that needs to be set and you have to re-init the array.

looking at HWU (hwu.netapp.com). doesn't look like X315A-R6 / X315_SMEGA04TA07 is compatiable with the DS4243 and the FAS2520. It does show compatiable with the DS4246 and FAS2520.

As far as re-using the drives, you can revert them to the orgional state.

https://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.dot-cm-cmpr-930%2Fstorage__encryption__disk__revert-to-original-state.html

What type of system did these drives come from (shelf model / controller model / ontap version) ?

GermanHermon · ‎2019-10-03

Whats the Boot Flag?

I mentioned i switched out the DS4243 for DS4246. I already tried the revert-to-original-state but it didnt work. I posted the output of that command above.

I don't know what environment these drives were in.

SpindleNinja · ‎2019-10-03

setenv bootarg.storageencryption.support true is the flag. (note: I don't think it's changed in ONTAP 9 anyway, I can't find the docs to support that it has, this is usually set at the factory when a controller is ordered with encrypted drives. It's been a long while since i've had to convert one)

note: Setting that will make non-encrypted drives not work.

GLENYU5820 · ‎2019-11-04

I recently just did this exercise. Could you try to run below? I assume your psid was taken from the disk label. > encryption disk revert-to-original-state -disk 0b.01.3 -psid 37FUU7QLD00000000LVCX6V53WE8YW1Z

GLENYU5820 · ‎2019-11-04

If it does not work, try this encryption disk revert-to-original-state -disk 1.01.3 -psid 37FUU7QLD00000000LVCX6V53WE8YW1Z