ONTAP-9.7- Remove "online" aggregate - disk are still attached to this aggregate

[Baptiste12]

Hi all,

I wanted to reuse a shelf and connect it to my ONTAP (9.7) cluster. The shelf was mounted successfully. The disks are attached to an old aggregate(aggr_DATA_8T) that has been deleted in my cluster (P1)."

But this old aggregate is still in the CLI configuration and not in the webpage (P2&P3)

My goal is to remove this association and attach the disks to my new aggregate.

When i'm trying to remove with "storage aggregate remove-stale-record -aggregate aggr_DATA_8T -nodename NETAPP-NXO-LAB-TLS-01" : i have :

 Failed to delete aggregate "aggr_DATA_8T". Reason: Aggregate 'aggr_DATA_8T' cannot be destroyed
because it is online. You will need to take it offline before destroying it.

So I decided to force it offline  with :  aggr offline -aggregate aggr_DATA_8T -force-offline true (P4)

But as you can see in P4 (entry doesn't exist)

If you have any ideas, I'm interested.

Regards

[Ontapforrum]

Step 1: Run the following cmd to check if the aggregate have foreign label:
::> system node run -node * -command sysconfig -r

Step 2: This will remove all foreign aggregates and return the drives as unzeroed spares.
::> set -privilege advanced
::*> storage aggregate remove-stale-record -nodename [node_name] -aggregate [foreign aggregate name]

KB: https://kb.netapp.com/onprem/ontap/hardware/How_to_remove_a_foreign_aggregate_(made_out_of_orphaned_disks)_in_ONTAP

If the steps above don't help, have a look at this related kb:
https://kb.netapp.com/onprem/ontap/hardware/How_to_remove_ownership_on_foreign_disks_in_clustered_Data_ONTAP

[Baptiste12]

Hi,

Step 1 : ::> system node run -node * -command sysconfig -r : 

--------------------------------------------

Aggregate aggr_DATA_8T (online, raid_dp, reconstruct, fast zeroed) (block checksums)
Plex /aggr_DATA_8T/plex0 (online, normal, active, pool0)
RAID group /aggr_DATA_8T/plex0/rg0 (reconstruction 5% completed, block checksums)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
dparity 0a.41.5 0a 41 5 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
parity 0a.41.6 0a 41 6 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
data 0a.41.9 0a 41 9 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
data 0a.41.10 0a 41 10 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264 (reconstruction 5% completed)
data 0a.41.0 0a 41 0 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
data 0a.41.7 0a 41 7 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
data 0a.41.11 0a 41 11 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
data 0a.41.1 0a 41 1 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
data 0a.41.2 0a 41 2 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
data 0a.41.4 0a 41 4 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
data 0a.41.8 0a 41 8 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264

-------------------------------------------------------------------------------------

I cannot force it offline.

[Ontapforrum]

The given aggr [Aggregate aggr_DATA_8T (online, raid_dp, reconstruct], is in reconstruction phase (5%), you cannot take it offline. Keep an eye on the progress. How much percentage is it showing now ?

[Baptiste12]

Now  :

Aggregate aggr_DATA_8T (online, raid_dp, fast zeroed) (block checksums)
Plex /aggr_DATA_8T/plex0 (online, normal, active, pool0)
RAID group /aggr_DATA_8T/plex0/rg0 (normal, block checksums)

Im trying to put offline : 

storage aggregate offline -aggregate aggr_DATA_8T

Error: command failed: Failed to check if aggr_DATA_8T is a foreign(unknown) aggregate. Reason: entry doesn't exist.

[Ontapforrum]

Have you tried the cmd I mentioned in my first reply, if not, please run it and let us know if it works.

::> set -privilege advanced
::*> storage aggregate remove-stale-record -nodename [node_name] -aggregate  aggr_DATA_8T

[Baptiste12]

Yes i did.

SRV-NETAPP::*> storage aggregate remove-stale-record -nodename NETAPP-NXO-LAB-TLS-02 -aggregate aggr_DATA_8T

Error: command failed: Failed to delete aggregate "aggr_DATA_8T". Reason: Aggregate 'aggr_DATA_8T' cannot be destroyed because it is online. You will need to take it offline before
destroying it.

[Ontapforrum]

Yes I just noticed, you had already tried this cmd in your first post, I see there the node name mentioned in the error says - NETAPP-NXO-LAB-TLS-01, but the command you just showed has 'NETAPP-NXO-LAB-TLS-02' ? 

what is the output:

::>system node show

::> aggr show

On both nodes

::> node run -node <node> -command sysconfig -r

[Baptiste12]

Yes because i tried to move aggr to the second node with (storage aggregate relocation start -node NETAPP-NXO-LAB-TLS-01 -destination NETAPP-NXO-LAB-TLS-02 -aggregate-list aggr_DATA_8T).

system node show
Node Health Eligibility Uptime Model Owner Location
--------- ------ ----------- ------------- ----------- -------- ---------------
NETAPP-NXO-LAB-TLS-01
true true 1 days 22:20 FAS2750 union
NETAPP-NXO-LAB-TLS-02
true true 1 days 22:30 FAS2750 union

---------------------------------------------

Aggregate Size Available Used% State #Vols Nodes RAID Status
--------- -------- --------- ----- ------- ------ ---------------- ------------
aggr0_uninetapp_01
159.9GB 7.75GB 95% online 1 NETAPP-NXO-LAB- raid_dp,
TLS-01 normal
aggr0_uninetapp_02
159.9GB 7.75GB 95% online 1 NETAPP-NXO-LAB- raid_dp,
TLS-02 normal
aggr_NXO_LAB_TLS
16.32TB 15.87TB 3% online 2 NETAPP-NXO-LAB- raid_dp,
TLS-01 normal

-------------------------------------------------------------------

node run -node NETAPP-NXO-LAB-TLS-01 -command sysconfig -r
Aggregate aggr_NXO_LAB_TLS (online, raid_dp, fast zeroed) (block checksums)
Plex /aggr_NXO_LAB_TLS/plex0 (online, normal, active, pool0)
RAID group /aggr_NXO_LAB_TLS/plex0/rg0 (normal, block checksums)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
dparity 0b.00.3P1 0b 0 3 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
parity 0b.00.7P1 0b 0 7 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.14P1 0b 0 14 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.15P1 0b 0 15 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.16P1 0b 0 16 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.20P1 0b 0 20 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.5P1 0b 0 5 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.9P1 0b 0 9 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.11P1 0b 0 11 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.13P1 0b 0 13 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.17P1 0b 0 17 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.21P1 0b 0 21 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.22P1 0b 0 22 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.23P1 0b 0 23 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.12P1 0b 0 12 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.0P1 0b 0 0 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)

RAID group /aggr_NXO_LAB_TLS/plex0/rg1 (normal, block checksums)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
dparity 0b.00.1P1 0b 0 1 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
parity 0b.00.4P1 0b 0 4 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.6P1 0b 0 6 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.8P1 0b 0 8 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
data 0b.00.10P1 0b 0 10 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)

Aggregate aggr0_uninetapp_01 (online, raid_dp) (block checksums)
Plex /aggr0_uninetapp_01/plex0 (online, normal, active, pool0)
RAID group /aggr0_uninetapp_01/plex0/rg0 (normal, block checksums)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
dparity 0b.00.0P2 0b 0 0 SA:A 0 SAS 10000 23956/49062912 23964/49079296
parity 0b.00.12P2 0b 0 12 SA:A 0 SAS 10000 23956/49062912 23964/49079296
data 0b.00.5P2 0b 0 5 SA:A 0 SAS 10000 23956/49062912 23964/49079296
data 0b.00.22P2 0b 0 22 SA:A 0 SAS 10000 23956/49062912 23964/49079296
data 0b.00.9P2 0b 0 9 SA:A 0 SAS 10000 23956/49062912 23964/49079296
data 0b.00.11P2 0b 0 11 SA:A 0 SAS 10000 23956/49062912 23964/49079296
data 0b.00.13P2 0b 0 13 SA:A 0 SAS 10000 23956/49062912 23964/49079296
data 0b.00.23P2 0b 0 23 SA:A 0 SAS 10000 23956/49062912 23964/49079296
data 0b.00.17P2 0b 0 17 SA:A 0 SAS 10000 23956/49062912 23964/49079296
data 0b.00.21P2 0b 0 21 SA:A 0 SAS 10000 23956/49062912 23964/49079296

Pool1 spare disks (empty)

Pool0 spare disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
Spare disks for block checksum
spare 0b.00.15P2 0b 0 15 SA:A 0 SAS 10000 23956/49062912 23964/49079296 (not zeroed)
spare 0b.00.2P1 0b 0 2 SA:A 0 SAS 10000 1118359/2290399744 1118367/2290416128 (not zeroed)
spare 0a.41.3 0a 41 3 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264 (not zeroed)

Partner disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
partner 0a.41.8 0a 41 8 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
partner 0a.41.4 0a 41 4 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
partner 0a.41.2 0a 41 2 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
partner 0a.41.1 0a 41 1 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
partner 0a.41.11 0a 41 11 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
partner 0a.41.7 0a 41 7 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
partner 0a.41.0 0a 41 0 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
partner 0a.41.10 0a 41 10 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
partner 0a.41.9 0a 41 9 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
partner 0a.41.6 0a 41 6 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264 (fast zeroed)
partner 0a.41.5 0a 41 5 SA:A 0 FSAS 7200 7486157/15331651072 7501160/15362376264
partner 0b.00.16P2 0b 0 16 SA:A 0 SAS 10000 0/0 23964/49079296
partner 0b.00.6P2 0b 0 6 SA:A 0 SAS 10000 0/0 23964/49079296
partner 0b.00.14P2 0b 0 14 SA:A 0 SAS 10000 0/0 23964/49079296
partner 0b.00.4P2 0b 0 4 SA:A 0 SAS 10000 0/0 23964/49079296
partner 0b.00.2P2 0b 0 2 SA:A 0 SAS 10000 0/0 23964/49079296
partner 0b.00.10P2 0b 0 10 SA:A 0 SAS 10000 0/0 23964/49079296
partner 0b.00.3P2 0b 0 3 SA:A 0 SAS 10000 0/0 23964/49079296
partner 0b.00.1P2 0b 0 1 SA:A 0 SAS 10000 0/0 23964/49079296
partner 0b.00.20P2 0b 0 20 SA:A 0 SAS 10000 0/0 23964/49079296
partner 0b.00.8P2 0b 0 8 SA:A 0 SAS 10000 0/0 23964/49079296
partner 0b.00.7P2 0b 0 7 SA:A 0 SAS 10000 0/0 23964/49079296

--------------------------------------------------------------------

node run -node NETAPP-NXO-LAB-TLS-02 -command sysconfig -r
Aggregate aggr0_uninetapp_02 (online, raid_dp) (block checksums)
Plex /aggr0_uninetapp_02/plex0 (online, normal, active, pool0)
RAID group /aggr0_uninetapp_02/plex0/rg0 (normal, block checksums)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
dparity 0b.00.7P2 0b 0 7 SA:B 0 SAS 10000 23956/49062912 23964/49079296
parity 0b.00.2P2 0b 0 2 SA:B 0 SAS 10000 23956/49062912 23964/49079296
data 0b.00.4P2 0b 0 4 SA:B 0 SAS 10000 23956/49062912 23964/49079296
data 0b.00.6P2 0b 0 6 SA:B 0 SAS 10000 23956/49062912 23964/49079296
data 0b.00.8P2 0b 0 8 SA:B 0 SAS 10000 23956/49062912 23964/49079296
data 0b.00.10P2 0b 0 10 SA:B 0 SAS 10000 23956/49062912 23964/49079296
data 0b.00.1P2 0b 0 1 SA:B 0 SAS 10000 23956/49062912 23964/49079296
data 0b.00.20P2 0b 0 20 SA:B 0 SAS 10000 23956/49062912 23964/49079296
data 0b.00.16P2 0b 0 16 SA:B 0 SAS 10000 23956/49062912 23964/49079296
data 0b.00.3P2 0b 0 3 SA:B 0 SAS 10000 23956/49062912 23964/49079296

Aggregate aggr_DATA_8T (online, raid_dp, fast zeroed) (block checksums)
Plex /aggr_DATA_8T/plex0 (online, normal, active, pool0)
RAID group /aggr_DATA_8T/plex0/rg0 (normal, block checksums)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
dparity 0a.41.5 0a 41 5 SA:B 0 FSAS 7200 7486157/15331651072 7501160/15362376264
parity 0a.41.6 0a 41 6 SA:B 0 FSAS 7200 7486157/15331651072 7501160/15362376264
data 0a.41.9 0a 41 9 SA:B 0 FSAS 7200 7486157/15331651072 7501160/15362376264
data 0a.41.10 0a 41 10 SA:B 0 FSAS 7200 7486157/15331651072 7501160/15362376264 (fast zeroed)
data 0a.41.0 0a 41 0 SA:B 0 FSAS 7200 7486157/15331651072 7501160/15362376264
data 0a.41.7 0a 41 7 SA:B 0 FSAS 7200 7486157/15331651072 7501160/15362376264
data 0a.41.11 0a 41 11 SA:B 0 FSAS 7200 7486157/15331651072 7501160/15362376264
data 0a.41.1 0a 41 1 SA:B 0 FSAS 7200 7486157/15331651072 7501160/15362376264
data 0a.41.2 0a 41 2 SA:B 0 FSAS 7200 7486157/15331651072 7501160/15362376264
data 0a.41.4 0a 41 4 SA:B 0 FSAS 7200 7486157/15331651072 7501160/15362376264
data 0a.41.8 0a 41 8 SA:B 0 FSAS 7200 7486157/15331651072 7501160/15362376264

Pool1 spare disks (empty)

Pool0 spare disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
Spare disks for block checksum
spare 0b.00.14P2 0b 0 14 SA:B 0 SAS 10000 23956/49062912 23964/49079296 (fast zeroed)

Partner disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
partner 0a.41.3 0a 41 3 SA:B 0 FSAS 7200 7486157/15331651072 7501160/15362376264
partner 0b.00.10P1 0b 0 10 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.8P1 0b 0 8 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.6P1 0b 0 6 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.4P1 0b 0 4 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.1P1 0b 0 1 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.0P1 0b 0 0 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.12P1 0b 0 12 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.23P1 0b 0 23 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.22P1 0b 0 22 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.21P1 0b 0 21 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.17P1 0b 0 17 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.13P1 0b 0 13 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.11P1 0b 0 11 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.9P1 0b 0 9 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.5P1 0b 0 5 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.20P1 0b 0 20 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.16P1 0b 0 16 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.15P1 0b 0 15 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.14P1 0b 0 14 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.7P1 0b 0 7 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.3P1 0b 0 3 SA:B 0 SAS 10000 1118359/2290399744 1118367/2290416128 (fast zeroed)
partner 0b.00.2P1 0b 0 2 SA:B 0 SAS 10000 0/0 1118367/2290416128
partner 0b.00.5P2 0b 0 5 SA:B 0 SAS 10000 0/0 23964/49079296
partner 0b.00.9P2 0b 0 9 SA:B 0 SAS 10000 0/0 23964/49079296
partner 0b.00.23P2 0b 0 23 SA:B 0 SAS 10000 0/0 23964/49079296
partner 0b.00.13P2 0b 0 13 SA:B 0 SAS 10000 0/0 23964/49079296
partner 0b.00.11P2 0b 0 11 SA:B 0 SAS 10000 0/0 23964/49079296
partner 0b.00.15P2 0b 0 15 SA:B 0 SAS 10000 0/0 23964/49079296
partner 0b.00.22P2 0b 0 22 SA:B 0 SAS 10000 0/0 23964/49079296
partner 0b.00.21P2 0b 0 21 SA:B 0 SAS 10000 0/0 23964/49079296
partner 0b.00.0P2 0b 0 0 SA:B 0 SAS 10000 0/0 23964/49079296
partner 0b.00.12P2 0b 0 12 SA:B 0 SAS 10000 0/0 23964/49079296
partner 0b.00.17P2 0b 0 17 SA:B 0 SAS 10000 0/0 23964/49079296

[Ontapforrum]

Does the disk belonging to the 'aggr_DATA_8T '  show - old owner ?

::> system node run -node <node name> -command "disk show -a"

Also,  does it show any volumes ?

::> vol show -aggr aggr_DATA_8T

[Baptiste12]

No old owner and no volume 😕

[Ontapforrum]

https://kb.netapp.com/onprem/ontap/hardware/How_to_remove_ownership_on_foreign_disks_in_clustered_Data_ONTAP

If the issue persists, return the shelf to the original storage system and clear the ownership (make them unowned). Then migrate the shelf to the new environment.