Insufficient spare disks 7-mode

[Geoff_T1]

Help!

I have inherited FAS240-2 and On Command manager is reporting there are insufficient spare disks in the system, the filer is fully loaded with 450GB drives and there is a disk shelf with 2TB drives. Sysconfig -r  is showing 2 spares  available (2TB) so i assume the system is complaining there are no 450GB drives available for spares?

Not sure what my options are here!

snrmnetap1> sysconfig -r
Aggregate aggr0 (online, raid_dp) (block checksums)
Plex /aggr0/plex0 (online, normal, active, pool0)
RAID group /aggr0/plex0/rg0 (normal, block checksums)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks ) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- ------------- - --------------
dparity 0a.00.3 0a 0 3 SA:A 0 SAS 10000 857000/175513 6000 858483/1758174768
parity 0a.00.23 0a 0 23 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
data 0a.00.4 0a 0 4 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
data 0a.00.1 0a 0 1 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
data 0a.00.2 0a 0 2 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
data 0a.00.8 0a 0 8 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
data 0a.00.9 0a 0 9 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
data 0a.00.10 0a 0 10 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
data 0a.00.11 0a 0 11 SA:A 0 SAS 10000 418000/856064 000 429247/879097968 (prefail)
data 0a.00.12 0a 0 12 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
data 0a.00.13 0a 0 13 SA:A 0 SAS 10000 418000/856064 000 429247/879097968

RAID group /aggr0/plex0/rg1 (normal, block checksums)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks ) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- ------------- - --------------
dparity 0a.00.22 0a 0 22 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
parity 0a.00.21 0a 0 21 SA:A 0 SAS 10000 418000/856064 000 420156/860480768
data 0a.00.5 0a 0 5 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
data 0a.00.14 0a 0 14 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
data 0a.00.15 0a 0 15 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
data 0a.00.16 0a 0 16 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
data 0a.00.17 0a 0 17 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
data 0a.00.18 0a 0 18 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
data 0a.00.19 0a 0 19 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
data 0a.00.20 0a 0 20 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
data 0a.00.6 0a 0 6 SA:A 0 SAS 10000 418000/856064 000 429247/879097968

Pool1 spare disks (empty)

Pool0 spare disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks ) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- ------------- - --------------
Spare disks for block checksum
spare 0b.01.8 0b 1 8 SA:B 0 BSAS 7200 1695466/34723 15904 1695759/3472914816
spare 0b.01.12 0b 1 12 SA:B 0 BSAS 7200 1695466/34723 15904 1695759/3472914816

Partner disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks ) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- ------------- - --------------
partner 0b.01.15 0b 1 15 SA:B 0 BSAS 7200 0/0 1695759/3472914816
partner 0b.01.14 0b 1 14 SA:B 0 BSAS 7200 0/0 1695759/3472914816
partner 0b.01.13 0b 1 13 SA:B 0 BSAS 7200 0/0 1695759/3472914816
partner 0a.00.7 0a 0 7 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
partner 0a.00.0 0a 0 0 SA:A 0 SAS 10000 418000/856064 000 429247/879097968
partner 0b.01.3 0b 1 3 SA:B 0 BSAS 7200 0/0 1695759/3472914816
partner 0b.01.0 0b 1 0 SA:B 0 BSAS 7200 0/0 1695759/3472914816
partner 0b.01.7 0b 1 7 SA:B 0 BSAS 7200 0/0 1695759/3472914816
partner 0b.01.5 0b 1 5 SA:B 0 BSAS 7200 0/0 1695759/3472914816
partner 0b.01.10 0b 1 10 SA:B 0 BSAS 7200 0/0 1695759/3472914816
partner 0b.01.2 0b 1 2 SA:B 0 BSAS 7200 0/0 1695759/3472914816
partner 0b.01.4 0b 1 4 SA:B 0 BSAS 7200 0/0 1695759/3472914816
partner 0b.01.1 0b 1 1 SA:B 0 BSAS 7200 0/0 1695759/3472914816
partner 0b.01.9 0b 1 9 SA:B 0 BSAS 7200 0/0 1695759/3472914816
partner 0b.01.6 0b 1 6 SA:B 0 BSAS 7200 0/0 1695759/3472914816
snrmnetap1> Fri Jul 1 10:55:10 BST [snrmnetap1:disk.ioFailed:error]: I/O operation failed despite several retries.

[RossC]

Hi @Geoff_T1 

Is the system reporting any failed disks (even on partner node)? 

You may want to dive through this KB article which walks through how sparing works for 7mode based systems, https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/How_does_Data_ONTAP_select_spares_for_aggregate_creation%2C_aggregate_... 

[Geoff_T1]

I had a more detailed look at this over the weekend and i realised what the issue was, The FAS has 2 controllers and the 2 spare disks were assigned to controller 2 hence why controller  1 was complaining that there are no spares available. i did quick disk reassign and all was good.

[RossC]

No worries @Geoff_T1 . Glad to hear you figured it out.