Re: I have a sparing problem where a 1TB SATA spare will not spare out a failed 750GB SATA disk on a...

rdunkerson · ‎2012-01-28

I have a sparing problem to where a 1TB SATA spare will not spare out a failed 750GB SATA disk on a FAS3040. All firmware is up to date on the disks so I am at a lost considering the rules for spares says a larger disk should spare into the aggregate.

Spare disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

spare 1d.18 1d 1 2 FC:B - ATA 7200 847555/1735794176 847827/1736350304

aborzenkov · ‎2012-01-28

Please show aggr status -r output.

rdunkerson · ‎2012-01-29

netapp3040b> aggr status -r

Aggregate t3_aggr1 (online, raid_dp) (block checksums)

Plex /t3_aggr1/plex0 (online, normal, active)

RAID group /t3_aggr1/plex0/rg0 (normal)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

dparity 2d.16 2d 1 0 FC:A - ATA 5400 274400/561971200 274540/562258784

parity 2b.32 2b 2 0 FC:B - ATA 7200 274400/561971200 274540/562258784

data 2d.48 2d 3 0 FC:A - ATA 5400 274400/561971200 274540/562258784

data 2d.64 2d 4 0 FC:A - ATA 5400 274400/561971200 274540/562258784

data 2d.17 2d 1 1 FC:A - ATA 7200 274400/561971200 274540/562258784

data 2b.33 2b 2 1 FC:B - ATA 7200 274400/561971200 274540/562258784

data 2b.49 2b 3 1 FC:B - ATA 5400 274400/561971200 274540/562258784

data 2b.65 2b 4 1 FC:B - ATA 5400 274400/561971200 274540/562258784

data 2d.18 2d 1 2 FC:A - ATA 5400 274400/561971200 274540/562258784

data 2d.34 2d 2 2 FC:A - ATA 5400 274400/561971200 274540/562258784

data 2d.50 2d 3 2 FC:A - ATA 5400 274400/561971200 274540/562258784

data 2b.66 2b 4 2 FC:B - ATA 5400 274400/561971200 274540/562258784

data 2d.26 2d 1 10 FC:A - ATA 5400 274400/561971200 274540/562258784

data 2d.35 2d 2 3 FC:A - ATA 5400 274400/561971200 274540/562258784

RAID group /t3_aggr1/plex0/rg1 (normal)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

dparity 2d.51 2d 3 3 FC:A - ATA 7200 274400/561971200 274540/562258784

parity 2b.67 2b 4 3 FC:B - ATA 5400 274400/561971200 274540/562258784

data 2d.36 2d 2 4 FC:A - ATA 5400 274400/561971200 274540/562258784

data 2d.20 2d 1 4 FC:A - ATA 5400 274400/561971200 274540/562258784

data 2b.52 2b 3 4 FC:B - ATA 5400 274400/561971200 274540/562258784

data 2b.68 2b 4 4 FC:B - ATA 5400 274400/561971200 274540/562258784

data 2b.37 2b 2 5 FC:B - ATA 5400 274400/561971200 274540/562258784

data 2b.21 2b 1 5 FC:B - ATA 7200 274400/561971200 274540/562258784

data 2d.53 2d 3 5 FC:A - ATA 7200 274400/561971200 274540/562258784

data 2b.69 2b 4 5 FC:B - ATA 7200 274400/561971200 274540/562258784

data 2b.38 2b 2 6 FC:B - ATA 7200 274400/561971200 274540/562258784

data 2d.22 2d 1 6 FC:A - ATA 5400 274400/561971200 274540/562258784

data 2d.54 2d 3 6 FC:A - ATA 7200 274400/561971200 274540/562258784

data 2b.70 2b 4 6 FC:B - ATA 5400 274400/561971200 274540/562258784

RAID group /t3_aggr1/plex0/rg2 (normal)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

dparity 2b.39 2b 2 7 FC:B - ATA 7200 274400/561971200 274540/562258784

parity 2d.71 2d 4 7 FC:A - ATA 5400 274400/561971200 274540/562258784

data 2d.23 2d 1 7 FC:A - ATA 7200 274400/561971200 274540/562258784

data 2b.55 2b 3 7 FC:B - ATA 5400 274400/561971200 274540/562258784

data 2b.40 2b 2 8 FC:B - ATA 7200 274400/561971200 274540/562258784

data 2b.72 2b 4 8 FC:B - ATA 7200 274400/561971200 274540/562258784

data 2b.24 2b 1 8 FC:B - ATA 5400 274400/561971200 274540/562258784

data 2b.56 2b 3 8 FC:B - ATA 5400 274400/561971200 274540/562258784

data 2d.41 2d 2 9 FC:A - ATA 5400 274400/561971200 274540/562258784

data 2b.73 2b 4 9 FC:B - ATA 5400 274400/561971200 274540/562258784

data 2d.25 2d 1 9 FC:A - ATA 5400 274400/561971200 274540/562258784

data 2b.57 2b 3 9 FC:B - ATA 5400 274400/561971200 274540/562258784

data 2d.42 2d 2 10 FC:A - ATA 5400 274400/561971200 274540/562258784

data 2b.74 2b 4 10 FC:B - ATA 5400 274400/561971200 274540/562258784

Aggregate t3_aggr0 (online, raid_dp, degraded) (block checksums)

Plex /t3_aggr0/plex0 (online, normal, active)

RAID group /t3_aggr0/plex0/rg0 (normal)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

dparity 1b.32 1b 2 0 FC:A - ATA 7200 635555/1301618176 635858/1302238304

parity 1b.33 1b 2 1 FC:A - ATA 7200 635555/1301618176 635858/1302238304

data 1d.29 1d 1 13 FC:B - ATA 7200 635555/1301618176 635858/1302238304

data 1b.35 1b 2 3 FC:A - ATA 7200 635555/1301618176 635858/1302238304

data 1b.36 1b 2 4 FC:A - ATA 7200 635555/1301618176 635858/1302238304

data 1b.37 1b 2 5 FC:A - ATA 7200 635555/1301618176 635858/1302238304

data 1b.26 1b 1 10 FC:A - ATA 7200 635555/1301618176 635858/1302238304

data 1b.39 1b 2 7 FC:A - ATA 7200 635555/1301618176 635858/1302238304

data 1b.40 1b 2 8 FC:A - ATA 7200 635555/1301618176 635858/1302238304

data 1d.24 1d 1 8 FC:B - ATA 7200 635555/1301618176 635858/1302238304

data 1b.42 1b 2 10 FC:A - ATA 7200 635555/1301618176 635858/1302238304

data 1b.43 1b 2 11 FC:A - ATA 7200 635555/1301618176 635858/1302238304

data 1b.44 1b 2 12 FC:A - ATA 7200 635555/1301618176 635858/1302238304

data 1d.16 1d 1 0 FC:B - ATA 7200 635555/1301618176 635858/1302238304

RAID group /t3_aggr0/plex0/rg1 (degraded)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

dparity 1d.17 1d 1 1 FC:B - ATA 7200 635555/1301618176 635858/1302238304

parity 1b.45 1b 2 13 FC:A - ATA 7200 635555/1301618176 635858/1302238304

data 1b.38 1b 2 6 FC:A - ATA 7200 635555/1301618176 635858/1302238304

data 1d.19 1d 1 3 FC:B - ATA 7200 635555/1301618176 635858/1302238304

data 1d.28 1d 1 12 FC:B - ATA 7200 635555/1301618176 635858/1302238304

data 1d.21 1d 1 5 FC:B - ATA 7200 635555/1301618176 635858/1302238304

data 1d.22 1d 1 6 FC:B - ATA 7200 635555/1301618176 635858/1302238304

data FAILED N/A 635555/1301618176

data 1d.20 1d 1 4 FC:B - ATA 7200 635555/1301618176 635858/1302238304

data 1d.25 1d 1 9 FC:B - ATA 7200 635555/1301618176 635858/1302238304

data 1b.41 1b 2 9 FC:A - ATA 7200 635555/1301618176 635858/1302238304

data 1b.34 1b 2 2 FC:A - ATA 7200 635555/1301618176 847827/1736350304

Aggregate t3_aggr3 (online, raid_dp) (block checksums)

Plex /t3_aggr3/plex0 (online, normal, active)

RAID group /t3_aggr3/plex0/rg0 (normal)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

dparity 2d.96 2d 6 0 FC:A - ATA 7200 423111/866531584 423889/868126304

parity 2b.97 2b 6 1 FC:B - ATA 7200 423111/866531584 423889/868126304

data 2b.98 2b 6 2 FC:B - ATA 7200 423111/866531584 423889/868126304

data 2d.99 2d 6 3 FC:A - ATA 7200 423111/866531584 423889/868126304

data 2d.100 2d 6 4 FC:A - ATA 7200 423111/866531584 423889/868126304

data 2d.101 2d 6 5 FC:A - ATA 7200 423111/866531584 423889/868126304

data 2b.102 2b 6 6 FC:B - ATA 7200 423111/866531584 423889/868126304

data 2b.103 2b 6 7 FC:B - ATA 7200 423111/866531584 423889/868126304

data 2b.93 2b 5 13 FC:B - ATA 7200 423111/866531584 423889/868126304

data 2b.105 2b 6 9 FC:B - ATA 7200 423111/866531584 423889/868126304

data 2d.106 2d 6 10 FC:A - ATA 7200 423111/866531584 423889/868126304

data 2d.107 2d 6 11 FC:A - ATA 7200 423111/866531584 423889/868126304

data 2d.108 2d 6 12 FC:A - ATA 7200 423111/866531584 423889/868126304

data 2d.104 2d 6 8 FC:A - ATA 7200 423111/866531584 423889/868126304

RAID group /t3_aggr3/plex0/rg1 (normal)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

dparity 2d.80 2d 5 0 FC:A - ATA 7200 423111/866531584 423889/868126304

parity 2b.81 2b 5 1 FC:B - ATA 7200 423111/866531584 423889/868126304

data 2b.82 2b 5 2 FC:B - ATA 7200 423111/866531584 423889/868126304

data 2d.83 2d 5 3 FC:A - ATA 7200 423111/866531584 423889/868126304

data 2d.84 2d 5 4 FC:A - ATA 7200 423111/866531584 423889/868126304

data 2d.85 2d 5 5 FC:A - ATA 7200 423111/866531584 423889/868126304

data 2b.86 2b 5 6 FC:B - ATA 7200 423111/866531584 423889/868126304

data 2b.87 2b 5 7 FC:B - ATA 7200 423111/866531584 423889/868126304

data 2d.88 2d 5 8 FC:A - ATA 7200 423111/866531584 423889/868126304

data 2b.89 2b 5 9 FC:B - ATA 7200 423111/866531584 423889/868126304

data 2b.90 2b 5 10 FC:B - ATA 7200 423111/866531584 423889/868126304

data 2d.91 2d 5 11 FC:A - ATA 7200 423111/866531584 423889/868126304

data 2d.92 2d 5 12 FC:A - ATA 7200 423111/866531584 423889/868126304

Spare disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

Spare disks for block or zoned checksum traditional volumes or aggregates

spare 2b.44 2b 2 12 FC:B - ATA 5400 274400/561971200 274540/562258784

spare 2b.58 2b 3 10 FC:B - ATA 5400 274400/561971200 274540/562258784

spare 2b.59 2b 3 11 FC:B - ATA 5400 274400/561971200 274540/562258784

spare 2d.28 2d 1 12 FC:A - ATA 5400 274400/561971200 274540/562258784

spare 2d.29 2d 1 13 FC:A - ATA 5400 274400/561971200 274540/562258784

spare 2d.60 2d 3 12 FC:A - ATA 5400 274400/561971200 274540/562258784

spare 2d.75 2d 4 11 FC:A - ATA 5400 274400/561971200 274540/562258784

spare 2d.76 2d 4 12 FC:A - ATA 5400 274400/561971200 274540/562258784

spare 2d.77 2d 4 13 FC:A - ATA 5400 274400/561971200 274540/562258784

spare 2b.45 2b 2 13 FC:B - ATA 7200 274400/561971200 274540/562258784

spare 2d.27 2d 1 11 FC:A - ATA 7200 274400/561971200 274540/562258784

spare 2d.43 2d 2 11 FC:A - ATA 7200 274400/561971200 274540/562258784

spare 2d.61 2d 3 13 FC:A - ATA 7200 274400/561971200 274540/562258784

spare 2d.109 2d 6 13 FC:A - ATA 7200 423111/866531584 423889/868126304

spare 1d.18 1d 1 2 FC:B - ATA 7200 847555/1735794176 847827/1736350304

Broken disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

not responding 2b.19 2b 1 3 FC:B - ATA 5400 274400/561971200 274540/562258784

failed 1d.23 1d 1 7 FC:B - ATA 7200 635555/1301618176 635858/1302238304

not responding 1d.27 1d 1 11 FC:B - ATA 7200 635555/1301618176 635858/1302238304

aborzenkov · ‎2012-01-29

Yes, looks like it ignores suitable spare. Do you have any suspicious messages in /etc/messages about it?

rdunkerson · ‎2012-01-29

Just the error that the raid roup is degraded because it can not spare in a disk out of the spares pool

chrism · ‎2012-01-30

You can add the disk in manually by throwing the "-f" flag at the end of the command to force it. It looks like that aggregate was initially created with 750GB SATA drives which is possibly why it is fighting back. While OnTap should be throwing it in and downsizing the drive, that's probably where it's choking.

Also, with your mix and match SATA aggregates I would also make sure that "options raid.rpm.ata.enable" is set to on so it plays nicely as well.

aborzenkov · ‎2012-01-30

Just a wild guess - may be outdated disk qualification package.

rdunkerson · ‎2012-01-31

I have the latest devices_qual_v2 file out in etc

rdunkerson · ‎2012-01-31

"options raid.rpm.ata.enable" is on but now this is weird we rplaced to 750GB drives that were in the broken disks but I can not see those disks in the slots for the aggregate. There are 2 of 1TB drives in shelf 1 bays 7 and 12 but I do not see them when I issue a sysconfig -r and note I can see them from a sysconfig -v.

Aggregate t3_aggr0 (online, raid_dp, degraded) (block checksums)
Plex /t3_aggr0/plex0 (online, normal, active)
RAID group /t3_aggr0/plex0/rg0 (normal)

      RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks)    Phys (MB/blks)
      --------- ------ ------------- ---- ---- ---- ----- --------------    --------------
      dparity   1b.32   1b    2   0   FC:A   - ATA   7200 635555/1301618176 635858/1302238304
      parity    1b.33   1b    2   1   FC:A   - ATA   7200 635555/1301618176 635858/1302238304
      data      1d.29   1d    1   13 FC:B   - ATA   7200 635555/1301618176 635858/1302238304
      data      1b.35   1b    2   3   FC:A   - ATA   7200 635555/1301618176 635858/1302238304
      data      1b.36   1b    2   4   FC:A   - ATA   7200 635555/1301618176 635858/1302238304
      data      1b.37   1b    2   5   FC:A   - ATA   7200 635555/1301618176 635858/1302238304
      data      1b.26   1b    1   10 FC:A   - ATA   7200 635555/1301618176 635858/1302238304
      data      1b.39   1b    2   7   FC:A   - ATA   7200 635555/1301618176 635858/1302238304
      data      1b.40   1b    2   8   FC:A   - ATA   7200 635555/1301618176 635858/1302238304
      data      1d.24   1d    1   8   FC:B   - ATA   7200 635555/1301618176 635858/1302238304
      data      1b.42   1b    2   10 FC:A   - ATA   7200 635555/1301618176 635858/1302238304
      data      1b.43   1b    2   11 FC:A   - ATA   7200 635555/1301618176 635858/1302238304
      data      1b.44   1b    2   12 FC:A   - ATA   7200 635555/1301618176 635858/1302238304
      data      1d.16   1d    1   0   FC:B   - ATA   7200 635555/1301618176 635858/1302238304

RAID group /t3_aggr0/plex0/rg1 (degraded)

      RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks)    Phys (MB/blks)
      --------- ------ ------------- ---- ---- ---- ----- --------------    --------------
      dparity   1d.17   1d    1   1   FC:B   - ATA   7200 635555/1301618176 635858/1302238304
      parity    1b.45   1b    2   13 FC:A   - ATA   7200 635555/1301618176 635858/1302238304
      data      1b.38   1b    2   6   FC:A   - ATA   7200 635555/1301618176 635858/1302238304
      data      1d.19   1d    1   3   FC:B   - ATA   7200 635555/1301618176 635858/1302238304
      data      1b.18   1b    1   2   FC:A   - ATA   7200 635555/1301618176 847827/1736350304
      data      1d.21   1d    1   5   FC:B   - ATA   7200 635555/1301618176 635858/1302238304
      data      1d.22   1d    1   6   FC:B   - ATA   7200 635555/1301618176 635858/1302238304
      data      FAILED          N/A                   635555/1301618176                                                       This raid group is not showing the disks in bay 7 and 12 of
      data      1d.20   1d    1   4   FC:B   - ATA   7200 635555/1301618176 635858/1302238304            shelf 1
      data      1d.25   1d    1   9   FC:B   - ATA   7200 635555/1301618176 635858/1302238304
      data      1b.41   1b    2   9   FC:A   - ATA   7200 635555/1301618176 635858/1302238304
      data      1b.34   1b    2   2   FC:A   - ATA   7200 635555/1301618176 847827/1736350304

I have a sparing problem where a 1TB SATA spare will not spare out a failed 750GB SATA disk on a FAS3040