Solved: 7-mode disk usage

XavW · ‎2022-05-26

I have an old clustered FAS8040 in 7-mode I'm trying to understand which has a bunch of chassis and drives (mixture of SSD and 4TB SATA).

Doing a "disk show" lists all the disks which are all part of a Pool0 pool, some of which are managed by controller A, the other by controller B. I have a total of 132 disks.

a> disk show
  DISK       OWNER      POOL 
------------ -------    -----
3c.00.9      b          Pool0
3c.00.21     b          Pool0
3c.00.7      b          Pool0
3c.00.3      a          Pool0
3c.00.5      a          Pool0
3c.00.15     a          Pool0
3c.00.19     b          Pool0
3c.00.11     b          Pool0
3c.00.1      a          Pool0
3c.00.23     b          Pool0
3c.00.13     a          Pool0
3c.00.17     a          Pool0
3c.00.0      a          Pool0
3c.00.8      b          Pool0
3c.00.2      a          Pool0
3c.00.4      a          Pool0
3c.00.22     b          FAILED
3c.00.12     a          Pool0
3c.00.14     a          Pool0
3c.00.18     b          Pool0
3c.00.20     b          Pool0
3c.00.16     a          Pool0
3c.00.10     b          Pool0
0d.12.11     a          Pool0
0d.12.0      a          Pool0
3d.10.2      b          Pool0
3d.11.7      b          Pool0
3d.10.1      b          Pool0
3d.10.8      b          Pool0
3d.10.21     a          Pool0
3d.10.14     a          Pool0
3d.10.18     a          Pool0
3d.10.4      b          Pool0
3d.10.5      b          Pool0
3d.11.2      b          Pool0
3d.10.13     a          Pool0
3d.11.4      b          Pool0
3d.11.5      b          Pool0
3d.11.8      b          Pool0
3d.11.3      b          Pool0
3d.10.17     a          Pool0
3d.10.19     a          Pool0
3d.10.7      b          Pool0
3d.10.3      b          Pool0
3d.10.12     a          Pool0
3d.10.10     b          Pool0
3d.11.9      b          Pool0
3d.11.0      b          Pool0
3d.10.6      b          Pool0
3d.10.23     a          Pool0
3d.11.11     b          Pool0
3d.10.16     a          Pool0
3d.10.15     a          Pool0
3d.11.6      b          Pool0
3d.11.1      b          Pool0
3c.00.6      b          Pool0
3d.10.20     a          Pool0
3d.10.9      b          Pool0
3d.10.22     a          Pool0
3d.10.11     b          Pool0
3d.11.10     b          Pool0
0d.12.5      a          Pool0
0d.12.12     a          Pool0
0d.12.3      a          Pool0
0d.12.2      a          Pool0
0d.12.20     b          Pool0
0d.12.10     a          Pool0
0d.12.21     b          Pool0
0d.12.23     b          Pool0
0d.12.8      a          Pool0
0d.12.17     a          Pool0
0d.12.22     b          Pool0
0d.12.9      a          Pool0
0d.12.15     a          Pool0
0d.12.4      a          Pool0
0d.12.6      a          Pool0
0d.12.7      a          Pool0
0d.12.1      a          Pool0
0d.12.14     a          Pool0
0d.12.13     a          Pool0
0d.12.19     b          Pool0
0d.12.16     a          Pool0
0d.12.18     b          Pool0
0d.13.7      a          Pool0
0d.13.18     b          Pool0
0d.13.20     b          Pool0
0d.13.13     b          Pool0
0d.13.4      a          Pool0
0d.13.17     b          Pool0
0d.13.15     b          Pool0
0d.13.9      a          Pool0
0d.13.22     b          Pool0
0d.13.0      a          Pool0
0d.13.2      a          Pool0
0d.13.11     a          Pool0
0d.13.16     b          Pool0
0d.13.12     b          Pool0
0d.13.1      a          Pool0
0d.13.23     b          Pool0
0d.13.14     b          Pool0
0d.13.19     b          Pool0
0d.13.8      a          Pool0
0d.13.5      a          Pool0
0d.13.6      a          Pool0
0d.13.10     a          Pool0
0d.13.3      a          Pool0
0d.13.21     b          Pool0
0a.01.19     a          Pool0
0a.01.18     a          Pool0
0a.01.5      a          Pool0
0a.01.22     a          Pool0
0a.01.23     a          Pool0
0a.01.20     a          Pool0
0a.01.9      a          Pool0
0a.01.16     a          Pool0
0a.01.12     a          Pool0
0a.01.17     a          Pool0
3d.10.0      a          Pool0
0a.01.3      a          Pool0
0a.01.6      b          Pool0
0a.01.10     b          Pool0
0a.01.14     b          Pool0
0a.01.11     b          Pool0
0a.01.0      b          Pool0
0a.01.8      b          Pool0
0a.01.15     b          Pool0
0a.01.1      b          Pool0
0a.01.13     b          Pool0
0a.01.2      b          Pool0
0a.01.4      b          Pool0
0a.01.7      b          Pool0
0a.01.21     a          Pool0

When I do an "aggr status -r aggr0" on each controller, I can see the usage of these disks in the aggregate and how they are broken up into different raid groups.

a> aggr status -r aggr0
Aggregate aggr0 (online, mixed_raid_type, hybrid) (block checksums)
  Plex /aggr0/plex0 (online, normal, active)
    RAID group /aggr0/plex0/rg0 (normal, block checksums, raid_dp)

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
      --------- ------          ------------- ---- ---- ---- ----- --------------    --------------
      dparity   0d.12.0         0d    12  0   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      parity    0d.13.0         0d    13  0   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.1         0d    12  1   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.1         0d    13  1   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.2         0d    13  2   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.3         0d    13  3   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.4         0d    13  4   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.5         0d    13  5   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.6         0d    13  6   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.7         0d    13  7   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.8         0d    13  8   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.9         0d    13  9   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.10        0d    13  10  SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.2         0d    12  2   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.3         0d    12  3   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.4         0d    12  4   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.5         0d    12  5   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.6         0d    12  6   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.7         0d    12  7   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.8         0d    12  8   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168

    RAID group /aggr0/plex0/rg1 (normal, block checksums, raid_dp)

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
      --------- ------          ------------- ---- ---- ---- ----- --------------    --------------
      dparity   0d.12.9         0d    12  9   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      parity    0d.12.10        0d    12  10  SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.12        0d    12  12  SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.11        0d    13  11  SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.14        0d    12  14  SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.15        0d    12  15  SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.16        0d    12  16  SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.17        0d    12  17  SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.12        3d    10  12  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.13        3d    10  13  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.14        3d    10  14  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.15        3d    10  15  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.16        3d    10  16  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.17        3d    10  17  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.18        3d    10  18  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.19        3d    10  19  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.20        3d    10  20  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.21        3d    10  21  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.22        3d    10  22  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.23        3d    10  23  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168

    RAID group /aggr0/plex0/rg2 (normal, block checksums, raid4)

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
      --------- ------          ------------- ---- ---- ---- ----- --------------    --------------
      parity    3c.00.0         3c    0   0   SA:B   0   SSD   N/A 762847/1562312192 763097/1562824368
      data      3c.00.1         3c    0   1   SA:B   0   SSD   N/A 762847/1562312192 763097/1562824368
      data      3c.00.2         3c    0   2   SA:B   0   SSD   N/A 762847/1562312192 763097/1562824368
      data      3c.00.3         3c    0   3   SA:B   0   SSD   N/A 762847/1562312192 763097/1562824368
      data      3c.00.4         3c    0   4   SA:B   0   SSD   N/A 762847/1562312192 763097/1562824368

b> aggr status -r aggr0
Aggregate aggr0 (online, mixed_raid_type, hybrid) (block checksums)
  Plex /aggr0/plex0 (online, normal, active)
    RAID group /aggr0/plex0/rg0 (normal, block checksums, raid_dp)

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
      --------- ------          ------------- ---- ---- ---- ----- --------------    --------------
      dparity   3d.11.8         3d    11  8   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      parity    3d.11.10        3d    11  10  SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.1         3d    10  1   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.18        0d    12  18  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.19        0d    12  19  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.20        0d    12  20  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.21        0d    12  21  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.22        0d    12  22  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.12        0d    13  12  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.13        0d    13  13  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.14        0d    13  14  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.15        0d    13  15  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.16        0d    13  16  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.17        0d    13  17  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.18        0d    13  18  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.19        0d    13  19  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.20        0d    13  20  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.21        0d    13  21  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.22        0d    13  22  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.2         3d    10  2   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168

    RAID group /aggr0/plex0/rg1 (normal, block checksums, raid4)

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
      --------- ------          ------------- ---- ---- ---- ----- --------------    --------------
      parity    3c.00.6         3c    0   6   SA:A   0   SSD   N/A 762847/1562312192 763097/1562824368
      data      3c.00.7         3c    0   7   SA:A   0   SSD   N/A 762847/1562312192 763097/1562824368
      data      3c.00.8         3c    0   8   SA:A   0   SSD   N/A 762847/1562312192 763097/1562824368
      data      3c.00.9         3c    0   9   SA:A   0   SSD   N/A 762847/1562312192 763097/1562824368
      data      3c.00.10        3c    0   10  SA:A   0   SSD   N/A 762847/1562312192 763097/1562824368

    RAID group /aggr0/plex0/rg2 (normal, block checksums, raid_dp)

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
      --------- ------          ------------- ---- ---- ---- ----- --------------    --------------
      dparity   3d.10.3         3d    10  3   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      parity    3d.10.4         3d    10  4   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.5         3d    10  5   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.6         3d    10  6   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.7         3d    10  7   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.8         3d    10  8   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.9         3d    10  9   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.10        3d    10  10  SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.10.11        3d    10  11  SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.11.1         3d    11  1   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.11.2         3d    11  2   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.11.3         3d    11  3   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.11.4         3d    11  4   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.11.5         3d    11  5   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.11.6         3d    11  6   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.11.7         3d    11  7   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.13.23        0d    13  23  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.11.9         3d    11  9   SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      0d.12.23        0d    12  23  SA:A   0  FSAS  7200 3807816/7798408704 3815447/7814037168
      data      3d.11.11        3d    11  11  SA:B   0  FSAS  7200 3807816/7798408704 3815447/7814037168

However, I still appear to have a bunch of drives which are part of the pool but don't come up as used in any of the RAID groups. If I total all the disks shown here, I come up with just 90 listed disks.

There are no unassigned disks.

How do I go about finding out why these disks are hidden and how can I allocate / use them?

The other question is I have one failed disk in 3c.00.22, however, it's not part of any of the disks listed in the aggregate. As the system is no longer under a support contract, I'm trying to ascertain the risk and how many disks I can afford to lose before data integrity becomes an issue. Critical data has been backed up, so it's not a disaster, just a risk assessment I'm trying to do. I guess it depends which combination of disks fail - more than two in a same RAID group and it goes down, but more drives can fail if spread across different RAID groups.

Any suggestions appreciated.

aladd · ‎2022-05-27

Collect an aggr status -r and sysconfig -r from both nodes.

This should help you shore up any unaccounted disks.

View solution in original post

paul_stejskal · ‎2022-05-26

I believe the term you're searching for is unassigned drives. I think it's disk show -n or -u, but it's been a few years. It may be priv set advanced to see it though. Pretty sure disk show -n will show unassigned drives.

aladd · ‎2022-05-26

disk show -n will display unassigned drives when ran from either node. from there you can identify the drives and assign them as needed.

XavW · ‎2022-05-27

Thank you both.

Strange, I get this:

a> disk show -n
disk show: No unassigned disks

So not sure what's happened to all those other drives!

aladd · ‎2022-05-27

pull a sysconfig -r, no additional options. This should show you available spares and broken disks as well.

XavW · ‎2022-05-27

I have and can see all the disks spread across the 5 different disk shelves.

Shelf 00 with 24 disks (containing one failed)

Shelf 01 with 24 disks

Shelf 10 with 24 disks

Shelf 11 with 12 disks

Shelf 12 with 24 disks

Shelf 13 with 24 disks

So a total of 132 disks.

What is unclear to me is how these disks are assigned to the aggregates / RAID sets.

Looking at each controller, as in my first post, you can see how they are used. But there are still a bunch which aren't listed in the aggr0. So I don't know if / how those extra drives are being used.

aladd · ‎2022-05-27

In 7-mode you have a root aggregate, which is where ONTAP lives and should be isolated from data aggregates. So the additional drives should be used to create a data aggregate. While 7-mode does allow you to store user data on the root aggr (aggr0) it is not ideal to do so, as it can cause data outages while troubleshooting.

The remainder of the drives not participating in an aggregate are spares that are used by ONTAP to either offload a misbehaving disk or perform a reconstruct from parity.

It seems like this was configured with only the root (aggr0).

XavW · ‎2022-05-27

Ahh, I've just seen another aggregate, an aggrSSD that I hadn't noticed before with 17 drives on each controller. So that adds another 34 bring me to a total of 124 drives. That leaves me with just 8 unaccounted drives...

aladd · ‎2022-05-27

Have you validated the output from both nodes?

XavW · ‎2022-05-27

Sorry, what do you mean by validated?

aladd · ‎2022-05-27

Collect an aggr status -r and sysconfig -r from both nodes.

This should help you shore up any unaccounted disks.

XavW · ‎2022-05-29

Thank you for this. This does include the spare disks which don't appear to show up anywhere else. By adding all of them up, I now reach the number of drives installed.