Aggregate max size calculations off by 2/16Tb (12.5%)?

[fletch2007]

I know this will all go away when we upgrade to 64bit - but this was very frustrating:

we have an existing  aggregate of 2 x DS4243 shelves (48 disks x 266gb right sized)

Ontap shows the existing capacity of 9.8Tb (via df -Ah)

We went to add one more shelf (24 x 266Gb) and found the size slightly larger than the 16Tb limit:

aggr add aggr2 -d 3d.02.0  3d.02.1  3d.02.2  3d.02.3  3d.02.4  3d.02.5  3d.02.6  3d.02.7  3d.02.8  3d.02.9  3d.02.10  3d.02.11  3d.02.12  3d.02.13  3d.02.14  3d.02.15  3d.02.16  3d.02.17  3d.02.18  3d.02.19  3d.02.20  3d.02.21

Note: preparing to add 20 data disks and 2 parity disks.

Continue? ([y]es, [n]o, or [p]review RAID layout) y

Aggregate size 16.08 TB exceeds limit 16.00 TB

File system size 16.08 TB exceeds maximum 15.99 TB

aggr add: Can not add specified disks to the aggregate because the aggregate size limit for this system type would be exceeded.

Ok, fine, we will add 21/24 disks instead - (overkill on the spares for now)

Now that is not the most frustrating part (remember, we will get this back when when we go 64bit)

The frustrating part is when we add 21 disks we ended up with not ~15.8Tb (as you'd expect 16.08 - .266 = 15.817Tb)

No we end up with only 14 (note the aggregate snapshot is disabled/zero):

df -Ah

Aggregate                total       used      avail capacity 

aggr2                     14TB     8324GB     6257GB      57% 

aggr2/.snapshot            0TB        0TB        0TB     ---%

Can someone explain why ontap appears to use two sets of books for these calculations

If there is overhead it should be included in the final useable calculations so these discrepencies are eliminated

thanks

Fletcher

http://vmadmin.info

[fletch2007]

The good news is our before and after VM benchmarks show a near linear spindle:vm throughput improvement with the addition of the 21 disks to the 46 disk aggregate:

http://www.vmadmin.info/2011/08/more-spindlesmore-vm-throughput.html

One open question is: will WAFL re-stripe the existing VM volumes over the newly added disk spindles over time?  (we CLONED the VM to ensure the VM was striped across all disks for the AFTER benchmark)

Will existing VMs need a similar operation to gain the full benefits of the added spindles?

thanks

[chriskranz]

That might make sense as although the Aggregate is sized on right sized capacity, the aggregate has a 10% overhead for WAFL. So your 15.8 TB storage loses 1.58TB due to this bringing it down to around the 14TB mark.

Yes, it can be frustrating if you aren't made aware of this upfront. There is also some challenges around the disk shelves being shown as 24x <insert disk size> rather than saying a disk shelf is approximately <insert usable size>. There are additional variables that can affect the usable size of a disk shelf (snapshots, parity, spares, so on), so it's a difficult marketing battle. Usually your supplier would help you make the calculations and see what usable size you would have, this should always be done with a new system, but I know is sometimes missed when adding more storage to an existing system as there are a few more variables to take in.

Regarding the question you pose on your website, WAFL will not automatically balance the load across the new spindles for you with you telling it to. You can run a reallocation against specific volumes to spread the data out, and you can also create a reallocation schedule where it'll only rebalance the load if it meets a certain threshold that deems beneficial (which you can also configure). This doesn't happen by default to give you a little more control over the performance hit of performing this, so it allows you to stagger this across different volumes and at different times to reduce the performance impact (it's a massive read scan as you can imagine).

Good to see the performance results though!