NetApp have expanded an aggregate which was previously solely 900GB disks with 1800GB disks, resulting in an aggregate with mixed disk sizes. This approach is alien to me and even though a new feature in ONTAP allow mixed disks in an aggregate to maintain their useable capacity I was still under the impression that mixed disk aggreates were to be avoided and best practice was an aggregate with a single disk-type (excluding hybrid aggs).
What are the pros and cons of this setup?
The advantage I suppose is the spindle count, however to my mind this will only be effective up to the point where the smaller disks become full upon which the RAID groups which contain the larger disks will start getting hot. Unless there is some sort of automatic/manual re-balancing (reallocate). Part of the issue I have is that how the mixed disk aggregates function doesn't seem to be documented, or at least I can't find it.
I'm thinking of backing out this config by splitting out the aggregate to two as I don't know what the consequences will be down the road.
new feature in ONTAP allow mixed disks in an aggregate
This was possible for as long as I remember (which is 15+ years). So there is nothing new. The obvious drawback of this approach is that large disks are utilized more than small so some data is striped across less number of disks. You cannot predict what data and it is impossible to give blanket statement about impact. You probably should avoid it for high load OLTP application; OTOH for simple file server my guess would be - nobody will notice.
I didn't realise that historically you could maintain the usebale size of larger disks within mixed aggregates, I always thought they were right-sized to the smaller capacity disks. I never really looked into it until now as it has never been an issue for me.
Here's the details of one of the aggregates in question, as you can see two different disk sizes displaying their usebale sizes, not right-sized to the smallest disk. It doesn't really matter if this is a new feature or not, I'm still not comfortable with it given the best practice. I was really after pros and cons, or anyone with experiense of using in this configuration. I'm tempted just to back it out as I have the space to be able to move things around at the moment.
As was previously pointed out, you aren't losing any capacity in this setup. The only time you lose disk capacity from multiple sizes is when multiple sizes are forced together in a raidgroup. This can happen on creation (generally accidentally) or when a larger size disk is pressed into service as a spare replacement for a compatible smaller size disk type: i.e. 900GB SAS disk fails, no spares available but a 1800GB SAS spare is, the 1800GB disk will be treated as a 900GB and takes over going forward. The only way to get capacity back in these situations is to manually fail out the "fake capacity" disk with a disk of the real desired capacity, then the larger disk disk to make it back into a full size spare. So long as you keep adequate spares of the size needed, you're good on a capacity front.
The key point in mixing disk sizes in an aggregate on raid boundaries is that performance measurement is unpredictable. Consider this metric - IOPs per GB. As a basic example and using round numbers (rather than right sized just to make the math simpler), assume 900GB and 1800GB drives with 15 data drives per RAID group, and 200 IOPs available per drive (about right for 10K drives).
Raidgroups that are made up of 900GB drives have 15 * 900 = 13500GB capacity and 3000 IOPs capability, or 0.22 IOPs per GB. As you might expect, raidgroups that are made up of 1800GB drives have 27000GB capacity and yet still the same 3000 IOPs capability, or only 0.11 IOPs per GB.
So in a mixed size aggregate, as data is spread across all the disks, you'll see some data accessed at 0.22 IOPs per GB processed and some at 0.11 IOPs per GB processed. Looks like you have roughly 11 aggregates split roughly 50/50. Since the larger disks are twice the size of the smaller, you have a roughly 1/3 - 2/3 split in capacity. Over time, 1/3 of your data will run at 0.22 IOPs per GB, and 2/3 will run at 0.11 IOPs per GB. So the addition of the larger disk slows down the entire aggregate on average.
So - questions: does this matter? That's up to you to answer. The addition of the 1800s to the same aggregate slows down the aggregate as a whole. Since data is written anywhere in the aggregate (WAFL), eventually even data today wholly residing on the 900GB disks will make it's way to the 1800s and potentially slow down. But, is your IO load heavy enough to make this an important consideration? Or, is it important to have a single management point (the aggregate) rather than say two separate aggregates?
The other way - two separate aggregates with just one disk type. Has the same issue with performance when measured "as a whole" across both aggregates - average IOPs capability of the system relative to the capacity of the system will go down. But, with two aggregates, you have fine tuning control available based on where you put specific volumes - the aggregate that can process data faster per GB or the one that is slower per GB. That may be what you want. Then again if you just need a lot of capacity in a single chunk - well that's what you need and a single aggregate gets it.
Also - it appears that the 1800GB disks are created in raid groups that are a different size than are the 900GB disks, if I've counted the disks carefully. That design also has the same performance pitfalls, though to a much lesser scale, as does multiple sizes in an aggregate.
To sum up - what you've created is an aggregate that will perform unpredictably over time depending on where any data is written. It willl generally slow down as a whole over time as capacity utilization increases and/or spreads over all the disks when measured against the total data throughput.
Unless there is a really good and well thought out reason to do it this way, I add vote to not mixing sizes as a best practice.
Hope this helps.
Lead Storage Engineer | Consilio, LLC
NCIE SAN Clustered, Data Protection
Kudos and accepted solutions are always appreciated.
Hi there! bit of an old thread, so you might not get too many responses here.
As I have heard it explained, ONTAP will schedule all writes towards the size of the smallest disks - larger capacity drives have better performance usually - meaning if it uses x% of time to write data, and it knows it can write "X" gb in a write cycle, meaning the amount of data it can write in any operation across all of the disks is reduced due to the smaller disks, meaning it can be slower to clear nvram checkpoints, which can lead to worse performance, sometimes significantly so.
I've mixed 8TB and 10TB drives in the same aggregate before, and it's fine.. but it was a massive aggregate and a sparse workload. I don't think I'd suggest mixing 900GB and 1800GB drives together.