nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Mixed disk size aggregates

tyrone_owen_1

FAS8080, ONTAP 8.3.1P1

NetApp have expanded an aggregate which was previously solely 900GB disks with 1800GB disks, resulting in an aggregate with mixed disk sizes. This approach is alien to me and even though a new feature in ONTAP allow mixed disks in an aggregate to maintain their useable capacity I was still under the impression that mixed disk aggreates were to be avoided and best practice was an aggregate with a single disk-type (excluding hybrid aggs).

What are the pros and cons of this setup?

The advantage I suppose is the spindle count, however to my mind this will only be effective up to the point where the smaller disks become full upon which the RAID groups which contain the larger disks will start getting hot. Unless there is some sort of automatic/manual re-balancing (reallocate). Part of the issue I have is that how the mixed disk aggregates function doesn't seem to be documented, or at least I can't find it.

I'm thinking of backing out this config by splitting out the aggregate to two as I don't know what the consequences will be down the road.

Thanks

Find more posts tagged with

FAS

Accepted answers

bobshouseofcards

As was previously pointed out, you aren't losing any capacity in this setup. The only time you lose disk capacity from multiple sizes is when multiple sizes are forced together in a raidgroup. This can happen on creation (generally accidentally) or when a larger size disk is pressed into service as a spare replacement for a compatible smaller size disk type: i.e. 900GB SAS disk fails, no spares available but a 1800GB SAS spare is, the 1800GB disk will be treated as a 900GB and takes over going forward. The only way to get capacity back in these situations is to manually fail out the "fake capacity" disk with a disk of the real desired capacity, then the larger disk disk to make it back into a full size spare. So long as you keep adequate spares of the size needed, you're good on a capacity front.

The key point in mixing disk sizes in an aggregate on raid boundaries is that performance measurement is unpredictable. Consider this metric - IOPs per GB. As a basic example and using round numbers (rather than right sized just to make the math simpler), assume 900GB and 1800GB drives with 15 data drives per RAID group, and 200 IOPs available per drive (about right for 10K drives).

Raidgroups that are made up of 900GB drives have 15 * 900 = 13500GB capacity and 3000 IOPs capability, or 0.22 IOPs per GB. As you might expect, raidgroups that are made up of 1800GB drives have 27000GB capacity and yet still the same 3000 IOPs capability, or only 0.11 IOPs per GB.

So in a mixed size aggregate, as data is spread across all the disks, you'll see some data accessed at 0.22 IOPs per GB processed and some at 0.11 IOPs per GB processed. Looks like you have roughly 11 aggregates split roughly 50/50. Since the larger disks are twice the size of the smaller, you have a roughly 1/3 - 2/3 split in capacity. Over time, 1/3 of your data will run at 0.22 IOPs per GB, and 2/3 will run at 0.11 IOPs per GB. So the addition of the larger disk slows down the entire aggregate on average.

So - questions: does this matter? That's up to you to answer. The addition of the 1800s to the same aggregate slows down the aggregate as a whole. Since data is written anywhere in the aggregate (WAFL), eventually even data today wholly residing on the 900GB disks will make it's way to the 1800s and potentially slow down. But, is your IO load heavy enough to make this an important consideration? Or, is it important to have a single management point (the aggregate) rather than say two separate aggregates?

The other way - two separate aggregates with just one disk type. Has the same issue with performance when measured "as a whole" across both aggregates - average IOPs capability of the system relative to the capacity of the system will go down. But, with two aggregates, you have fine tuning control available based on where you put specific volumes - the aggregate that can process data faster per GB or the one that is slower per GB. That may be what you want. Then again if you just need a lot of capacity in a single chunk - well that's what you need and a single aggregate gets it.

Also - it appears that the 1800GB disks are created in raid groups that are a different size than are the 900GB disks, if I've counted the disks carefully. That design also has the same performance pitfalls, though to a much lesser scale, as does multiple sizes in an aggregate.

To sum up - what you've created is an aggregate that will perform unpredictably over time depending on where any data is written. It willl generally slow down as a whole over time as capacity utilization increases and/or spreads over all the disks when measured against the total data throughput.

Unless there is a really good and well thought out reason to do it this way, I add vote to not mixing sizes as a best practice.

Hope this helps.

Bob Greenwald

Lead Storage Engineer | Consilio, LLC

NCIE SAN Clustered, Data Protection

Kudos and accepted solutions are always appreciated.

All comments

aborzenkov

new feature in ONTAP allow mixed disks in an aggregate

This was possible for as long as I remember (which is 15+ years). So there is nothing new. The obvious drawback of this approach is that large disks are utilized more than small so some data is striped across less number of disks. You cannot predict what data and it is impossible to give blanket statement about impact. You probably should avoid it for high load OLTP application; OTOH for simple file server my guess would be - nobody will notice.

tyrone_owen_1

I didn't realise that historically you could maintain the usebale size of larger disks within mixed aggregates, I always thought they were right-sized to the smaller capacity disks. I never really looked into it until now as it has never been an issue for me.

Is there anything documented about the behaviour?

nasmanrox

I agree with Tyron. Unless it's hybrid aggr, I do not believe it's recommended to mix the size. Your large size disk will only function as the smaller size

aborzenkov

I do not believe it's recommended to mix the size.

That's true.

Your large size disk will only function as the smaller size

And that's wrong, sorry. This applies to a single raid group, not to the whole aggregate. You can have multiple raid groups with different disk sizes; in each raid group whole disks will be used.