FAS2040 Aggregate Limits & External Shelf Throughput

[dtrujillo63]

My company is looking to buy a few netapp products and I have a question that you guys might be able to help with.

We're currently looking to get a FAS2040 paired with a fully populated 24 disk shelf running 2TB SATA drives. According to the SE working my account, I will be able to create one aggregate for all 24 drives. Now according to the netapp website, the maximum aggregate is 16 disks and or a volume size of 16TBs. Since I'm going to run 24 x 2TB disks, will I have to create multiple aggregates? I hope not since you lose two parity disks per aggregate.

Another question I have, what's the throughput from the FAS2040 controller to a external disk shelf? Again, according to the SE it's 12GB/s (he said it's four 3GB/s paths in one cable?). Now if I look at the product information on the netapp website, all it says is 3GB/s.

NetApp should either train their SEs to provide correct information, or update their product information. I'm confused.

Thanks!

[scottgelb]

It is correct that the 3GB SAS port has 4 lanes, so you have 12GB of throughput on that port (3GB*4 lanes) and the cable has all 4 lanes so you only have one cable.

For the Aggregate Limits.  Depending on if you are running ONTAP 7.3.x or 8.0.x.  It is recommended to use 2TB drives with ONTAP 8.0.x for the larger 64 bit aggregates.  You can have multiple raid groups in an aggregate, but with ONTAP 7.3, you would have only 1 RAID group in the 2TB aggregate and with ONTAP 8.0.x you could have 1 or 2 RAID groups in the 2TB aggregate.  The limit you listed below of 16 disks and 16TB is for ONTAP 7.3, not 8.0.x.

7.3

Max Aggr Size is 16TB right sized disk on any platform.  The maximum number of disks in a 2TB aggregate is 11 disks and the maximum number of data disks is 9 disks (9 Data + 2 Parity).  With 9 data disks you will yield about 13.09TB without volume snapshots.

The maximum raid group size (not aggregate size) for SATA  and RAID_DP is 16 disks (14 data + 2 parity) and the default is 14 (12 data + 2 parity)

8.0.1

The FAS2040 has a max aggr size of 30TB right sized disk (depending on the platform, the limits are 30,50,70 or 100TB in this release).  The maximum number of disks in a 2TB 64 bit (2040) aggregate is 22 disks and the maximum number of data disks is 18 disks.  You could create a 20 Drive aggregate with (18 Data + 2 parity), but often we see 2*(9D+2P).  The latter has a smaller raid group and keeps raid group sizes even, but does cost 2 extra drives.  The debate on this would be rebuild times (but with rapid raid recovery less of an issue) and statistics of disk failure with a larger raid group.  With 18 data disks you will yield about 26.2TB without volume snapshots.

The maximum raid group size (not aggregate size) for SATA and RAID_DP was increased to 20 disks with 8.0.1 (18 data + 2 parity).

So....with 24 drives on the system and assuming ONTAP 8.0.1, if you have 22 disks in the aggregate (2*(9D+2P) and 2 hot spares, you could use the whole shelf on the FAS2040...but since you max out at 20 or 22 disks (depending on 1 or 2 raid groups) and you need spares, you won't be able to put all 24 drives in the aggregate but it is close.

[dtrujillo63]

It was my understanding that an aggregate was just another word for a raid group? I guess since ONTAP 7.3 only allowed one raid group in an aggregate, it did act as another word for a raid group right? Now with ONTAP 8.0, your saying that an aggregate will support two raid groups. Does that mean that it will logically group the two raid groups to display the total usable space as one? What about from a performance stand point? Will my data be striped across both raid groups in the one aggregate? I wanted to get the 24 drive shelf because I thought all the drives (minus spares and parity) were going to spin my data which in turn would provide better performance. The solution we're looking to purchase is an empty FAS2040 with dual controllers, plus two DS4243 shelves. One with 24 * 2TB SATA drives for my tier 2 & 3 data, and one with 24 * 450GB 15K SAS drives for tier 1 data. How would all of this effect my SAS shelf?

Here's how it was playing out in my head before I read your post. I would create one aggregate per shelf. Each aggregate would be configured with two hot spares plus a 22 disk RAID-DP group.

What configuration would be "best" practice for the hardware I plan on getting?

[scottgelb]

An aggregate can contain many raid groups....a max of 150 per aggregate and 400 per controller.  For best performance, we like to keep the RAID groups even sized or within 1 drive of each other.  If you stripe across 18 data disks in one raid group, or 18 data disks in two raid groups (9 data per raid group), I would expect very similar performance.  ONTAP stripes across all disks in an aggregate so the data would be written across all data disks...there is some overhead for each raid group but a very small percent.

Assuming you are using 8.0.1 (not 7.3)... Here is a way you could create one aggregate per controller.  There is a lot of debate over having a separate root aggregate, but for smaller 2040 systems we often don't create separate root.  You can find that debate on this community site and it has a lot of good input from several members.  I do like that you want to put SATA on one node and SAS on the other.  That prevents CPs for SATA from slowing down SAS under heavier workloads.  Based on 24 disks, you could use near the same RAID layout for SATA and SAS since you don't have that many disks.

Controller 1 - 24x 2TB SATA    default raid group size for raid_dp is 14 (12D+2P) and max raid group size (8.0.1) is 20 (18D+2P)

spares  2-4 disks     (depends on if you have 1 or two raid groups below)..

aggr0  ~24TB usable after 5% aggregate reserve and no volume snapshots

     rg0     9D+2P

     rg1     9D+2P

Alternatively...you could have 20 drives in one raid group for the same usable... and leave 4 spares...but a more conservative way is above.. 

     rg0     18D+2P

Controller 2 - 24x 450GB SAS   default raid group size for raid_dp is 16 (14D+2P) and max raid group size is 28

spares   2 disks

aggr0   ~ 6.1TB usable after 5% aggregate reserve and no volume snapshots

     rg0     9D+2P

     rg1     9D+2P

alternatively...you could fit the 22 drives in one aggregate... ~ 6.78TB usable after 5% aggregate reserve and no volume snapshots

     rg0     20D+2P

[peter_kazakov]

I have a question here.

All this (having two raid groups in a single aggregate) applies when the storage is in Active-Pasive.

Can it be done, in Active-Active Configuration, with each controller holds one RAID group, but they are merged in a single aggregation?

My idea is to have the throughput of both controllers, and to have all space allocated in a single aggregation

Is that possible with the new ONTAP 8.0.1?

[dtrujillo63]

Unfortunately, this cannot be done. Think of an Active/Active configuration as two separate filers. Each controller must have its own drives, aggregates, and raid groups.

[KEITH4472]

Scott,

I am also building a FAS2040 with an external DS4243 shelf with 24 x 450GB SAS drives for one of our regional offices and was wondering whether to go with 2 x 9+2 RAID groups or a single 20+2 RAID group.

Apart from the difference in available space and a longer rebuild time for the larger single RAID group are there any drawbackups with going for the single larger RAID group.  I'll be running OnTAP 8.0.1 with a 32 bit aggregate with a mixture of user files (home directories, groups areas etc) plus up to 40 VMs accessed via NFS.

Thanks

Keith

[scottgelb]

If 8.0.1, I would go with a 64-bit aggregate just to future proof the setup.  The risk is the statistics of hitting a double disk failure with a remaining bit error on a drive or a triple disk failure with a raid group that size... for rebuild time, we are seeing rapid raid recovery for a large number of rebuilds (pre-fail of drives and a copy of the drive) so most rebuilds may not have any extra time regardless of rg size unless the drive has a hard failure and needs a full raid rebuild.  For rapid raid recovery you have to have at least 2 spares of that drive type which is what you will have below.  Assuming 5% aggr reserve and 20% snapshot reserve, the larger raid group will get you about 556GB additional usable.

If you really need that space, then I would decide based on the usable requirement...if less space works then I would be more conservative with smaller raid groups...and if you know you are going to add disks later, you can add disks to existing raid groups.  If not adding disks later, then 22 might be the way to go in a single raid group.

[KEITH4472]

Thanks for your helpful advise Scott. However, I may stick with a 32 bit

aggregate for 2 reasons:

1. We will be storing a large number of smaller files and NetApp recommend

32 bit for such use, particularly on a smaller controller.

2. We have 2 clusters in our Data Centres running 7.3, and although we plan

to upgrade to 8.0.1 at some point over the next 12 months, I expect that we

will stay with 32 bit aggregates for now as I don't have the space or the

downtime to rebuild the existing arrays (mixture of 300GB FC and 1TB SATA).

Although I only expect to use QSM I would like to retain the ability to do

full volume mirroring, so I may wait for NetApp to release the promised

converter from 32 to 64 bit aggregates.

So this is a tricky one for me as this is our first system on OnTap 8 and

whichever way I go I can see drawbacks. The only areas where I can see 64

bit aggregates helping me will be to load my 1TB SATA shelves more evenly.

Regards

Keith

[scottgelb]

Eventually we may need everything 64bit, but I have had an 8.1 peek and there is a conversion from 32bit...but until it is officially available as a Release Candidate and we see it I won't say anymore since QA code doesn't guarantee what will hit the RC and GA releases...but safe to say if there isn't conversion it will cause a lot of stir in the field.  If you have any plans for compression then that is another reason for 64bit.  Eventually I think we'll see 32bit aggrs go the way of traditional volumes, but for now your plan makes sense and will help your performance on a smaller controller with 32bit aggrs.

[scottgelb]

To clarify, if you have 20 raid groups or 5 raid groups all with the same amount of disks, all the space is available to the aggregate.  This has always been the case...just with 2TB drives 7.3 couldn't fit more than one raid group in an aggregate.

[dtrujillo63]

Is ONTAP 8.0.x not included by default? If it is, are future updates available free of charge?

[scottgelb]

It depends on how and when it is ordered.  A 2040 will ship with GA code.. and 8.0.1 is now GA.  But it could ship with a 7.3 ONTAP.  Upgrades are included in the software maintenance which is a maintenance line item on your proposal and with software maintenance there is no additional fee for ONTAP upgrades.

[dtrujillo63]

I actually just placed the order earlier today. I did get SupportEdge premium, but saw nothing related to software maintenance. If it's not included, they better throw it in. I'm not liking the fact that there seems to be a fee for EVERYTHING.

Thanks for being so helpful by the way. I really like the community thus far.

[scottgelb · Accepted solution]

SupportEdge is maintenance so you should be good to go.

Typos Sent on Blackberry Wireless

[ccolht]

Each RG requires 2 parity so more RGs means less usable space.