Aggregate size calculation changes

janjacobbakker · ‎2008-07-11

Hi Folks,

I'm trying to design a storage solution.

A FAS3020 with 3 shelves full with 42x 300GB FC disks.

Default the Raid Group size is 16 (14 + 2 spare) de max raid group size is 28.

Does anyone have some best practice information? The NOW site hasn't much info on that.

Kind regards

scott_lau · ‎2010-02-26

Building new aggregates is all fine and dandy, but what do I do in a case like this:

Aggregate RAID Size 16 with 15x621G disks

We have added a 14TB shelf (1TB disks) and need to expand the aggregate mentioned above. I don't want to use the entire shelf for this aggregate since I have other aggregates that have the similar setup. I would like to know the best scenario for the best performance with the mixed disks???

Thanks all!

Scott L.

redtail1981 · ‎2010-05-04

Hi guys,

After reading so much i'm pretty confused on the Raid-Group and Max data disk for Aggregate part..

If i have a FAS3140HA with 3 x DS243 populated with 24 x 1 TB Disks, what would be the most recommended configurations ?

Should i create 6 x Aggr with RG size of 10 Data Disks each ?

	Raid Group	Parity	Total	Controller
Aggr0	10	2	12	A
Aggr1	10	2	12	A
Aggr2	9	2	11	A
Aggr3	9	2	11	B
Aggr4	9	2	11	B
Aggr5	9	2	11	B

Additionally 2 x HotSpares for each Controller

Or should i create 3 x Aggr with Both RG of size 8 and remaining Aggr with 10 Data Disks ?

	Raid Group1	Raid Group2	Parity	Total	Controller
Aggr0	8	8	4	20	A
Aggr1	8	8	4	20	A
Aggr2	8	8	4	20	B
Aggr3	6		2	8	B

Additionally 2 x HotSpares for each Controller

Btw if i have an additional 3140 with the same amount of disk shelves would i be able to use the same configurations ?

Thanks in advance.

radek_kubka · ‎2010-05-04

Hi and welcome to the Communities!

First of all - what RAID level are we talking about? The maximums specified here are talking about data disks only, so it makes a difference to the disk allocations whether we use RAID-4 or RAID-DP:

http://now.netapp.com/NOW/knowledge/docs/ontap/rel73/html/ontap/rnote/rel_notes/reference/r_oc_rn_feat73_aggr-size-max-drives.html

I assume you want to go for the default RAID-DP settings, so extra two parity drives for each RAID group.

You can actually stick to the default RAID group size for SATA, i.e. 14 & effectively end up with following setup:

	RG 1	RG 2	HotSpare	Data Drives	Total	Controller
Aggr0	14	9	-	19	23	A
Aggr1	14	8	-	18	22	A
Aggr0	14	9	-	19	23	B
Hot spare	-	-	2	-	2	A
Hot spare	-	-	2	-	2	B

Hot spares are per controller, not per aggregate & you can also have the same aggregate name on both controllers.

For a non-clustered FAS3140 you can have almost identical setup, but all drives will be obviously assigned to the same controller (in this case it has to be be aggr0, aggr1 & aggr2). You can also have one less hot spare, thus adding one data disk to the smallest of the aggregates.

Regards,
Radek

redtail1981 · ‎2010-05-04

Hi Radek,

Thanks for your inputs, btw is the following configurations possible as well ?

Yup its in Raid-DP, Think i input the wrong field should be Parity instead of Hot-spares.. my bad...

	RG1	RG2	Parity	Total	Controller
Aggr0	10	9	4	23	A
Aggr1	9	9	4	22	A
Aggr0	10	9	4	23	B
HotSpare	-	-	-	2	A
HotSpare	-	-	-	2	B

As i read there seems to be a limitation of 19 Data Disks for firmwares after 7.3.x, is that right ?

Does having the Raid Groups to be similiar size helps in the performance as well ?

Or filling the first RG to 14 and giving 5 data disks to the remaining RG is a better choice ?

I have read some people actually recommend craving an Aggr0 with 3 disks only ? Is it recommended ?

Sorry for the many questions..

Thanks and Regards,

HongWei

radek_kubka · ‎2010-05-04

Yep, you have to be on 7.3 onwards to stick 19x 1TB data drives into one aggregate.

And yep, having fairly even RAID group sizes does help - configuring this though requires slightly more effort, but still - this isn't a rocket science . You can find a detailed know-how just below, in a post from Scott:

http://communities.netapp.com/message/4148#4148

Regards,

Radek

aflores_ibm · ‎2010-05-04

HongWei,

Those requirements are correct but you might also have to take under considerations what you are deploying on that system. For example, A-SIS has its own constraints and benefits that will impact your RAID design. Having a segregated aggregate for managing the root volume is great but few are the enterprise that will not look twice at assigning 3 disks of 1TB for hosting only the root volume. If you would deploy the root volume and other data within that 3 disks RAID, the performance requirement should be minimal as you will not have the benefit of having multiple striped disks. Another basic solution is to host the root volume and guarantee its space within the larger aggregate. Rebuilt time on a 3 disks aggregate should also be considered on large disks.

Having balanced RAID groups within the containing aggregate seems to be the best approach assuming that you know before hand how many shelves and disks you will be given as well as the process and services you will have to support.

Regards,

Allain

redtail1981 · ‎2010-05-04

Hi Radek and Allain,

Thank you both for your inputs

Will deploy the Filer with the config, i've post before hand and for the single controller options below are the config i have in mind:

As i feel, it might be better to fill up all the aggregates and leaving an additional spares. Or is there any other considerations that i might be missing out ?

	RG1	RG2	Parity	Total
Aggr0	10	9	4	23
Aggr1	10	9	4	23
Aggr2	10	9	4	23
HotSpare	-	-	-	3

Best Regards,

HongWei

rkaramchedu1 · ‎2010-05-05

Consider this (I tend to do math in a round about way, so please bear with me... )

Total # of drives = 3 x 24 = 72

# of Hot Spares = 4 (To enable disk maintenance center, 2 hot spares are required per controller)

# of Active Drives = 68

# of active drives/controller = 34

Using RG Size of 12, you get 2 aggregates per controller

Aggregate 1 (RG0 = 10D+2P, RG1 = 9D+2P). Total # drives in Aggregate 1 = 23 drives
Aggregate 2 (RG0 = 9D+2P). Later, you can add another 12 drives to this aggregate (10D+2P) and get another aggregate with maximum efficiency.

With the default RG Size of 14, in order to maximize spindles in a 16TB aggregate, I'll have to have these RG layouts

12D+2P
7D+2P

Not sure if I like that. Does this make sense?

redtail1981 · ‎2010-05-05

Hi rkaramchedu1,

I dun quite get it, you are proposing 2 different ways of craving the Raid Groups Depending on the Raid Size ?

Or for the second calculation is for the Controller B's Aggregates ?

Anyway if we considers trying to max the Raid Group to Raid group default of 14.

We will not be able to max out the aggregate and will end up having to create additional aggregate and inducing addtional Parity Disks, am i right ?

Regards,

HongWei

rkaramchedu1 · ‎2010-05-06

I dun quite get it, you are proposing 2 different ways of craving the Raid Groups Depending on the Raid Size ?

Not really. If you specify a raid group size of 12 when creating a drive and select 23 drives to be added to the aggregate, it will automatically created the two raid groups. IIRC, there's also a "dry run" option with a "-n", that will show the layout of what it would do, without actually doing it.

In order to get as close to the 16TB limit as possible, the configuration is to include 23 drives (and also limit parity drives to 4) as indicated above.

redtail1981 · ‎2010-05-06

Hi rkaramchedu1,

I have another question in mind since we are at the Aggregate design topic

As i know the max possible Raid Group size is 16, it is possible for us to create an Aggregate with single Raid Group 14 Data + 2 Parity am i right ?

Will it in turn churn out better performance as it has more spindles ?

What is the advantages i have, in maximizing an Aggregate to 23 Disks, other then having a single larger Aggregate to manage from ?

Regards,

HongWei

rkaramchedu1 · ‎2010-05-06

I have another question in mind since we are at the Aggregate design topic

As i know the max possible Raid Group size is 16, it is possible for us to create an Aggregate with single Raid Group 14 Data + 2 Parity am i right ?

Will it in turn churn out better performance as it has more spindles ?

What is the advantages i have, in maximizing an Aggregate to 23 Disks, other then having a single larger Aggregate to manage from ?

Yes you can create a aggregate with 16 drives with RG Size set to 16.. Remember that an aggregate will have multiple raid groups underneath it as it grows bigger or as it is created. As I said before, if you want to maximize for aggregate capacity and get the aggregate size as close to 16TB as possible, then the optimal configuration of that aggregate would be not with the default raid group size of 14 but with a raid group size of 12. Just doing some quick math here with the raid group sizes and for 23 drives

RG Size = 10. 3 Raid groups with 3 drives in the last raid group.

RG Size = 11. 2 Raid groups with 1 drive left over

RG Size = 12, 2 Raid groups with 11 drives in last raid group

RG Size = 13, 2 Raid groups with 10 drives in last raid group

RG Size = 14, 2 Raid groups with 9 drives in last raid group

RG Size = 15, 2 Raid groups with 8 drives in last raid group

RG Size = 16, 2 Raid groups with 7 drives in last raid group

Given all of these combinations, I'd take RG Size of 12, which has a even balance of disks in my raid groups. The aggregate will perform better because it has more spindles and it has a more balanced and uniform raid layout.

aflores_ibm · ‎2010-05-06

I would recommend a different view of the equation:

Total Disks	RAID size	Number of RAID	RAID employed Disks	Spares	DP Used Disks	Disks Used for DP and Spares
72	4	18	72	0	36	36
72	5	14	70	2	28	30
72	6	12	72	0	24	24
72	7	10	70	2	20	22
72	8	9	72	0	18	18
72	9	8	72	0	16	16
72	10	7	70	2	14	16
72	11	6	66	6	12	18
72	12	6	72	0	12	12
72	13	5	65	7	10	17
72	14	5	70	2	10	12
72	15	4	60	12	8	20
72	16	4	64	8	8	16

Using 12 disks per RAID would leave you with no spares. Using software ownership, you can define RAID group of 16 and maximize you usable allocation space. Also you would maximize performance within the RAID group and leave a miminum of disks used for internal system requirements.

Just a though

redtail1981 · ‎2010-05-06

Hi but is another sense you will not be able to fill up the aggregate to its max of 16TB and for some configs which you indicated the number of spares seems to be either too much or too little ?

Best Regards

aflores_ibm · ‎2010-05-07

HongWei,

I've redefined the table so you might see the results a bit more clearly. I'm attaching it in an xls format as well so you can have a better view of it.

The rows in red font are not recommended due to the low amount of spare disks compare to the amount of RAID-DP group and RAID sizes.

The rows in red font and stricked though are impossible due to the '0' amount of spare disks.

the rows containing bold fonts are the remaining solution for balanced RAID-DP groups. Some, while yielding a maximum return of usable space, would require more manual or customized maintenance as explained below the table.

Please keep in mind that NetApp recommendation are to 14 disks per RAID. This table is just an example to define balanced RAID-DP deployment over the specified environment.

Total Disks	RAID-DP Group		Disk Deployment
Total Disks	Max # of group overall	# Disks per group	# of Disks hosting Data	# of Disks hosting DP	# of Disks assign for spare
72	4	16	56	8	8
72	4	15	52	8	12
72	5	14	60	10	2
72	5	13	55	10	7
72	6	12	60	12	0
72	6	11	54	12	6
72	7	10	56	14	2
72	8	9	56	16	0
72	9	8	54	18	0
72	10	7	50	20	2
72	12	6	48	24	0
72	14	5	42	28	2
72	18	4	36	36	0
72	24	3	24	48	0

Basically, with a 12 disk RAID group over 72 SATA disks and using the maximum amount of RAID groups [6], you would have no spares left and, therefore, would have to reduce that number by one leaving 5 RAID groups and 12 spares. You can decide to create an new reduced aggregate out of that amount if you which.

On the other hand, using 16 disks would yield 4 RAID groups with 8 spares. You would have to remove the disks auto-assignment and define disk ownership manually to transfer more disks to one controller [A] then to the the other [B]. You would have to disable auto-assign permanently and do disk ownership yourself or via a script.

Regards,

Allain

cebulrdcis · ‎2008-09-25

Hi Guyz,

need your expert recommendations on what raid group size do you recommend on a 1TB SATA Shelve? I have 3 x 1TB NetApp shelf to be used as my snapvault target ( secondary storage).

Also, I have some 1TB shelves on my primary storage for CIFS Shares.. and I dont know if I used the best practice for raid group size.I used raidsize=14 to maximize the capacity limit..

hoping im correct..

scottgelb · ‎2008-09-25

Depending on if the ONTAP version is 7.2.x or 7.3.x, here is what I have been doing for 1TB drives...based on the maximum drives supported per aggregate for each ontap version.

7.2.x 19 total, disks supported (15 data max per aggr) [ (RG0: 8D+2P) + (RG1: 7D+2P) ]

7.3.x 23 total disks supported (19 data max per aggr) [ (RG0: 10D+2P) + (RG1: 9D+2P) ]'

Since we need two raid groups to max the aggregate size (4 parity disks needed to get to max size), it's best to keep the raid groups as even as possible.. for example, with 7.3 using the default raid group size, you could go with rg0: 14D+2P, then rg1: 5D+2P - but that wouldn't be as good as 10D+2P and 9D+2P over two raid groups...

The trick with ONTAP 7.2 to do this is... create a 10 drive aggregate.. change the raidsize option to 10, then add 9 more drives to max it out..changing to 10 will force a new raid group. For example:

aggr create aggrname 10
aggr options aggrname raidsize 10
aggr add aggrname 9

Similarly, you can do the same with ONTAP 7.3 by creating with 12 drives, then change the raidsize option to 12, then add 11 more drives to max it out.. Additionally, 7.3 added a new option to create a new raid group with "-g new"

aggr create aggrname 12
aggr options aggrname raidsize 12
aggr add aggrname 11

OR

aggr create aggrname 12
aggr add -g new aggrname 11 #-g forces a new raid group with the next 11 drives... prior to 7.3 -g is supported but only for existing raid groups..explicity use "new" as the raid group name and it will create a new raid group

cebulrdcis · ‎2008-09-25

thanks scott..

I try this one in our new 3 x 1TB shelf for our Secondary storage (FAS3020)

once again thank you..

rkaramchedu1 · ‎2008-10-15

The more I read this thread, the more confusing it becomes.

Let me stick to PRE-7.3 for a second.

The table I saw above says optimal RAID group size for a 1TB SATA Drive is 17. First off, for SATA, the max RAID group size allowed is 16. So 17 is not even a workable option.
Aggregate Size. From my readings, work and understanding, the total size of an aggregate was computed by adding together the raw size of all disks in the aggregate, including parity disks, and regardless of the amount of disk space available to be used in each data disk. What I am now confused about now is the use of the term "raw size". I believe it is the "physical capacity" might be the correct term. More specifically, it is the output from the "sysconfig -r" command from under the "Phys (MB/blks)" column.

For e.g. for a 300GB FC Disk Drive, the Physical Size is 274845 MB = 268.40 GB. So for an aggregate limit of 16TB, the total # of drives = 16 * 1024 / 268.40 = 61 drives (parity and data together).

Am I correct ?

cebulrdcis · ‎2008-12-12

For your reference,

http://now.netapp.com/NOW/knowledge/docs/ontap/rel73/html/ontap/rnote/rel_notes/concept/c_oc_rn_feat73_aggr-size.html

Aggregate size calculation changes

In Data ONTAP 7.3 and later releases, the total size of an aggregate is computed using the usable size of its data disks rather than the raw size of all of its disks.

In previous releases of Data ONTAP, the total size of an aggregate was computed by adding together the raw size of all disks in the aggregate, including parity disks, and regardless of the amount of disk space available to be used in each data disk. This method of computing the aggregate size could result in an aggregate exceeding the maximum allowable size even though the amount of usable space in that aggregate was much smaller than the maximum allowable aggregate size.

In Data ONTAP 7.3 and later releases, only the "used size" (as reported by the sysconfig -r command) of data disks is used to calculate the total aggregate size. If you were prevented from adding disks to an aggregate due to aggregate size constraints in an earlier version of Data ONTAP, you might be able to add more disks to that aggregate after upgrading to Data ONTAP 7.3.

Maximum data drives per 16-TB aggregate
With the aggregate size calculation changes present in Data ONTAP 7.3, you can include more data drives in an aggregate without exceeding the aggregate size limit.

amiller_1 · ‎2008-12-13

Ah....right....I should have remembered the emphasis on data disks.

For anyone following along, in my example above, with 7.3 I could now have 45 disks total (i.e. 3 RAID groups with (15) 500 GB drives in each -- that would be 39 data disks and 6 parity/dual parity disks).

Raid group size recommendation

Aggregate size calculation changes