First CDOT install - FAS2520 with ADP and Data Aggr spares?

[SherIT]

Hi,

My first post here and first time with CDOT (8.3).

System:

2520 HA with 12x900SAS

1 x Shelf with 12x900SAS & 4x400SSD

Ran through System Manager and did not assign drives which has given me a 50/50 split across the 2 nodes resulting in each controller having;

6x836GB (Shelf)

6x693GB (ADP partitions)

2x372GB (Shelf SSD)

So I would like to assign both the 6 Shelf drives and the 6 ADP partitions to the same aggregate ideally. When I create my first aggr it shows that I can have 6 drives from the ADP Partions on either controller which is setup with RAID-DP (which I would like) but this shows that there are no assigned spare drives and does not let me adjust this in System Manager. If I choose the shelf drives then it only lets me have 5 of the 6 with an assigned spare which is what I woudl expect form the ADP partitions as well. Am I missing something here? I know ADP onyl assigns 5 of the 6 drives to the root aggr, does this mean that the remaining spare is actually a spare for all ADP based drives as opposed to just a spare for ADP based aggrs? There fore the spare is taken care of already and I should just stop worrying and build  away?

Thanks for reading and fingers crossed you can put my mind to rest.

Thanks,

James

[aborzenkov]

Whole disks and partitions must be in different raid groups, so you won't lose capacity.

[SherIT]

OK, the strage thing is it lets me create a RG with the ADP drives and then lets me add capacity to it with the shelf drives without a second RG;

I would rather not split the ADP and Shelf drives in to different raid groups on eahc controller to conserve space if I can but preferable to me would be to use the option of having all ADP partitions on one controller and all shelf drives on the other but just not sure how this will react to an ADP drive loss

[scottgelb]

What is the cli output of aggregate show -instance ?  also run node run * sysconfig -V

It works to mix the drives but you get right sizing in that raid group.

[scottgelb]

The drives are assigned equally between the nodes... haven't tried and don't think we would try to reassign a partition...since root is already on the partition then leaving a data partition for the data aggregate.. 

Typically I setup a RG if just ADP per node (leaving 0-2 spares depending on additional shelves or not), then add an additional raid group of physical non-partition drives.

[scottgelb]

It is a little confusing, but if only ADP partitioned drives, then you should leave a virtual spare for the data partitioned aggregate.  For storage pool with SSD for flashpool, you leave a physical spare since the 4 allocation units don't include a virtual spare.  For ADP, if you have additonal shelves outside of ADP with the same drive type, you could go without an ADP spare on the data paritioned aggregate since you have a physical spare available.  I think Skip's preso at Insight this year showed an example of 3 shelves and no virtual spare. 

In your case, you have 6 partitoned drives on each controller and no physical drives that are not partitioned...so I would create aggregates of 4 or 5 drives to leave 1 or 2 spares...1 is fine to match the root aggr partition.  How many total HDD (12?) and  SSD (only 2?) do you have?

[SherIT]

Hey, thanks for replying!

So if I create an aggr of 5 drives on each contoller with the partitioned drives it will leave 1 not used, will this act as a spare if one of my partitioned drives failes then? Just a little concerened in the way theta System Manager shows 0 spares when I create this:

In total we have 12 partitioned drives in the filer plus 12 in the shelf and 4 SSDs which I would like to create a flash pool for cashing hot data. Can I add drives from the shelf in to the same aggregate as the partitioned drives after creating it?

Thanks again,

James

[scottgelb]

correct that will leave one virtual spare..but don't see disk choice and if it will grab phsyical or virtual... you can choose the -disksize parameter to force it to the slightly smaller partitioned drives.  And as long as you have a physical spare, you may go without a virtual spare...

[SherIT]

Thanks Scott, that helps alot. Just to make sure I am understanding correctly, by allocating all 6 of the partitioned drives to a RG and then assinging 5 out of 6 in the shelf RGs on each controller the shelf spare will effectively be global for both raid groups and will fill in for an ADP drive if one fails as well?  If so, I will do some testing to sanity check everything before any data goes on there, I know ADP doesn't span shelves but would be good if it can utilise a shelf disk in a failure scenario and give me some precious GBs back!

[scottgelb]

Correct.. a spare is good to replace a physical or partitioned drive.  I would prefer 2 spares though but you have so few disks so many may disagree with that.  Remember that StoragePools though don't have a virtual spare so make sure only to use 3 of the SSD in the storagepool.

[SherIT]

Hey,

I have been experimenting with the raid groups and creating aggregates today and am not sure how I want to proceed at the moment. These are the options I have come up with if I split drives evenly across controllers:

1. config is per controller;

6xADP drives to create aggr, add capacity of 5xshelf drives into same RG - This I think treats the shelf drives as the same size as the ADP drives meaning I lose 150GB from each of the shelf drives as well

= 1 spare, 2 parity and 9 x 693.22GB == 5.4TB Aggregate

TOTAL Storage 10.8TB

2. config is per controller;

5xshelf drives to create aggr, add capacity of 6xADP drives (have to be in sep RG due to smaller size)   - as the aggr is created with the larger shelf drives the ADP drives have to go in to a separate RG.

= 1 spare, 2 ADP parity, 2 Shelf Parity. 3 x 836.91GB + 3 x 693.22GB == 4TB Aggregate

TOTAL Storage 8TB

Clearly option 1 is more efficient even if I do lose 150GB per drive!

Now, what are my options if I dont do an even split? Can I assign;

12xshelf drives to one controller = 1 spare, 2 parity. 9 x 836.91GB == 6.6TB in Shelf aggr

12xADP drives to other controller = 1 spare, 2 parity. 9 x 693.22GB == 5.4TB in ADP aggr

TOTAL Storage 12TB

Does the above work as far as spares? (i.e. can the ADP spare cover one of the drives that holds the root aggr/partition on either controller?). Are there any other suggestions to maximise storage without losing RAID-DP and spares?

Thanks,

[scottgelb]

Just reread...you have 12x partitioned... so 6x per node then 12x extra physical disks and 4x SSD. 

I would create a paritioned aggregate of 5 paritioned drives on each node and leave a virtual spare per node

Then you have a choice on the 12x physical drives...assign 6 to each node, or all 12 to one node... either add a new raid group of physical drives to the partitioned aggregate, or a new aggregate.  But depends how much space you need per aggregate.  Make sure to leave at least 1 physical spare...usually 2 but on this small of a system that might not make sense.

Create a storage pool of 3x SSD leaving 1 physical spare..you get by default 2 allocation units per controller and can override that default if you need 3 or 4 on one node instead.