New system setup advice

[chris_davis]

I just installed a FAS2240-4 along with 3 DS4246 shelves full of 2 TB SATA disks.

This filer is intended to be used as a SnapMirror destination for my production filer.

My production filer is a 2240-4 with 1 DS4246 shelf.  The filer has 24 2 TB SATA disks carved out for Exchange 2010 and our file servers.  26.19 TB total.
The DS4246 shelf has 24 600 GB SAS disks used for my VMware servers. 8.65 TB total.
One controller of the filer is over the filer's disks, the other controller is for the shelf.

Previously I was backing these up to an older 2040 with one shelf but I replaced that with the 2240 and shelves mentioned above.

I'm looking for advice on how to split up the available space.

The filer is currently configured with half of the filer's disks assigned to one controller and the other half to the other controller.

This is what I'm thinking:

Controller A: Assigned to all of the filer's disks and 2 shelves.  This will give me roughly 72 TB of disk space after RAID-DP and 2 spares are taken out.  I will use this space to receive data from the SATA aggr on the source filer (Exchange and File Server data).

Controller B: Assigned to the 3rd shelf.  I will get about 24 TB from this shelf which will be for my VMware SnapMirror.

What is the best method of reassigning my disks?

The disks for my shelves are currently unassigned and the filer's disks are split between the two controllers.

Is there anything I'm missing that I'm not thinking about?

[resqme914]

You seem to have enough disks so that you don't have to play games trying to maximize useable space.  What I would suggest is that you configure your disks so that you can take advantage of both controllers which is what you seem to have planned.  It looks good.

To reassign the disks , use "disk assign -s unowned" on the controller where a disk is currently owned.  Once you have unassigned all the disks you want to move to the other controller, just go into the other controller and do a "disk assign all".  HTH.

[chris_davis]

What are your thoughts, for Controller A, assigning the filer's disks and the disks from two of the shelves to one aggregate and letting the filer create the needed raid groups?  Will this give me enough protection against disk failure?  Would this help or hurt performance?

[resqme914 · Accepted solution]

I tend to prefer minimal number of large aggregates to having lots of smaller aggregates.  With your environment, which is fairly small, I would say have just one aggregate on controller A (assuming they're all SATA).  Having more disks in the aggregate will give you better performance, in general.

In terms of raid groups (raid-dp), I also tend to go larger and trying to maximize my space.  If I had 72 SATA disks, I would have one raid-dp aggregate, with rg size 17, which will leave 4 spares.  If you don't wish to leave that many spares, you can use rg size 18, but leave two of the raidgroups with only 17 disks each.   HTH.

[chris_davis]

That's exactly what I'm working on now.

I tried creating one aggregate out of my 72 disks but it looks like there's a limit of a 43 disk aggr (60 TB) on the 2240 filer.  Also, SATA seems to limit RGs to 16 disks, so for a 24 disk shelf, I would need to create 12 disk RGs to prevent my RGs from spanning multiple shelves.  Does that sound correct?

I've been at this all day reconfiguring my other controller (just 24 disks, which ended up being 14 and 8 disk RGs (because I didn't specify RG size) with 2 spares) so I'm a little fuzzy but I'm thinking that I should limit myself to one aggr per filer or shelf using 2 raid groups and leave 2 disks out for the spares.  That will give me 6 spares but I'm not sure if I can do any better.

[chris_davis]

Instead of taking the word of another forum post I looked it up myself:

FAS2240

Max Aggr size: 120 TB

Max RG size: 20 (18 + 2 parity) ... which leaves each shelf with 4 spares??

I'm going to work on this tomorrow to see if I can reorganize the disks on this filer so that I can blow out most of the raid groups while still keeping the OS intact and rebuilding it using 20 disks per RG.

[resqme914]

I'm sorry I didn't check the limitations of the 2240.  SATA also has lower number of disks per raidgroup, but I thought it was 20.  Maybe that's just for larger FAS models.  Anyway, check the NetApp Hardware Universe for the limits of your model.  Try not to have unbalanced raid groups; i.e. the other controller you mentioned has a 14 disk rg and an 8 disk rg...  I would redo that if possible.

Also, I wanted to ask you... why are you averse to spanning shelves?  There's really no reason why that should be an issue.  The disks comprising a raidgroup can be on different shelves (obviously assuming symmetry).  As a matter of fact, in a very large configuration, one could argue that the best design would have one disk per shelf comprising its raid groups.

[chris_davis]

Regarding spanning RGs on multiple shelves, I've read on several other forum posts to avoid doing that due to degraded performance.

If an 8/14 RG split is really that bad I'll work on getting that corrected today.  It won't be an issue at all, just time consuming.

According to this document:

http://www.netapp.com/us/products/storage-systems/fas2200/fas2200-tech-specs.aspx

I can have a maximum of 20 disks per RG (18 +2 for parity) and a max of 120TB for my aggregate.

From what I'm reading, this seems to be the preferred method:

Filer, Shelf 1, and Shelf 2 = RG of 9+2, and 10+2 = 23 disks and 1 hot spare per storage node

Total of roughly 57 TB after RAID penalty, etc. and 3 hot spares

Or

Filer = 2 RGs of 10+2 = 24 disks

Shelf 1 = 2 RGs of 10+2 = 24 disks

Shelf 2 = 2 RGs of 9+2 = 22 disks

Total of roughly 58 TB after RAID penalty, etc. and 2 hot spares

I think I'm going to go with Option 2 since I have 4-hour support on these.

I've read that going past 14 disks per RG actually hurts performance, even though you have more spindles.

Do you see anything I'm missing?

[radek_kubka]

Regarding spanning RGs on multiple shelves, I've read on several other forum posts to avoid doing that due to degraded performance.

Where exactly? I was always under impression it really doesn't matter from performance point of view. It can make a difference though if someone plans to remove some shelves from a setup in a future - keeping an aggregate within a shelf (or shelves) boundaries makes it much easier.

Radek

[resqme914]

IF... the max aggr size on your model is 120TB, and IF the max rgsize of SATA is 20...  I'd stick to my recommendation of rgsize 17 (or 18).  You'd have to have two aggregates so as not to exceed the 120TB limit.  You can either do two equal size aggregates composed of two rg's each (of size 17 or 18 disks),  Or you could create one aggr with three rg's, and a second, smaller aggregate with one rg.  I would stop worrying about keeping the aggregates/raid groups within shelf boundaries.  The only reason you'd do that is if you planned to physically move them in the future. 

[chris_davis]

After going back and researching it's really not a big deal to span shelves.  In fact, it's almost recommended.

[chris_davis]

What do you think of this....

Since this is going to be used as a backup filer, I analyzed the space being used by the source resources.

Exchange is using about 13 TB, and our file servers, when moved over, will consume about 10 TB.

I have 72 2TB disks and decided on four 17-disk raid groups leaving me with 4 hot spares.

aggr create aggr_exchange -r 17 34

aggr creeate aggr_fileserver -r 17 34

Obviously, my RGs are being forced to span multiple shelves which I've put research into and it's a non-issue and even recommended to split them up.

Now, in order to get to that point, I had to create a small 3 disk aggr to put the root volume on.  These 3 disks are now needed so I can use them as hot spares but on which aggr should I move the root vol?  I know it doesn't really matter, but just out of curiosity where would you put yours?

Also, the NetApp Usable Space Calculator recommends my RGs to be three 18-disk RGs and one 14-disk RG.  I know it's just a tool a user wrote, but it seems odd that it wouldn't try to get the user to spread out their RGs evenly.

Thanks a ton for your help.  I will make sure to mark your answers as Correct and Helpful if possible.

[resqme914]

I'd put the root vol into the "fileserver" aggr just because that's probably going to be the less busy aggregate.