How many disks can be connect to a SAS loop?

[renqingrong]

I know that  max 84 FC disks can be connected per FC loop,

How many disks for SAS loops?

for example if  FAS3170A is with 840 disks, How many SAS HBA need?

[rkaramchedu1]

For SAS (DS4243), NetApp uses the term stack, as opposed to a loop (loop is a term used for DS14 disk shelves)

With the exception of FAS2XXX systems, each SAS stack supports a maximum of 10 disk shelves, which brings the # of disk drives to 240.

I believe for FAS2XXX, the limit is 4 DS4243 per stack, which brings the # of disk drives to 96.

For a FAS3170A with 840 disk drives, you'll need 3.5 loops (4 loops). So per controller, you'll need 8 ports (4 x 2), which is 2 quad port SAS HBAs per controller.

[amiller_1]

He....was just discussing this with a customer earlier this week. Some details are.... (a lot overlapping with the previous post)

• Loop --> Stack.
• 84 Disks/6 shelves in an FC loop (related to maximum devices in an arbitrated FC loop (i.e. FCAL) and the loop speed ran at 4 Gig FC maximum. With MPHA that worked out to 8 Gig FC for the disk loop backend. Given you needed 2 FC ports per 84 disks, you'd very quickly become slot-bound on the mid-range 31xx boxes (like really, really quickly....was hard to max out the 3160 or 3170 for instance if you needed any other expansion cards -- 10 GigE, PAM, etc.)
• 240 disks/10 shelves in a SAS stack (i.e. just about 3x disks per stack as an FC loop). Each SAS port actually provides 4 lanes running 3 GigE per lane -- i.e. 12 Gig per connector. With MPHA that's 24 Gig per stack (i.e. again about 3x the bandwidth...which makes perfect sense ).
• Given a quad-port SAS HBA, you can handle 2 x SAS stacks -- 480 disks. You do want to make sure to follow the System Configuration Guide for putting the SAS cards in (as 3 of the 31x0 slots are higher speed than the 4th slot...just follow the guide for slot order and you'll be golden).
• Given (2) SAS HBA's, you can now max out even a 3170's disk count with just 2 cards....leaving 2 slots open for 10 GigE or PAM....quite wonderful really.
• You can't mix SAS & SATA drives inside the same shelf but you CAN mix them inside the same stack -- this is huge and much more flexibible than before (should only do one transition between SAS and SATA within a stack).
• The DS4243 do have an Alternate Control Path -- ACP -- which runs over a separate GigE link -- the onboard 31x0 ports are great for that (especially if doing 10 GigE). ACP isn't required but recommended.
• The 2040 can only handle (4) DS4243 shelves for 96 disks....quite the odd limitation given that the 2040 maxes out at 136 disks (you have to use some DS14 shelves to hit that basically).
• All of this in the docs on NOW of course....

[rkaramchedu1]

Great post - added a lot of clarity to my response. Thank you!

You can't mix SAS & SATA drives inside the same shelf but you CAN mix them inside the same stack -- this is huge and much more flexibible than before (should only do one transition between SAS and SATA within a stack).

I think this point itself might warrant a separate thread. This ability of mix-n-match SAS/SATA shelves within a stack might well bring forth it's own set of interesting scenarios. While I do agree that they'll support mixing and matching shelves in a stack - I am sure there are some best practices and some more will come out.

(sort of like, they gave us the rope, we can chose to use it or hang ourselves with it )

For e.g. 1 filer, 3 DS4243 SATA and 3 DS4243 SAS Shelves. Which is better in the following scenarios?

1. Create two stacks and put 3 SATA in one and 3 SAS in the other
2. Create 1 stack and group 3 SAS together and 3 SATA together (one crossover point between SAS and SATA)
3. Create 1 stack and randomly stack disk shelves (SAS+SATA+SAS+SATA+SAS....)

If ports are not an issue (yeah - right!), then the order might be 1), 2). But if they are, then I would reckon that 2) would be more preferable than 3) and I also recall reading somewhere about this.

[scottgelb]

The recommendation is to NOT switch switch drive types more than once in a stack. So SAS --> SAS --> SATA then you should only add sata after... It would be interesting to see if support will let us hot break the middle of a stack connection to insert shelves in the middle to keep disk types contiguous in the stack... however this often isn't feasible since shelf-ids are already set and shelves likely have to move in the rack.

I often keep all FC/SAS on one head and all SATA on the other head in configurations...we have several customers with high performance needs and have sometimes had issues where the CP (consistency ever 10 seconds or nvram half full) to FC drives have higher latency since they have to wait for the CP to complete to all drives.  For many this isn't an issue and perfstat is the best way to see what is happening (and the GSC performance team does a great job helping identify this edge case).

[amiller_1]

Yeah....the internal guides were clear about the best practice being to only have one SAS/SATA transition in a stack but did leave open what exactly happens if you don't (I'm more wondering for curiosity....the best practice is quite clear.

What I'm thinking of right now actually is to put the controller at the bottom of top or bottom of the rack and put the disk shelves in the middle -- so you can either grow up with one type of disk or grow up with another type of disk (I already will do the same thing in DS14 racks actually). The whole hot-break question is very interesting though.

Alternatively, there are longer SAS cables and the only requirement is that the shelves be connected up in the right order....which doesn't necessarily have to correspond to their physical location in the rack. That would lead to messier cabling but does ultimately give enough flexibility for almost any scenario I think.

What will be interesting though is the physical rack space -- if you d a 31x0 system, you're at 6U already. Given a standard 42U rack, that only leaves you with enough space for 10 DS4243 shelves....makes me think we'll see the first SAS stack often be 9 shelves. The next stack (in an adjacent rack) will grow up to 10 shelves/40U.

[adamfox]

Only one switch is the recommendation, but it's not required.  It will work fine with multiples.  I checked internally as well.

[amiller_1]

Hmm....very good to know. So then...if my choice is between multiple transitions (i.e. existing environment where not laid out well physically) vs. a nasty cabling mess, which is practically better?

[dominicb]

Hi Adam,

I can find plenty of statements that there should be only one transition from SAS to SATA in a stack but I can't find anything to tell me why that is.

If this is not required why is it recommended?

Thanks
Dominic

[mcope]

Unfortunately, Adam Fox is no longer with NetApp.

At the time he wrote this comment, the only supported configuration was one crossover per SAS stack.  A few months ago that limitation was removed.

It's still a best practice to avoid mixing SAS and SATA in a stack when you can and limiting the number of crossovers when you can't.  You always want to try to design disk hardware and software ownership to segregate I/O for slow drives from the I/O for fast drives.  Think of it like allowing a Toyota Prius cars into a NASCAR race - at some point the Prius is going to create a chokepoint and slow everyone behind them.

[dominicb]

Thanks Michael

[mcope]

All the documents and diagrams I've seen for systems with only SAS shelves have the controller about 1/2 - 3/4 of the way up in the cabinet (e.g, U24+ rather than U19+).  Placing the controller in the top of the cabinet may give you a few more feet/meters on the SAS cable, but then you have to stand on a ladder to do system maintenance.  Placing the controller at the bottom of the cabinet is never a good idea because the controller(s) become the world's most expensive vacuum cleaners (and I've seen systems complain about being too cold if they are too close to the A/C outlets)

[amiller_1]

He he...expensive vacuum cleaners. If anything, that feels like a blinding statement of the obvious....blinding because it seems stunningly obvious but it never crossed my mind until you mentioned it.

I'll have to ponder where I want to stick controllers now....I kind of like the 3/4 up the rack for easy maintenance/installs/card adds...so hmm....hmm....