Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

gmilazzoitag · ‎2012-01-25

Looking and reading at documentation regarding usage of ATTO SAS-FC bridge on Metrocluster I've always read about the need to have 4 bridges (as in the pic here) to connect and double the path to the stack SAS up to completation of one of this stack (seven DS4243 shelves).

I should setup a fabric MC where I will use both SAS and SATA disks in my DS4243 shelves: exactly as in the pic with one site with both 1xSAS+1xSATA shelves and the second site only with 1xSAS.

It now happens that configuring in PE/CE the needed hardware it give me EIGHT bridges for somewhere somebody says that this is best practice to have separate stack for SAS and SATA...what?!? This violates the principle that the SAS backend can do: a multiplexed one of four 3 Gbps links each

As far I know a stack SAS can be formed by both SAS and SATA disks mixing the DS4243 (up to ten shelves) and the ATTO manual too repeat this. No issue at all.

Why should my customer but other 4 ATTO bridges (9000 € each price list!)?

Regards

radek_kubka · ‎2012-01-25

Hi,

I think technically speaking you are correct.

PartnerEdge always used to have its own ways of 'best practices' around different things, like throwing in extra SAS HBAs on 3200 platforms.

I believe you can always build MetroCluster setup manually (i.e. as a non-MC configuration) & add additional components via "Add parts" - just run the outcome through your NetApp resource to make sure they are happy with that as well!

Regards,

Radek

gmilazzoitag · ‎2012-01-25

Hi,

thanks for prompt answer. I agree with you but I would prefer that this kind of answer come from some NetApp representative.

pspringfield · ‎2012-01-27

We recently raised exactly the same query with the NetApp Partner Helpdesk this week and received the reply below.

As an observation, if there is any single person within the channel whose advice I would trust, it is Radek.

Hope this helps,

Pete Springfield

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Question -

.... The challenge is that the specification it spits out includes 8 of the new SAS Bridges, my assumption is that you can mix both SAS and SATA disk within a stack (pg 19 of the attached PDF) and would only require 4 of the SAS bridges in total.

Answer -

yes, that's correct. SAS and SATA shelves can be mixed in the same stack, which would require only four SAS Bridges in total.

According to the MetroCluster Product Frequently Asked Questions,
MC (DOT 8.1.0 release) do support SAS and SATA drives in separate DS4243 shelves (but in the same stack). However, this is not supported as a configured system at POS. If someone wants a combination of SAS and SATA drives with DS4243 the same stack, they will have to order an MC configuration with SAS drives and then order DS4243 with SATA drives as add-on or vice-versa.

I hope this answers your question.

Kind regards

NetApp SE Helpdesk
Phone: +44 207 280 3754
partner-helpdesk-uk@netapp.com
www.netapp.com

gmilazzoitag · ‎2012-01-27

No doubts on Radek answer but the "official" channel is what I need to convince account that plays with configurator that I don't need all that stuff!

Many thanks

pspringfield · ‎2012-01-27

I'm with you there.

All the best, Pete

MUNSTERENYA · ‎2013-02-07

I'm in the process of buying a metro cluster solution. Today we've both SAS and SATA disks in the system. Our NetApp Partner advice us not to mix SAS and SATA in the same stack and came up with a solution including 8 ATTO 6500N bridges (exactly as the drawing in the gmilazzoitag posted earlier in the thread). But looking to the tech. spec the bridge has 2 SAS ports A & B. Can't I use both SAS ports in the bridge? port A for a stack of SAS disks and port B for a stack of SATA disks. Then it should be enough with 4 bridges. Or is that impossible?

Regards

aborzenkov · ‎2013-02-07

No, you cannot use both SAS ports. You need 2 bridges per loop on each site.

MUNSTERENYA · ‎2013-02-08

OK, thanx for answer. Do you have any explenation why I can't use both ports? In what situation is it possible to use both ports then?

aborzenkov · ‎2013-02-08

No, I do not have explanations. That’s what is supported by NetApp.

radek_kubka · ‎2013-02-08

There is even more interesting question here - why create separate SAS & SATA stacks?

In theory mixing different disk types in the same stack is supported, but there is a very vague recommendation from NetApp about "minimising the number of cross-over points". So if we have a mixed stack ending with the "wrong" disk type, a future expansion may be problematic due to creating a second cross-over point.

That being said, I haven't seen any firm statement saying 1 cross-over is fine, but 2 (or 3, or 4) are bad.

Any views on it?

thomas_glodde · ‎2013-02-08

Back in the days of FC backends you where forced to use seperate stacks for FC & S-ATA, so some ppl tend to stick with it and so does the QuoteEdge. Usualy a stack should be owned by one filerhead only as well. Both of these recommendations do not stand for SAS cabeling anymore. The reason why you cannot use both ports on a bridge is that ONTAP cannot handle more than 2 pathes to a single disk and for failover purposes if one of the bridges dies, so 4 is the minimum. Be aware that disk autoassign wont work in such a configuration, at least not now, maybe they will fix it in a newer ontap version.

radek_kubka · ‎2013-02-08

Both of these recommendations do not stand for SAS cabeling anymore.

Well, that's what common sense tells us.

However, there is this paragraph in Storage Subsystem FAQ:

"HOW MANY CROSSOVER POINTS BETWEEN SATA AND SAS CAN I IMPLEMENT IN MY DS4243 STACK?

The term “crossover point” refers to the number of times a DS4243 storage shelf makes the transition from

SATA to SAS (or vice versa) within a single stack. Multiple crossover points per stack are supported.

NetApp recommends minimizing the number of crossover points when initially deploying a storage

configuration. As storage configurations grow, it is likely that additional crossover points will be introduced

into the storage configuration. This is acceptable; however, by reducing the number of crossover points in

your stacks, you reduce the variability of the storage configuration. As is true in all computer systems today,

increased variability can add difficulty when troubleshooting issues and also raises the possibility of

encountering unknown issues."

As woolly, as you can get

thomas_glodde · ‎2013-02-11

When using DS4243 disk shelves, electronics etc. are the same, so mixing shouldnt be an issue here. Besides SAS is a point to point protocol, so even mixing within 1 shelf shouldnt be an issue (other vendors do so).

But you do not need more then one crossover point anyway if you choose your shelf ids wisely. You can have 10 shelves in a stack and shelfids should be unique, so start with 10, 20 and 30.

10 SAS

20 S-ATA

30 S-ATA

if you expand:

8 SAS

9 SAS

10 SAS

20 S-ATA

30 S-ATA

31 S-ATA

32 S-ATA

So only 1 crosover needed.

radek_kubka · ‎2013-02-11

SAS is a point to point protocol, so even mixing within 1 shelf shouldnt be an issue (other vendors do so).

Couldn't agree more - one million dollar question: why NetApp can't (don't want to) do it?

you do not need more then one crossover point anyway if you choose your shelf ids wisely.

In an ideal world - yes. But usually you have all types of constraints, like physical space in a rack, cable lengths, etc. Or you just have one "mixed" stack & can't introduce another one (e.g. due to having only 4 ATTO bridges ). I agree multiple cross-over points shouldn't make any difference for the overall health of the solution, yet this statement in the Storage Subsystem FAQ hints it is somewhat treated with suspicion.

thomas_glodde · ‎2013-02-12

For SAS/S-ATA i could image they dont want to add a gazillion possible options to the quoting edge tool resulting in less experienced users creating not field feasible solutions (as happens still alot with FAS2220 machines, eg 12 disks with snaplock in an ha-pair)

For the crossover stuff, i think they just dont have any experience yet if it causes errors or not or maybe they have a problem with acp in that mix.

klikar1982 · ‎2013-05-23

Hi All,

I have a little bit different view why is netapp recommending 8 bridges.

I can not provide direct link to documentation, but as far as I remember, netapp best practice recommends to separate local and remote disk pools into different stacks. So because we are talking about metrocluster configuration which should utilize 2 disks pools in 2 stacks and you need 2 bridges per stack for redundancy, you end up with 4 bridges per location, 8 bridges for whole metrocluster solution.

In our production environment, we have installations with only 4 bridges, so we are mixing disks for different pools in 1 stack.

thomas_glodde · ‎2013-05-24

kil, that was for old DS14 FC AL topologies. In regards of new SAS Technology and ATTO SAStoFC bridges mixing stacks/pools etc is totaly fine & supported.

aborzenkov · ‎2013-05-24

Well, there is difference between “supported” and “recommended”. And NetApp quoting tool does not necessary agrees with either, adding third possibility ☺

thomas_glodde · ‎2013-05-24

sure, its recommended to use 8 atto bridges which adds like 25,000$USD to your configuration compared to 4 bridges 😉

radek_kubka · ‎2013-05-24

Yeah, presumably it all comes from this cryptic recommendation to 'minimize the number of crossover points' I quoted above.