topic Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks in ONTAP Discussions

Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

gmilazzoitag — Thu, 05 Jun 2025 06:36:39 GMT

Looking and reading at documentation regarding usage of ATTO SAS-FC bridge on Metrocluster I've always read about the need to have 4 bridges (as in the pic here) to connect and double the path to the stack SAS up to completation of one of this stack (seven DS4243 shelves).

I should setup a fabric MC where I will use both SAS and SATA disks in my DS4243 shelves: exactly as in the pic with one site with both 1xSAS+1xSATA shelves and the second site only with 1xSAS.

It now happens that configuring in PE/CE the needed hardware it give me EIGHT bridges for somewhere somebody says that this is best practice to have separate stack for SAS and SATA...what?!? This violates the principle that the SAS backend can do: a multiplexed one of four 3 Gbps links each

As far I know a stack SAS can be formed by both SAS and SATA disks mixing the DS4243 (up to ten shelves) and the ATTO manual too repeat this. No issue at all.

Why should my customer but other 4 ATTO bridges (9000 € each price list!)?

Regards

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

radek_kubka — Wed, 25 Jan 2012 12:53:33 GMT

Hi,

I think technically speaking you are correct.

PartnerEdge always used to have its own ways of 'best practices' around different things, like throwing in extra SAS HBAs on 3200 platforms.

I believe you can always build MetroCluster setup manually (i.e. as a non-MC configuration) & add additional components via "Add parts" - just run the outcome through your NetApp resource to make sure they are happy with that as well!

Regards,

Radek

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

gmilazzoitag — Wed, 25 Jan 2012 13:50:25 GMT

Hi,

thanks for prompt answer. I agree with you but I would prefer that this kind of answer come from some NetApp representative.

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

pspringfield — Fri, 27 Jan 2012 12:05:50 GMT

We recently raised exactly the same query with the NetApp Partner Helpdesk this week and received the reply below.

As an observation, if there is any single person within the channel whose advice I would trust, it is Radek.

Hope this helps,

Pete Springfield

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Question -

.... The challenge is that the specification it spits out includes 8 of the new SAS Bridges, my assumption is that you can mix both SAS and SATA disk within a stack (pg 19 of the attached PDF) and would only require 4 of the SAS bridges in total.

Answer -

yes, that's correct. SAS and SATA shelves can be mixed in the same stack, which would require only four SAS Bridges in total.

According to the MetroCluster Product Frequently Asked Questions,
MC (DOT 8.1.0 release) do support SAS and SATA drives in separate DS4243 shelves (but in the same stack). However, this is not supported as a configured system at POS. If someone wants a combination of SAS and SATA drives with DS4243 the same stack, they will have to order an MC configuration with SAS drives and then order DS4243 with SATA drives as add-on or vice-versa.

I hope this answers your question.

Kind regards

NetApp SE Helpdesk
Phone: +44 207 280 3754
partner-helpdesk-uk@netapp.com
www.netapp.com

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

gmilazzoitag — Fri, 27 Jan 2012 12:17:30 GMT

No doubts on Radek answer but the "official" channel is what I need to convince account that plays with configurator that I don't need all that stuff!

Many thanks

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

pspringfield — Fri, 27 Jan 2012 17:25:42 GMT

I'm with you there.

All the best, Pete

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

MUNSTERENYA — Thu, 07 Feb 2013 12:29:06 GMT

I'm in the process of buying a metro cluster solution. Today we've both SAS and SATA disks in the system. Our NetApp Partner advice us not to mix SAS and SATA in the same stack and came up with a solution including 8 ATTO 6500N bridges (exactly as the drawing in the gmilazzoitag posted earlier in the thread). But looking to the tech. spec the bridge has 2 SAS ports A & B. Can't I use both SAS ports in the bridge? port A for a stack of SAS disks and port B for a stack of SATA disks. Then it should be enough with 4 bridges. Or is that impossible?

Regards

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

aborzenkov — Thu, 07 Feb 2013 12:32:45 GMT

No, you cannot use both SAS ports. You need 2 bridges per loop on each site.

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

MUNSTERENYA — Fri, 08 Feb 2013 08:23:04 GMT

OK, thanx for answer. Do you have any explenation why I can't use both ports? In what situation is it possible to use both ports then?

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

aborzenkov — Fri, 08 Feb 2013 08:30:43 GMT

No, I do not have explanations. That’s what is supported by NetApp.

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

radek_kubka — Fri, 08 Feb 2013 11:32:41 GMT

There is even more interesting question here - why create separate SAS & SATA stacks?

In theory mixing different disk types in the same stack is supported, but there is a very vague recommendation from NetApp about "minimising the number of cross-over points". So if we have a mixed stack ending with the "wrong" disk type, a future expansion may be problematic due to creating a second cross-over point.

That being said, I haven't seen any firm statement saying 1 cross-over is fine, but 2 (or 3, or 4) are bad.

Any views on it?

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

thomas_glodde — Fri, 08 Feb 2013 15:19:31 GMT

Back in the days of FC backends you where forced to use seperate stacks for FC & S-ATA, so some ppl tend to stick with it and so does the QuoteEdge. Usualy a stack should be owned by one filerhead only as well. Both of these recommendations do not stand for SAS cabeling anymore. The reason why you cannot use both ports on a bridge is that ONTAP cannot handle more than 2 pathes to a single disk and for failover purposes if one of the bridges dies, so 4 is the minimum. Be aware that disk autoassign wont work in such a configuration, at least not now, maybe they will fix it in a newer ontap version.

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

radek_kubka — Fri, 08 Feb 2013 16:34:00 GMT

Both of these recommendations do not stand for SAS cabeling anymore.

Well, that's what common sense tells us.

However, there is this paragraph in Storage Subsystem FAQ:

"HOW MANY CROSSOVER POINTS BETWEEN SATA AND SAS CAN I IMPLEMENT IN MY DS4243 STACK?

The term “crossover point” refers to the number of times a DS4243 storage shelf makes the transition from

SATA to SAS (or vice versa) within a single stack. Multiple crossover points per stack are supported.

NetApp recommends minimizing the number of crossover points when initially deploying a storage

configuration. As storage configurations grow, it is likely that additional crossover points will be introduced

into the storage configuration. This is acceptable; however, by reducing the number of crossover points in

your stacks, you reduce the variability of the storage configuration. As is true in all computer systems today,

increased variability can add difficulty when troubleshooting issues and also raises the possibility of

encountering unknown issues."

As woolly, as you can get

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

thomas_glodde — Mon, 11 Feb 2013 15:54:51 GMT

When using DS4243 disk shelves, electronics etc. are the same, so mixing shouldnt be an issue here. Besides SAS is a point to point protocol, so even mixing within 1 shelf shouldnt be an issue (other vendors do so).

But you do not need more then one crossover point anyway if you choose your shelf ids wisely. You can have 10 shelves in a stack and shelfids should be unique, so start with 10, 20 and 30.

10 SAS

20 S-ATA

30 S-ATA

if you expand:

8 SAS

9 SAS

10 SAS

20 S-ATA

30 S-ATA

31 S-ATA

32 S-ATA

So only 1 crosover needed.

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

radek_kubka — Mon, 11 Feb 2013 16:52:36 GMT

SAS is a point to point protocol, so even mixing within 1 shelf shouldnt be an issue (other vendors do so).

Couldn't agree more - one million dollar question: why NetApp can't (don't want to) do it?

you do not need more then one crossover point anyway if you choose your shelf ids wisely.

In an ideal world - yes. But usually you have all types of constraints, like physical space in a rack, cable lengths, etc. Or you just have one "mixed" stack & can't introduce another one (e.g. due to having only 4 ATTO bridges ). I agree multiple cross-over points shouldn't make any difference for the overall health of the solution, yet this statement in the Storage Subsystem FAQ hints it is somewhat treated with suspicion.

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

thomas_glodde — Tue, 12 Feb 2013 20:15:24 GMT

For SAS/S-ATA i could image they dont want to add a gazillion possible options to the quoting edge tool resulting in less experienced users creating not field feasible solutions (as happens still alot with FAS2220 machines, eg 12 disks with snaplock in an ha-pair)

For the crossover stuff, i think they just dont have any experience yet if it causes errors or not or maybe they have a problem with acp in that mix.

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

klikar1982 — Thu, 23 May 2013 11:57:56 GMT

Hi All,

I have a little bit different view why is netapp recommending 8 bridges.

I can not provide direct link to documentation, but as far as I remember, netapp best practice recommends to separate local and remote disk pools into different stacks. So because we are talking about metrocluster configuration which should utilize 2 disks pools in 2 stacks and you need 2 bridges per stack for redundancy, you end up with 4 bridges per location, 8 bridges for whole metrocluster solution.

In our production environment, we have installations with only 4 bridges, so we are mixing disks for different pools in 1 stack.

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

thomas_glodde — Fri, 24 May 2013 09:01:44 GMT

kil, that was for old DS14 FC AL topologies. In regards of new SAS Technology and ATTO SAStoFC bridges mixing stacks/pools etc is totaly fine & supported.

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

aborzenkov — Fri, 24 May 2013 09:04:23 GMT

Well, there is difference between “supported” and “recommended”. And NetApp quoting tool does not necessary agrees with either, adding third possibility ☺

Re: Why 8 ATTO bridges? Fabric Metrocluster with SAS stacks

thomas_glodde — Fri, 24 May 2013 11:23:18 GMT

sure, its recommended to use 8 atto bridges which adds like 25,000$USD to your configuration compared to 4 bridges 😉