3270 - Local SAS Ports Best Practices (Use Cases...)?

[chriszurich]

After recently receiving my new 3270 controller pair I had a few suprise's...

There is an additional issue which I either missed when reviewing the documentation or is possibly not mentioned at all.  The issue I speak of is that with the new 10GbE interconnect (HA Active/Active) change there is no longer 4 onboard fibre channel ports available per controller.  If you wish to cluster a pair on 3200's series arrays you will only have two 4GbS FC ports per controller pair.  While I understand that an FC card can easily be added to expansion ports this seems like a glaring issue for those of us who wish to use our new 3200 as a multi-purpose SAN/NAS box.  Plus this goes against existing best practices which state that FC target traffic should be isolated to either a single card or the local FC ports but not both.

I'd also like to get feedback on how others are planning on using the two local SAS ports built into each controller.  With the cabling diagrams I've reviewed as well as the best practice documents I see no real use for them unless I wish to build out a non-redundant SAS stack.  In my environment this is not an option since uptime and performance is my main concern.

[BrendonHiggins]

I have my FAS3140 pair cabled via quad port SAS HBA in each filer via the A&C port on each card.  In the very unlikley event of a card failure the filer can route the traffic via the internal interconnect.  Not perfect but good enough while the problem card is replaced.

Hope it helps

Bren

PS - Have you seen this yet?  http://now.netapp.com/NOW/knowledge/docs/hardware/filer/215-05500_A0.pdf

[chriszurich]

Saw the cabling guide and read it a couple months back.  Now that I actually have the hardware I'll read through it again.  

[aborzenkov · Accepted solution]

Two SAS ports per controller is enough to build redundant MP-HA stack with 240 disks. If you need more, you could add more SAS HBAs ...

[radek_kubka]

Hi,

this seems like a glaring issue for those of us who wish to use our new 3200 as a multi-purpose SAN/NAS box.

To me 2x 2 FC ports is just enough to provide resilient connectivity via two fabrics - each port from each controller goes to fabric A & B respectively.

this goes against existing best practices which state that FC target traffic should be isolated to either a single card or the local FC ports but not both.

This is no longer the case with ONTAP 7.3.2 or higher:

http://communities.netapp.com/message/44248#44248

Regards,
Radek

[chriszurich]

Thanks for the input guys,  I really appreciate the links and corrections.  I guess one of my concerns is that the onboard SAS ports reside on the same asic.  So if the asic fails a controller failover will be initiated where as with use of two four port cards I can layout the redundant drops such that the card is no longer a single point of failure. So I just re-read this statement and while true it can be mitigated by mixing the drops between a local 4 port SAS card and the onboard ports. 

Radek, within the linked post bellow the one you mentioned expands upon my concern regarding mixing local and expansion FC cards.  Since the local ports are 4Gb/s and an expansion card would be 8Gb/s this would mean that I could not simply add the expansion card and use both the local and expansion card ports.  This would mean that my local 4Gb/s ports would be useless since I'm running in single image mode which includes all local target fc ports within the same group.

Please correct me if this functionality has changed.

[radek_kubka]

Since the local ports are 4Gb/s and an expansion card would be 8Gb/s this would mean that I could not simply add the expansion card and use both the local and expansion card ports.

Yep - if you opt for 8GBit cards (which seems to be rather obvious choice these days).

In theory you can use 4Gbit HBAs (X1128A-R6 or X1130A-R6), so mixing all FC ports would be fine.

That said, I am not trying to blindly prove my point & actually I've been always surprised by the fact that 32x0 onboard ports are 4Gbit only (http://communities.netapp.com/message/42960#42960 - you've seen this already as your post is in this thread as well)

Regards,

Radek

[aborzenkov]

Same speed rule applies only to target ports. You can have initiator ports of different speed, or mix target and initiator ports of different speed. So if you add 8Gb/s target HBA, you can use on board for shelf connection just fine.

[chriszurich]

So I see your point but would like to reiterate that the Fibre Channel ports on this filer will never be used as initiators (disk initiators to be exact).  All of the shelves I plan on attaching are of the SAS varient so my only use for FC ports is as targets for host I/O. 

Soon I'll have the box fully cabled and will start my baseline performance testing.  We'll see how hard I can punish the two local 4Gb FC ports...