Mixing FC Target Ports Onboard and Expansion

[erick_moore]

We have a 3170 with a single quad port FC expansion card (actually we have 2, one per controller).  We have no plans for adding any more than the 2 FC loops already installed (we are going with SAS for all our new shelves), and we don't plan on using more than 2 of the onboard ports per controller as initiators.  Our installer configured 2 of the onboard ports as targets along with 2 of the ports in the quad port HBA so we would have loop redundancy in the event we lost a PCI slot.  I was just taking the NetApp NCIE-SAN practice exam, and it stated that target ports should never be mixed between the expansion cards and the onboard cards.  Any thoughts on this matter?

[radek_kubka]

Hi Eric,

Have you read this thread?

http://communities.netapp.com/message/7600#7600

Regards,

Radek

[erick_moore]

I can't get to that link for some reason.  It is giving me an access is denied error.

[radek_kubka]

It's within Partner & Field Community, that's probably why.

This is the bottom line from Skip Shapiro from NetApp:

If any on-board ports are used as  targets, you cannot use any expansion target HBAs.  All target ports  used either either onboard or on expansion cards; you can't use both  types together.

However, you can mix onboard ports configured as initiators and initiator expansion HBAs.

[ekashpureff]

Eric -

It's my understanding that add on cards are configured at the factory to be initiator or target ports, and can not be re-configured.

Motherboard ports can be reconfigured. This requires a re-boot.

I haven't seen any best practice documents that say you shouldn't use motherboard target ports as well as add on card target ports.

I hope this response has been helpful to you.

At your service,

Eugene E. Kashpureff
ekashp@kashpureff.org
Fastlane NetApp Instructor and Independent Consultant
http://www.fastlaneus.com/ http://www.linkedin.com/in/eugenekashpureff

(P.S. I appreciate points for helpful or correct answers.)

[scottgelb]

There are some rules that are often broken with Initiator/Target and also with Disk/Tape Initiator ports.

NetApp/Partner SEs have an updated fc rules powerpoint in the net2 download and I submitted some edits a while ago...a great document and summary below

1) It is ok to have initiators mixed onboard and expansion slots

2) Expansion cards are hard set as initiator or target, but onboard can be swapped (already mentioned)

3) It is not ok to mix Targets onboard and expansion slots.  Either all onboard or all expansion slots for targets

4) It is ok to mix port speeds on the same card or asic

5) You cannot put ports on the same expansion hba ASIC (1a/1b for example) to both disk and tape (1a to disk and 1b to tape for example), but you can put disk on 1a/1b and tape on 1c/1d.  However, if onboard ports it is OK (0a disk, 0b tape)... This rule is often broken since it works but is not supported on expansion hbas.  (kb22038)

[aborzenkov]

Re. 3 – it is not what the latest ppt (edition 4) says:

Can I have target expansion cards installed in the system and still use some of my onboard ports for fibre channel SAN Host connectivity (targets)?

It Depends

7.3.2 or greater – Yes

7.3.1.1 or less - No

[scottgelb]

I think that needs a fix in the new one.. I don't know of support added in 7.3.2 for on/offboard targets concurrent.. Need to have Bryan fix this one..or clarify if a change was made, but not correct I don't think.

Typos Sent on Blackberry Wireless

[aborzenkov]

Well, I am afraid it is a bit more complicated ...

7.3.1.1 BSAG:

Considerations

If you install target expansion adapters on the system, you must configure the onboard adapters as

initiators.

7.3.3 BSAG:

The whole paragrah "Consderations" is no more there. The only limitation that is mentioned is total number of target HBAs and the rule that all target adapters must be of the same speed: you cannot mix 2-Gb and 4-Gb adapters on the same system

The presentation mentions this rule only for target expansion cards:

Can I mix expansion target cards in the same system of different speeds (X1130 4G & X1131 8G)? NO

But if the rule of speed mix is for all target HBAs, it means - you can use onboard, as long as speed matches. Given that current HBAs are 8Gb/s, we have no way to mix them

I'd love if someone from NetApp gave final word.

[scottgelb]

And 8.0.1 rc2 bsag it says you have to unconfigure onboard on an ha pair when adding target adapters... It has always been either all onboard or all expansion target ports without a mix, but the documentation doesn't all match up... I will forward this link to Steve Botkin and he will be able to confirm here.

You are upgrading from a 2-Gb onboard adapter to a 4-Gb target expansion adapter or 4-Gb
onboard adapter to a 8-Gb target expansion adapter. Because you cannot mix 2-Gb and 4-Gb
adapters or 4-Gb and 8-Gb adapters on the same system, or on two systems in an HA pair, you
must set the onboard adapters to unconfigured before installing the target expansion adapter.

[sbotkin]

The requirements for the mixing of Initators and Targets both onboard ports and HBA's has changed with Data ONTAP 7.3.2.

The mixing of onboard Target cards with Target HBA's is not supported in any version of Data ONTAP before 7.3.2.  The supported configurations are documented in the FC & iSCSI Configuration Guide on the NOW  site at : https://now.netapp.com/NOW/knowledge/docs/san/fcp_iscsi_config/QuickRef/fc_iscsi_config_guide_72.pdf on page 22.   If a Target HBA is installed in a FAS solution running Data ONTAP 7.3.1.1 or LOWER then the onboard ports will have to be all Initiators, none can be used as Target devices.

The  mixing of onboard and HBA's for Target's and Initiators has changed, it  is supported now with systems running Data ONTAP 7.3.2 or higher.  The  changes to this are documented in the FC & iSCSI Configuration Guide  for Data ONTAP 7.3.2+ which can be found at : https://now.netapp.com/NOW/knowledge/docs/san/fcp_iscsi_config/QuickRef/fc_iscsi_config_guide_73.pdf on page 22.  With Data ONTAP 7.3.2 it is possible to have onboard ports  configured as targets and to also have target HBA's installed in the  controllers.  The System Configuration Guide will show the number of  Target HBA's that are supported for any FAS solution by choosing Data  ONTAP 7.3.2 or higher : http://now.netapp.com/NOW/knowledge/docs/hardware/NetApp/syscfg/scdot732/index.htm

The  HBA's that are used in the FAS solutions are supported with all of  their ports either being Initiator's or Target's, they can NOT be split  like the onboard ports on the FAS solutions.  The HBA is ordered as  either an Initiator or Target HBA.  This is a hardware requirement of  the HBA's.

The Initiators Targets - The Rules (Edition 4) has the correct information in it for those that have access to the Net2 Document Repository.

Hope this helps...

Steve

[aborzenkov]

Thank you, but it is still unclear (both in fc_iscsi_config_guide_73.pdf and in presentaion) whether I can mix onboard and expansion card of different speed. Both documents state:

All expansion adapters should be the same speed

So once more direct question - can I mix onboard 4Gb/s target port with expansion 8Gb/s target HBA?

Thank you!

[jooss]

It's not currently supported to mix the speeds.

Page 38 of the 8.0 FC/iSCSI config guide does document that for the FAS3200:

------

Each 32xx controller supports 4-Gb and 8-Gb FC target expansion adapters. Note that the 8-Gb

expansion adapters cannot be combined with 4-Gb targets (whether using expansion adapters or

onboard).

------

Regarding he BSAG comment:

-----

Considerations

If you install target expansion adapters on the system, you must configure the onboard adapters as

initiators.

-----

This was a left over from when it was not allowed to mix onboard and add-in expansion target ports.

I've marked that and we'll work on getting that straighted out.

We'll also work on getting the statements on the mixing of speeds to be more definitive.

Cheers,

Rick

[lmunro_hug]

Hi, bit of thread necromancy here… 🙂

I am also studying for the NCIE: SAN and in the practice questions this topic comes up. I think the answers must be based on old data as they say the right answer is the third one down in the screenshot, Is this still correct? If you look at the Fibre Channel and iSCSI Configuration Guide for the Data ONTAP 8.0 page 14 https://library.netapp.com/ecm/ecm_get_file/ECMM1280845 you can see that target ports are supported on both the onboard an expansion ports.

Can anyone from NetApp confirm that the practice test is wrong? Dont want to be marked wrong if something related comes up in the exam.

Thanks

Luke