OK to Connect NetApp Controller's 10g Interfaces to FEX Ports?

[MRJORDANG]

Hello,

We use NFS datastores (vmware) over 10g pretty heavily and I recently noticed our NetApp's RX PAUSE frame counters (RECEIVE - Xoff & Xon) steadily increasing throughout the day and increasing rapidly during times of high utilization.    Upon further investigation I discovered that our 10g ports are connected to FEX ports rather than the parent Nexus 5K switch.   

By default, Flow Control send and receive are disabled on each port on the 5K.   On the other hand, Flow Control send is enabled on the FEX ports (receive is off).    We also have Flow Control send and receive enabled on our NetApp's 10g interfaces.  I know best practice, per NetApp, is to disable Flow Control throughout the environment (switches, NetApp controllers, ESXi hosts).

 All of the FlexPod design docs I've seen that involve Nexus 5K's and FEX's show the NetApp controllers connected directly to the 5K's.      However, I can't find any documentation that recommends against connecting the 10g interfaces on NetApp controllers to FEX ports.  

Should we NOT connect our NetApp controller's 10g interfaces to FEX ports?   I assume we should NOT because there is some level of oversubscription (sharing) with FEX ports which could result in PAUSE frames.    This assumption also seems to align with the documentation I've found on NetApp and Cisco's sites that show NetApp controllers connected to the 5K's.    

Another person seems to have gone down this road as well and ended up moving their 10g interfaces off the FEX to the parent 5K switch.   Just trying to determine if it is a best practice of some sort to avoid connecting the NetApp's 10g interfaces to FEX ports.

https://supportforums.cisco.com/discussion/12219911/flow-control-cisco-nexus-fabric-extenders

Thanks,

MRJG

[rarvind]

Hello MRJG,

Connecting NetApp storage to the FEX in a FlexPod environment is not a validated design.

I would recommend you to connect the NetApp storage to the Nexus 5K switches.

Yes, it is a NetApp best practise to disable flow control on the interfaces.

Also, could you please explain the reason for connecting the storage to the FEX, are you running out of free ports on the Nexus?

Thanks

Arvind.

[MRJORDANG]

Hi RARVIND,

You are correct - I believe the decision was made to connect the NetApp to the FEX because of a lack of available ports on the 5K.   

You mention connecting the NetApp to the FEX in a FlexPod environment is not a validated design.     Is this also a general recommendation (in a non-FlexPod environment) because of the sharing/oversubscription/flow control that takes place on a FEX?    

Disregard the previous question if the criteria for a FlexPod environment is simply Netapp+Nexus (5K) equipment.   In that case, we are a FlexPod shop despite not having our design officially validated/certified.   All of our equipment was purchased separately - not as a 'FlexPod' package, if you will.

Thanks,

MRJG

[rarvind]

Hi MRJG,

The FlexPod environment will also include the Cisco UCS Compute layer along with the NetApp Storage and Nexus switches.

The FEX can be used to connect the Cisco UCS C-series Rack Mount servers to the Fabric Interconnects.

We do not use the FEX ports for Storage connectivity primarily because there is enough availability of ports on the Nexus 5K, as these switches are not shared outside the FlexPod environment.

I am not aware of a written statement that recommends for or against connecting NetApp Storage to a FEX, I will check on this and let you know.

In the meanwhile, you can try to improve the performance by increasing the number of uplinks from the FEX to the Nexus 5K. (If you have not done that already )

As you know, you will not be able to acheive the same level of performance with the FEX when compared to the 5Ks. All the traffic will need to go upstream to the 5K irrespective of the initiator/target being connected to the same FEX or otherwise. Thus, affecting the throughput of your uplinks.

Thanks

Arvind.

[rarvind]

Hi MRJG,

Can you help me with the below details -

1. FEX Model

2. Nexus 5K Model

3. NX-OS Version

Thanks

Arvind.