Flow Control?

[HendersonD]

In the NetApp and VMWare Virtual Infrastructure 3 Storage Best Practice guide on page 39
http://www.netapp.com/us/library/technical-reports/tr-3428.html

It talks about using Flow Control. My setup is this:

ESX HostßàCisco switch in IBM BladeCenterßàCisco 6509 Core SwitchßàNetapp Filers

The guide says to set the items at the end of this chain (ESX host and Filer) to Flow Control Send ON, Receive OFF

The guides says to set switch ports to “Desired” or if that is not available Send OFF, Receive ON

I dug around quite a bit trying to find out what my Flow Control configuration is now and if need be change it to the recommendations. In Filerview I found where to set these. Right now my iSCSI interfaces and my Public Network interfaces are set to Full which I think means that Send and Receive are both ON. A few questions:

1.       Do I change both the iSCSI interface and the Public Network interface in the filer to Send ON, Receive Off?

2.       Where do you find the configuration of Flow Control on the ESX host? I have dug all over the place and cannot find what my current settings are or how to change them if they are not correct. I am using VMWare’s Enhanced VMXNET adapter for all of my VMs and ESX3.5

3.       Since I have two switches between my ESX host and my Filer, do I assume the recommendation is the same for switches (Send OFF, Receive ON)?

[Dhirendra]

Hello,

If you set flowcontrol to full on interfaces it mean partener can also send and receive flow control frames. If you have more hosts which are accessing in environment then it will create more traffic and performance will be degraded.. By default Flowcontrol is set to Full for complete send and receive files.. So it is recommended to set Flowcontrol as None so that no send and recieve of flowcontrol frames.. which will increase performance..

---

[spmkayser]

Hijacking this thread a bit. Is anyone aware of why exactly one should employ the recommended flow control settings of "filer&hosts: send only" and "switch: receive only"? What's this trying to prevent? Why does it matter whether the filer sends or receives a PAUSE frame? Isn't communication between switch and filer paused in both cases?

[klem]

Take a look at section 6, page 22 of TR-3802, where we discuss the benefits of enabling send flow control on an end host. Basically, modern switches have additional queuing and buffering available so the end device is able to pause traffic and create back-pressure on the switch. The reverse isn't always true.

http://media.netapp.com/documents/tr-3802.pdf

Keep in mind that there are other better methods of dealing with congestion in a network such as QoS which will give you much more fine grain controls than Flow Control.

[chriskranz]

Is that a definitive answer on the flow control? I've spoken to NetApp engineers before, and the performance team in the UK and I've always been told that flow control should be set to full. When I asked about the doc suggesting send on, receive off, they said this must be a typo and was incorrect.

Just want to make sure as we seem to get different opinions.

[klem]

If you have flow control send disabled on the switch, it will not matter if you have the FAS controller set to full or send only since it will never receive a pause frame from the switch. So, either setting (full or send on) will work fine in your environment.

[chriskranz]

But what impact does setting flow control to full across all devices have? (as most people would by default in my experience)

[klem]

Within the network fabric, better methods such as implementing QoS policies or even using a higher level loss based mechanism such as TCP are available and preferred to flow control. When using flow control at an edge device, full mode will likely work. However, it's preferred to put the back-pressure on the faster device, which in theory has more buffers available to deal with the additional stored packets. Here's the main problem: If you have multiple traffic flows going to the storage controller (very likely), sending a traditional Ethernet PAUSE frame to the controller will pause all traffic flows! That means you'll have essentially the same quality of service for all traffic flows to the controller. Priority Flow Control (PFC) was developed in the new DCB capable switches to specifically to help address this problem.

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-542809_ns783_Networking_Solutions_White_Paper.html

[baselinept]

Hi,

Have you tried the ethtool?

For instance, the command 'ethtool -A vmnic0 rx on tx on' would enable flow control for transmit and receive on nic0.

Cheers