As part of a migration to NFS datastores from VMFS and FCP LUNs, I am designing my new LAN configuration. I have selected the cross-switch Etherchannel design recommend in the best practice doc which requires only 2 NIC ports from the quad port NICs I have purchased for this project. The quad port FCP HBAs we use have ports paired on the same ASIC chip so that so that A & C must be used to ensure no single point of failure. I have been looking for a similar rule on the quad port NIC cards but can not see anything in the documentation.
There is a good thread about this topic, but it's in the Sales Engineering space (not accessible for end-users - don't shoot the messenger ). The very informative bottom line from Steve Botkin is below though:
Systems with only two onboard ports share a single Ethernet chip.
Systems with four onboard ports are split across two separate Ethernet chip sets internal to the controller.
The QUAD Ethernet port ADD-IN cards all share a single Ethernet chip.
If an Ethernet chip used in a controller or a NIC card ever fails then a controller failover will occur so splitting access across chips is not necessary.
Ethernet chip sets used can run at full bandwidth for all of the ports using those chips.