I have a FAS2040. With respect to the Gigabit Ethernet Ports (e0a,e0b,e0c,e0d)
Right now I have only one physical switch over which I need to run NFS to these ports. What is the best design to get as much performance and redundancy as possible? I can use all 4 ports if I need to but for the moment having one physical switch is a constraint that I can't get around. Should I use only 2 of these ports or all 4?
Make sure you spread out the physical connections on the switch so you're utilizing the different ASICS and not flooding the first one by putting all the connections on the first 10 ports.
Do some research on the switch so you know which ports belong to which ASICS.
LACP VIFs is the way to go. Make sure you add an alias IP to each VIF on the 2040 so each VIF has 2 IP addresses. This is needed to load balance the NFS traffic between the 2 physical connections on the 2040 from the ESXi hosts when using IP Hash in vSphere.
You have one datastore mounted to one IP address and another to the second IP address.