We have a LACP VIF on each controller dedicated for iSCSI which plugs into 2 stacked Juniper EX4200s with LACP enabled on the LAG.
On the ESX side we have 2 NICs in a vSwitch plugging into the same stacked switch and have setup a LAG with LACP turned off (VMware doesn't support LACP - only static LAG) and load balancing based on IP Hash.
Is this the best setup for performance and redundancy or should we be looking at something else? I've read somewhere about MPIO?
Does NetApp have a best practice guide for iSCSI on the FAS3210s and ESXi 4.1?
You could use either LACP or native iSCSI Multipathing but there are different considerations for each.
1. LACP - In case of LACP, the network handles the redundancy. For load balancing, the key is to use multiple IP aliases on the VIF created on the NetApp storage. The number of IP aliases should be equal to number of physical ports in the VIF. In this case each datastore will have one path but load balancing is achieved by creating multiple datastores which have to be spread across the configured IP aliases (therefore different physical ports). The best practice is to use Virtual Storage Console (NetApp Plugin into vCenter) for provisioning the datastores since it automatically maps the datastores to the available aliases. This works similar to NFS and gives easier migration option in case one is looking at migrating to NFS in the future.
2. Multipathing: iSCSI has an additional option of using native multipathing available in vSphere. In this case the native multipathing takes care of both redundancy and load balancing. This is described in section 3.8 of TR3749 (link above)