I've been using NFS on Netapps via 10G redundant ethernet links (active/standby) for about six months now using ESX 4.1. It rocks. I've not seen utilization on the Netapps approach 4 Gig yet, and IOPs are still relatively low (~200 VMs and climbing fast).
But I'm being forced to put up some servers utilizing 8 x 1 Gig links in a manner similar to your environment.
I can't imagine why anyone would use iSCSI rather than NFS anymore, partularly now that "Load based on physical port" is a teaming option in vSphere Enterprise plus.
I've run tests with a vmware distributed switch (not the vNexus) on a single portgroup and vlan with physical uplinks to TWO DIFFERENT SWITCHS without creating a L2 loop. If works fine without 802.3ad LACP, or proprietary Cisco Etherchannel, or spanning tree. In fact the docs say "Load Based on Physical Port" is not compatible with LACP or Etherchannel. VMware takes care of the routing, putting VM1 on the first link, then VM2 on the second link, etc. It then looks for any physical port which goes over 70% bandwidth for an extended period of time, and starts to move VMs off of that port. Now, that's extremely cool. I haven't found anywhere I can change the 70% configuration, and I only know about it from reading.
So, you put the same (multiple) vlan tags on two different ports on two different physical switches, and you have an active/active load balanced team (LBT). You can scale to four or six uplinks balanced across physical swithes as well. This means that high end switches which can do LACP (or Etherchannel) across two switches are no longer necessary.
In my new design, which will house Oracle and MSSQL VMs (don't ask), I'm going active/standby to two different switches for management console, and two 3 x 1 Gig uplinks to different physical switches for both data and storage, no LACP at all, and that's it. NFS has its own portgroup, and with ESX 4.1 I have QOS or whatever VMware calls it, so I assign priority to IP storage and let it fly.
You do have to be careful configuring your portgroups. Hope this gives you some ideas. VMware + Netapps + NFS is really very nice. I suppose you could throw iSCSI in there as well.