Is it okay to run NFS and iSCSI on the same vif? Yes it sure is.
Would I do this? Probably, however as I like to use jumbo frames for iSCSI and normal frames for NFS I might possible use different VIFs for iSCSI and NFS traffic. Everyone's enviornment is different, so this might not work for you.
Is NFS better than iSCSI? Depends on what you're trying to achieve, for VMware workloads; absolutely!
This biggest setback that you guys have pointed out; is if you want to use jumbo frames, this will not be possible. But otherwise, this is not a huge issue? I'm asking this because I have few ports to work with like Radek pointed out.
You can have multiple vlans on top of a VIF, one for NFS and one for iSCSI for example. On the NetApp, IP addresses and MTUs are defined at the VLAN level. Your switching infrastructure must be configured to match. Switch implementations vary. I recently configured an Extreme switch with Jumbo frames on one VLAN and standard on another. I am sure this is also supported on recent enterprise class switches. In the case of our Extreme switch, the MTU was defined at the port level and had to be as large or larger than the largest VLAN MTU and then the MTUs for each VLAN were defined per VLAN. One thing to watch out for is low end managed switches that may not have large enough port buffers to efficiently support jumbo frames or perhaps do not allow the definition of enough VLANS. We saw an inexpensive switch recently that only allowed 4 VLANs to be defined and this was not adequate for an ideal VMware VLAN configuration.