iSCSI configuration help

[lindloffc]

Currently I have the following setup on a FAS - 3140.

Head 1, single top level VIF that has 2 - 10gig interfaces (e4a, e4b).  The main IP is 10.50.1.1 and a single alias of 10.50.1.2

Head 2, single top level VIF that has 2 - 10gig interfaces (e4a, e4b).  The main IP is 10.50.1.3 and a single alias of 10.50.1.4

These interfaces are plugged into a couple of Nexus 5548's, the VIF's are using LACP into the Nexus switches.

Volumes are shared out via NFS to VMware servers.

Our ESXi 4.1 servers have dual 10gig nics for storage, and they are setup with teaming in ESXi to use IP hash.  Each of the hosts have a single IP on the same 10.50.1.x subnet, which is VLAN 50 for us.

All of that works great.  I mounted the NFS volumes on ESX, by staggering the target IP's of the Filer Heads and I see almost equal traffic on and interfaces of the VIF's.

******Now we are migrating to Hyper V (dont ask) So I must move to iSCSI volumes/LUN's.  I want to use MPIO on the Windows 2008 R2 hosts. 

I have read some NetApp docs and Microsoft docs that talk about using multiple subnets for host and target to make sure MPIO works properly.  However I have seen setup (screen shots) that show all interfaces, using MPIO on the same subnet.  In my current setup above we use the same subnet with NFS/VMware.  So I am confused???? 

Can I use my current configuration and just use the same subnet and then on my Windows 2008 R2 hosts just IP each of the two NIC's on the same subnet?  Or should I do the following?

Head 1, single top level VIF that has 2 - 10gig interfaces (e4a, e4b).  The main IP would be 10.50.1.1 and I would add an alias of 10.50.2.1 ?

Head 1, single top level VIF that has 2 - 10gig interfaces (e4a, e4b).  The main IP would be 10.50.1.2 and I would add an alias of 10.50.2.2 ?

On the Windows 2008 R2 host I would then setup NIC 1 with 10.50.1.6 and NIC 2 with 10.50.2.6 pointing each NIC at a different Target IP when setting up the MS iSCSI initiator (with MPIO installed)??

Would I still use LACP on the VIF's?

Thanks for any help!

[aborzenkov]

just use the same subnet and then on my Windows 2008 R2 hosts just IP each of the two NIC's on the same subnet?

In this case all outgoing traffic will be sent via one interface only, according to routing table. This is simply how TCP/IP communication works. Not only it won't offer any load balancing (I really do not think it is an issue with 10G); the main problem is it may defeat any error recovery attempted by MPIO stack.

In storage world established practice is to use two completely physically independent fabrics for conneciton; then storage stack on host provides full end-to-end error detection and recovery (and load balancing).

Head 1, single top level VIF that has 2 - 10gig interfaces (e4a, e4b).   The main IP would be 10.50.1.1 and I would add an alias of 10.50.2.1 ?

This means you will run two IP networks inside of single physical VLAN. Which now goes against networking best practices ; and and has the same issue of defeating the main purpose of MPIO - to have several independent connections.

But in your case (having single switch stack) it could be enough for load balancing purposes.

[lindloffc]

So you are saying neither of my proposed solutions will work

So I found these...

http://communities.netapp.com/blogs/ethernetstorageguy/2009/04/04/multimode-vif-survival-guide

and

http://communities.netapp.com/message/23645

Both of which are examples of having everything on the same subnet/vlan.  Also I was reading another doc that walked through the MPIO setup with screen shots on the Windows side, it used the same subnet as well.  It showed that when making the connections in the Microsoft iSCSI setup to choose Host IP 1 to Target IP 1 and on the second connection choose Host IP 2 to Target IP 2.....example only had two NIC's on each side.  My guess is that the MPIO software handles the communications, even with everything on the same subnet??

It is all confusing at best.

[aborzenkov · Accepted solution]

So you are saying neither of my proposed solutions will work

No, I'm just saying that I would prefer slightly different setup

Also I was reading another doc that walked through the MPIO setup with  screen shots on the Windows side, it used the same subnet as well.

Nobody is perfect

My guess is that the MPIO software handles the communications, even with everything on the same subnet?

One point to note - source address selection is not equal to outgoing interface selection. So it is perfectly valid to have session with source S1 going via interface with address S2. You cannot tell which route your packets take by looking on netstat (or similar) output.

Good news is that on W2k8 it could actually work. I am not able to find relevant Microsoft documentation, but the article http://technet.microsoft.com/en-us/magazine/2007.09.cableguy.aspx describes differences between weak and strong host models. In particular:

If the source address has been specified, the source interface is known. The source interface is assigned the source address.

That said, you still have single point of failure - your LAN switch (stack). You still have single VLAN that handles all traffic. Any issues on this VLAN that disrupt traffic will affect both pathes in your MPIO setup. So as already mentioned, you will probably have load balancing but you will not actually have complete fault tolerance.

That is not worse then it was with NFS. It is just that iSCSI allows to do better ...

[lindloffc]

Thanks for your input.  I do see the single VLAN as a weak link.  I do have dual Nexus switches with interconnects and each host has single dedicated iSCSI NIC in each of the Nexus switches.

I could trunk the ports to the NetApp and add/tag multiple VLAN's to it, then put each of the host NIC's on different subnets, and add an alias to VIF's on that subnet to give me more than one VLAN??

I need to do a lot of testing, copy large files, pull a twinax cable in the middle of the copy.....etc.

[shaunjurr]

Hi,

Your challenge is a familiar one because of how the different protocols achieve redundance. NFSv3 can't do multiple networks, so Ethernet protocols are used to get redundancy.  iSCSI has Multi-pathing which offers both redundancy and increased bandwidth.  With MPIO you need to have 2 subnets. Since you already have your 10GE links setup for Ethernet redundancy, you don't have enough nic's left to setup a sort of "IP-SAN" connection redundancy where you might choose to have interfaces/vlans preferred to certain switches... where vif's are not really necessary. 

The only good solution is to have another pair of 10GE interfaces, but barring that, just add a second vlan to get 2 iSCSI subnets (easier to move them later as vlans probably than stuffing 2 subnets into one vlan) and put your iscsi traffic there.  It works.  You won't get traffic balanced perfectly but you get all of the advantages and perhaps a good basis to migrate things at a later date.