Hey there-
I have a problem with one of our NetApp filers routing traffic out the wrong interface in some specific instances, and having spent a few days banging my head against the first layer of NetApp support, I thought I'd check here to see if anybody has a good idea or has run into this issue before (I did find another thread about iSCSI traffic going out the wrong interface that sounded similar to my problem, but no solution).
Our setup is a pair of 3240s, netapp-a and netapp-b, which are configured identically (theoretically, at least). They each have e0M interfaces configured, and a multimode vif with a bunch of vlans trunked down. NFS traffic is not routed at all here, so all hosts that mount must have an interface on one of the vlans trunked down.
95% of the time this works fine, however we have a number of Solaris 10 hosts that cannot mount filesystems from netapp-b, but can from netapp-a. When I watch the traffic via snoop and pktt, the mount dialog starts and at some point netapp-b starts responding on e0M:
pktt on vif-100
18:12:55.396707 vlan 100, p 0, IP nfsclient.ourdomain.org.945 > netapp-b-100.ourdomain.org.nfs: Flags [S], seq 4057553784, win 32781, options [mss 1460,nop,nop,TS val 379612810 ecr 0,nop,wscale 6,nop,nop,sackOK], length 0
18:12:55.396755 vlan 100, p 4, IP netapp-b-100.ourdomain.org.nfs > nfsclient.ourdomain.org.945: Flags [S.], seq 2382367927, ack 4057553785, win 65535, options [mss 1460,nop,nop,sackOK,nop,wscale 1,nop,nop,TS val 204358559 ecr 379612810], length 0
18:12:55.396949 vlan 100, p 0, IP nfsclient.ourdomain.org.945 > netapp-b-100.ourdomain.org.nfs: Flags [.], ack 1, win 32783, options [nop,nop,TS val 379612810 ecr 204358559], length 0
18:12:55.397250 vlan 100, p 0, IP nfsclient.ourdomain.org.1322256560 > netapp-b-100.ourdomain.org.nfs: 112 null
18:12:55.397289 vlan 100, p 4, IP netapp-b-100.ourdomain.org.nfs > nfsclient.ourdomain.org.1322256560: reply ok 32 null
18:12:55.397412 vlan 100, p 0, IP nfsclient.ourdomain.org.945 > netapp-b-100.ourdomain.org.nfs: Flags [.], ack 37, win 32783, options [nop,nop,TS val 379612810 ecr 204358559], length 0
18:12:55.397509 vlan 100, p 0, IP nfsclient.ourdomain.org.945 > netapp-b-100.ourdomain.org.nfs: Flags [F.], seq 117, ack 37, win 32783, options [nop,nop,TS val 379612810 ecr 204358559], length 0
18:12:55.397520 vlan 100, p 4, IP netapp-b-100.ourdomain.org.nfs > nfsclient.ourdomain.org.945: Flags [.], ack 118, win 33580, options [nop,nop,TS val 204358559 ecr 379612810], length 0
18:12:55.397526 vlan 100, p 4, IP netapp-b-100.ourdomain.org.nfs > nfsclient.ourdomain.org.945: Flags [F.], seq 37, ack 118, win 33580, options [nop,nop,TS val 204358559 ecr 379612810], length 0
18:12:55.397663 vlan 100, p 0, IP nfsclient.ourdomain.org.945 > netapp-b-100.ourdomain.org.nfs: Flags [.], ack 38, win 32783, options [nop,nop,TS val 379612810 ecr 204358559], length 0
18:12:55.399980 vlan 100, p 0, IP nfsclient.ourdomain.org.50151 > netapp-b-100.ourdomain.org.sunrpc: UDP, length 56
18:13:10.402218 vlan 100, p 0, IP nfsclient.ourdomain.org.50151 > netapp-b-100.ourdomain.org.sunrpc: UDP, length 56
You can see at the end packets coming in from the client, and nothing going out. However, if you run a pktt on e0M:
18:12:55.396849 IP netapp-b-100.ourdomain.org.sunrpc > nfsclient.ourdomain.org.50151: UDP, length 28
18:13:10.399088 IP netapp-b-100.ourdomain.org.sunrpc > nfsclient.ourdomain.org.50151: UDP, length 28
The default route on the filer is e0M, but also there is a network routing table entry for that subdomain, as well as a routing table entry for the client itself.
When I first ran into this issue I thought it was a client-OS issue, since it only seems to be affecting our Solaris 10 hosts- all of our linux hosts seem to be fine. But given that this is only happening on one of our filers, and the traffic is obviously going out the wrong interface on the broken one, I'm thinking this is a problem on the NetApp side, either a bug or a configuration error.
Any thoughts/comments/help/encouragement?
-c
