FAS and V-Series Storage Systems Discussions

Re: Asymmetrical routing in filer?

Hey John-

   To answer your questions:

1) Yes, the solaris hosts are on the same vlan as the filer.  I keep trying to explain to NetApp support that these packets shouldn't be routing at all, but they don't seem to believe me.

2) Yes, I am using vlan tagging on the filer.

3) The route table shows the correct entries for both the client (Solaris box) and the subnet it is on (the filer also has an interface on that net).

4) I have tried using proto=tcp as an arg to mount, with no change in behavior by the filer.

I'll post specifics (route table, ifconfig, etc) when I get my VPN back up and functional.

        -c

Re: Asymmetrical routing in filer?

Hey all-

   Sorry for the delay in response; I had to put this issue aside for a bit, but picked it back up again about a week ago (broke my hand, and I tell you, being a sys admin with one hand to type with is a PITA).    Anyway, I had temporarily mitigated this issue back in December by setting the interfaces.blocked.nfs option to "e0M'.  As soon as I did that, the asymmetrical routing went away, and all of our clients, regardless of OS, were able to mount all volumes just fine.

  I opened up a new ticket with NetApp last week (they auto-closed it back at the end of Dec, when I had my accident).  In the meantime, they had auto-removed all of the packet traces and other data I had uploaded for the original ticket, so I planned on clearing the interfaces.blocked.nfs option, and repeating my pktt/tcpdumps. I did this on Saturday, and for the life of me the problem just does not happen any more.  I've been going since Saturday night with that option off, I'm back to mounting vol0 on my management host over e0M, and life seems fine.  What changed between Dec and now?  Nothing- I haven't touched a thing, outside of creating a new volume or two.  No networking changes, no upgrades, no reboots/failovers, nada.

  So at this point I guess I can't really say much. I've closed the ticket with NetApp, and I'll keep an eye on whether or not this crops up again. If I do, I'll try to remember to follow up to this post.

-c

Re: Asymmetrical routing in filer?

I suspect it may be the default route.. netstat -rn will show the default route likely set to the interface you want and e0M was down so it couldn’t attach the route… a reboot may change that. ONTAP binds the route to an interface. I just had a head swap last weekend where we had an odd route tied to the wrong interface… the right interface on the same network as the route was just down at the time of the route add… when the interface was brought up the route didn’t work to that interface until we deleted and re-added the route.

Re: Asymmetrical routing in filer?

So, are you saying that netstat -rn would show the route going out interface X, but instead the traffic would go out interface Y?  Because that's what I was seeing- the routing table looked perfect, and in fact the first 6 packets or so coming out of the filer would go out the correct interface, but all of the sudden the packets would switch over to going out e0M (with no change to the routing table).

I upgraded the OS on the filers back in mid-December, and verified the problem existed after the upgrade. The filers have not been rebooted since then (so any changes to the interfaces.blocked.nfs option have been since the last reboot).   That said, I should plan on a reboot this weekend just to see if the problem comes back- good idea.

Message was edited for spelling by: Chad Lake

Re: Asymmetrical routing in filer?

I've seen that with interfaces up/down and routes attached to the wrong interfaces.  How did the reboot go?

Re: Asymmetrical routing in filer?

OK, I finally got a chance to reboot the filer, and the problem did not re-emerge.  I tell you, I don't think I've ever been so disappointed to have everything working like it should....

Re: Asymmetrical routing in filer?

agreed...consistency even when broken is always better to know.  netstat -rn is key to looking at things... I have had a few customers in the last 2 weeks have network issues and found their secondary vifs (ifgrps) did not come up and routing tied to their other vifs and the only way to fix is to up the correct vif, then delete and re-add the same route (or route -f or reboot but that is disruptive) to get it working.  A different issue but similar in how routes bind to an interface.

Re: Asymmetrical routing in filer?

When I was having problems the routing table always appeared correct.  And even the first part of the mount request would go over the correct interface (~10-12 packets or so) until abruptly switching to going out over e0M (with no change in the routing table).

If it crops up again I'll try to remember to come back to this thread with an update.  Thanks everybody for their help with this!

Re: Asymmetrical routing in filer?

That's right...even with fastpath and blocking traffic to e0M for data with interface.blocked and still did that...very odd.

Forums