Since ONTAP adds the gateway to a physical interface it can get confusing sometimes. Do you have "options ip.fastpath.enable" set to off ? It sounds like the default on was changed to off. When on the response goes out the same port it came in on. We used to turn it off pretty on most installs but with e0M it is sometimes easier to leave it on in order to not have to modify routing which could affect the data network.
With fastpath enabled, any user traffic that comes in the data interface will go back out that interface and not use routing (exception is an nfs mount request which doesn't use fastpath until after the mount completes we have found). I agree the static route gets ugly when it is outside the management network in this case.
Is routed on or turned off? Do you have a case open and have you run a packet trace (pktt) to check what is going in/out of e0M? This might be a burt support can track and see if fixed.
I think it applies to all protocol including management traffic...anything that comes in from a host. We did find that nfs mount requests use routing, not fastpath...a customer with 2 interfaces on the same subnet... mount request came in .3 and the response out .2 and the client wasn't able to mount...but once mounted all traffic in .3 came back out .3. I agree completely...It would be nice to have a full description of fastpath and what it does or doesn't do with routing for all scenarios and protocols.
But for e.g Redhat Linux the behaviour is different. If you do not explicitely specifiy proto=tcp in the mount options, Linux will use UDP. In some redhat releases it will switch to TCP if the reply from the filer does not arrive. If you explicitely specify proto=tcp the Redhat will immediately use TCP in the mount request and fastpath will be used for the reply!
In general fastpath will be used for TCP connections and only for UDP data traffic. The reply to ICMP traffic will not use fastpath.
Another issue one should be aware of is that fastpath only applies to connections initiated by a client. This is important to know if you use iSCSI and your initiators are not in the same network as your filer. With iSCSI the netapp filer regularly sends out a keep alive. Since this is not client initiated, fastpath is not applicable and the filer routing table will be used. If the filer does not explicitely have an entry for your initiator subnet, the default gateway will be used and depending on your netwerk setup you run the risk of the keep alive not ariving at the initiator. When this happens the filer will drop the iSCSI connection after 10 failed keep alive attempts!
Let’s be honest – the problem you have is in no way specific to NetApp and will happen on every other system I have worked with. There is no system that would by default ignore routing table for random destinations. If you have requirement that specific networks must be reached via specific gateway you have to tell system about it. There is no workaround. Really.