Ok, I think I might have this.
Node1
e0M -> 10.140.15.211/24
StorLAN -> 10.140.21.211/24
ProdLAN -> 10.140.11.211/24
Default route -> 10.140.11.254
Node1
e0M -> 10.140.15.212/24
StorLAN -> 10.140.21.212/24
ProdLAN -> 10.140.11.212/24
Default route -> 10.140.11.254
What this config tells me is this:
UNLESS you have a client IP on 10.140.15.x, you cannot ping this interface.
UNLESS you have a client IP on 10.140.21.x, you cannot ping this interface.
The default gateway is 10.140.11.254 which is on the ProdLAN ifgrp.
ANY communication INTO the NetApp, will result in going OUT THE ProdLAN interface.
That is why your Pings are failing. Host 10.140.31.x going to the NetApp (no matter the incoming interface), if it makes it, returns out the ProdLAN where it may not be able to find it end unit.
What you need to do is determine which interface you really want to be your default Gateway.
Then re-order your /etc/rc files.
ifconfig <interface> blah.....
route add <blah>
ifconfig #2
ifconfig #3
You have defined the default to the be 10.140.11.254 on ProdLAN, yet according to "route" it is trying to use 10.140.15.254 on e0M.
Figure which one, correct it and reboot the nodes.