different vfilers on different VLANs, how to configure route

[SIMURGH]

We have multiple vfilers in a 7-mode filer, but they are on different vlans.

e.g.  vfiler1 is on mvif1-801, ip addr is 10.68.173.1

       vfiler4 is on mvif1-804, ip addr is 10.68.237.1

According to routing table, default gateway for them is 10.68.173.254.  Since they are on the same physical interface, there can only be one default gateway.

After a code upgrade and reboot, vfiler4 fails to connect to any DNS server, thus all AD authentication failed, rendering the CIFS share unusable. 

This understandable as vfiler4 being in 10.68.237 subnet cannot reach the 10.68.173.254 default gateway, and therefore any network destination without the routing.  It should be using 10.68.237.254 as gateway instead to reach the DNS servers.  

The dilemma is that I cannot add a route because the routing table is used by all vfilers.  If I add one using 10.68.237.254 as gateway, other vfilers in mvif1-801 will also use this but they won't be able to reach it.

Something was lost during the reboot that was not written into the rc file.  I don't know how it worked before.  What should I add in the routing table?

Please help.  Thanks.

[MarcSchindler]

Hello,

I think you forgot to add the routing entries in the /etc/rc

Check this...

Modify the routing table for the vFiler unit by entering the following command:
vfiler run vfiler_name route add [inet] default gateway_metric
vfiler run vfiler_name route add [inet6] default gateway_metric

Example
The following example shows how to add a route to the routing table used by the vFiler unit named vFiler1:
vfiler run vFiler1 route add default 1.2.3.4 1

For each interface used by the vFiler unit, add the following information to the /etc/rc file on the hosting storage system:
The same ifconfig command that was used to configure the interface as Up.
The same route command that was used to add a route to the routing table.
This configures the interfaces as Up and enforces the route commands across reboots.
Optional: If the hosting storage system is part of an HA pair, edit the /etc/rc file in each partner of the HA pair to define a partner interface for each interface that the vFiler unit uses.

Example
The following command shows how to ensure that interface e10 has a partner e10 interface: ifconfig e10 partner e10
Alternatively, you can run the setup command from the storage system and enter the partner interface for each interface.
Attention: If you use the storage system setup command to automatically define a partner interface, it clears all existing information about vFiler unit configuration in /etc/rc file.

Optional: If the IPspace does not have a default gateway, establish a route to the default gateway in the IPspace that any vFiler in the IPspace can use by entering the following command:
vfiler run vfiler_name route add default gateway_metric

Example
The following command shows how to add a route to the default gateway for the IPspace used by the vFiler unit named vFiler1:
vfiler run vFiler1 route add default 1.2.3.1 1
vfiler run vFiler1 route add inet6 default fc20::abcd 1

source: https://library.netapp.com/ecmdocs/ECMP1155585/html/GUID-8FC8613F-F480-40C8-95F0-E9E7199709DE.html

[SIMURGH]

Hi Marc

I have tried something like that.  Even though I add the route using the vfiler run command, it ends up in the global routing table, which means all vfilers will use this route.  For example, the default gateway is bound to the physical interface, not vfiler, so I can't have different default gateways (or routes for that matter) for different vfilers.  

[sheelnidhig]

You need to use IPSapce for each routed network.

So in this case you can have one IPSpace per vFiler which is dedicated for a VLAN. 

,Sheel

[scottgelb]

What is the subnet mask?  Can you email the output of your rc and hosts files?  and the output of netstat -rn.  It sounds like a net route is missing that would be needed after the default route for the first vfiler.  I agree with the other post that separate IPspaces would be good to have...although you are already setup on the default-ipspace and there is no changing without destroying and recreating the vfilers manually so probably best to just get it back working the way it was before then remediate later if it makes sense... You are correct that a vfiler run route add is the same as a route add in this case since you are in the default ipspace when running in the vfiler context.

[SIMURGH]

Hi

Yes they were always in the same default IP space.  And recreating it in a different one wasn't really viable. Subnet masks are 255.255.255.0  I am sending you the netstat -rn, rc and host file in a separate message.

Thanks.

[scottgelb]

It looks like you don't have a route to the 237 gateway from the 237 destination network...be careful though as you mentioned since adding a network route will affect all vfilers in the same IPspace.

route add net 10.67.237.0/24 10.67.237.254 1

Also the separate lines for nfo are not required...nfo can be on the same line as the command if you want to consolidate.  Although not a bad idea to keep separate if you want to find it quicker as it is.

[SIMURGH]

Hi

Yes we noticed that too, but what I don't get is:  This route basically says to get to a destination of  10.67.237.0/24, use 10.67.237.254 as gateway, but does it help if the source is in this network, while the destination they are trying to get to is 10.68.10.x?

Thanks.

[scottgelb]

I wonder how it worked before...you could add that route, but like you mentioned, the other VLANs that need to get to 10.68.10. would go through the 237 network and don't have an interface on that network... The only other workaround that would be a bit arduous are route add host entries for each exact interface to get to the proper gateway outbound.  

To see what changed, you could go back through older autosupports and look and compare netstat output... meanwhile, I would add a host route to get to dns from the 237 interface without affecting other vfilers..  Ideally, I would recreate the environment using IPspaces... if this were cDOT (for when you migrate down the road) we would be able to do this without IPspaces since each SVM (vfiler equivalent) gets its own routing table even if sharing the ipspace with other SVMs.