Multistore Failover and Networking

[mfhlmr001]

Over the weekend I tested failover on one of our cluster nodes running multistore (dtw-netapp3 to dtw-netapp4) and although the failover completed the network failed. The 3 vfilers all started on the partner but the network interfaces were all down. I saw the following messages on the console but there was not enough detail to determine the exact problem.

vif: nas is not mapped to a local vif

add net default: gateway 192.168.100.161

PIP: protocol filter cannot be inspected (not applicable in partner mode)

Fri Sep  9 23:21:53 EDT [dtw-netapp4: net.ifconfig.takeoverError:warning]: WARNING: 3 errors detected during network takeover processing WARNING: Some network clients may not be able to access the cluster during takeover

Fri Sep  9 23:21:53 EDT [dtw-netapp4: cf.rsrc.takeoverOpFail:error]: Cluster monitor: takeover during ifconfig_2 failed; takeover continuing...

Fri Sep  9 23:21:54 EDT [vfiler3_apm@dtw-netapp3/dtw-netapp4: vFiler.initialized:notice]: vfiler "vfiler3_apm" initialized.

Fri Sep  9 23:21:54 EDT [vfiler2_onstar@dtw-netapp3/dtw-netapp4: vFiler.initialized:notice]: vfiler "vfiler2_onstar" initialized.

Fri Sep  9 23:21:54 EDT [vfiler1@dtw-netapp3/dtw-netapp4: vFiler.initialized:notice]: vfiler "vfiler1" initialized.

Fri Sep  9 23:21:54 EDT [dtw-netapp3/dtw-netapp4: proto_init02:info]: Vfiler discovery complete

It appears that there is a problem with the vif (nas) and none of the vlans came up.

The vif is defined the same on both controllers and I have defined the primary and partner interfaces and the same ipspaces on both controllers. I have a very limited window to test this again so I would appreciate any help indetermining what I am missing.

Thank you.

dtw-netapp3> rdfile /etc/rc

ifconfig e0a `hostname`-e0a mediatype auto flowcontrol full netmask  255.255.255.248 partner 192.168.100.166

vif create multi nas -b ip e3a e3b

vlan create nas 969

vlan add nas 970

vlan add nas 967

ifconfig nas-967 192.168.93.129 up netmask 255.255.255.224 partner nas-967 mtusize 9000 -wins

ifconfig nas-969 192.168.60.50 up netmask 255.255.255.192 partner nas-969 mtusize 9000 -wins

ifconfig nas-970 192.168.99.129 up netmask 255.255.255.240 partner nas-970 mtusize 9000 -wins

route add inet default 192.168.100.161 4

routed off

options dns.enable off

options nis.enable off

savecore

dtw-netapp4> rdfile /etc/rc

ifconfig e0a `hostname`-e0a mediatype auto flowcontrol full netmask  255.255.255.248 partner 192.168.100.165

vif create multi nas -b ip e3a e3b

vlan create nas 969

vlan add nas 967

vlan add nas 970

ifconfig nas-967 partner nas-967

ifconfig nas-969 partner nas-969

ifconfig nas-970 partner nas-970

route add inet default 192.168.100.161 4

routed off

options dns.enable off

options nis.enable off

savecore

dtw-netapp3> ipspace list

Number of ipspaces configured: 4

default-ipspace                   (e0M e0a e0b)

APM                               (nas-967)

Onstar                            (nas-970)

vf1-ipspace                       (nas-969)

dtw-netapp4> ipspace list

Number of ipspaces configured: 4

default-ipspace                   (e0M e0a e0b)

APM                               (nas-967)

Onstar                            (nas-970)

vf1-ipspace                       (nas-969)

dtw-netapp3> ifconfig -a

e0M: flags=0x210c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

        ether 00:a0:98:26:18:36 (auto-unknown-cfg_down) flowcontrol full

e0a: flags=0x2d4c867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

        inet 192.168.100.165 netmask-or-prefix 0xfffffff8 broadcast 192.168.100.167

        partner inet 192.168.100.166 (not in use)

        ether 00:a0:98:26:18:34 (auto-1000t-fd-up) flowcontrol full

e0b: flags=0x250c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

        ether 00:a0:98:26:18:35 (auto-unknown-cfg_down) flowcontrol full

e3a: flags=0x8bd0a867<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 9000

        ether 02:a0:98:26:18:36 (auto-10g_sr-fd-up) flowcontrol full

        trunked nas

e3b: flags=0x8bd0a867<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 9000

        ether 02:a0:98:26:18:36 (auto-10g_sr-fd-up) flowcontrol full

        trunked nas

lo: flags=0x1948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 8160

        inet 127.0.0.1 netmask-or-prefix 0xff000000 broadcast 127.0.0.1

        ether 00:00:00:00:00:00 (VIA Provider)

nas: flags=0xa3d0a863<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 9000

        ether 02:a0:98:26:18:36 (Enabled virtual interface)

nas-969: flags=0x794a863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM,NOWINS> mtu 9000

        inet 192.168.60.50 netmask-or-prefix 0xffffffc0 broadcast 192.168.60.63

        partner nas-969 (not in use)

        ether 02:a0:98:26:18:36 (Enabled virtual interface)

nas-970: flags=0x794a863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM,NOWINS> mtu 9000

        inet 192.168.99.129 netmask-or-prefix 0xfffffff0 broadcast 192.168.99.143

        partner nas-970 (not in use)

        ether 02:a0:98:26:18:36 (Enabled virtual interface)

nas-967: flags=0x794a863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM,NOWINS> mtu 9000

        inet 192.168.93.129 netmask-or-prefix 0xffffffe0 broadcast 192.168.93.159

        partner nas-967 (not in use)

        ether 02:a0:98:26:18:36 (Enabled virtual interface)

dtw-netapp4> ifconfig -a

e0M: flags=0x210c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

        ether 00:a0:98:26:18:22 (auto-unknown-cfg_down) flowcontrol full

e0a: flags=0x2d4c867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

        inet 192.168.100.166 netmask-or-prefix 0xfffffff8 broadcast 192.168.100.167

        partner inet 192.168.100.165 (not in use)

        ether 00:a0:98:26:18:20 (auto-1000t-fd-up) flowcontrol full

e0b: flags=0x250c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

        ether 00:a0:98:26:18:21 (auto-unknown-cfg_down) flowcontrol full

e3a: flags=0x8b50a866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 1500

        ether 02:a0:98:26:18:22 (auto-unknown-cfg_down) flowcontrol full

        trunked nas

e3b: flags=0x8b50a866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 1500

        ether 02:a0:98:26:18:22 (auto-unknown-cfg_down) flowcontrol full

        trunked nas

lo: flags=0x1948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 8160

        inet 127.0.0.1 netmask-or-prefix 0xff000000 broadcast 127.0.0.1

        ether 00:00:00:00:00:00 (VIA Provider)

nas: flags=0xa2408862<BROADCAST,RUNNING,MULTICAST,VLAN> mtu 1500

        ether 02:a0:98:26:18:22 (Disabled virtual interface)

nas-969: flags=0x2008862<BROADCAST,RUNNING,MULTICAST> mtu 1500

        partner nas-969 (not in use)

        ether 02:a0:98:26:18:22 (Disabled virtual interface)

nas-967: flags=0x2008862<BROADCAST,RUNNING,MULTICAST> mtu 1500

        partner nas-967 (not in use)

        ether 02:a0:98:26:18:22 (Disabled virtual interface)

nas-970: flags=0x2008862<BROADCAST,RUNNING,MULTICAST> mtu 1500

        partner nas-970 (not in use)

        ether 02:a0:98:26:18:22 (Disabled virtual interface)

[lovik_netapp]

Did you try checking gateway as the only thing which may be missing is vfiler specific default gateway entry, other than that everything looks correct in your rc file.

I would recommend opening a support case with netapp global support to troubleshoo the issue.

Cheers.

[scottgelb · Accepted solution]

After takeover, does "vfiler status -a" show the interfaces as Unconfigured or are they configured?

Also...on a hunch here... set mtusize 9000 on the dtw-netapp4 rc file ifconfig as well as the partner parameter.  They don't match for failover...I would edit rc and also run ifconfig to set mtusize manually since rc only takes on reboot.

dtw-netapp4

ifconfig nas-967 partner nas-967 mtusize 9000

ifconfig nas-969 partner nas-969  mtusize 9000

ifconfig nas-970 partner nas-970  mtusize 9000

One other possible issue (might not be an issue) is no default route for each of the non-default ipspaces.  If all hosts are on the same subnet it isn't needed, but in most cases the default route is needed for each IPspace as well.  If you need to get to different subnets, then the rc file (rc file in vfiler0 since there is no rc per vfiler) would need "vfiler run vfilername route add default ..."...for example..

dtw-netapp3

ifconfig nas-967 192.168.93.129 up netmask 255.255.255.224 partner nas-967 mtusize 9000 -wins

ifconfig nas-969 192.168.60.50 up netmask 255.255.255.192 partner nas-969 mtusize 9000 -wins

ifconfig nas-970 192.168.99.129 up netmask 255.255.255.240 partner nas-970 mtusize 9000 -wins

route add inet default 192.168.100.161 4

vfiler run vfiler1 route add default x.x.x.x n

vfiler run vfiler2_onstar route add default x.x.x.x n

vfiler run vfiler3_apm route add default x.x.x.x n

routed off

options dns.enable off

options nis.enable off

savecore

[mfhlmr001]

Thanks for the suggestions. I will report the results after my next test.

Regards,

Larry

[scottgelb]

Cool.. let us know.  I would look at both vfiler status -a before and after takeover and also netstat -rn from the vfiler context or vfiler run to see the network is connected... I think the mtu will help...then if you need routes or not.

[mfhlmr001]

Adding mtusize 9000 to the dtw-netapp4 ifconfig partner statements corrected the problem. We tested failover and it worked without a problem.

Thanks for the help!

Larry 

[scottgelb]

Very good. Thank you for the update.

Typos Sent on Blackberry Wireless