topic Multistore Failover and Networking in ONTAP Discussions

Multistore Failover and Networking

mfhlmr001 — Thu, 05 Jun 2025 06:45:58 GMT

Over the weekend I tested failover on one of our cluster nodes running multistore (dtw-netapp3 to dtw-netapp4) and although the failover completed the network failed. The 3 vfilers all started on the partner but the network interfaces were all down. I saw the following messages on the console but there was not enough detail to determine the exact problem.

vif: nas is not mapped to a local vif

add net default: gateway 192.168.100.161

PIP: protocol filter cannot be inspected (not applicable in partner mode)

Fri Sep 9 23:21:53 EDT [dtw-netapp4: net.ifconfig.takeoverError:warning]: WARNING: 3 errors detected during network takeover processing WARNING: Some network clients may not be able to access the cluster during takeover

Fri Sep 9 23:21:53 EDT [dtw-netapp4: cf.rsrc.takeoverOpFail:error]: Cluster monitor: takeover during ifconfig_2 failed; takeover continuing...

Fri Sep 9 23:21:54 EDT [vfiler3_apm@dtw-netapp3/dtw-netapp4: vFiler.initialized:notice]: vfiler "vfiler3_apm" initialized.

Fri Sep 9 23:21:54 EDT [vfiler2_onstar@dtw-netapp3/dtw-netapp4: vFiler.initialized:notice]: vfiler "vfiler2_onstar" initialized.

Fri Sep 9 23:21:54 EDT [vfiler1@dtw-netapp3/dtw-netapp4: vFiler.initialized:notice]: vfiler "vfiler1" initialized.

Fri Sep 9 23:21:54 EDT [dtw-netapp3/dtw-netapp4: proto_init02:info]: Vfiler discovery complete

It appears that there is a problem with the vif (nas) and none of the vlans came up.

The vif is defined the same on both controllers and I have defined the primary and partner interfaces and the same ipspaces on both controllers. I have a very limited window to test this again so I would appreciate any help indetermining what I am missing.

Thank you.

dtw-netapp3> rdfile /etc/rc

ifconfig e0a `hostname`-e0a mediatype auto flowcontrol full netmask 255.255.255.248 partner 192.168.100.166

vif create multi nas -b ip e3a e3b

vlan create nas 969

vlan add nas 970

vlan add nas 967

ifconfig nas-967 192.168.93.129 up netmask 255.255.255.224 partner nas-967 mtusize 9000 -wins

ifconfig nas-969 192.168.60.50 up netmask 255.255.255.192 partner nas-969 mtusize 9000 -wins

ifconfig nas-970 192.168.99.129 up netmask 255.255.255.240 partner nas-970 mtusize 9000 -wins

route add inet default 192.168.100.161 4

routed off

options dns.enable off

options nis.enable off

savecore

dtw-netapp4> rdfile /etc/rc

ifconfig e0a `hostname`-e0a mediatype auto flowcontrol full netmask 255.255.255.248 partner 192.168.100.165

vif create multi nas -b ip e3a e3b

vlan create nas 969

vlan add nas 967

vlan add nas 970

ifconfig nas-967 partner nas-967

ifconfig nas-969 partner nas-969

ifconfig nas-970 partner nas-970

route add inet default 192.168.100.161 4

routed off

options dns.enable off

options nis.enable off

savecore

dtw-netapp3> ipspace list

Number of ipspaces configured: 4

default-ipspace (e0M e0a e0b)

APM (nas-967)

Onstar (nas-970)

vf1-ipspace (nas-969)

dtw-netapp4> ipspace list

Number of ipspaces configured: 4

default-ipspace (e0M e0a e0b)

APM (nas-967)

Onstar (nas-970)

vf1-ipspace (nas-969)

dtw-netapp3> ifconfig -a

e0M: flags=0x210c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 00:a0:98:26:18:36 (auto-unknown-cfg_down) flowcontrol full

e0a: flags=0x2d4c867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

inet 192.168.100.165 netmask-or-prefix 0xfffffff8 broadcast 192.168.100.167

partner inet 192.168.100.166 (not in use)

ether 00:a0:98:26:18:34 (auto-1000t-fd-up) flowcontrol full

e0b: flags=0x250c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 00:a0:98:26:18:35 (auto-unknown-cfg_down) flowcontrol full

e3a: flags=0x8bd0a867<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 9000

ether 02:a0:98:26:18:36 (auto-10g_sr-fd-up) flowcontrol full

trunked nas

e3b: flags=0x8bd0a867<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 9000

ether 02:a0:98:26:18:36 (auto-10g_sr-fd-up) flowcontrol full

trunked nas

lo: flags=0x1948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 8160

inet 127.0.0.1 netmask-or-prefix 0xff000000 broadcast 127.0.0.1

ether 00:00:00:00:00:00 (VIA Provider)

nas: flags=0xa3d0a863<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 9000

ether 02:a0:98:26:18:36 (Enabled virtual interface)

nas-969: flags=0x794a863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM,NOWINS> mtu 9000

inet 192.168.60.50 netmask-or-prefix 0xffffffc0 broadcast 192.168.60.63

partner nas-969 (not in use)

ether 02:a0:98:26:18:36 (Enabled virtual interface)

nas-970: flags=0x794a863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM,NOWINS> mtu 9000

inet 192.168.99.129 netmask-or-prefix 0xfffffff0 broadcast 192.168.99.143

partner nas-970 (not in use)

ether 02:a0:98:26:18:36 (Enabled virtual interface)

nas-967: flags=0x794a863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM,NOWINS> mtu 9000

inet 192.168.93.129 netmask-or-prefix 0xffffffe0 broadcast 192.168.93.159

partner nas-967 (not in use)

ether 02:a0:98:26:18:36 (Enabled virtual interface)

dtw-netapp4> ifconfig -a

e0M: flags=0x210c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 00:a0:98:26:18:22 (auto-unknown-cfg_down) flowcontrol full

e0a: flags=0x2d4c867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

inet 192.168.100.166 netmask-or-prefix 0xfffffff8 broadcast 192.168.100.167

partner inet 192.168.100.165 (not in use)

ether 00:a0:98:26:18:20 (auto-1000t-fd-up) flowcontrol full

e0b: flags=0x250c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 00:a0:98:26:18:21 (auto-unknown-cfg_down) flowcontrol full

e3a: flags=0x8b50a866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 1500

ether 02:a0:98:26:18:22 (auto-unknown-cfg_down) flowcontrol full

trunked nas

e3b: flags=0x8b50a866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 1500

ether 02:a0:98:26:18:22 (auto-unknown-cfg_down) flowcontrol full

trunked nas

lo: flags=0x1948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 8160

inet 127.0.0.1 netmask-or-prefix 0xff000000 broadcast 127.0.0.1

ether 00:00:00:00:00:00 (VIA Provider)

nas: flags=0xa2408862<BROADCAST,RUNNING,MULTICAST,VLAN> mtu 1500

ether 02:a0:98:26:18:22 (Disabled virtual interface)

nas-969: flags=0x2008862<BROADCAST,RUNNING,MULTICAST> mtu 1500

partner nas-969 (not in use)

ether 02:a0:98:26:18:22 (Disabled virtual interface)

nas-967: flags=0x2008862<BROADCAST,RUNNING,MULTICAST> mtu 1500

partner nas-967 (not in use)

ether 02:a0:98:26:18:22 (Disabled virtual interface)

nas-970: flags=0x2008862<BROADCAST,RUNNING,MULTICAST> mtu 1500

partner nas-970 (not in use)

ether 02:a0:98:26:18:22 (Disabled virtual interface)

Multistore Failover and Networking

lovik_netapp — Tue, 13 Sep 2011 02:01:06 GMT

Did you try checking gateway as the only thing which may be missing is vfiler specific default gateway entry, other than that everything looks correct in your rc file.

I would recommend opening a support case with netapp global support to troubleshoo the issue.

Cheers.

Multistore Failover and Networking

scottgelb — Tue, 13 Sep 2011 04:37:16 GMT

After takeover, does "vfiler status -a" show the interfaces as Unconfigured or are they configured?

Also...on a hunch here... set mtusize 9000 on the dtw-netapp4 rc file ifconfig as well as the partner parameter. They don't match for failover...I would edit rc and also run ifconfig to set mtusize manually since rc only takes on reboot.

dtw-netapp4

ifconfig nas-967 partner nas-967 mtusize 9000

ifconfig nas-969 partner nas-969 mtusize 9000

ifconfig nas-970 partner nas-970 mtusize 9000

One other possible issue (might not be an issue) is no default route for each of the non-default ipspaces. If all hosts are on the same subnet it isn't needed, but in most cases the default route is needed for each IPspace as well. If you need to get to different subnets, then the rc file (rc file in vfiler0 since there is no rc per vfiler) would need "vfiler run vfilername route add default ..."...for example..

dtw-netapp3

ifconfig nas-967 192.168.93.129 up netmask 255.255.255.224 partner nas-967 mtusize 9000 -wins

ifconfig nas-969 192.168.60.50 up netmask 255.255.255.192 partner nas-969 mtusize 9000 -wins

ifconfig nas-970 192.168.99.129 up netmask 255.255.255.240 partner nas-970 mtusize 9000 -wins

route add inet default 192.168.100.161 4

vfiler run vfiler1 route add default x.x.x.x n

vfiler run vfiler2_onstar route add default x.x.x.x n

vfiler run vfiler3_apm route add default x.x.x.x n

routed off

options dns.enable off

options nis.enable off

savecore

Multistore Failover and Networking

mfhlmr001 — Tue, 13 Sep 2011 21:39:02 GMT

Thanks for the suggestions. I will report the results after my next test.

Regards,

Larry

Multistore Failover and Networking

scottgelb — Wed, 14 Sep 2011 00:31:44 GMT

Cool.. let us know. I would look at both vfiler status -a before and after takeover and also netstat -rn from the vfiler context or vfiler run to see the network is connected... I think the mtu will help...then if you need routes or not.

Multistore Failover and Networking

mfhlmr001 — Mon, 26 Sep 2011 18:16:26 GMT

Adding mtusize 9000 to the dtw-netapp4 ifconfig partner statements corrected the problem. We tested failover and it worked without a problem.

Thanks for the help!

Larry

Re: Multistore Failover and Networking

scottgelb — Mon, 26 Sep 2011 18:33:05 GMT

Very good. Thank you for the update.

Typos Sent on Blackberry Wireless