Multistore failover fails

JOHANNESWOLF · ‎2013-04-05

Hi together,

got a massive problem when i failover a customer MetroCluster 3140. All the cifs-vfilers didnt failover.

The customer creates the vfilers with Systemmanager 2.0 😞 so i repair the rc file and the ipspace list, but this didnt solve the issue.

Everytime after a failover when i check the vfiler status -r on partner side there stand unconfigured

The ipadresses are correct and the vfilers running, but the ipadress didnt get matched on a interface.

Maybe you have a solution?

Here are ifconfig -a / ipspacelist /etc/rc and etc/hosts

Thanks

Joe

hostname NetApp03

ifgrp create lacp VIF1 e2a e2b

vlan create VIF1 106 107 108 117 251 301 401 503

ifconfig VIF1-301 192.168.242.105 netmask 255.255.255.0 partner VIF1-301

ifconfig VIF1-107 192.168.7.105 netmask 255.255.255.0 partner VIF1-107

ifconfig VIF1-401 192.168.240.125 netmask 255.255.255.0 partner VIF1-401

ifconfig VIF1-251 192.168.201.105 netmask 255.255.255.0 partner VIF1-251

ifconfig VIF1-117 partner VIF1-117

ifconfig VIF1-106 partner VIF1-106

ifconfig VIF1-503 partner VIF1-503

ifconfig VIF1-108 partner VIF1-108

route add default 192.168.242.1 1

vfiler run Firstxpstore route add default 192.168.7.1 1

vfiler run DSIMstore route add default 192.168.201.1 1

vfiler run NMSstore route add default 192.168.240.1 1

routed on

options dns.domainname xxx.xxx

options dns.enable on

options nis.enable off

-----------------------------------------------------------------------------------------------------------

NetApp03> ipspace list

Number of ipspaces configured: 8

default-ipspace (e0a e0b e0M losk VIF1 VIF1-301)

DSIMstore (VIF1-251)

Firstxpstore (VIF1-107)

nmsstore (VIF1-401)

Edmstore (VIF1-108)

EEOstore (VIF1-117)

Firstxtstore (VIF1-106)

Scalastore (VIF1-503)

-----------------------------------------------------------------------------------------------------------

e0a: flags=0x2f08866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 00:a0:98:0b:6a:4c (auto-1000t-fd-up) flowcontrol full

e0b: flags=0x2f08866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 00:a0:98:0b:6a:4d (auto-1000t-fd-up) flowcontrol full

e2a: flags=0x89f08867<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 1500

ether 02:a0:98:0b:6a:4c (auto-10g_sr-fd-up) flowcontrol full

trunked VIF1

e2b: flags=0x89f08867<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 1500

ether 02:a0:98:0b:6a:4c (auto-10g_sr-fd-up) flowcontrol full

trunked VIF1

e0M: flags=0x2b08866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 00:a0:98:0b:6a:4e (auto-100tx-fd-up) flowcontrol full

lo: flags=0x1b48049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 8160

inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1

ether 00:00:00:00:00:00 (VIA Provider)

losk: flags=0x40a400c9<UP,LOOPBACK,RUNNING> mtu 9188

inet 127.0.20.1 netmask 0xff000000 broadcast 127.0.20.1

VIF1: flags=0xa2f08863<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 1500

ether 02:a0:98:0b:6a:4c (Enabled interface groups)

VIF1-107: flags=0x2b48863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

inet 192.168.7.105 netmask 0xffffff00 broadcast 192.168.7.255

partner VIF1-107 (not in use)

ether 02:a0:98:0b:6a:4c (Enabled interface groups)

VIF1-108: flags=0x2308862<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 02:a0:98:0b:6a:4c (Enabled interface groups)

VIF1-117: flags=0x2308862<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 02:a0:98:0b:6a:4c (Enabled interface groups)

VIF1-251: flags=0x2b48863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

inet 192.168.201.105 netmask 0xffffff00 broadcast 192.168.201.255

partner VIF1-251 (not in use)

ether 02:a0:98:0b:6a:4c (Enabled interface groups)

VIF1-301: flags=0x2b48863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

inet 192.168.242.105 netmask 0xffffff00 broadcast 192.168.242.255

partner VIF1-301 (not in use)

ether 02:a0:98:0b:6a:4c (Enabled interface groups)

VIF1-401: flags=0x2b48863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

inet 192.168.240.125 netmask 0xffffff00 broadcast 192.168.240.255

partner VIF1-401 (not in use)

ether 02:a0:98:0b:6a:4c (Enabled interface groups)

VIF1-503: flags=0x2308862<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 02:a0:98:0b:6a:4c (Enabled interface groups)

VIF1-106: flags=0x2308862<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 02:a0:98:0b:6a:4c (Enabled interface groups)

----------------------------------------------------------------------------------------------------------------------

127.0.0.1 localhost

192.168.242.105 NetApp03 NetApp03-VIF1

# 0.0.0.0 NetApp03-e0M

# 0.0.0.0 NetApp03-e2a

# 0.0.0.0 NetApp03-e2b

192.168.115.10 xxxxxx mailhost

192.168.242.106 NetApp04

172.16.20.125 NetApp03-VIF1-503

192.168.4.140 NetApp03-VIF1-108

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

hostname NetApp04

ifgrp create lacp VIF1 e2a e2b

vlan create VIF1 106 107 108 117 251 301 401 503

ifconfig VIF1-301 192.168.242.106 netmask 255.255.255.0 partner VIF1-301

ifconfig VIF1-117 192.168.117.106 netmask 255.255.255.0 partner VIF1-117

ifconfig VIF1-106 192.168.6.106 netmask 255.255.255.0 partner VIF1-106

ifconfig VIF1-503 172.16.20.125 netmask 255.255.255.0 partner VIF1-503

ifconfig VIF1-108 192.168.4.140 netmask 255.255.255.128 partner VIF1-108

ifconfig VIF1-107 partner VIF1-107

ifconfig VIF1-401 partner VIF1-401

ifconfig VIF1-251 partner VIF1-251

route add default 192.168.242.1 1

vfiler run Firstxtstore route add default 192.168.6.1 1

vfiler run EEOstore route add default 192.168.201.1 1

vfiler run Scalastore route add default 172.16.20.1 1

vfiler run Edmstore route add default 192.168.4.128 1

routed on

options dns.domainname xxx.xxx

options dns.enable on

options nis.enable off

----------------------------------------------------------------------------------------------------------

NetApp04> ipspace list

Number of ipspaces configured: 8

default-ipspace (e0a e0b e0M losk VIF1 VIF1-301)

DSIMstore (VIF1-251)

Edmstore (VIF1-108)

EEOstore (VIF1-117)

Firstxpstore (VIF1-107)

Firstxtstore (VIF1-106)

nmsstore (VIF1-401)

Scalastore (VIF1-503)

------------------------------------------------------------------------------------------------------------

e0a: flags=0x2f08866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 00:a0:98:0b:6a:3c (auto-1000t-fd-up) flowcontrol full

e0b: flags=0x2f08866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 00:a0:98:0b:6a:3d (auto-1000t-fd-up) flowcontrol full

e2a: flags=0x89f08867<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 1500

ether 02:a0:98:0b:6a:3c (auto-10g_sr-fd-up) flowcontrol full

trunked VIF1

e2b: flags=0x89f08867<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 1500

ether 02:a0:98:0b:6a:3c (auto-10g_sr-fd-up) flowcontrol full

trunked VIF1

e0M: flags=0x2b08866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 00:a0:98:0b:6a:3e (auto-100tx-fd-up) flowcontrol full

lo: flags=0x1b48049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 8160

inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1

ether 00:00:00:00:00:00 (VIA Provider)

losk: flags=0x40a400c9<UP,LOOPBACK,RUNNING> mtu 9188

inet 127.0.20.1 netmask 0xff000000 broadcast 127.0.20.1

VIF1: flags=0xa2f08863<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 1500

ether 02:a0:98:0b:6a:3c (Enabled interface groups)

VIF1-106: flags=0x2b48863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

inet 192.168.6.106 netmask 0xffffff00 broadcast 192.168.6.255

partner VIF1-106 (not in use)

ether 02:a0:98:0b:6a:3c (Enabled interface groups)

VIF1-107: flags=0x2308862<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

partner VIF1-107 (not in use)

ether 02:a0:98:0b:6a:3c (Enabled interface groups)

VIF1-108: flags=0x2b48863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

inet 192.168.4.140 netmask 0xffffff80 broadcast 192.168.4.255

partner VIF1-108 (not in use)

ether 02:a0:98:0b:6a:3c (Enabled interface groups)

VIF1-117: flags=0x2b48863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

inet 192.168.117.106 netmask 0xffffff00 broadcast 192.168.117.255

partner VIF1-117 (not in use)

ether 02:a0:98:0b:6a:3c (Enabled interface groups)

VIF1-251: flags=0x2308862<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

partner VIF1-251 (not in use)

ether 02:a0:98:0b:6a:3c (Enabled interface groups)

VIF1-301: flags=0x2b48863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

inet 192.168.242.106 netmask 0xffffff00 broadcast 192.168.242.255

partner VIF1-301 (not in use)

ether 02:a0:98:0b:6a:3c (Enabled interface groups)

VIF1-401: flags=0x2308862<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

partner VIF1-401 (not in use)

ether 02:a0:98:0b:6a:3c (Enabled interface groups)

VIF1-503: flags=0x2b48863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

inet 172.16.20.125 netmask 0xffffff00 broadcast 172.16.20.255

partner VIF1-503 (not in use)

ether 02:a0:98:0b:6a:3c (Enabled interface groups)

-----------------------------------------------------------------------------------------------------------

#Auto-generated by setup Thu Dec 30 16:37:00 GMT 2010

127.0.0.1 localhost

192.168.242.106 NetApp04 NetApp04-VIF1

# 0.0.0.0 NetApp04-e0M

# 0.0.0.0 NetApp04-e2a

# 0.0.0.0 NetApp04-e2b

192.168.115.10 xxxxx mailhost

192.168.242.105 NetApp03

192.168.242.208 NetApp04-e2a

192.168.242.20 NetApp04-e2b

172.16.20.106 NetApp04-VIF1-503

192.168.4.140 NetApp04-VIF1-108

brauntvr2swiss · ‎2013-04-05

Hi Joe

Could you run a vfiler status -r on both Filers and put the output in this thread?

Does all vfiers Fails (unconfigured) during a takeover or only a few?

regards

Thomas

pauli · ‎2013-04-05

Hi Joe,

it is clear that you got in Trouble! On Controller 03 there area few Partner IF missing in ifconfig!

VIF1-108: flags=0x2308862<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 02:a0:98:0b:6a:4c (Enabled interface groups)

VIF1-117: flags=0x2308862<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 02:a0:98:0b:6a:4c (Enabled interface groups)

VIF1-503: flags=0x2308862<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 02:a0:98:0b:6a:4c (Enabled interface groups)

VIF1-106: flags=0x2308862<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

ether 02:a0:98:0b:6a:4c (Enabled interface groups)

Even if you have them in rc file, if the are not set, you will get in Trouble durring Failover.

after the giveack, there are set because they are in rc file. Check with ifconfig -a!

BTW: I always tell my customer to add an IP address to the VIFS, even if they are not used! So they can ping from

Controller to Controller to check if the Network Interfaces work and if the vlans are Setup on both side!

This prevent from Problems like the vlans are not configured on the Switch on the side where the Controller is with "normally" no IP address.

Thanks,

Horst

JOHANNESWOLF · ‎2013-04-05

Sry the customer has no direct internet connection and i cant get some output from my office.

But i know that all vfilers are in unconfigured status during a takeover. I´ll send the customer a mail, maybe he can send me the informations.

regards

joe