Multistore failover

[ccolht]

I'm trying to wrap my head around vfiler configs in HA pairs. I've fount bits and pieces in various docs but haven't found a comprehensive guide.

Exactly how does a vfiler failover between nodes? What do I need to make this work? Could someone provide example rc files?

Here is where I'm at:

FilerA has several vlans on vif0: 101, 102 103 104

FilerB has the same vlans defined

FilerA has several vfilers with ipspaces:

vfA1, ipspace ip-vfA1 vif0-101 10.0.0.1/24

vfA2, ipspace ip-vfA2 vif0-102 10.0.2.1/24

FilerB has several vfilers with ipspaces:

vfB1, ipspace ip-vfB1 vif0-103 10.0.0.1/8

vfB2, ipspace ip-vfB2 vif0-104 10.0.3.1/24

All ipspaces on both filers with same interfaces

The ifconfig lines in the rc files specify the partner interfaces, e.g.

on FilerA.:ifconfig vif0-102 10.0.2.1 netmask 255.255.255.0 partner vif0-102

on FilerB: ifconfig vif0-102 10.0.2.4 netmask 255.255.255.0 partner vif0-102

The owning filer's rc file has the default route for each vfiler specified with a vfiler run and a default route for default-ipspace.

When I failover, the interfaces don't work on the vfilers. I see this on the console of the takeover filer:

Mon Apr  4 17:35:38 GMT [FilerB: net.ifconfig.noPartner:error]: ifconfig: 'vif0-102' cannot be configured: Address does not match any partner interface.

Does the partner need an ip assigned to it's interface (on the same subnet as the primary)? I tried this but it didn't work:

FilerB> ifconfig vif0-102 10.0.2.4 netmask 255.255.255..0
ifconfig: can't assign address to vif0-102, no vfiler in this interface's ipspace (ip-vfA2) has this address

So it looks like the rc file ifconfigs won't work the way I have them on the partners.

If I try to configure a shared interface in System Manager on the partner, it insists on an ip but fails because there is no vfiler on the partner in that ipspace. Everything in the default-ipspace works fine.

[scottgelb]

They key thing is that the ipspace exists on both controllers.  Then the vlan must be assigned to that ipspace.  On the partner, you don't need an ip and can have just the partner statement in the ifconfig.  I wouldn't use system manager for this.  Also, you likely will need to add "vfiler run vfilername route add default x.x.x.x 1" to each rc file for each vFlier if you need a route out (if not a flat network).  I usually see this issue when the failover partner interface isn't assigned to the same ipspace as the partner controller.

[ccolht]

So my primary rc file looks like this:

...

ifconfig vif0-102 10.0.2.2 netmask 255.255.255.0 mtusize 1500 trusted -wins partner vif0-102
vfiler run vfA1 route add default 10.0.2.99 1
...

and the partner's rc file looks like this?

...

ifconfig vif0-102 partner vif0-102

...

ipspaces, vlans and interfaces are identical on both filers.

[scottgelb · Accepted solution]

That should work.. Test a failover.

Typos Sent on Blackberry Wireless

[ccolht]

that did the trick. Failed over and back again successfully.

Thanks!

So to summarize,

Match ipspaces

Match  interfaces (vlans, etc)

For each vfiler:

     In primary /etc/rc

          set ip and partner interface

               ifconfig vif0-102 10.0.2.1 netmask 255.255.255.0 partner vif0-102

          set vfiler's default route in vfiler0 /etc/rc if needed

               vfiler run vfA1 route add default 10.0.2.99 1

     In secondary /etc/rc

          set partner interface

               ifconfig vif0-102 partner vif0-102

[scottgelb]

Perfect the rule of thumb is always create ipspaces with the same name on both controllers and assign the partner interface to the ipspace.. Interface names don't need to be the same but it is a best practice we follow to make things easy. We also prefer putting the ip address in hosts instead if rc and reference the name in the ifconfig.

Typos Sent on Blackberry Wireless

[fletch2007]

I'm getting these errors on cci-na02  (where I initiated the "cf takeover" to test):

cci01-vif0 is not mapped to a local ifgrp

ipspace command cannot be run in partner mode

ipspace command cannot be run in partner mode

ifconfig: 'cci01-vif0-205' cannot be configured: Address does not match any partner interface.

ifconfig: cci01-vif0-205: no such interface

Do the ifgrp names need to match?  What am I missing?  - thanks

Here is the cci-na02 /etc/rc

#Auto-generated by zapi_net_config Mon Mar  4 11:26:51 PST 2013

hostname cci-na02

ifgrp create single cci02-vif0 e1b e2b

vlan create  cci02-vif0 205

ipspace create v205

ipspace assign v205 cci02-vif0-205

vlan create cci02-vif0 3162

ifconfig cci02-vif0-3162 `hostname`-cci02-vif0-3162 netmask 255.255.254.0 partner cci01-vif0-3162 up

route add net default 10.39.22.1 1

options dns.enable on

options nis.enable off

savecore

cci-na01's /etc/rc

#Auto-generated by zapi_net_config Mon Mar  4 11:26:51 PST 2013

hostname cci-na01

ifgrp create single cci01-vif0 e1b e2b

vlan create  cci01-vif0 205

ipspace create v205

ipspace assign v205 cci01-vif0-205

vlan create cci01-vif0 3162

ifconfig cci01-vif0-3162 `hostname`-cci01-vif0-3162 netmask 255.255.254.0 partner cci02-vif0-3162 up

ifconfig cci01-vif0-205 `hostname`-cci01-vif0-205 netmask 255.255.255.0 partner cci02-vif0-205 up

vfiler run cci-vf-02 route add net default 172.25.124.1 1

route add net default 10.39.252.1 1

options dns.enable on

options nis.enable off

savecore

[scottgelb]

When you run Ipspace list, are the interfaces in the same Ipspace? Ifgrp name doesn't matter as long as setup as partner interfaces to each other. But Ipspace assignment to the same Ipspace name does matter.

Sent from my iPhone 5

[fletch2007]

cci01-vif0-205 and cci02-vif0-205 are in the same ipspace "v205" - is there a TR doc on this? - thanks

cci-na01> ipspace list

Number of ipspaces configured: 2

default-ipspace                   (e0a e0b e1a e2a e3a e3b e0M losk cci01-vif0 cci01-vif0-3162)

v205                              (cci01-vif0-205)

cci-na02> ipspace list

Number of ipspaces configured: 2

default-ipspace                   (e0a e0b e1a e2a e3a e3b e0M losk cci02-vif0 cci02-vif0-3162)

v205                              (cci02-vif0-205)

[scottgelb]

There are some old TRs but the best info is the data ontap multistore guides then the labs I have posted on here. Also LoD has the Lans now too.

Sent from my iPhone 5

[scottgelb]

ipspace create and ipspace assign are persistent in the registry so are not needed (will error as redundant) in the rc file.

I see the issue I think…  The cci-na02 rc file is missing the ifconfig for cci02-vif0-205… even though no IP on it it still needs to be setup for partner failover.

add the line to ccio2 rc file

ifconfig ccio2-vif0-205 partner cci01-vif0-205

[fletch2007]

Yes, adding that line resulted in a successful (vfiler NFS remained uninterrupted while the parent node rebooted)

While in takeover mode I issue a "vfiler status" on cci-na02:

cci-na02(takeover)*> vfiler status

vfiler0                          running

I expect to see the vfiler cci-vf-02 it took over, but I don't.

Similarly I issue "ifconfig -a" on cci-na02 and expect to see the ip address of the taken over vfiler, but I do not.

Is this hidden in the registry too?

thanks

[scottgelb]

Type "partner vfiler status" to see the virtual takeover node vfilers.

Sent from my iPhone 5

[brauntvr2swiss]

Hi

You have to type "partner" on the console, then you are on the failovered Filer...

Regards

Thomas

[fletch2007]

yes - prefixing with partner showed vfiler and ifconfig as expected

thanks!

[mdvillanueva]

Hi Scott,

Can you please help me identify what is going on with setup. I have a vfiler which has its own ipspace and the interface is a vlan-tagged interface. the interface is not failing over.

Here is some lines in my filer where the vfiler is created:

RC

====

vlan create NET1 345

ifconfig NET1-345 192.168.45.130 netmask 255.255.255.0 partner NET1-345 up

vfiler run vfiler1 route add default 192.168.45.1 1

Here is for the partner filer:

RC

==

vlan create NET1 345

ifconfig NET1-345 partner NET1-345 up

I don't understand why it does not work when failed over. During the takeover, I will type 'partner' to and then I ran ifconfig -a. I don't see the NET1-345.

I saw this in the log during the takeover:

ifconfig: 'NET1-345' cannot be configured: Address does not match any partner interface

[mdvillanueva]

could it be that it is failing because I put "up" at the end of my ifconfig statement?

[scottgelb]

Is the Ipspace present on the partner and was this vlan added to that Ipspace? Ipspace list to check. I suspect that is the issue

Sent from my iPhone 5

[mdvillanueva]

It is there.

here is the filer owner of the vfiler

filer2> ipspace list

Number of ipspaces configured: 2

default-ipspace (e0M losk NET1 NET1-350 NET1-348)

zylab_ipspace (NET1-345)

here is the partner

filer1> ipspace list

Number of ipspaces configured: 2

default-ipspace (e0M losk NET1 NET1-350 NET1-348)

zylab_ipspace (NET1-345)

I am not sure what I am missing.

[scottgelb]

Do you have vlan create in the rc file?

Sent from my iPhone 5

[mdvillanueva]

Yes I do have it on both filers.

Do I need to enter anything on host file?