How to REIP a Netapp's primary interface without any downtime?

[JLundfelt]

Hello.,

As part of an organization change we need to change the IP address of everything in our environment, to a new corporate network range. I was wondering if our plan to REIP the primary routable interface on each of our NetApp’s was solid. Here is the typical controller-

Type	Controller	Interface	Network Address	Network Mask	Type	Status	In/Sec	Out/Sec	Protocols
Active/Active Controller	irv-gdc-san1a.corp.myco.com	vif1-96	10.1.96.100	255.255.255.0	VIF	Up	694780	1745400	CIFS
Active/Active Controller	irv-gdc-san1a.corp.myco.com	losk	127.0.20.1	255.0.0.0	Unknown	Up	0	1	LOOPBACK
Active/Active Controller	irv-gdc-san1a.corp.myco.com	vif2-230	10.1.230.50	255.255.255.0	VIF	Up	52099700	23208100	iSCSI
Active/Active Controller	irv-gdc-san1a.corp.myco.com	vif2-231	10.1.231.50	255.255.255.0	VIF	Up	2099880	1475370	NFS
Active/Active Controller	irv-gdc-san1a.corp.myco.com	vif2-232	10.1.232.50	255.255.255.0	VIF	Up	75679	6013	NFS

Routing tables

Internet:

Destination      Gateway            Flags     Refs     Use  Interface

default          10.1.96.1          UGS        33 3318967135  vif1-96

10.1.96/24       link#14            UC          0        0  vif1-96

10.1.96.1        c4:7d:4f:b1:35:40  UHL         1        0  vif1-96

irv-gdc-san1a-vi 2:a0:98:12:e8:ac   UHL         0      443  lo

10.1.96.255      ff:ff:ff:ff:ff:ff  UHL         0   317283  vif1-96

The A record points to 10.1.96.100, and the tentative plan was to -

• Add new VLAN to existing VIF1 interface. [vlan create vif1 26]
• Add new IP Address (10.228.26.100) to new VLAN [Ifconfig Vif1-26 10.228.26.100 netmask 255.255.255.0]
• Verify connectivity by testing CIFS / RPC to new IP address
• Change A record(s)
• Confirm network traffic [netstat -I vif1-96 -w 1 –x]
• Change default route
• Confirm network traffic [netstat -I vif1-26 -w 1 –x]
• Remove existing vif1-96 VLAN

Attached are our hosts, and RC file

[billshaffer]

This looks like it will work - as far as changing the controller IP.  But I think you're still going to run into CIFS/NFS disconnects.  I'm pretty sure that NFS filehandles won't move to the new IP, and you'd have to do a remount on the clients.  CIFS will certainly complain when the existing interface is taken offline.

You will also need to consider your NFS exports - if the network changes are happening on the client side as well, then the entire /etc/exports file becomes deprecated as soon as you make the switch.

I'm not completely clear on how CIFS registers with AD, or whether the same machine on a different IP will confuse AD or not, but that is another thing to consider.

You could test this - create your new vlan, and create another A record to point to the existing one.  Mount a couple shares on a test box pointing to that new A record, change the A record, as see what happens....  You'd have to do something with the routing on the new vlan - a static host route to your test box, maybe?

Bill

[JLundfelt]

Yeah. I am scheduled to test tomorrow. The good news is only CIFS, and MGMT (DFM / SSH / SNMP / ETC) run over the primary interface. iSCSI and NFS have their own interfaces as I was illustrating above-

Controller	Interface	Network Address	Network Mask	Protocols
irv-gdc-san1a.corp.myco.com	vif1-96	10.1.96.100	255.255.255.0	CIFS
irv-gdc-san1a.corp.myco.com	losk	127.0.20.1	255.0.0.0	LOOPBACK
irv-gdc-san1a.corp.myco.com	vif2-230	10.1.230.50	255.255.255.0	iSCSI
irv-gdc-san1a.corp.myco.com	vif2-231	10.1.231.50	255.255.255.0	NFS
irv-gdc-san1a.corp.myco.com	vif2-232	10.1.232.50	255.255.255.0	NFS

The first interface is the primary, and the one we will be changing. I am not worried how thie will impact AD / Kerberos so much, as I am how I can have two routes simultaneously while I wait for the DNS record to update, and workstations and servers to start using the new interface. Wouldn't I need to change my plan to-

• Add new VLAN to existing VIF1 interface. [vlan create vif1 26]
• Add new IP Address (10.228.26.100) to new VLAN [Ifconfig Vif1-26 10.228.26.100 netmask 255.255.255.0]
• Verify connectivity by testing CIFS / RPC to new IP address
• Change A record(s)
• Confirm network traffic [netstat -I vif1-96 -w 1 –x]
• Change default route  add new route with lower metric [route add host 10.228.26.100 10.228.26.1 10]
• Confirm network traffic [netstat -I vif1-26 -w 1 –x]
• Remove existing vif1-96 VLAN

[billshaffer · Accepted solution]

For the test, at least, you don't have to worry much about the multiple routes - a host route will take precedent over a default route.  The man page for na_route says "metric" has to be 0 if the destination is on a directly attached network; not sure if it would let you put a non-zero metric for a destination on the same network.

For the actual cutover, I do see your dilemma - if the client addresses aren't changing, how do you satisfy incoming traffic from both vlans?  Like I said though, CIFS is going to be a problem anyways because it is connection oriented.  I don't see any way around refreshing the CIFS share on the clients to get them to recognize the new IP.  And in addition to having to wait for DNS to propagate, you  need to worry about the DNS cache on the windows boxes.  I've seen plenty of occasions when I had to flush the cache in order to get windows to see a new IP.

Bill

[JLundfelt]

So we ended up having some downtime. It was a good thing we were connected through the RLM, We are going to either try and source the RC file next time, or paste the following commands to expedite-

•	Change default route
o	arp -a
o	route -f
o	route delete default
o	route add default 10.228.26.1 1
o	route add net 10.228.26/24 10.228.26.1 1

Will let everyone know what the next outage window duration is.

Thanks,

Jon

[billshaffer]

What's the purpose of the net route?  If an interface exists on that network, it should by default route that subnet through the interface?

Bill

[aborzenkov]

I am not that sure. Routes are tied to interfaces, so it may happen that when you down interface with current default route, you are left without any default route at all. Need to test.

[JLundfelt]

I just confirmed it's 'optional', I needed it when i had problems recreating a LIF

[JLundfelt]

So for anyone else that needs to REIP the primary interface on a NetApp, here was our final full list of commands-

vlan create vif1 426

ifconfig vif1-426 10.228.26.101 netmask 255.255.255.0 -wins partner vif1-96

-CHANGE DNS RECORD- Wait for replication (flush client DNS cache Get-nacifssession  |%{psexec \\$($_.hostname) ipconfig /flushdns}

arp -a

route -f

route delete default

route add default 10.228.26.1 1

ifconfig vif1-96 down

vlan delete vif1 96

options cf.hw_assist.partner.address 10.228.26.100

dfm host set irv-gdc-san1b.prod.myco.com hostPrimaryAddress=10.228.26.101