New CDOT System will not recognize default gateway for node managemen ports

[NEWNETAPP987]

Hi,

I installed a new 8.2.1 CDOT system.  I upgraded it from 8.2.1 7 mode.  At first all the nodes worked on the network properly.  Then I rebooted node 1 and I could not ping the node management ports from remote site anymore.  I got on the SP and everything looked fine.

I rebooted again and got these messages

route: writing to routing socket: Network is unreachable
add net 0.0.0.0: gateway 10.26.6.1: Network is unreachable
Apr 25 16:35:23 Battery charge capacity: 3584 mA*hr. Power outage protection flash de-staging cycles: 54

It won't add the default gateway on boot up.  I can't ping the eom management ports on each node.  But, this, is the strange part, I can ping the cluster management IP on the same management network.  

Does anyone have any idea why it won't add the default gateway when it boots?

Thanks

[shatfield]

Physical interfaces don't have IP addresses in CDOT. The node management logical interface may have failed over to a different port.

Try a net int show, find the interface node management is sitting on, and see if that port is on the right network.  If its correct, check the routing groups.

The default failover groups caused all sorts of havoc prior to FS.0

And that message at boot is usually a red herring.

[JGPSHNTAP]

I agree - Show us net int show

Also, i would recommend 8.3

also, type route show

and check your broadcast domains

[NEWNETAPP987]

blf_cdr_smnap_cluster1::> net int show
(network interface show)
Logical Status Network Current Current Is
Vserver Interface Admin/Oper Address/Mask Node Port Home
----------- ---------- ---------- ------------------ ------------- ------- ----
blf_cdr_smnap_cluster1
cluster_mgmt up/up 10.28.8.60/24 blf_cdr_smnap_cluster1-01
e0i true
blf_cdr_smnap_cluster1-01
clus1 up/up 169.254.210.140/16 blf_cdr_smnap_cluster1-01
e0a true
clus2 up/up 169.254.57.205/16 blf_cdr_smnap_cluster1-01
e0c true
mgmt1 up/up 10.28.8.61/24 blf_cdr_smnap_cluster1-01
e0M true
blf_cdr_smnap_cluster1-02
clus1 up/up 169.254.231.186/16 blf_cdr_smnap_cluster1-02
e0a true
clus2 up/up 169.254.94.37/16 blf_cdr_smnap_cluster1-02
e0c true
mgmt1 up/up 10.28.8.62/24 blf_cdr_smnap_cluster1-02
e0M true
7 entries were displayed.

blf_cdr_smnap_cluster1::>

blf_cdr_smnap_cluster1::> network routing-groups route show
Routing
Vserver Group Destination Gateway Metric
--------- --------- --------------- --------------- ------
blf_cdr_smnap_cluster1
c10.28.8.0/24
0.0.0.0/0 10.28.8.1 20
blf_cdr_smnap_cluster1-01
n10.28.8.0/24
0.0.0.0/0 10.28.8.1 10
blf_cdr_smnap_cluster1-02
n10.28.8.0/24
0.0.0.0/0 10.28.8.1 10
3 entries were displayed.

I think I founhd the problem but donj't know how to fix it.  10.28.8.60 does work.   It has the mac address of the 10.28.8.1 gatwway show for the eoi interface.

EOM does not have a mac address show for the 10.28.8.1 gateway.  I think I need to have the network group change something on the switch but I don't know what to tell them to change

blf_cdr_smnap_cluster1::> node run -node blf_cdr_smnap_cluster1-01 route -gsn

Routing tables

Routing group: __default_grp

Internet:
Destination Gateway Flags Refs Use Interface
127.0.0.1 127.0.0.1 UH 0 0 lo
127.0.10.1 127.0.20.1 UHS 5 286 losk

Routing group: blf_cdr_smnap_cluster1-01_c169.254.0.0/16

Internet:
Destination Gateway Flags Refs Use Interface
169.254 link#1 UC 0 0 e0c =>
169.254 link#3 UC 0 0 e0a
169.254.57.205 0:a0:98:5e:38:98 UHL 0 2580 lo
169.254.94.37 0:a0:98:5e:19:b2 UHL 2 72196 e0c =>
169.254.94.37 0:a0:98:5e:19:b0 UHL 18 35650 e0a
169.254.210.140 0:a0:98:5e:38:96 UHL 0 2792 lo
169.254.231.186 0:a0:98:5e:19:b2 UHL 24 15225 e0c =>
169.254.231.186 0:a0:98:5e:19:b0 UHL 0 78162 e0a

Routing group: blf_cdr_smnap_cluster1-01_n10.28.8.0/24

Internet:
Destination Gateway Flags Refs Use Interface
default 10.28.8.1 UGS 0 0 e0M
10.28.8/24 link#6 UC 0 0 e0M
10.28.8.1 link#6 UHL 1 0 e0M

Routing group: blf_cdr_smnap_cluster1_c10.28.8.0/24

Internet:
Destination Gateway Flags Refs Use Interface
default 10.28.8.1 UGS 0 2806 e0i
10.28.8/24 link#7 UC 0 0 e0i
10.28.8.1 0:23:4:55:4c:dd UHL 1 0 e0i

blf_cdr_smnap_cluster1::>

[aborzenkov]

If you want cluster_mgmt LIF be on e0i interface, this interface should be connected to the 10.28.8.60/24 network (broadcast domain, VLAN, however it is called by your networking equipment). But my guess is that you simply need to create failover group for cluster_mgmt LIF that consists of e0M ports on all nodes.

Anyway right now you just need to move cluster_mgmt LIF to e0M port to get access again.