network trace route from two nodes to cluster-mgmt-lif failed.

[netappmagic]

I have 6 nodes in the cluster, two nodes in the HA failed to cluster-mgmt-lif when I do "network traceroute -node prod-node1 -destination cluster_mgmt_ip". I am fine on all the other 4 nodes. 

Can somebody here please shed the light on this issue please?

[bmccullough]

It could be any number of issues, storage side or switch side.  Below assumes that cluster management IP and node management IPs are on same subnet / VLAN / Broadcast Domain.

1) verify that the LIFs configured and where you expect them to be; IP addresses, Subnets, Ports.

::> network interface show -role node-mgmt|cluster-mgmt

2) verify that the ports the LIFs are currently residing on ports that are in the correct broadcast domains.

::> network port show

3) identify the network switch ports and see if actual the switch ports connected have configurations match server side config.

::> system node run -node * -command cdpd show-neighbors

[TMACMD]

First do this (turn oncdp/lldp):

system node run -node * options cdpd.enable on
system node run -node * options lldp.enable on

Wait for 3 minutes for LLDP/CDP to talk.

Then do this (assuming all your MGMT and ClusMGMT are on e0M):

network device-discovery show -port e0M

This should show which switches and switch ports you e0M ports are connected to. Maybe you are connected to a wrong switch. Maybe the access VLAN is set incorrectly at the switch port.

Try testing to the gateway

network ping -lif <node-mgmt> -vserver <admin-svm> -destination <gateway_ip>

Repeat for each NODE-MGMT

Check your Broadcast-Domains

broadcast-domain show

You should see one (unless you renamed it, it will be called Default) that includes the e0M ports.

Make sure ***ALL THE PORTS*** in that broadcast domain are correct! When adding in new nodes, *ALL* ports get added to the Default broadcast-domain. You may need to remove ports that should not be there!

Check the LIFs failover ports:

network interface show -role node-mgmt|cluster-mgmt -failover

VERIFY the ports listed are correct. If not, the broadcast-domain is incorrect!

Finally, check failover-policy:

network interface show -role node-mgmt|cluster-mgmt -fields failover-polcy

All the NODE-MGMT LIFs should be local-only. The Cluster-MGMT could be system-defined. I would modify to broadcast-domain-wide:

network interface modify -lif <cluster_mgmt> -vserver <admin_svm> -failover-policy broadcast-domain-wide

Hope that helps. If you still have trouble, please think about posting some of the out from the commands above here.