Community

Subscribe
Highlighted

FAS3250 interconnect error

Hi!

We try to set up a FAS3250 cluster (7mode).

- System is two chassis configuration.

- Both filers have cluster licenses

- Every filer has partner-sysid (printenv partner-sysid)

- We use ports c0a and c0b for connection between filers (c0a -> c0a and c0b -> c0b)

- Each filer can see others filer disks (sysconfig-r output)

- cluster is enabled

The problem is that we have interconnect error. Cables are connected and there is no connection, we also try other cables and there is still no connection between filers.

Here are errors:

netappDEMO01*> cf status

netappDEMO02 is up, takeover disabled because of reason (interconnect error)

netappDEMO01 has disabled takeover by netappDEMO02 (interconnect error)

RDMA Interconnect is down (Link 0 down, Link 1 down).

netappDEMO01*>

netappDEMO01*> cf monitor

  current time: 22Feb2013 10:06:25

  UP 20:26:04, partner 'netappDEMO02', cluster monitor enabled

  RDMA Interconnect is down (Link 0 down, Link 1 down), takeover capability off-line (interconnect error)

  takeover by partner off-line (interconnect error)

  partner update TAKEOVER_DISABLED (22Feb2013 10:06:22)

        netappDEMO01*> ic status

        Link 0: down

        Link 1: down

        IC RDMA connection : down


                slot 0: Dual 10G Ethernet Controller T320E-SFP/KR

                Device Type:        CT-FE-3

                Version Number:     T3-SRAM1.1.0-BR1016-02-01-FW7.7.209-DR04

                Serial Number:      jb04050693

                c0a MAC Address:    00:a0:98:38:02:fe (auto-unknown-enabling)

                c0b MAC Address:    00:a0:98:38:02:ff (auto-unknown-enabling)

netappDEMO01*> ic config show

Interconnect Type: iWARP

  Local System ID: 20145XXXXXX

Partner System ID: 20145XXXXX

c0a: flags=0x174e867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 9000 PRIVATE

        inet 192.168.1.126 netmask 0xffffff00 broadcast 192.168.1.255

        ether 00:a0:98:38Fri Feb:03:fe (auto-unknown-enabling) flowcontrol full

c0b: flags=0x174e867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 9000 PRIVATE

        inet 192.168.2.177 netmask 0xffffff00 broadcast 192.168.2.255

        ether 00:a0:98:38:03:ff (auto-unknown-enabling) flowcontro 22 10:l full

In syslogs we get this errror: Failover is licensed but takeover of partner is disabled due to reason : interconnect error.

The question is how to enable or repair this connection between two filers? They are aware of each other but there is no link between them.

Does anybody know what is the problem here?

Thank you!

With regards,

Klemen

Re: FAS3250 interconnect error

Is it new installation or it worked before?

Re: FAS3250 interconnect error

New system. It was shipped as clustered ontap.

Also, we have done "set-defaults".

Klemen

Re: FAS3250 interconnect error

i have also try this, result is the same...

netappDEMO01*> ic link off 0

Fri FebFri Feb 22 10 22 10::37:50 CET [netappDEMO01:net.if.mgmt.sameSubnet:warning]: ifconfig: IP address '192.168.8.201' configured on dedicated management port 'e0M' is on the same subnet as IP address '192.168.8.200' configured on data port e0a. Management IP addresses must be o37:50 Cn dedicated management subnets.

ET [netappDEMO01:net.if.mgmt.sameSubnet:warning]: ifconfig: IP address '192.168.8.201' configured on dedicated management port 'e0M' is on the same subnet as IP address '192.1netappDEMO01*> 68.8.200' configured Fri Feb 22 10:37:50 Con dataET [netappDEMO01:netif.linkInfo:info]: Ethernet c0a: Link configured down.

port e0a. Management IP addresses must be on dedicated management subnets.

Fri Feb 22 10:37:50 CET [netappDEMO01:netif.linkInfo:info]: Ethernet c0a: Link configured down.

netappDEMO01*>

netappDEMO01*> ic link off 1

netappDEMO01*> Fri FebFri Feb 22 10:37:56 CET [neta 22 10Smiley TonguepDEMO01:netif.linkInfo:info]: Ethernet c0b: Link configured down.

37:56 CET [netappDEMO01:netif.linkInfo:info]: Ethernet c0b: Link configured down.

netappDEMO01*> ic link on 1

netappDEMO01*> Fri FebFri Feb 22 10:38:31 CET [ne 22 10:tappDEMO01:net.if.mgmt.sameSubnet:warning]: ifconfig: IP address '192.168.8.201' configured on dedicated management port 'e0M' is on the same subnet as IP address '192.168.8.200' configured on data port e0a. Management IP addresses must be on dedicated ma38:31 Cnagement subnets.

ET [netappDEMO01:net.if.mgmt.sameSubnet:warning]: ifconfig: IP address '192.168.8.201' configured on dedicated management port 'e0M' is on the same subnet as IP address '192.168.8.200' configured on data port e0a. Management IP addresses must be on dedicated management subnets.

ic link on 0

netappDEMO01*> ic link on 0

interface 0 is already up

netappDEMO01*>

netappDEMO01*>

netappDEMO01*> Fri FebFri Feb 22 10:38:39 CET [netappDEMO01:netif.linkInfo:info 22 10:]: Ethernet c0b: Link configured down.

38:39 CET [netappDEMO01:netif.linkInfo:info]: Ethernet c0b: Link configured down.

netappDEMO01*> Fri FebFri Feb 22 10:38:42 CET [netapp 22 10Smiley Very HappyEMO01:netif.linkInfo:info]: Ethernet c0a: Link configured down.

38:42 CET [netappDEMO01:netif.linkInfo:info]: Ethernet c0a: Link configured down.

netappDEMO01*>

netappDEMO01*> ic status

        Link 0: down

        Link 1: down

        IC RDMA connection : down

netappDEMO01*>

Re: FAS3250 interconnect error

Have you verified that correct ports are connected? No kidding, mistakes happen.

Have you verified you used the correct cables? How they are marked?

I would cf disable, removing CF license, reboot, verify systems came up as standalone and then re-enabling

Re: FAS3250 interconnect error

hi!

i have checked: c0a --> c0a and c0b --> c0b (still no lights)

i have done all as you said, result is the same:

netappDEMO01> cf status

netappDEMO02 is up, takeover disabled because of reason (interconnect error)

netappDEMO01 has disabled takeover by netappDEMO02 (interconnect error)

RDMA Interconnect is down (Link 0 down, Link 1 down).

netappDEMO01>

netappDEMO01> cf monitor

  current time: 22Feb2013 15:29:09

  UP 00:03:08, partner 'netappDEMO02', cluster monitor enabled

  RDMA Interconnect is down (Link 0 down, Link 1 down), takeover capability off-line (interconnect error)

  takeover by partner off-line (interconnect error)

  partner update TAKEOVER_DISABLED (22Feb2013 15:29:09)

netappDEMO01>

netappDEMO01> ic status

netappDEMO01*> ic status

        Link 0: down

        Link 1: down

        IC RDMA con

with regards,

Klemen

Re: FAS3250 interconnect error

Hi - First thing i would check is the IP addresses used for you interfaces. The console is display these error messages:

netappDEMO01*> Fri FebFri Feb 22 10:38:31 CET [ne 22 10:tappDEMO01:net.if.mgmt.sameSubnet:warning]: ifconfig: IP address '192.168.8.201' configured on dedicated management port 'e0M' is on the same subnet as IP address '192.168.8.200' configured on data port e0a. Management IP addresses must be on dedicated ma38:31 Cnagement subnets.

ET [netappDEMO01:net.if.mgmt.sameSubnet:warning]: ifconfig: IP address '192.168.8.201' configured on dedicated management port 'e0M' is on the same subnet as IP address '192.168.8.200' configured on data port e0a. Management IP addresses must be on dedicated management subnets.

ic link on 0

it appears as though you have IP Address set on the management interface (e0M) which is in the same subnet as one set physical interface e0a. This will cause problems as the appliance wont no where to route your IC traffic, or your IC traffic is routing out of your e0a interface!. The management nic's e0m, need there own subnet. ie 192.168.8.x and other interfaces or interface groups need a separate one, ie 192.168.9.x

Re: FAS3250 interconnect error

what is output of the ifconfig -a command if you can post it please?

Re: FAS3250 interconnect error

hi!

i have removed e0a port on both systems. The same problem.

output:

netappDEMO02> ifconfig -a

e0a: flags=0x2f0c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

        ether 00:a0:98:38:02:fc (auto-1000t-fd-up) flowcontrol full

e0b: flags=0x270c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

        ether 00:a0:98:38:02:fd (auto-unknown-down) flowcontrol full

e1a: flags=0x170c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,LRO> mtu 1500

        ether 90:e2:ba:2f:83:2c (auto-unknown-down) flowcontrol full

e1b: flags=0x170c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,LRO> mtu 1500

        ether 90:e2:ba:2f:83:2d (auto-unknown-down) flowcontrol full

e2a: flags=0x170c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,LRO> mtu 1500

        ether 90:e2:ba:2f:7b:3c (auto-unknown-down) flowcontrol full

e2b: flags=0x170c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,LRO> mtu 1500

        ether 90:e2:ba:2f:7b:3d (auto-unknown-down) flowcontrol full

c0a: flags=0x174e867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 9000 PRIVATE

        inet 192.168.1.70 netmask 0xffffff00 broadcast 192.168.1.255

        ether 00:a0:98:38:02:fe (auto-unknown-enabling) flowcontrol full

c0b: flags=0x174e867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 9000 PRIVATE

        inet 192.168.2.158 netmask 0xffffff00 broadcast 192.168.2.255

        ether 00:a0:98:38:02:ff (auto-unknown-enabling) flowcontrol full

e0M: flags=0x2b4c867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM,MGMT_PORT> mtu 1500

        inet 192.168.8.203 netmask 0xffffff00 broadcast 192.168.8.255 noddns

        partner inet 192.168.8.201 (not in use)

        ether 00:a0:98:38:03:00 (auto-100tx-fd-up) flowcontrol full

e0P: flags=0x2b4c867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM,ACP_PORT> mtu 1500 PRIVATE

        inet 192.168.1.176 netmask 0xfffffc00 broadcast 192.168.3.255 noddns

        ether 00:a0:98:38:03:01 (auto-100tx-fd-up) flowcontrol full

lo: flags=0x1b48049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 8160

        inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1

        ether 00:00:00:00:00:00 (RNIC Provider)

losk: flags=0x40a400c9<UP,LOOPBACK,RUNNING> mtu 9188

        inet 127.0.20.1 netmask 0xff000000 broadcast 127.0.20.1

netappDEMO02>

netappDEMO01*> ifconfig -a

e0a: flags=0x2f0c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

        ether 00:a0:98:38:03:fc (auto-1000t-fd-up) flowcontrol full

e0b: flags=0x270c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500

        ether 00:a0:98:38:03:fd (auto-unknown-down) flowcontrol full

e1a: flags=0x170c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,LRO> mtu 1500

        ether 90:e2:ba:2e:37:08 (auto-unknown-down) flowcontrol full

e1b: flags=0x170c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,LRO> mtu 1500

        ether 90:e2:ba:2e:37:09 (auto-unknown-down) flowcontrol full

e2a: flags=0x170c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,LRO> mtu 1500

        ether 90:e2:ba:2f:82:f0 (auto-unknown-down) flowcontrol full

e2b: flags=0x170c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,LRO> mtu 1500

        ether 90:e2:ba:2f:82:f1 (auto-unknown-down) flowcontrol full

c0a: flags=0x174e867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 9000 PRIVATE

        inet 192.168.1.230 netmask 0xffffff00 broadcast 192.168.1.255

        ether 00:a0:98:38:03:fe (auto-unknown-enabling) flowcontrol full

c0b: flags=0x174e867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 9000 PRIVATE

        inet 192.168.2.9 netmask 0xffffff00 broadcast 192.168.2.255

        ether 00:a0:98:38:03:ff (auto-unknown-enabling) flowcontrol full

e0M: flags=0x2b4c867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM,MGMT_PORT> mtu 1500

        inet 192.168.8.201 netmask 0xffffff00 broadcast 192.168.8.255 noddns

        partner inet 192.168.8.203 (not in use)

        ether 00:a0:98:38:04:00 (auto-100tx-fd-up) flowcontrol full

e0P: flags=0x2b4c867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM,ACP_PORT> mtu 1500 PRIVATE

        inet 192.168.1.167 netmask 0xfffffc00 broadcast 192.168.3.255 noddns

        ether 00:a0:98:38:04:01 (auto-100tx-fd-up) flowcontrol full

lo: flags=0x1b48049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 8160

        inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1

        ether 00:00:00:00:00:00 (RNIC Provider)

losk: flags=0x40a400c9<UP,LOOPBACK,RUNNING> mtu 9188

        inet 127.0.20.1 netmask 0xff000000 broadcast 127.0.20.1

netappDEMO01*>

netappDEMO01*> ic config show

Interconnect Type: iWARP

  Local System ID: 2014xxxxxx

Partner System ID: 2014xxxxx

c0a: flags=0x174e867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 9000 PRIVATE

        inet 192.168.1.230 netmask 0xffffff00 broadcast 192.168.1.255

        ether 00:a0:98:38:03:fe (auto-unknown-down) flowcontrol full

c0b: flags=0x174e867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 9000 PRIVATE

        inet 192.168.2.9 netmask 0xffffff00 broadcast 192.168.2.255

        ether 00:a0:98:38:03:ff (auto-unknown-down) flowcontrol full

netappDEMO01*>

Klemen

Re: FAS3250 interconnect error

You can send the output of the following command

sysconfig -a 0