topic Re: Interconnect issues - FAS3270 in ONTAP Discussions

Interconnect issues - FAS3270

John_Blackwell — Wed, 04 Jun 2025 12:22:14 GMT

Hello All,

Trying to help a customer out convert a pair of controllers that were in a metro cluster into a HA pair, FAS3270 with IOXM.

We are using new disks so wiped out all disk ownership and installed the new disks, completed a option 4 and installed ONTAP on both, then 1 of the 2 nodes died and would not boot so replaced it with a new node, completed the node replacement process and got the node back up, enabled CF from the first time and it wouldnt syncronise, did a CF monitor and both links showed as down, this is where it gets odd.

Currently running 1 cable between c0a, looking for a 2nd cable now, we do see a green light.

1st node -

dersan03*> ic status
        Link 0: down
        Link 1: down
        IC RDMA connection : down

dersan03*> ic config show
Interconnect Type: iWARP
Local System ID: 1575077239
Partner System ID: 1574064502
c0a: flags=0x1f4e867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 9000 PRIVATE
        inet 192.168.1.174 netmask 0xffffff00 broadcast 192.168.1.255
        ether 00:a0:98:1d:83:c4 (auto-10g_twinax-fd-up) flowcontrol full
c0b: flags=0x174e867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 9000 PRIVATE
        inet 192.168.2.246 netmask 0xffffff00 broadcast 192.168.2.255
        ether 00:a0:98:1d:83:c5 (auto-unknown-enabling) flowcontrol full

Node2 -

dersan04*> ic status
        Link 0: down
        Link 1: down
        cfo_rv connection state :                       NOT CONNECTED
        cfo_rv nic used :                               0
        cfo_rv connection role :                        source_rv
        cfo_rv1 connection state :                      NOT CONNECTED
        cfo_rv1 nic used :                              1
        cfo_rv1 connection role :                       none
        cfo_rv2 connection state :                      NOT CONNECTED
        cfo_rv2 nic used :                              1
        cfo_rv2 connection role :                       not applicable

dersan04*> ic config show

Error: Command not supported on this platform.

I get the feeling there is a misconfig on node02 but ha-config for both chassis and controller is set to ha, psm-cc-config is unset and the partner system ID's are correct.

What am I missing, crying in pain over here, both nodes running 8.2.4P6, confirmed part numbers between the 2 controllers are the same but the revision number is different.

Thanks in advance.

John

Re: Interconnect issues - FAS3270

TMACMD — Thu, 25 Jul 2019 19:55:39 GMT

For kicks...

when you re-initialized the system, at the LOADER prompt, did you "set-defaults"?

Maybe there is a ENV VAR set that is messing things up?

From MAINT mode, you did verify that "ha-config show" on both nodes looks correct?

For controller and chassis (ha) ?

Re: Interconnect issues - FAS3270

John_Blackwell — Thu, 25 Jul 2019 20:47:48 GMT

I didnt, that is a good thing to try though, once I have done that do I have to reset the partner ID?

Yes, ha looks set correct to me

*> ha-config show
Chassis HA configuration: ha
Controller HA configuration: ha

Attached are the printenv from each node, I dont see anything suspecious.

To me it looks like node2 is trying to use the internal chassis HA connections opposed to c0a/c0b but I dont know how to get it to change

Re: Interconnect issues - FAS3270

TMACMD — Thu, 25 Jul 2019 21:00:30 GMT

If possible, start over.

Boot both into maintenance mode.

Make sure both are ha for the the chassis and controller.

If not, use the ha-config command to set both sides to ha.

Then at the loader prompt on both sides:

set-defaults

Follow the directions in this link: https://kb.netapp.com/app/answers/answer_view/a_id/1030427

Hopefully that helps

Re: Interconnect issues - FAS3270

John_Blackwell — Thu, 25 Jul 2019 21:06:59 GMT

Thanks for this, yes I will give it a go tomorrow and let you know how I get on.

John

Re: Interconnect issues - FAS3270

John_Blackwell — Mon, 29 Jul 2019 08:18:20 GMT

Turns out there was an FCVI card in slot 1 which was taking precident over the onboard cards, removed the card and all started working.

Cheers for your help