ACP module connection-error after ONTAP 9.1 upgrade

[alessice]

Hi,

I have upgrade my FAS2554 from 8.3 to 9.1. All works fine but about every 5 minutes in the Events Log I found this debug messages:

acp.command.connect: Could not establish connection with 192.168.2.145.

acp.command.connect: Could not establish connection with 192.168.2.145.

I read that starting from ONTAP 9 ACP move from out-of-band to in-band for communications, but I have no idea how to switch or solve for my case.

Here my configurations:

cluster2::> storage shelf acp module show
Node Module Name State
----------------- ----------------- ----------------------
cluster2-node01 1.0.B connection-error
1.0.A connection-error
cluster2-node02 1.0.B connection-error
1.0.A connection-error
4 entries were displayed.

cluster2::> storage shelf acp show
Node Channel Connectivity
------------------ --------------- ----------------------
cluster2-node01 out-of-band full-connectivity
cluster2-node02 out-of-band full-connectivity
2 entries were displayed.

cluster2::> storage shelf acp show -instance

Node: cluster2-node01
Channel: out-of-band
Enable Status: true
Port: e0P
IP Address: 192.168.2.55
Subnet: 192.168.0.1
Netmask: 255.255.252.0
Connection Status: full-connectivity

Node: cluster2-node02
Channel: out-of-band
Enable Status: true
Port: e0P
IP Address: 192.168.1.253
Subnet: 192.168.0.1
Netmask: 255.255.252.0
Connection Status: full-connectivity
2 entries were displayed.

cluster2::> storage shelf acp module show -instance

Node: cluster2-node01
Module Name: 1.0.B
Adapter Name: 0a
MAC Address: 00:a0:98:c6:ea:91
IOM Type: iom6e
Shelf Serial Number: SHJHU1702000451
IP Address: 192.168.2.145
Protocol Version: 2.1.01.22
Firmware Version: 01.22
Assigner ID: 537067641
State: connection-error
Last Contact: 514

Node: cluster2-node01
Module Name: 1.0.A
Adapter Name: 0a
MAC Address: 00:a0:98:c6:ed:91
IOM Type: iom6e
Shelf Serial Number: SHJHU1702000451
IP Address: 192.168.1.145
Protocol Version: 2.1.01.22
Firmware Version: 01.22
Assigner ID: 537067641
State: active
Last Contact: 36

Node: cluster2-node02
Module Name: 1.0.B
Adapter Name: 0b
MAC Address: 00:a0:98:c6:ea:91
IOM Type: iom6e
Shelf Serial Number: SHJHU1702000451
IP Address: 192.168.2.145
Protocol Version: 2.1.01.22
Firmware Version: 01.22
Assigner ID: 537067641
State: connection-error
Last Contact: 514

Node: cluster2-node02
Module Name: 1.0.A
Adapter Name: 0b
MAC Address: 00:a0:98:c6:ed:91
IOM Type: iom6e
Shelf Serial Number: SHJHU1702000451
IP Address: 192.168.1.145
Protocol Version: 2.1.01.22
Firmware Version: 01.22
Assigner ID: 537067641
State: active
Last Contact: 36
4 entries were displayed.

cluster2::> system service-processor show
IP Firmware
Node Type Status Configured Version IP Address
------------- ---- ----------- ------------ --------- -------------------------
cluster2-node01
SP online true 2.4.1P2 192.168.7.210
cluster2-node02
SP online true 2.4.1P2 192.168.7.211
2 entries were displayed.

I read the KB https://kb.netapp.com/app/answers/answer_view/a_id/1029778 but I have no understand if for my setup is sufficient an

storage shelf acp configure -is-enabled true -channel in-band

or can cause problems if firmware or similar are not at the right version.

Thanks

[SpindleNinja]

you post the output of the following: 

storage shelf acp module show -fields module-name,protocol-version,firmware-version,shelf-serial-number,iom-type,state  

[SpindleNinja]

you post the output of the following: 

storage shelf acp module show -fields module-name,protocol-version,firmware-version,shelf-serial-number,iom-type,state

[alessice]

Thanks, here te output:

cluster2::> storage shelf acp module show -fields module-name,protocol-version,firmware-version,shelf-serial-number,iom-type,state
node            module-name protocol-version firmware-version shelf-serial-number iom-type state            
--------------- ----------- ---------------- ---------------- ------------------- -------- ---------------- 
cluster2-node01 1.0.B       2.1.01.22        01.22            SHJHU1702000451     iom6e    connection-error 
cluster2-node01 1.0.A       2.1.01.22        01.22            SHJHU1702000451     iom6e    connection-error 
cluster2-node02 1.0.B       2.1.01.22        01.22            SHJHU1702000451     iom6e    connection-error 
cluster2-node02 1.0.A       2.1.01.22        01.22            SHJHU1702000451     iom6e    connection-error 
4 entries were displayed.

[SpindleNinja]

based on the firmware levels,  doesn't look like you can move to inband.     

Just to confirm, Is this just internal disks or is there a shelf attatched? 

have you opened a support case with netapp yet? 

[alessice]

This is a single FAS2554 in HA with 24 internal disk, no external shelf.

From ActiveIQ all firmware are marked as update. For now I haven't open a case.

[SpindleNinja]

have you tried a manual update of the lastest firmwares?  

https://mysupport.netapp.com/NOW/cgi-bin/shelffwdnld.cgi/download/tools/diskshelf/bin/all_shelf_fw 

[GidonMarcus]

Hi All

i can't find a reference for it now. but i had similar issue with two  FAS2240 also going from 8.3.2 to 9.1 where ACP gone to no-connection. the solution was to turn ACP off and on again.

i think there was a bug open for it, but i can't find it...

Gidi

[AlexDawson · Accepted solution]

BURT ID 1121636

After the Data ONTAP software upgrade from 8.3.x to 9.1.Px, ONTAP might report
 that the Alternate Control Path (ACP) processor on IOM6/IOM6E is no longer
 active, and it is reported as a "connection-error."
 You can verify this issue by using the "storage shelf acp module show" command
 to display the state of all ACP processors on IOM modules.
 cluster1::> storage shelf acp module show
 Node               Module Name        State
 -----------------  -----------------  ----------------------
 cluster1-01        1.10.B             active
                     1.11.B             active
                     1.11.A             active
                     1.10.A             active
                     1.0.A              connection-error
                     1.0.B              active
 cluster1-02        1.10.B             active
                     1.11.B             active
                     1.11.A             active
                     1.10.A             active
                     1.0.A              connection-error
                     1.0.B              active

 1. Disable ACP
 cluster1::> storage shelf acp configure -is-enabled false
 2. Wait for 5 minutes to allow the ACP to reset completely.
 3. Enable out-of-band ACP again and use the default IP address. ACP provides the
 IP address.
 cluster1::> storage shelf acp configure -is-enabled true -channel out-of-band

[alessice]

Thanks @AlexDawson 

the bug report exactly my issue, I will try to disable/enable ACP and let you know if it works and solve my issue.

[AlexDawson]

Sounds good!

[alessice]

I'm happy, now ACP works fine:

cluster2::> storage shelf acp module show          
Node               Module Name        State
-----------------  -----------------  ----------------------
cluster2-node01    1.0.B              active
                   1.0.A              active
cluster2-node02    1.0.B              active
                   1.0.A              active
4 entries were displayed.