Solved: Re: Unable to update ACP on shelf running inband

Stormont · ‎2020-05-05

We have a DS2246 connected to an FAS8200 operating in in-band mode and the cluster thinks needs to be updated, but all attempts to do so never update it. (although Protocol Version and Firmware versions of all zeros seems weird). What other options do we have to update this?

Oriole::storage firmware shelf*> storage shelf acp module show -instance -node Oriole-03

Node: Oriole-03

Module Name: 3.0.B

Adapter Name: 2d

MAC Address: 00:a0:98:49:5e:a4

IOM Type: iom6

Shelf Serial Number: 6000832241

Protocol Version: 0.0.00.00

Firmware Version: 00.00

State: firmware-update-required

Last Contact: 82491

Power Cycle Count: 0

Power Off Count: 0

Power On Count: 0

Expander Reset Count: 0

Expander Power Cycle Count: 0

Node: Oriole-03

Module Name: 3.0.A

Adapter Name: 2d

MAC Address: 00:a0:98:49:5e:cc

IOM Type: iom6

Shelf Serial Number: 6000832241

Protocol Version: 0.0.00.00

Firmware Version: 00.00

State: firmware-update-required

Last Contact: 82491

Power Cycle Count: 0

Power Off Count: 0

Power On Count: 0

Expander Reset Count: 0

Expander Power Cycle Count: 0

set -priv adv

storage firmware download -node Oriole-03 -package-url http://172.22.17.34/NetApp/all_shelf_fw.zip

Oriole::*> system node run -node Oriole-03 -command storage download acp

Downloading ACP firmware can take several minutes and will not disrupt client access during that time.

However, normal ACP recovery capabilities will not be available while the firmware upgrade is in progress.

Are you sure you want to continue with ACP processor firmware update? y [storage download acp] Firmware will not be downloaded to module 0a.00.B as it does not have ACPP.

[storage download acp] Firmware will not be downloaded to module 0a.00.A as it does not have ACPP.

[storage download acp] Firmware will not be downloaded to module 0c.04.A as it does not have ACPP.

[storage download acp] Firmware will not be downloaded to module 0c.04.B as it does not have ACPP.

[storage download acp] Firmware will not be downloaded to module 0a.01.A as it does not have ACPP.

[storage download acp] Firmware will not be downloaded to module 0a.01.B as it does not have ACPP.

[storage download acp] Firmware will not be downloaded to module 2d.00.B as ACPP state is not active, Current ACPP state is 0x9.

[storage download acp] Firmware will not be downloaded to module 2d.00.A as ACPP state is not active, Current ACPP state is 0x9.

[storage download acp] Firmware will not be downloaded to module 0c.05.B as it does not have ACPP.

[storage download acp] Firmware will not be downloaded to module 0c.05.A as it does not have ACPP.

Waiting for a while (even days) then shows this:

storage shelf acp module show -fields firmware-version,module-name,node Oriole::*> storage shelf acp module show -fields firmware-version,module-name,node

node module-name firmware-version

--------- ----------- ----------------

Oriole-01 1.0.B 02.12

Oriole-01 1.0.A 02.12

Oriole-02 1.0.B 02.12

Oriole-02 1.0.A 02.12

Oriole-03 3.0.B 00.00

Oriole-03 3.0.A 00.00

Oriole-03 2.1.B -

Oriole-03 2.0.A -

Oriole-03 2.1.A -

Oriole-03 2.0.B -

Oriole-03 5.5.B -

Oriole-03 5.5.A -

Oriole-03 5.4.A -

Oriole-03 5.4.B -

Oriole-04 3.0.B 00.00

Oriole-04 3.0.A 00.00

Oriole-04 2.1.B -

Oriole-04 2.0.A -

Oriole-04 2.1.A -

Oriole-04 2.0.B -

Oriole-04 5.5.B -

Oriole-04 5.5.A -

Oriole-04 5.4.A -

Oriole-04 5.4.B -

24 entries were displayed.

Stormont · ‎2020-05-18

This bug was supposedly created to resolve the issue we are seeing:

https://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1050364

However, there are apparently times when it doesn't get out of the 0x9 state, so bug https://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1310583 was created to fix that.

One thing I did find is that all three shelves that are still running 0191 are connected to the same adapter (one DS2246 on one loop connected to 2c and 2d and two DS4246 shelves connected to 2a and 2b). Since this is on an FAS8200 that doesn't support OOB and I can't imagine that both adapters in both nodes need to be physically reseated, we must have the bug where they won't get out of the 0x9 state.

View solution in original post

SpindleNinja · ‎2020-05-05

what's the output of

storage shelf acp show -instance

Stormont · ‎2020-05-05

Oriole::> storage shelf acp show -instance

Node: Oriole-01
Channel: out-of-band
Enable Status: true
Port: e0P
IP Address: 192.168.3.62
Subnet: 192.168.0.1
Netmask: 255.255.252.0
Connection Status: full-connectivity

Node: Oriole-02
Channel: out-of-band
Enable Status: true
Port: e0P
IP Address: 192.168.3.12
Subnet: 192.168.0.1
Netmask: 255.255.252.0
Connection Status: full-connectivity

Node: Oriole-03
Channel: in-band
Enable Status: true
Connection Status: active

Node: Oriole-04
Channel: in-band
Enable Status: true
Connection Status: active
4 entries were displayed.

________________________________

Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error.

SpindleNinja · ‎2020-05-05

Thanks, so yeah. That is a bit odd... I would try a IOM re-seat, but open a ticket with support as a CYA.

Stormont · ‎2020-05-05

They suggested I switch to out of band mode and then try the update again. I'm not sure if the shelf was ever cabled for OOB mode, so I'm not sure what that might do.

________________________________

Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error.

SpindleNinja · ‎2020-05-05

Was it ever headswapped from something with e0P?

Go for the toggle though, there shouldn't be any ill effects.

Stormont · ‎2020-05-05

No, it was not (we purchased it new and then added it to the cluster)

________________________________

Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error.

SpindleNinja · ‎2020-05-06

Have they ever had an upgrade before?

Also, if you just run "storage shelf acp module show -fields firmware-version" does it show any firmware versions?

And see if it shows up here too:

node run -node node-3 -command sysconfig -a

Stormont · ‎2020-05-06

I’m a bit confused by this one. For a different shelf (a DS212), “storage shelf acp module show -fields firmware-version” shows the following:

node firmware-version

--------- ----------------

Oriole-01 02.12

But “sysconfig –a” shows a different version

“Shelf 1: DS212-12 Firmware rev. IOM12 A: 0250 IOM12 B: 0250”

Shouldn’t both commands show some form of “0250” instead of “02.12” for the one and “0250” for the other?

For the shelf showing all “00”, “storage shelf acp module show -fields firmware-version” shows the following:

node firmware-version

--------- ----------------

Oriole-03 00.00

While “sysconfig –a” shows:

Shelf 6000832241: DS2246 Firmware rev. IOM6 A: 0191 IOM6 B: 0191

aborzenkov · ‎2020-05-06

@Stormont wrote:

Shouldn’t both commands show some form of “0250” instead of “02.12” for the one and “0250” for the other?

There are ACP firmware and IOM firmware. They are different things. In-band ACP requires ACP firmware at least 2.09. 2.12 is the latest ACP firmware version for IOM6 as of now.

Stormont · ‎2020-05-07

I was told that the shelf most likely won't go to OOB mode if it is not cabled that way and that a reset of each ACP module is the next thing to try. Based on the output below, are these the correct commands?

Oriole-03*> acpadmin ibacp_list_all

Number of ACP modules 10

MAC SES Last Contact Protocol Assigner Shelf Current Inband IOM shelf

Address bit (seconds ago) Version ACPA ID S/N State ID Type flag

---------------------------------------------------------------------------------------------------------------

00:a0:98:49:5e:a4 00 245613 0.0.00.00 0 6000832241 0x9 2d.00.B IOM6 0x20

00:a0:98:49:5e:cc 00 245613 0.0.00.00 0 6000832241 0x9 2d.00.A IOM6 0x20

Oriole-04*> acpadmin ibacp_list_all

Number of ACP modules 10

MAC SES Last Contact Protocol Assigner Shelf Current Inband IOM shelf

Address bit (seconds ago) Version ACPA ID S/N State ID Type flag

---------------------------------------------------------------------------------------------------------------

00:a0:98:49:5e:a4 00 245240 0.0.00.00 0 6000832241 0x9 2c.00.B IOM6 0x20

00:a0:98:49:5e:cc 00 245240 0.0.00.00 0 6000832241 0x9 2c.00.A IOM6 0x20

Based on that, would we be looking at running these commands:

node run -node Oriole-03

priv set diag

For a given Module A: sasadmin expander_cli 2d.00.A “acpmgr_tp 3 0”

For a given Module B: sasadmin expander_cli 2d.00.B “acpmgr_tp 3 1”

node run -node Oriole-04

For a given Module A: sasadmin expander_cli 2c.00.A “acpmgr_tp 3 0”

For a given Module B: sasadmin expander_cli 2c.00.B “acpmgr_tp 3 1”

amans · ‎2020-05-17

So looks like, you have been suggested to reset the SES bit. I have hardly seen this work, but certainly give it a try.

In addition, 0212 ACP FW is bundled with ONTAP 9.7 and above. I wonder, why the ACP did not upgrade for you, this would require further investigation from support.

Stormont · ‎2020-05-18

This bug was supposedly created to resolve the issue we are seeing:

https://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1050364

However, there are apparently times when it doesn't get out of the 0x9 state, so bug https://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1310583 was created to fix that.

One thing I did find is that all three shelves that are still running 0191 are connected to the same adapter (one DS2246 on one loop connected to 2c and 2d and two DS4246 shelves connected to 2a and 2b). Since this is on an FAS8200 that doesn't support OOB and I can't imagine that both adapters in both nodes need to be physically reseated, we must have the bug where they won't get out of the 0x9 state.

KristianT · ‎2021-02-01

Hi Gents,

I encountered this issue on FAS8040 with DS4246 shelves running OnTap 9.7. Seem this is known bug for in-band configuration. Work-around is to configure out-of-band acp, it should trigger automatic update if you have new packages. Give it few minutes and then switch in to in-band.

Have a nice day,

Kristian