ONTAP Discussions

Unable to update ACP on shelf running inband

Stormont
7,434 Views

We have a DS2246 connected to an FAS8200 operating in in-band mode and the cluster thinks needs to be updated, but all attempts to do so never update it. (although Protocol Version and Firmware versions of all zeros seems weird).  What other options do we have to update this?

 

 

Oriole::storage firmware shelf*> storage shelf acp module show -instance -node Oriole-03

 

                        Node: Oriole-03

                 Module Name: 3.0.B

                Adapter Name: 2d

                 MAC Address: 00:a0:98:49:5e:a4

                    IOM Type: iom6

         Shelf Serial Number: 6000832241

            Protocol Version: 0.0.00.00

            Firmware Version: 00.00

                       State: firmware-update-required

                Last Contact: 82491

           Power Cycle Count: 0

             Power Off Count: 0

              Power On Count: 0

        Expander Reset Count: 0

  Expander Power Cycle Count: 0

 

                        Node: Oriole-03

                 Module Name: 3.0.A

                Adapter Name: 2d

                 MAC Address: 00:a0:98:49:5e:cc

                    IOM Type: iom6

         Shelf Serial Number: 6000832241

            Protocol Version: 0.0.00.00

            Firmware Version: 00.00

                       State: firmware-update-required

               Last Contact: 82491

           Power Cycle Count: 0

             Power Off Count: 0

              Power On Count: 0

        Expander Reset Count: 0

  Expander Power Cycle Count: 0

 

set -priv adv

storage firmware download -node Oriole-03 -package-url http://172.22.17.34/NetApp/all_shelf_fw.zip

Oriole::*> system node run -node Oriole-03 -command storage download acp

 

Downloading ACP firmware can take several minutes and will not disrupt client access during that time.

However, normal ACP recovery capabilities will not be available while the firmware upgrade is in progress.

 

Are you sure you want to continue with ACP processor firmware update? y [storage download acp] Firmware will not be downloaded to module 0a.00.B as it does not have ACPP.

[storage download acp] Firmware will not be downloaded to module 0a.00.A as it does not have ACPP.

[storage download acp] Firmware will not be downloaded to module 0c.04.A as it does not have ACPP.

[storage download acp] Firmware will not be downloaded to module 0c.04.B as it does not have ACPP.

[storage download acp] Firmware will not be downloaded to module 0a.01.A as it does not have ACPP.

[storage download acp] Firmware will not be downloaded to module 0a.01.B as it does not have ACPP.

[storage download acp] Firmware will not be downloaded to module 2d.00.B as ACPP state is not active, Current ACPP state is 0x9.

[storage download acp] Firmware will not be downloaded to module 2d.00.A as ACPP state is not active, Current ACPP state is 0x9.

[storage download acp] Firmware will not be downloaded to module 0c.05.B as it does not have ACPP.

[storage download acp] Firmware will not be downloaded to module 0c.05.A as it does not have ACPP.

 

 

Waiting for a while (even days) then shows this:

 

storage shelf acp module show -fields firmware-version,module-name,node Oriole::*> storage shelf acp module show -fields firmware-version,module-name,node

node      module-name firmware-version

--------- ----------- ----------------

Oriole-01 1.0.B       02.12

Oriole-01 1.0.A       02.12

Oriole-02 1.0.B       02.12

Oriole-02 1.0.A       02.12

Oriole-03 3.0.B       00.00

Oriole-03 3.0.A       00.00

Oriole-03 2.1.B       -

Oriole-03 2.0.A       -

Oriole-03 2.1.A       -

Oriole-03 2.0.B       -

Oriole-03 5.5.B       -

Oriole-03 5.5.A       -

Oriole-03 5.4.A       -

Oriole-03 5.4.B       -

Oriole-04 3.0.B       00.00

Oriole-04 3.0.A       00.00

Oriole-04 2.1.B       -

Oriole-04 2.0.A       -

Oriole-04 2.1.A       -

Oriole-04 2.0.B       -

Oriole-04 5.5.B       -

Oriole-04 5.5.A       -

Oriole-04 5.4.A       -

Oriole-04 5.4.B       -

24 entries were displayed.

1 ACCEPTED SOLUTION

Stormont
6,830 Views

This bug was supposedly created to resolve the issue we are seeing:

https://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1050364

 

However, there are apparently times when it doesn't get out of the 0x9 state, so bug https://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1310583 was created to fix that.

 

One thing I did find is that all three shelves that are still running 0191 are connected to the same adapter (one DS2246 on one loop connected to 2c and 2d and two DS4246 shelves connected to 2a and 2b).  Since this is on an FAS8200 that doesn't support OOB and I can't imagine that both adapters in both nodes need to be physically reseated, we must have the bug where they won't get out of the 0x9 state.

 

View solution in original post

13 REPLIES 13

SpindleNinja
7,426 Views

what's the output of 

 

storage shelf acp show -instance

Stormont
7,425 Views
Oriole::> storage shelf acp show -instance

Node: Oriole-01
Channel: out-of-band
Enable Status: true
Port: e0P
IP Address: 192.168.3.62
Subnet: 192.168.0.1
Netmask: 255.255.252.0
Connection Status: full-connectivity

Node: Oriole-02
Channel: out-of-band
Enable Status: true
Port: e0P
IP Address: 192.168.3.12
Subnet: 192.168.0.1
Netmask: 255.255.252.0
Connection Status: full-connectivity

Node: Oriole-03
Channel: in-band
Enable Status: true
Connection Status: active

Node: Oriole-04
Channel: in-band
Enable Status: true
Connection Status: active
4 entries were displayed.

________________________________

Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error.

SpindleNinja
7,389 Views

Thanks,  so yeah.  That is a bit odd...     I would try a IOM re-seat, but open a ticket with support as a CYA. 

 

Stormont
7,376 Views
They suggested I switch to out of band mode and then try the update again. I'm not sure if the shelf was ever cabled for OOB mode, so I'm not sure what that might do.




________________________________

Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error.

SpindleNinja
7,369 Views

Was it ever headswapped from something with e0P?   

 

Go  for the toggle though,  there shouldn't be any ill effects. 

 

Stormont
7,360 Views
No, it was not (we purchased it new and then added it to the cluster)




________________________________

Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error.

SpindleNinja
7,317 Views

Have they ever had an upgrade before?    

 

Also, if you just run "storage shelf acp module show -fields firmware-version"   does it show any firmware versions?   

 

And see if it shows up here too: 

 

node run -node node-3 -command sysconfig -a

Stormont
7,297 Views

                I’m a bit confused by this one.  For a different shelf (a DS212), “storage shelf acp module show -fields firmware-version” shows the following:

node      firmware-version

--------- ----------------

Oriole-01 02.12

Oriole-01 02.12

 

                But “sysconfig –a” shows a different version

 

“Shelf 1: DS212-12  Firmware rev. IOM12 A: 0250  IOM12 B: 0250”

 

 

                Shouldn’t both commands show some form of “0250” instead of “02.12” for the one and “0250” for the other?

 

 

                For the shelf showing all “00”, “storage shelf acp module show -fields firmware-version” shows the following:

node      firmware-version

--------- ----------------

Oriole-03 00.00

 

                While “sysconfig –a” shows:

 

Shelf 6000832241: DS2246  Firmware rev. IOM6 A: 0191  IOM6 B: 0191

aborzenkov
7,227 Views

@Stormont wrote:

Shouldn’t both commands show some form of “0250” instead of “02.12” for the one and “0250” for the other?


There are ACP firmware and IOM firmware. They are different things. In-band ACP requires ACP firmware at least 2.09. 2.12 is the latest ACP firmware version for IOM6 as of now.

Stormont
7,177 Views

I was told that the shelf most likely won't go to OOB mode if it is not cabled that way and that a reset of each ACP module is the next thing to try.  Based on the output below, are these the correct commands?

 

Oriole-03*> acpadmin ibacp_list_all

Number of ACP modules 10

MAC                SES    Last Contact  Protocol   Assigner    Shelf             Current Inband   IOM     shelf

Address            bit    (seconds ago) Version    ACPA ID     S/N               State   ID       Type    flag

---------------------------------------------------------------------------------------------------------------

00:a0:98:49:5e:a4  00      245613      0.0.00.00   0           6000832241        0x9    2d.00.B  IOM6     0x20

00:a0:98:49:5e:cc  00      245613      0.0.00.00   0           6000832241        0x9    2d.00.A  IOM6     0x20

 

 

Oriole-04*> acpadmin ibacp_list_all

Number of ACP modules 10

MAC                SES    Last Contact  Protocol   Assigner    Shelf             Current Inband   IOM     shelf

Address            bit    (seconds ago) Version    ACPA ID     S/N               State   ID       Type    flag

---------------------------------------------------------------------------------------------------------------

00:a0:98:49:5e:a4  00      245240      0.0.00.00   0           6000832241        0x9    2c.00.B  IOM6     0x20

00:a0:98:49:5e:cc  00      245240      0.0.00.00   0           6000832241        0x9    2c.00.A  IOM6     0x20

 

                Based on that, would we be looking at running these commands:

 

node run -node Oriole-03

priv set diag

  1. For a given Module A: sasadmin expander_cli  2d.00.A “acpmgr_tp 3 0”

<wait 10 minutes>

  1. For a given Module B: sasadmin expander_cli 2d.00.B “acpmgr_tp 3 1”

<wait 10 minutes>

node run -node Oriole-04

  1. For a given Module A: sasadmin expander_cli  2c.00.A “acpmgr_tp 3 0”

<wait 10 minutes>

  1. For a given Module B: sasadmin expander_cli 2c.00.B “acpmgr_tp 3 1”

 

amans
7,059 Views

So looks like, you have been suggested to reset the SES bit.  I have hardly seen this work, but certainly  give it a try.

 

In addition, 0212 ACP FW is bundled with ONTAP 9.7 and above. I wonder, why the ACP did not upgrade for you, this would require further investigation from support.

 

 

 

 

Stormont
6,831 Views

This bug was supposedly created to resolve the issue we are seeing:

https://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1050364

 

However, there are apparently times when it doesn't get out of the 0x9 state, so bug https://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1310583 was created to fix that.

 

One thing I did find is that all three shelves that are still running 0191 are connected to the same adapter (one DS2246 on one loop connected to 2c and 2d and two DS4246 shelves connected to 2a and 2b).  Since this is on an FAS8200 that doesn't support OOB and I can't imagine that both adapters in both nodes need to be physically reseated, we must have the bug where they won't get out of the 0x9 state.

 

KristianT
4,902 Views

Hi Gents,

 

I encountered this issue on FAS8040 with DS4246 shelves running OnTap 9.7. Seem this is known bug for in-band configuration. Work-around is to configure out-of-band acp, it should trigger automatic update if you have new packages. Give it few minutes and then switch in to in-band.

 

Have a nice day,

Kristian

Public