FAS8020 Node offline due to failed service processor firmware update during ONTAP update

[Vallinayagam]

One of our lab FAS8020 controller (B) is currently offline due to a failed Service processor firmware update during ONTAP update from 9.1P2 to 9.3P15. The ONTAP 9.3P15 and SP firmware on the other controller (A) is updated without any issues. 

pln-n2-nas2a::> version -node *

pln-n2-nas2a-01:
NetApp Release 9.3P15: Tue Jul 16 19:40:35 UTC 2019

pln-n2-nas2a-02:
NetApp Release 9.3P15: Tue Jul 16 19:40:35 UTC 2019

2 entries were displayed.

After the failed firmware update the SP on controller B was rebooted and it never came up due to mismatch with ONTAP version.

pln-n2-nas2a::> service-processor show
(system service-processor show)
IP Firmware
Node Type Status Configured Version IP Address
------------- ---- ----------- ------------ --------- -------------------------
pln-n2-nas2a-01
SP online true 3.7P1 xx.xx.xx.xx
pln-n2-nas2a-02
SP rebooting true 3.3 xx.xx.xx.xx
2 entries were displayed.

Errors noticed on the event log:

8/31/2019 01:58:13 pln-n2-nas2a-02 ALERT sp.firmware.incompatible: The SP firmware version 3.3 is not compatible with this version of Data ONTAP.
8/31/2019 01:58:13 pln-n2-nas2a-02 ERROR sp.servprocd.upd.error: SP update error: SP firmware update failure has been detected.

The node was brought down automatically  since the SP is down.

8/31/2019 02:13:23 pln-n2-nas2a-02 EMERGENCY sp.ipmi.lost.shutdown: SP heartbeat stopped and cannot be recovered. To prevent hardware damage and data loss, the system will shut down in 2 minutes.

pln-n2-nas2a::system service-processor*> show
IP Firmware
Node Type Status Configured Version IP Address
------------- ---- ----------- ------------ --------- -------------------------
pln-n2-nas2a-01
SP online true 3.7P1 xx.xx.xx.xx
pln-n2-nas2a-02
- node-offline
- - xx.xx.xx.xx
2 entries were displayed.

Is there a way to review the controller B /SP from this situation? Any suggestion on this is much appreciated. Thanks.

[OntapCore · Accepted solution]

Hello Vallinayagam,

We recommend that you go ahead and attempt to upgrade the Service Processor manually via the SP prompt.

Please follow the below steps:

1) Navigate to https://mysupport.netapp.com/ and log into with your NetApp user account.

2) Once logged in, at the top of the screen you should see a tab that says "Downloads" and a select "Firmware" from the drop-down box.

3) On the next page, click "System Firmware & Diagnostics".

4) Select your FAS platform (FAS8020) and click "Go!".

5) Select the radial option "Service Processor Image for installation from the SP prompt".

6) Click "Proceed to Download" next to the SP Firmware 3.7P1 image.

7) After this, you will be taken to the download page that also contains instructions on how to install the SP firmware.

If you still have issues after updating the SP firmware on controller B, please open a NetApp Support case for further investigation.

[RajeshPanda]

@Vallinayagam  can you review the recommendeation given by our experts?

 If the above answer was useful to you, please consider Accepting it as a solution, this might help someone with similar issue.
Thank you!

[Vallinayagam]

That helped. We did a power reset and performed the SP firmware upgrade as per the provided procedure and the controller B SP is back up. Thank you for the suggestions.