FAS2650 single controller - Shelf FW update failure

[sraudonis]

Hi,

i'm using a FAS2650 in a single controller configuration. Now i'm seeing manny shelf firmware update error messages:

ERROR sfu.downloadSummaryErrors: Shelf firmware updated on 0 shelves, failed to update on 1 shelf.
ERROR sfu.downloadError: [storage download shelf]: Error downloading firmware file IOM12E.0210.SFW on ?.shelf0.General firmware update error.
ERROR sfu.SASDownloadFailed: [storage download shelf]: Download failed on IOM12E.0210.SFW on disk shelf controller module B on disk shelf ?.0.
ERROR scsi.cmd.checkCondition: Enclosure services device 0b.00.99: Check Condition: CDB 0x1d: Sense Data SCSI:illegal request - (0x5 - 0x35 0x4 0x0)(16).
ERROR sfu.firmwareDownrev: Disk shelf firmware needs to be updated on 1 disk shelf.

He is reporting "module B", but it is a single controller:

Shelf 0: DS224-12 Firmware rev. IOM12E A: 0201 B: ----

How can i update the firmware?

Kind regards

Stefan

[Ontapforrum]

Hi,

Please raise a ticket with NetApp Technical Support.

https://kb.netapp.com/Advice_and_Troubleshooting/Flash_Storage/AFF_Series/Shelf_firmware_failed_to_update

Please note that for - FAS2600 series single-path configurations, you must halt your system to re-seat or replace an faulty IOM12 Module. Therefore you will need to plan a downtime. If it's a dual-path, you can try re-seating IO Module non-disruptively.

Thanks!

[sraudonis]

Hi,

but this is the internal IOM12E not a shelf...

I will try a power off of the complete system first and if this not helps i will open a ticket...

[Ontapforrum]

ok, give it a try. Internal Shelf is a single-path, so no other option but to take it down. Try shutting down gracefully and power it up and see if that error goes away.

[sraudonis]

During startup he isn't update, because i rebooted a few days ago during update from 9.7p6 to 9.7p7 and the error i noticed already before that reboot.

So i will test what happens when removing the power...

[sraudonis]

No change, i shutdown the system and swithed it off. After switching on i searched in maintenance mode if there are firmware update commands available but i found nothing.

O.k. i try to open a case...

[amans]

Hello Stefan,

This is a BUG.

https://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1333095

Regards,

Aman

[sraudonis]

Hi Aman,

o.k. thank you for that information, but:

"If you want to eliminate the error messages, after new IOM12E/IOM6E firmware has been installed and updated, you can use the (priv: advanced) "storage firmware shelf delete" command to delete the firmware files from the filesystem."

After, but how to install in module A?

He currently shows that 201 is installed and 210 is available...

Kind regards

Stefan

[andris]

If the 0210 files are correctly installed, a reboot would probably result in the A module being updated. Then, it will complain again, but you can remove the 0210 file at that point.

[andris]

A less disruptive approach might be to remove the 0210 file, wait 15 minutes, then re-add the 0210 file again. If the timing works out, it might update A before trying B.

[sraudonis]

Timing... o.k. i will try, later and will tell the result...

I'm thinking already about to plugin a 2nd controller.

[andris]

There's no downside to making the system HA...

[sraudonis]

@andris, the downside is the higher power consumption...

I'm using the system here in my homeoffice, a non productive test and playing environment. 

And i made already some reboots, and the module wasn't updated. I will try now to uninstall and reinstall the FW files.

@maffo, i have already opened a case 2 days ago, but with prio 4, as i wrote, it's a not productive environment... I don't want to prevent the support from helping others with more important problems...

[sraudonis]

O.k. timing is bad, i tryed removed the FW files and reinstalled them. Then i tryed a update and reoot but no change...

Other question: When i insert the 2nd controller, what status of tis controller is needed that the other controller has his path to module B? (loader, manintenance mode, running ONTAP)

[sraudonis · Accepted solution]

I used the lunch break for a test and here is a workaround:

1. Insert a 2nd controller and press ctrl-c to stop autoboot, leave it in the loader prompt.

2. Plug in one SAS cable from controller A 0a to controller B 0b

3. In sysconfig -a you should now see disks on both SAS HBA's and FW infos for both IOM's.

4. Start a FW update and wait.

5. After a few minutes the FW update finishes on both modules. Now the 2nd controller can be removed.

All can be done online and isn't disruptive. It will only result in a Autosupport message when the 2nd controller is removed, because the disks have lost the redundancy...

[sraudonis]

Additional note:

The symptom of burt 1333095 is still visible, so i deleted the FW files. And this ok, because now the installed FW is the latest...

[maffo]

hi @sraudonis 

as mentioned previously you are unfortunately hitting BURT 1333095 : since the system has a single embedded IOM and it has upgraded successfully. You can use the "storage firmware shelf delete" command to delete IOM12E software from ONTAP in order to stop these messages from occurring.

If you are still having issues, please feel free to open a technical case to continue troubleshooting with us.