Problem after ontap upgrade 7.32 - 7.351

[mhellberg]

Hi All,

Was helping a customer with a ontap upgrade.

The installation went fine, no errors.  Before the reboot,disk firmware etc. was upgraded.

I rebooted the system, waited 10 minutes but the system did´nt come back up.

Went down to the serverroom, connected thru serial and the system had stoped at the boot menu:

(1)    Normal boot.

(2)    Boot without /etc/rc.

(3)    Change password.

(4)    Initialize all disks.

No errros before that what i could see. i choosed nr:1 and the system booted fine.

So i run version:

NetApp   Release 7.3.2: Thu Oct 15 04:17:39 PDT 2009

Still the old version?

If i run version -b:

1:/x86_elf/kernel/primary.krn:   OS 7.3.5.1

Rebooted the filer again. Same problem. The boot menu first, then if choosing number one, the old version.

Any one got a clue?

[cobham_rich]

Hi,

I am planning this same upgrade. Did you get a resolution to your problem, or anything I need to observe to avoid the same issue?

Thanks,

Rich.

[mhellberg]

Hi,

No resolution, have opened a case with netapp but nothing yet...

[cobham_rich]

I better make sure the RLM interface is up then as I'm doing mine remotley!! I'll let you know if it works or ends up the same.

Cheers.

[mhellberg]

The sugestion from netapp was to  run the software download command again.

I´ll going to try that in a couple of days.

Good luck!

[chriszurich]

After the download command completed and upon reboot did you run "update_flash" from the Loader prompt?

Link to 7.3.5.1 upgrade guide. If step 11 is not executed the kernel will not be upgraded and the version command will always display the previous version.

https://now.netapp.com/NOW/knowledge/docs/ontap/rel7351/html/ontap/upgrade/frameset.html

[michelvankessel]

I had this problem before. Just do the update again. Via RLM/CONSOLE that way you can observe the boot process  Kind regards,  Michel van Kessel

[scottgelb]

has to be an enviroment option like autoboot... I would push the latest controller/diags (extract tar from now site, priv set advanced ; download -d, halt, update-flash, set-defaults, bye)... set-defaults should fix a change in the boot option that might be causing this...run by support too.  If firmware/diags are all up to date, just do the set-defaults and bye to change the variables.

[thomas_glodde]

set-defaults might reset your onboard target san ports to initiator, be careful.

[scottgelb]

Without fcadmin config -t initiator, I have never seen that happen.

Typos Sent on Blackberry Wireless

[thomas_glodde]

FAS3140 MetroCluster, i did NDU ontap update as well as update_flash, issued a set-defaults and my onboard 0b and 0d ports went to initiator again (as shipped from factory). had to offline them in maintenance mode and reset them to target.

offtopic: have you noticed that after a power outage filers wont boot up again although autoboot = true is configured??

[scottgelb]

Never have seen that or heard of that.

Typos Sent on Blackberry Wireless

[mhellberg]

Hi,

I run the download command again and then halted the system.

The LOADER boot environment gives me the message: Warning: The CompactFlash contains newer firmware image..

I run update_flash and then bye.

The system boots to the same startmenu as before. When choosing "normal boot" , the system boots 7.32

[michelvankessel]

Which filer do you have? FAS31xx series? if that is true, you shouldn't HALT the filer, but REBOOT it. The FAS31xx series auto upgrade flash and ontap on reboot! Important is to follow the boot process via RLM or CONSOLE. It will show you any error messages I would execute the "software update" command again with the -r switch and then reboot the filer. hope it helps

[mhellberg]

Fas2040

[michelvankessel]

same for the FAS2040

[mhellberg]

i actually used reboot the first time, but maby something went wrong there.

But if i look at the installguide https://now.netapp.com/NOW/knowledge/docs/ontap/rel7351/html/ontap/upgrade/GUID-CA91213B-3534-45C4-AAFE-692DE3E5AB1E.html

it´s says "Halt"...

[michelvankessel]

Some filer models have an environment variable called " AUTO_FW_UPDATE " that is by default ON. Anyway, if you connect to the console you should see some kind of message why the new kernel is not booting. In your case you can also use the BMC. Regards Michel van Kessel

[mhellberg]

Thanks for the reminder of the bmc log!

Cant find any direct errors. This is the events since the Halt:

The boot media contains a newer firmware image (6.1).

Automatic firmware update skipped (ONTAP halted).

Warning: The boot media contains a newer firmware image (6.1).

Please run 'update_flash' at the Loader prompt to update your

system firmware (6.0).

LOADER-B> update_flash

New BIOS Version: 6.1

New Loader Version: 2.1

Saving Primary Image to Backup flash device

Programming ----------------------------.+-- done. 1048576 bytes written

Updating Primary Boot Flash

Programming .+----.+.+.+------.+.+.+.+.+.+.+ done. 1048576 bytes written

LOADER-B> bye

AMI BIOS8 Modular BIOS

Copyright (C) 1985-2009,  American Megatrends, Inc. All Rights Reserved

Portions Copyright (C) 2009 NetApp, Inc. All Rights Reserved

BIOS Version 6.1

......................................

Boot Loader version 2.1

Copyright (C) 2000-2003 Broadcom Corporation.

Portions Copyright (C) 2002-2008 NetApp

4096MB RAM installed

CPU Type: Intel(R) Xeon(R) CPU                  @ 1.66GHz

Starting AUTOBOOT press Ctrl-C to abort...

Loading backup/X86_ELF/kernel/primary.krn:.......0x200000/41406484 0x297d014/14083868 0x36eb730/1732450 0x3892692/6 Entry at 0x00200000

Starting program at 0x00200000

Press CTRL-C for special boot menu

Mon Feb 28 12:13:20 GMT [fci.initialization.failed:error]: Initialization failed on Fibre Channel adapter 0b.

Mon Feb 28 12:13:20 GMT [fci.initialization.failed:error]: Initialization failed on Fibre Channel adapter 0a.

NetApp Release 7.3.2: Thu Oct 15 04:17:39 PDT 2009

Copyright (c) 1992-2009 NetApp.

Starting boot on Mon Feb 28 12:13:09 GMT 2011

Mon Feb 28 12:13:20 GMT [nvram.battery.turned.on:info]: The NVRAM battery is turned ON. It is turned OFF during system shutdown.

Mon Feb 28 12:13:23 GMT [diskown.isEnabled:info]: software ownership has been enabled for this system

(1)  Normal boot.

(2)  Boot without /etc/rc.

(3)  Change password.

(4)  Initialize owned disks (24 disks are owned by this filer).

(4a) Same as option 4, but create a flexible root volume.

(5)  Maintenance mode boot.

I have reopen the case with netapp support. They sugested that i should redownload the file from the web and try the upgrade again... I suppose i must give that a try, but have to wait a week for next maintenance window.

[michelvankessel · Accepted solution]

as you can see the backup kernel is loaded (Loading backup/X86_ELF/kernel/primary.krn) As suggested by Netapp support, I would download the .exe file to the /etc/software directory again and run the update again. Good luck, Michel van Kessel

[scottgelb]

Did you printenv to check environment? Did you run set-defaults ?

Typos Sent on Blackberry Wireless