Getting Failed to install Certificate. Reason: "The certificate has expired.". when upgrading ONTAP

[craig_schuer]

I am trying to upgrade our lab cluster from 9.1P8 to 9.3P7 and it is failing on the last startup phase of the upgrade. It is a 2 node switched cluster with FAS8060 controllers. I eventually want to upgrade to 9.5x to test for our production environment. I've tried doing an additional takeover/giveback to see if that would clear the error. 

Here's what I'm seeing. Google searches have come up empty so far...

lab-cluster01::*> version
NetApp Release 9.1P8: Wed Aug 30 13:33:41 UTC 2017

Info: The output from the version command above may not be correct because
upgrade is in progress or has failed in one or more nodes in the cluster.
Use the "upgrade-revert show" command in advanced mode to view the status
of upgrade.

lab-cluster01::*> upgrade-revert show
(system node upgrade-revert show)

Node: lab-cluster01-03 Status:
complete

Status Message: The upgrade is complete.

Vers Phase Status Upgrade Phase Status Message
---- ---------- -------- ------------------------------------------------------
510 pre-root applied No upgrade is required for this phase.
510 pre-apps applied Upgrade successful.
510 post-apps applied Upgrade successful.
700 pre-root applied No upgrade is required for this phase.
700 pre-apps applied Upgrade successful.
700 post-apps applied Upgrade successful.

Node: lab-cluster01-04 Status:
aborted

Status Message: The upgrade aborted. Contact support personnel for the upgrade repair procedure.

Vers Phase Status Upgrade Phase Status Message
---- ---------- -------- ------------------------------------------------------
510 pre-root applied No upgrade is required for this phase.
510 pre-apps applied Upgrade successful.
510 post-apps applied Upgrade successful.
700 pre-root applied No upgrade is required for this phase.
700 pre-apps applied Upgrade successful.
700 post-apps aborted Failed to install Certificate. Reason: "The certificate has expired.".
12 entries were displayed.

lab-cluster01::*> node image show
Is Is Install
Node Image Default Current Version Date
-------- ------- ------- ------- ------------------------- -------------------
lab-cluster01-03
image1 false false 9.1P8 10/24/2019 12:51:32
image2 true true 9.3P7 10/24/2019 14:17:20
lab-cluster01-04
image1 false false 9.1P8 10/24/2019 12:52:36
image2 true true 9.3P7 10/24/2019 14:18:28
4 entries were displayed.

I don't get what certificate I'm supposed to update... 

Please help!

Thanks

[TMACMD]

Why did you not use the most current version of ONTAP 9.3 (P16)?

This is the ONE AND ONLY bug fixed in ONTAP 9.3P14.

This is a very well know issue. Support should be able to help

[craig_schuer]

I used 9.3P7 because we are using that version on our production systems. 

[craig_schuer]

Is going to P14 (or above) the only way to clear this error? I have never seen this before. 

[TMACMD]

From the bug:

 In ONTAP 9.2, a set of trusted root CA certificates were introduced in 
 ONTAP's certificate management so that the admin SVM can allow applications 
 running in ONTAP to seamlessly establish TLS connections to external entities.
 All these certificates get installed on every new cluster installation starting
 from 9.2 and on all upgrades from 9.1 or below to 9.2+.
 
 Each certificate has an expiration date associated with it and as of July 06, 
 2019, one of the default certificates has expired. As a general security principle,
 ONTAP does not allow installation of expired certificates and as a result, the 
 upgrades and new installs have been impacted for the above mentioned releases.
 
 To be specific, new installations are impacted at the creation of the cluster
 and for upgraded systems, ONTAP can serve data but no new features are available.

 Contact NetApp Technical Support for assistance.

[TMACMD · Accepted solution]

https://mysupport.netapp.com//NOW/cgi-bin/bol?Type=Detail&Display=1250500

[craig_schuer]

I'll try upgrading to 9.3P14

[craig_schuer]

Upgraded to 9.3P14 last week and error went away.