Unable to access via HTTPS, OnCommand System Manager after SSL certificate Renew, FAS2240 9.1P20

[Meadastian]

Hello

We are trying to renew an expired Sysmgr certificate for access the Oncommand sys man. on a Fas2240 with ontap 9.1P20.

We used the netapp kb (https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/How_to_renew_a_Self-Signed_SSL_certificate_in_ONTAP_9#), we checked all the steps. In this moment we have a new self signed certificate assigned to the manager SVM "fasclu" it is "Server enabled", but when browsing the mgmt cluster ip, or a singel node mgmt ip, we get a ERR_CONNECTION_CLOSED.

We have already rebooted one node, but nothing changes. If we enable the HTTP access, we are able to logon to oncommand.

Also a new certificate was issued but nothing.

Did you ever experienced anithing like this? The same operation with the same commands was issued on a FAS220 with same ontap and all went ok.

Thank you.

[hamdani]

Look up these KB's

ERR_CONNECTION_CLOSED Observed when accessing System Manager
How to tackle common issues with trying to upload image into System Manager

After upgrading ONTAP, System Manager and API are failing

[Meadastian]

thank you hamdani, but the first KB link to the kb that we used for renewing the certificate, the second kb link to third kb that say to renew the certificate as showed in the first kb.

We have already searched the kb's for this problems, but we did not resolve.

Also we have already renewed an expired certificate with this kbs with no problem on a Fas2220, it'only the 2240 that has the problem...

Any idea? thank you very much. i appreciate.

[hamdani]

let us confirm the latest certificate serial number and make sure "server-enabled" for ssl show command, also confirm the cluster health

example: Putty into cluster console and please confirm the commands below

cluster show

cluster ring show

sec cert show -common-name fasclu
ssl show -vserver fasclu  -fields vserver,server-enabled,common-name,client-enabled,serial
services web show -vserver fasclu -name sysmgr

[Meadastian]

Thank you

in the meantime we tried to use our internal CA for generating the certificate but same result

here the output of the commands (i am hiding the internal domain name only)

fasclu::> cluster show
Node Health Eligibility
--------------------- ------- ------------
fasclu-01 true true
fasclu-02 true true
2 entries were displayed.

fasclu::> sec cert show -common-name fasclu.domain.it
(security certificate show)
Vserver Serial Number Common Name Type
---------- --------------- -------------------------------------- ------------
fasclu 7000000E27198D8DADAC392340000300000E27
fasclu.domain.it server
Certificate Authority: domain CA
Expiration Date: Thu Aug 29 11:58:36 2024

fasclu::> ssl show -vserver fasclu -fields vserver,server-enabled,common-name,client-enabled,serial
(security ssl show)
vserver serial common-name server-enabled client-enabled
------- -------------------------------------- ------------------------- -------------- --------------
fasclu 7000000E27198D8DADAC392340000300000E27 fasclu.domain.it true false

fasclu::> services web show -vserver fasclu -name sysmgr

Vserver: fasclu
Service Name: sysmgr
Type of Vserver: admin
Version of Web Service: 1.0.0
Description of Web Service: OnCommand System Manager
Long Description of Web Service: The OnCommand System Manager web service
Service Requirements: -
Default Authorized Roles: admin, readonly
Enabled: true
SSL Only: true

[hamdani]

Thanks for the command output and they look good and i have found this error in mgwd log file can we confirm if this is still happening

Error:

00000023.0000062f 00001c8c Mon Aug 29 2022 10:28:07 +02:00 [kern_mgwd:info:1680] 0x83906bc00: 86ffff0000000039: ERR: job_manager::rdb_util: create_rw_transaction:src/job_rec_int.cc:618 create_rw_transaction(job_manager job tx loop): Failed to get transaction. Node "fasclu-02" on ring "Management" is offline. Check the health of the cluster using the "cluster show" command. For further assistance, contact technical support. (393271)

Run these command:

set d; row 0;

cluster ring show

If you see the mgmt ring is offline, please create a case with NetApp helpdesk so we will check the health of the cluster db's.

[Meadastian]

Hello hamdani, i think it's all online from the output...

fasclu::*> cluster ring show
Node UnitName Epoch DB Epoch DB Trnxs Master Online
--------- -------- -------- -------- -------- --------- ---------
fasclu-01 mgmt 15 15 3561 fasclu-01 master
fasclu-01 vldb 14 14 20 fasclu-01 master
fasclu-01 vifmgr 13 13 48 fasclu-01 master
fasclu-01 bcomd 13 13 4 fasclu-01 master
fasclu-01 crs 13 13 1 fasclu-01 master
fasclu-02 mgmt 15 15 3561 fasclu-01 secondary
fasclu-02 vldb 14 14 20 fasclu-01 secondary
fasclu-02 vifmgr 13 13 48 fasclu-01 secondary
fasclu-02 bcomd 13 13 4 fasclu-01 secondary
fasclu-02 crs 13 13 1 fasclu-01 secondary
10 entries were displayed.

[Meadastian]

hello Hamdani, do you have any other idea?

We have 3 netapp, this fas 2240 is the only with this type of problem. It is very strange that we can't restore the https access. are there any log that we can analize for troubleshoot this problem?

thank you very much. Sebastian

[hamdani · Accepted solution]

Sebastian, i think you should create a case

The logs shows the following and we have also seen expired vserver certificates

[Wed Aug 31 00:22:01.538982 2022] [ssl:notice] [pid 7050:tid 34431100160] [client 192.168.0.9:49966] [vserver 4294967295] Setting server certificate chain file /mroot/etc/vserver_4294967295/certificates/ssl/server/5+42000003F8141743C3FFB31D720003000003F8+fasclu.meadinformatica.it/chain.pem

[Wed Aug 31 00:22:01.539047 2022] [ssl:notice] [pid 7050:tid 34431100160] [client 192.168.0.9:49966] [vserver 4294967295] Certificate-based client authentication is not configured for this vserver

[Wed Aug 31 00:22:01.539645 2022] [ssl:emerg] [pid 7050:tid 34431100160] AH01903: Failed to configure CA certificate chain!

See if the KB help?

How to install a Certificate Authority (CA) signed certificate in ONTAP for System Manager use

KB: https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/How_to_install_a_Certificate_Authority_(CA)_signed_certificate_in_ONTA...

ONTAP System Manager is not reachable after certificate changes

KB: https://kb.netapp.com/Advice_and_Troubleshooting/Data_Infrastructure_Management/ONTAP_System_Manager/ONTAP_System_Manager_is_not_reachable_after_certi...

if the KB didnt resolve the issue, please create a case with NetApp help desk and provide them the information collected here and the command output

"security cert show" from your cluster