OnCommand API Services use ca-issued certificate with non-default password

[acjackson]

Hi,

I'm trying to use a ca-issued certificate instead of a self-signed certificate.

If I use a Java Keystore File (JKS) with the default password 'changeit' everything works as expected, but if I'm trying to use a non-default password I get the following error

2017-08-23 14:40:05,233 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service jboss.server.controller.management.security_realm.SSLRealm.key-manager: org.jboss.msc.service.StartException in service jboss.server.controller.management.security_realm.SSLRealm.key-manager: JBAS015229: Unable to start service
at org.jboss.as.domain.management.security.FileKeystore.load(FileKeystore.java:154)
at org.jboss.as.domain.management.security.FileKeyManagerService.start(FileKeyManagerService.java:119)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [rt.jar:1.8.0_73]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.8.0_73]
at java.lang.Thread.run(Unknown Source) [rt.jar:1.8.0_73]
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
at sun.security.provider.JavaKeyStore.engineLoad(Unknown Source) [rt.jar:1.8.0_73]
at sun.security.provider.JavaKeyStore$JKS.engineLoad(Unknown Source) [rt.jar:1.8.0_73]
at sun.security.provider.KeyStoreDelegator.engineLoad(Unknown Source) [rt.jar:1.8.0_73]
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(Unknown Source) [rt.jar:1.8.0_73]
at java.security.KeyStore.load(Unknown Source) [rt.jar:1.8.0_73]
at org.jboss.as.domain.management.security.FileKeystore.load(FileKeystore.java:113)
... 6 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
... 12 more

According to the Installation and Setup Guide, configuring /opt/netapp/essentials/jboss/standalone/configuration/standalone.xml should be enough

<system-properties>
    <property name="apiserver.keystore.keypassword" value="NEW_PASSWORD" />
    <property name="apiserver.keystore.storepassword" value="NEW_PASSWORD" />
</system-properties>

As I was getting the above error, I've tried changing password in /opt/netapp/api-server/api-tools/config/keystore-config.properties

apiserver.keystore.keypassword="NEW_PASSWORD"
apiserver.keystore.storepassword="NEW_PASSWORD"

and added this in /opt/netapp/essentials/jboss/standalone/configuration/standalone.xml

<security-realm name="SSLRealm">
    <server-identities>
        <ssl>
            <keystore path="apiservice/keystore.jks" relative-to="jboss.server.config.dir" keystore-password="NEW_PASSWORD" key-password="NEW_PASSWORD" alias="server"/>
        </ssl>
    </server-identities>
</security-realm>

Maybe someone managed to get it working and can help me out  

[Ladislav_Hajzer · Accepted solution]

==========================================================================
CONTAINER - Generate SSL CSR
==========================================================================

[1] By Default, the SSL certificate named keystore.jks is in the directory /opt/netapp/essentials/standalone/configuration/apiservice/
[2] Create backup of original keystore file
[3] Remove original keystore file
[4] Change default password for keystore file in keystore configuration file
[5] Change default password for keystore file in JAVA application "Netapp API services"
[6] Generate new keystore "keystore.jks" and key pair
[7] List content of the keystore "keystore.jks"
[8] Generate CSR For Private Key (alias) "hostname"
==========================================================================
[1]# cd /opt/netapp/essentials/jboss/standalone/configuration/apiservice/
[2]# cp ./keystore.jks ./keystore.jks.old
[3]# rm ./keystore.jks
[4]# vi /opt/netapp/api-server/api-tools/config/keystore-config.properties
--------------------------------------------------------------------------------------------------------
apiserver.keystore.keypassword=<KEY_PASSWORD>
apiserver.keystore.storepassword=<KEYSTORE_PASSWORD>
--------------------------------------------------------------------------------------------------------
[5]# vi /opt/netapp/essentials/jboss/standalone/configuration/standalone-full.xml
--------------------------------------------------------------------------------------------------------
...
<system-properties>
<property name="apiserver.keystore.keypassword" value="<KEY_PASSWORD>"/>
<property name="apiserver.keystore.storepassword" value="<KEYSTORE_PASSWORD>"/>
</system-properties>
...
--------------------------------------------------------------------------------------------------------
[6]# keytool -genkey -alias hostname -keyalg RSA -keystore keystore.jks -keysize 2048
--------------------------------------------------------------------------------------------------------
keytool -genkey -alias hostname -keyalg RSA -keystore keystore.jks -keysize 2048
Enter keystore password:
Re-enter new password:
What is your first and last name?
[Unknown]: hostname.domain.com
What is the name of your organizational unit?
[Unknown]:
What is the name of your organization?
[Unknown]: COMPANY
What is the name of your City or Locality?
[Unknown]: CITY
What is the name of your State or Province?
[Unknown]:
What is the two-letter country for this unit?
[Unknown]: CX
Is CN=hostname.domain.com, OU=Unknown, O=COMPANY, L=CITY, ST=Unknown, C=CX correct?
[no]: yes

Enter key password for <hostname>
(RETURN if same as keystore password): ENTER

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.jks -CXststoretype pkcs12".
--------------------------------------------------------------------------------------------------------
[7]# keytool -list -v -keystore keystore.jks
--------------------------------------------------------------------------------------------------------
[8]# keytool -certreq -alias hostname -file hostname.csr -keystore keystore.jks
--------------------------------------------------------------------------------------------------------

==========================================================================
CONTAINER - Import Root/CA certificates
==========================================================================

!!! NOTE !!!
If Root/CA certificates are already exists in system-wide CA keystore ($JAVA_HOME/jre/lib/security/cacerts),
then we don't need to do this. In our case we need only step [2].
!!! NOTE !!!

[1] Import CA certificate "CA1" to keystore "keystore.jks"
[2] Import CA certificate "CA2" to keystore "keystore.jks"
--------------------------------------------------------------------------------------------------------
# /opt/netapp/essentials/jboss/standalone/configuration/apiservice/
[1]# keytool -importcert -trustcacerts -file CA1.crt -alias CA1 -keystore keystore.jks
[2]# keytool -importcert -trustcacerts -file CA2.crt -alias CA2 -keystore keystore.jks
--------------------------------------------------------------------------------------------------------

==========================================================================
CONTAINER - Import signed SSL certificate
==========================================================================

Import signed SSL certificate to keystore "keystore.jks"
--------------------------------------------------------------------------------------------------------
# /opt/netapp/essentials/jboss/standalone/configuration/apiservice/
# keytool -importcert -trustcacerts -file hostname.crt -alias hostname -keystore keystore.jks
--------------------------------------------------------------------------------------------------------

Restart API services
--------------------------------------------------------------------------------------------------------
# /etc/init.d/apiserver restart
--------------------------------------------------------------------------------------------------------

LINKz
==========================================================================

2014 - Java Keytool Essentials: Working with Java Keystores
https://www.digitalocean.com/community/tutorials/java-keytool-essentials-working-with-java-keystores

2008 - The Most Common Java Keytool Keystore Commands
https://www.sslshopper.com/article-most-common-java-keytool-keystore-commands.html

[Ladislav_Hajzer]

Hi all.

LIttle correction for step [6].

😉

[6] Generate new keystore "keystore.jks" and key pair
--------------------------------------------------------------------------------------------------------
[6]# keytool -genkey -alias hostname -keyalg RSA -keystore keystore.jks -keysize 2048
--------------------------------------------------------------------------------------------------------
keytool -genkey -alias hostname -keyalg RSA -keystore keystore.jks -keysize 2048
Enter keystore password:
Re-enter new password:
What is your first and last name?
[Unknown]: hostname.domain.com
What is the name of your organizational unit?
[Unknown]:
What is the name of your organization?
[Unknown]: COMPANY
What is the name of your City or Locality?
[Unknown]: CITY
What is the name of your State or Province?
[Unknown]:
What is the two-letter country for this unit?
[Unknown]: CX
Is CN=hostname.domain.com, OU=Unknown, O=COMPANY, L=CITY, ST=Unknown, C=CX correct?
[no]: yes

Enter key password for <hostname>
(RETURN if same as keystore password): <KEY_PASSWORD>

[marcinlub]

I did (almost) the same - without re-creating the keystore. 

When I'm listing the content of the keystore it list all properly (keytool -list -v -keystore /opt/netapp/essentials/jboss/standalone/configuration/apiservice/keystore.jks)

however jboss failes to start with below error. 

Is it really required to create new keystore from scratch ? Why can't I use the one that exists?

Or what else I might be doing wrong?

BTW the mamual is very badly written when it comes to this (also some paths are wrong in manual)

JBOSS error:

2018-03-01 09:46:34,758 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-3) MSC000001: Failed to start service jboss.server.controller.management.security_realm.SSLRealm.key-manager: org.jboss.msc.service.StartException in service jboss.server.controller.management.security_realm.SSLRealm.key-manager: Failed to start service
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_161]
Caused by: java.lang.IllegalStateException: org.jboss.msc.service.StartException in anonymous service: WFLYDM0018: Unable to start service
        at org.jboss.as.domain.management.security.FileKeyManagerService.loadKeyStore(FileKeyManagerService.java:193)
        at org.jboss.as.domain.management.security.AbstractKeyManagerService.createKeyManagers(AbstractKeyManagerService.java:125)
        at org.jboss.as.domain.management.security.AbstractKeyManagerService.start(AbstractKeyManagerService.java:83)
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]
        ... 3 more
Caused by: org.jboss.msc.service.StartException in anonymous service: WFLYDM0018: Unable to start service
        at org.jboss.as.domain.management.security.FileKeystore.load(FileKeystore.java:153)
        at org.jboss.as.domain.management.security.FileKeyManagerService.loadKeyStore(FileKeyManagerService.java:189)
        ... 7 more
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780) [rt.jar:1.8.0_161]
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) [rt.jar:1.8.0_161]
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) [rt.jar:1.8.0_161]
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) [rt.jar:1.8.0_161]
        at java.security.KeyStore.load(KeyStore.java:1445) [rt.jar:1.8.0_161]
        at org.jboss.as.domain.management.security.FileKeystore.load(FileKeystore.java:112)
        ... 8 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778) [rt.jar:1.8.0_161]
        ... 13 more

[Ladislav_Hajzer]

Hi.

Marcinlub> I did (almost) the same - without re-creating the keystore.

LH> Almost, but you added new keystore configuration to file [1] and in my opinion this is not necessarry because this configuration is already in file [2]. Enough is to change passwords in files [2] and [3].

Marcinlub> When I'm listing the content of the keystore it list all properly

LH> Yes of course, but you still use old Netapp password "changeit", which is password for Keystore, but also for key.

Marcinlub> Is it really required to create new keystore from scratch ? Why can't I use the one that exists?

LH> No, it is not required, you can use existing keystore, but if you want to change default passwords for keystore and key you must also change this (because files [2] and [3] only use those passwords). If you want to change the password for JAVA keystore you can do this as in [4]. If you want to change password for the specific key which is stored in keystore you can do this as in [5].

Best regards

L.H.

[1] /opt/netapp/essentials/jboss/standalone/configuration/standalone.xml

[2] /opt/netapp/essentials/jboss/standalone/configuration/standalone-full.xml

[3] /opt/netapp/api-server/api-tools/config/keystore-config.properties

[4] JAVA keystore - change keystore password

# cd /opt/netapp/essentials/jboss/standalone/configuration/apiservice/

# keytool -storepasswd -keystore keystore.jks

[5] JAVA keystore - change password for specific key

# cd /opt/netapp/essentials/jboss/standalone/configuration/apiservice/

# keytool -keystore keystore.jks -alias <key_alias> -keypasswd

NOTE: <key_alias> is alias for specific key which is stored in keystore.