topic Re: VSC 5.0 SSL Handshake Failures in VMware Solutions Discussions

VSC 5.0 SSL Handshake Failures

JBROADWAY — Wed, 04 Jun 2014 07:35:40 GMT

I'd just like to document a problem and a work around / solution I found with VSC 5.0 and vCenter 5.5 in case anyone else is experiencing something similar as I did not see anything for this in the bug tracker for VSC.

Setup:

vCenter 5.5 on win2k8r2

VSC on a separate win2k8r2 server

cDOT 8.2.1

Problems encountered:

Unable to to backup systems from within vCenter,

Clicking on "Backup and recovery Configuration" option in the VSC plugin in vCenter gives an "unable to connect to virtual storage console" error

Running commands such as "smvi backup list" as well as other smvi commands from the VSC server produces an error and throws java exceptions about handshake failures and a connection has been closed in the logs.

Specific issue (found using Wireshark):

After the tcp three way handshake, VSC by default sends a SSL client hello handshake proposing SSLv3 (0x0300) to vCenter. vCenter would then FIN,ACK the connection immediately causing VSC to issue a fatal alert handshake failure packet back to vCenter and the client programs to fail / error out. There was also periodic background traffic between the two servers that I noticed would always propose and negotiate at TLSv1 so I guessed that forcing the client initiated connections to negotiate at TLSv1 might fix the issue and it did.

Solution / Workaround:

Open

%Programfiles%\Netapp\Virtual Storage Console\smvi\server\etc\wrapper.conf

and find the "wrapper.java.additional.X" statements (should be 4), add the following statement.

wrapper.java.additional.5=-Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2

Open

%Programfiles%\Netapp\Virtual Storage Console\wrapper\wrapper.conf

and find the "wrapper.java.additional.X" statements (should be 7) add following additional statement.

wrapper.java.additional.8=-Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2

Restart both VSC services or reboot.

VSC will now propose TLSv1.2 (0x0303) by default and vCenter will negotiate the connection down to TLSv1.0 (0x0301). If you remove TLSv1 from those statements, you'll get the handshake failures again as it appears vCenter 5.5 will only use TLSv1. The "Backup and Recovery" configuration will now display in vCenter, smvi commands complete successfully and backup jobs can be created from within vCenter.

I had added the -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 in a couple other of places (control panel java applet for both the system jre and the one located in the VSC installation folder) and the smvi.bat / .cmd files but the two wrapper.conf files appear to be the only place where you can force specific java run time options for VSC.

Re: VSC 5.0 SSL Handshake Failures

CSEIZMAIR — Fri, 13 Jun 2014 16:07:09 GMT

Thanks for the detailed fix. It also helped me with the same errors.

Re: VSC 5.0 SSL Handshake Failures

rubinsed1 — Wed, 25 Mar 2015 20:26:55 GMT

This didn't help me with getting VSC to communicate with the filers over TLS per the vulnerabilities listed in https://kb.netapp.com/support/index?page=content&id=9010008 and sslv3 being disabled on them.

I wonder if there is some other way to get that to work.

Re: VSC 5.0 SSL Handshake Failures

Coldfirex — Thu, 25 Jun 2015 16:17:08 GMT

Worked here with VSC 4.2.2 and VCenter 5.1 u3 that is set to use TLSv1. Thanks!

Re: VSC 5.0 SSL Handshake Failures

JCobe — Wed, 30 Dec 2015 16:56:59 GMT

Used this fixing a problem after updating to vCenter 5.5U3B. I had to remove the TLS v1.1 and 1.2 arguments though. The lines looked like this:

wrapper.java.additional.5=-Dhttps.protocols=TLSv1

I was getting a HTTP 500 error (Failed to retrieve backup list). The Netapp Interop matrix doesnt have 5.5U3B in the listing and probably should be since 5.5U3B removes SSLv3.

Re: VSC 5.0 SSL Handshake Failures

dkomanek — Wed, 13 Jan 2016 22:41:59 GMT

It resolved my problem after upgrading vCenter to 5.5 update 3b, thank you very much.

It was necessary to mention only TLSv1 in the configuration files (as is mentioned in one of the replies to this thread) and to restart the whole windows server. Restarting VSC server service was not sufficient in my case.

Thanks again,

Re: VSC 5.0 SSL Handshake Failures

KvK — Wed, 20 Jan 2016 21:01:06 GMT

Hello,

Thanks for this info.

The same problems occurred with VSC 4.2.2 when we updated vCenter 5.5 U3 to vCenter 5.5 U3b

I used your work around / solution to solv our problem

Regards,

René

Re: VSC 5.0 SSL Handshake Failures

thigian007 — Fri, 29 Jan 2016 22:57:32 GMT

I had this same exact problem after upgrading VCenter to 5.5 Update 3b. this should be an official Netapp article. Worked like a charm.

Re: VSC 5.0 SSL Handshake Failures

inva — Thu, 04 Feb 2016 06:23:38 GMT

Does the work around work, if the netapp ontap version is 7.3.5.1 ? i tried, it didn't work , and i have VSC 4.2.2 / vSphere 5.5 U3B

Re: VSC 5.0 SSL Handshake Failures

GenosMan — Thu, 04 Feb 2016 12:03:05 GMT

Inva,

That is quite an old version of VSC which may be part of the problem, but I would check to make sure any newer versions work with the version of OnTap you have deployed before updating VSC.

Re: VSC 5.0 SSL Handshake Failures

georgevj — Mon, 15 Feb 2016 09:45:13 GMT

This is documented in the following KB article

https://kb.netapp.com/support/index?page=content&id=2026327

Re: VSC 5.0 SSL Handshake Failures

ThomasHoffmannKBB — Fri, 19 Feb 2016 07:46:16 GMT

Thank you for this solution. It helped me to resolve the Problem as described.

Best regerds

Thomas

Re: VSC 5.0 SSL Handshake Failures

geluyan — Fri, 21 Oct 2016 09:52:51 GMT

solved the same issue by one of our customer. thx! This is one for my documention 😄