VSC 5.0 SSL Handshake Failures

[JBROADWAY]

I'd just like to document a problem and a work around / solution I found with VSC 5.0 and vCenter 5.5 in case anyone else is experiencing something similar as I did not see anything for this in the bug tracker for VSC.

Setup:

vCenter 5.5 on win2k8r2

VSC on a separate win2k8r2 server

cDOT 8.2.1

Problems encountered:

Unable to to backup systems from within vCenter,

Clicking on "Backup and recovery Configuration" option in the VSC plugin in vCenter gives an "unable to connect to virtual storage console"  error

Running commands such as "smvi backup list" as well as other smvi commands from the VSC server produces an error and throws java exceptions about handshake failures and a connection has been closed in the logs.

Specific issue (found using Wireshark):

After the tcp three way handshake, VSC by default sends a SSL client hello handshake proposing SSLv3 (0x0300) to vCenter.  vCenter would then FIN,ACK the connection immediately causing VSC to issue a fatal alert handshake failure packet back to vCenter and the client programs to fail / error out.  There was also periodic background traffic between the two servers that I noticed would always propose and negotiate at TLSv1 so I guessed that forcing the client initiated connections to negotiate at TLSv1 might fix the issue and it did.

Solution / Workaround:

Open

%Programfiles%\Netapp\Virtual Storage Console\smvi\server\etc\wrapper.conf

and find the "wrapper.java.additional.X" statements (should be 4), add the following statement.

wrapper.java.additional.5=-Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2

Open

%Programfiles%\Netapp\Virtual Storage Console\wrapper\wrapper.conf

and find the "wrapper.java.additional.X" statements (should be 7) add following additional statement.

wrapper.java.additional.8=-Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2

Restart both VSC services or reboot.

VSC will now propose TLSv1.2 (0x0303) by default and vCenter will negotiate the connection down to TLSv1.0 (0x0301).  If you remove TLSv1 from those statements, you'll get the handshake failures again as it appears vCenter 5.5 will only use TLSv1.  The "Backup and Recovery" configuration will now display in vCenter, smvi commands complete successfully and backup jobs can be created from within vCenter.

I had added the -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 in a couple other of places (control panel java applet for both the system jre and the one located in the VSC installation folder) and the smvi.bat / .cmd files but the two wrapper.conf files appear to be the only place where you can force specific java run time options for VSC.

[CSEIZMAIR]

Thanks for the detailed fix. It also helped me with the same errors.

[rubinsed1]

This didn't help me with getting VSC to communicate with the filers over TLS per the vulnerabilities listed in https://kb.netapp.com/support/index?page=content&id=9010008 and sslv3 being disabled on them.

I wonder if there is some other way to get that to work.

[Coldfirex]

Worked here with VSC 4.2.2 and VCenter 5.1 u3 that is set to use TLSv1.  Thanks!

[JCobe]

Used this fixing a problem after updating to vCenter 5.5U3B. I had to remove the TLS v1.1 and 1.2 arguments though. The lines looked like this:

wrapper.java.additional.5=-Dhttps.protocols=TLSv1

I was getting a HTTP 500 error (Failed to retrieve backup list). The Netapp Interop matrix doesnt have 5.5U3B in the listing and probably should be since 5.5U3B removes SSLv3.

[dkomanek]

It resolved my problem after upgrading vCenter to 5.5 update 3b, thank you very much.

It was necessary to mention only TLSv1 in the configuration files (as is mentioned in one of the replies to this thread) and to restart the whole windows server. Restarting VSC server service was not sufficient in my case.

Thanks again,

D.

[KvK]

Hello,

Thanks for this info.

The same problems occurred with VSC 4.2.2 when we updated vCenter 5.5 U3 to  vCenter 5.5 U3b

I used your work around / solution to solv our problem 

Regards,

René

[inva]

Does the work around work, if the netapp ontap version is 7.3.5.1 ? i tried, it didn't work , and i have VSC 4.2.2 / vSphere 5.5 U3B

[GenosMan]

Inva,

That is quite an old version of VSC which may be part of the problem, but I would check to make sure any newer versions work with the version of OnTap you have deployed before updating VSC.

[thigian007]

I had this same exact problem after upgrading VCenter to 5.5 Update 3b. this should be an official Netapp article. Worked like a charm.

[georgevj]

This is documented in the following KB article

https://kb.netapp.com/support/index?page=content&id=2026327

[ThomasHoffmannKBB]

Thank you for this solution. It helped me to resolve the Problem as described.

Best regerds

Thomas

[geluyan]

solved the same issue by one of our customer. thx! This is one for my documention 😄