I'd just like to document a problem and a work around / solution I found with VSC 5.0 and vCenter 5.5 in case anyone else is experiencing something similar as I did not see anything for this in the bug tracker for VSC.
vCenter 5.5 on win2k8r2
VSC on a separate win2k8r2 server
Unable to to backup systems from within vCenter,
Clicking on "Backup and recovery Configuration" option in the VSC plugin in vCenter gives an "unable to connect to virtual storage console" error
Running commands such as "smvi backup list" as well as other smvi commands from the VSC server produces an error and throws java exceptions about handshake failures and a connection has been closed in the logs.
Specific issue (found using Wireshark):
After the tcp three way handshake, VSC by default sends a SSL client hello handshake proposing SSLv3 (0x0300) to vCenter. vCenter would then FIN,ACK the connection immediately causing VSC to issue a fatal alert handshake failure packet back to vCenter and the client programs to fail / error out. There was also periodic background traffic between the two servers that I noticed would always propose and negotiate at TLSv1 so I guessed that forcing the client initiated connections to negotiate at TLSv1 might fix the issue and it did.
Solution / Workaround:
%Programfiles%\Netapp\Virtual Storage Console\smvi\server\etc\wrapper.conf
and find the "wrapper.java.additional.X" statements (should be 4), add the following statement.
%Programfiles%\Netapp\Virtual Storage Console\wrapper\wrapper.conf
and find the "wrapper.java.additional.X" statements (should be 7) add following additional statement.
Restart both VSC services or reboot.
VSC will now propose TLSv1.2 (0x0303) by default and vCenter will negotiate the connection down to TLSv1.0 (0x0301). If you remove TLSv1 from those statements, you'll get the handshake failures again as it appears vCenter 5.5 will only use TLSv1. The "Backup and Recovery" configuration will now display in vCenter, smvi commands complete successfully and backup jobs can be created from within vCenter.
I had added the -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 in a couple other of places (control panel java applet for both the system jre and the one located in the VSC installation folder) and the smvi.bat / .cmd files but the two wrapper.conf files appear to be the only place where you can force specific java run time options for VSC.
This didn't help me with getting VSC to communicate with the filers over TLS per the vulnerabilities listed in https://kb.netapp.com/support/index?page=content&id=9010008 and sslv3 being disabled on them.
I wonder if there is some other way to get that to work.
Used this fixing a problem after updating to vCenter 5.5U3B. I had to remove the TLS v1.1 and 1.2 arguments though. The lines looked like this:
I was getting a HTTP 500 error (Failed to retrieve backup list). The Netapp Interop matrix doesnt have 5.5U3B in the listing and probably should be since 5.5U3B removes SSLv3.
It resolved my problem after upgrading vCenter to 5.5 update 3b, thank you very much.
It was necessary to mention only TLSv1 in the configuration files (as is mentioned in one of the replies to this thread) and to restart the whole windows server. Restarting VSC server service was not sufficient in my case.
Does the work around work, if the netapp ontap version is 126.96.36.199 ? i tried, it didn't work , and i have VSC 4.2.2 / vSphere 5.5 U3B
That is quite an old version of VSC which may be part of the problem, but I would check to make sure any newer versions work with the version of OnTap you have deployed before updating VSC.
This is documented in the following KB article