I'd just like to document a problem and a work around / solution I found with VSC 5.0 and vCenter 5.5 in case anyone else is experiencing something similar as I did not see anything for this in the bug tracker for VSC.
vCenter 5.5 on win2k8r2
VSC on a separate win2k8r2 server
Unable to to backup systems from within vCenter,
Clicking on "Backup and recovery Configuration" option in the VSC plugin in vCenter gives an "unable to connect to virtual storage console" error
Running commands such as "smvi backup list" as well as other smvi commands from the VSC server produces an error and throws java exceptions about handshake failures and a connection has been closed in the logs.
Specific issue (found using Wireshark):
After the tcp three way handshake, VSC by default sends a SSL client hello handshake proposing SSLv3 (0x0300) to vCenter. vCenter would then FIN,ACK the connection immediately causing VSC to issue a fatal alert handshake failure packet back to vCenter and the client programs to fail / error out. There was also periodic background traffic between the two servers that I noticed would always propose and negotiate at TLSv1 so I guessed that forcing the client initiated connections to negotiate at TLSv1 might fix the issue and it did.
VSC will now propose TLSv1.2 (0x0303) by default and vCenter will negotiate the connection down to TLSv1.0 (0x0301). If you remove TLSv1 from those statements, you'll get the handshake failures again as it appears vCenter 5.5 will only use TLSv1. The "Backup and Recovery" configuration will now display in vCenter, smvi commands complete successfully and backup jobs can be created from within vCenter.
I had added the -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 in a couple other of places (control panel java applet for both the system jre and the one located in the VSC installation folder) and the smvi.bat / .cmd files but the two wrapper.conf files appear to be the only place where you can force specific java run time options for VSC.
It resolved my problem after upgrading vCenter to 5.5 update 3b, thank you very much.
It was necessary to mention only TLSv1 in the configuration files (as is mentioned in one of the replies to this thread) and to restart the whole windows server. Restarting VSC server service was not sufficient in my case.