The off-box OnCommand System Manager is no longer supported. We would encourage people to learn and simulate with the new ONTAP 9 environment, with built in, web based system manager.
If you're getting a 500 server error when launching it, it is probably an issue with Java Runtime Environment, potentially due to an incompatibility. There are logs in the install directory - bootlog.log I believe.
This document may have more troubleshooting information - https://kb.netapp.com/app/answers/answer_view/a_id/1032155/~/how-to-troubleshoot-issues-with-oncommand-system-manager-3.x-