The plugins are part of the agent so no agent = no plugins.
In SC 3.x we used to have a built in local agent which would bypass our real agent if you didnt want to use one, however since we moved to java this is no longer possible so as of 4.0 an agent is required to do anything with a plugin.
We are coming out with a new agent in SC 4.1 which is built for scalability, the 4.0 agent is same agent we always had which works great but it is only single threaded. Meaning you can only do one backup at a time, if you get multiple parallel backups things wont work well. If this is case you probably should stay on SC 3.6P1 until SC 4.1 is available due to restriction in 4.0 of having the agent.
On any host that has access to vCenter. The scAgent checks if the port is open so not sure what is happening there but all the agent setup does is allow you to accept license and configures start script.
You can touch a file called "LICENSE" in scAgent root dir and then update path in bin/scAgent start script. The you should be able to start agent. This is workaround if you arent able to figure out this issue.
The issue I am thinking is the plugin makes some ZAPI calls to storage so the agent/plugin must be installed somewhere where that can happen. However this is no different than previous versions of SC. The only difference is the agent is running as a daemon, that is it. The agent can be installed on same server as SC server.