The host service package is out of support for a long time.
However, it is a bit more than only having a valid certificate installed on the UM server. After creating the certificate for the dfm server you need to do the same for dfm service.
On the Vcenter where you have installed the host package you need to execute the script which is in the folder and afterwards disable authentication for the host service in PowerShell. Then you add the host service. I cannot say which step exactly you missed or if it is not compatible anymore.
Since host service is used at own risk I would consider alternatives.
The host agent is not for Vcenter. It is for Monitoring Hosts directly or use them for OSSV.
For adding something to the vCenter you would need the OnCommand Host Package.
As I already mentioned, it is EOS, needs WS2008 and up to ESX 5.1
OnCommand® Unified Manager Host Package 1.3 for Windows® - Description
This page contains an overview of OnCommand Unified Manager Host Package 1.3 and its installation requirements for Windows. The Host Package supports the monitoring, backup, and recovery of VMware® virtual objects. This release does not include any GUI components.
When you are ready to proceed with the installation of OnCommand Host Package 1.3, click the link at the bottom of the page to go to the software download page.
Important: If you are installing both the Core Package and the Host Package, you must install the Core Package software before installing the Host Package software. After you install both packages, you must configure a host service with the DataFabric® Manager server before you can create your backup jobs. See theOnCommand Unified Manager Installation and Setup Guidefor instructions.