could you implent a special receive-connector, set the scope to your SM-Host and allow anomyous authentication. If you just want to send notifications to your home-domains ther´s no need to enable the open-relay-option for that host.
In order to determine what the exact error is you will need to check the SMSP Logs.
The manager log is at: C:\Program Files\NetApp\SnapManager for SharePoint\VaultServer\Control\logs
And the media log is at: C:\Program Files\NetApp\SnapManager for SharePoint\VaultServer\Media\logs
And agent log is written to windows event.
Since you are having problems with email notification setup, you need check the “control high level.log” in manager log folder. If you check this log there should be additional details to help troubleshoot.