Just to add some additional encounters of this error message ... and my solution.
In my case, the same error was occuring on a WFA 4.1rc1 (Win2012R2) system doing Credential TESTs against a set of clusters where some worked and some did not. The interesting thing was a side-by-side WFA 3.1p2 system could Credentail TEST against all those same systems just fine .. both ONTAP 8.3.2 and all ONTAP 9.0 systems also.
After reading Matt's comment above, I double checked my PTR records in DNS. It turned out that 2 of my ONTAP 9.0 systems DID NOT have reverse lookup PTR records (by accident) and those where the ones failing with the error. Adding the PTR records resolved the problem.
So it appears WFA 4.1rc1 is using a newer protocol option of SSL that requires and depends on PTR records for the clusters. I'm sure a security export could comment with further details ... but PTR reccords are always good to have 🙂 ... but I can imagine lots of lingering configurations out there that are working fine today but uncover this issue when upgrading to WFA 4.x ... caused by some bad DNS practices.
I experienced the same issue in a lab environment and found the root cause was a DNS\Name resolution issue (Not firewall related). My WFA server had multiple network interfaces and I had to ensure the NIC binding order was correct and that it was able to query the DNS server. Ensure you have A and PTR record for the credentials you are adding then try adding it using the FQDN rather than the IP Address.
If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.