I'm raising this in hope that we can resolve an issue that we have been facing for about a year now. We have had an open case with Microsoft regarding this but have so far been unsuccessful in resolving our issue.
We have numerous Windows Servers running Server 2012/2016 and are in a Windows Failover Cluster. The witness for these clusters is a fileshare on the NetApp. The clusters all appear to be working as normal. They failover the roles perfectly and we dont have any issues in connectivity between the hosts or witness etc. These are live clusters and have been in place for a few years without any issues.
We also are using Microsoft System Center Operations Manager 2019 which is monitoring the health of these systems and cluster state. - This is what first raised our issue.
For some reason on each of the cluster nodes in the SMBWitnessClient/WitnessClientAdmin event log we are getting the following errors multiple times per hour
"Witness Client received error (The remote procedure call failed.) from witness server CLUSTERNODENAME for NetName \\CLUSTERRESOURCE"
Now it does this multiple times per hour for each and every cluster resource on every cluster node. Each time it happens it creates an alert in SCOM notifying us of this. I know it says it receieved an error from the Witness Server (Which is a fileshare on the netapp) however we do also get events in the log to say
"Witness Client Successfully registered with Witness Server CLUSTERNODENAME for notification on NetName \\CLUSTERRESOURCE\ with IP Address X.X.X.X" (IP Address of the cluster resource.
Has anyone come across this before or seen anything like this? I believe this is something more to do with the NetApp than a Windows issue. It doesn't appear to be causing any issues however it is becoming increasingly frustrating.
Witness works by using remote procedure calls (RPCs) instead of waiting for Transmission Control Protocol (TCP) system calls. When an SMB 3 client connects to a CA share, witness informs the client where it can find the same data in replicated CA shares on other nodes. The SMB 3 client registers this information, and, should a failure occur, rather than waiting for a network timeout to occur, witness notifies the client of the failure and initiates a transparent failover to quickly establish a new session on the alternate node.
Witness is enabled by default and an essential requirement for continuously available shares.
Note: ONTAP does not support witness for use cases other than Hyper-V over SMB and SQL over SMB.