Events sent to Splunk only from one node

[nfantinato]

Hello,

we have configured audit logs to be sent via syslog to a Splunk server using command:

cluster log-forwarding create -destination xx.xx.xx.xx -port 514 -protocol tcp-unencrypted -verify-server false -facility user

but it seems that logs are sent only from node 1 of the Netapp storage array. So node 2 always results as it is not sending anything to Splunk.

It is normal? I mean, are all logs normally sent only from node 1?

The storage is a FAS8200, Ontap version is 9.7P17

Thanks in advance for any information.

Regards.

[hmoubara]

Hello @nfantinato 

Can you run the below command and check if there is connection between the node the syslog server and also review history of events that were set to be forwarded to the server.

cluster::> event notification destination check -node <node-name> -destination-name <>

cluster::> event notification history show -node  <node-name> -destination-name <>

https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/Event_forwarding_to_a_Syslog_server

Thanks 

[nfantinato]

Hello @hmoubara,

thank you for your reply. Because we found a bit complex to set correct filters in event notification, instead of those commands you indicated we've run the following one for both nodes:
cluster log-forwarding statistics show -node <node_name> -destination 161.27.170.14 -port 514

and those statistics show no errors and all messages correctly sent. So everything on Netapp side seems to work well, but on Spunk side no logs arrive from node 2. And this beahaviour happens only for some storages, not all.

Is it better to use event notification command instead of cluster log-forwarding?

Thank you.

[hmoubara]

Hello @nfantinato 

Sorry i missed read your original post. You referring to audit logging which you would use cluster log-forwarding as the event notification if for ems logs that was generated by the system.

So regarding the issue receiving logs only from one node, is most likely since the cluster-mgmt lif live on the working node. Try moving the cluster-mgmt lif to node 2 and see if you are getting logs forwarded to the syslog server.

Thanks 

[nfantinato]

Hello @hmoubara,

thank you for your reply.

Unfortunately we are can't move LIFs or our monitoring tools get crazy. We tried to compare settings of two different storages, one sending logs from both nodes and the other sending logs only from node 1 and they seem the same.

Maybe with event notification commands, as you suggested initially, we can reach a deeper level of customization.