ONTAP Discussions
ONTAP Discussions
We have 4 NetApp's, all version 9.16. These are government systems, so we have no way to upload logs or configs, we'll have to do this the hard way.
3 of the 4 simply will not send syslogs to the internal syslog server. As far as we can tell, the 1 NetApp that is working, is configured exactly the same way as the 3 that are not working.
We have been through several guides and posts on the Google machine and have come up empty on everything tried.
- We have a working filter, tested with the 'event filter test' command
- We have valid syslog destinations in IP address format (though, just to be 100% sure, we also have DNS configured and working)
- The syslog destinations have the correct filters applied
- We can generate a test event and see it in the ONTAP event log (we've been using monitor.volume.nearfull)
- Using the event history show -destination syslog_1 (syslog_1 being our defined dest) we see absolutely nothing
- This is confirmed with a tcpdump command on the syslog server itself seeing no packets
- It's as if the syslog service never gets notified that it needs to send a syslog
- We can ping and traceroute the syslog IP address (and even the DNS name) from the ONTAP CLI
At this point, we're down to a suggestion to login to the systemshell and reset notifiyd. We are, however, pretty nervous about doing so, and since 3 of our 4 devices don't work, it seems like this is not the right thing to be mucking with.
Does anyone have anything on this topic. syslog configurations are pretty darn simple, usually, and ONTAP9 doesn't really seem to be any different. Is there some obscure option, somewhere, that needs to be enabled or something?
Solved! See The Solution
Have you checked the network interface service policies on 3 clusters don't send to syslog server. Please with the working one. If syslog traffic is sent via management interface, then management interface must have right service policy applied and rules (FW) applied.
network interface show -fields service-policy -vserver <your_cluster>
Then check the returned sevice-policy has syslog-forwarding policy applied
network interface service-policy show -vserver <your_cluster>
Sample out put will be like this (includes syslog forwarding)
<your_cluster>
default-management management-core: 0.0.0.0/0
management-autosupport: 0.0.0.0/0
management-ssh: 0.0.0.0/0
management-https: 0.0.0.0/0
management-ems: 0.0.0.0/0
management-ntp-client: 0.0.0.0/0
management-dns-client: 0.0.0.0/0
management-ad-client: 0.0.0.0/0
management-ldap-client: 0.0.0.0/0
management-nis-client: 0.0.0.0/0
management-http: 0.0.0.0/0
backup-ndmp-control: 0.0.0.0/0
management-snmp-server: 0.0.0.0/0
management-ntp-server: 0.0.0.0/0
management-log-forwarding: 0.0.0.0/0
You can also test the notification destination using the command:
event notification destination check -destination-name syslog_1
You can also try to create a different destination (i.e email) and use it to test.
If using different destination works, then it will be connectivity issue between the storage and syslog server where you can use netapp tcpdump to debug.
The check -destination only works for tcp connections. I didn't specify in my post, but we are trying to use udp, though, I think we could do tcp if it comes to it.
Have you checked the network interface service policies on 3 clusters don't send to syslog server. Please with the working one. If syslog traffic is sent via management interface, then management interface must have right service policy applied and rules (FW) applied.
network interface show -fields service-policy -vserver <your_cluster>
Then check the returned sevice-policy has syslog-forwarding policy applied
network interface service-policy show -vserver <your_cluster>
Sample out put will be like this (includes syslog forwarding)
<your_cluster>
default-management management-core: 0.0.0.0/0
management-autosupport: 0.0.0.0/0
management-ssh: 0.0.0.0/0
management-https: 0.0.0.0/0
management-ems: 0.0.0.0/0
management-ntp-client: 0.0.0.0/0
management-dns-client: 0.0.0.0/0
management-ad-client: 0.0.0.0/0
management-ldap-client: 0.0.0.0/0
management-nis-client: 0.0.0.0/0
management-http: 0.0.0.0/0
backup-ndmp-control: 0.0.0.0/0
management-snmp-server: 0.0.0.0/0
management-ntp-server: 0.0.0.0/0
management-log-forwarding: 0.0.0.0/0
Yes, thank you - this is definitely the problem. We had a rouge service-policy in there (for reasons I wish I could tell you) and changing that to the default-management, which is 0.0.0.0/0 on everything, and I immediately start seeing traffic on the tcpdump on the syslog server.
Now, I can't get syslog to actually log the traffic, but that's another problem - packets are now getting there so thank you very much.