Tracking Snapmirror Transfer Times - Forwarding Snapmirror Logs to a Central Log Correlation System (syslog Splunk)

[jurphville]

We utilize Splunk to aggregate logs accross our enterprise.  Currently my Netapps are using syslog to forward messages/console logs, but I would also like to get snapmirror logs.  Is this something that can be done via syslog?  Is there another way that someone is accomplishing this?

My current syslog.conf file is...

*.info        /dev/console

*.info        /etc/messages

*.info        @mySplunkServer

I'm most concerned w/ being able to track snapmirror transfer times to be sure we are within our data protection policies.  What other methods are people using to accomplish this?

[pascalduk]

I do not have actual hands on experience with splunk myself, but I am told that splunk can import (log) files. With this method you could import the /etc/log/snapmirror log file from the root volume. Problem will be the format of the snapmirror log file as each individual transfer has for the request, actual start and end a seperate entry. I doubt splunk can be used to analyze this.

An option would be to use the snapmirror log parser from the netapp utility toolchest (https://now.netapp.com/eservice/toolchest?toolid=422)  to summarize the data for you first and import the results in splunk.

[jurphville]

You are correct, Splunk can ingest log files, and that is an option, but I have been hesitant to do that because I don't want to introduce a process that could "mess" w/ the Netapps/Snapmirror.  Splunk *should* be able to read the log files w/o disruption, but I'd love to hear from anyone who is already doing that w/ success before implementing.  For me, the ideal scenario would be where the Netapp handles to the log forwarding through a built in mechanism, rather than a third party action.

To your point about multiple logs per transfer, Splunk actually has a built in mechanism for handling that.  I can give Splunk the parameters that define a transaction (Start="Request" End="End" by Filer:Vol) and Splunk will dynamically combine the multiple events into a single transaction and automatically calculate the duration of the transfer.  I can then pull out statistics, create graphs, and combine that into a dashboard.

I have seen the Snapmirror Log Parser, but that script requires that the logs are already local to the script, and doesn't address how the logs are pulled out of the Netapp.  In that case, I would assume a basic copy/paste.

[pascalduk]

There is unfortunaly no other option that read the snapmirror log file, but I am pretty sure that it will not be a problem. I have my own scripts processing netapp log files for years now without any problem.

jurphville wrote:
 To your point about multiple logs per transfer, Splunk actually has a built in mechanism for handling that.  I can give Splunk the parameters that define a transaction (Start="Request" End="End" by Filer:Vol) and Splunk will dynamically combine the multiple events into a single transaction and automatically calculate the duration of the transfer.  I can then pull out statistics, create graphs, and combine that into a dashboard.

That is very interesting as we probably implement splunk at my company next year and this will help me get rid off custom scripts to create the graphs of transfer times and amount transferred.

[aborzenkov]

There is unfortunaly no other option that read the snapmirror log file

Well ... there still exists SNMP and API to pull data from filer. Actually, API is pobably the only way to monitor current transfer progress, because log is written only after transfer is completed (basically, it corresponds to "last transfer" in snapmirror status) and SNMP does not seem to provide "progress" value.