Typically within a business an instance of Splunk would be indexing varied forms of data at copious volumes. A few examples would be Windows registry, event logs, application web logs, Linux configuration syslog, application web logs, and database audits.
Forwarding all possible logs into Splunk can be hugely beneficial towards visibility, however in some cases a user may not be interested in particular logs, and may only want to index specific logs. For instance a common scenario would be based around compliance when recording Windows Security events of which a Splunk administrator may only be interested in logging and reporting user log-on and/or log-off activity. Any other events may not be of interest/needed, which means filtering out these unwanted events would be favourable.
Within the back end of Splunk’s configurable depths, an administrator can modify two configuration files called props.conf and transforms.conf. This results in a way of filtering unwanted data before being indexed. This blog will provide an example of how to achieve pre-index filtering in Splunk with the use of props.conf and transforms.conf.
PROPS.CONF AND TRANSFORMS.CONF
Some of the most common uses for props.conf are as follows:
1. When experiencing multiline events, props.conf can be configured for linebreaking 2. Configuration to recognise timestamps 3. Create segmentation between events 4. A way of overriding the automated host and source type matching built into Splunk 5. Advanced regex overriding based on host and source type configuration 6. Renaming source types 7. Ability to anonymise particular types of data feed such as bank card details, etc 8. Re-routing of particular events when a user may have multiple indexes