The syslog-ng application can compare the contents of the log messages to a database of predefined message patterns. This can be used for many different tasks:
Using patterndb™ also needs some patterns. These can easily be created based on:
By comparing log messages to known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and for example can label the messages as user login, application crash, file transfer, etc. events.
In addition to classifying messages, you can also add different tags which can be used later for filtering messages, for example, to collect messages tagged as user_login to a separate file or to perform conditional post processing on the tagged messages.
The classification functionality of the pattern database was originally inspired by the logcheck project, but the syslog-ng aproach has the following advantages:
To find the pattern that matches a particular message, syslog-ng uses a method called longest prefix match radix tree. This means that syslog-ng creates a tree structure of the available patterns, where the different characters available in the patterns for a given position are the branches of the tree.
The following patterns describe the same message:
Accepted password for bazsi from 10.50.0.247 port 42156 ssh2
A regular expression matching this message from the logcheck project:
Accepted \ (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam) \ for [^[:space:]]+ from [^[:space:]]+ port [0-9]+( (ssh|ssh2))?
A syslog-ng database pattern for this message:
Accepted @QSTRING:auth_method: @ for@QSTRING:username: @from\ @QSTRING:client_addr: @port @NUMBER:port:@ ssh2
Using patterns one can also extract important information and create name value pairs from data found in log messages. These can be used for many different tasks: removing sensitive information from log files, create files or database tables dynamically, etc.
Name value pairs can also help to standardize log information. For example IP addresses can fulfill many different functions in log messages: it could be the IP address of the sender host, or the source or destination of an action. By using standardized field names for IP addresses, one could easily find all connections originating from a given IP address or all connection attempts to a botnets C&C machine.
Balabit joined CEE as a board member, which was working on a common taxonomy and dictionary for log messages. This effort is now in hibernation, and still without an extensive list of field names to be used in patterns. While there is still hope, that the CEE effort will be relived, for now we suggest to use common sense when giving field names.
In the previous example, “username” is the name, which will receive the value of the authenticated username used in the given ssh session. This value can be used to create a separate log file for each user, can be rewritten to anonymize logs, tag administrative users differently, etc.
Recent versions of syslog-ng also make real time event correlation possible. This can be useful in many different situations. For example important data for a single event is often scattered into multiple syslog messages. Also login and logout events are often logged far away from each other, even in different log files, making log analysis difficult. Using correlation these can be collected into a single new message.
For details check the in depth article at http://lwn.net/Articles/424459/ or the documentation.
There are many docs and blogs helping to write patterns for syslog-ng. First of all there is detailed documentation:
And there are also many related blogs:
A good way of starting to write patterns is to look at documentation and blogs and patterns in the Balabit patterndb™ repository at https://github.com/balabit/syslog-ng-patterndb Most of these patterns are coming from Balabit syslog-ng integration work, some of them are syslog-ng user contributions. You can also contribute by sending patterns or fixes for existing patterns to the syslog-ng mailing list ( https://lists.balabit.hu/mailman/listinfo/syslog-ng ).