Classification

The syslog-ng application can compare the contents of the log messages to a database of predefined message patterns. This can be used for many different tasks:

• real-time log message classification (like identifying the message)
• extracting important information from messages (for example, usernames, IP addresses not included in the message header)
• real-time event correlation (create a single message from multiple related log messages)

Using patterndb™ also needs some patterns. These can easily be created based on:

• the guidelines in The syslog-ng Administrator Guide
• various blog posts
• samples from the patterndb™ project

Real-time log message classification

By comparing log messages to known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and for example can label the messages as user login, application crash, file transfer, etc. events.

In addition to classifying messages, you can also add different tags which can be used later for filtering messages, for example, to collect messages tagged as user_login to a separate file or to perform conditional post processing on the tagged messages.

The classification functionality of the pattern database was originally inspired by the logcheck project, but the syslog-ng aproach has the following advantages:

• The syslog-ng patterns are much easier to write and maintain than the regular expressions used by logcheck.
• It is much easier to understand syslog-ng pattens than regular expressions.
• Pattern matching based on regular expressions is computationally very intensive, especially when the number of patterns increases. The solution used by syslog-ng can be performed real-time, and is independent from the number of patterns, so it scales much better.

To find the pattern that matches a particular message, syslog-ng uses a method called longest prefix match radix tree. This means that syslog-ng creates a tree structure of the available patterns, where the different characters available in the patterns for a given position are the branches of the tree.

Examples

The following patterns describe the same message:
Accepted password for bazsi from 10.50.0.247 port 42156 ssh2

A regular expression matching this message from the logcheck project:
Accepted \ (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam) \ for [^[:space:]]+ from [^[:space:]]+ port [0-9]+( (ssh|ssh2))?

A syslog-ng database pattern for this message:
Accepted @QSTRING:auth_method: @ for@QSTRING:username: @from\ @QSTRING:client_addr: @port @NUMBER:port:@ ssh2