The syslog-ng 3.0 Administrator Guide

Document version: Tenth
Release date: January 8, 2010
Latest version available at the BalaBit Documentation Page

3.9. Classifying messages

To classify messages using a pattern database, include a db_parser() statement in your syslog-ng configuration file using the following syntax:

Declaration:
        parser <identifier> {db_parser(file("<database_filename>"));};

Note that using the parser in a log statement only performs the classification, but does not automatically do anything with the results of the classification.

[Example] Example 3.39. Defining pattern databases

The following statement uses the database located at /opt/syslog-ng/var/db/patterndb.xml.

parser pattern_db {
            db_parser(
                file("/opt/syslog-ng/var/db/patterndb.xml")
            );
            };

To apply the patterns on the incoming messages, include the parser in a log statement:

log { 
            source(s_all);
            parser(pattern_db);            
            destination( di_messages_class);
            };
[Note] Note

The default location of the pattern database file is /opt/syslog-ng/var/run/patterndb.xml. The file option of the db-parser statement can be used to specify a different file, thus different db-parser statements can use different pattern databases. Later versions of syslog-ng will be able to dynamically generate a main database from separate pattern database files.

[Example] Example 3.40. Using classification results

The following destination separates the log messages into different files based on the class assigned to the pattern that matches the message (e.g., Violation and Security type messages are stored in a separate file), and also adds the ID of the matching rule to the message: 

destination di_messages_class {
        file("/var/log/messages-${.classifier.class}"       
        template("${.classifier.rule_id};${S_UNIXTIME};${SOURCEIP};${HOST};${PROGRAM};${PID};${MSG}\n")
        template_escape(no)
    );
};

Sample pattern databases are available at the BalaBit Download page http://www.balabit.com/network-security/syslog-ng/log-server-appliance/. However, these are not directly usable in syslog-ng 3.0.x, because they are formatted according to the second version (V2) of the pattern database format, which is supported only by the syslog-ng Store Box (SSB) appliance version 1.0.x. The syslog-ng 3.0.x OSE and PE applications only support the first version (V1) of the pattern database; support for the V2 and V3 pattern database formats will be available in syslog-ng 3.1. In the meantime, you can create your own pattern database: see Section 8.6.2.3, “Creating pattern databases” for details.

3.9.1. Using parser results in filters and templates

3.9.1.1. Filtering messages based on classification

The results of message classification and parsing can be used in custom filters and file and database templates as well. There are two built-in macros in syslog-ng that allow you to use the results of the classification: the .classifier.class macro contains the class assigned to the message (e.g., violation, security, or unknown), while the .classifier.rule_id macro contains the identifier of the message pattern that matched the message.

[Example] Example 3.41. Using classification results for filtering messages

To filter on a specific message class, create a filter that checks the .classifier_class macro, and use this filter in a log statement.

filter fi_class_violation {
                        match("violation"
                        value(".classifier.class")
                        type("string")
                        );
                        };
log { 
                        source(s_all);
                        parser(pattern_db);
                        filter(fi_class_violation);
                        destination(di_class_violation);
                        };

Filtering on the unknown class selects messages that did not match any rule of the pattern database. Routing these messages into a separate file allows you to periodically review new or unknown messages.

To filter on messages matching a specific classification rule, create a filter that checks the .classifier_rule_id macro. The unique identifier of the rule (e.g., e1e9c0d8-13bb-11de-8293-000c2922ed0a) is the id attribute of the rule in the XML database.

filter fi_class_rule {
                        match("e1e9c0d8-13bb-11de-8293-000c2922ed0a"
                        value(".classifier_rule_id")
                        type("string")
                        );
                        };

The message-segments parsed by the pattern parsers can also be used as macros as well. To accomplish this, you have to add a name to the parser, and then you can use this name as a macro that refers to the parsed value of the message.

[Example] Example 3.42. Using pattern parsers as macros

For example, you want to parse messages of an application that look like "Transaction: <type>.", where <type> is a string that has different values (e.g., refused, accepted, incomplete, etc.). To parse these messages, you can use the following pattern:

'Transaction: @ESTRING::.@'

Here the @ESTRING@ parser parses the message until the next full stop character. To use the results in a filter or a filename template, include a name in the parser of the pattern, e.g.:

'Transaction: @ESTRING:TRANSACTIONTYPE:.@'

After that, add a custom template to the logpath that uses this template. For example, to select every accepted transaction, use the following custom filter in the log path:

match("accepted" value("TRANSACTIONTYPE"));
[Note] Note

The above macros can be used in database columns and filename templates as well, if you create custom templates for the destination or logspace.

Use a consistent naming scheme for your macros, for example, APPLICATIONNAME_MACRONAME.

© 2007-2008 BalaBit IT Security
Please send your comments or documentation bugs to: documentation@balabit.com

Prev  Up  Next
3.8. Parsing messages  Home | Contents  3.10. Rewriting messages