The syslog-ng 3.0 Administrator Guide

Document version: Tenth
Release date: January 8, 2010
Latest version available at the BalaBit Documentation Page

3.6. Filters

Filters perform log routing within syslog-ng: a message passes the filter if the filter expression is true for the particular message. If a log statement includes filters, the messages are sent to the destinations only if they pass all filters of the log path. For example, a filter can select only the messages originating from a particular host. Complex filters can be created using filter functions and logical boolean expressions.

To define a filter, add a filter statement to the syslog-ng configuration file using the following syntax:

filter <identifier> { expression; };

The expression may contain the following elements:

[Example] Example 3.33. A simple filter statement

The following filter statement selects the messages that contain the word deny and come from the host example.

filter demo_filter { host("example") and match("deny"); };

For the filter to have effect, include it in a log statement:

log demo_filteredlog{
        source(s1); source(s2); 
        filter(demo_filter);
        destination(d1); destination(d2); };

The host(), match(), and program() filter functions accept regular expressions as parameters.

filter demo_regexp_filter { host("system.*1") and match("deny"); };

To limit the scope of a filter function, add the value() option and set its scope with a macro. For example, to limit the scope of the match() filter to the text part of the message, use:

match("keyword" value("MESSAGE"))

The value() parameter accepts both built-in macros and user-defined ones created with a parser. Do not prefix the macros with the $ sign. For details on macros and parsers, see Section 3.7, “Templates and macros” and Section 3.8, “Parsing messages”.

[Note] Note

When a log statement includes multiple filter statements, syslog-ng sends a message to the destination only if all filters are true for the message. In other words, the filters are connected with the logical AND operator. In the following example, no message arrives to the destination, because the filters are exclusive (the hostname of a client cannot be example1 and example2 at the same time):

filter demo_filter1 { host("example1"); };
filter demo_filter2 { host("example2"); };

log demo_filteredlog{
	source(s1); source(s2); 
	filter(demo_filter1); filter(demo_filter2);
	destination(d1); destination(d2); };

To select the messages that come from either host example1 or example2, use a single filter expression:

filter demo_filter { host("example1") or host("example2"); };
                
log demo_filteredlog{
	source(s1); source(s2); 
	filter(demo_filter);
	destination(d1); destination(d2); };

Use the not operator to invert filters, for example, to select the messages that were not sent by host example1: 

filter demo_filter { not host("example1"); };

However, to select the messages that were not sent by host example1 or example2, you have to use the and operator (that's how boolean logic works):

filter demo_filter { not host("example1") and not host("example2"); };

Alternatively, you can use parentheses to avoid this confusion:

filter demo_filter { not (host("example1") or host("example2")); };

In the extended regular expressions, the characters ()[].*?+^$| are used as special symbols. Therefore, these characters have to be preceded with a backslash (\) if they are meant literally. For example, the \$40 expression matches the $40 string. Backslashes have to be escaped as well if they are meant literally. For example, the \\d expression matches the \d string.

[Tip] Tip

If you use single quotes in, you do not need to escape the backslash, e.g. match("\\.") is equivalent to match('\.').

By default, all regular expressions are case sensitive. To disable the case sensitivity of the expression, add the flags(ignore-case) option to the regular expression.

filter demo_regexp_insensitive { host("system" flags(ignore-case)); };

For details on regular expressions, see Section 8.8, “Regular expressions”.

[Note] Note

In regular expressions, the asterisk (*) character means 0, 1 or any number of the previous expression. For example, in the f*ilter expression the asterisk means 0 or more f letters. This expression matches for the following strings: ilter, filter, ffilter, etc. To achieve the wildcard functionality commonly represented by the asterisk character in other applications, use .* in your expressions, e.g., f.*ilter.

The level() filter can select messages corresponding to a single importance level, or a level-range. To select messages of a specific level, use the name of the level as a filter parameter, e.g., use the following to select warning messages:

level(warning)

To select a range of levels, include the beginning and the ending level in the filter, separated with two dots (..). For example, to select every message of error or higher level, use the following filter:

level(err..emerg)

Similarly, messages sent by a range of facilities can also be selected. Note that this is only possible when using the name of the facilities. It is not possible to select ranges the numerical codes of the facilities.

facility(local0..local5)

For a complete list of the available levels and facilities, see Section 8.4, “Filter functions”.

For a complete description on the above functions, see Section 8.4, “Filter functions”.

3.6.1. Optimizing regular expressions in filters

Some filter functions accept regular expressions as parameters. But evaluating general regular expressions puts a high load on the CPU, which can cause problems when the message traffic is very high. Often the regular expression can be replaced with simple filter functions and logical operators. Using simple filters and logical operators, the same effect can be achieved at a much lower CPU load.

[Example] Example 3.34. Optimizing regular expressions in filters

Suppose you need a filter that matches the following error message logged by the xntpd NTP daemon:

xntpd[1567]: time error -1159.777379 is too large (set clock manually);

The following filter uses regular expressions and matches every instance and variant of this message.

filter f_demo_regexp {
    program("demo_program") and
    match("time error .* is too large .* set clock manually"); };

Segmenting the match() part of this filter into separate match() functions greatly improves the performance of the filter. 

filter f_demo_optimized_regexp {
    program("demo_program") and
    match("time error") and 
    match("is too large") and 
    match("set clock manually"); };

© 2007-2008 BalaBit IT Security
Please send your comments or documentation bugs to: documentation@balabit.com

Prev  Up  Next
3.5. Log paths  Home | Contents  3.7. Templates and macros