The syslog-ng 3.0 Administrator Guide

Document version: Tenth
Release date: January 8, 2010
Latest version available at the BalaBit Documentation Page

3.5. Log paths

Log paths determine what happens with the incoming log messages. Messages coming from the sources listed in the log statement and matching all the filters are sent to the listed destinations.

To define a log path, add a log statement to the syslog-ng configuration file using the following syntax:

log {
    source(s1); source(s2); ... 
    optional_element(filter1|parser1|rewrite1); optional_element(filter2|parser2|rewrite2);... 
    destination(d1); destination(d2); ... 
    flags(flag1[, flag2...]);
    };
[Warning] Warning

Log statements are processed in the order they appear in the configuration file, thus the order of log paths may influence what happens to a message, especially when using filters and log flags.

[Example] Example 3.29. A simple log statement

The following log statement sends all messages arriving to the localhost to a remote server.

source s_localhost { tcp(ip(127.0.0.1) port(1999) ); };
destination d_tcp { tcp("10.1.2.3" port(1999); localport(999)); };
log { source(s_localhost); destination(d_tcp); };

All matching log statements are processed by default, and the messages are sent to every matching destination by default. So a single log message might be sent to the same destination several times, provided the destination is listed in several log statements, and it can be also sent to several different destinations.

This default behavior can be changed using the flags() parameter. Flags apply to individual log paths; they are not global options. The following flags available in syslog-ng:

[Warning] Warning

The final, fallback, and catchall flags apply only for the top-level log paths, they have no effect on embedded log paths.

[Example] Example 3.30. Using log path flags

Let's suppose that you have two hosts (myhost_A and myhost_B) that run two applications each (application_A and application_B), and you collect the log messages to a central syslog-ng server. On the server, you create two log paths:

  • one that processes only the messages sent by myhost_A; and

  • one that processes only the messages sent by application_A.

This means that messages sent by application_A running on myhost_A will be processed by both log paths, and the messages of application_B running on myhost_B will not be processed at all.

  • If you add the final flag to the first log path, then only this log path will process the messages of myhost_A, so the second log path will receive only the messages of application_A running on myhost_B.

  • If you create a third log path that includes the fallback flag, it will process the messages not processed by the first two log paths, in this case, the messages of application_B running on myhost_B.

  • Adding a fourth log path with the catchall flag would process every message received by the syslog-ng server.

    log { source(s_localhost); destination(d_file); flags(catchall); };

For details on the individual flags, see Section 8.3, “Log path flags”. The effect and use of the flow-control flag is detailed in Section 2.13, “Managing incoming and outgoing messages with flow-control”.

3.5.1. Using embedded log statements

Embedded log statements (see Section 2.2.1, “Embedded log statements” ) re-use the results of processing messages (e.g., the results of filtering or rewriting) to create complex log paths. Embedded log statements use the same syntax as regular log statements, but they cannot contain additional sources. To define embedded log statements, use the following syntax:

log {
    source(s1); source(s2); ... 
    
    optional_element(filter1|parser1|rewrite1); 
    optional_element(filter2|parser2|rewrite2);... 
    
    destination(d1); destination(d2); ... 
    
    flags(flag1[, flag2...]); 
    
    #embedded log statement
    log
        {
        optional_element(filter1|parser1|rewrite1); 
        optional_element(filter2|parser2|rewrite2);... 
        destination(d1); destination(d2); ...
        #another embedded log statement
        log
        {
            optional_element(filter1|parser1|rewrite1); 
            optional_element(filter2|parser2|rewrite2);... 
            destination(d1); destination(d2); ...};
        };    
        };
[Warning] Warning

The final, fallback, and catchall flags apply only for the top-level log paths, they have no effect on embedded log paths.

[Example] Example 3.31. Using embedded log paths

The following log path sends every message to the d_file1 and the d_file2 destinations.

log { source(s_localhost); destination(d_file1); destination(d_file2); };

The next example is equivalent with the one above, but uses an embedded log statement.

log { source(s_localhost); destination(d_file1); 
                    log {destination(d_file2); };
};

The following example sends every message coming from the host 192.168.1.1 into the d_file1 destination, and sends every message coming from the host 192.168.1.1 and containing the string example into the d_file2 destination.

log { source(s_localhost); host(192.168.1.); destination(d_file1); 
                    log {message("example"); destination(d_file2); };
};

The following example collects logs from multiple source groups and uses the source() filter in the embedded log statement to select messages of the s_network source group.

log { source(s_localhost); source(s_network); destination(d_file1); 
                    log {source(s_network); destination(d_file2); };
                    };

3.5.2. Configuring flow-control

For details on how flow-control works, see Section 2.13, “Managing incoming and outgoing messages with flow-control”. The summary of the main points is as follows:

  • The syslog-ng application normally reads a maximum of log_fetch_limit() number of messages from a source.

  • From TCP and unix-stream sources, syslog-ng reads a maximum of log_fetch_limit() from every connection of the source. The number of connections to the source is set using the max_connections() parameter.

  • Every destination has an output buffer (log_fifo_size()).

  • Flow-control uses a control window to determine if there is free space in the output buffer for new messages. Every source has its own control window; log_iw_size() parameter sets the size of the control window.

  • When a source accepts multiple connections, the messages of every connection use the same control window.

  • The output buffer must be larger than the control window of every source that logs to the destination.

  • If the control window is full, syslog-ng stops reading messages from the source until some messages are successfully sent to the destination.

  • If the output buffer becomes full, and neither disk-buffering nor flow-control is used, messages may be lost.

[Note] Note

If you modify the max_connections() or the log_fetch_limit() parameter, do not forget to adjust the log_iw_size() and log_fifo_size() parameters accordingly.

[Example] Example 3.32. Sizing parameters for flow-control

Suppose that syslog-ng has a source that must accept up to 300 parallel connections. Such situation can arise when a network source receives connections from many clients, or if many applications log to the same socket. Therefore, set the max_connections() parameter of the source to 300. However, the log_fetch_limit() (default value: 10) parameter applies to every connection of the source individually, while the log_iw_size() (default value: 100) parameter applies to the source. In a worst-case scenario, the destination does not accept any messages, while all 300 connections send at least log_fetch_limit() number of messages to the source during every poll loop. Therefore, the control window must accommodate at least max_connections()*log_fetch_limit() messages to be able to read every incoming message of a poll loop. In the current example this means that (log_iw_size() should be greater than 300*10=3000. If the control window is smaller than this value, the control window might fill up with messages from the first connections — causing syslog-ng to read only one message of the last connections in every poll loop.

The output buffer of the destination must accommodate at least log_iw_size() messages, but use a greater value: in the current example 3000*10=30000 messages. That way all incoming messages of ten poll loops fit in the output buffer. If the output buffer is full, syslog-ng does not read any messages from the source until some messages are successfully sent to the destination.

source s_localhost { 
            tcp(ip(127.0.0.1) port(1999) max-connections(300)); };
            destination d_tcp { 
            tcp("10.1.2.3" port(1999); localport(999)); log_fifo_size(30000); };
            log { source(s_localhost); destination(d_tcp); flags(flow-control); };

If other sources send messages to this destination, than the output buffer must be further increased. For example, if a network host with maximum 100 connections also logs into the destination, than increase the log_fifo_size() by 10000.

source s_localhost { 
            tcp(ip(127.0.0.1) port(1999) max-connections(300)); };
            source s_tcp { 
            tcp(ip(192.168.1.5) port(1999) max-connections(100)); };
            destination d_tcp { 
            tcp("10.1.2.3" port(1999); localport(999)); log_fifo_size(40000); };
            log { source(s_localhost); destination(d_tcp); flags(flow-control); };

See also Section 7.2, “Handling lots of parallel connections”.

© 2007-2008 BalaBit IT Security
Please send your comments or documentation bugs to: documentation@balabit.com

Prev  Up  Next
3.4. Destinations and destination drivers  Home | Contents  3.6. Filters