The syslog-ng Administrator Guide

This section describes the internal message-processing model of syslog-ng, as well as the flow-control feature that can prevent message losses. To use flow-control, the flow-control flag must be enabled for the particular log path.

The syslog-ng application monitors (polls) the sources defined in its configuration file, periodically checking each source for messages. When a log message is found in one of the sources, syslog-ng polls every source and reads the available messages. These messages are processed and put into the output buffer of syslog-ng (also called fifo). From the output buffer, the operating system sends the messages to the appropriate destinations.

In large-traffic environments many messages can arrive during a single poll loop, therefore syslog-ng reads only a fixed number of messages from each source. The log_fetch_limit() option specifies the number of messages read during a poll loop from a single source.

Figure 8.1. Managing log messages in syslog-ng

	Note
	The log_fetch_limit() parameter can be set as a global option, or for every source individually.

Every destination has its own output buffer. The output buffer is needed because the destination might not be able to accept all messages immediately. The log_fifo_size() parameter sets the size of the output buffer. The output buffer must be larger than the log_fetch_limit() of the sources, to ensure that every message read during the poll loop fits into the output buffer. If the log path sends messages to a destination from multiple sources, the output buffer must be large enough to store the incoming messages of every source.

TCP and unix-stream sources can receive the logs from several incoming connections (e.g., many different clients or applications). For such sources, syslog-ng reads messages from every connection, thus the log_fetch_limit() parameter applies individually to every connection of the source.

The flow-control of syslog-ng introduces a control window to the source that tracks how many messages can syslog-ng accept from the source. Every message that syslog-ng reads from the source lowers the window size by one; every message that syslog-ng successfully sends from the output buffer increases the window size by one. If the window is full (i.e., its size decreases to zero), syslog-ng stops reading messages from the source. The initial size of the control window is by default 100: the log_fifo_size() must be larger than this value in order for flow-control to have any effect. If a source accepts messages from multiple connections, all messages use the same control window.

When flow-control is used, every source has its own control window. As a worst-case situation, the output buffer of the destination must be set to accommodate all messages of every control window, that is, the log_fifo_size() of the destination must be greater than number_of_sources*log_iw_size(). This applies to every source that sends logs to the particular destination, thus if two sources having several connections and heavy traffic send logs to the same destination, the control window of every connection of both sources must fit into the output buffer of the destination. Otherwise, syslog-ng does not activate the flow-control, and messages may be lost.

The summary of the main points is as follows:

	Note
	If you modify the max_connections() or the log_fetch_limit() parameter, do not forget to adjust the log_iw_size() and log_fifo_size() parameters accordingly.

Figure 8.2. Managing log messages of TCP sources in syslog-ng

Example 8.1. Sizing parameters for flow-control

Suppose that syslog-ng has a source that must accept up to 300 parallel connections. Such situation can arise when a network source receives connections from many clients, or if many applications log to the same socket. Therefore, set the max_connections() parameter of the source to 300. However, the log_fetch_limit() (default value: 10) parameter applies to every connection of the source individually, while the log_iw_size() (default value: 100) parameter applies to the source. In a worst-case scenario, the destination does not accept any messages, while all 300 connections send at least log_fetch_limit() number of messages to the source during every poll loop. Therefore, the control window must accommodate at least max_connections()*log_fetch_limit() messages to be able to read every incoming message of a poll loop. In the current example this means that (log_iw_size() should be greater than 300*10=3000. If the control window is smaller than this value, the control window might fill up with messages from the first connections — causing syslog-ng to read only one message of the last connections in every poll loop. The output buffer of the destination must accommodate at least log_iw_size() messages, but use a greater value: in the current example 3000*10=30000 messages. That way all incoming messages of ten poll loops fit in the output buffer. If the output buffer is full, syslog-ng does not read any messages from the source until some messages are successfully sent to the destination. source s_localhost { tcp(ip(127.0.0.1) port(1999) max-connections(300)); }; destination d_tcp { tcp("10.1.2.3" port(1999); localport(999)); log_fifo_size(30000); }; log { source(s_localhost); destination(d_tcp); flags(flow-control); }; If other sources send messages to this destination, than the output buffer must be further increased. For example, if a network host with maximum 100 connections also logs into the destination, than increase the log_fifo_size() by 10000. source s_localhost { tcp(ip(127.0.0.1) port(1999) max-connections(300)); }; source s_tcp { tcp(ip(192.168.1.5) port(1999) max-connections(100)); }; destination d_tcp { tcp("10.1.2.3" port(1999); localport(999)); log_fifo_size(40000); }; log { source(s_localhost); destination(d_tcp); flags(flow-control); };

See also Section 8.1, “Handling lots of parallel connections”.

8.3.1. Flow-control and multiple destinations

Using flow-control on a source has an important side-effect if the messages of the source are sent to multiple destinations. If flow-control is in use and one of the destinations cannot accept the messages, the other destinations do not receive any messages either, because syslog-ng stops reading the source. For example, if messages from a source are sent to a remote server and also stored locally in a file, and the network connection to the server becomes unavailable, neither the remote server nor the local file will receive any messages. This side-effect of the flow-control can be avoided by using the disk-based buffering feature of syslog-ng Premium Edition.

	Note
	Creating separate log paths for the destinations that use the same flow-controlled source does not avoid the problem.