The syslog-ng Open Source Edition 3.1 Administrator Guide

The syslog-ng application is a flexible and highly scalable system logging application that is ideal for creating centralized and trusted logging solutions. The main features of syslog-ng are summarized below.

The syslog-ng application is not log analysis software. It can filter log messages and select only the ones matching certain criteria. It can even convert the messages and restructure them to a predefined format, or parse the messages and segment them into different fields. But syslog-ng cannot interpret and analyze the meaning behind the messages, or recognize patterns in the occurrence of different messages.

Log messages contain information about the events happening on the hosts. Monitoring system events is essential for security and system health monitoring reasons.

The original syslog protocol separates messages based on the priority of the message and the facility sending the message. These two parameters alone are often inadequate to consistently classify messages, as many applications might use the same facility — and the facility itself is not even included in the log message. To make things worse, many log messages contain unimportant information. The syslog-ng application helps you to select only the really interesting messages, and forward them to a central server.

Company policies or other regulations often require log messages to be archived. Storing the important messages in a central location greatly simplifies this process.

For details on how can you use syslog-ng to comply with various regulations, see the Regulatory compliance and system logging whitepaper available at http://www.balabit.com/support/documentation/

Version 3.1 of syslog-ng Open Source Edition includes the following main features:

The syslog-ng application is used worldwide by companies and institutions who collect and manage the logs of several hosts, and want to store them in a centralized, organized way. Using syslog-ng is particularly advantageous for:

The following is a list of public references — companies who use syslog-ng in their production environment:

The syslog-ng Open Source Edition application is highly portable and is known to run on a wide range of hardware architectures (x86, x86_64, SUN Sparc, PowerPC 32 and 64, Alpha) and operating systems, including Linux, BSD, Solaris, IBM AIX, HP-UX, Mac OS X, Cygwin, Tru64, and others.

Typically, syslog-ng is used to manage log messages and implement centralized logging, where the aim is to collect the log messages of several devices on a single, central log server. The different devices — called syslog-ng clients — all run syslog-ng, and collect the log messages from the various applications, files, and other sources. The clients send all important log messages to the remote syslog-ng server, where the server sorts and stores them.

The syslog-ng application reads incoming messages and forwards them to the selected destinations. The syslog-ng application can receive messages from files, remote hosts, and other sources.

Log messages enter syslog-ng in one of the defined sources, and are sent to one or more destinations.

Sources and destinations are independent objects; log paths define what syslog-ng does with a message, connecting the sources to the destinations. A log path consists of one or more sources and one or more destinations; messages arriving to a source are sent to every destination listed in the log path. A log path defined in syslog-ng is called a log statement.

Optionally, log paths can include filters. Filters are rules that select only certain messages, for example, selecting only messages sent by a specific application. If a log path includes filters, syslog-ng sends only the messages satisfying the filter rules to the destinations set in the log path.

Other optional elements that can appear in log statements are parsers and rewriting rules. Parsers segment messages into different fields to help processing the messages, while rewrite rules modify the messages by adding, replacing, or removing parts of the messages.

The following procedure illustrates the route of a log message from its source on the syslog-ng client to its final destination on the central syslog-ng server.

2.2.1. Procedure – The route of a log message in syslog-ng

Figure 2.1. The route of a log message

1. A device or application sends a log message to a source on the syslog-ng client. For example, an Apache web server running on Linux enters a message into the /var/log/apache file.

2. The syslog-ng client running on the web server reads the message from its /var/log/apache source.

3. The syslog-ng client processes the first log statement that includes the /var/log/apache source.

4. The syslog-ng client performs optional operations (message filtering, parsing, and rewriting) on the message; for example, it compares the message to the filters of the log statement (if any). If the message complies with all filter rules, syslog-ng sends the message to the destinations set in the log statement, for example, to the remote syslog-ng server.

   	Warning
   	Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.

   	Note
   	The syslog-ng client sends a message to all matching destinations by default. As a result, a message may be sent to a destination more than once, if the destination is used in multiple log statements. To prevent such situations, use the final flag in the destination statements. See Table 6.1, Log statement flags for details.

5. The syslog-ng client processes the next log statement that includes the /var/log/apache source, repeating Steps 3-4.

6. The message sent by the syslog-ng client arrives to a source set in the syslog-ng server.

7. The syslog-ng server reads the message from its source and processes the first log statement that includes that source.

8. The syslog-ng server performs optional operations (message filtering, parsing, and rewriting) on the message; for example, it compares the message to the filters of the log statement (if any). If the message complies with all filter rules, syslog-ng sends the message to the destinations set in the log statement.

   	Warning
   	Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.

9. The syslog-ng server processes the next log statement, repeating Steps 7-9.

	Note
	The syslog-ng application can stop reading messages from its sources if the destinations cannot process the sent messages. This feature is called flow-control and is detailed in Section 2.12, Managing incoming and outgoing messages with flow-control.

2.2.2. Embedded log statements

Starting from version 3.0, syslog-ng can handle embedded log statements (also called log pipes). Embedded log statements are useful for creating complex, multi-level log paths with several destinations and use filters, parsers, and rewrite rules.

For example, if you want to filter your incoming messages based on the facility parameter, and then use further filters to send messages arriving from different hosts to different destinations, you would use embedded log statements.

Figure 2.2. Embedded log statement

Embedded log statements include sources — and usually filters, parsers, rewrite rules, or destinations — and other log statements that can include filters, parsers, rewrite rules, and destinations. The following rules apply to embedded log statements:

• Only the beginning (also called top-level) log statement can include sources.

• Embedded log statements can include multiple log statements on the same level (i.e., a top-level log statement can include two or more log statements).

• Embedded log statements can include several levels of log statements (i.e., a top-level log statement can include a log statement that includes another log statement, and so on).

• Only another log statement can follow an embedded log statement, filters or other rules cannot.

• Embedded log statements that are on the same level receive the same messages from the higher-level log statement. For example, if the top-level log statement includes a filter, the lower-level log statements receive only the messages that pass the filter.

Figure 2.3. Embedded log statements

Embedded log filters can be used to optimize the processing of log messages, for example, to re-use the results of filtering and rewriting operations.

The syslog-ng Open Source Edition application has three typical operation scenarios: Client, Server, and Relay.

2.3.1. Client mode

Figure 2.4. Client-mode operation

In client mode, syslog-ng collects the local logs generated by the host and forwards them through a network connection to the central syslog-ng server or to a relay. Clients often also log the messages locally into files.

2.3.2. Relay mode

Figure 2.5. Relay-mode operation

In relay mode, syslog-ng receives logs through the network from syslog-ng clients and forwards them to the central syslog-ng server using a network connection. Relays also log the messages from the relay host into a local file, or forward these messages to the central syslog-ng server.

2.3.3. Server mode

Figure 2.6. Server-mode operation

In server mode, syslog-ng acts as a central log-collecting server. It receives messages from syslog-ng clients and relays over the network, and stores them locally in files, or passes them to other applications, e.g., log analyzers.

The syslog-ng application uses the following objects:

For details on the above objects, see Section 4.2, Defining global objects.

The syslog-ng application supports messages originating from different timezones. The original syslog protocol does not include timezone information, but syslog-ng provides a solution by extending the syslog protocol to include the timezone in the log messages. The syslog-ng application also enables administrators to supply timezone information for legacy devices which do not support the protocol extension.

Timezone information is associated with messages entering syslog-ng is selected using the following algorithm:

The syslog-ng application receives the timezone and daylight saving information from the operating system it is installed on. If the operating system handles daylight saving correctly, so does syslog-ng.

The syslog-ng application can send and receive log messages securely over the network using the Transport Layer Security (TLS) protocol. TLS is an encryption protocol over the TCP/IP network protocol, so it can be used only with TCP-based sources and destinations ( tcp() and tcp6()).

TLS uses certificates to authenticate and encrypt the communication, as illustrated on the following figure:

Figure 2.7. Certificate-based authentication

The client authenticates the server by requesting its certificate and public key. Optionally, the server can also request a certificate from the client, thus mutual authentication is also possible.

In order to use TLS encryption in syslog-ng, the following elements are required:

When using mutual authentication to verify the identity of the clients, the following elements are required:

Mutual authentication ensures that the syslog-ng server accepts log messages only from authorized clients.

See Section 4.12, Encrypting log messages with TLS for details on configuring TLS communication in syslog-ng.

The syslog-ng application can dynamically create filenames, directories, or names of database tables using macros that help you organize your log messages. Macros refer to a property or a part of the log message, for example, the $HOST macro refers to the name or IP address of the client that sent the log message, while $DAY is the day of the month when syslog-ng has received the message. Using these macros in the path of the destination log files allows you for example to collect the logs of every host into separate files for every day.

A set of macros can be defined as a template object and used in multiple destinations.

Another use of macros and templates is to customize the format of the syslog message, for example to add elements of the message header to the message text. Note that if a message uses the IETF-syslog format, only the text of the message can be customized, the structure of the header is fixed.

For details on using templates and macros, see Section 4.7, Templates and macros and Section 6.5, Macros.

The filters and default macros of syslog-ng work well on the headers and metainformation of the log messages, but are rather limited when processing the content of the messages. Parsers can segment the content of the messages into name-value pairs, and these names can be used as user-defined macros. Subsequent filtering or other type of processing of the message can use these custom macros to refer to parts of the message.

Parsers are global objects most often used together with filters and rewrite rules. For details on using parsers, see Section 4.8, Parsing messages and Section 6.6, Message parsers.

The syslog-ng application can rewrite parts of the messages using rewrite rules. Rewrite rules are global objects similar to parsers and filters and can be used in log paths. The syslog-ng application has two methods to rewrite parts of the log messages: replacing (setting) a part of the message to a fix value, and a general search-and-replace mode.

Substitution completely replaces a specific part of the message that is referenced using a built-in or user-defined macro.

General rewriting searches for a string in the entire message (or only a part of the message specified by a macro) and replaces it with another string. Optionally, this replacement string can be a template that contains macros.

For details on using rewrite rules, see Section 4.10, Rewriting messages and Section 6.7, Rewriting messages.

The syslog-ng application can compare the contents of the received log messages to predefined message patterns. By comparing the messages to the known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and for example can label the messages as user login, application crash, file transfer, etc. events.

To find the pattern that matches a particular message, syslog-ng uses a method called longest prefix match radix tree. This means that syslog-ng creates a tree structure of the available patterns, where the different characters available in the patterns for a given position are the branches of the tree.

To classify a message, syslog-ng selects the first character of the message (the text of message, not the header), and selects the patterns starting with this character, other patterns are ignored for the rest of the process. After that, the second character of the message is compared to the second character of the selected patterns. Again, matching patterns are selected, and the others discarded. This process is repeated until a single pattern completely matches the message, or no match is found. In the latter case, the message is classified as unknown, otherwise the class of the matching pattern is assigned to the message.

To make the message classification more flexible and robust, the patterns can contain pattern parsers: elements that match on a set of characters. For example, the NUMBER parser matches on any integer or hexadecimal number (for example 1, 123, 894054, 0xFFFF, etc.). Other pattern parsers match on various strings and IP addresses. For the details of available pattern parsers, see Section 6.6.2.1, Using pattern parsers.

The functionality of the pattern database is similar to that of the logcheck project, but it is much easier to write and maintain the patterns used by syslog-ng, than the regular expressions used by logcheck. Also, it is much easier to understand syslog-ng pattens than regular expressions.

Pattern matching based on regular expressions is computationally very intensive, especially when the number of patterns increases. The solution used by syslog-ng can be performed real-time, and is independent from the number of patterns, so it scales much better. The following patterns describe the same message: Accepted password for bazsi from 10.50.0.247 port 42156 ssh2

A regular expression matching this message from the logcheck project: Accepted \ (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam) \ for [^[:space:]]+ from [^[:space:]]+ port [0-9]+( (ssh|ssh2))?

A syslog-ng database pattern for this message: Accepted @QSTRING:auth_method: @ for@QSTRING:username: @from\ @QSTRING:client_addr: @port @NUMBER:port:@ ssh2

For details on using pattern databases to classify log messages, see Section 4.9, Classifying messages and Section 6.6.2, Pattern databases.

2.11.1. The structure of the pattern database

The pattern database is organized as follows:

Figure 2.8. The structure of the pattern database

• The pattern database consists of rulesets. A ruleset consists of a Program Pattern and a set of rules: the rules of a ruleset are applied to log messages if the name of the application that sent the message matches the Program Pattern of the ruleset. The name of the application (the content of the $PROGRAM macro) is compared to the Program Patterns of the available rulesets, and then the rules of the matching rulesets are applied to the message.

• The Program Pattern can be a string that specifies the name of the appliation or the beginning of its name (e.g., to match for sendmail, the program pattern can be sendmail, or just send), and the Program Pattern can contain pattern parsers. Note that pattern parsers are completely independent from the syslog-ng parsers used to segment messages. Additionally, every rule has a unique identifier: if a message matches a rule, the identifier of the rule is stored together with the message.

• Rules consist of a message pattern and a class. The Message Pattern is similar to the Program Pattern, but is applied to the message part of the log message (the content of the $MESSAGE macro). If a message pattern matches the message, the class of the rule is assigned to the message (e.g., Security, Violation, etc.).

• Rules can also contain additional information about the matching messages, such as the description of the rule, an URL, name-value pairs, or free-form tags.

• Patterns can consist of literals (keywords, or rather, keycharacters) and pattern parsers.

  	Note
  	If the $PROGRAM part of a message is empty, rules with an empty Program Pattern are used to classify the message. If the same Program Pattern is used in multiple rulesets, the rules of these rulesets are merged, and every rule is used to classify the message. Note that message patterns must be unique within the merged rulesets, but the currently only one ruleset is checked for uniqueness.

2.11.2. How pattern matching works

Figure 2.9. Applying patterns

The followings describe how patterns work. This information applies to program patterns and message patterns alike, even though message patterns are used to illustrate the procedure.

Patterns can consist of literals (keywords, or rather, keycharacters) and pattern parsers. Pattern parsers attempt to parse a sequence of characters according to certain rules.

	Note
	Wildcards and regular expressions cannot be used in patterns. The @ character must be escaped, i.e., to match for this character, you have to write @@ in your pattern. This is required because pattern parsers of syslog-ng are enclosed between @ characters.

When a new message arrives, syslog-ng attempts to classify it using the pattern database. The available patterns are organized alphabetically into a tree, and syslog-ng inspects the message character-by-character, starting from the beginning. This approach ensures that only a small subset of the rules must be evaluated at any given step, resulting in high processing speed. Note that the speed of classifying messages is practically independent from the total number of rules.

For example, if the message begins with the Apple string, only patterns beginning with the character A are considered. In the next step, syslog-ng selects the patterns that start with Ap, and so on, until there is no more specific pattern left.

Note that literal matches take precedence over pattern parser matches: if at a step there is a pattern that matches the next character with a literal, and another pattern that would match it with a parser, the pattern with the literal match is selected. Using the previous example, if at the third step there is the literal pattern Apport and a pattern parser Ap@STRING@, the Apport pattern is matched, even if the pattern parser would result in a better match.

If there are two parsers at the same level (e.g., Ap@STRING@ and Ap@QSTRING@), it is random which pattern is applied (technically, the one that is loaded first). However, if the selected parser cannot parse at least one character of the message, the other parser is used. But having two different parsers at the same level is extremely rare, so the impact of this limitation is much less than it appears.

This section describes the internal message-processing model of syslog-ng, as well as the flow-control feature that can prevent message losses. To use flow-control, the flow-control flag must be enabled for the particular log path.

The syslog-ng application monitors (polls) the sources defined in its configuration file, periodically checking each source for messages. When a log message is found in one of the sources, syslog-ng polls every source and reads the available messages. These messages are processed and put into the output buffer of syslog-ng (also called fifo). From the output buffer, the operating system sends the messages to the appropriate destinations.

In large-traffic environments many messages can arrive during a single poll loop, therefore syslog-ng reads only a fixed number of messages from each source. The log_fetch_limit() option specifies the number of messages read during a poll loop from a single source.

Figure 2.10. Managing log messages in syslog-ng

	Note
	The log_fetch_limit() parameter can be set as a global option, or for every source individually.

Every destination has its own output buffer. The output buffer is needed because the destination might not be able to accept all messages immediately. The log_fifo_size() parameter sets the size of the output buffer. The output buffer must be larger than the log_fetch_limit() of the sources, to ensure that every message read during the poll loop fits into the output buffer. If the log path sends messages to a destination from multiple sources, the output buffer must be large enough to store the incoming messages of every source.

TCP and unix-stream sources can receive the logs from several incoming connections (for example many different clients or applications). For such sources, syslog-ng reads messages from every connection, thus the log_fetch_limit() parameter applies individually to every connection of the source.

Figure 2.11. Managing log messages of TCP sources in syslog-ng

The flow-control of syslog-ng introduces a control window to the source that tracks how many messages can syslog-ng accept from the source. Every message that syslog-ng reads from the source lowers the window size by one; every message that syslog-ng successfully sends from the output buffer increases the window size by one. If the window is full (i.e., its size decreases to zero), syslog-ng stops reading messages from the source. The initial size of the control window is by default 100: the log_fifo_size() must be larger than this value in order for flow-control to have any effect. If a source accepts messages from multiple connections, all messages use the same control window.

When flow-control is used, every source has its own control window. As a worst-case situation, the output buffer of the destination must be set to accommodate all messages of every control window, that is, the log_fifo_size() of the destination must be greater than number_of_sources*log_iw_size(). This applies to every source that sends logs to the particular destination. Thus if two sources having several connections and heavy traffic send logs to the same destination, the control window of both sources must fit into the output buffer of the destination. Otherwise, syslog-ng does not activate the flow-control, and messages may be lost.

The syslog-ng application handles outgoing messages the following way:

Figure 2.12. Handling outgoing messages in syslog-ng PE

2.12.1. Flow-control and multiple destinations

Using flow-control on a source has an important side-effect if the messages of the source are sent to multiple destinations. If flow-control is in use and one of the destinations cannot accept the messages, the other destinations do not receive any messages either, because syslog-ng stops reading the source. For example, if messages from a source are sent to a remote server and also stored locally in a file, and the network connection to the server becomes unavailable, neither the remote server nor the local file will receive any messages.

	Note
	Creating separate log paths for the destinations that use the same flow-controlled source does not avoid the problem.

As of October 2009, the following release policy applies to syslog-ng Open Source Edition:

Note

Releases of the feature branch are tested just like the stable releases; they are not "unstable" development snapshots. The difference between earlier major releases and current feature releases is the smaller number of features contained in a release, and the shorter support periods. If an unstable snapshot or alpha/beta/rc release is released for public testing, it is always marked explicitly as such.

	Warning
	Downgrading from a feature release to an earlier (and thus unsupported) feature release, or to the stable release is officially not supported, but usually works as long as your syslog-ng OSE configuration file is appropriate for the old syslog-ng OSE version. However, persistent data like the position of the last processed message in a file source will be probably lost.

Multiple syslog-ng servers can be run in fail-over mode. The syslog-ng application does not include any internal support for this, as clustering support must be implemented on the operating system level. A tool that can be used to create UNIX clusters is Heartbeat (see http://www.linux-ha.org/ for details).

During the course of a message from the sending application to the final destination of the message, there are a number of locations where a message may be lost, even though syslog-ng does its best to avoid message loss. Usually losing messages can be avoided with careful planning and proper configuration of syslog-ng and the hosts running syslog-ng. The following list shows the possible locations where messages may be lost, and provides methods to minimize the risk of losing messages.

	Note
	The following list covers the main possibilities of losing messages, but does not take into account the possible use of flow-control (see Section 2.12, Managing incoming and outgoing messages with flow-control). This topic will be addressed in more detail in the future releases of this guide.

The following sections describe the structure of log messages. Currently there are two standard syslog message formats:

2.16.1. BSD-syslog or legacy-syslog messages

This section describes the format of a syslog message, according to the legacy-syslog or BSD-syslog protocol (see RFC 3164 http://www.ietf.org/rfc/rfc3164.txt). A syslog message consists of the following parts:

• PRI

• HEADER

• MSG

The total message cannot be longer than 1024 bytes.

The following is a sample syslog message: <133>Feb 25 14:09:07 webserver syslogd: restart. The message corresponds to the following format: <priority>timestamp hostname application: message. The different parts of the message are explained in the following sections.

	Note
	The syslog-ng application supports longer messages as well. For details, see the log_msg_size() option in Section 6.9, Global options. However, it is not recommended to enable messages larger than the packet size when using UDP destinations.

2.16.1.1. The PRI message part

The PRI part of the syslog message (known as Priority value) represents the Facility and Severity of the message. Facility represents the part of the system sending the message, while severity marks its importance. The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity. The possible facility and severity values are presented below.

	Note
	Facility codes may slightly vary between different platforms. The syslog-ng application accepts facility codes as numerical values as well.

Numerical Code	Facility
0	kernel messages
1	user-level messages
2	mail system
3	system daemons
4	security/authorization messages
5	messages generated internally by syslogd
6	line printer subsystem
7	network news subsystem
8	UUCP subsystem
9	clock daemon
10	security/authorization messages
11	FTP daemon
12	NTP subsystem
13	log audit
14	log alert
15	clock daemon
16-23	locally used facilities (local0-local7)

Table 2.1. syslog Message Facilities

The following table lists the severity values.

Numerical Code	Severity
0	Emergency: system is unusable
1	Alert: action must be taken immediately
2	Critical: critical conditions
3	Error: error conditions
4	Warning: warning conditions
5	Notice: normal but significant condition
6	Informational: informational messages
7	Debug: debug-level messages

Table 2.2. syslog Message Severities

2.16.1.2. The HEADER message part

The HEADER part contains a timestamp and the hostname (without the domain name) or the IP address of the device. The timestamp field is the local time in the Mmm dd hh:mm:ss format, where:

• Mmm is the English abbreviation of the month: Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec.

• dd is the day of the month on two digits. If the day of the month is less than 10, the first digit is replaced with a space. (For example Aug 7.)

• hh:mm:ss is the local time. The hour (hh) is represented in a 24-hour format. Valid entries are between 00 and 23, inclusive. The minute (mm) and second (ss) entries are between 00 and 59 inclusive.

	Note
	The syslog-ng application supports other timestamp formats as well, like ISO, or the PIX extended format. For details, see the ts_format() option in Section 6.9, Global options.

2.16.1.3. The MSG message part

The MSG part contains the name of the program or process that generated the message, and the text of the message itself. The MSG part is usually in the following format: program[pid]: message text.

2.16.2. IETF-syslog messages

This section describes the format of a syslog message, according to the IETF-syslog protocol (see RFC 5424-5428 http://tools.ietf.org/html/rfc5424).A syslog message consists of the following parts:

• HEADER (includes the PRI as well)

• STRUCTURED-DATA

• MSG

The following is a sample syslog message:^[1]

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

The message corresponds to the following format:

<priority>VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG

In this example, the Facility has the value of 4, severity is 2, so PRI is 34. The VERSION is 1. The message was created on 11 October 2003 at 10:14:15pm UTC, 3 milliseconds into the next second. The message originated from a host that identifies itself as "mymachine.example.com". The APP-NAME is "su" and the PROCID is unknown. The MSGID is "ID47". The MSG is "'su root' failed for lonvick...", encoded in UTF-8. The encoding is defined by the BOM. There is no STRUCTURED-DATA present in the message, this is indicated by "-" in the STRUCTURED-DATA field. The MSG is "'su root' failed for lonvick...".

The HEADER part of the message must be in plain ASCII format, the parameter values of the STRUCTURED-DATA part must be in UTF-8, while the MSG part should be in UTF-8. The different parts of the message are explained in the following sections.

2.16.2.1. The PRI message part

The PRI part of the syslog message (known as Priority value) represents the Facility and Severity of the message. Facility represents the part of the system sending the message, while severity marks its importance. The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity. The possible facility and severity values are presented below.

	Note
	Facility codes may slightly vary between different platforms. The syslog-ng application accepts facility codes as numerical values as well.

Numerical Code	Facility
0	kernel messages
1	user-level messages
2	mail system
3	system daemons
4	security/authorization messages
5	messages generated internally by syslogd
6	line printer subsystem
7	network news subsystem
8	UUCP subsystem
9	clock daemon
10	security/authorization messages
11	FTP daemon
12	NTP subsystem
13	log audit
14	log alert
15	clock daemon
16-23	locally used facilities (local0-local7)

Table 2.3. syslog Message Facilities

The following table lists the severity values.

Numerical Code	Severity
0	Emergency: system is unusable
1	Alert: action must be taken immediately
2	Critical: critical conditions
3	Error: error conditions
4	Warning: warning conditions
5	Notice: normal but significant condition
6	Informational: informational messages
7	Debug: debug-level messages

Table 2.4. syslog Message Severities

2.16.2.2. The HEADER message part

The HEADER part contains the following elements:

• VERSION: Version number of the syslog protocol standard. Currently this can only be 1.

• ISOTIMESTAMP: The time when the message was generated in the ISO 8601 compatible standard timestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example: 2006-06-13T15:58:00.123+01:00.

• HOSTNAME: The machine that originally sent the message.

• APPLICATION: The device or application that generated the message

• PID: The process name or process ID of the syslog application that sent the message. It is not necessarily the process ID of the application that generated the message.

• MESSAGEID: The ID number of the message.

	Note
	The syslog-ng application supports other timestamp formats as well, like ISO, or the PIX extended format. The timestamp used in the IETF-syslog protocol is derived from RFC3339, which is based on ISO8601. For details, see the ts_format() option in Section 6.9, Global options.

2.16.2.3. The STRUCTURED-DATA message part

The STRUCTURED-DATA message part may contain meta- information about the syslog message, or application-specific information such as traffic counters or IP addresses. STRUCTURED-DATA consists of data blocks enclosed in brackets ([]). Every block include the ID of the block, and one or more name=value pairs. The syslog-ng application automatically parses the STRUCTURED-DATA part of syslog messages, which can be referenced in macros (see Section 6.5, Macros for details). An example STRUCTURED-DATA block looks like:

[exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"]

2.16.2.4. The MSG message part

The MSG part contains the text of the message itself. The encoding of the text must be UTF-8 if the BOM character is present in the message. If the message does not contain the BOM character, the encoding is treated as unknown. Usually messages arriving from legacy sources do not include the BOM character.

This section describes how to install the syslog-ng application interactively using the binary installer. The installer has a simple interface: use the TAB or the arrow keys of your keyboard to navigate between the options, and Enter to select an option.

	Note
	The installer stops the running syslogd application if it is running, but its components are not removed. The /etc/init.d/sysklogd init script is automatically renamed to /etc/init.d/sysklogd.backup. Rename this file to its original name if you want to remove syslog-ng or restart the syslogd package.

3.1.1. Installing syslog-ng in client or relay mode

Complete the following steps to install syslog-ng Open Source Edition on clients or relays. See Section 2.3, Modes of operation for details on the different operation modes of syslog-ng.

3.1.1.1. Procedure – Installing syslog-ng in client or relay mode

1. Enable the executable attribute for the installer using the chmod +x syslog-ng-<edition>-<version>-<OS>-<platform>.run, then start the installer as root using the ./syslog-ng-<edition>-<version>-<OS>-<platform>.run command. (Note that the exact name of the file depends on the operating system and platform.) Wait until the package is uncompressed and the welcome screen appears, then select Continue.

   

   Figure 3.1. The welcome screen

   
   

2. Accepting the EULA: You can install syslog-ng only if you understand and accept the terms of the End-User License Agreement (EULA). The full text of the EULA can be displayed during installation by selecting the Show EULA option, and is also available in this guide for convenience at Appendix 2, GNU General Public License. Select Accept to accept the EULA and continue the installation.

   If you do not accept the terms of the EULA for some reason, select Reject to cancel installing syslog-ng.

3. Detecting platform and operating system: The installer attempts to automatically detect your oprating system and platform. If the displayed information is correct, select Yes. Otherwise select Exit to abort the installation, and verify that your platform is supported. See Section 1.6, Supported platforms for a list of supported platforms. If your platform is supported but not detected correctly, contact your local distributor, reseller, or the BalaBit Support Team. See Section 5, Contact and support information for contact details.

   

   Figure 3.2. Platform detection

   
   

4. Upgrading: The syslog-ng installer can automatically detect if you have previously installed a version of syslog-ng on your system. To use the configuration file of this previous installation, select Yes. To ignore the old configuration file and create a new one, select No.

   Note that if you decide to use your existing configuration file, the installer automatically checks it for syntax error and displays a list of warnings and errors if it finds any problems.

   

   Figure 3.3. Upgrading syslog-ng

   
   

5. Generating a new configuration file: The installer displays some questions to generate a new configuration file.

   1. Remote sources: Select Yes to accept log messages from the network. TCP, UDP, and SYSLOG messages on every interface will be automatically accepted.

      

      Figure 3.4. Accepting remote messages

      
      

   2. Remote destinations: Enter the IP address or hostname of your logserver or relay and select OK.

      

      Figure 3.5. Forwarding messages to the logserver

      
      

   	Note
   	Accepting remote messages and forwarding them to a logserver means that syslog-ng will start in relay mode.

6. After the installation is finished, add the /opt/syslog-ng/bin and /opt/syslog-ng/sbin directories to your search PATH environment variable. That way you can use syslog-ng and its related tools without having to specify the full pathname. Add the following line to your shell profile:

   PATH=/opt/syslog-ng/bin:$PATH 

	Note
	The native logrotation tools do not send a SIGHUP to syslog-ng after rotating the log files, causing syslog-ng to write into files already rotated. To solve this problem, the syslog-ng init script links the /var/run/syslog.pid file to syslog-ng's pid. Also, on Linux, the install.sh script symlinks the initscript of the original syslog daemon to syslog-ng's initscript.

3.1.2. Installing syslog-ng in server mode

Complete the following steps to install syslog-ng on logservers. See Section 2.3, Modes of operation for details on the different operation modes of syslog-ng.

3.1.2.1. Procedure – Installing syslog-ng in server mode

1. Enable the executable attribute for the installer using the chmod +x syslog-ng-<edition>-<version>-<OS>-<platform>.run, then start the installer as root using the ./syslog-ng-<edition>-<version>-<OS>-<platform>.run command. (Note that the exact name of the file depends on the operating system and platform.) Wait until the package is uncompressed and the welcome screen appears, then select Continue.

   

   Figure 3.6. The welcome screen

   
   

2. Accepting the EULA: You can install syslog-ng only if you understand and accept the terms of the End-User License Agreement (EULA). The full text of the EULA can be displayed during installation by selecting the Show EULA option, and is also available in this guide for convenience at Appendix 2, GNU General Public License. Select Accept to accept the EULA and continue the installation.

   If you do not accept the terms of the EULA for some reason, select Reject to cancel installing syslog-ng.

3. Detecting platform and operating system: The installer attempts to automatically detect your oprating system and platform. If the displayed information is correct, select Yes. Otherwise select Exit to abort the installation, and verify that your platform is supported. See Section 1.6, Supported platforms for a list of supported platforms. If your platform is supported but not detected correctly, contact your local distributor, reseller, or the BalaBit Support Team. See Section 5, Contact and support information for contact details.

   

   Figure 3.7. Platform detection

   
   

4. Upgrading: The syslog-ng installer can automatically detect if you have previously installed a version of syslog-ng on your system. To use the configuration file of this previous installation, select Yes. To ignore the old configuration file and create a new one, select No.

   Note that if you decide to use your existing configuration file, the installer automatically checks it for syntax error and displays a list of warnings and errors if it finds any problems.

   

   Figure 3.8. Upgrading syslog-ng

   
   

5. Generating a new configuration file: The installer displays some questions to generate a new configuration file.

   1. Remote sources: Select Yes to accept log messages from the network. TCP, UDP, and SYSLOG messages on every interface will be automatically accepted.

      

      Figure 3.9. Accepting remote messages

      
      

   2. Remote destinations: Enter the IP address or hostname of your logserver or relay and select OK.

      

      Figure 3.10. Forwarding messages to the logserver

      
      

   	Note
   	Accepting remote messages and forwarding them to a logserver means that syslog-ng will start in relay mode.

6. After the installation is finished, add the /opt/syslog-ng/bin and /opt/syslog-ng/sbin directories to your search PATH environment variable. That way you can use syslog-ng and its related tools without having to specify the full pathname. Add the following line to your shell profile:

   PATH=/opt/syslog-ng/bin:$PATH 

	Note
	The native logrotation tools do not send a SIGHUP to syslog-ng after rotating the log files, causing syslog-ng to write into files already rotated. To solve this problem, the syslog-ng init script links the /var/run/syslog.pid file to syslog-ng's pid. Also, on Linux, the install.sh script symlinks the initscript of the original syslog daemon to syslog-ng's initscript.

3.1.3. Installing syslog-ng without user-interaction

The syslog-ng application can be installed in silent mode without any user-interaction by specifying the required parameters from the command line. Answers to every question of the installer can be set in advance using command-line parameters.

./syslog-ng-<version>.run -- [options]

	Warning
	The -- characters between the executable and the parameters are mandatory, like in the following example: ./syslog-ng-3.0.1b-solaris-10-sparc-client.run -- --accept-eula

To display the list of parameters, execute the ./syslog-ng-<version>.run -- --h command. Currently the following options are available:

• --accept-eula or -a: Accept the EULA.

• --upgrade | -u: Perform automatic upgrade — use the configuration file from an existing installation.

• --remote <destination host>: Send logs to the specified remote server. Not available when performing an upgrade.

• --network: Accept messages from the network. Not available when performing an upgrade.

• --configuration <file>: Use the specified configuration file.

To install syslog-ng on operating systems that use the Red Hat Package Manager (RPM), complete the following steps. Installing syslog-ng automatically replaces the original syslog service. The following supported operating systems use RPM:

To install syslog-ng on operating systems that use the Debian Software Package (deb) format, complete the following steps. The following supported operating systems use this format:

To compile syslog-ng Open Source Edition (OSE) from the source code, complete the following steps. Alternatively, you can use the precompiled binary packages. Precompiled binary packages are available for free for the supported Linux and BSD platforms at http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/upgrades/.

For information on configuring syslog-ng, see the Chapter 4, Configuring syslog-ng.

If you need to uninstall syslog-ng for some reason, you have the following options:

Complete the following steps to configure your Microsoft SQL Server to enable remote logins and accept log messages from syslog-ng.

The syslog-ng application is configured by editing the syslog-ng.conf file. Use any regular text editor application to modify the file. The precompiled syslog-ng packages include sample configuration files as well.

Every syslog-ng configuration file must begin with a line containing the version information of syslog-ng. For syslog-ng version 3.1, this line looks like:

@version:3.1

Versioning the configuration file was introduced in syslog-ng 3.0. If the configuration file does not contain the version information, syslog-ng assumes that the file is for syslog-ng version 2.x. In this case it interprets the configuration and sends warnings about the parts of the configuration that should be updated. Version 3.0 and later will correctly operate with configuration files of version 2.x, but the default values of certain parameters have changed since 3.0.

All identifiers, option names and attributes, and any other strings used in the syslog-ng configuration file are case sensitive. Objects must be defined before they are referenced in another statement.

	Example 4.1. A simple configuration file
	The following is a very simple configuration file for syslog-ng: it collects the internal messages of syslog-ng and the messages from /dev/log into the /var/log/messages_syslog-ng.log file. @version:3.0 source s_local { unix-stream("/dev/log"); internal(); }; destination d_file {file("/var/log/messages_syslog-ng.log"); }; log { source(s_local); destination(d_file); };

	Tip
	Before activating a new configuration, check that your configuration file is syntactically correct using the syslog-ng --syntax-only command. To activate the configuration, reload the configuration of syslog-ng using the /etc/init.d/syslog-ng reload command.

The syslog-ng.conf file is located under the /opt/syslog-ng/etc/ directory.

	Note
	Earlier versions of syslog-ng OSE stored the configuration file in different directories, depending on the platform; typically under /etc/syslog-ng/.

4.1.1. Including configuration files

The syslog-ng application supports including external files in its configuration file, so parts of its configuration can be managed separately. To include the contents of a file in the syslog-ng configuration, use the following syntax

include "filename";

This imports the entire file into the configuration of syslog-ng, at the location of the include statement. If you specify a directory, syslog-ng will try to include every file in alphabetic order. When including configuration files, consider the following points:

• If an object is defined twice (for example the original syslog-ng configuration file and the file imported into this configuration file both define the same option, source, or other object), then the object that is defined later in the configuration file will be effective. For example, if you set a global option at the beginning of the configuration file, and later include a file that defines the same option with a different value, then the option defined in the imported file will be used.

• Files can be embedded into each other: the included files can contain include statements as well, up to a maximum depth of 15 levels.

• Include statements can only be used at top level of the configuration file. For example, the following is correct:

  @version:3.0
  include "example.conf";

  But the following is not:

  source s_example {
      include "example.conf"
       };

	Warning
	The syslog-ng application will not start if it cannot find a file that is to be included in its configuration. Always double-check the filenames, paths, and access rights when including configuration files, and use the --syntax-only command-line option to check your configuration.

Global objects (for example sources, destinations, log paths, or filters) are defined in the syslog-ng configuration file. Object definitions consist of the following elements:

The syntax is summarized as follows:

type identifier { parameters };

Objects have parameters; some of them are required, others are optional. Required parameters are positional, meaning that they must be specified in a defined order. Optional arguments can be specified in any order using the option(value) format. If a parameter (optional or required) is not specified, its default value is used. The parameters and their default values are listed in the reference section of the particular object. See Chapter 6, Reference for details.

	Example 4.2. Using required and optional parameters
	The unix-stream() source driver has a single required argument: the name of the socket to listen on. Optional parameters follow the socket name in any order, so the following source definitions have the same effect: source s_demo_stream1 { unix-stream("/dev/log" max-connections(10) group(log)); }; source s_demo_stream2 { unix-stream("/dev/log" group(log) max-connections(10)); };

To add comments to the configuration file, start a line with # and write your comments. These lines are ignored by syslog-ng.

# Comment: This is a stream source
source s_demo_stream { 
           unix-stream("/dev/log" max-connections(10) group(log)); };

A source is where syslog-ng receives log messages. Sources consist of one or more drivers, each defining where and how messages are received.

To define a source, add a source statement to the syslog-ng configuration file using the following syntax:

source <identifier> { source-driver(params); source-driver(params); ... };

	Example 4.3. A simple source statement
	The following source statement receives messages on the TCP port 1999 of the interface having the 10.1.2.3 IP address. source s_demo_tcp { tcp(ip(10.1.2.3) port(1999)); };

	Example 4.4. A source statement using two source drivers
	The following source statement receives messages on the 1999 TCP port and the 1999 UDP port of the interface having the 10.1.2.3 IP address. source s_demo_two_drivers { tcp(ip(10.1.2.3) port(1999)); udp(ip(10.1.2.3) port(1999)); };

	Example 4.5. Setting default priority and facility
	If the message received by the source does not have a proper syslog header, you can use the default-facility() and default-priority() options to set the facility and priority of the messages. Note that these values are applied only to messages that do not set these parameters in their header. source headerless_messages { udp(default-facility(syslog) default-priority(emerg)); };

Define a source only once. The same source can be used in several log paths. Duplicating sources causes syslog-ng to open the source (TCP/IP port, file, etc.) more than once, which might cause problems. For example, include the /dev/log file source only in one source statement, and use this statement in more than one log path if needed.

To collect log messages on a specific platform, it is important to know how the native syslogd communicates on that platform. The following table summarizes the operation methods of syslogd on some of the tested platforms:

Each possible communication mechanism has a corresponding source driver in syslog-ng. For example, to open a unix socket with SOCK_DGRAM style communication use the driver unix-dgram. The same socket using the SOCK_STREAM style — as used under Linux — is called unix-stream.

	Example 4.6. Source statement on a Linux based operating system
	The following source statement collects the following log messages: internal(): Messages generated by syslog-ng. udp(ip(0.0.0.0) port(514)): Messages arriving to the 514/UDP port of any interface of the host. unix-stream("/dev/log");: Messages arriving to the /dev/log socket. source s_demo { internal(); udp(ip(0.0.0.0) port(514)); unix-stream("/dev/log"); };

The following table lists the source drivers available in syslog-ng.

For a complete description of the parameters of the above drivers, see Section 6.1, Source drivers.

4.3.1. Collecting internal messages

All messages generated internally by syslog-ng use this special source. To collect warnings, errors and notices from syslog-ng itself, include this source in one of your source statements.

internal()

The syslog-ng application will issue a warning upon startup if none of the defined log paths reference this driver.

	Example 4.7. Using the internal() driver
	source s_local { internal(); };

4.3.1.1. Log statistics

Periodically, syslog-ng sends a message containing statistics about the received messages, and about any lost messages since the last such message. It includes a processed entry for every source and destination, listing the number of messages received or sent, and a dropped entry including the IP address of the server for every destination where syslog-ng has lost messages. The center(received) entry shows the total number of messages received from every configured sources.

The following is a sample log statistics message for a configuration that has a single source (s_local) and a network and a local file destination (d_network and d_local, respectively). All incoming messages are sent to both destinations.

Log statistics;
                    dropped='tcp(AF_INET(192.168.10.1:514))=6439',
                    processed='center(received)=234413',
                    processed='destination(d_tcp)=234413',
                    processed='destination(d_local)=234413',
                    processed='source(s_local)=234413'

Log statistics can be also retrieved on-demand using one of the following options:

• Use the socat application: echo STATS | socat -vv UNIX-CONNECT:/opt/syslog-ng/var/run/syslog-ng.ctl -

• If you have an OpenBSD-style netcat application installed, use the echo STATS | nc -U var/run/syslog-ng.ctl command. Note that the netcat included in most Linux distributions is a GNU-style version that is not suitable to query the statistics of syslog-ng.

• Starting from syslog-ng Open Source Edition version 3.1, syslog-ng Open Source Edition includes the syslog-ng-ctl utility. Use the syslog-ng-ctl stats command.

The statistics include a list of source groups and destinations, as well as the number of processed messages for each. The verbosity of the statistics can be set using the stats_level() option. See Section 6.9, Global options for details. An example output is shown below.

src.internal;s_all#0;;a;processed;6445
src.internal;s_all#0;;a;stamp;1268989330
destination;df_auth;;a;processed;404
destination;df_news_dot_notice;;a;processed;0
destination;df_news_dot_err;;a;processed;0
destination;d_ssb;;a;processed;7128
destination;df_uucp;;a;processed;0
source;s_all;;a;processed;7128
destination;df_mail;;a;processed;0
destination;df_user;;a;processed;1
destination;df_daemon;;a;processed;1
destination;df_debug;;a;processed;15
destination;df_messages;;a;processed;54
destination;dp_xconsole;;a;processed;671
dst.tcp;d_network#0;10.50.0.111:514;a;dropped;5080
dst.tcp;d_network#0;10.50.0.111:514;a;processed;7128
dst.tcp;d_network#0;10.50.0.111:514;a;stored;2048
destination;df_syslog;;a;processed;6724
destination;df_facility_dot_warn;;a;processed;0
destination;df_news_dot_crit;;a;processed;0
destination;df_lpr;;a;processed;0
destination;du_all;;a;processed;0
destination;df_facility_dot_info;;a;processed;0
center;;received;a;processed;0
destination;df_kern;;a;processed;70
center;;queued;a;processed;0
destination;df_facility_dot_err;;a;processed;0

The statistics are semicolon separated; every line contains statistics for a particular object (for example source, destination, tag, etc.). The statistics have the following fields:

1. The type of the object (for example dst.file, tag, src.facility)

2. The ID of the object used in the syslog-ng configuration file, for example d_internal or source.src_tcp. The #0 part means that this is the first destination in the destination group.

3. The instance ID (destination) of the object, for example the filename of a file destination, or the name of the application for a program source or destination.

4. The status of the object. One of the following:

   • a - active. At the time of quering the statistics, the source or the destination was still alive (it continuously received statistical data).

   • d - dynamic. Such objects may not be continuously available, for example, like statistics based on the sender's hostname.

   • o - This object was once active, but stopped receiving messages. (For example a dynamic object may disappear and become orphan.)

5. The type of the statistics:

   • processed: The number of messages that successfully reached their destination.

   • dropped: The number of dropped messages — syslog-ng OSE could not send the messages to the destination and the output buffer got full, so messages were lost.

   • stored: The number of messages stored in the message queue, waiting to be sent to the destination.

   • suppressed: The number of suppressed messages (if the suppress() feature is enabled).

   • stamp: The UNIX timestamp of the last message sent to the destination.

6. The number of such messages.

	Note
	Note that certain statistics are available only if the stats-level() option is set to a higher value.

4.3.2. Collecting messages from text files

Collects log messages from plain-text files, for example from the logfiles of an Apache webserver.

The syslog-ng application notices if a file is renamed or replaced with a new file, so it can correctly follow the file even if logrotation is used. When syslog-ng is restarted, it records the position of the last sent log message, and continues to send messages from this position after the restart.

The file driver has a single required parameter specifying the file to open. For the list of available optional parameters, see Section 6.1.2, file().

Declaration:
                file(filename);

	Example 4.8. Using the file() driver
	source s_file { file("/var/log/messages"); };

The kernel usually sends log messages to a special file (/dev/kmsg on BSDs, /proc/kmsg on Linux). The file() driver reads log messages from such files. The syslog-ng application can periodically check the file for new log messages if the follow_freq() option is set.

Note

On Linux, the klogd daemon can be used in addition to syslog-ng to read kernel messages and forward them to syslog-ng. klogd used to preprocess kernel messages to resolve symbols etc., but as this is deprecated by ksymoops there is really no point in running both klogd and syslog-ng in parallel. Also note that running two processes reading /proc/kmsg at the same time might result in dead-locks. When using syslog-ng to read messages from the /proc/kmsg file, syslog-ng automatically disables the follow_freq() parameter to avoid blocking the file.

4.3.3. Collecting messages from named pipes

The pipe driver opens a named pipe with the specified name and listens for messages. It is used as the native message delivery protocol on HP-UX.

The pipe driver has a single required parameter, specifying the filename of the pipe to open. For the list of available optional parameters, see Section 6.1.3, pipe().

Declaration:
                pipe(filename);

	Note
	As of syslog-ng Open Source Edition 3.0.2, pipes are created automatically. In earlier versions, you had to create the pipe using the mkfifo(1) command.

Pipe is very similar to the file() driver, but there are a few differences, for example pipe() opens its argument in read-write mode, therefore it is not recommended to be used on special files like /proc/kmsg.

	Warning
	It is not recommended to use pipe() on anything else than real pipes.

	Example 4.9. Using the pipe() driver
	source s_pipe { pipe("/dev/pipe" pad_size(2048)); };

4.3.4. Collecting messages on Sun Solaris

Solaris uses its STREAMS framework to send messages to the syslogd process. Solaris 2.5.1 and above uses an IPC called door in addition to STREAMS, to confirm the delivery of a message. The syslog-ng application supports the IPC mechanism via the door() option (see below).

	Note
	The sun-streams() driver must be enabled when the syslog-ng application is compiled (see ./configure --help).

The sun-streams() driver has a single required argument specifying the STREAMS device to open, and the door() option. For the list of available optional parameters, see Section 6.1.5, sun-streams() driver.

Declaration:
        sun-streams(name_of_the_streams_device door(filename_of_the_door));

	Example 4.10. Using the sun-streams() driver
	source s_stream { sun-streams("/dev/log" door("/etc/.syslog_door")); };

4.3.5. Collecting messages using the IETF syslog protocol

The syslog() driver enables to receive messages from the network using the new standard syslog protocol and message format (also called IETF-syslog protocol; described in RFC 5424-28, see Section 2.16.2, IETF-syslog messages). UDP, TCP, and TLS-encrypted TCP can all be used to transport the messages.

For the list of available optional parameters, see Section 6.1.6, syslog().

Declaration:
            syslog(ip() port() transport() options());

Example 4.11. Using the syslog() driver

TCP source listening on the localhost on port 1999. source s_syslog { syslog(ip(127.0.0.1) port(1999) transport("tcp")); }; UDP source with defaults. source s_udp { syslog( transport("udp")); }; Encrypted source where the client is also authenticated. See Section 6.10, TLS options for details on the encryption settings. source s_syslog_tls{ syslog( ip(10.100.20.40) transport("tls") tls( peer-verify(required-trusted) ca_dir('/opt/syslog-ng/etc/syslog-ng/keys/ca.d/') key_file('/opt/syslog-ng/etc/syslog-ng/keys/server_privatekey.pem') cert_file('/opt/syslog-ng/etc/syslog-ng/keys/server_certificate.pem') ) );};

4.3.6. Collecting messages from remote hosts using the BSD syslog protocol

The tcp(), tcp6(), udp(), udp6() drivers can receive messages from the network using the TCP and UDP networking protocols. The tcp6() and udp6() drivers use the IPv6 network protocol, while tcp() and udp() use IPv4.

UDP is a simple datagram oriented protocol, which provides "best effort service" to transfer messages between hosts. It may lose messages, and no attempt is made at the protocol level to retransmit such lost messages. The BSD-syslog protocol traditionally uses UDP.

TCP provides connection-oriented service, which basically means that the path of the messages is flow-controlled. Along this path, each message is acknowledged, and retransmission is done for lost packets. Generally it is safer to use TCP, because lost connections can be detected, and no messages get lost, assuming that the TCP connection does not break. When a TCP connection is broken the 'in-transit' messages that were sent by syslog-ng but not yet received on the other side are lost. (Basically these messages are still sitting in the socket buffer of the sending host and syslog-ng has no information about the fate of these messages).

The tcp() and udp() drivers do not have any required parameters. By default they bind to the 0.0.0.0:514 address, which means that syslog-ng will listen on all available interfaces, port 514. To limit accepted connections to only one interface, use the localip() parameter as described below. For the list of available optional parameters, see Section 6.1.7, tcp(), tcp6(), udp() and udp6().

Declaration:
                tcp([options]);
                udp([options]);

	Note
	The tcp port 514 is reserved for use with rshell, so select a different port if syslog-ng and rshell is used at the same time.

If you specify a multicast bind address to udp() and udp6(), syslog-ng will automatically join the necessary multicast group. TCP does not support multicasting.

The syslog-ng application supports TLS (Transport Layer Security, also known as SSL) for the tcp() and tcp6() drivers. See the TLS-specific options below and Section 4.12, Encrypting log messages with TLS for details. For the list of available optional parameters, see Section 6.1.7, tcp(), tcp6(), udp() and udp6().

	Tip
	The syslog() driver also supports TLS-encrypted connections.

Example 4.12. Using the udp() and tcp() drivers

A simple udp() source with default settings. source s_udp { udp(); };# An UDP source with default settings. A TCP source listening on the localhost interface, with a limited number of connections allowed. source s_tcp { tcp(ip(127.0.0.1) port(1999) max-connections(10)); }; A TCP source listening on a TLS-encrypted channel. source s_tcp { tcp(ip(127.0.0.1) port(1999) tls(peer-verify('required-trusted') key_file('/opt/syslog-ng/etc/syslog-ng/syslog-ng.key') cert_file('/opt/syslog-ng/etc/syslog-ng/syslog-ng.crt'))); }; A TCP source listening for messages using the IETF-syslog message format. Note that for transferring IETF-syslog messages, generally you are recommended to use the syslog() driver on both the client and the server, as it uses both the IETF-syslog message format and the protocol. See Section 4.3.5, Collecting messages using the IETF syslog protocol for details. source s_tcp_syslog { tcp(ip(127.0.0.1) port(1999) flags(syslog-protocol) ); };

4.3.7. Collecting messages from UNIX domain sockets

The unix-stream() and unix-dgram() drivers open an AF_UNIX socket and start listening on it for messages. The unix-stream() driver is primarily used on Linux and uses SOCK_STREAM semantics (connection oriented, no messages are lost); while unix-dgram() is used on BSDs and uses SOCK_DGRAM semantics: this may result in lost local messages if the system is overloaded.

To avoid denial of service attacks when using connection-oriented protocols, the number of simultaneously accepted connections should be limited. This can be achieved using the max-connections() parameter. The default value of this parameter is quite strict, you might have to increase it on a busy system.

Both unix-stream and unix-dgram have a single required argument that specifies the filename of the socket to create. For the list of available optional parameters, see Section 6.1.8, unix-stream() and unix-dgram()

Declaration: 
                unix-stream(filename [options]);
                unix-dgram(filename [options]);

	Note
	syslogd on Linux originally used SOCK_STREAM sockets, but some distributions switched to SOCK_DGRAM around 1999 to fix a possible DoS problem. On Linux you can choose to use whichever driver you like as syslog clients automatically detect the socket type being used.

The difference between the unix-stream and unix-dgram drivers is similar to the difference between the TCP and UDP network protocols. Use the following guidelines to select which driver to use in a particular situation:

Choose unix-stream if you would choose TCP (stream) instead of UDP (datagram). The unix-stream driver offers the following features:

• Increased reliability

• Ordered delivery of messages

• Client-side notification of failures

Choose unix-dgram if you would choose TCP (stream) over UDP (datagram). The unix-dgram driver offers the following features:

• Decreased possibility of Dos by opening too many connections (a local vulnerability)

• Less overhead

However, the client does not notice if a message is lost when using the unix-dgram driver.

	Example 4.13. Using the unix-stream() and unix-dgram() drivers
	source s_stream { unix-stream("/dev/log" max-connections(10)); }; source s_dgram { unix-dgram("/var/run/log"); };

A destination is where a log message is sent if the filtering rules match. Similarly to sources, destinations consist of one or more drivers, each defining where and how messages are sent.

	Tip
	If no drivers are defined for a destination, all messages sent to the destination are discarded. This is equivalent to omitting the destination from the log statement.

To define a destination, add a destination statement to the syslog-ng configuration file using the following syntax:

destination <identifier> { 
                destination-driver(params); destination-driver(params); ... };

	Example 4.14. A simple destination statement
	The following destination statement sends messages to the TCP port 1999 of the 10.1.2.3 host. destination d_demo_tcp { tcp("10.1.2.3" port(1999)); }; If name resolution is configured, the hostname of the target server can be used as well. destination d_tcp { tcp("target_host" port(1999); localport(999)); };

The following table lists the destination drivers available in syslog-ng.

For detailed list of driver parameters, see Section 6.2, Destination drivers.

4.4.1. Storing messages in plain-text files

The file driver is one of the most important destination drivers in syslog-ng. It allows to output messages to the specified text file, or to a set of files.

The destination filename may include macros which get expanded when the message is written, thus a simple file() driver may create several files. For more information on available macros see Section 6.5, Macros.

If the expanded filename refers to a directory which does not exist, it will be created depending on the create_dirs() setting (both global and a per destination option).

The file() has a single required parameter that specifies the filename that stores the log messages. For the list of available optional parameters, see Section 6.2.1, file().

Declaration:
                file(filename options());

	Example 4.15. Using the file() driver
	destination d_file { file("/var/log/messages" ); };

	Example 4.16. Using the file() driver with macros in the file name and a template for the message
	destination d_file { file("/var/log/$YEAR.$MONTH.$DAY/messages" template("$HOUR:$MIN:$SEC $TZ $HOST [$LEVEL] $MSG $MSG\n") template_escape(no)); };

	Note
	When using the file() destination, update the configuration of your log rotation program to rotate these files. Otherwise, the log files can become very large.

Warning

Since the state of each created file must be tracked by syslog-ng, it consumes some memory for each file. If no new messages are written to a file within 60 seconds (controlled by the time_reap() global option), it is closed, and its state is freed. Exploiting this, a DoS attack can be mounted against the system. If the number of possible destination files and its needed memory is more than the amount available on the syslog-ng server. The most suspicious macro is $PROGRAM, where the number of possible variations is rather high. Do not use the $PROGRAM macro in insecure environments.

4.4.2. Sending messages to named pipes

The pipe() driver sends messages to a named pipe like /dev/xconsole.

The pipe driver has a single required parameter, specifying the filename of the pipe to open. The filename can include macros. For the list of available optional parameters, see Section 6.2.2, pipe().

Declaration:
                pipe(filename);

	Warning
	As of syslog-ng Open Source Edition 3.0.2, pipes are created automatically. In earlier versions, you had to create the pipe using the mkfifo(1) command.

	Example 4.17. Using the pipe() driver
	destination d_pipe { pipe("/dev/xconsole"); };

4.4.3. Sending messages to external applications

The program() driver starts an external application or script and sends the log messages to its standard input (stdin).

The program() driver has a single required parameter, specifying a program name to start. The program is executed with the help of the current shell, so the command may include both file patterns and I/O redirections. For the list of available optional parameters, see Section 6.2.3, program().

Declaration: 
                program(command_to_run);

	Note
	The syslog-ng application automatically restarts the external program if it exits for reliability reasons. However it is not recommended to launch programs for single messages, because if the message rate is high, launching several instances of an application might overload the system, resulting in Denial of Service.

Note that the message format does not include the priority and facility values by default. To add these values, specify a template for the program destination, as shown in the following example.

	Example 4.18. Using the program() destination driver
	destination d_prog { program("/bin/script" template("<$PRI>$DATE $HOST $MSG\n"); ); };

4.4.4. Storing messages in an SQL database

The sql() driver sends messages into an SQL database. Currently the Microsoft SQL (MSSQL), MySQL, Oracle, PostgreSQL, and SQLite databases are supported.

The sql() driver has the following required parameters:

type

Type:	mssql, mysql, oracle, pgsql, or sqlite3
Default:	n/a

Description: Specifies the type of the database, i.e., the DBI database driver to use. Use the mssql option to send logs to an MSSQL database. See the examples of the databases on the following sections for details.

database

Type:	string
Default:	n/a

Description: Name of the database that stores the logs.

table

Type:	string
Default:	n/a

Description: Name of the database table to use (can include macros). When using macros, note that some databases limit the length of table names.

columns

Type:	string list
Default:	"date", "facility", "level", "host", "program", "pid", "message"

Description: Name of the columns storing the data in fieldname [dbtype] format. The [dbtype] parameter is optional, and specifies the type of the field. By default, syslog-ng creates text columns. Note that not every database engine can index text fields.

values

Type:	string list
Default:	"${R_YEAR}-${R_MONTH}-${R_DAY} ${R_HOUR}:${R_MIN}:${R_SEC}", "$FACILITY", "$LEVEL", "$HOST", "$PROGRAM", "$PID", "$MSGONLY"

Description:The parts of the message to store in the fields specified in the columns parameter.

For the list of available optional parameters, see Section 6.2.4, sql().

Declaration: 
    sql(database_type host_parameters database_parameters [options]);

Warning

The syslog-ng application requires read and write access to the SQL table, otherwise it cannot verify that the destination table exists. Currently the syslog-ng application has default schemas for the different databases and uses these defaults if the database schema (for example columns and column types) is not defined in the configuration file. However, these schemas will be deprecated and specifying the exact database schema will be required in later versions of syslog-ng.

Note

In addition to the standard syslog-ng packages, the sql() destination requires database-specific packages to be installed. These packages are automatically installed by the binary syslog-ng installer. The sql() driver is currently not available for every platform that is supported by syslog-ng. For a list of platforms that support the sql() driver, visit http://www.balabit.com/network-security/syslog-ng/central-syslog-server/.

The table and value parameters can include macros to create tables and columns dynamically (see Section 6.5, Macros for details).

	Warning
	When using macros in table names, note that some databases limit the maximum allowed length of table names. Consult the documentation of the database for details.

Inserting the records into the database is performed by a separate thread. The syslog-ng application automatically performs the escaping required to insert the messages into the database.

Example 4.19. Using the sql() driver

The following example sends the log messages into a PostgreSQL database running on the logserver host. The messages are inserted into the logs database, the name of the table includes the exact date and the name of the host sending the messages. The syslog-ng application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges. destination d_sql { sql(type(pgsql) host("logserver") username("syslog-ng") password("password") database("logs") table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}") columns("datetime", "host", "program", "pid", "message") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes("datetime", "host", "program", "pid", "message")); }; The following example specifies the type of the database columns as well: destination d_sql { sql(type(pgsql) host("logserver") username("syslog-ng") password("password") database("logs") table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}") columns("datetime varchar(16)", "host varchar(32)", "program varchar(20)", "pid varchar(8)", "message varchar(200)") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes("datetime", "host", "program", "pid", "message")); };

4.4.4.1. Using the sql() driver with an Oracle database

The Oracle sql destination has some special aspects that are important to note.

• The hostname of the database server is set in the tnsnames.ora file, not in the host parameter of the sql() destination.

  Make sure to set the Oracle-related environment variables properly, so syslog-ng and the Oracle client will find the file. The following variables must be set: ORACLE_BASE, ORACLE_HOME, and ORACLE_SID. See the documentation of the Oracle Instant Client for details.

• As certain database versions limit the maximum length of table names, macros in the table names should be used with care.

• In the current version of syslog-ng PE, the types of database columns must be explicitly set for the Oracle destination. The column used to store the text part of the syslog messages should be able to store messages as long as the longest message permitted by syslog-ng, therefore it is usually recommended to use the varchar2 or clob column type. (The maximum length of the messages can be set using the log_msg_size() option.) See the following example for details.

Example 4.20. Using the sql() driver with an Oracle database

The following example sends the log messages into an Oracle database running on the logserver host, which must be set in the /etc/tnsnames.ora file. The messages are inserted into the LOGS database, the name of the table includes the exact date when the messages were sent. The syslog-ng application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges. destination d_sql { sql(type(oracle) username("syslog-ng") password("password") database("LOGS") table("msgs_${R_YEAR}${R_MONTH}${R_DAY}") columns("datetime varchar(16)", "host varchar(32)", "program varchar(32)", "pid varchar(8)", "message varchar2") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes("datetime", "host", "program", "pid", "message")); }; The Oracle Instant Client retrieves the address of the database server from the /etc/tnsnames.ora file. Edit or create this file as needed for your configuration. A sample is provided below. LOGS = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP) (HOST = logserver) (PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = EXAMPLE.SERVICE) ) )

4.4.4.2. Using the sql() driver with a Microsoft SQL database

The mssql database driver can access Microsoft SQL (MSSQL) destinations. This driver has some special aspects that are important to note.

• The date format used by the MSSQL database must be explicitly set in the /etc/locales.conf file of the syslog-ng server. See the following example for details.

• As certain database versions limit the maximum length of table names, macros in the table names should be used with care.

• In the current version of syslog-ng PE, the types of database columns must be explicitly set for the MSSQL destination. The column used to store the text part of the syslog messages should be able to store messages as long as the longest message permitted by syslog-ng. The varchar column type can store maximum 4096 bytes-long messages. The maximum length of the messages can be set using the log_msg_size() option. See the following example for details.

• Remote access for SQL users must be explicitly enabled on the Microsoft Windows host running the Microsoft SQL Server. See Section 3.6, Configuring Microsoft SQL Server to accept logs from syslog-ng for details.

Example 4.21. Using the sql() driver with an MSSQL database

The following example sends the log messages into an MSSQL database running on the logserver host. The messages are inserted into the syslogng database, the name of the table includes the exact date when the messages were sent. The syslog-ng application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges. destination d_mssql { sql(type(mssql) host("logserver") port("1433") username("syslogng") password("syslogng") database("syslogng") table("msgs_${R_YEAR}${R_MONTH}${R_DAY}")columns("datetime varchar(16)", "host varchar(32)", "program varchar(32)", "pid varchar(8)", "message varchar(4096)") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes("datetime", "host", "program", "pid")); }; The date format used by the MSSQL database must be explicitly set in the /etc/locales.conf file of the syslog-ng server. Edit or create this file as needed for your configuration. A sample is provided below. [default] date = "%Y-%m-%d %H:%M:%S"

4.4.5. Sending messages to a remote logserver using the IETF-syslog protocol

The syslog() driver sends messages to a remote host (for example a syslog-ng server or relay) on the local intranet or internet using the new standard syslog protocol developed by IETF (see Section 2.16.2, IETF-syslog messages for details about the new protocol). The protocol supports sending messages using the UDP, TCP, or the encrypted TLS networking protocols.

The required arguments of the driver are the address of the destination host (where messages should be sent). The transport method (networking protocol) is optional, syslog-ng uses the TCP protocol by default. For the list of available optional parameters, see Section 6.2.5, syslog().

Declaration:
                syslog(host transport [options]);

	Note
	Note that the syslog destination driver has required parameters, while the source driver defaults to the local bind address, and every parameter is optional.

The udp transport method automatically sends multicast packets if a multicast destination address is specified. The tcp and tls methods do not support multicasting.

	Note
	The default ports for the different transport protocols are as follows: UDP — 514; TLS — 6514.

Example 4.22. Using the syslog() driver

destination d_tcp { syslog(ip("10.1.2.3") transport("tcp") port(1999) localport(999)); }; If name resolution is configured, the hostname of the target server can be used as well. destination d_tcp { syslog(ip("target_host") transport("tcp") port(1999) localport(999)); }; Send the log messages using TLS encryption and use mutual authentication. See Section 6.10, TLS options for details on the encryption and authentication options. destination d_syslog_tls{ syslog("10.100.20.40" transport("tls") port(6514) tls(peer-verify(required-trusted) ca_dir('/opt/syslog-ng/etc/syslog-ng/keys/ca.d/') key_file('/opt/syslog-ng/etc/syslog-ng/keys/client_key.pem') cert_file('/opt/syslog-ng/etc/syslog-ng/keys/client_certificate.pem')) );};

4.4.6. Sending messages to a remote logserver using the legacy BSD-syslog protocol

The tcp(), tcp6(), udp(), and udp6() drivers send messages to another host (for example a syslog-ng server or relay) on the local intranet or internet using the UDP or TCP protocol. The tcp6() and udp6() drivers use the IPv6 network protocol.

All four drivers have a single required parameter specifying the destination host address, where messages should be sent. For the list of available optional parameters, see Section 6.2.6, tcp(), tcp6(), udp(), and udp6().

The udp() and udp6() drivers automatically send multicast packets if a multicast destination address is specified. The tcp() and tcp6() drivers do not support multicasting.

Declaration:
                tcp(host [options]);
                udp(host [options]);
                tcp6(host [options]);
                udp6(host [options]);

Example 4.23. Using the tcp() driver

destination d_tcp { tcp("10.1.2.3" port(1999) localport(999)); }; If name resolution is configured, the hostname of the target server can be used as well. destination d_tcp { tcp("target_host" port(1999) localport(999)); }; To send messages using the IETF-syslog message format without using the IETF-syslog protocol, enable the syslog-protocol flag: destination d_tcp { tcp("10.1.2.3" port(1999) flags(syslog-protocol) ); }; (To use the IETF-syslog protocol, see Section 6.2.5, syslog().)

4.4.7. Sending messages to UNIX domain sockets

The unix-stream() and unix-dgram() drivers send messages to a UNIX domain socket in either SOCK_STREAM or SOCK_DGRAM mode.

Both drivers have a single required argument specifying the name of the socket to connect to. For the list of available optional parameters, see Section 6.2.7, unix-stream() & unix-dgram().

Declaration: 
                unix-stream(filename [options]);
                unix-dgram(filename [options]);

	Example 4.24. Using the unix-stream() driver
	destination d_unix_stream { unix-stream("/var/run/logs"); };

4.4.8. usertty()

This driver writes messages to the terminal of a logged-in user.

The usertty() driver has a single required argument, specifying a username who should receive a copy of matching messages. Use the asterisk * to specify every user currently logged in to the system.

Declaration: 
    usertty(username);

The usertty() does not have any further options nor does it support templates.

	Example 4.25. Using the usertty() driver
	destination d_usertty { usertty("root"); };

Log paths determine what happens with the incoming log messages. Messages coming from the sources listed in the log statement and matching all the filters are sent to the listed destinations.

To define a log path, add a log statement to the syslog-ng configuration file using the following syntax:

log {
    source(s1); source(s2); ... 
    optional_element(filter1|parser1|rewrite1); optional_element(filter2|parser2|rewrite2);... 
    destination(d1); destination(d2); ... 
    flags(flag1[, flag2...]);
    };

	Warning
	Log statements are processed in the order they appear in the configuration file, thus the order of log paths may influence what happens to a message, especially when using filters and log flags.

	Example 4.26. A simple log statement
	The following log statement sends all messages arriving to the localhost to a remote server. source s_localhost { tcp(ip(127.0.0.1) port(1999) ); }; destination d_tcp { tcp("10.1.2.3" port(1999); localport(999)); }; log { source(s_localhost); destination(d_tcp); };

All matching log statements are processed by default, and the messages are sent to every matching destination by default. So a single log message might be sent to the same destination several times, provided the destination is listed in several log statements, and it can be also sent to several different destinations.

This default behavior can be changed using the flags() parameter. Flags apply to individual log paths; they are not global options. The following flags available in syslog-ng:

	Warning
	The final, fallback, and catchall flags apply only for the top-level log paths, they have no effect on embedded log paths.

Example 4.27. Using log path flags

Let's suppose that you have two hosts (myhost_A and myhost_B) that run two applications each (application_A and application_B), and you collect the log messages to a central syslog-ng server. On the server, you create two log paths: one that processes only the messages sent by myhost_A; and one that processes only the messages sent by application_A. This means that messages sent by application_A running on myhost_A will be processed by both log paths, and the messages of application_B running on myhost_B will not be processed at all. If you add the final flag to the first log path, then only this log path will process the messages of myhost_A, so the second log path will receive only the messages of application_A running on myhost_B. If you create a third log path that includes the fallback flag, it will process the messages not processed by the first two log paths, in this case, the messages of application_B running on myhost_B. Adding a fourth log path with the catchall flag would process every message received by the syslog-ng server. log { source(s_localhost); destination(d_file); flags(catchall); };

For details on the individual flags, see Section 6.3, Log path flags. The effect and use of the flow-control flag is detailed in Section 2.12, Managing incoming and outgoing messages with flow-control.

4.5.1. Using embedded log statements

Embedded log statements (see Section 2.2.2, Embedded log statements ) re-use the results of processing messages (for example the results of filtering or rewriting) to create complex log paths. Embedded log statements use the same syntax as regular log statements, but they cannot contain additional sources. To define embedded log statements, use the following syntax:

log {
    source(s1); source(s2); ... 
    
    optional_element(filter1|parser1|rewrite1); 
    optional_element(filter2|parser2|rewrite2);... 
    
    destination(d1); destination(d2); ... 
    
    #embedded log statement
    log
        {
        optional_element(filter1|parser1|rewrite1); 
        optional_element(filter2|parser2|rewrite2);... 
        destination(d1); destination(d2); ...
        #another embedded log statement
        log
        {
            optional_element(filter1|parser1|rewrite1); 
            optional_element(filter2|parser2|rewrite2);... 
            destination(d1); destination(d2); ...};
        };    
    #set flags after the embedded log statements
    flags(flag1[, flag2...]); 
        };

	Warning
	The final, fallback, and catchall flags apply only for the top-level log paths, they have no effect on embedded log paths.

Example 4.28. Using embedded log paths

The following log path sends every message to the d_file1 and the d_file2 destinations. log { source(s_localhost); destination(d_file1); destination(d_file2); }; The next example is equivalent with the one above, but uses an embedded log statement. log { source(s_localhost); destination(d_file1); log {destination(d_file2); }; }; The following example sends every message coming from the host 192.168.1.1 into the d_file1 destination, and sends every message coming from the host 192.168.1.1 and containing the string example into the d_file2 destination. log { source(s_localhost); host(192.168.1.); destination(d_file1); log {message("example"); destination(d_file2); }; }; The following example collects logs from multiple source groups and uses the source() filter in the embedded log statement to select messages of the s_network source group. log { source(s_localhost); source(s_network); destination(d_file1); log {source(s_network); destination(d_file2); }; };

4.5.2. Configuring flow-control

For details on how flow-control works, see Section 2.12, Managing incoming and outgoing messages with flow-control. The summary of the main points is as follows:

• The syslog-ng application normally reads a maximum of log_fetch_limit() number of messages from a source.

• From TCP and unix-stream sources, syslog-ng reads a maximum of log_fetch_limit() from every connection of the source. The number of connections to the source is set using the max_connections() parameter.

• Every destination has an output buffer (log_fifo_size()).

• Flow-control uses a control window to determine if there is free space in the output buffer for new messages. Every source has its own control window; log_iw_size() parameter sets the size of the control window.

• When a source accepts multiple connections, the messages of every connection use the same control window.

• The output buffer must be larger than the control window of every source that logs to the destination.

• If the control window is full, syslog-ng stops reading messages from the source until some messages are successfully sent to the destination.

• If the output buffer becomes full, and neither disk-buffering nor flow-control is used, messages may be lost.

	Note
	If you modify the max_connections() or the log_fetch_limit() parameter, do not forget to adjust the log_iw_size() and log_fifo_size() parameters accordingly.

Example 4.29. Sizing parameters for flow-control

Suppose that syslog-ng has a source that must accept up to 300 parallel connections. Such situation can arise when a network source receives connections from many clients, or if many applications log to the same socket. Therefore, set the max_connections() parameter of the source to 300. However, the log_fetch_limit() (default value: 10) parameter applies to every connection of the source individually, while the log_iw_size() (default value: 100) parameter applies to the source. In a worst-case scenario, the destination does not accept any messages, while all 300 connections send at least log_fetch_limit() number of messages to the source during every poll loop. Therefore, the control window must accommodate at least max_connections()*log_fetch_limit() messages to be able to read every incoming message of a poll loop. In the current example this means that (log_iw_size() should be greater than 300*10=3000. If the control window is smaller than this value, the control window might fill up with messages from the first connections — causing syslog-ng to read only one message of the last connections in every poll loop. The output buffer of the destination must accommodate at least log_iw_size() messages, but use a greater value: in the current example 3000*10=30000 messages. That way all incoming messages of ten poll loops fit in the output buffer. If the output buffer is full, syslog-ng does not read any messages from the source until some messages are successfully sent to the destination. source s_localhost { tcp(ip(127.0.0.1) port(1999) max-connections(300)); }; destination d_tcp { tcp("10.1.2.3" port(1999); localport(999)); log_fifo_size(30000); }; log { source(s_localhost); destination(d_tcp); flags(flow-control); }; If other sources send messages to this destination, than the output buffer must be further increased. For example, if a network host with maximum 100 connections also logs into the destination, than increase the log_fifo_size() by 10000. source s_localhost { tcp(ip(127.0.0.1) port(1999) max-connections(300)); }; source s_tcp { tcp(ip(192.168.1.5) port(1999) max-connections(100)); }; destination d_tcp { tcp("10.1.2.3" port(1999); localport(999)); log_fifo_size(40000); }; log { source(s_localhost); destination(d_tcp); flags(flow-control); };

See also Section 5.2, Handling lots of parallel connections.

The following sections describe how to select and filter log messages.

4.6.1. Using filters

Filters perform log routing within syslog-ng: a message passes the filter if the filter expression is true for the particular message. If a log statement includes filters, the messages are sent to the destinations only if they pass all filters of the log path. For example, a filter can select only the messages originating from a particular host. Complex filters can be created using filter functions and logical boolean expressions.

To define a filter, add a filter statement to the syslog-ng configuration file using the following syntax:

filter <identifier> { expression; };

The expression may contain the following elements:

• The functions listed in Section facility(). Some of the functions accept extended regular expressions as parameters.

• The boolean operators and, or, not.

• Parentheses to mark the precedence of the operators when using complex filters.

Example 4.30. A simple filter statement

The following filter statement selects the messages that contain the word deny and come from the host example. filter demo_filter { host("example") and match("deny" value("MESSAGE")); }; For the filter to have effect, include it in a log statement: log demo_filteredlog { source(s1); source(s2); filter(demo_filter); destination(d1); destination(d2); }; The host(), match(), and program() filter functions accept regular expressions as parameters. filter demo_regexp_filter { host("system.*1") and match("deny" value("MESSAGE")); };

The value() parameter limits the scope of a filter function to the scope of a macro. For example, to limit the scope of the match() filter to the text part of the message, use:

match("keyword" value("MESSAGE"))

The value() parameter accepts both built-in macros and user-defined ones created with a parser. Do not prefix the macros with the $ sign. For details on macros and parsers, see Section 4.7, Templates and macros and Section 4.8, Parsing messages.

Note

When a log statement includes multiple filter statements, syslog-ng sends a message to the destination only if all filters are true for the message. In other words, the filters are connected with the logical AND operator. In the following example, no message arrives to the destination, because the filters are exclusive (the hostname of a client cannot be example1 and example2 at the same time): filter demo_filter1 { host("example1"); }; filter demo_filter2 { host("example2"); }; log demo_filteredlog { source(s1); source(s2); filter(demo_filter1); filter(demo_filter2); destination(d1); destination(d2); }; To select the messages that come from either host example1 or example2, use a single filter expression: filter demo_filter { host("example1") or host("example2"); }; log demo_filteredlog { source(s1); source(s2); filter(demo_filter); destination(d1); destination(d2); }; Use the not operator to invert filters, for example, to select the messages that were not sent by host example1: filter demo_filter { not host("example1"); }; However, to select the messages that were not sent by host example1 or example2, you have to use the and operator (that's how boolean logic works): filter demo_filter { not host("example1") and not host("example2"); }; Alternatively, you can use parentheses to avoid this confusion: filter demo_filter { not (host("example1") or host("example2")); };

In the extended regular expressions, the characters ()[].*?+^$| are used as special symbols. Therefore, these characters have to be preceded with a backslash (\) if they are meant literally. For example, the \$40 expression matches the $40 string. Backslashes have to be escaped as well if they are meant literally. For example, the \\d expression matches the \d string.

	Tip
	If you use single quotes in, you do not need to escape the backslash, for example match("\\.") is equivalent to match('\.').

By default, all regular expressions are case sensitive. To disable the case sensitivity of the expression, add the flags(ignore-case) option to the regular expression.

filter demo_regexp_insensitive { host("system" flags(ignore-case)); };

For details on regular expressions, see Section 6.8, Regular expressions.

Note

In regular expressions, the asterisk (*) character means 0, 1 or any number of the previous expression. For example, in the f*ilter expression the asterisk means 0 or more f letters. This expression matches for the following strings: ilter, filter, ffilter, etc. To achieve the wildcard functionality commonly represented by the asterisk character in other applications, use .* in your expressions, for example f.*ilter.

The level() filter can select messages corresponding to a single importance level, or a level-range. To select messages of a specific level, use the name of the level as a filter parameter, for example use the following to select warning messages:

level(warning)

To select a range of levels, include the beginning and the ending level in the filter, separated with two dots (..). For example, to select every message of error or higher level, use the following filter:

level(err..emerg)

Similarly, messages sent by a range of facilities can also be selected. Note that this is only possible when using the name of the facilities. It is not possible to select ranges the numerical codes of the facilities.

facility(local0..local5)

For a complete list of the available levels and facilities, see Section 6.4, Filter functions.

For a complete description on the above functions, see Section 6.4, Filter functions.

4.6.2. Optimizing regular expressions in filters

Some filter functions accept regular expressions as parameters. But evaluating general regular expressions puts a high load on the CPU, which can cause problems when the message traffic is very high. Often the regular expression can be replaced with simple filter functions and logical operators. Using simple filters and logical operators, the same effect can be achieved at a much lower CPU load.

Example 4.31. Optimizing regular expressions in filters

Suppose you need a filter that matches the following error message logged by the xntpd NTP daemon: xntpd[1567]: time error -1159.777379 is too large (set clock manually); The following filter uses regular expressions and matches every instance and variant of this message. filter f_demo_regexp { program("demo_program") and match("time error .* is too large .* set clock manually"); }; Segmenting the match() part of this filter into separate match() functions greatly improves the performance of the filter. filter f_demo_optimized_regexp { program("demo_program") and match("time error") and match("is too large") and match("set clock manually"); };

4.6.3. Tagging messages

Starting with syslog-ng 3.1, it is also possible to label the messages with custom tags. Tags are simple labels, identified by their names, which must be unique. Currently syslog-ng can tag a message at two different places:

• at the source when the message is received; and

• when the message matches a pattern in the pattern database. For details on using the pattern database, see Section 4.9, Classifying messages, for details on creating tags in the pattern database, see Section 6.6.2.3, Creating pattern databases.

When syslog-ng receives a message, it automatically adds the .source.<id_of_the_source_statement> tag to the message. Use the tags() option of the source to add custom tags, and the tags() option of the filters to select only specific messages.

Note

Tagging messages and also filtering on the tags is very fast, much faster then other types of filters. Tags are available locally, that is, if you add tags to a message on the client, these tags will not be available on the server. To include the tags in the message, use the $TAGS macro in a template. Alternatively, if you are using the IETF-syslog message format, you can include the $TAGS macro in the .SDATA.meta part of the message. Note that the $TAGS macro is available only in syslog-ng OSE 3.1.1 and later.

	Example 4.32. Adding tags and filtering messages with tags
	source s_tcp { tcp(ip(192.168.1.1) port(1514) tags("tcp", "router")); }; Use the tags() option of the filters to select only specific messages: filter f_tcp { tags(".source.s_tcp"); }; filter f_router { tags("router"); };

The syslog-ng application allows you to define message templates, and reference them from every object that can use a template. Templates can be used to create standard message formats or filenames. Templates can reference one or more macros (for example date, the hostname, etc.). See Section 6.5, Macros for a list of macros available in syslog-ng Open Source Edition. Fields from the structured data (SD) part of messages using the new IETF-syslog standard can also be used as macros.

Template objects have a single option called template_escape, which is disabled by default (template_escape(no)). This behavior is useful when the messages are passed to an application that cannot handle escaped characters properly. Enabling template escaping (template_escape(yes)) causes syslog-ng to escape the ' and " characters from the messages.

	Note
	In versions 2.1 and earlier, the template_escape() option was enabled by default.

Macros can be included by prefixing the macro name with a $ sign, just like in Bourne compatible shells. Regarding braces around macro names, the following two formats are equivalent "$MSG" and "${MSG}".

Default values for macros can also be specified by appending the :- characters and the default value to the macro, for example

${HOST:-default_hostname}

The macros related to the date of the message (for example: ISODATE, HOUR, etc.) have two further versions each: one with the S_ and one with the R_ prefix (for example: S_DATE and R_DATE ). The S_DATE macro represents the date found in the log message, i.e. when the message was sent by the original application. R_DATE is the date when syslog has received the message.

DATE equals either S_DATE or R_DATE, depending on the global option set in the now deprecated use_time_recvd() parameter (see Section 6.9, Global options).

	Warning
	The hostname-related macros (FULLHOST, FULLHOST_FROM, HOST, and HOST_FROM) do not have any effect if the keep_hostname() option is disabled.

By default, syslog-ng sends messages using the following template: $ISODATE $HOST $MSGHDR$MSG\n. (The $MSGHDR$MSG part is written together because the $MSGHDR macro includes a trailing whitespace.)

	Note
	Earlier versions of syslog-ng used templates and scripts to send log messages into SQL databases. Starting from version 2.1, syslog-ng natively supports direct database access using the sql() destination. See Section 6.2.4, sql() for details.

Example 4.33. Using templates

The following template (t_demo_filetemplate) adds the date of the message and the name of the host sending the message to the beginning of the message text. The template is then used in a file destination: messages sent to this destination (d_file) will use the message format defined in the template. template t_demo_filetemplate { template("$ISODATE $HOST $MSG\n"); template_escape(no); }; destination d_file { file("/var/log/messages" template(t_demo_filetemplate)); }; Templates can also be used inline, if they are used only at a single location. The following destination is equivalent with the previous example: destination d_file { file ("/var/log/messages" template("$ISODATE $HOST $MSG\n") template_escape(no) ); };

The syslog-ng application can separate parts of log messages (i.e., the contents of the $MSG macro) to named fields (columns). These fields act as user-defined macros that can be referenced in message templates, file- and tablenames, etc.

Parsers are similar to filters: they must be defined in the syslog-ng configuration file and used in the log statement.

	Note
	The order of filters, rewriting rules, and parsers in the log statement is important, as they are processed sequentially.

To create a parser, define the columns of the message, the delimiter or separator characters, and optionally the characters that are used to escape the delimiter characters (quote-pairs). For the list of parser parameters, see Section 6.6, Message parsers.

Declaration:
            parser parser_name {
            csv-parser(column1, column2, ...)
            delimiters()
            quote-pairs()
            };

Column names work like macros. Always use a prefix to identify the columns of the parsers, for example MYPARSER1.COLUMN1, MYPARSER2.COLUMN2, etc. Column names starting with a dot (for example .HOST) are reserved for use by syslog-ng.

	Example 4.34. Segmenting hostnames separated with a dash
	The following example separates hostnames like example-1 and example-2 into two parts. parser p_hostname_segmentation { csv-parser(columns("HOSTNAME.NAME", "HOSTNAME.ID") delimiters("-") flags(escape-none) template("${HOST}")); }; destination d_file { file("/var/log/messages-${HOSTNAME.NAME:-examplehost}"); }; log { source(s_local); parser(p_hostname_segmentation); destination(d_file);};

Example 4.35. Parsing Apache log files

The following parser processes the log of Apache web servers and separates them into different fields. Apache log messages can be formatted like: "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T %v" Here is a sample message: 192.168.1.1 - - [31/Dec/2007:00:17:10 +0100] "GET /cgi-bin/example.cgi HTTP/1.1" 200 2708 "-" "curl/7.15.5 (i4 86-pc-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8c zlib/1.2.3 libidn/0.6.5" 2 example.balabit To parse such logs, the delimiter character is set to a single whitespace (delimiters(" ")). Whitespaces between quotes and brackets are ignored (quote-pairs('""[]')). parser p_apache { csv-parser(columns("APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME", "APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS", "APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT", "APACHE.PROCESS_TIME", "APACHE.SERVER_NAME") flags(escape-double-char,strip-whitespace) delimiters(" ") quote-pairs('""[]') ); }; The results can be used for example to separate log messages into different files based on the APACHE.USER_NAME field. If the field is empty, the nouser name is assigned. log { source(s_local); parser(p_apache); destination(d_file);}; }; destination d_file { file("/var/log/messages-${APACHE.USER_NAME:-nouser}"); };

Multiple parsers can be used to split a part of an already parsed message into further segments.

Example 4.36. Segmenting a part of a message

The following example splits the timestamp of a parsed Apache log message into separate fields. parser p_apache_timestamp { csv-parser(columns("APACHE.TIMESTAMP.DAY", "APACHE.TIMESTAMP.MONTH", "APACHE.TIMESTAMP.YEAR", "APACHE.TIMESTAMP.HOUR", "APACHE.TIMESTAMP.MIN", "APACHE.TIMESTAMP.MIN", "APACHE.TIMESTAMP.ZONE") delimiters("/: ") flags(escape-none) template("${APACHE.TIMESTAMP}")); }; log { source(s_local); log { parser(p_apache); parser(p_apache_timestamp); destination(d_file);}; };

To classify messages using a pattern database, include a db_parser() statement in your syslog-ng configuration file using the following syntax:

Declaration:
        parser <identifier> {db_parser(file("<database_filename>"));};

Note that using the parser in a log statement only performs the classification, but does not automatically do anything with the results of the classification.

	Example 4.37. Defining pattern databases
	The following statement uses the database located at /opt/syslog-ng/var/db/patterndb.xml. parser pattern_db { db_parser( file("/opt/syslog-ng/var/db/patterndb.xml") ); }; To apply the patterns on the incoming messages, include the parser in a log statement: log { source(s_all); parser(pattern_db); destination( di_messages_class); };

	Note
	The default location of the pattern database file is /opt/syslog-ng/var/run/patterndb.xml. The file option of the db-parser statement can be used to specify a different file, thus different db-parser statements can use different pattern databases. Later versions of syslog-ng will be able to dynamically generate a main database from separate pattern database files.

Example 4.38. Using classification results

The following destination separates the log messages into different files based on the class assigned to the pattern that matches the message (for example Violation and Security type messages are stored in a separate file), and also adds the ID of the matching rule to the message: destination di_messages_class { file("/var/log/messages-${.classifier.class}" template("${.classifier.rule_id};${S_UNIXTIME};${SOURCEIP};${HOST};${PROGRAM};${PID};${MSG}\n") template_escape(no) ); };

To create your own pattern databases see Section 6.6.2.3, Creating pattern databases.

4.9.2. Using parser results in filters and templates

4.9.2.1. Filtering messages based on classification

The results of message classification and parsing can be used in custom filters and file and database templates as well. There are two built-in macros in syslog-ng OSE that allow you to use the results of the classification: the .classifier.class macro contains the class assigned to the message (for example violation, security, or unknown), while the .classifier.rule_id macro contains the identifier of the message pattern that matched the message.

Example 4.39. Using classification results for filtering messages

To filter on a specific message class, create a filter that checks the .classifier_class macro, and use this filter in a log statement. filter fi_class_violation { match("violation" value(".classifier.class") type("string") ); }; log { source(s_all); parser(pattern_db); filter(fi_class_violation); destination(di_class_violation); }; Filtering on the unknown class selects messages that did not match any rule of the pattern database. Routing these messages into a separate file allows you to periodically review new or unknown messages. To filter on messages matching a specific classification rule, create a filter that checks the .classifier_rule_id macro. The unique identifier of the rule (for example e1e9c0d8-13bb-11de-8293-000c2922ed0a) is the id attribute of the rule in the XML database. filter fi_class_rule { match("e1e9c0d8-13bb-11de-8293-000c2922ed0a" value(".classifier_rule_id") type("string") ); };

The message-segments parsed by the pattern parsers can also be used as macros as well. To accomplish this, you have to add a name to the parser, and then you can use this name as a macro that refers to the parsed value of the message.

Example 4.40. Using pattern parsers as macros

For example, you want to parse messages of an application that look like "Transaction: <type>.", where <type> is a string that has different values (for example refused, accepted, incomplete, etc.). To parse these messages, you can use the following pattern: 'Transaction: @ESTRING::.@' Here the @ESTRING@ parser parses the message until the next full stop character. To use the results in a filter or a filename template, include a name in the parser of the pattern, for example: 'Transaction: @ESTRING:TRANSACTIONTYPE:.@' After that, add a custom template to the logpath that uses this template. For example, to select every accepted transaction, use the following custom filter in the log path: match("accepted" value("TRANSACTIONTYPE"));

	Note
	The above macros can be used in database columns and filename templates as well, if you create custom templates for the destination or logspace. Use a consistent naming scheme for your macros, for example, APPLICATIONNAME_MACRONAME.

The syslog-ng application can rewrite parts of log messages: it can search and replace text, and also set a specific field to a specified value. Rewriting messages is often used in conjunction with message parsing Section 4.8, Parsing messages.

Rewrite rules are similar to filters: they must be defined in the syslog-ng configuration file and used in the log statement.

	Note
	The order of filters, rewriting rules, and parsers in the log statement is important, as they are processed sequentially.

To create replace a part of the log message, define the string or regular expression to replace, the string to replace the original text (macros can be used as well), and the field of the message that the rewrite rule should process. Substitution rules can operate on any value available via macros, for example HOST, MESSAGE, PROGRAM, or any user-defined macros created using parsers (see Section 6.6, Message parsers for details). The only exceptions are the FACILITY, SEVERITY, TAGS, and the date-related fields, which cannot be rewritten. Substitution rules use the following syntax:

Declaration:
            rewrite <name_of_the_rule>
                {subst("<string or regular expression to find>", "<replacement string>", value(<field name>), flags());};

A single substitution rule can include multiple substitutions that are applied sequentially to the message. Note that rewriting rules must be included in the log statement to have any effect.

	Tip
	For case-insensitive searches, add the flags(ignore-case) option; to replace every occurrence of the string, add flags(global) option.

Example 4.41. Using substitution rules

The following example replaces the first occurrence of the string IP in the text of the message with the string IP-Address. rewrite r_rewrite_subst{subst("IP", "IP-Address", value("MESSAGE"));}; To replace every occurrence, use: rewrite r_rewrite_subst{subst("IP", "IP-Address", value("MESSAGE"), flags("global"));}; Multiple substitution rules are applied sequentially; the following rules replace the first occurrence of the string IP with the string IP-Addresses. rewrite r_rewrite_subst{subst("IP", "IP-Address", value("MESSAGE")); subst("Address", "Addresses", value("MESSAGE"));};

To set a field of the message to a specific value, define the string to include in the message, and the field where it should be included. Setting a field can operate on any value available via macros, for example HOST, MESSAGE, PROGRAM, or any user-defined macros created using parsers (see Section 6.6, Message parsers for details). The only exceptions are the FACILITY, SEVERITY, TAGS, and the date-related fields, which cannot be rewritten. Note that the rewrite operation completely replaces any previous value of that field. Use the following syntax:

Declaration:
            rewrite <name_of_the_rule>
                {set("<string to include>", value(<field name>));};

Example 4.42. Setting message fields to a particular value

The following example sets the HOST field of the message to myhost. rewrite r_rewrite_set{set("myhost", value("HOST"));}; The following example sets the sequence ID field of the RFC5424-formatted (IETF-syslog) messages to a fixed value. rewrite r_sd { set("55555" value(".SDATA.meta.sequenceId")); }; It is also possible to set the value of a field that does not exist yet, and create a new name-value pair that is associated with the message. The following example created the MODIFIED field and sets its value to yes. If you use the $MODIFIED macro in a template or SQL table, its value will be yes for every message that was processed with this rewrite rule, and empty for every other message. rewrite r_rewrite_set{set("yes", value("MODIFIED"));};

The syslog-ng application has a number of global options governing DNS usage, the timestamp format used, and other general points. Each option may have parameters, similarly to driver specifications. To set global options, add an option statement to the syslog-ng configuration file using the following syntax:

options { option1(params); option2(params); ... };

	Example 4.43. Using global options
	To disable domain name resolving, add the following line to the syslog-ng configuration file: options { use_dns(no); };

For a detailed list of the available options, see Section 6.9, Global options. See Chapter 5, Best practices and examples for important global options and recommendations on their use.

This section describes how to configure TLS encryption in syslog-ng. For the concepts of using TLS in syslog-ng, see Section 2.7, Secure logging using TLS.

Create an X.509 certificate for the syslog-ng server.

Note

The subject_alt_name parameter (or the Common Name parameter if the subject_alt_name parameter is empty) of the server's certificate must contain the hostname or the IP address (as resolved from the syslog-ng clients and relays) of the server (for example syslog-ng.example.com). Alternatively, the Common Name or the subject_alt_name parameter can contain a generic hostname, for example *.example.com. Note that if the Common Name of the certificate contains a generic hostname, do not specify a specific hostname or an IP address in the subject_alt_name parameter.

Complete the following steps on every syslog-ng client host. Examples are provided using both the legacy BSD-syslog protocol (using the tcp() driver) and the new IETF-syslog protocol standard (using the syslog() driver):

Complete the following steps on the syslog-ng server:

For the details of the available tls() options, see Section 6.10, TLS options.

This section describes how to configure mutual authentication between the syslog-ng server and the client. Configuring mutual authentication is similar to configuring TLS (see Section 4.12, Encrypting log messages with TLS), but the server verifies the identity of the client as well. Therefore, each client must have a certificate, and the server must have the certificate of the CA that issued the certificate of the clients. For the concepts of using TLS in syslog-ng, see Section 2.7, Secure logging using TLS.

Complete the following steps on every syslog-ng client host. Examples are provided using both the legacy BSD-syslog protocol (using the tcp() driver) and the new IETF-syslog protocol standard (using the syslog() driver):

Complete the following steps on the syslog-ng server:

For the details of the available tls() options, see Section 6.10, TLS options.

To configure syslog-ng on a client host, complete the following steps:

	Example 4.49. A simple configuration for clients
	The following is a simple configuration file that collects local log messages and forwards them to a logserver using the IETF-syslog protocol. @version:3.0 options { mark_freq(30); }; source s_local { unix-stream("/dev/log"); internal(); }; destination d_syslog_tcp { syslog("192.168.1.1" transport("tcp") port(2010)); }; log { source(s_local);destination(d_syslog_tcp); };

To configure syslog-ng on a relay host, complete the following steps:

In relay mode, syslog-ng cannot write messages received from network sources into files; the file() destination is disabled. The following sources are network sources: syslog(), tcp(), tcp6(), udp(), udp6().

Example 4.50. A simple configuration for relays

The following is a simple configuration file that collects local and incoming log messages and forwards them to a logserver using the IETF-syslog protocol. @version:3.0 options { mark_freq(30); keep_hostname(yes); }; source s_local { unix-stream("/dev/log"); internal(); }; source s_network { syslog(transport(tcp))}; destination d_syslog_tcp { syslog("192.168.1.5" transport("tcp") port(2010) ); }; log { source(s_local); source(s_network); destination(d_syslog_tcp); };

To configure syslog-ng on a server host, complete the following steps:

Example 4.51. A simple configuration for servers

The following is a simple configuration file for syslog-ng Open Source Edition that collects incoming log messages and stores them in a text file. @version:3.0 options { time_reap(30); mark_freq(10); keep_hostname(yes); }; source s_local { unix-stream("/dev/log"); internal();}; source s_network { syslog(transport(tcp))}; destination d_logs { file( "/var/log/syslog-ng/logs.txt" owner("root") group("root") perm(0777) ); }; log { source(s_local); source(s_network); destination(d_logs); };

This section provides tips and guidelines about troubleshooting problems related to syslog-ng.

Tip

As a general rule, first try to get logging the messages to a local file. Once this is working, you know that syslog-ng is running correctly and receiving messages, and you can proceed to forwarding the messages to the server. If the syslog-ng server does not receive the messages, use tcpdump or a similar packet sniffer tool on the client to verify that the messages are sent correctly, and on the server to verify that it receives the messages. If syslog-ng is closing the connections for no apparent reason, be sure to check the log messages of syslog-ng. You might also want to run syslog-ng with the --verbose or --debug command-line options for more-detailed log messages. Starting from syslog-ng OSE version 3.1, you can enable these messages without restarting syslog-ng using the syslog-ng-ctl verbose --set=on command. See the syslog-ng-ctl man page for details at syslog-ng-ctl(1). Similarly, build up encrypted connections step-by-step: first create a working unencrypted (for example TCP) connection, then add TLS encryption, and finally client authentication if needed.

This section provides general tips and recommendations on using syslog-ng. Some of the recommendations are detailed in the subsequent sections.

When syslog-ng is receiving messages from a large number of TCP or unix-stream connections, the CPU usage of syslog-ng might increase even if the number of messages is low. By default, syslog-ng processes every message when it is received. To reduce the CPU usage, process the incoming messages in batches. To accomplish this, instruct syslog-ng to wait for a short time before processing a message. During this period additional messages might arrive that can be processed together with the original message. To process log messages in batches, set the time_sleep() option (measured in milliseconds) to a non-zero value. Include the following line in your syslog-ng configuration:

options { time_sleep(20); };

	Note
	It is not recommended to increase the time_sleep() parameter above 100ms, as that might distort timestamps, slow down syslog-ng, and cause messages to be dropped. When modifying the time_sleep() option, also adjust the log_fetch_limit() and log_fifo_size() options accordingly.

The max_connections() parameter limits the number of parallel connections for the source.

If adjusting the time_sleep() option is not desired for some reason, an alternative solution is to use unix-stream(), udp() and unix-dgram() sources instead of tcp() connections.

This section provides tips on optimizing the performance of syslog-ng. Optimizing the performance is important for syslog-ng hosts that handle large traffic.

The syslog-ng application can resolve the hostnames of the clients and include them in the log messages. However, the performance of syslog-ng is severely degraded if the domain name server is unaccessible or slow. Therefore, it is not recommended to resolve hostnames in syslog-ng. If you must use name resolution from syslog-ng, consider the following:

	Note
	Domain name resolution is important mainly in relay and server mode.

To collect logs from a chroot using a syslog-ng client running on the host, complete the following steps:

Figure 5.1. Collecting logs from chroot

	Note
	You need to set up timezone information within your chroot as well. This usually means creating a symlink to /etc/localtime.

The syslog-ng application can replace both the syslogd and klogd daemons on Linux hosts. To replace klogd, complete the following steps:

If the clients run syslog-ng, then use the ISO timestamp, because it includes timezone information. That way you do not need to adjust the recv_time_zone() parameter of syslog-ng.

If you want syslog-ng to output timestamps in Unix (POSIX) time format, use the S_UNIXTIME and R_UNIXTIME macros. You do not need to change any of the timezone related parameters, because the timestamp information of incoming messages is converted to Unix time internally, and Unix time is a timezone-independent time representation. (Actually, Unix time measures the number of seconds elapsed since midnight of Coordinated Universal Time (UTC) January 1, 1970, but does not count leap seconds.)

To skip the processing of a message without sending it to a destination, create a log statement with the appropriate filters, but do not include any destination in the statement, and use the final flag.

	Example 5.1. Skipping messages
	The following log statement drops all debug level messages without any further processing. filter demo_debugfilter { level(debug); }; log { source(s_all); filter(demo_debugfilter); flags(final); };