The syslog-ng Premium Edition 3.2 Administrator Guide

The syslog-ng application is a flexible and highly scalable system logging application that is ideal for creating centralized and trusted logging solutions. The main features of syslog-ng are summarized below.

The syslog-ng application is not log analysis software. It can filter log messages and select only the ones matching certain criteria. It can even convert the messages and restructure them to a predefined format, or parse the messages and segment them into different fields. But syslog-ng cannot interpret and analyze the meaning behind the messages, or recognize patterns in the occurrence of different messages.

Log messages contain information about the events happening on the hosts. Monitoring system events is essential for security and system health monitoring reasons.

The original syslog protocol separates messages based on the priority of the message and the facility sending the message. These two parameters alone are often inadequate to consistently classify messages, as many applications might use the same facility — and the facility itself is not even included in the log message. To make things worse, many log messages contain unimportant information. The syslog-ng application helps you to select only the really interesting messages, and forward them to a central server.

Company policies or other regulations often require log messages to be archived. Storing the important messages in a central location greatly simplifies this process.

For details on how can you use syslog-ng to comply with various regulations, see the Regulatory compliance and system logging whitepaper available here.

Version 3.2 of syslog-ng Premium Edition includes the following main features. For a complete list of changes, see the release notes at the syslog-ng PE Download Page.

Version 3.2 of syslog-ng Premium Edition includes the following important changes:

The syslog-ng application is used worldwide by companies and institutions who collect and manage the logs of several hosts, and want to store them in a centralized, organized way. Using syslog-ng is particularly advantageous for:

The following is a list of public references — companies who use syslog-ng in their production environment:

The syslog-ng Premium Edition application is officially supported on the following platforms. Note that the following table is for general reference only, and is not always accurate about the supported platforms and options available for specific platforms. The latest version of this table is available here.

The central syslog-ng server cannot be installed on Microsoft Windows platforms. The syslog-ng Agent for Windows application is capable of forwarding eventlog messages to the central server. The syslog-ng Agent for Windows application is available as part of syslog-ng Premium Edition on the x86 and x86_64 architectures for the following operating systems:

For details about the syslog-ng Agent for Windows application, see The syslog-ng Agent for Windows 3.2 Administrator Guide.

The central syslog-ng server can be installed on the IBM System i platform, but the syslog-ng Agent for IBM System i is needed to collect the native logs of IBM System i. For details about the syslog-ng Agent for IBM System i, see The syslog-ng Agent for IBM System i Administrator Guide. The syslog-ng Agent for IBM System i is a commercial product independent from syslog-ng Premium Edition, and must be licensed separately.

Typically, syslog-ng is used to manage log messages and implement centralized logging, where the aim is to collect the log messages of several devices on a single, central log server. The different devices — called syslog-ng clients — all run syslog-ng, and collect the log messages from the various applications, files, and other sources. The clients send all important log messages to the remote syslog-ng server, where the server sorts and stores them.

The syslog-ng application reads incoming messages and forwards them to the selected destinations. The syslog-ng application can receive messages from files, remote hosts, and other sources.

Log messages enter syslog-ng in one of the defined sources, and are sent to one or more destinations.

Sources and destinations are independent objects; log paths define what syslog-ng does with a message, connecting the sources to the destinations. A log path consists of one or more sources and one or more destinations; messages arriving to a source are sent to every destination listed in the log path. A log path defined in syslog-ng is called a log statement.

Optionally, log paths can include filters. Filters are rules that select only certain messages, for example, selecting only messages sent by a specific application. If a log path includes filters, syslog-ng sends only the messages satisfying the filter rules to the destinations set in the log path.

Other optional elements that can appear in log statements are parsers and rewriting rules. Parsers segment messages into different fields to help processing the messages, while rewrite rules modify the messages by adding, replacing, or removing parts of the messages.

The following procedure illustrates the route of a log message from its source on the syslog-ng client to its final destination on the central syslog-ng server.

2.2.1. Procedure – The route of a log message in syslog-ng

Figure 2.1. The route of a log message

1. A device or application sends a log message to a source on the syslog-ng client. For example, an Apache web server running on Linux enters a message into the /var/log/apache file.

2. The syslog-ng client running on the web server reads the message from its /var/log/apache source.

3. The syslog-ng client processes the first log statement that includes the /var/log/apache source.

4. The syslog-ng client performs optional operations (message filtering, parsing, and rewriting) on the message; for example, it compares the message to the filters of the log statement (if any). If the message complies with all filter rules, syslog-ng sends the message to the destinations set in the log statement, for example, to the remote syslog-ng server.

   	Warning
   	Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.

   	Note
   	The syslog-ng client sends a message to all matching destinations by default. As a result, a message may be sent to a destination more than once, if the destination is used in multiple log statements. To prevent such situations, use the final flag in the destination statements. See Table 6.1, Log statement flags for details.

5. The syslog-ng client processes the next log statement that includes the /var/log/apache source, repeating Steps 3-4.

6. The message sent by the syslog-ng client arrives to a source set in the syslog-ng server.

7. The syslog-ng server reads the message from its source and processes the first log statement that includes that source.

8. The syslog-ng server performs optional operations (message filtering, parsing, and rewriting) on the message; for example, it compares the message to the filters of the log statement (if any). If the message complies with all filter rules, syslog-ng sends the message to the destinations set in the log statement.

   	Warning
   	Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.

9. The syslog-ng server processes the next log statement, repeating Steps 7-9.

	Note
	The syslog-ng application can stop reading messages from its sources if the destinations cannot process the sent messages. This feature is called flow-control and is detailed in Section 2.13, Managing incoming and outgoing messages with flow-control.

2.2.2. Embedded log statements

Starting from version 3.0, syslog-ng can handle embedded log statements (also called log pipes). Embedded log statements are useful for creating complex, multi-level log paths with several destinations and use filters, parsers, and rewrite rules.

For example, if you want to filter your incoming messages based on the facility parameter, and then use further filters to send messages arriving from different hosts to different destinations, you would use embedded log statements.

Figure 2.2. Embedded log statement

Embedded log statements include sources — and usually filters, parsers, rewrite rules, or destinations — and other log statements that can include filters, parsers, rewrite rules, and destinations. The following rules apply to embedded log statements:

• Only the beginning (also called top-level) log statement can include sources.

• Embedded log statements can include multiple log statements on the same level (that is, a top-level log statement can include two or more log statements).

• Embedded log statements can include several levels of log statements (that is, a top-level log statement can include a log statement that includes another log statement, and so on).

• Only another log statement can follow an embedded log statement, filters or other rules cannot.

• Embedded log statements that are on the same level receive the same messages from the higher-level log statement. For example, if the top-level log statement includes a filter, the lower-level log statements receive only the messages that pass the filter.

Figure 2.3. Embedded log statements

Embedded log filters can be used to optimize the processing of log messages, for example, to re-use the results of filtering and rewriting operations.

The syslog-ng Premium Edition application has three distinct modes of operation: Client, Server, and Relay. The syslog-ng application running on a host determines the mode of operation automatically based on the license and the configuration file.

	Note
	Microsoft Windows based hosts can run only the syslog-ng agent. The syslog-ng agent operates only in client mode.

2.3.1. Client mode

Figure 2.4. Client-mode operation

In client mode, syslog-ng collects the local logs generated by the host and forwards them through a network connection to the central syslog-ng server or to a relay. Clients can also log the messages locally into files.

No license file is required to run syslog-ng in client mode.

2.3.2. Relay mode

Figure 2.5. Relay-mode operation

In relay mode, syslog-ng receives logs through the network from syslog-ng clients and forwards them to the central syslog-ng server using a network connection. Relays can also log the messages from the relay host into a local file, or forward these messages to the central syslog-ng server.

Relays cannot write messages received from the network into local files, only buffer the messages to the hard disk when disk-based buffering is used.

No license file is required to run syslog-ng in relay mode.

2.3.3. Server mode

Figure 2.6. Server-mode operation

In server mode, syslog-ng acts as a central log-collecting server. It receives messages from syslog-ng clients and relays over the network, and stores them locally in files, or passes them to other applications, for example log analyzers.

Running syslog-ng Premium Edition in server mode requires a license file. The license determines how many individual hosts can connect to the server.

The syslog-ng application uses the following objects:

For details on the above objects, see Section 4.2, Defining global objects.

The syslog-ng application supports messages originating from different timezones. The original syslog protocol does not include timezone information, but syslog-ng provides a solution by extending the syslog protocol to include the timezone in the log messages. The syslog-ng application also enables administrators to supply timezone information for legacy devices which do not support the protocol extension.

Timezone information is associated with messages entering syslog-ng is selected using the following algorithm:

The syslog-ng application receives the timezone and daylight saving information from the operating system it is installed on. If the operating system handles daylight saving correctly, so does syslog-ng.

The syslog-ng application can send and receive log messages securely over the network using the Transport Layer Security (TLS) protocol. TLS is an encryption protocol over the TCP/IP network protocol, so it can be used only with TCP-based sources and destinations ( tcp() and tcp6()).

TLS uses certificates to authenticate and encrypt the communication, as illustrated on the following figure:

Figure 2.7. Certificate-based authentication

The client authenticates the server by requesting its certificate and public key. Optionally, the server can also request a certificate from the client, thus mutual authentication is also possible.

In order to use TLS encryption in syslog-ng, the following elements are required:

When using mutual authentication to verify the identity of the clients, the following elements are required:

Mutual authentication ensures that the syslog-ng server accepts log messages only from authorized clients.

For details on configuring TLS communication in syslog-ng, see Section 4.13, Encrypting log messages with TLS.

The Premium Edition of syslog-ng can store log messages securely in encrypted, compressed and timestamped binary files. Timestamps can be requested from an external Timestamping Authority (TSA).

Logstore files consist of individual chunks, every chunk can be encrypted, compressed, and timestamped separately. Chunks contain compressed log messages and header information needed for retrieving messages from the logstore file.

The syslog-ng PE application generates an SHA-1 hash for every chunk to verify the integrity of the chunk. The hashes of the chunks are chained together to prevent injecting chunks into the logstore file. The syslog-ng application can encrypt the logstore using various algorithms, using the aes128 encryption algorithm in CBC mode and the hmac-sha1 hashing (HMAC) algorithm as default.

2.8.1. Journal files

Starting with syslog-ng Premium Edition 3.2, syslog-ng PE processes log messages into a journal file before writing them to the logstore file. That way logstore files are consistent even if syslog-ng PE crashes unexpectedly, avoiding losing messages. Note that this does not protect against losing messages if the operating system crashes.

A journal file is automatically created for every logstore file that syslog-ng PE opens. A journal file consists of journal blocks which store the log messages. When a journal block fills up with messages, syslog-ng PE writes the entire block into the logstore file and starts to reuse the journal block (one journal block becomes one chunk in the logstore file). If the messages cannot be written to the logstore file (for example, because the disk becomes unaccessible, or file operations are slow), messages are put to the next journal block (syslog-ng PE uses four blocks by default). When all journal blocks become full, syslog-ng PE will stop processing incoming traffic. syslog-ng PE starts accepting messages to the logstore file again when the first journal block is successfully written to the logstore file. If syslog-ng PE receives a HUP or STOP signal, or no new message arrives into the logstore for the period set in the time_reap() parameter, it writes every journal block to the logstore.

Warning

If a particular logstore destination receives messages at a constant but very low message rate (for example, a 100-byte message every 30 seconds), messages do not get written to the logstore file for a long time, because the journal block does not get full, and messages are more frequent than the time_reap() time. This becomes a problem when using logrotate to rotate the logstore files, because log messages will not be in the files they are expected. To avoid this situation, either use time-based macros in the filenames of the logstore files, or send a HUP signal to syslog-ng PE right before rotating the logstore files. When every block of a journal becomes full and syslog-ng PE stops processing incoming traffic, it will not read new messages at all until a block is successfully written to the related logstore file. This is in contrast with flow-control, where only messages from the source related to the particular destination are not processed. The messages in the journal file are in plain-text format: they are neither encrypted nor compressed. However, this is not a problem because normally the journal file is available only in the memory of the host, it is written to disk only if syslog-ng PE crashes. When syslog-ng PE is restarted, it automatically processes the journal files to the logstore files, unless a particular logstore file is not part of configuration of syslog-ng PE. Such orphaned journal files can be processed with the lgstool recover command. For details on processing orphaned journal files, see the section called “The recover command”.

	Note
	Journal files are located in the same folder as the logstore file. The name of the journal file is the same as the logstore file with .jor suffix added. For example, the journal file for messages.lgsis messages.lgs.jor.

The syslog-ng PE application uses a separate journal file for every logstore file; every journal file is processed by a separate thread. The journal files are mapped into the memory. The journal of an individual logstore file uses up to journal_block_size()*journal_block_count() memory address, which is 4MB by default. However, if you have several logstore files open in parallel (for example, you are collecting log messages from 500 hosts and storing them in separate files for every host, and the hosts are continuously sending messages) the memory requirements for journaling rise quickly (to ~2GB for the 500 hosts). To limit the memory use of journals, adjust the logstore_journal_shmem_threshold() global option (by default, it is 512 MB).

If the memory required for the journal files exceeds the logstore_journal_shmem_threshold() limit, syslog-ng PE will store only a single journal block of every journal file in the memory, and — if more blocks are needed for a journal — store the additional blocks on the hard disk. Opening new logstore files means allocating memory for one new journal block for every new file. In extreme situations involving large traffic, this can lead to syslog-ng PE consuming the entire memory of the system. Adjust the journal_block_size() and your file-naming conventions as needed to avoid such situations. For details on logstore journals, see Section 2.8.1, Journal files.

	Warning
	If you have a large amount of open logstore files in parrallel (for example, you are using the $HOST or $PROGRAM macros in your filenames) consider lowering the journal_block_size() to avoid syslog-ng PE consuming the entire memory of the system.

Example 2.1. Calculating memory usage of logstore journals

If you are using the default settings (4 journal blocks for every logstore journal, one block is 1MB, logstore_journal_shmem_threshold() is 512MB), this means that syslog-ng PE will allocate 4MB memory for every open logstore file, up to 512MB if you have 128 open logstore files. Opening a new logstore file would require 4 more megabytes of memory for journaling, bringing the total required memory to 516MB, which is above the logstore_journal_shmem_threshold(). In this case, syslog-ng PE switches to storing only a single journal block in the memory, lowering the memory requirements of journaling to 129MB. However, opening more and more logstore files wil require more and more memory, and this is not limited, except when syslog-ng PE reaches the maximum number of files that can be open (as set in the --fd-limit command-line option).

The syslog-ng application can dynamically create filenames, directories, or names of database tables using macros that help you organize your log messages. Macros refer to a property or a part of the log message, for example, the $HOST macro refers to the name or IP address of the client that sent the log message, while $DAY is the day of the month when syslog-ng has received the message. Using these macros in the path of the destination log files allows you for example to collect the logs of every host into separate files for every day.

A set of macros can be defined as a template object and used in multiple destinations.

Another use of macros and templates is to customize the format of the syslog message, for example to add elements of the message header to the message text. Note that if a message uses the IETF-syslog format, only the text of the message can be customized, the structure of the header is fixed.

For details on using templates and macros, see Section 4.7, Templates and macros and Section 6.5, Macros.

The filters and default macros of syslog-ng work well on the headers and metainformation of the log messages, but are rather limited when processing the content of the messages. Parsers can segment the content of the messages into name-value pairs, and these names can be used as user-defined macros. Subsequent filtering or other type of processing of the message can use these custom macros to refer to parts of the message.

Parsers are global objects most often used together with filters and rewrite rules. For details on using parsers, see Section 4.8, Parsing messages and Section 6.6, Message parsers.

The syslog-ng application can rewrite parts of the messages using rewrite rules. Rewrite rules are global objects similar to parsers and filters and can be used in log paths. The syslog-ng application has two methods to rewrite parts of the log messages: replacing (setting) a part of the message to a fix value, and a general search-and-replace mode.

Substitution completely replaces a specific part of the message that is referenced using a built-in or user-defined macro.

General rewriting searches for a string in the entire message (or only a part of the message specified by a macro) and replaces it with another string. Optionally, this replacement string can be a template that contains macros.

For details on using rewrite rules, see Section 4.10, Rewriting messages and Section 6.7, Rewriting messages.

The syslog-ng application can compare the contents of the received log messages to predefined message patterns. By comparing the messages to the known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and for example can label the messages as user login, application crash, file transfer, and so on events.

To find the pattern that matches a particular message, syslog-ng uses a method called longest prefix match radix tree. This means that syslog-ng creates a tree structure of the available patterns, where the different characters available in the patterns for a given position are the branches of the tree.

To classify a message, syslog-ng selects the first character of the message (the text of message, not the header), and selects the patterns starting with this character, other patterns are ignored for the rest of the process. After that, the second character of the message is compared to the second character of the selected patterns. Again, matching patterns are selected, and the others discarded. This process is repeated until a single pattern completely matches the message, or no match is found. In the latter case, the message is classified as unknown, otherwise the class of the matching pattern is assigned to the message.

To make the message classification more flexible and robust, the patterns can contain pattern parsers: elements that match on a set of characters. For example, the NUMBER parser matches on any integer or hexadecimal number (for example 1, 123, 894054, 0xFFFF, etc.). Other pattern parsers match on various strings and IP addresses. For the details of available pattern parsers, see Section 6.6.2.1, Using pattern parsers.

The functionality of the pattern database is similar to that of the logcheck project, but it is much easier to write and maintain the patterns used by syslog-ng, than the regular expressions used by logcheck. Also, it is much easier to understand syslog-ng pattens than regular expressions.

Pattern matching based on regular expressions is computationally very intensive, especially when the number of patterns increases. The solution used by syslog-ng can be performed real-time, and is independent from the number of patterns, so it scales much better. The following patterns describe the same message: Accepted password for bazsi from 10.50.0.247 port 42156 ssh2

A regular expression matching this message from the logcheck project: Accepted \ (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam) \ for [^[:space:]]+ from [^[:space:]]+ port [0-9]+( (ssh|ssh2))?

A syslog-ng database pattern for this message: Accepted @QSTRING:auth_method: @ for@QSTRING:username: @from\ @QSTRING:client_addr: @port @NUMBER:port:@ ssh2

For details on using pattern databases to classify log messages, see Section 4.9, Classifying messages and Section 6.6.2, Pattern databases.

2.12.1. The structure of the pattern database

The pattern database is organized as follows:

Figure 2.8. The structure of the pattern database

• The pattern database consists of rulesets. A ruleset consists of a Program Pattern and a set of rules: the rules of a ruleset are applied to log messages if the name of the application that sent the message matches the Program Pattern of the ruleset. The name of the application (the content of the $PROGRAM macro) is compared to the Program Patterns of the available rulesets, and then the rules of the matching rulesets are applied to the message.

• The Program Pattern can be a string that specifies the name of the appliation or the beginning of its name (for example to match for sendmail, the program pattern can be sendmail, or just send), and the Program Pattern can contain pattern parsers. Note that pattern parsers are completely independent from the syslog-ng parsers used to segment messages. Additionally, every rule has a unique identifier: if a message matches a rule, the identifier of the rule is stored together with the message.

• Rules consist of a message pattern and a class. The Message Pattern is similar to the Program Pattern, but is applied to the message part of the log message (the content of the $MESSAGE macro). If a message pattern matches the message, the class of the rule is assigned to the message (for example Security, Violation, and so on).

• Rules can also contain additional information about the matching messages, such as the description of the rule, an URL, name-value pairs, or free-form tags.

• Patterns can consist of literals (keywords, or rather, keycharacters) and pattern parsers.

  	Note
  	If the $PROGRAM part of a message is empty, rules with an empty Program Pattern are used to classify the message. If the same Program Pattern is used in multiple rulesets, the rules of these rulesets are merged, and every rule is used to classify the message. Note that message patterns must be unique within the merged rulesets, but the currently only one ruleset is checked for uniqueness.

2.12.2. Pattern matching

Figure 2.9. Applying patterns

The followings describe how patterns work. This information applies to program patterns and message patterns alike, even though message patterns are used to illustrate the procedure.

Patterns can consist of literals (keywords, or rather, keycharacters) and pattern parsers. Pattern parsers attempt to parse a sequence of characters according to certain rules.

	Note
	Wildcards and regular expressions cannot be used in patterns. The @ character must be escaped, that is, to match for this character, you have to write @@ in your pattern. This is required because pattern parsers of syslog-ng are enclosed between @ characters.

When a new message arrives, syslog-ng attempts to classify it using the pattern database. The available patterns are organized alphabetically into a tree, and syslog-ng inspects the message character-by-character, starting from the beginning. This approach ensures that only a small subset of the rules must be evaluated at any given step, resulting in high processing speed. Note that the speed of classifying messages is practically independent from the total number of rules.

For example, if the message begins with the Apple string, only patterns beginning with the character A are considered. In the next step, syslog-ng selects the patterns that start with Ap, and so on, until there is no more specific pattern left.

Note that literal matches take precedence over pattern parser matches: if at a step there is a pattern that matches the next character with a literal, and another pattern that would match it with a parser, the pattern with the literal match is selected. Using the previous example, if at the third step there is the literal pattern Apport and a pattern parser Ap@STRING@, the Apport pattern is matched, even if the pattern parser would result in a better match.

If there are two parsers at the same level (for example Ap@STRING@ and Ap@QSTRING@), it is random which pattern is applied (technically, the one that is loaded first). However, if the selected parser cannot parse at least one character of the message, the other parser is used. But having two different parsers at the same level is extremely rare, so the impact of this limitation is much less than it appears.

This section describes the internal message-processing model of syslog-ng, as well as the flow-control feature that can prevent message losses. To use flow-control, the flow-control flag must be enabled for the particular log path.

The syslog-ng application monitors (polls) the sources defined in its configuration file, periodically checking each source for messages. When a log message is found in one of the sources, syslog-ng polls every source and reads the available messages. These messages are processed and put into the output buffer of syslog-ng (also called fifo). From the output buffer, the operating system sends the messages to the appropriate destinations.

In large-traffic environments many messages can arrive during a single poll loop, therefore syslog-ng reads only a fixed number of messages from each source. The log_fetch_limit() option specifies the number of messages read during a poll loop from a single source.

Figure 2.10. Managing log messages in syslog-ng

	Note
	The log_fetch_limit() parameter can be set as a global option, or for every source individually.

Every destination has its own output buffer. The output buffer is needed because the destination might not be able to accept all messages immediately. The log_fifo_size() parameter sets the size of the output buffer. The output buffer must be larger than the log_fetch_limit() of the sources, to ensure that every message read during the poll loop fits into the output buffer. If the log path sends messages to a destination from multiple sources, the output buffer must be large enough to store the incoming messages of every source.

TCP and unix-stream sources can receive the logs from several incoming connections (for example many different clients or applications). For such sources, syslog-ng reads messages from every connection, thus the log_fetch_limit() parameter applies individually to every connection of the source.

Figure 2.11. Managing log messages of TCP sources in syslog-ng

The flow-control of syslog-ng introduces a control window to the source that tracks how many messages can syslogng accept from the source. Every message that syslog-ng reads from the source decreases the number of free slots by one; every message that syslog-ng successfully sends from the output buffer increases the number of free slots by one. If the window is full (that is, there are no free slots), syslog-ng stops reading messages from the source. The initial size of the control window is by default 100: the log_fifo_size() must be larger than this value in order for flow-control to have any effect. If a source accepts messages from multiple connections, all messages use the same control window.

When flow-control is used, every source has its own control window. As a worst-case situation, the output buffer of the destination must be set to accommodate all messages of every control window, that is, the log_fifo_size() of the destination must be greater than number_of_sources*log_iw_size(). This applies to every source that sends logs to the particular destination. Thus if two sources having several connections and heavy traffic send logs to the same destination, the control window of both sources must fit into the output buffer of the destination. Otherwise, syslog-ng does not activate the flow-control, and messages may be lost.

	Note
	Flow-control can be used together with the disk-based buffering feature of syslog-ng PE. For details, see Section 2.14, Using disk-based buffering.

2.13.1. Flow-control and multiple destinations

Using flow-control on a source has an important side-effect if the messages of the source are sent to multiple destinations. If flow-control is in use and one of the destinations cannot accept the messages, the other destinations do not receive any messages either, because syslog-ng stops reading the source. For example, if messages from a source are sent to a remote server and also stored locally in a file, and the network connection to the server becomes unavailable, neither the remote server nor the local file will receive any messages. This side-effect of the flow-control can be avoided by using the disk-based buffering feature of syslog-ng Premium Edition.

	Note
	Creating separate log paths for the destinations that use the same flow-controlled source does not avoid the problem.

The Premium Edition of syslog-ng stores messages on the local hard disk if the central log server or the network connection to the server becomes unavailable. The syslog-ng application automatically sends the stored messages to the server when the connection is reestablished. The disk buffer is used as a queue: when the connection to the server is reestablished, syslog-ng sends the messages to the server in the order they were received.

	Note
	Disk-based buffering can be used in conjunction with flow-control. For details, see Section 2.13, Managing incoming and outgoing messages with flow-control.

Disk buffers can be used with tcp(), tcp6(), syslog() (when using the tcp or tls transport methods), and sql() destinations. Every such destination uses a separate disk buffer (similarly to the output buffers controlled by log_fifo_size()). The hard disk space is not pre-allocated, so ensure that there is always enough free space to store the disk buffers even when the disk buffers are full.

If syslog-ng is restarted (using the /etc/init.d/syslog-ng restart command), it automatically saves any unsent messages of the disk buffer and the output queue. After the restart, syslog-ng sends the saved messages to the server. In other words, the disk buffer is persistent. The disk buffer is also resistant to syslog-ng crashes.

The syslog-ng application handles outgoing messages the following way:

Figure 2.12. Handling outgoing messages in syslog-ng PE

Starting with syslog-ng Premium Edition 3.2., syslog-ng PE can detect if the remote server of a network destination becomes unaccessible, and start sending messages to a secondary server. Multiple failover servers can be configured; so if the secondary server becomes unaccessible as well, syslog-ng PE will switch to the third server in the list, and so on. If there are no more failover servers left, syslog-ng PE returns to the beginning of a list and attempts to connect to the primary server.

When syslog-ng PE starts up, it will always try to connect to the primary server first, but once it fails over to a secondary server, it will not automatically attempt to return to the primary server even if it becomes available. However, if the configuration of syslog-ng PE is reloaded, it will attempt to connect the primary server.

If syslog-ng PE uses TLS-encryption to communicate with the remote server, syslog-ng PE checks the certificate of the failover server as well. The certificates of the failover servers should match their domain names or IP addresses — for details, see Section 4.13, Encrypting log messages with TLS. Note that when mutual authentication is used, the syslog-ng PE client sends the same certificate to every server.

The primary server and the failover servers must be accessible with the same communication method: it is not possible to use different destination drivers or options for the different servers.

	Note
	Client-side failover works only for TCP-based connections, that is, the tcp() (including TLS-encrypted connections) and the syslog() destination driver (excluding UDP transport). Client-side failover is not supported in the sql() driver, even though it may use a TCP connection to access a remote database.

For details on configuring failover servers, see Example 6.34, Specifying failover servers for tcp() destinations and Example 6.30, Specifying failover servers for syslog() destinations.

As of October 2009, the following release policy applies to syslog-ng Premium Edition:

Note

Releases of the feature branch are tested just like the stable releases; they are not "unstable" development snapshots. The difference between earlier major releases and current feature releases is the smaller number of features contained in a release, and the shorter support periods. If an unstable snapshot or alpha/beta/rc release is released for public testing, it is always marked explicitly as such.

Warning

Downgrading from a feature release to an earlier (and thus unsupported) feature release, or to the stable release is not supported: this means that once you upgrade a system from a stable release (for example 1.0) to a feature release (for example 1.1), you will have to keep upgrading to the new feature releases until the next stable version release (for example 2.0) is published, or risk using an unsupported product.

The syslog-ng Premium Edition application is licensed on a per-host basis: the syslog-ng server accepts connections only from the number of individual hosts (also called log source hosts) specified in its license file.

A log source host is a host or network device (including syslog-ng clients and relays) that sends logs to the syslog-ng server. Log source hosts can be servers, routers, desktop computers, or other devices capable of sending syslog messages or running syslog-ng. Log source hosts are identified by their IP addresses, so virtual machines and vhosts are separately counted. Licenses are available for 5, 10, 25, 50, 100, 150, 200, 250, 300, 500, 750, 1000, and unlimited number of log source hosts.

Warning

If the actual IP address of the host differs from the IP address received by looking up its IP address from its hostname in the DNS, the syslog-ng server counts them as two different hosts. The chain_hostnames() option of syslog-ng can interfere with the way syslog-ng counts the log source hosts, causing syslog-ng to think there are more hosts logging to the central server. As chain_hostnames() is a deprecated option, disable it on your log sources to avoid any problems related to license counting.

Buying a syslog-ng server license permits you to perform the following:

Example 2.2. Counting log source hosts

Let's say that you have two facilities (for example data centers or server farms), and you have 80 AIX servers and 20 Microsoft Windows host at Facility 1, and 5 HP-UX servers and 40 Debian servers at Facility 2. That is 145 hosts altogether. If you want to collect the log messages of these host to a single logserver, then you need a syslog-ng PE license that allows you to accept logs from at least 145 hosts. (In practice this means you have to buy a license for 150 hosts.) If you want each facility to have its own logserver, and do not want to have a central server that collects the log messages of both facilities, you need two separate licenses: a license for 100 hosts at Facility 1, and a license for at least 45 hosts at Facility 2 (actually you have to buy license for 50 hosts). If you want each facility to have its own local logserver that stores the logs locally, and also want to have a central logserver that collects every log message independently from the two local logserver, you need three licenses: a license for 100 hosts at Facility 1, and a license for at least 45 hosts at Facility 2, and a license for the central logserver. The size of the license on the central logserver should be 100 (the hosts at Facility 1) + 45 (the hosts at Facility 2) + 2 (the two local logservers at each facility) = 147 — practically thats another 150-host license. Note If, for example, the 40 Debian servers at Facility 2 are each running 3 virtual hosts, then the total number of hosts at Facility 2 is 125, and the license sizes should be calculated accordingly.

Multiple syslog-ng servers can be run in fail-over mode. The syslog-ng application does not include any internal support for this, as clustering support must be implemented on the operating system level. A tool that can be used to create UNIX clusters is Heartbeat (for details, see Linux-HA).

During the course of a message from the sending application to the final destination of the message, there are a number of locations where a message may be lost, even though syslog-ng does its best to avoid message loss. Usually losing messages can be avoided with careful planning and proper configuration of syslog-ng and the hosts running syslog-ng. The following list shows the possible locations where messages may be lost, and provides methods to minimize the risk of losing messages.

	Note
	The following list covers the main possibilities of losing messages, but does not take into account the possible use of flow-control (see Section 2.13, Managing incoming and outgoing messages with flow-control). This topic will be addressed in more detail in the future releases of this guide.

The following sections describe the structure of log messages. Currently there are two standard syslog message formats:

2.20.1. BSD-syslog or legacy-syslog messages

This section describes the format of a syslog message, according to the legacy-syslog or BSD-syslog protocol (see RFC 3164). A syslog message consists of the following parts:

• PRI

• HEADER

• MSG

The total message cannot be longer than 1024 bytes.

The following is a sample syslog message: <133>Feb 25 14:09:07 webserver syslogd: restart. The message corresponds to the following format: <priority>timestamp hostname application: message. The different parts of the message are explained in the following sections.

	Note
	The syslog-ng application supports longer messages as well. For details, see the log_msg_size() option in Section 6.9, Global options. However, it is not recommended to enable messages larger than the packet size when using UDP destinations.

2.20.1.1. The PRI message part

The PRI part of the syslog message (known as Priority value) represents the Facility and Severity of the message. Facility represents the part of the system sending the message, while severity marks its importance. The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity. The possible facility and severity values are presented below.

	Note
	Facility codes may slightly vary between different platforms. The syslog-ng application accepts facility codes as numerical values as well.

Numerical Code	Facility
0	kernel messages
1	user-level messages
2	mail system
3	system daemons
4	security/authorization messages
5	messages generated internally by syslogd
6	line printer subsystem
7	network news subsystem
8	UUCP subsystem
9	clock daemon
10	security/authorization messages
11	FTP daemon
12	NTP subsystem
13	log audit
14	log alert
15	clock daemon
16-23	locally used facilities (local0-local7)

Table 2.1. syslog Message Facilities

The following table lists the severity values.

Numerical Code	Severity
0	Emergency: system is unusable
1	Alert: action must be taken immediately
2	Critical: critical conditions
3	Error: error conditions
4	Warning: warning conditions
5	Notice: normal but significant condition
6	Informational: informational messages
7	Debug: debug-level messages

Table 2.2. syslog Message Severities

2.20.1.2. The HEADER message part

The HEADER part contains a timestamp and the hostname (without the domain name) or the IP address of the device. The timestamp field is the local time in the Mmm dd hh:mm:ss format, where:

• Mmm is the English abbreviation of the month: Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec.

• dd is the day of the month on two digits. If the day of the month is less than 10, the first digit is replaced with a space. (For example Aug 7.)

• hh:mm:ss is the local time. The hour (hh) is represented in a 24-hour format. Valid entries are between 00 and 23, inclusive. The minute (mm) and second (ss) entries are between 00 and 59 inclusive.

	Note
	The syslog-ng application supports other timestamp formats as well, like ISO, or the PIX extended format. For details, see the ts_format() option in Section 6.9, Global options.

2.20.1.3. The MSG message part

The MSG part contains the name of the program or process that generated the message, and the text of the message itself. The MSG part is usually in the following format: program[pid]: message text.

2.20.2. IETF-syslog messages

This section describes the format of a syslog message, according to the IETF-syslog protocol (see RFC 5424-5428).A syslog message consists of the following parts:

• HEADER (includes the PRI as well)

• STRUCTURED-DATA

• MSG

The following is a sample syslog message:^[1]

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

The message corresponds to the following format:

<priority>VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG

In this example, the Facility has the value of 4, severity is 2, so PRI is 34. The VERSION is 1. The message was created on 11 October 2003 at 10:14:15pm UTC, 3 milliseconds into the next second. The message originated from a host that identifies itself as "mymachine.example.com". The APP-NAME is "su" and the PROCID is unknown. The MSGID is "ID47". The MSG is "'su root' failed for lonvick...", encoded in UTF-8. The encoding is defined by the BOM. There is no STRUCTURED-DATA present in the message, this is indicated by "-" in the STRUCTURED-DATA field. The MSG is "'su root' failed for lonvick...".

The HEADER part of the message must be in plain ASCII format, the parameter values of the STRUCTURED-DATA part must be in UTF-8, while the MSG part should be in UTF-8. The different parts of the message are explained in the following sections.

2.20.2.1. The PRI message part

The PRI part of the syslog message (known as Priority value) represents the Facility and Severity of the message. Facility represents the part of the system sending the message, while severity marks its importance. The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity. The possible facility and severity values are presented below.

	Note
	Facility codes may slightly vary between different platforms. The syslog-ng application accepts facility codes as numerical values as well.

Numerical Code	Facility
0	kernel messages
1	user-level messages
2	mail system
3	system daemons
4	security/authorization messages
5	messages generated internally by syslogd
6	line printer subsystem
7	network news subsystem
8	UUCP subsystem
9	clock daemon
10	security/authorization messages
11	FTP daemon
12	NTP subsystem
13	log audit
14	log alert
15	clock daemon
16-23	locally used facilities (local0-local7)

Table 2.3. syslog Message Facilities

The following table lists the severity values.

Numerical Code	Severity
0	Emergency: system is unusable
1	Alert: action must be taken immediately
2	Critical: critical conditions
3	Error: error conditions
4	Warning: warning conditions
5	Notice: normal but significant condition
6	Informational: informational messages
7	Debug: debug-level messages

Table 2.4. syslog Message Severities

2.20.2.2. The HEADER message part

The HEADER part contains the following elements:

• VERSION: Version number of the syslog protocol standard. Currently this can only be 1.

• ISOTIMESTAMP: The time when the message was generated in the ISO 8601 compatible standard timestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example: 2006-06-13T15:58:00.123+01:00.

• HOSTNAME: The machine that originally sent the message.

• APPLICATION: The device or application that generated the message

• PID: The process name or process ID of the syslog application that sent the message. It is not necessarily the process ID of the application that generated the message.

• MESSAGEID: The ID number of the message.

	Note
	The syslog-ng application supports other timestamp formats as well, like ISO, or the PIX extended format. The timestamp used in the IETF-syslog protocol is derived from RFC3339, which is based on ISO8601. For details, see the ts_format() option in Section 6.9, Global options.

2.20.2.3. The STRUCTURED-DATA message part

The STRUCTURED-DATA message part may contain meta- information about the syslog message, or application-specific information such as traffic counters or IP addresses. STRUCTURED-DATA consists of data blocks enclosed in brackets ([]). Every block include the ID of the block, and one or more name=value pairs. The syslog-ng application automatically parses the STRUCTURED-DATA part of syslog messages, which can be referenced in macros (for details, see Section 6.5, Macros). An example STRUCTURED-DATA block looks like:

[exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"]

2.20.2.4. The MSG message part

The MSG part contains the text of the message itself. The encoding of the text must be UTF-8 if the BOM character is present in the message. If the message does not contain the BOM character, the encoding is treated as unknown. Usually messages arriving from legacy sources do not include the BOM character.

This section describes how to install the syslog-ng application interactively using the binary installer. The installer has a simple interface: use the TAB or the arrow keys of your keyboard to navigate between the options, and Enter to select an option.

	Note
	The installer stops the running syslogd application if it is running, but its components are not removed. The /etc/init.d/sysklogd init script is automatically renamed to /etc/init.d/sysklogd.backup. Rename this file to its original name if you want to remove syslog-ng or restart the syslogd package.

Note

The .run installer copies temporary files under the /tmp directory, requiring about 15-70MB free space. If there is not enough free space under /tmp, before starting the installer, set a different directory using the TMPDIR environment variable. In Bourne and Korn-compatible shells, issue the following commands: # TMPDIR=/var/tmp # export TMPDIR In C shell, issue the following command: # setenv TMPDIR /var/tmp

3.1.1. Installing syslog-ng in client or relay mode

Complete the following steps to install syslog-ng Premium Edition on clients or relays. For details on the different operation modes of syslog-ng, see Section 2.3, Modes of operation.

3.1.1.1. Procedure – Installing syslog-ng in client or relay mode

1. Login to your MyBalabit account and download the syslog-ng installer package.

2. Enable the executable attribute for the installer using the chmod +x syslog-ng-<edition>-<version>-<OS>-<platform>.run, then start the installer as root using the ./syslog-ng-<edition>-<version>-<OS>-<platform>.run command. (Note that the exact name of the file depends on the operating system and platform.) Wait until the package is uncompressed and the welcome screen appears, then select Continue.

   

   Figure 3.1. The welcome screen

   
   

3. Accepting the EULA: You can install syslog-ng only if you understand and accept the terms of the End-User License Agreement (EULA). The full text of the EULA can be displayed during installation by selecting the Show EULA option, and is also available in this guide for convenience at Appendix 2, BalaBit syslog-ng Premium Edition License contract . Select Accept to accept the EULA and continue the installation.

   If you do not accept the terms of the EULA for some reason, select Reject to cancel installing syslog-ng.

4. Detecting platform and operating system: The installer attempts to automatically detect your oprating system and platform. If the displayed information is correct, select Yes. Otherwise select Exit to abort the installation, and verify that your platform is supported. For a list of supported platforms, see Section 1.6, Supported platforms. If your platform is supported but not detected correctly, contact your local distributor, reseller, or the BalaBit Support Team. For contact details, see Section 5, Contact and support information.

   

   Figure 3.2. Platform detection

   
   

5. Locating the license: Since you are installing syslog-ng in client or relay mode, simply select OK. For details on the different operation modes of syslog-ng, see Section 2.3, Modes of operation.

6. Upgrading: The syslog-ng installer can automatically detect if you have previously installed a version of syslog-ng on your system. To use the configuration file of this previous installation, select Yes. To ignore the old configuration file and create a new one, select No.

   Note that if you decide to use your existing configuration file, the installer automatically checks it for syntax error and displays a list of warnings and errors if it finds any problems.

   

   Figure 3.3. Upgrading syslog-ng

   
   

7. Generating a new configuration file: The installer displays some questions to generate a new configuration file.

   1. Remote sources: Select Yes to accept log messages from the network. TCP, UDP, and SYSLOG messages on every interface will be automatically accepted.

      

      Figure 3.4. Accepting remote messages

      
      

   2. Remote destinations: Enter the IP address or hostname of your logserver or relay and select OK.

      

      Figure 3.5. Forwarding messages to the logserver

      
      

   	Note
   	Accepting remote messages and forwarding them to a logserver means that syslog-ng will start in relay mode.

8. After the installation is finished, add the /opt/syslog-ng/bin and /opt/syslog-ng/sbin directories to your search PATH environment variable. That way you can use syslog-ng and its related tools without having to specify the full pathname. Add the following line to your shell profile:

   PATH=/opt/syslog-ng/bin:$PATH 

	Note
	The native logrotation tools do not send a SIGHUP to syslog-ng after rotating the log files, causing syslog-ng to write into files already rotated. To solve this problem, the syslog-ng init script links the /var/run/syslog.pid file to syslog-ng's pid. Also, on Linux, the install.sh script symlinks the initscript of the original syslog daemon to syslog-ng's initscript.

	Note
	The native logrotation tools do not send a SIGHUP to syslog-ng after rotating the log files, causing syslog-ng to write into files already rotated. To solve this problem, the syslog-ng init script links the /var/run/syslog.pid file to syslog-ng's pid. Also, on Linux, the install.sh script symlinks the initscript of the original syslog daemon to syslog-ng's initscript.

3.1.3. Installing syslog-ng without user-interaction

The syslog-ng application can be installed in silent mode without any user-interaction by specifying the required parameters from the command line. Answers to every question of the installer can be set in advance using command-line parameters.

./syslog-ng-premium-edition-<version>.run -- [options]

	Warning
	The -- characters between the executable and the parameters are mandatory, like in the following example: ./syslog-ng-premium-edition-3.0.1b-solaris-10-sparc-client.run -- --accept-eula -l /var/tmp/license.txt

To display the list of parameters, execute the ./syslog-ng-premium-edition-<version>.run -- --h command. Currently the following options are available:

• --accept-eula or -a: Accept the EULA.

• --license-file <file> or -l <file>: Path to the license file.

• --upgrade | -u: Perform automatic upgrade — use the configuration file and license file from an existing installation.

• --remote <destination host>: Send logs to the specified remote server. Not available when performing an upgrade.

• --network: Accept messages from the network. Not available when performing an upgrade.

• --configuration <file>: Use the specified configuration file.

If you need to uninstall syslog-ng for some reason, you have the following options:

The syslog-ng application is configured by editing the syslog-ng.conf file. Use any regular text editor application to modify the file. The precompiled syslog-ng packages include sample configuration files as well.

Every syslog-ng configuration file must begin with a line containing the version information of syslog-ng. For syslog-ng version 3.2, this line looks like:

@version:3.2

Versioning the configuration file was introduced in syslog-ng 3.0. If the configuration file does not contain the version information, syslog-ng assumes that the file is for syslog-ng version 2.x. In this case it interprets the configuration and sends warnings about the parts of the configuration that should be updated. Version 3.0 and later will correctly operate with configuration files of version 2.x, but the default values of certain parameters have changed since 3.0.

All identifiers, option names and attributes, and any other strings used in the syslog-ng configuration file are case sensitive. Objects must be defined before they are referenced in another statement.

	Example 4.1. A simple configuration file
	The following is a very simple configuration file for syslog-ng: it collects the internal messages of syslog-ng and the messages from /dev/log into the /var/log/messages_syslog-ng.log file. @version:3.0 source s_local { unix-stream("/dev/log"); internal(); }; destination d_file {file("/var/log/messages_syslog-ng.log"); }; log { source(s_local); destination(d_file); };

	Tip
	Before activating a new configuration, check that your configuration file is syntactically correct using the syslog-ng --syntax-only command. To activate the configuration, reload the configuration of syslog-ng using the /etc/init.d/syslog-ng reload command.

The syslog-ng.conf and license.txt files are located under the /opt/syslog-ng/etc/ directory.

	Note
	Earlier versions of syslog-ng PE stored the configuration and license files under different directories, depending on the platform; typically under /etc/syslog-ng/.

On Microsoft Windows platforms the syslog-ng agent stores its configuration in the system registry, and can be configured from a graphical interface. For details, see The syslog-ng Agent for Windows Administrator Guide.

4.1.1. Including configuration files

The syslog-ng application supports including external files in its configuration file, so parts of its configuration can be managed separately. To include the contents of a file in the syslog-ng configuration, use the following syntax

include "filename";

This imports the entire file into the configuration of syslog-ng, at the location of the include statement. If you specify a directory, syslog-ng will try to include every file in alphabetic order. When including configuration files, consider the following points:

• If an object is defined twice (for example the original syslog-ng configuration file and the file imported into this configuration file both define the same option, source, or other object), then the object that is defined later in the configuration file will be effective. For example, if you set a global option at the beginning of the configuration file, and later include a file that defines the same option with a different value, then the option defined in the imported file will be used.

• Files can be embedded into each other: the included files can contain include statements as well, up to a maximum depth of 15 levels.

• Include statements can only be used at top level of the configuration file. For example, the following is correct:

  @version:3.0
  include "example.conf";

  But the following is not:

  source s_example {
      include "example.conf"
       };

	Warning
	The syslog-ng application will not start if it cannot find a file that is to be included in its configuration. Always double-check the filenames, paths, and access rights when including configuration files, and use the --syntax-only command-line option to check your configuration.

bin/logchksign "cfg-fingerprint='832ef664ff79df8afc66cd955c0c8aaa3c343f31', cfg-nonce-ndx='0', cfg-signature='785223cfa19ad52b855550be141b00306347b0a9'"

Global objects (for example sources, destinations, log paths, or filters) are defined in the syslog-ng configuration file. Object definitions consist of the following elements:

The syntax is summarized as follows:

type identifier { parameters };

Objects have parameters; some of them are required, others are optional. Required parameters are positional, meaning that they must be specified in a defined order. Optional arguments can be specified in any order using the option(value) format. If a parameter (optional or required) is not specified, its default value is used. The parameters and their default values are listed in the reference section of the particular object. For details, see Chapter 6, Reference.

	Example 4.2. Using required and optional parameters
	The unix-stream() source driver has a single required argument: the name of the socket to listen on. Optional parameters follow the socket name in any order, so the following source definitions have the same effect: source s_demo_stream1 { unix-stream("/dev/log" max-connections(10) group(log)); }; source s_demo_stream2 { unix-stream("/dev/log" group(log) max-connections(10)); };

To add comments to the configuration file, start a line with # and write your comments. These lines are ignored by syslog-ng.

# Comment: This is a stream source
source s_demo_stream { 
           unix-stream("/dev/log" max-connections(10) group(log)); };

A source is where syslog-ng receives log messages. Sources consist of one or more drivers, each defining where and how messages are received.

To define a source, add a source statement to the syslog-ng configuration file using the following syntax:

source <identifier> { source-driver(params); source-driver(params); ... };

	Example 4.3. A simple source statement
	The following source statement receives messages on the TCP port 1999 of the interface having the 10.1.2.3 IP address. source s_demo_tcp { tcp(ip(10.1.2.3) port(1999)); };

	Example 4.4. A source statement using two source drivers
	The following source statement receives messages on the 1999 TCP port and the 1999 UDP port of the interface having the 10.1.2.3 IP address. source s_demo_two_drivers { tcp(ip(10.1.2.3) port(1999)); udp(ip(10.1.2.3) port(1999)); };

	Example 4.5. Setting default priority and facility
	If the message received by the source does not have a proper syslog header, you can use the default-facility() and default-priority() options to set the facility and priority of the messages. Note that these values are applied only to messages that do not set these parameters in their header. source headerless_messages { udp(default-facility(syslog) default-priority(emerg)); };

Define a source only once. The same source can be used in several log paths. Duplicating sources causes syslog-ng to open the source (TCP/IP port, file, and so on) more than once, which might cause problems. For example, include the /dev/log file source only in one source statement, and use this statement in more than one log path if needed.

To collect log messages on a specific platform, it is important to know how the native syslogd communicates on that platform. The following table summarizes the operation methods of syslogd on some of the tested platforms:

Each possible communication mechanism has a corresponding source driver in syslog-ng. For example, to open a unix socket with SOCK_DGRAM style communication use the driver unix-dgram. The same socket using the SOCK_STREAM style — as used under Linux — is called unix-stream.

	Example 4.6. Source statement on a Linux based operating system
	The following source statement collects the following log messages: internal(): Messages generated by syslog-ng. udp(ip(0.0.0.0) port(514)): Messages arriving to the 514/UDP port of any interface of the host. unix-stream("/dev/log");: Messages arriving to the /dev/log socket. source s_demo { internal(); udp(ip(0.0.0.0) port(514)); unix-stream("/dev/log"); };

The following table lists the source drivers available in syslog-ng.

For a complete description of the parameters of the above drivers, see Section 6.1, Source drivers.

4.3.1. Collecting internal messages

All messages generated internally by syslog-ng use this special source. To collect warnings, errors and notices from syslog-ng itself, include this source in one of your source statements.

internal()

The syslog-ng application will issue a warning upon startup if none of the defined log paths reference this driver.

	Example 4.7. Using the internal() driver
	source s_local { internal(); };

4.3.1.1. Log statistics

Periodically, syslog-ng sends a message containing statistics about the received messages, and about any lost messages since the last such message. It includes a processed entry for every source and destination, listing the number of messages received or sent, and a dropped entry including the IP address of the server for every destination where syslog-ng has lost messages. The center(received) entry shows the total number of messages received from every configured sources.

The following is a sample log statistics message for a configuration that has a single source (s_local) and a network and a local file destination (d_network and d_local, respectively). All incoming messages are sent to both destinations.

Log statistics;
                    dropped='tcp(AF_INET(192.168.10.1:514))=6439',
                    processed='center(received)=234413',
                    processed='destination(d_tcp)=234413',
                    processed='destination(d_local)=234413',
                    processed='source(s_local)=234413'

Log statistics can be also retrieved on-demand using one of the following options:

• Use the socat application: echo STATS | socat -vv UNIX-CONNECT:/opt/syslog-ng/var/run/syslog-ng.ctl -

• If you have an OpenBSD-style netcat application installed, use the echo STATS | nc -U var/run/syslog-ng.ctl command. Note that the netcat included in most Linux distributions is a GNU-style version that is not suitable to query the statistics of syslog-ng.

• Starting from syslog-ng Premium Edition version 3.1, syslog-ng Premium Edition includes the syslog-ng-ctl utility. Use the syslog-ng-ctl stats command.

The statistics include a list of source groups and destinations, as well as the number of processed messages for each. The verbosity of the statistics can be set using the stats_level() option. For details, see Section 6.9, Global options. An example output is shown below.

src.internal;s_all#0;;a;processed;6445
src.internal;s_all#0;;a;stamp;1268989330
destination;df_auth;;a;processed;404
destination;df_news_dot_notice;;a;processed;0
destination;df_news_dot_err;;a;processed;0
destination;d_ssb;;a;processed;7128
destination;df_uucp;;a;processed;0
source;s_all;;a;processed;7128
destination;df_mail;;a;processed;0
destination;df_user;;a;processed;1
destination;df_daemon;;a;processed;1
destination;df_debug;;a;processed;15
destination;df_messages;;a;processed;54
destination;dp_xconsole;;a;processed;671
dst.tcp;d_network#0;10.50.0.111:514;a;dropped;5080
dst.tcp;d_network#0;10.50.0.111:514;a;processed;7128
dst.tcp;d_network#0;10.50.0.111:514;a;stored;2048
destination;df_syslog;;a;processed;6724
destination;df_facility_dot_warn;;a;processed;0
destination;df_news_dot_crit;;a;processed;0
destination;df_lpr;;a;processed;0
destination;du_all;;a;processed;0
destination;df_facility_dot_info;;a;processed;0
center;;received;a;processed;0
destination;df_kern;;a;processed;70
center;;queued;a;processed;0
destination;df_facility_dot_err;;a;processed;0

The statistics are semicolon separated; every line contains statistics for a particular object (for example source, destination, tag, and so on). The statistics have the following fields:

• The type of the object (for example dst.file, tag, src.facility)

• The ID of the object used in the syslog-ng configuration file, for example d_internal or source.src_tcp. The #0 part means that this is the first destination in the destination group.

• The instance ID (destination) of the object, for example the filename of a file destination, or the name of the application for a program source or destination.

• The status of the object. One of the following:

  • a - active. At the time of quering the statistics, the source or the destination was still alive (it continuously received statistical data).

  • d - dynamic. Such objects may not be continuously available, for example, like statistics based on the sender's hostname.

  • o - This object was once active, but stopped receiving messages. (For example a dynamic object may disappear and become orphan.)

• The type of the statistics:

  • processed: The number of messages that successfully reached their destination.

  • dropped: The number of dropped messages — syslog-ng PE could not send the messages to the destination and the output buffer got full, so messages were lost.

  • stored: The number of messages stored in the message queue, waiting to be sent to the destination.

  • suppressed: The number of suppressed messages (if the suppress() feature is enabled).

  • stamp: The UNIX timestamp of the last message sent to the destination.

• The number of such messages

	Note
	Certain statistics are available only if the stats-level() option is set to a higher value. When receiving messages with non-standard facility values (that is, higher than 23), these messages will be listed as other facility instead of their facility number.

4.3.2. Collecting messages from text files

Collects log messages from plain-text files, for example from the logfiles of an Apache webserver.

The syslog-ng application notices if a file is renamed or replaced with a new file, so it can correctly follow the file even if logrotation is used. When syslog-ng is restarted, it records the position of the last sent log message, and continues to send messages from this position after the restart.

The file driver has a single required parameter specifying the file to open. For the list of available optional parameters, see Section 6.1.2, file().

Declaration:
                file(filename);

In syslog-ng PE, the filename (but not the pathname) may include wildcard characters (for example *). Note that when using wildcards in filenames, always set how often syslog-ng should check the file for new messages using the follow_freq() parameter.

When using wildcards, syslog-ng PE monitors every matching file, and can receive new log messages from any of the files. However, monitoring (polling) many files (that is, more than ten) has a significant overhead and may affect performance. On Linux this overhead is not so significant, because syslog-ng PE uses the inotify feature of the kernel.

	Example 4.8. Using the file() driver
	source s_file { file("/var/log/messages"); };

	Example 4.9. Using wildcards in the filename
	The following example monitors every file with the .log extension in the /var/application directory for log messages. Note that only syslog-ng PE supports wildcards in the file and pathnames. source s_file { file("/var/application/*.log" follow_freq(1));};

	Example 4.10. Monitoring multiple directories
	The following example reads files having the .log extension from the /var/application/ directory and its subdirectories. Note that only syslog-ng PE supports recursive directory handling and wildcards in the file and pathnames. source s_file_subdirectories { file("/var/application/*.log" recursive(yes) follow_freq(1) log_fetch_limit(100) );};

The kernel usually sends log messages to a special file (/dev/kmsg on BSDs, /proc/kmsg on Linux). The file() driver reads log messages from such files. The syslog-ng application can periodically check the file for new log messages if the follow_freq() option is set.

Note

On Linux, the klogd daemon can be used in addition to syslog-ng to read kernel messages and forward them to syslog-ng. klogd used to preprocess kernel messages to resolve symbols etc., but as this is deprecated by ksymoops there is really no point in running both klogd and syslog-ng in parallel. Also note that running two processes reading /proc/kmsg at the same time might result in dead-locks. When using syslog-ng to read messages from the /proc/kmsg file, syslog-ng automatically disables the follow_freq() parameter to avoid blocking the file.

4.3.3. Collecting messages from named pipes

The pipe driver opens a named pipe with the specified name and listens for messages. It is used as the native message delivery protocol on HP-UX.

The pipe driver has a single required parameter, specifying the filename of the pipe to open. For the list of available optional parameters, see Section 6.1.3, pipe().

Declaration:
                pipe(filename);

	Note
	As of syslog-ng Premium Edition 3.0.3, pipes are created automatically. In earlier versions, you had to create the pipe using the mkfifo(1) command.

Pipe is very similar to the file() driver, but there are a few differences, for example pipe() opens its argument in read-write mode, therefore it is not recommended to be used on special files like /proc/kmsg.

	Warning
	It is not recommended to use pipe() on anything else than real pipes.

	Example 4.11. Using the pipe() driver
	source s_pipe { pipe("/dev/pipe" pad_size(2048)); };

4.3.4. Collecting messages on Sun Solaris

Solaris uses its STREAMS framework to send messages to the syslogd process. Solaris 2.5.1 and above uses an IPC called door in addition to STREAMS, to confirm the delivery of a message. The syslog-ng application supports the IPC mechanism via the door() option (see below).

The sun-streams() driver has a single required argument specifying the STREAMS device to open, and the door() option. For the list of available optional parameters, see Section 6.1.5, sun-streams() driver.

Declaration:
        sun-streams(name_of_the_streams_device door(filename_of_the_door));

	Example 4.12. Using the sun-streams() driver
	source s_stream { sun-streams("/dev/log" door("/etc/.syslog_door")); };

4.3.5. Collecting messages using the IETF syslog protocol

The syslog() driver enables to receive messages from the network using the new standard syslog protocol and message format (also called IETF-syslog protocol; described in RFC 5424-28, see Section 2.20.2, IETF-syslog messages). UDP, TCP, and TLS-encrypted TCP can all be used to transport the messages.

For the list of available optional parameters, see Section 6.1.6, syslog().

Declaration:
            syslog(ip() port() transport() options());

Example 4.13. Using the syslog() driver

TCP source listening on the localhost on port 1999. source s_syslog { syslog(ip(127.0.0.1) port(1999) transport("tcp")); }; UDP source with defaults. source s_udp { syslog( transport("udp")); }; Encrypted source where the client is also authenticated. For details on the encryption settings, see Section 6.10, TLS options. source s_syslog_tls{ syslog( ip(10.100.20.40) transport("tls") tls( peer-verify(required-trusted) ca_dir('/opt/syslog-ng/etc/syslog-ng/keys/ca.d/') key_file('/opt/syslog-ng/etc/syslog-ng/keys/server_privatekey.pem') cert_file('/opt/syslog-ng/etc/syslog-ng/keys/server_certificate.pem') ) );};

4.3.6. Collecting messages from remote hosts using the BSD syslog protocol

The tcp(), tcp6(), udp(), udp6() drivers can receive messages from the network using the TCP and UDP networking protocols. The tcp6() and udp6() drivers use the IPv6 network protocol, while tcp() and udp() use IPv4.

UDP is a simple datagram oriented protocol, which provides "best effort service" to transfer messages between hosts. It may lose messages, and no attempt is made at the protocol level to retransmit such lost messages. The BSD-syslog protocol traditionally uses UDP.

TCP provides connection-oriented service, which basically means that the path of the messages is flow-controlled. Along this path, each message is acknowledged, and retransmission is done for lost packets. Generally it is safer to use TCP, because lost connections can be detected, and no messages get lost, assuming that the TCP connection does not break. When a TCP connection is broken the 'in-transit' messages that were sent by syslog-ng but not yet received on the other side are lost. (Basically these messages are still sitting in the socket buffer of the sending host and syslog-ng has no information about the fate of these messages).

The tcp() and udp() drivers do not have any required parameters. By default they bind to the 0.0.0.0:514 address, which means that syslog-ng will listen on all available interfaces, port 514. To limit accepted connections to only one interface, use the localip() parameter as described below. For the list of available optional parameters, see Section 6.1.7, tcp(), tcp6(), udp() and udp6().

Declaration:
                tcp([options]);
                udp([options]);

	Note
	The tcp port 514 is reserved for use with rshell, so select a different port if syslog-ng and rshell is used at the same time.

If you specify a multicast bind address to udp() and udp6(), syslog-ng will automatically join the necessary multicast group. TCP does not support multicasting.

The syslog-ng Premium Edition application supports TLS (Transport Layer Security, also known as SSL) for the tcp() and tcp6() drivers. See the TLS-specific options below and Section 4.13, Encrypting log messages with TLS for details. For the list of available optional parameters, see Section 6.1.5, sun-streams() driver.

Example 4.14. Using the udp() and tcp() drivers

A simple udp() source with default settings. source s_udp { udp(); };# An UDP source with default settings. A TCP source listening on the localhost interface, with a limited number of connections allowed. source s_tcp { tcp(ip(127.0.0.1) port(1999) max-connections(10)); }; A TCP source listening on a TLS-encrypted channel. source s_tcp { tcp(ip(127.0.0.1) port(1999) tls(peer-verify('required-trusted') key_file('/opt/syslog-ng/etc/syslog-ng/syslog-ng.key') cert_file('/opt/syslog-ng/etc/syslog-ng/syslog-ng.crt'))); }; A TCP source listening for messages using the IETF-syslog message format. Note that for transferring IETF-syslog messages, generally you are recommended to use the syslog() driver on both the client and the server, as it uses both the IETF-syslog message format and the protocol. For details, see Section 4.3.5, Collecting messages using the IETF syslog protocol. source s_tcp_syslog { tcp(ip(127.0.0.1) port(1999) flags(syslog-protocol) ); };

4.3.7. Collecting messages from UNIX domain sockets

The unix-stream() and unix-dgram() drivers open an AF_UNIX socket and start listening on it for messages. The unix-stream() driver is primarily used on Linux and uses SOCK_STREAM semantics (connection oriented, no messages are lost); while unix-dgram() is used on BSDs and uses SOCK_DGRAM semantics: this may result in lost local messages if the system is overloaded.

To avoid denial of service attacks when using connection-oriented protocols, the number of simultaneously accepted connections should be limited. This can be achieved using the max-connections() parameter. The default value of this parameter is quite strict, you might have to increase it on a busy system.

Both unix-stream and unix-dgram have a single required argument that specifies the filename of the socket to create. For the list of available optional parameters, see Section 6.1.8, unix-stream() and unix-dgram()

Declaration: 
                unix-stream(filename [options]);
                unix-dgram(filename [options]);

	Note
	syslogd on Linux originally used SOCK_STREAM sockets, but some distributions switched to SOCK_DGRAM around 1999 to fix a possible DoS problem. On Linux you can choose to use whichever driver you like as syslog clients automatically detect the socket type being used.

The difference between the unix-stream and unix-dgram drivers is similar to the difference between the TCP and UDP network protocols. Use the following guidelines to select which driver to use in a particular situation:

Choose unix-stream if you would choose TCP (stream) instead of UDP (datagram). The unix-stream driver offers the following features:

• Increased reliability

• Ordered delivery of messages

• Client-side notification of failures

Choose unix-dgram if you would choose TCP (stream) over UDP (datagram). The unix-dgram driver offers the following features:

• Decreased possibility of Dos by opening too many connections (a local vulnerability)

• Less overhead

However, the client does not notice if a message is lost when using the unix-dgram driver.

	Example 4.15. Using the unix-stream() and unix-dgram() drivers
	source s_stream { unix-stream("/dev/log" max-connections(10)); }; source s_dgram { unix-dgram("/var/run/log"); };

A destination is where a log message is sent if the filtering rules match. Similarly to sources, destinations consist of one or more drivers, each defining where and how messages are sent.

	Tip
	If no drivers are defined for a destination, all messages sent to the destination are discarded. This is equivalent to omitting the destination from the log statement.

To define a destination, add a destination statement to the syslog-ng configuration file using the following syntax:

destination <identifier> { 
                destination-driver(params); destination-driver(params); ... };

	Example 4.16. A simple destination statement
	The following destination statement sends messages to the TCP port 1999 of the 10.1.2.3 host. destination d_demo_tcp { tcp("10.1.2.3" port(1999)); }; If name resolution is configured, the hostname of the target server can be used as well. destination d_tcp { tcp("target_host" port(1999); localport(999)); };

The following table lists the destination drivers available in syslog-ng.

For detailed list of driver parameters, see Section 6.2, Destination drivers.

4.4.1. Storing messages in plain-text files

The file driver is one of the most important destination drivers in syslog-ng. It allows to output messages to the specified text file, or to a set of files.

The destination filename may include macros which get expanded when the message is written, thus a simple file() driver may create several files. For more information on available macros see Section 6.5, Macros.

If the expanded filename refers to a directory which does not exist, it will be created depending on the create_dirs() setting (both global and a per destination option).

The file() has a single required parameter that specifies the filename that stores the log messages. For the list of available optional parameters, see Section 6.2.1, file().

Declaration:
                file(filename options());

	Example 4.17. Using the file() driver
	destination d_file { file("/var/log/messages" ); };

	Example 4.18. Using the file() driver with macros in the file name and a template for the message
	destination d_file { file("/var/log/$YEAR.$MONTH.$DAY/messages" template("$HOUR:$MIN:$SEC $TZ $HOST [$LEVEL] $MSG $MSG\n") template_escape(no)); };

	Note
	When using the file() destination, update the configuration of your log rotation program to rotate these files. Otherwise, the log files can become very large.

Warning

Since the state of each created file must be tracked by syslog-ng, it consumes some memory for each file. If no new messages are written to a file within 60 seconds (controlled by the time_reap() global option), it is closed, and its state is freed. Exploiting this, a DoS attack can be mounted against the system. If the number of possible destination files and its needed memory is more than the amount available on the syslog-ng server. The most suspicious macro is $PROGRAM, where the number of possible variations is rather high. Do not use the $PROGRAM macro in insecure environments.

4.4.2. Storing messages in encrypted files

The logstore() driver stores log messages in binary files that can be encrypted, compressed, checked for integrity, and timestamped by an external Timestamping Authority (TSA). Otherwise, it is very similar to the file() destination.

Logstore files consist of individual chunks, every chunk can be encrypted, compressed, and timestamped separately. Chunks contain compressed log messages and header information needed for retrieving messages from the logstore file.

To display the contents of a logstore file, use the lgstool (formerly called logcat) command supplied with syslog-ng, for example lgstool cat /var/log/messages.lgs. Log messages available in the journal file of the logstore (but not yet written to the logstore file itself) are displayed as well.

To display the contents of encrypted log files, specify the private key of the certificate used to encrypt the file, for example lgstool cat -k private.key /var/log/messages.lgs. The contents of the file are sent to the standard output, so it is possible to use grep and other tools to find particular log messages, for example lgstool cat /var/log/messages.lgs |grep 192.168.1.1. For further details, see lgstool(1).

	Tip
	The lgstool utility is available for Microsoft Windows operating systems at the syslog-ng Premium Edition Download Page.

For files that are in use by syslog-ng, the last chunk that is open cannot be read. The syslog-ng PE application generates an SHA-1 hash for every chunk to verify the integrity of the chunk. The hashes of the chunks are chained together to prevent injecting chunks into the logstore file. The syslog-ng application can use various encryption and hashing (HMAC) algorithms, using the aes128 in CBC mode and hmac-sha1 by default. For other algorithms, see Section cipher() and Section digest().

	Warning
	If the computer running syslog-ng Premium Edition crashes, an unclosed chunk remains at the end of the logstore file. This chunk is marked as broken, its data stays there but is not shown by lgstool.

The destination filename may include macros which get expanded when the message is written, thus a simple logstore() driver may create several files. For more information on available macros see Section 6.5, Macros.

If the expanded filename refers to a directory which does not exist, it will be created depending on the create_dirs() setting (both global and a per destination option).

The logstore() has a single required parameter that specifies the filename that stores the log messages. For the list of available optional parameters, see Section 6.2.2, logstore().

Declaration:
                logstore(filename options());

Example 4.19. Using the logstore() driver

A simple example saving and compressing log messages. destination d_logstore { logstore("/var/log/messages.lgs" compress(5) ); }; A more detailed example that encrypts messages, modifies the parameters for closing chunks, and sets file privileges. destination d_logstore { logstore("/var/log/messages-logstore.lgs" encrypt_certificate("/opt/syslog-ng/etc/syslog-ng/keys/10-100-20-40/public-certificate-of-the-server.pem") owner("balabit") group("balabit") perm(0777) ); }; The URL to the Timestamping Authority and if needed, the OID of the timestamping policy can be set as global options, or also per logstore destination. The following example specifies the URL and the OID as global options: options {         timestamp_url("http://10.50.50.50:8080/");         timestamp_policy("0.4.0.2023.1.1"); };

	Note
	When using the logstore() destination, update the configuration of your log rotation program to rotate these files. Otherwise, the log files can become very large.

Warning

Since the state of each created file must be tracked by syslog-ng, it consumes some memory for each file. If no new messages are written to a file within 60 seconds (controlled by the time_reap() global option), it is closed, and its state is freed. Exploiting this, a DoS attack can be mounted against the system. If the number of possible destination files and its needed memory is more than the amount available on the syslog-ng server. The most suspicious macro is $PROGRAM, where the number of possible variations is rather high. Do not use the $PROGRAM macro in insecure environments.

4.4.3. Sending messages to named pipes

The pipe() driver sends messages to a named pipe like /dev/xconsole.

The pipe driver has a single required parameter, specifying the filename of the pipe to open. The filename can include macros. For the list of available optional parameters, see Section 6.2.3, pipe().

Declaration:
                pipe(filename);

	Warning
	As of syslog-ng Premium Edition 3.0.3, pipes are created automatically. In earlier versions, you had to create the pipe using the mkfifo(1) command.

	Example 4.20. Using the pipe() driver
	destination d_pipe { pipe("/dev/xconsole"); };

4.4.4. Sending messages to external applications

The program() driver starts an external application or script and sends the log messages to its standard input (stdin).

The program() driver has a single required parameter, specifying a program name to start. The program is executed with the help of the current shell, so the command may include both file patterns and I/O redirections. For the list of available optional parameters, see Section 6.2.4, program().

Declaration: 
                program(command_to_run);

	Note
	The syslog-ng application automatically restarts the external program if it exits for reliability reasons. However it is not recommended to launch programs for single messages, because if the message rate is high, launching several instances of an application might overload the system, resulting in Denial of Service.

Note that the message format does not include the priority and facility values by default. To add these values, specify a template for the program destination, as shown in the following example.

	Example 4.21. Using the program() destination driver
	destination d_prog { program("/bin/script" template("<$PRI>$DATE $HOST $MSG\n"); ); };

4.4.5. Storing messages in an SQL database

The sql() driver sends messages into an SQL database. Currently the Microsoft SQL (MSSQL), MySQL, Oracle, PostgreSQL, and SQLite databases are supported.

	Note
	In order to use the sql() destination, syslog-ng Premium Edition must run in server mode. Typically, only the central syslog-ng Premium Edition server uses the sql() destination.

The sql() driver has the following required parameters:

type

Type:	mssql, mysql, oracle, pgsql, or sqlite3
Default:	n/a

Description: Specifies the type of the database, that is, the DBI database driver to use. Use the mssql option to send logs to an MSSQL database. See the examples of the databases in the following sections for details.

database

Type:	string
Default:	n/a

Description: Name of the database that stores the logs.

table

Type:	string
Default:	n/a

Description: Name of the database table to use (can include macros). When using macros, note that some databases limit the length of table names.

columns

Type:	string list
Default:	"date", "facility", "level", "host", "program", "pid", "message"

Description: Name of the columns storing the data in fieldname [dbtype] format. The [dbtype] parameter is optional, and specifies the type of the field. By default, syslog-ng creates text columns. Note that not every database engine can index text fields.

values

Type:	string list
Default:	"${R_YEAR}-${R_MONTH}-${R_DAY} ${R_HOUR}:${R_MIN}:${R_SEC}", "$FACILITY", "$LEVEL", "$HOST", "$PROGRAM", "$PID", "$MSGONLY"

Description:The parts of the message to store in the fields specified in the columns parameter.

For the list of available optional parameters, see Section 6.2.5, sql().

Declaration: 
    sql(database_type host_parameters database_parameters [options]);

Warning

The syslog-ng application requires read and write access to the SQL table, otherwise it cannot verify that the destination table exists. Currently the syslog-ng application has default schemas for the different databases and uses these defaults if the database schema (for example columns and column types) is not defined in the configuration file. However, these schemas will be deprecated and specifying the exact database schema will be required in later versions of syslog-ng.

	Note
	In addition to the standard syslog-ng packages, the sql() destination requires database-specific packages to be installed. These packages are automatically installed by the binary syslog-ng installer. The sql() driver is currently not available for every platform that is supported by syslog-ng. For a list of platforms that support the sql() driver, click here.

The table and value parameters can include macros to create tables and columns dynamically (for details, see Section 6.5, Macros).

	Warning
	When using macros in table names, note that some databases limit the maximum allowed length of table names. Consult the documentation of the database for details.

Inserting the records into the database is performed by a separate thread. The syslog-ng application automatically performs the escaping required to insert the messages into the database.

Example 4.22. Using the sql() driver

The following example sends the log messages into a PostgreSQL database running on the logserver host. The messages are inserted into the logs database, the name of the table includes the exact date and the name of the host sending the messages. The syslog-ng application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges. destination d_sql { sql(type(pgsql) host("logserver") username("syslog-ng") password("password") database("logs") table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}") columns("datetime", "host", "program", "pid", "message") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes("datetime", "host", "program", "pid", "message")); }; The following example specifies the type of the database columns as well: destination d_sql { sql(type(pgsql) host("logserver") username("syslog-ng") password("password") database("logs") table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}") columns("datetime varchar(16)", "host varchar(32)", "program varchar(20)", "pid varchar(8)", "message varchar(200)") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes("datetime", "host", "program", "pid", "message")); };

4.4.5.1. Using the sql() driver with an Oracle database

The Oracle sql destination has some special aspects that are important to note.

• The hostname of the database server is set in the tnsnames.ora file, not in the host parameter of the sql() destination.

  Make sure to set the Oracle-related environment variables properly, so syslog-ng and the Oracle client will find the file. The following variables must be set: ORACLE_BASE, ORACLE_HOME, and ORACLE_SID. See the documentation of the Oracle Instant Client for details.

• As certain database versions limit the maximum length of table names, macros in the table names should be used with care.

• In the current version of syslog-ng PE, the types of database columns must be explicitly set for the Oracle destination. The column used to store the text part of the syslog messages should be able to store messages as long as the longest message permitted by syslog-ng, therefore it is usually recommended to use the varchar2 or clob column type. (The maximum length of the messages can be set using the log_msg_size() option.) For details, see the following example.

Example 4.23. Using the sql() driver with an Oracle database

The following example sends the log messages into an Oracle database running on the logserver host, which must be set in the /etc/tnsnames.ora file. The messages are inserted into the LOGS database, the name of the table includes the exact date when the messages were sent. The syslog-ng application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges. destination d_sql { sql(type(oracle) username("syslog-ng") password("password") database("LOGS") table("msgs_${R_YEAR}${R_MONTH}${R_DAY}") columns("datetime varchar(16)", "host varchar(32)", "program varchar(32)", "pid varchar(8)", "message varchar2") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes("datetime", "host", "program", "pid", "message")); }; The Oracle Instant Client retrieves the address of the database server from the /etc/tnsnames.ora file. Edit or create this file as needed for your configuration. A sample is provided below. LOGS = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP) (HOST = logserver) (PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = EXAMPLE.SERVICE) ) )

4.4.5.2. Using the sql() driver with a Microsoft SQL database

The mssql database driver can access Microsoft SQL (MSSQL) destinations. This driver has some special aspects that are important to note.

• The date format used by the MSSQL database must be explicitly set in the /etc/locales.conf file of the syslog-ng server. For details, see the following example.

• As certain database versions limit the maximum length of table names, macros in the table names should be used with care.

• In the current version of syslog-ng PE, the types of database columns must be explicitly set for the MSSQL destination. The column used to store the text part of the syslog messages should be able to store messages as long as the longest message permitted by syslog-ng. The varchar column type can store maximum 4096 bytes-long messages. The maximum length of the messages can be set using the log_msg_size() option. For details, see the following example.

• Remote access for SQL users must be explicitly enabled on the Microsoft Windows host running the Microsoft SQL Server. For details, see Procedure 3.5, Configuring Microsoft SQL Server to accept logs from syslog-ng.

Example 4.24. Using the sql() driver with an MSSQL database

The following example sends the log messages into an MSSQL database running on the logserver host. The messages are inserted into the syslogng database, the name of the table includes the exact date when the messages were sent. The syslog-ng application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges. destination d_mssql { sql(type(mssql) host("logserver") port("1433") username("syslogng") password("syslogng") database("syslogng") table("msgs_${R_YEAR}${R_MONTH}${R_DAY}")columns("datetime varchar(16)", "host varchar(32)", "program varchar(32)", "pid varchar(8)", "message varchar(4096)") values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY") indexes("datetime", "host", "program", "pid")); }; The date format used by the MSSQL database must be explicitly set in the /etc/locales.conf file of the syslog-ng server. Edit or create this file as needed for your configuration. A sample is provided below. [default] date = "%Y-%m-%d %H:%M:%S"

4.4.6. Sending messages to a remote logserver using the IETF-syslog protocol

The syslog() driver sends messages to a remote host (for example a syslog-ng server or relay) on the local intranet or internet using the new standard syslog protocol developed by IETF (for details about the new protocol, see Section 2.20.2, IETF-syslog messages). The protocol supports sending messages using the UDP, TCP, or the encrypted TLS networking protocols.

The required arguments of the driver are the address of the destination host (where messages should be sent). The transport method (networking protocol) is optional, syslog-ng uses the TCP protocol by default. For the list of available optional parameters, see Section 6.2.6, syslog().

Declaration:
                syslog(host transport [options]);

	Note
	Note that the syslog destination driver has required parameters, while the source driver defaults to the local bind address, and every parameter is optional.

The udp transport method automatically sends multicast packets if a multicast destination address is specified. The tcp and tls methods do not support multicasting.

	Note
	The default ports for the different transport protocols are as follows: UDP — 514; TLS — 6514.

Example 4.25. Using the syslog() driver

destination d_tcp { syslog(ip("10.1.2.3") transport("tcp") port(1999) localport(999)); }; If name resolution is configured, the hostname of the target server can be used as well. destination d_tcp { syslog(ip("target_host") transport("tcp") port(1999) localport(999)); }; Send the log messages using TLS encryption and use mutual authentication. For details on the encryption and authentication options, see Section 6.10, TLS options. destination d_syslog_tls{ syslog("10.100.20.40" transport("tls") port(6514) tls(peer-verify(required-trusted) ca_dir('/opt/syslog-ng/etc/syslog-ng/keys/ca.d/') key_file('/opt/syslog-ng/etc/syslog-ng/keys/client_key.pem') cert_file('/opt/syslog-ng/etc/syslog-ng/keys/client_certificate.pem')) );};

4.4.7. Sending messages to a remote logserver using the legacy BSD-syslog protocol

The tcp(), tcp6(), udp(), and udp6() drivers send messages to another host (for example a syslog-ng server or relay) on the local intranet or internet using the UDP or TCP protocol. The tcp6() and udp6() drivers use the IPv6 network protocol.

All four drivers have a single required parameter specifying the destination host address, where messages should be sent. For the list of available optional parameters, see Section 6.2.7, tcp(), tcp6(), udp(), and udp6().

The udp() and udp6() drivers automatically send multicast packets if a multicast destination address is specified. The tcp() and tcp6() drivers do not support multicasting.

Declaration:
                tcp(host [options]);
                udp(host [options]);
                tcp6(host [options]);
                udp6(host [options]);

Example 4.26. Using the tcp() driver

destination d_tcp { tcp("10.1.2.3" port(1999) localport(999)); }; If name resolution is configured, the hostname of the target server can be used as well. destination d_tcp { tcp("target_host" port(1999) localport(999)); }; To send messages using the IETF-syslog message format without using the IETF-syslog protocol, enable the syslog-protocol flag: destination d_tcp { tcp("10.1.2.3" port(1999) flags(syslog-protocol) ); }; For details on how to use the IETF-syslog protocol, see Section 6.2.6, syslog().

4.4.8. Sending messages to UNIX domain sockets

The unix-stream() and unix-dgram() drivers send messages to a UNIX domain socket in either SOCK_STREAM or SOCK_DGRAM mode.

Both drivers have a single required argument specifying the name of the socket to connect to. For the list of available optional parameters, see Section 6.2.8, unix-stream() & unix-dgram().

Declaration: 
                unix-stream(filename [options]);
                unix-dgram(filename [options]);

	Example 4.27. Using the unix-stream() driver
	destination d_unix_stream { unix-stream("/var/run/logs"); };

4.4.9. usertty()

This driver writes messages to the terminal of a logged-in user.

The usertty() driver has a single required argument, specifying a username who should receive a copy of matching messages. Use the asterisk * to specify every user currently logged in to the system.

Declaration: 
    usertty(username);

The usertty() does not have any further options nor does it support templates.

	Example 4.28. Using the usertty() driver
	destination d_usertty { usertty("root"); };

Log paths determine what happens with the incoming log messages. Messages coming from the sources listed in the log statement and matching all the filters are sent to the listed destinations.

To define a log path, add a log statement to the syslog-ng configuration file using the following syntax:

log {
    source(s1); source(s2); ... 
    optional_element(filter1|parser1|rewrite1); optional_element(filter2|parser2|rewrite2);... 
    destination(d1); destination(d2); ... 
    flags(flag1[, flag2...]);
    };

	Warning
	Log statements are processed in the order they appear in the configuration file, thus the order of log paths may influence what happens to a message, especially when using filters and log flags.

	Example 4.29. A simple log statement
	The following log statement sends all messages arriving to the localhost to a remote server. source s_localhost { tcp(ip(127.0.0.1) port(1999) ); }; destination d_tcp { tcp("10.1.2.3" port(1999); localport(999)); }; log { source(s_localhost); destination(d_tcp); };

All matching log statements are processed by default, and the messages are sent to every matching destination by default. So a single log message might be sent to the same destination several times, provided the destination is listed in several log statements, and it can be also sent to several different destinations.

This default behavior can be changed using the flags() parameter. Flags apply to individual log paths; they are not global options. The following flags available in syslog-ng:

	Warning
	The final, fallback, and catchall flags apply only for the top-level log paths, they have no effect on embedded log paths.

Example 4.30. Using log path flags

Let's suppose that you have two hosts (myhost_A and myhost_B) that run two applications each (application_A and application_B), and you collect the log messages to a central syslog-ng server. On the server, you create two log paths: one that processes only the messages sent by myhost_A; and one that processes only the messages sent by application_A. This means that messages sent by application_A running on myhost_A will be processed by both log paths, and the messages of application_B running on myhost_B will not be processed at all. If you add the final flag to the first log path, then only this log path will process the messages of myhost_A, so the second log path will receive only the messages of application_A running on myhost_B. If you create a third log path that includes the fallback flag, it will process the messages not processed by the first two log paths, in this case, the messages of application_B running on myhost_B. Adding a fourth log path with the catchall flag would process every message received by the syslog-ng server. log { source(s_localhost); destination(d_file); flags(catchall); };

For details on the individual flags, see Section 6.3, Log path flags. The effect and use of the flow-control flag is detailed in Section 2.13, Managing incoming and outgoing messages with flow-control.

4.5.1. Using embedded log statements

Embedded log statements (see Section 2.2.2, Embedded log statements) re-use the results of processing messages (for example the results of filtering or rewriting) to create complex log paths. Embedded log statements use the same syntax as regular log statements, but they cannot contain additional sources. To define embedded log statements, use the following syntax:

log {
    source(s1); source(s2); ... 
    
    optional_element(filter1|parser1|rewrite1); 
    optional_element(filter2|parser2|rewrite2);... 
    
    destination(d1); destination(d2); ... 
    
    #embedded log statement
    log
        {
        optional_element(filter1|parser1|rewrite1); 
        optional_element(filter2|parser2|rewrite2);... 
        destination(d1); destination(d2); ...
        #another embedded log statement
        log
        {
            optional_element(filter1|parser1|rewrite1); 
            optional_element(filter2|parser2|rewrite2);... 
            destination(d1); destination(d2); ...};
        };    
    #set flags after the embedded log statements
    flags(flag1[, flag2...]); 
        };

	Warning
	The final, fallback, and catchall flags apply only for the top-level log paths, they have no effect on embedded log paths.

Example 4.31. Using embedded log paths

The following log path sends every message to the d_file1 and the d_file2 destinations. log { source(s_localhost); destination(d_file1); destination(d_file2); }; The next example is equivalent with the one above, but uses an embedded log statement. log { source(s_localhost); destination(d_file1); log {destination(d_file2); }; }; The following example sends every message coming from the host 192.168.1.1 into the d_file1 destination, and sends every message coming from the host 192.168.1.1 and containing the string example into the d_file2 destination. log { source(s_localhost); host(192.168.1.); destination(d_file1); log {message("example"); destination(d_file2); }; }; The following example collects logs from multiple source groups and uses the source() filter in the embedded log statement to select messages of the s_network source group. log { source(s_localhost); source(s_network); destination(d_file1); log {source(s_network); destination(d_file2); }; };

4.5.2. Configuring flow-control

For details on how flow-control works, see Section 2.13, Managing incoming and outgoing messages with flow-control. The summary of the main points is as follows:

• The syslog-ng application normally reads a maximum of log_fetch_limit() number of messages from a source.

• From TCP and unix-stream sources, syslog-ng reads a maximum of log_fetch_limit() from every connection of the source. The number of connections to the source is set using the max_connections() parameter.

• Every destination has an output buffer (log_fifo_size()).

• Flow-control uses a control window to determine if there is free space in the output buffer for new messages. Every source has its own control window; log_iw_size() parameter sets the size of the control window.

• When a source accepts multiple connections, the messages of every connection use the same control window.

• The output buffer must be larger than the control window of every source that logs to the destination.

• If the control window is full, syslog-ng stops reading messages from the source until some messages are successfully sent to the destination.

• If the output buffer becomes full, and neither disk-buffering nor flow-control is used, messages may be lost.

	Note
	If you modify the max_connections() or the log_fetch_limit() parameter, do not forget to adjust the log_iw_size() and log_fifo_size() parameters accordingly.

Example 4.32. Sizing parameters for flow-control

Suppose that syslog-ng has a source that must accept up to 300 parallel connections. Such situation can arise when a network source receives connections from many clients, or if many applications log to the same socket. Therefore, set the max_connections() parameter of the source to 300. However, the log_fetch_limit() (default value: 10) parameter applies to every connection of the source individually, while the log_iw_size() (default value: 100) parameter applies to the source. In a worst-case scenario, the destination does not accept any messages, while all 300 connections send at least log_fetch_limit() number of messages to the source during every poll loop. Therefore, the control window must accommodate at least max_connections()*log_fetch_limit() messages to be able to read every incoming message of a poll loop. In the current example this means that (log_iw_size() should be greater than 300*10=3000. If the control window is smaller than this value, the control window might fill up with messages from the first connections — causing syslog-ng to read only one message of the last connections in every poll loop. The output buffer of the destination must accommodate at least log_iw_size() messages, but use a greater value: in the current example 3000*10=30000 messages. That way all incoming messages of ten poll loops fit in the output buffer. If the output buffer is full, syslog-ng does not read any messages from the source until some messages are successfully sent to the destination. source s_localhost { tcp(ip(127.0.0.1) port(1999) max-connections(300)); }; destination d_tcp { tcp("10.1.2.3" port(1999); localport(999)); log_fifo_size(30000); }; log { source(s_localhost); destination(d_tcp); flags(flow-control); }; If other sources send messages to this destination, than the output buffer must be further increased. For example, if a network host with maximum 100 connections also logs into the destination, than increase the log_fifo_size() by 10000. source s_localhost { tcp(ip(127.0.0.1) port(1999) max-connections(300)); }; source s_tcp { tcp(ip(192.168.1.5) port(1999) max-connections(100)); }; destination d_tcp { tcp("10.1.2.3" port(1999); localport(999)); log_fifo_size(40000); }; log { source(s_localhost); destination(d_tcp); flags(flow-control); };

See also Section 5.2, Handling lots of parallel connections.

The following sections describe how to select and filter log messages.

4.6.1. Using filters

Filters perform log routing within syslog-ng: a message passes the filter if the filter expression is true for the particular message. If a log statement includes filters, the messages are sent to the destinations only if they pass all filters of the log path. For example, a filter can select only the messages originating from a particular host. Complex filters can be created using filter functions and logical boolean expressions.

To define a filter, add a filter statement to the syslog-ng configuration file using the following syntax:

filter <identifier> { expression; };

The expression may contain the following elements:

• The functions listed in Section facility(). Some of the functions accept extended regular expressions as parameters.

• The boolean operators and, or, not.

• Parentheses to mark the precedence of the operators when using complex filters.

Example 4.33. A simple filter statement

The following filter statement selects the messages that contain the word deny and come from the host example. filter demo_filter { host("example") and match("deny" value("MESSAGE")); }; For the filter to have effect, include it in a log statement: log demo_filteredlog { source(s1); source(s2); filter(demo_filter); destination(d1); destination(d2); }; The host(), match(), and program() filter functions accept regular expressions as parameters. filter demo_regexp_filter { host("system.*1") and match("deny" value("MESSAGE")); };

The value() parameter limits the scope of a filter function to the scope of a macro. For example, to limit the scope of the match() filter to the text part of the message, use:

match("keyword" value("MESSAGE"))

The value() parameter accepts both built-in macros and user-defined ones created with a parser. Do not prefix the macros with the $ sign. For details on macros and parsers, see Section 4.7, Templates and macros and Section 4.8, Parsing messages.

Note

When a log statement includes multiple filter statements, syslog-ng sends a message to the destination only if all filters are true for the message. In other words, the filters are connected with the logical AND operator. In the following example, no message arrives to the destination, because the filters are exclusive (the hostname of a client cannot be example1 and example2 at the same time): filter demo_filter1 { host("example1"); }; filter demo_filter2 { host("example2"); }; log demo_filteredlog { source(s1); source(s2); filter(demo_filter1); filter(demo_filter2); destination(d1); destination(d2); }; To select the messages that come from either host example1 or example2, use a single filter expression: filter demo_filter { host("example1") or host("example2"); }; log demo_filteredlog { source(s1); source(s2); filter(demo_filter); destination(d1); destination(d2); }; Use the not operator to invert filters, for example, to select the messages that were not sent by host example1: filter demo_filter { not host("example1"); }; However, to select the messages that were not sent by host example1 or example2, you have to use the and operator (that's how boolean logic works): filter demo_filter { not host("example1") and not host("example2"); }; Alternatively, you can use parentheses to avoid this confusion: filter demo_filter { not (host("example1") or host("example2")); };

In the extended regular expressions, the characters ()[].*?+^$| are used as special symbols. Therefore, these characters have to be preceded with a backslash (\) if they are meant literally. For example, the \$40 expression matches the $40 string. Backslashes have to be escaped as well if they are meant literally. For example, the \\d expression matches the \d string.

	Tip
	If you use single quotes in, you do not need to escape the backslash, for example match("\\.") is equivalent to match('\.').

By default, all regular expressions are case sensitive. To disable the case sensitivity of the expression, add the flags(ignore-case) option to the regular expression.

filter demo_regexp_insensitive { host("system" flags(ignore-case)); };

For details on regular expressions, see Section 6.8, Regular expressions.

Note

In regular expressions, the asterisk (*) character means 0, 1 or any number of the previous expression. For example, in the f*ilter expression the asterisk means 0 or more f letters. This expression matches for the following strings: ilter, filter, ffilter, and so on. To achieve the wildcard functionality commonly represented by the asterisk character in other applications, use .* in your expressions, for example f.*ilter.

The level() filter can select messages corresponding to a single importance level, or a level-range. To select messages of a specific level, use the name of the level as a filter parameter, for example use the following to select warning messages:

level(warning)

To select a range of levels, include the beginning and the ending level in the filter, separated with two dots (..). For example, to select every message of error or higher level, use the following filter:

level(err..emerg)

Similarly, messages sent by a range of facilities can also be selected. Note that this is only possible when using the name of the facilities. It is not possible to select ranges using the numerical codes of the facilities.

facility(local0..local5)

For a complete list of the available levels and facilities, see Section 6.4, Filter functions.

For a complete description on the above functions, see Section 6.4, Filter functions.

4.6.2. Optimizing regular expressions in filters

Some filter functions accept regular expressions as parameters. But evaluating general regular expressions puts a high load on the CPU, which can cause problems when the message traffic is very high. Often the regular expression can be replaced with simple filter functions and logical operators. Using simple filters and logical operators, the same effect can be achieved at a much lower CPU load.

Example 4.34. Optimizing regular expressions in filters

Suppose you need a filter that matches the following error message logged by the xntpd NTP daemon: xntpd[1567]: time error -1159.777379 is too large (set clock manually); The following filter uses regular expressions and matches every instance and variant of this message. filter f_demo_regexp { program("demo_program") and match("time error .* is too large .* set clock manually"); }; Segmenting the match() part of this filter into separate match() functions greatly improves the performance of the filter. filter f_demo_optimized_regexp { program("demo_program") and match("time error") and match("is too large") and match("set clock manually"); };

4.6.3. Tagging messages

Starting with syslog-ng 3.1, it is also possible to label the messages with custom tags. Tags are simple labels, identified by their names, which must be unique. Currently syslog-ng can tag a message at two different places:

• at the source when the message is received; and

• when the message matches a pattern in the pattern database. For details on using the pattern database, see Section 4.9, Classifying messages, for details on creating tags in the pattern database, see Section 6.6.2.3, Creating pattern databases.

When syslog-ng receives a message, it automatically adds the .source.<id_of_the_source_statement> tag to the message. Use the tags() option of the source to add custom tags, and the tags() option of the filters to select only specific messages.

Note

Tagging messages and also filtering on the tags is very fast, much faster then other types of filters. Tags are available locally, that is, if you add tags to a message on the client, these tags will not be available on the server. To include the tags in the message, use the $TAGS macro in a template. Alternatively, if you are using the IETF-syslog message format, you can include the $TAGS macro in the .SDATA.meta part of the message. Note that the $TAGS macro is available only in syslog-ng PE 3.1.1 and later.

	Example 4.35. Adding tags and filtering messages with tags
	source s_tcp { tcp(ip(192.168.1.1) port(1514) tags("tcp", "router")); }; Use the tags() option of the filters to select only specific messages: filter f_tcp { tags(".source.s_tcp"); }; filter f_router { tags("router"); };

The syslog-ng application allows you to define message templates, and reference them from every object that can use a template. Templates can be used to create standard message formats or filenames. Templates can reference one or more macros (for example date, the hostname, and so on). For a list of macros available in the Linux/Unix versions of syslog-ng see Section 6.5, Macros. For the macros of the syslog-ng Agent for Windows application, see Section 3.6, Customizing the message format in The syslog-ng Agent for Windows 3.2 Administrator Guide. Fields from the structured data (SD) part of messages using the new IETF-syslog standard can also be used as macros.

Template objects have a single option called template_escape, which is disabled by default (template_escape(no)). This behavior is useful when the messages are passed to an application that cannot handle escaped characters properly. Enabling template escaping (template_escape(yes)) causes syslog-ng to escape the ' and " characters from the messages.

	Note
	In versions 2.1 and earlier, the template_escape() option was enabled by default.

Macros can be included by prefixing the macro name with a $ sign, just like in Bourne compatible shells. Regarding braces around macro names, the following two formats are equivalent "$MSG" and "${MSG}".

Default values for macros can also be specified by appending the :- characters and the default value to the macro, for example

${HOST:-default_hostname}

	Note
	For the macros of the syslog-ng Agent for Windows application, see Section 3.6, Customizing the message format in The syslog-ng Agent for Windows 3.2 Administrator Guide.

The macros related to the date of the message (for example: ISODATE, HOUR, and so on) have two further versions each: one with the S_ and one with the R_ prefix (for example: S_DATE and R_DATE). The S_DATE macro represents the date found in the log message, that is, when the message was sent by the original application. R_DATE is the date when syslog has received the message.

DATE equals either S_DATE or R_DATE, depending on the global option set in the now deprecated use_time_recvd() parameter (see Section 6.9, Global options).

	Warning
	The hostname-related macros (FULLHOST, FULLHOST_FROM, HOST, and HOST_FROM) do not have any effect if the keep_hostname() option is disabled.

By default, syslog-ng sends messages using the following template: $ISODATE $HOST $MSGHDR$MSG\n. (The $MSGHDR$MSG part is written together because the $MSGHDR macro includes a trailing whitespace.)

	Note
	Earlier versions of syslog-ng used templates and scripts to send log messages into SQL databases. Starting from version 2.1, syslog-ng natively supports direct database access using the sql() destination. See Section 6.2.5, sql() for details.

Example 4.36. Using templates

The following template (t_demo_filetemplate) adds the date of the message and the name of the host sending the message to the beginning of the message text. The template is then used in a file destination: messages sent to this destination (d_file) will use the message format defined in the template. template t_demo_filetemplate { template("$ISODATE $HOST $MSG\n"); template_escape(no); }; destination d_file { file("/var/log/messages" template(t_demo_filetemplate)); }; Templates can also be used inline, if they are used only at a single location. The following destination is equivalent with the previous example: destination d_file { file ("/var/log/messages" template("$ISODATE $HOST $MSG\n") template_escape(no) ); };

The syslog-ng application can separate parts of log messages (that is, the contents of the $MSG macro) to named fields (columns). These fields act as user-defined macros that can be referenced in message templates, file- and tablenames, and so on.

Parsers are similar to filters: they must be defined in the syslog-ng configuration file and used in the log statement.

	Note
	The order of filters, rewriting rules, and parsers in the log statement is important, as they are processed sequentially.

To create a parser, define the columns of the message, the delimiter or separator characters, and optionally the characters that are used to escape the delimiter characters (quote-pairs). For the list of parser parameters, see Section 6.6, Message parsers.

Declaration:
            parser parser_name {
            csv-parser(column1, column2, ...)
            delimiters()
            quote-pairs()
            };

Column names work like macros. Always use a prefix to identify the columns of the parsers, for example MYPARSER1.COLUMN1, MYPARSER2.COLUMN2, and so on. Column names starting with a dot (for example .HOST) are reserved for use by syslog-ng.

	Example 4.37. Segmenting hostnames separated with a dash
	The following example separates hostnames like example-1 and example-2 into two parts. parser p_hostname_segmentation { csv-parser(columns("HOSTNAME.NAME", "HOSTNAME.ID") delimiters("-") flags(escape-none) template("${HOST}")); }; destination d_file { file("/var/log/messages-${HOSTNAME.NAME:-examplehost}"); }; log { source(s_local); parser(p_hostname_segmentation); destination(d_file);};

Example 4.38. Parsing Apache log files

The following parser processes the log of Apache web servers and separates them into different fields. Apache log messages can be formatted like: "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T %v" Here is a sample message: 192.168.1.1 - - [31/Dec/2007:00:17:10 +0100] "GET /cgi-bin/example.cgi HTTP/1.1" 200 2708 "-" "curl/7.15.5 (i4 86-pc-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8c zlib/1.2.3 libidn/0.6.5" 2 example.balabit To parse such logs, the delimiter character is set to a single whitespace (delimiters(" ")). Whitespaces between quotes and brackets are ignored (quote-pairs('""[]')). parser p_apache { csv-parser(columns("APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME", "APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS", "APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT", "APACHE.PROCESS_TIME", "APACHE.SERVER_NAME") flags(escape-double-char,strip-whitespace) delimiters(" ") quote-pairs('""[]') ); }; The results can be used for example to separate log messages into different files based on the APACHE.USER_NAME field. If the field is empty, the nouser name is assigned. log { source(s_local); parser(p_apache); destination(d_file);}; }; destination d_file { file("/var/log/messages-${APACHE.USER_NAME:-nouser}"); };

Multiple parsers can be used to split a part of an already parsed message into further segments.

Example 4.39. Segmenting a part of a message

The following example splits the timestamp of a parsed Apache log message into separate fields. parser p_apache_timestamp { csv-parser(columns("APACHE.TIMESTAMP.DAY", "APACHE.TIMESTAMP.MONTH", "APACHE.TIMESTAMP.YEAR", "APACHE.TIMESTAMP.HOUR", "APACHE.TIMESTAMP.MIN", "APACHE.TIMESTAMP.MIN", "APACHE.TIMESTAMP.ZONE") delimiters("/: ") flags(escape-none) template("${APACHE.TIMESTAMP}")); }; log { source(s_local); log { parser(p_apache); parser(p_apache_timestamp); destination(d_file);}; };

To classify messages using a pattern database, include a db_parser() statement in your syslog-ng configuration file using the following syntax:

Declaration:
        parser <identifier> {db_parser(file("<database_filename>"));};

Note that using the parser in a log statement only performs the classification, but does not automatically do anything with the results of the classification.

	Example 4.40. Defining pattern databases
	The following statement uses the database located at /opt/syslog-ng/var/db/patterndb.xml. parser pattern_db { db_parser( file("/opt/syslog-ng/var/db/patterndb.xml") ); }; To apply the patterns on the incoming messages, include the parser in a log statement: log { source(s_all); parser(pattern_db); destination( di_messages_class); };

	Note
	The default location of the pattern database file is /opt/syslog-ng/var/run/patterndb.xml. The file option of the db-parser statement can be used to specify a different file, thus different db-parser statements can use different pattern databases. Later versions of syslog-ng will be able to dynamically generate a main database from separate pattern database files.

Example 4.41. Using classification results

The following destination separates the log messages into different files based on the class assigned to the pattern that matches the message (for example Violation and Security type messages are stored in a separate file), and also adds the ID of the matching rule to the message: destination di_messages_class { file("/var/log/messages-${.classifier.class}" template("${.classifier.rule_id};${S_UNIXTIME};${SOURCEIP};${HOST};${PROGRAM};${PID};${MSG}\n") template_escape(no) ); };

For details on how to create your own pattern databases see Section 6.6.2.3, Creating pattern databases.

4.9.2. Using parser results in filters and templates

4.9.2.1. Filtering messages based on classification

The results of message classification and parsing can be used in custom filters and file and database templates as well. There are two built-in macros in syslog-ng PE that allow you to use the results of the classification: the .classifier.class macro contains the class assigned to the message (for example violation, security, or unknown), while the .classifier.rule_id macro contains the identifier of the message pattern that matched the message.

Example 4.42. Using classification results for filtering messages

To filter on a specific message class, create a filter that checks the .classifier_class macro, and use this filter in a log statement. filter fi_class_violation { match("violation" value(".classifier.class") type("string") ); }; log { source(s_all); parser(pattern_db); filter(fi_class_violation); destination(di_class_violation); }; Filtering on the unknown class selects messages that did not match any rule of the pattern database. Routing these messages into a separate file allows you to periodically review new or unknown messages. To filter on messages matching a specific classification rule, create a filter that checks the .classifier.rule_id macro. The unique identifier of the rule (for example e1e9c0d8-13bb-11de-8293-000c2922ed0a) is the id attribute of the rule in the XML database. filter fi_class_rule { match("e1e9c0d8-13bb-11de-8293-000c2922ed0a" value(".classifier.rule_id") type("string") ); };

The message-segments parsed by the pattern parsers can also be used as macros as well. To accomplish this, you have to add a name to the parser, and then you can use this name as a macro that refers to the parsed value of the message.

Example 4.43. Using pattern parsers as macros

For example, you want to parse messages of an application that look like "Transaction: <type>.", where <type> is a string that has different values (for example refused, accepted, incomplete, and so on). To parse these messages, you can use the following pattern: 'Transaction: @ESTRING::.@' Here the @ESTRING@ parser parses the message until the next full stop character. To use the results in a filter or a filename template, include a name in the parser of the pattern, for example: 'Transaction: @ESTRING:TRANSACTIONTYPE:.@' After that, add a custom template to the logpath that uses this template. For example, to select every accepted transaction, use the following custom filter in the log path: match("accepted" value("TRANSACTIONTYPE"));

	Note
	The above macros can be used in database columns and filename templates as well, if you create custom templates for the destination or logspace. Use a consistent naming scheme for your macros, for example, APPLICATIONNAME_MACRONAME.

The syslog-ng application can rewrite parts of log messages: it can search and replace text, and also set a specific field to a specified value. Rewriting messages is often used in conjunction with message parsing (see Section 4.8, Parsing messages).

Rewrite rules are similar to filters: they must be defined in the syslog-ng configuration file and used in the log statement.

	Note
	The order of filters, rewriting rules, and parsers in the log statement is important, as they are processed sequentially.

To create replace a part of the log message, define the string or regular expression to replace, the string to replace the original text (macros can be used as well), and the field of the message that the rewrite rule should process. Substitution rules can operate on any value available via macros, for example HOST, MESSAGE, PROGRAM, or any user-defined macros created using parsers (see Section 6.6, Message parsers for details). The only exceptions are the FACILITY, SEVERITY, TAGS, and the date-related fields, which cannot be rewritten. Substitution rules use the following syntax:

Declaration:
            rewrite <name_of_the_rule>
                {subst("<string or regular expression to find>", "<replacement string>", value(<field name>), flags());};

A single substitution rule can include multiple substitutions that are applied sequentially to the message. Note that rewriting rules must be included in the log statement to have any effect.

	Tip
	For case-insensitive searches, add the flags(ignore-case) option; to replace every occurrence of the string, add flags(global) option.

Example 4.44. Using substitution rules

The following example replaces the first occurrence of the string IP in the text of the message with the string IP-Address. rewrite r_rewrite_subst{subst("IP", "IP-Address", value("MESSAGE"));}; To replace every occurrence, use: rewrite r_rewrite_subst{subst("IP", "IP-Address", value("MESSAGE"), flags("global"));}; Multiple substitution rules are applied sequentially; the following rules replace the first occurrence of the string IP with the string IP-Addresses. rewrite r_rewrite_subst{subst("IP", "IP-Address", value("MESSAGE")); subst("Address", "Addresses", value("MESSAGE"));};

To set a field of the message to a specific value, define the string to include in the message, and the field where it should be included. Setting a field can operate on any value available via macros, for example HOST, MESSAGE, PROGRAM, or any user-defined macros created using parsers (for details, see Section 6.6, Message parsers). The only exceptions are the FACILITY, SEVERITY, TAGS, and the date-related fields, which cannot be rewritten. Note that the rewrite operation completely replaces any previous value of that field. Use the following syntax:

Declaration:
            rewrite <name_of_the_rule>
                {set("<string to include>", value(<field name>));};

Example 4.45. Setting message fields to a particular value

The following example sets the HOST field of the message to myhost. rewrite r_rewrite_set{set("myhost", value("HOST"));}; The following example sets the sequence ID field of the RFC5424-formatted (IETF-syslog) messages to a fixed value. rewrite r_sd { set("55555" value(".SDATA.meta.sequenceId")); }; It is also possible to set the value of a field that does not exist yet, and create a new name-value pair that is associated with the message. The following example created the MODIFIED field and sets its value to yes. If you use the $MODIFIED macro in a template or SQL table, its value will be yes for every message that was processed with this rewrite rule, and empty for every other message. rewrite r_rewrite_set{set("yes", value("MODIFIED"));};

The syslog-ng application has a number of global options governing DNS usage, the timestamp format used, and other general points. Each option may have parameters, similarly to driver specifications. To set global options, add an option statement to the syslog-ng configuration file using the following syntax:

options { option1(params); option2(params); ... };

	Example 4.46. Using global options
	To disable domain name resolving, add the following line to the syslog-ng configuration file: options { use_dns(no); };

For a detailed list of the available options, see Section 6.9, Global options. For important global options and recommendations on their use, see Chapter 5, Best practices and examples.

To enable disk-based buffering, use the log_disk_fifo_size() parameter in the destination to set the size of the disk buffer in bytes. Note that this value applies to every destination separately; every destination will have its own diskbuffer file, even if the parameter is set as a global option. For details on how disk-based buffering works, see Section 2.14, Using disk-based buffering. Disk buffers can be used with tcp(), tcp6(), syslog() (when using the tcp or tls transport methods), and sql() destinations. The number of messages that the disk buffer can store depends on the size (length) of the actual messages. The maximum length of a message is limited by the log_msg_size() parameter, which is 8192 bytes by default.

The disk buffer is located under /opt/syslog-ng/var/ on every platform.

Example 4.47. Enabling disk-based buffering

The following example turns on disk-based buffering for the destination. The size of the disk buffer is 4 194 304 bytes (4 megabytes). In a worst-case situation, using the default value of the log_msg_size() parameter (8192 bytes), this disk buffer can store at least 512 messages. Typical log messages are about 300-500 bytes long, so a disk buffer of 4 megabytes can store over 8000 messages. Set the size of the disk buffer based on the average size and number of messages, and the longest estimated downtime of the server. destination d_tcp { tcp("10.1.2.3" port(1999) log_disk_fifo_size(4194304)); };

This section describes how to configure TLS encryption in syslog-ng. For the concepts of using TLS in syslog-ng, see Section 2.7, Secure logging using TLS.

Create an X.509 certificate for the syslog-ng server.

Note

The subject_alt_name parameter (or the Common Name parameter if the subject_alt_name parameter is empty) of the server's certificate must contain the hostname or the IP address (as resolved from the syslog-ng clients and relays) of the server (for example syslog-ng.example.com). Alternatively, the Common Name or the subject_alt_name parameter can contain a generic hostname, for example *.example.com. Note that if the Common Name of the certificate contains a generic hostname, do not specify a specific hostname or an IP address in the subject_alt_name parameter.

Complete the following steps on every syslog-ng client host. Examples are provided using both the legacy BSD-syslog protocol (using the tcp() driver) and the new IETF-syslog protocol standard (using the syslog() driver):

Complete the following steps on the syslog-ng server:

For the details of the available tls() options, see Section 6.10, TLS options.

This section describes how to configure mutual authentication between the syslog-ng server and the client. Configuring mutual authentication is similar to configuring TLS (see Section 4.13, Encrypting log messages with TLS), but the server verifies the identity of the client as well. Therefore, each client must have a certificate, and the server must have the certificate of the CA that issued the certificate of the clients. For the concepts of using TLS in syslog-ng, see Section 2.7, Secure logging using TLS.

Complete the following steps on every syslog-ng client host. Examples are provided using both the legacy BSD-syslog protocol (using the tcp() driver) and the new IETF-syslog protocol standard (using the syslog() driver):

Complete the following steps on the syslog-ng server:

For details on the available tls() options, see Section 6.10, TLS options.

The syslog-ng Premium Edition server operates only if a valid license file is present on the host. The license file is called license.txt, and is located in the same directory as the syslog-ng configuration file.

	Warning
	The license.txt file must be readable to the user running the syslog-ng process.

To install a license file, copy it to the directory where the configuration file is stored. For the location of the license file, see Section 4.1, The syslog-ng configuration file.

To upgrade a license file, simply overwrite the old license file with the new one.

	Note
	The license file is needed only when running syslog-ng Premium Edition in server mode.

This section provides tips and guidelines about troubleshooting problems related to syslog-ng. Troubleshooting the syslog-ng Agent for Windows application is discussed in Chapter 4, Troubleshooting syslog-ng Agent for Windows in The syslog-ng Agent for Windows 3.2 Administrator Guide.

Tip

As a general rule, first try to get logging the messages to a local file. Once this is working, you know that syslog-ng is running correctly and receiving messages, and you can proceed to forwarding the messages to the server. If the syslog-ng server does not receive the messages, use tcpdump or a similar packet sniffer tool on the client to verify that the messages are sent correctly, and on the server to verify that it receives the messages. If syslog-ng is closing the connections for no apparent reason, be sure to check the log messages of syslog-ng. You might also want to run syslog-ng with the --verbose or --debug command-line options for more-detailed log messages. Starting from syslog-ng PE version 3.1, you can enable these messages without restarting syslog-ng using the syslog-ng-ctl verbose --set=on command. For details, see the syslog-ng-ctl man page at syslog-ng-ctl(1). Similarly, build up encrypted connections step-by-step: first create a working unencrypted (for example TCP) connection, then add TLS encryption, and finally client authentication if needed.

This section provides general tips and recommendations on using syslog-ng. Some of the recommendations are detailed in the subsequent sections.

When syslog-ng is receiving messages from a large number of TCP or unix-stream connections, the CPU usage of syslog-ng might increase even if the number of messages is low. By default, syslog-ng processes every message when it is received. To reduce the CPU usage, process the incoming messages in batches. To accomplish this, instruct syslog-ng to wait for a short time before processing a message. During this period additional messages might arrive that can be processed together with the original message. To process log messages in batches, set the time_sleep() option (measured in milliseconds) to a non-zero value. Include the following line in your syslog-ng configuration:

options { time_sleep(20); };

	Note
	It is not recommended to increase the time_sleep() parameter above 100ms, as that might distort timestamps, slow down syslog-ng, and cause messages to be dropped. When modifying the time_sleep() option, also adjust the log_fetch_limit() and log_fifo_size() options accordingly.

The max_connections() parameter limits the number of parallel connections for the source.

If adjusting the time_sleep() option is not desired for some reason, an alternative solution is to use unix-stream(), udp() and unix-dgram() sources instead of tcp() connections.

This section provides tips on optimizing the performance of syslog-ng. Optimizing the performance is important for syslog-ng hosts that handle large traffic.

The syslog-ng application can resolve the hostnames of the clients and include them in the log messages. However, the performance of syslog-ng is severely degraded if the domain name server is unaccessible or slow. Therefore, it is not recommended to resolve hostnames in syslog-ng. If you must use name resolution from syslog-ng, consider the following:

	Note
	Domain name resolution is important mainly in relay and server mode.