The syslog-ng 3.0 Administrator Guide

The syslog-ng application is a flexible and highly scalable system logging application that is ideal for creating centralized and trusted logging solutions. The main features of syslog-ng are summarized below.

The syslog-ng application is not log analysis software. It can filter log messages and select only the ones matching certain criteria. It can even convert the messages and restructure them to a predefined format, or parse the messages and segment them into different fields. But syslog-ng cannot interpret and analyze the meaning behind the messages, or recognize patterns in the occurrence of different messages.

Log messages contain information about the events happening on the hosts. Monitoring system events is essential for security and system health monitoring reasons.

The original syslog protocol separates messages based on the priority of the message and the facility sending the message. These two parameters alone are often inadequate to consistently classify messages, as many applications might use the same facility — and the facility itself is not even included in the log message. To make things worse, many log messages contain unimportant information. The syslog-ng application helps you to select only the really interesting messages, and forward them to a central server.

Company policies or other regulations often require log messages to be archived. Storing the important messages in a central location greatly simplifies this process.

For details on how can you use syslog-ng to comply with various regulations, see the Regulatory compliance and system logging whitepaper available at http://www.balabit.com/support/documentation/

Version 3.0 of syslog-ng includes the following main features:

Version 3.0 of syslog-ng includes the following important changes:

The syslog-ng application is used worldwide by companies and institutions who collect and manage the logs of several hosts, and want to store them in a centralized, organized way. Using syslog-ng is particularly advantageous for:

The following is a list of public references — companies who use syslog-ng in their production environment:

The syslog-ng PE application is officially supported on the following platforms. Note that the following table is for general reference only, and is not always accurate about the supported platforms and options available for specific platforms. The latest version of this table is available at http://www.balabit.com/network-security/syslog-ng/central-syslog-server/.

The central syslog-ng server cannot be installed on Microsoft Windows platforms. The syslog-ng Agent for Windows capable of forwarding eventlog messages to the central server is available on the x86 and x86_64 architecture for Microsoft Windows XP, Microsoft Windows 2003 Server, Microsoft Windows Vista, and Microsoft Windows 2008 Server. The syslog-ng Agent is available only in syslog-ng Premium Edition.

The central syslog-ng server can be installed on the IBM System i platform, but the syslog-ng Agent for IBM System i is needed to collect the native logs of IBM System i (see Chapter 6, Collecting logs from IBM System i). The syslog-ng Agent for IBM System i is a commercial product independent from both syslog-ng OSE and PE, and must be licensed separately.

For syslog-ng OSE, precompiled binary packages are available for free for the supported Linux and BSD platforms at http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/upgrades/. Precompiled binary packages for HP-UX, IBM AIX, and Solaris are available for an annual fee at the BalaBit webshop at http://www.balabit.com/. For the list of available platforms, see http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/support/.

Typically, syslog-ng is used to manage log messages and implement centralized logging, where the aim is to collect the log messages of several devices on a single, central log server. The different devices — called syslog-ng clients — all run syslog-ng, and collect the log messages from the various applications, files, and other sources. The clients send all important log messages to the remote syslog-ng server, where the server sorts and stores them.

The syslog-ng application reads incoming messages and forwards them to the selected destinations. The syslog-ng application can receive messages from files, remote hosts, and other sources.

Log messages enter syslog-ng in one of the defined sources, and are sent to one or more destinations.

Sources and destinations are independent objects; log paths define what syslog-ng does with a message, connecting the sources to the destinations. A log path consists of one or more sources and one or more destinations; messages arriving to a source are sent to every destination listed in the log path. A log path defined in syslog-ng is called a log statement.

Optionally, log paths can include filters. Filters are rules that select only certain messages, for example, selecting only messages sent by a specific application. If a log path includes filters, syslog-ng sends only the messages satisfying the filter rules to the destinations set in the log path.

Other optional elements that can appear in log statements are parsers and rewriting rules. Parsers segment messages into different fields to help processing the messages, while rewrite rules modify the messages by adding, replacing, or removing parts of the messages.

The following procedure illustrates the route of a log message from its source on the syslog-ng client to its final destination on the central syslog-ng server.

The syslog-ng Premium Edition application has three distinct modes of operation: Client, Server, and Relay. The syslog-ng application running on a host determines the mode of operation automatically based on the license and the configuration file.

The syslog-ng application uses the following objects:

For details on the above objects, see Section 3.2, Defining global objects.

The syslog-ng application supports messages originating from different timezones. The original syslog protocol does not include timezone information, but syslog-ng provides a solution by extending the syslog protocol to include the timezone in the log messages. The syslog-ng application also enables administrators to supply timezone information for legacy devices which do not support the protocol extension.

Timezone information is associated with messages entering syslog-ng is selected using the following algorithm:

The syslog-ng application receives the timezone and daylight saving information from the operating system it is installed on. If the operating system handles daylight saving correctly, so does syslog-ng.

The syslog-ng application can send and receive log messages securely over the network using the Transport Layer Security (TLS) protocol. TLS is an encryption protocol over the TCP/IP network protocol, so it can be used only with TCP-based sources and destinations ( tcp() and tcp6()).

TLS uses certificates to authenticate and encrypt the communication, as illustrated on the following figure:

The client authenticates the server by requesting its certificate and public key. Optionally, the server can also request a certificate from the client, thus mutual authentication is also possible.

In order to use TLS encryption in syslog-ng, the following elements are required:

When using mutual authentication to verify the identity of the clients, the following elements are required:

Mutual authentication ensures that the syslog-ng server accepts log messages only from authorized clients.

See Section 3.13, Encrypting log messages with TLS for details on configuring TLS communication in syslog-ng.

The Premium Edition of syslog-ng can store log messages securely in encrypted, compressed and timestamped binary files. Timestamps can be requested from an external Timestamping Authority (TSA).

Logstore files consist of individual chunks, every chunk can be encrypted, compressed, and timestamped separately. Chunks contain log message data, chunk size defaults to 128k (about 1MB worth of compressed logs). Chunks are closed when their size reaches the limit set in the chunk_size parameter, or when the time limit set in the chunk_time parameter expires and a new message arrives. Specifically, when a new message arrives to the logstore, syslog-ng checks if chunk_time time has elapsed since the last message has arrived. If it has, then the old chunk is closed and the new message is written into a new chunk.

The syslog-ng PE application generates an SHA-1 hash for every chunk to verify the integrity of the chunk. The hashes of the chunks are chained together to prevent injecting chunks into the logstore file. The syslog-ng application can encrypt the logstore using the aes128 algorithm in CBC mode; the hashing (HMAC) algorithm is hmac-sha1. Currently it is not possible to use other algorithms.

The syslog-ng application can dynamically create filenames, directories, or names of database tables using macros that help you organize your log messages. Macros refer to a property or a part of the log message, for example, the $HOST macro refers to the name or IP address of the client that sent the log message, while $DAY is the day of the month when syslog-ng has received the message. Using these macros in the path of the destination log files allows you for example to collect the logs of every host into separate files for every day.

A set of macros can be defined as a template object and used in multiple destinations.

Another use of macros and templates is to customize the format of the syslog message, for example to add elements of the message header to the message text. Note that if a message uses the IETF-syslog format, only the text of the message can be customized, the structure of the header is fixed.

For details on using templates and macros, see Section 3.7, Templates and macros and Section 8.5, Macros.

The filters and default macros of syslog-ng work well on the headers and metainformation of the log messages, but are rather limited when processing the content of the messages. Parsers can segment the content of the messages into name-value pairs, and these names can be used as user-defined macros. Subsequent filtering or other type of processing of the message can use these custom macros to refer to parts of the message.

Parsers are global objects most often used together with filters and rewrite rules. For details on using parsers, see Section 3.8, Parsing messages and Section 8.6, Message parsers.

The syslog-ng application can rewrite parts of the messages using rewrite rules. Rewrite rules are global objects similar to parsers and filters and can be used in log paths. The syslog-ng application has two methods to rewrite parts of the log messages: replacing (setting) a part of the message to a fix value, and a general search-and-replace mode.

Substitution completely replaces a specific part of the message that is referenced using a built-in or user-defined macro.

General rewriting searches for a string in the entire message (or only a part of the message specified by a macro) and replaces it with another string Optionally, this replacement string can be a template that contains macros.

For details on using rewrite rules, see Section 3.10, Rewriting messages and Section 8.7, Rewriting messages.

The syslog-ng application can compare the contents of the received log messages to predefined message patterns. By comparing the messages to the known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and for example can label the messages as user login, application crash, file transfer, etc. events.

To find the pattern that matches a particular message, syslog-ng uses a method called longest prefix match radix tree. This means that syslog-ng creates a tree structure of the available patterns, where the different characters available in the patterns for a given position are the branches of the tree. This is also illustrated on the following figure:

To classify a message, syslog-ng selects the first character of the message (the text of message, not the header), and selects the patterns starting with this character, other patterns are ignored for the rest of the process. After that, the second character of the message is compared to the second character of the selected patterns. Again, matching patterns are selected, and the others discarded. This process is repeated until a single pattern completely matches the message, or no match is found. In the latter case, the message is classified as unknown, otherwise the class of the matching pattern is assigned to the message.

To make the message classification more flexible and robust, the patterns can contain pattern parsers: elements that match on a set of characters. For example, the NUMBER parser matches on any integer numbers (e.g., 1, 123, 894054, etc.). Other pattern parsers match on various strings and IP addresses. For the details of available pattern parsers, see Section 8.6.2.1, Using pattern parsers.

The functionality of the pattern database is similar to that of the logcheck project, but it is much easier to write and maintain the patterns used by syslog-ng, than the regular expressions used by logcheck. Also, it is much easier to understand syslog-ng pattens than regular expressions.

Pattern matching based on regular expressions is computationally very intensive, especially when the number of patterns increases. The solution used by syslog-ng can be performed real-time, and is independent from the number of patterns, so it scales much better. The following patterns describe the same message: Accepted password for bazsi from 10.50.0.247 port 42156 ssh2

A regular expression matching this message from the logcheck project: Accepted \ (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam) \ for [^[:space:]]+ from [^[:space:]]+ port [0-9]+( (ssh|ssh2))?

A syslog-ng database pattern for this message: Accepted @QSTRING:auth_method: @ for@QSTRING:username: @from\ @QSTRING:client_addr: @port @NUMBER:port:@ ssh2

For details on using pattern databases to classify log messages, see Section 3.9, Classifying messages and Section 8.6.2, Pattern databases.

This section describes the internal message-processing model of syslog-ng, as well as the flow-control feature that can prevent message losses. To use flow-control, the flow-control flag must be enabled for the particular log path.

The syslog-ng application monitors (polls) the sources defined in its configuration file, periodically checking each source for messages. When a log message is found in one of the sources, syslog-ng polls every source and reads the available messages. These messages are processed and put into the output buffer of syslog-ng (also called fifo). From the output buffer, the operating system sends the messages to the appropriate destinations.

In large-traffic environments many messages can arrive during a single poll loop, therefore syslog-ng reads only a fixed number of messages from each source. The log_fetch_limit() option specifies the number of messages read during a poll loop from a single source.

Every destination has its own output buffer. The output buffer is needed because the destination might not be able to accept all messages immediately. The log_fifo_size() parameter sets the size of the output buffer. The output buffer must be larger than the log_fetch_limit() of the sources, to ensure that every message read during the poll loop fits into the output buffer. If the log path sends messages to a destination from multiple sources, the output buffer must be large enough to store the incoming messages of every source.

TCP and unix-stream sources can receive the logs from several incoming connections (e.g., many different clients or applications). For such sources, syslog-ng reads messages from every connection, thus the log_fetch_limit() parameter applies individually to every connection of the source.

The flow-control of syslog-ng introduces a control window to the source that tracks how many messages can syslog-ng accept from the source. Every message that syslog-ng reads from the source lowers the window size by one; every message that syslog-ng successfully sends from the output buffer increases the window size by one. If the window is full (i.e., its size decreases to zero), syslog-ng stops reading messages from the source. The initial size of the control window is by default 100: the log_fifo_size() must be larger than this value in order for flow-control to have any effect. If a source accepts messages from multiple connections, all messages use the same control window.

When flow-control is used, every source has its own control window. As a worst-case situation, the output buffer of the destination must be set to accommodate all messages of every control window, that is, the log_fifo_size() of the destination must be greater than number_of_sources*log_iw_size(). This applies to every source that sends logs to the particular destination. Thus if two sources having several connections and heavy traffic send logs to the same destination, the control window of both sources must fit into the output buffer of the destination. Otherwise, syslog-ng does not activate the flow-control, and messages may be lost.

The Premium Edition of syslog-ng stores messages on the local hard disk if the central log server or the network connection to the server becomes unavailable. The syslog-ng application automatically sends the stored messages to the server when the connection is reestablished. The disk buffer is used as a queue: when the connection to the server is reestablished, syslog-ng sends the messages to the server in the order they were received.

Disk buffers can be used with tcp(), tcp6(), syslog() (when using the tcp or tls transport methods), and sql() destinations. Every such destination uses a separate disk buffer (similarly to the output buffers controlled by log_fifo_size()). The hard disk space is not pre-allocated, so ensure that there is always enough free space to store the disk buffers even when the disk buffers are full.

If syslog-ng is restarted (using the /etc/init.d/syslog-ng restart command), it automatically saves any unsent messages of the disk buffer and the output queue. After the restart, syslog-ng sends the saved messages to the server. In other words, the disk buffer is persistent.

The syslog-ng application handles outgoing messages the following way:

The syslog-ng Premium Edition application is licensed on a per-host basis: the syslog-ng server accepts connections only from the number of individual hosts (also called log source hosts) specified in its license file.

A log source host is a host or network device (including syslog-ng clients and relays) that sends logs to the syslog-ng server. Log source hosts can be servers, routers, desktop computers, or other devices capable of sending syslog messages or running syslog-ng. Log source hosts are identified by their IP addresses, so virtual machines and vhosts are separately counted. Licenses are available for 5, 10, 25, 50, 100, 150, 200, 250, 300, 500, 750, 1000, and unlimited number of log source hosts.

Buying a syslog-ng server license permits you to perform the following:

The syslog-ng Open Source Edition application is distributed under version 2 of the GNU General Public License. See Appendix 3, GNU General Public License for details.

As of October 2009, the following release policy applies to syslog-ng:

Multiple syslog-ng servers can be run in fail-over mode. The syslog-ng application does not include any internal support for this, as clustering support must be implemented on the operating system level. A tool that can be used to create UNIX clusters is Heartbeat (see http://www.linux-ha.org/ for details).

During the course of a message from the sending application to the final destination of the message, there are a number of locations where a message may be lost, even though syslog-ng does its best to avoid message loss. Usually losing messages can be avoided with careful planning and proper configuration of syslog-ng and the hosts running syslog-ng. The following list shows the possible locations where messages may be lost, and provides methods to minimize the risk of losing messages.

The following sections describe the structure of log messages. Currently there are two standard syslog message formats:

The syslog-ng application is configured by editing the syslog-ng.conf file. Use any regular text editor application to modify the file. The precompiled syslog-ng packages include sample configuration files as well.

Every syslog-ng configuration file must begin with a line containing the version information of syslog-ng. For syslog-ng version 3.0, this line looks like:

@version:3.0

Versioning the configuration file was introduced in syslog-ng 3.0. If the configuration file does not contain the version information, syslog-ng assumes that the file is for syslog-ng version 2.x. In this case it interprets the configuration and sends warnings about the parts of the configuration that should be updated. Version 3.0 and later will correctly operate with configuration files of version 2.x, but the default values of certain parameters have changed since 3.0.

All identifiers, option names and attributes, and any other strings used in the syslog-ng configuration file are case sensitive. Objects must be defined before they are referenced in another statement.

The syslog-ng.conf and license.txt files are located under the /opt/syslog-ng/etc/ directory.

On Microsoft Windows platforms the syslog-ng agent stores its configuration in the system registry, and can be configured from a graphical interface. See Chapter 5, Collecting logs from Windows hosts for details.

Global objects (e.g., sources, destinations, log paths, or filters) are defined in the syslog-ng configuration file. Object definitions consist of the following elements:

The syntax is summarized as follows:

type identifier { parameters };

Objects have parameters; some of them are required, others are optional. Required parameters are positional, meaning that they must be specified in a defined order. Optional arguments can be specified in any order using the option(value) format. If a parameter (optional or required) is not specified, its default value is used. The parameters and their default values are listed in the reference section of the particular object. See Chapter 8, Reference for details.

To add comments to the configuration file, start a line with # and write your comments. These lines are ignored by syslog-ng.

# Comment: This is a stream source
source s_demo_stream { 
           unix-stream("/dev/log" max-connections(10) group(log)); };

A source is where syslog-ng receives log messages. Sources consist of one or more drivers, each defining where and how messages are received.

To define a source, add a source statement to the syslog-ng configuration file using the following syntax:

source <identifier> { source-driver(params); source-driver(params); ... };

Define a source only once. The same source can be used in several log paths. Duplicating sources causes syslog-ng to open the source (TCP/IP port, file, etc.) more than once, which might cause problems. For example, include the /dev/log file source only in one source statement, and use this statement in more than one log path if needed.

To collect log messages on a specific platform, it is important to know how the native syslogd communicates on that platform. The following table summarizes the operation methods of syslogd on some of the tested platforms:

Each possible communication mechanism has a corresponding source driver in syslog-ng. For example, to open a unix socket with SOCK_DGRAM style communication use the driver unix-dgram. The same socket using the SOCK_STREAM style — as used under Linux — is called unix-stream.

The following table lists the source drivers available in syslog-ng.

For a complete description of the parameters of the above drivers, see Section 8.1, Source drivers.

A destination is where a log message is sent if the filtering rules match. Similarly to sources, destinations consist of one or more drivers, each defining where and how messages are sent.

To define a destination, add a destination statement to the syslog-ng configuration file using the following syntax:

destination <identifier> { 
                destination-driver(params); destination-driver(params); ... };

The following table lists the destination drivers available in syslog-ng.

For detailed list of driver parameters, see Section 8.2, Destination drivers.

Log paths determine what happens with the incoming log messages. Messages coming from the sources listed in the log statement and matching all the filters are sent to the listed destinations.

To define a log path, add a log statement to the syslog-ng configuration file using the following syntax:

log {
    source(s1); source(s2); ... 
    optional_element(filter1|parser1|rewrite1); optional_element(filter2|parser2|rewrite2);... 
    destination(d1); destination(d2); ... 
    flags(flag1[, flag2...]);
    };

All matching log statements are processed by default, and the messages are sent to every matching destination by default. So a single log message might be sent to the same destination several times, provided the destination is listed in several log statements, and it can be also sent to several different destinations.

This default behavior can be changed using the flags() parameter. Flags apply to individual log paths; they are not global options. The following flags available in syslog-ng:

For details on the individual flags, see Section 8.3, Log path flags. The effect and use of the flow-control flag is detailed in Section 2.13, Managing incoming and outgoing messages with flow-control.

Filters perform log routing within syslog-ng: a message passes the filter if the filter expression is true for the particular message. If a log statement includes filters, the messages are sent to the destinations only if they pass all filters of the log path. For example, a filter can select only the messages originating from a particular host. Complex filters can be created using filter functions and logical boolean expressions.

To define a filter, add a filter statement to the syslog-ng configuration file using the following syntax:

filter <identifier> { expression; };

The expression may contain the following elements:

The value() parameter limits the scope of a filter function to the scope of a macro. For example, to limit the scope of the match() filter to the text part of the message, use:

match("keyword" value("MESSAGE"))

The value() parameter accepts both built-in macros and user-defined ones created with a parser. Do not prefix the macros with the $ sign. For details on macros and parsers, see Section 3.7, Templates and macros and Section 3.8, Parsing messages.

In the extended regular expressions, the characters ()[].*?+^$| are used as special symbols. Therefore, these characters have to be preceded with a backslash (\) if they are meant literally. For example, the \$40 expression matches the $40 string. Backslashes have to be escaped as well if they are meant literally. For example, the \\d expression matches the \d string.

By default, all regular expressions are case sensitive. To disable the case sensitivity of the expression, add the flags(ignore-case) option to the regular expression.

filter demo_regexp_insensitive { host("system" flags(ignore-case)); };

For details on regular expressions, see Section 8.8, Regular expressions.

The level() filter can select messages corresponding to a single importance level, or a level-range. To select messages of a specific level, use the name of the level as a filter parameter, e.g., use the following to select warning messages:

level(warning)

To select a range of levels, include the beginning and the ending level in the filter, separated with two dots (..). For example, to select every message of error or higher level, use the following filter:

level(err..emerg)

Similarly, messages sent by a range of facilities can also be selected. Note that this is only possible when using the name of the facilities. It is not possible to select ranges the numerical codes of the facilities.

facility(local0..local5)

For a complete list of the available levels and facilities, see Section 8.4, Filter functions.

For a complete description on the above functions, see Section 8.4, Filter functions.

The syslog-ng application allows you to define message templates, and reference them from every object that can use a template. Templates can be used to create standard message formats or filenames. Templates can reference one or more macros (e.g., date, the hostname, etc.). See Section 8.5, Macros for a list of macros available in the Linux/Unix versions of syslog-ng, and Section 5.6, Customizing the message format for the macros of the syslog-ng Agent for Windows application. Fields from the structured data (SD) part of messages using the new IETF-syslog standard can also be used as macros.

Template objects have a single option called template_escape, which is disabled by default (template_escape(no)). This behavior is useful when the messages are passed to an application that cannot handle escaped characters properly. Enabling template escaping (template_escape(yes)) causes syslog-ng to escape the ' and " characters from the messages.

Macros can be included by prefixing the macro name with a $ sign, just like in Bourne compatible shells. Regarding braces around macro names, the following two formats are equivalent "$MSG" and "${MSG}".

Default values for macros can also be specified by appending the :- characters and the default value to the macro, e.g.,

${HOST:-default_hostname}

The macros related to the date of the message (e.g.: ISODATE, HOUR, etc.) have two further versions each: one with the S_ and one with the R_ prefix (e.g.: S_DATE and R_DATE ). The S_DATE macro represents the date found in the log message, i.e. when the message was sent by the original application. R_DATE is the date when syslog has received the message.

DATE equals either S_DATE or R_DATE, depending on the global option set in the now deprecated use_time_recvd() parameter (see Section 8.9, Global options).

By default, syslog-ng sends messages using the following template: $ISODATE $HOST $MSGHDR$MSG\n. (The $MSGHDR$MSG part is written together because the $MSGHDR macro includes a trailing whitespace.)

The syslog-ng application can separate parts of log messages (i.e., the contents of the $MSG macro) to named fields (columns). These fields act as user-defined macros that can be referenced in message templates, file- and tablenames, etc.

Parsers are similar to filters: they must be defined in the syslog-ng configuration file and used in the log statement.

To create a parser, define the columns of the message, the delimiter or separator characters, and optionally the characters that are used to escape the delimiter characters (quote-pairs). For the list of parser parameters, see Section 8.6, Message parsers.

Declaration:
            parser parser_name {
            csv-parser(column1, column2, ...)
            delimiters()
            quote-pairs()
            };

Column names work like macros. Always use a prefix to identify the columns of the parsers, e.g., MYPARSER1.COLUMN1, MYPARSER2.COLUMN2, etc. Column names starting with a dot (e.g., .HOST) are reserved for use by syslog-ng.

Multiple parsers can be used to split a part of an already parsed message into further segments.

To classify messages using a pattern database, include a db_parser() statement in your syslog-ng configuration file using the following syntax:

Declaration:
        parser <identifier> {db_parser(file("<database_filename>"));};

Note that using the parser in a log statement only performs the classification, but does not automatically do anything with the results of the classification.

Sample pattern databases are available at the BalaBit Download page http://www.balabit.com/network-security/syslog-ng/log-server-appliance/. However, these are not directly usable in syslog-ng 3.0.x, because they are formatted according to the second version (V2) of the pattern database format, which is supported only by the syslog-ng Store Box (SSB) appliance version 1.0.x. The syslog-ng 3.0.x OSE and PE applications only support the first version (V1) of the pattern database; support for the V2 and V3 pattern database formats will be available in syslog-ng 3.1. In the meantime, you can create your own pattern database: see Section 8.6.2.3, Creating pattern databases for details.

The syslog-ng application can rewrite parts of log messages: it can search and replace text, and also set a specific field to a specified value. Rewriting messages is often used in conjunction with message parsing Section 3.8, Parsing messages.

Rewrite rules are similar to filters: they must be defined in the syslog-ng configuration file and used in the log statement.

To create replace a part of the log message, define the string or regular expression to replace, the string to replace the original text (macros can be used as well), and the field of the message that the rewrite rule should process. Substitution rules can operate on any value available via macros, e.g., HOST, MESSAGE, PROGRAM, or any user-defined macros created using parsers (see Section 8.6, Message parsers for details.). Substitution rules use the following syntax:

Declaration:
            rewrite <name_of_the_rule>
                {subst("<string or regular expression to find>", "<replacement string>", value(<field name>), flags());};

A single substitution rule can include multiple substitutions that are applied sequentially to the message. Note that rewriting rules must be included in the log statement to have any effect.

To set a field of the message to a specific value, define the string to include in the message, and the field where it should be included. Setting a field can operate on any value available via macros, e.g., HOST, MESSAGE, PROGRAM, or any user-defined macros created using parsers (see Section 8.6, Message parsers for details.). Note that this operation completely replaces any previous value of that field. Use the following syntax:

Declaration:
            rewrite <name_of_the_rule>
                {set("<string to include>", value(<field name>));};

The syslog-ng application has a number of global options governing DNS usage, the timestamp format used, and other general points. Each option may have parameters, similarly to driver specifications. To set global options, add an option statement to the syslog-ng configuration file using the following syntax:

options { option1(params); option2(params); ... };

For a detailed list of the available options, see Section 8.9, Global options. See Chapter 7, Best practices and examples for important global options and recommendations on their use.

To enable disk-based buffering, use the log_disk_fifo_size() parameter in the destination to set the size of the disk buffer in bytes. Note that this value applies to every destination separately; every destination will have its own diskbuffer file, even if the parameter is set as a global option. For details on how disk-based buffering works, see Section 2.14, Using disk-based buffering. Disk buffers can be used with tcp(), tcp6(), syslog() (when using the tcp or tls transport methods), and sql() destinations. The number of messages that the disk buffer can store depends on the size (length) of the actual messages. The maximum length of a message is limited by the log_msg_size() parameter, which is 8192 bytes by default.

The disk buffer is located under /opt/syslog-ng/var/ on every platform.

This section describes how to configure TLS encryption in syslog-ng. For the concepts of using TLS in syslog-ng, see Section 2.7, Secure logging using TLS.

Create an X.509 certificate for the syslog-ng server.

Complete the following steps on every syslog-ng client host. Examples are provided using both the legacy BSD-syslog protocol (using the tcp() driver) and the new IETF-syslog protocol standard (using the syslog() driver):

Complete the following steps on the syslog-ng server:

For the details of the available tls() options, see Section 8.10, TLS options.

This section describes how to configure mutual authentication between the syslog-ng server and the client. Configuring mutual authentication is similar to configuring TLS (see Section 3.13, Encrypting log messages with TLS), but the server verifies the identity of the client as well. Therefore, each client must have a certificate, and the server must have the certificate of the CA that issued the certificate of the clients. For the concepts of using TLS in syslog-ng, see Section 2.7, Secure logging using TLS.

Complete the following steps on every syslog-ng client host. Examples are provided using both the legacy BSD-syslog protocol (using the tcp() driver) and the new IETF-syslog protocol standard (using the syslog() driver):

Complete the following steps on the syslog-ng server:

For the details of the available tls() options, see Section 8.10, TLS options.

To configure syslog-ng on a client host, complete the following steps:

To configure syslog-ng on a relay host, complete the following steps:

In relay mode, syslog-ng cannot write messages received from network sources into files; the file() destination is disabled. The following sources are network sources: syslog(), tcp(), tcp6(), udp(), udp6().

To configure syslog-ng on a server host, complete the following steps:

The syslog-ng Premium Edition server operates only if a valid license file is present on the host. The license file is called license.txt, and is located in the same directory as the syslog-ng configuration file.

To install a license file, copy it to the directory where the configuration file is stored. See Section 3.1, The syslog-ng configuration file for the location of the license file.

To upgrade a license file, simply overwrite the old license file with the new one.

This section provides tips and guidelines about troubleshooting problems related to syslog-ng. Troubleshooting the syslog-ng Agent for Windows application is discussed in Section 5.10, Troubleshooting syslog-ng Agent for Windows.

This section describes how to install the syslog-ng application interactively using the binary installer. The installer has a simple interface: use the TAB or the arrow keys of your keyboard to navigate between the options, and Enter to select an option.

To install syslog-ng on operating systems that use the Red Hat Package Manager (RPM), complete the following steps. Installing syslog-ng automatically replaces the original syslog service. The following supported operating systems use RPM:

To install syslog-ng on operating systems that use the Debian Software Package (deb) format, complete the following steps. The following supported operating systems use this format:

To compile syslog-ng Open Source Edition (OSE) from the source code, complete the following steps. Alternatively, you can use the precompiled binary packages. Precompiled binary packages are available for free for the supported Linux and BSD platforms at http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/upgrades/. Precompiled binary packages for HP-UX, IBM AIX, and Solaris are available for an annual fee at the BalaBit webshop at http://www.balabit.com/. When you buy a binary package, you automatically receive the latest version of syslog-ng OSE for your platform, and all updates for a year.

For information on configuring syslog-ng, see the Chapter 3, Configuring syslog-ng.

If you need to uninstall syslog-ng for some reason, you have the following options:

Complete the following steps to configure your Microsoft SQL Server to enable remote logins and accept log messages from syslog-ng.