The syslog-ng 3.0 Administrator Guide

Document version: Tenth
Release date: January 8, 2010
Latest version available at the BalaBit Documentation Page

8.2. Destination drivers

Destination drivers output log messages to somewhere outside syslog-ng e.g., to a file or a network socket.

8.2.1. file()

The file driver outputs messages to the specified text file, or to a set of files.

The destination filename may include macros which get expanded when the message is written, thus a simple file() driver may create several files. For more information on available macros see Section 8.5, “Macros”.

[Warning] Warning

When creating several thousands separate log files, syslog-ng might not be able to open the required number of files. This might happen for example when using the $HOST macro in the filename while receiving messages from a large number of hosts. To overcome this problem, adjust the --fd-limit comman-line parameter of syslog-ng or the global ulimit parameter of your host. For setting the --fd-limit comman-line parameter of syslog-ng see the syslog-ng(8) manual page. For setting the ulimit parameter of the host, see the documentation of your operating system.

The file() destination has the following options:

Name Type Default Description
create_dirs() yes or no no Enable creating non-existing directories.
dir_group() string root The group of directories created by syslog-ng.
dir_owner() string root The owner of directories created by syslog-ng.
dir_perm() number 0600 The permission mask of directories created by syslog-ng. Log directories are only created if a file after macro expansion refers to a non-existing directory, and directory creation is enabled (see the create_dirs() option below). For octal numbers prefix the number with 0, e.g., use 0755 for rwxr-xr-x.
flags() no_multi_line, syslog-protocol empty set

Flags influence the behavior of the driver.

The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line.

The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver.
flush_lines() number Use global setting. Specifies how many lines are flushed to a destination at a time. Syslog-ng waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the network, but also increases message latency. The latency can be limited by the use of the flush_timeout option.
flush_timeout() time in milliseconds Use global setting. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. See the flush_lines option for more information.
frac_digits() number 0 The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format.. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.
fsync() yes or no no Forces an fsync() call on the destination fd after each write. Note: enabling this option may seriously degrade performance.
group() string root Set the group of the created file to the one specified.
local_time_zone() name of the timezone or the timezone offset The local timezone. Sets the timezone used when expanding filename and tablename templates. The timezone can be specified as using the name of the (e.g., time_zone("Europe/Budapest")), or as the timezone offset (e.g., +01:00). The valid timezone names are listed under the /usr/share/zoneinfo directory.
log_fifo_size() number Use global setting. The number of entries in the output buffer (output fifo).
overwrite_if_older() number 0 If set to a value higher than 0, syslog-ng checks when the file was last modified before starting to write into the file. If the file is older than the specified amount of time (in seconds), then syslog-ng removes the existing file and opens a new file with the same name. In combination with e.g., the $WEEKDAY macro, this can be used for simple log rotation, in case not all history has to be kept. (Note that in this weekly log rotation example if its Monday 00:01, then the file from last Monday is not seven days old, because it was probably last modified shortly before 23:59 last Monday, so it is actually not even six days old. So in this case, set the overwrite_if_older() parameter to a-bit-less-than-six-days, for example, to 518000 seconds.
owner() string root Set the owner of the created file to the one specified.
perm() number 0600 The permission mask of the file if it is created by syslog-ng. For octal numbers prefix the number with 0, e.g., use 0755 for rwxr-xr-x.
suppress() seconds 0 (disabled) If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.
template() string A format conforming to the default logfile format. Specifies a template defining the logformat to be used in the destination. Macros are described in Section 8.5, “Macros”. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
template_escape() yes or no no Turns on escaping ' and " in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
throttle() number 0 Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time_zone() timezone offset in seconds unspecified Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.
ts_format() rfc3164, bsd, rfc3339, iso rfc3164 Override the global timestamp format (set in the global ts_format() parameter) for the specific destination. See also Section 7.7, “A note on timezones and timestamps”.

Table 8.8. Options for file()


[Example] Example 8.12. Using the file() driver 
destination d_file { file("/var/log/messages" ); };
[Example] Example 8.13. Using the file() driver with macros in the file name and a template for the message 
destination d_file {
        file("/var/log/$YEAR.$MONTH.$DAY/messages"
             template("$HOUR:$MIN:$SEC $TZ $HOST [$LEVEL] $MSG $MSG\n")
             template_escape(no));
};

8.2.2. logstore()

The logstore driver stores log messages in binary files that can be encrypted, compressed, checked for integrity, and timestamped by an external Timestamping Authority (TSA). Otherwise, it is very similar to the file() destination.

To display the contents of a logstore file, use the logcat command supplied with syslog-ng, e.g., logcat /var/log/messages.lgs.

The destination filename may include macros which get expanded when the message is written, thus a simple logstore() driver may create several files. For more information on available macros see Section 8.5, “Macros”.

[Warning] Warning

When creating several thousands separate log files, syslog-ng might not be able to open the required number of files. This might happen for example when using the $HOST macro in the filename while receiving messages from a large number of hosts. To overcome this problem, adjust the --fd-limit comman-line parameter of syslog-ng or the global ulimit parameter of your host. For setting the --fd-limit comman-line parameter of syslog-ng see the syslog-ng(8) manual page. For setting the ulimit parameter of the host, see the documentation of your operating system.

The logstore() has a single required parameter that specifies the filename that stores the log messages.

Declaration:
    logstore(filename options());

The logstore() destination has the following options:

Name Type Default Description
chunk_size() number 128 Size of a logstore chunk in kilobytes. Note that this size refers to the compressed size of the chunk. Also, the gzip library used for compressing the messages has a 32k long buffer; messages may not appear in the actual logfile until this buffer is not filled. Logstore chunks are closed when they reach the specified size, or when the time limit set in chunk_time expires.
chunk_time() number 5 Time limit in seconds: syslog-ng PE closes the chunk if no new messages arrive until the time limit expires. Logstore chunks are closed when the time limit expires, or when they reach the size specified in the chunk_size parameter. If the time limit set in the time_reap parameter expires, the entire file is closed.
compress() number between 0-9 3 Compression level. 0 means uncompressed files, while 1-9 is the compression level used by gzip (9 means the highest but slowest compression, 3 is usually a good compromise).
create_dirs() yes or no no Enable creating non-existing directories.
dir_group() string root The group of the directories created by syslog-ng.
dir_owner() string root The owner of directories created by syslog-ng.
dir_perm() number 0600 The permission mask of directories created by syslog-ng. Log directories are only created if a file after macro expansion refers to a non-existing directory, and directory creation is enabled (see the create_dirs() option below). For octal numbers prefix the number with 0, e.g., use 0755 for rwxr-xr-x.
encrypt_certificate() filename none Name of a file, that contains an X.509 certificate (and the public key) in PEM format. The syslog-ng application uses this certificate to encrypt the logstore files which can be decrypted using the private key of the certificate.
flags() no_multi_line, syslog-protocol empty set

Flags influence the behavior of the driver.

The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line.

The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver.
flush_lines() number Use global setting. Specifies how many lines are flushed to a destination at a time. Syslog-ng waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the network, but also increases message latency. The latency can be limited by the use of the flush_timeout option.
flush_timeout() time in milliseconds Use global setting. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. See the flush_lines option for more information.
frac_digits() number 0 The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format.. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.
fsync() yes or no no Forces an fsync() call on the destination fd after each write. Note: enabling this option may seriously degrade performance.
group() string root Set the group of the created file to the one specified.
log_fifo_size() number Use global setting. The number of entries in the output buffer (output fifo).
owner() string root Set the owner of the created file to the one specified.
perm() number 0600 The permission mask of the file if it is created by syslog-ng. For octal numbers prefix the number with 0, e.g., use 0755 for rwxr-xr-x.
suppress() seconds 0 (disabled) If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.
template() string A format conforming to the default logfile format. Specifies a template defining the logformat to be used in the destination. Macros are described in Section 8.5, “Macros”. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
template_escape() yes or no no Turns on escaping ' and " in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
throttle() number 0 Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time_reap() number 60 The time to wait in seconds before an idle destination file is closed.
timestamp-freq() number in seconds Use global setting. The minimum time that should expire between two timestamping requests. When syslog-ng closes a chunk, it checks how much time has expired since the last timestamping request: if it is higher than the value set in the timestamp-freq parameter, it requests a new timestamp from the authority set in the timestamp-url parameter.
timestamp-url() string Use global setting. The URL of the Timestamping Authority used to request timestamps to sign logstore chunks. Note that syslog-ng currently supports only Timestamping Authorities that conform to RFC3161 Internet X.509 Public Key Infrastructure Time-Stamp Protocol, other protocols like Microsoft Authenticode Timestamping are not supported.
time_zone() timezone offset in seconds unspecified Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.
ts_format() rfc3164, bsd, rfc3339, iso rfc3164 Override the global timestamp format (set in the global ts_format() parameter) for the specific destination. See also Section 7.7, “A note on timezones and timestamps”.

Table 8.9. Options for logstore()


[Example] Example 8.14. Using the logstore() driver

A simple example saving and compressing log messages.

destination d_logstore { file("/var/log/messages.lgs" compress(5) ); };

A more detailed example that encrypts messages, modifies the parameters for closing chunks, and sets file privileges.

destination d_logstore { logstore("/var/log/messages-logstore.lgs"
    encrypt_certificate("/opt/syslog-ng/etc/syslog-ng/keys/10-100-20-40/public-certificate-of-the-server.pem")			
    chunk_size(100) 
    chunk_time(5)
    owner("balabit")
    group("balabit")
    perm(0777)
); };

8.2.3. pipe()

This driver sends messages to a named pipe like /dev/xconsole.

The pipe driver has a single required parameter, specifying the filename of the pipe to open. The filename can include macros.

Declaration:
    pipe(filename);
[Warning] Warning

As of syslog-ng Open Source Edition 3.0.2, pipes are created automatically. In earlier versions, you had to create the pipe using the mkfifo(1) command.

The pipe() destination has the following options:

Name Type Default Description
flags() no_multi_line, syslog-protocol empty set

Flags influence the behavior of the driver.

The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line.

The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver.
flush_lines() number Use global setting. Specifies how many lines are flushed to a destination at a time. Syslog-ng waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the network, but also increases message latency. The latency can be limited by the use of the flush_timeout option.
flush_timeout() time in milliseconds Use global setting. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. See the flush_lines option for more information.
frac_digits() number 0 The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format.. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.
fsync() yes or no no Forces an fsync() call on the destination fd after each write. Note: enabling this option may seriously degrade performance.
group() string root Set the group of the pipe to the one specified.
log_fifo_size() number Use global setting. The number of entries in the output buffer (output fifo).
owner() string root Set the owner of the pipe to the one specified.
perm() number 0600 The permission mask of the pipe. For octal numbers prefix the number with '0', e.g.: use 0755 for rwxr-xr-x.
suppress() seconds 0 (disabled) If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.
template() string A format conforming to the default logfile format. Specifies a template defining the logformat to be used in the destination. Macros are described in Section 8.5, “Macros”. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
template_escape() yes or no no Turns on escaping ' and " in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
throttle() number 0 Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time_zone() timezone offset in seconds unspecified Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.
ts_format() rfc3164, bsd, rfc3339, iso rfc3164 Override the global timestamp format (set in the global ts_format() parameter) for the specific destination. See also Section 7.7, “A note on timezones and timestamps”.

Table 8.10. Options for pipe()


[Example] Example 8.15. Using the pipe() driver 
destination d_pipe { pipe("/dev/xconsole"); };

8.2.4. program()

This driver starts an external application or script and sends the log messages to its standard input (stdin).

The program() driver has a single required parameter, specifying a program name to start.

Declaration: 
    program(command_to_run);

The program() destination has the following options:

Name Type Default Description
flags() no_multi_line, syslog-protocol empty set

Flags influence the behavior of the driver.

The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line.

The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver.
flush_lines() number Use global setting. Specifies how many lines are flushed to a destination at a time. Syslog-ng waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the network, but also increases message latency. The latency can be limited by the use of the flush_timeout option.
flush_timeout() time in milliseconds Use global setting. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. See the flush_lines option for more information.
frac_digits() number 0 The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format.. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.
fsync() yes or no no Forces an fsync() call on the destination fd after each write. Note: enabling this option may seriously degrade performance.
log_fifo_size() number Use global setting. The number of entries in the output buffer (output fifo).
suppress() seconds 0 (disabled) If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.
template() string A format conforming to the default logfile format. Specifies a template defining the logformat to be used in the destination. Macros are described in Section 8.5, “Macros”. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
template_escape() yes or no no Turns on escaping ' and " in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
throttle() number 0 Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time_zone() timezone offset in seconds unspecified Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.
ts_format() rfc3164, bsd, rfc3339, iso rfc3164 Override the global timestamp format (set in the global ts_format() parameter) for the specific destination. See also Section 7.7, “A note on timezones and timestamps”.

Table 8.11. Options for program()


[Example] Example 8.16. Using the program() destination driver 
destination d_prog { program("/bin/script" template("<$PRI>$DATE $HOST $MSG\n"); };

8.2.5. sql()

This driver sends messages into an SQL database. The sql() driver has the following required parameters: type, database, table, columns, values.

Declaration: 
    sql(database_type host_parameters database_parameters [options]);

The sql() destination has the following options:

Name Type Default Description
columns string list "date", "facility", "level", "host", "program", "pid", "message" Name of the columns storing the data in fieldname [dbtype] format. The [dbtype] parameter is optional, and specifies the type of the field. By default, syslog-ng creates text columns. Note that not every database engine can index text fields.
database string n/a Name of the database that stores the logs.
frac_digits() number 0 The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format.. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.
host hostname or IP address n/a Hostname of the database server. Note that Oracle destinations do not use this parameter, but retrieve the hostname from the /etc/tnsnames.ora file.
indexes string list "date", "facility", "host", "program" The list of columns that are indexed by the database to speed up searching. To disable indexing for the destination, include the empty indexes() parameter in the destination, simply omitting the indexes parameter will cause syslog-ng to request indexing on the default columns.
local_time_zone() name of the timezone or the timezone offset The local timezone. Sets the timezone used when expanding filename and tablename templates. The timezone can be specified as using the name of the (e.g., time_zone("Europe/Budapest")), or as the timezone offset (e.g., +01:00). The valid timezone names are listed under the /usr/share/zoneinfo directory.
log_disk_fifo_size() number 0 Size of the hard disk space in bytes that is used as disk buffer. Available only in syslog-ng Premium Edition when using the tcp(), tcp6(), syslog() (when using the tcp or tls transport methods), and sql() destinations. Can be also defined as a global option. See Section 2.14, “Using disk-based buffering” for details on using the disk buffer.
log_fifo_size() number Use global setting. The number of entries in the output buffer (output fifo).
null string   If the content of a column matches the string specified in the null() parameter, the contents of the column will be replaced with an SQL NULL value. If unset (by default), the option does not match on any string. See the Example 8.20, “Using SQL NULL values” for details.
password string n/a Password of the database user.
table string n/a Name of the database table to use (can include macros). When using macros, note that some databases limit the length of table names.
time_zone() timezone offset in seconds unspecified Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.
type mssql, mysql, oracle, pgsql, or sqlite3 n/a Specifies the type of the database, i.e., the DBI database driver to use. Use the mssql option to send logs to an MSSQL database. See the examples of the databases on the following sections for details.
username string n/a Name of the database user.
values string list "${R_YEAR}-${R_MONTH}-${R_DAY} ${R_HOUR}:${R_MIN}:${R_SEC}", "$FACILITY", "$LEVEL", "$HOST", "$PROGRAM", "$PID", "$MSGONLY" The parts of the message to store in the fields specified in the columns parameter.

Table 8.12. Options for sql()


[Note] Note

If you specify host="localhost", syslog-ng will use a socket to connect to the local database server. Use host="127.0.0.1" to force TCP communication between syslog-ng and the local database server.

To specify the socket to use, set and export the MYSQL_UNIX_PORT environment variable, e.g., MYSQL_UNIX_PORT=/var/lib/mysql/mysql.sock; export MYSQL_UNIX_PORT.

[Example] Example 8.17. Using the sql() driver

The following example sends the log messages into a PostgreSQL database running on the logserver host. The messages are inserted into the logs database, the name of the table includes the exact date and the name of the host sending the messages. The syslog-ng application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges.

destination d_sql { 
  sql(type(pgsql)
  host("logserver") username("syslog-ng") password("password")
  database("logs")
  table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}")
  columns("datetime", "host", "program", "pid", "message")
  values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY")
  indexes("datetime", "host", "program", "pid", "message"));
  };

The following example specifies the type of the database columns as well:

destination d_sql { 
  sql(type(pgsql)
  host("logserver") username("syslog-ng") password("password")
  database("logs")
  table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}")
  columns("datetime varchar(16)", "host varchar(32)", "program  varchar(20)", "pid varchar(8)", "message  varchar(200)")
  values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY")
  indexes("datetime", "host", "program", "pid", "message"));
};
[Example] Example 8.18. Using the sql() driver with an Oracle database

The following example sends the log messages into an Oracle database running on the logserver host, which must be set in the /etc/tnsnames.ora file. The messages are inserted into the LOGS database, the name of the table includes the exact date when the messages were sent. The syslog-ng application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges.

destination d_sql { 
  sql(type(oracle)
  username("syslog-ng") password("password")
  database("LOGS")
  table("msgs_${R_YEAR}${R_MONTH}${R_DAY}")
  columns("datetime varchar(16)", "host varchar(32)", "program varchar(32)", "pid varchar(8)", "message varchar2")                        
  values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY")
  indexes("datetime", "host", "program", "pid", "message"));
  };

The Oracle Instant Client retrieves the address of the database server from the /etc/tnsnames.ora file. Edit or create this file as needed for your configuration. A sample is provided below.

LOGS =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)
(HOST = logserver)
(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = EXAMPLE.SERVICE)
)
)
[Example] Example 8.19. Using the sql() driver with an MSSQL database

The following example sends the log messages into an MSSQL database running on the logserver host. The messages are inserted into the syslogng database, the name of the table includes the exact date when the messages were sent. The syslog-ng application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges.

destination d_mssql { 
sql(type(mssql) host("logserver") port("1433") 
  username("syslogng") password("syslogng") database("syslogng") 
  table("msgs_${R_YEAR}${R_MONTH}${R_DAY}")columns("datetime varchar(16)", "host varchar(32)",
  "program varchar(32)", "pid varchar(8)", "message varchar(4096)")
  values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY")
  indexes("datetime", "host", "program", "pid", "message"));
  };

The date format used by the MSSQL database must be explicitly set in the /etc/locales.conf file of the syslog-ng server. Edit or create this file as needed for your configuration. A sample is provided below.

[default]
date = "%Y-%m-%d %H:%M:%S"
[Example] Example 8.20. Using SQL NULL values

The null() parameter of the SQL driver can be used to replace the contents of a column with a special SQL NULL value. To replace every column that contains an empty string with NULL, use the null("") option, e.g.,

destination d_sql { 
                    sql(type(pgsql)
                    host("logserver") username("syslog-ng") password("password")
                    database("logs")
                    table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}")
                    columns("datetime", "host", "program", "pid", "message")
                    values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY")
                    indexes("datetime", "host", "program", "pid", "message")
                    null(""));                    
                    };

To replace only a specific column (e.g., pid) if it is empty, assign a default value to the column, and use this default value in the null() parameter:

destination d_sql { 
                    sql(type(pgsql)
                    host("logserver") username("syslog-ng") password("password")
                    database("logs")
                    table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}")
                    columns("datetime", "host", "program", "pid", "message")
                    values("$R_DATE", "$HOST", "$PROGRAM", "${PID:-@@NULL@@}", "$MSGONLY")
                    indexes("datetime", "host", "program", "pid", "message")
                    null("@@NULL@@"));                    
                    };

Ensure that the default value you use does not appear in the actual log messages, because other occurrences of this string will be replaced with NULL as well.

8.2.6. syslog()

The syslog() driver sends messages to a remote host (e.g., a syslog-ng server or relay) on the local intranet or internet using the new standard syslog protocol developed by IETF (see Section 2.18.2, “IETF-syslog messages” for details about the protocol). The protocol supports sending messages using the UDP, TCP, or the encrypted TLS networking protocols.

The required arguments of the driver are the address of the destination host (where messages should be sent) and the transport method (networking protocol).

The udp transport method automatically sends multicast packets if a multicast destination address is specified. The tcp and tls methods do not support multicasting.

Declaration:
    syslog(host transport [options]);

These destinations have the following options:

Name Type Default Description
flags() no_multi_line, syslog-protocol empty set

Flags influence the behavior of the driver.

The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line.

The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver.
flush_lines() number Use global setting. Specifies how many lines are flushed to a destination at a time. Syslog-ng waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the network, but also increases message latency. The latency can be limited by the use of the flush_timeout option.
flush_timeout() time in milliseconds Use global setting. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. See the flush_lines option for more information.
frac_digits() number 0 The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format.. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.
fsync() yes or no no Forces an fsync() call on the destination fd after each write. Note: enabling this option may seriously degrade performance.
ip_tos() number 0 Specifies the Type-of-Service value of outgoing packets.
ip_ttl() number 0 Specifies the Time-To-Live value of outgoing packets.
keep-alive() yes or no yes Specifies whether connections to destinations should be closed when syslog-ng is restarted (upon the receipt of a SIGHUP signal). Note that this applies to the client (destination) side of the syslog-ng connections, server-side (source) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the source. When the keep-alive option is enabled, syslog-ng saves the contents of the output queue of the destination when receiving a HUP signal, reducing the risk of losing messages.
localip() string 0.0.0.0 The IP address to bind to before connecting to target.
localport() number 0 The port number to bind to. Messages are sent from this port.
log_disk_fifo_size() number 0 Size of the hard disk space in bytes that is used as disk buffer. Available only in syslog-ng Premium Edition when using the tcp(), tcp6(), syslog() (when using the tcp or tls transport methods), and sql() destinations. Can be also defined as a global option. See Section 2.14, “Using disk-based buffering” for details on using the disk buffer.
log_fifo_size() number Use global setting. The number of entries in the output buffer (output fifo).
port() or destport() number 601 The port number to connect to. Note that the default port numbers used by syslog-ng do not comply with the latest RFC which was published after the release of syslog-ng 3.0.2, therefore the default port numbers will change in the future releases.
so_broadcast() yes or no no This option controls the SO_BROADCAST socket option required to make syslog-ng send messages to a broadcast address. See the socket(7) manual page for details.
so_keepalive() yes or no no Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. See the socket(7) manual page for details.
so_rcvbuf() number 0 Specifies the size of the socket receive buffer in bytes. See the socket(7) manual page for details.
so_sndbuf() number 0 Specifies the size of the socket send buffer in bytes. See the socket(7) manual page for details.
spoof_source() yes or no no Enables source address spoofing. This means that the host running syslog-ng generates UDP packets with the source IP address matching the original sender of the message. It is useful when you want to perform some kind of preprocessing via syslog-ng then forward messages to your central log management solution with the source address of the original sender. This option only works for UDP destinations though the original message can be received by TCP as well. This option is only available if syslog-ng was compiled using the --enable-spoof-source configuration option.
suppress() seconds 0 (disabled) If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.
template() string A format conforming to the default logfile format. Specifies a template defining the logformat to be used in the destination. Macros are described in Section 8.5, “Macros”. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
template_escape() yes or no no Turns on escaping ' and " in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
throttle() number 0 Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time_zone() timezone offset in seconds unspecified Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.
tls() tls options n/a This option sets various TLS specific options like key/certificate files and trusted CA locations. TLS can be used only with the tcp transport protocols. See Section 8.10, “TLS options” for more information.
transport udp, tcp, or tls tcp Specifies the protocol used to receive messages from the source.
ts_format() rfc3164, bsd, rfc3339, iso rfc3164 Override the global timestamp format (set in the global ts_format() parameter) for the specific destination. See also Section 7.7, “A note on timezones and timestamps”.

Table 8.13. Options for syslog() destinations


[Example] Example 8.21. Using the syslog() driver 
destination d_tcp { syslog(ip"10.1.2.3" transport("tcp") port(1999); localport(999)); };

If name resolution is configured, the hostname of the target server can be used as well.

destination d_tcp { syslog(ip"target_host" transport("tcp") port(1999); localport(999)); };

Send the log messages using TLS encryption and use mutual authentication. See Section 8.10, “TLS options” for details on the encryption and authentication options.

destination d_syslog_tls{ 
                syslog("10.100.20.40"
                transport("tls")
                port(6514)
                tls(peer-verify(required-trusted)
                ca_dir('/opt/syslog-ng/etc/syslog-ng/keys/ca.d/')
                key_file('/opt/syslog-ng/etc/syslog-ng/keys/client_key.pem')
                cert_file('/opt/syslog-ng/etc/syslog-ng/keys/client_certificate.pem'))
                );};

8.2.7. tcp(), tcp6(), udp(), and udp6(),

This driver sends messages to another host on the local intranet or internet using the UDP or TCP protocol. The tcp6() and udp6() drivers use the IPv6 network protocol.

Both drivers have a single required argument specifying the destination host address, where messages should be sent, and several optional parameters. Note that this differs from source drivers, where local bind address is implied, and none of the parameters are required.

The udp() and udp6() drivers automatically send multicast packets if a multicast destination address is specified. The tcp() and tcp6() drivers do not support multicasting.

Declaration:
    tcp(host [options]);
    udp(host [options]);
    tcp6(host [options]);
    udp6(host [options]);

These destinations have the following options:

Name Type Default Description
flags() no_multi_line, syslog-protocol empty set

Flags influence the behavior of the driver.

The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line.

The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver.
flush_lines() number Use global setting. Specifies how many lines are flushed to a destination at a time. Syslog-ng waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the network, but also increases message latency. The latency can be limited by the use of the flush_timeout option.
flush_timeout() time in milliseconds Use global setting. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. See the flush_lines option for more information.
frac_digits() number 0 The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format.. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.
fsync() yes or no no Forces an fsync() call on the destination fd after each write. Note: enabling this option may seriously degrade performance.
ip_tos() number 0 Specifies the Type-of-Service value of outgoing packets.
ip_ttl() number 0 Specifies the Time-To-Live value of outgoing packets.
keep-alive() yes or no yes Specifies whether connections to destinations should be closed when syslog-ng is restarted (upon the receipt of a SIGHUP signal). Note that this applies to the client (destination) side of the syslog-ng connections, server-side (source) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the source. When the keep-alive option is enabled, syslog-ng saves the contents of the output queue of the destination when receiving a HUP signal, reducing the risk of losing messages.
localip() string 0.0.0.0 The IP address to bind to before connecting to target.
localport() number 0 The port number to bind to. Messages are sent from this port.
log_disk_fifo_size() number 0 Size of the hard disk space in bytes that is used as disk buffer. Available only in syslog-ng Premium Edition when using the tcp(), tcp6(), syslog() (when using the tcp or tls transport methods), and sql() destinations. Can be also defined as a global option. See Section 2.14, “Using disk-based buffering” for details on using the disk buffer.
log_fifo_size() number Use global setting. The number of entries in the output buffer (output fifo).
port() or destport() number 514 The port number to connect to. Note that the default port numbers used by syslog-ng do not comply with the latest RFC which was published after the release of syslog-ng 3.0.2, therefore the default port numbers will change in the future releases.
so_broadcast() yes or no no This option controls the SO_BROADCAST socket option required to make syslog-ng send messages to a broadcast address. See the socket(7) manual page for details.
so_keepalive() yes or no no Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. See the socket(7) manual page for details.
so_rcvbuf() number 0 Specifies the size of the socket receive buffer in bytes. See the socket(7) manual page for details.
so_sndbuf() number 0 Specifies the size of the socket send buffer in bytes. See the socket(7) manual page for details.
spoof_source() yes or no no Enables source address spoofing. This means that the host running syslog-ng generates UDP packets with the source IP address matching the original sender of the message. It is useful when you want to perform some kind of preprocessing via syslog-ng then forward messages to your central log management solution with the source address of the original sender. This option only works for UDP destinations though the original message can be received by TCP as well. This option is only available if syslog-ng was compiled using the --enable-spoof-source configuration option.
suppress() seconds 0 (disabled) If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.
template() string A format conforming to the default logfile format. Specifies a template defining the logformat to be used in the destination. Macros are described in Section 8.5, “Macros”. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
template_escape() yes or no no Turns on escaping ' and " in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
throttle() number 0 Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time_zone() timezone offset in seconds unspecified Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.
tls() tls options n/a This option sets various TLS specific options like key/certificate files and trusted CA locations. TLS can be used only with the tcp transport protocols. See Section 8.10, “TLS options” for more information.
ts_format() rfc3164, bsd, rfc3339, iso rfc3164 Override the global timestamp format (set in the global ts_format() parameter) for the specific destination. See also Section 7.7, “A note on timezones and timestamps”.

Table 8.14. Options for tcp, tcp6, udp, and udp6 destinations


[Example] Example 8.22. Using the tcp() driver 
destination d_tcp { tcp("10.1.2.3" port(1999); localport(999)); };

If name resolution is configured, the hostname of the target server can be used as well.

destination d_tcp { tcp("target_host" port(1999); localport(999)); };

To send messages using the IETF-syslog message format, enable the syslog-protocol flag:

destination d_tcp { tcp("10.1.2.3" port(1999); flags(syslog-protocol) };
[Example] Example 8.23. Enabling disk-based buffering

The following example turns on disk-based buffering for the destination. The size of the disk buffer is 4 194 304 bytes (4 megabytes). In a worst-case situation, using the default value of the log_msg_size() parameter (8192 bytes), this disk buffer can store at least 512 messages. Typical log messages are about 300-500 bytes long, so a disk buffer of 4 megabytes can store over 8000 messages. Set the size of the disk buffer based on the average size and number of messages, and the longest estimated downtime of the server. 

destination d_tcp { 
                    tcp("10.1.2.3" port(1999) log_disk_fifo_size(4194304)); };

8.2.8. unix-stream() & unix-dgram()

These drivers send messages to a unix socket in either SOCK_STREAM or SOCK_DGRAM mode.

Both drivers have a single required argument specifying the name of the socket to connect to. 

Declaration: 
    unix-stream(filename [options]);
    unix-dgram(filename [options]);

The unix-stream() and unix-dgram() destinations have the following options:

Name Type Default Description
flags() no_multi_line, syslog-protocol empty set

Flags influence the behavior of the driver.

The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line.

The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver.
flush_lines() number Use global setting. Specifies how many lines are flushed to a destination at a time. Syslog-ng waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the network, but also increases message latency. The latency can be limited by the use of the flush_timeout option.
flush_timeout() time in milliseconds Use global setting. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. See the flush_lines option for more information.
frac_digits() number 0 The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format.. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.
fsync() yes or no no Forces an fsync() call on the destination fd after each write. Note: enabling this option may seriously degrade performance.
log_fifo_size() number Use global setting. The number of entries in the output buffer (output fifo).
keep-alive() yes or no yes Specifies whether connections to destinations should be closed when syslog-ng is restarted (upon the receipt of a SIGHUP signal). Note that this applies to the client (destination) side of the syslog-ng connections, server-side (source) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the source. When the keep-alive option is enabled, syslog-ng saves the contents of the output queue of the destination when receiving a HUP signal, reducing the risk of losing messages.
so_broadcast() yes or no no This option controls the SO_BROADCAST socket option required to make syslog-ng send messages to a broadcast address. See the socket(7) manual page for details.
so_keepalive() yes or no no Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. See the socket(7) manual page for details.
so_rcvbuf() number 0 Specifies the size of the socket receive buffer in bytes. See the socket(7) manual page for details.
so_sndbuf() number 0 Specifies the size of the socket send buffer in bytes. See the socket(7) manual page for details.
suppress() seconds 0 (disabled) If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.
template() string A format conforming to the default logfile format. Specifies a template defining the logformat to be used in the destination. Macros are described in Section 8.5, “Macros”. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
template_escape() yes or no no Turns on escaping ' and " in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
throttle() number 0 Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time_zone() timezone offset in seconds unspecified Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.
ts_format() rfc3164, bsd, rfc3339, iso rfc3164 Override the global timestamp format (set in the global ts_format() parameter) for the specific destination. See also Section 7.7, “A note on timezones and timestamps”.

Table 8.15. Options for unix-stream() and unix-dgram()


[Example] Example 8.24. Using the unix-stream() driver 
destination d_unix_stream { unix-stream("/var/run/logs"); };

8.2.9. usertty()

This driver writes messages to the terminal of a logged-in user.

The usertty() driver has a single required argument, specifying a username who should receive a copy of matching messages. 

Declaration: 
    usertty(username);

The usertty() does not have any further options nor does it support templates.

[Example] Example 8.25. Using the usertty() driver 
destination d_usertty { usertty("root"); };

© 2007-2008 BalaBit IT Security
Please send your comments or documentation bugs to: documentation@balabit.com

Prev  Up  Next
8.1. Source drivers  Home | Contents  8.3. Log path flags