The syslog-ng 3.0 Administrator Guide

Document version: Thirteenth
Release date: September 23, 2010
Latest version available at the BalaBit Documentation Page

8.2.2. logstore()

The logstore driver stores log messages in binary files that can be encrypted, compressed, checked for integrity, and timestamped by an external Timestamping Authority (TSA). Otherwise, it is very similar to the file() destination.

To display the contents of a logstore file, use the logcat command supplied with syslog-ng, e.g., logcat /var/log/messages.lgs.

The destination filename may include macros which get expanded when the message is written, thus a simple logstore() driver may create several files. For more information on available macros see Section 8.5, “Macros”.

[Warning] Warning

When creating several thousands separate log files, syslog-ng might not be able to open the required number of files. This might happen for example when using the $HOST macro in the filename while receiving messages from a large number of hosts. To overcome this problem, adjust the --fd-limit comman-line parameter of syslog-ng or the global ulimit parameter of your host. For setting the --fd-limit comman-line parameter of syslog-ng see the syslog-ng(8) manual page. For setting the ulimit parameter of the host, see the documentation of your operating system.

The logstore() has a single required parameter that specifies the filename that stores the log messages.

Declaration:
    logstore(filename options());

The logstore() destination has the following options:

Name Type Default Description
chunk_size() number 128 Size of a logstore chunk in kilobytes. Note that this size refers to the compressed size of the chunk. Also, the gzip library used for compressing the messages has a 32k long buffer; messages may not appear in the actual logfile until this buffer is not filled. Logstore chunks are closed when they reach the specified size, or when the time limit set in chunk_time expires.
chunk_time() number 5 Time limit in seconds: syslog-ng PE closes the chunk if no new messages arrive until the time limit expires. Logstore chunks are closed when the time limit expires, or when they reach the size specified in the chunk_size parameter. If the time limit set in the time_reap parameter expires, the entire file is closed.
compress() number between 0-9 3 Compression level. 0 means uncompressed files, while 1-9 is the compression level used by gzip (9 means the highest but slowest compression, 3 is usually a good compromise).
create_dirs() yes or no no Enable creating non-existing directories.
dir_group() string root The group of the directories created by syslog-ng.
dir_owner() string root The owner of directories created by syslog-ng.
dir_perm() number 0600 The permission mask of directories created by syslog-ng. Log directories are only created if a file after macro expansion refers to a non-existing directory, and directory creation is enabled (see the create_dirs() option below). For octal numbers prefix the number with 0, e.g., use 0755 for rwxr-xr-x.
encrypt_certificate() filename none Name of a file, that contains an X.509 certificate (and the public key) in PEM format. The syslog-ng application uses this certificate to encrypt the logstore files which can be decrypted using the private key of the certificate.
flags() no_multi_line, syslog-protocol empty set

Flags influence the behavior of the driver.

The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line.

The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver.
flush_lines() number Use global setting. Specifies how many lines are flushed to a destination at a time. Syslog-ng waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the network, but also increases message latency. The latency can be limited by the use of the flush_timeout option.
flush_timeout() time in milliseconds Use global setting. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. See the flush_lines option for more information.
frac_digits() number 0 The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format.. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.
fsync() yes or no no Forces an fsync() call on the destination fd after each write. Note: enabling this option may seriously degrade performance.
group() string root Set the group of the created file to the one specified.
owner() string root Set the owner of the created file to the one specified.
perm() number 0600 The permission mask of the file if it is created by syslog-ng. For octal numbers prefix the number with 0, e.g., use 0755 for rwxr-xr-x.
suppress() seconds 0 (disabled) If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.
template() string A format conforming to the default logfile format. Specifies a template defining the logformat to be used in the destination. Macros are described in Section 8.5, “Macros”. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.
template_escape() yes or no no Turns on escaping ' and " in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.
throttle() number 0 Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.
time_reap() number 60 The time to wait in seconds before an idle destination file is closed.
timestamp-freq() number in seconds Use global setting. The minimum time that should expire between two timestamping requests. When syslog-ng closes a chunk, it checks how much time has expired since the last timestamping request: if it is higher than the value set in the timestamp-freq parameter, it requests a new timestamp from the authority set in the timestamp-url parameter.
timestamp-url() string Use global setting. The URL of the Timestamping Authority used to request timestamps to sign logstore chunks. Note that syslog-ng currently supports only Timestamping Authorities that conform to RFC3161 Internet X.509 Public Key Infrastructure Time-Stamp Protocol, other protocols like Microsoft Authenticode Timestamping are not supported.
time_zone() timezone in +/-HH:MM format unspecified Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.
ts_format() rfc3164, bsd, rfc3339, iso rfc3164 Override the global timestamp format (set in the global ts_format() parameter) for the specific destination. See also Section 7.7, “A note on timezones and timestamps”.

Table 8.9. Options for logstore()


[Example] Example 8.14. Using the logstore() driver

A simple example saving and compressing log messages.

destination d_logstore { logstore("/var/log/messages.lgs" compress(5) ); };

A more detailed example that encrypts messages, modifies the parameters for closing chunks, and sets file privileges.

destination d_logstore { logstore("/var/log/messages-logstore.lgs"
    encrypt_certificate("/opt/syslog-ng/etc/syslog-ng/keys/10-100-20-40/public-certificate-of-the-server.pem")			
    chunk_size(100) 
    chunk_time(5)
    owner("balabit")
    group("balabit")
    perm(0777)
); };
Prev  Up  Next
8.2.1. file()  Home | Contents  8.2.3. pipe()

© 2007-2010 BalaBit IT Security
Please send your comments or documentation bugs to: documentation@balabit.com