The syslog-ng 3.0 Administrator Guide

Document version: Thirteenth
Release date: September 23, 2010
Latest version available at the BalaBit Documentation Page

8.1.2. file()

Collects log messages from plain-text files. The file driver has a single required parameter specifying the file to open.

Declaration:
    file(filename);

In syslog-ng PE, the filename (but not the pathname) may include wildcard characters (e.g., *). Note that when using wildcards in filenames, always set how often syslog-ng should check the file for new messages using the follow_freq() parameter.

When using wildcards, syslog-ng PE monitors every matching file, and can receive new log messages from any of the files. However, monitoring (polling) many files (i.e., more than ten) has a significant overhead and may affect performance. On Linux this overhead is not so significant, because syslog-ng PE uses the inotify feature of the kernel.

[Note] Note

If the message does not have a proper syslog header, syslog-ng treats messages received from files as sent by the kern facility. Use the default-facility and default-priority options in the source definition to assign a different facility if needed.

The file() driver has the following options:

Name Type Default Description
default-facility() facility string kern This parameter assigns a facility value to the messages received from the file source, if the message does not specify one.
default-priority() priority string   This parameter assigns an emergency level to the messages received from the file source, if the message does not specify one.
file filename with path   The file to read messages from. Note that only syslog-ng PE supports wildcards in the filename (but not in the pathname). To monitor the subdirectories as well, use the recursive option.
encoding() string   Specifies the characterset (encoding, e.g., UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command.
flags() empty-lines, kernel, no-multi-line, no-parse, store-legacy-msghdr, syslog-protocol, validate-utf8 empty set

Specifies the log parsing options of the source.

Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ng removes empty lines automatically.

The kernel flag makes the source default to the LOG_KERN | LOG_CRIT priority if not specified otherwise.

The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line.

By default, syslog-ng parses incoming messages as syslog messages. If a source does not send properly formatted messages, use the no-parse flag to disable message parsing for the source. As a result, syslog-ng will generate a new syslog header and put the entire incoming message into the MSG part of the syslog message.

The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. Other information (timestamp, host, etc.) is added automatically. This flag is useful for parsing files not complying to the syslog format.

If the store-legacy-msghdr flag is enabled, syslog-ng stores the original incoming header of the log message. This is useful of the original format of a non-syslog-compliant message must be retained (syslog-ng automatically corrects minor header errors, e.g., adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). Note that store-legacy-msghdr should be enabled when receiving messages from syslog-ng Agent for Windows clients that use the Snare-compatible mode.

The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard. Note that this flag is not needed for the syslog driver.

The validate-utf8 flag enables encoding-verification for messages formatted according to the new IETF syslog standard (see Section 2.19.2, “IETF-syslog messages” for details). If the BOM character is missing, but the message is otherwise UTF-8 compliant, syslog-ng automatically adds the BOM character to the message.
follow_freq() number 1 Indicates that the source should be checked periodically instead of being polled. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, syslog-ng will not attempt to use poll() on the file, but checks whether the file changed every time the follow_freq() interval (in seconds) has elapsed. Floating-point numbers (e.g., 1.5) can be used as well.
keep_timestamp() yes or no yes Specifies whether syslog-ng should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
log_fetch_limit() number The value specified by the global log_fetch_limit() option, which defaults to 10. The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log_fetch_limit() is too high.
log_iw_size() number 100 The size of the initial window, this value is used during flow control.
log_msg_size() number Use the global log_msg_size() option, which defaults to 8192. Specifies the maximum length of incoming log messages. Uses the value of the global option if not specified.
log_prefix() (DEPRECATED) string   A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux. NOTE: This option is deprecated. Use program_override() instead.
optional() yes or no   Instruct syslog-ng to ignore the error if a specific source cannot be initialized. No other attempts to initialize the source will be made until the configuration is reloaded. This option currently applies to the pipe(), unix-dgram, and unix-stream drivers.
pad_size() number 0 Specifies input padding. Some operating systems (such as HP-UX) pad all 0 messages to block boundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). Syslog-ng will pad reads from the associated device to the number of bytes set in pad_size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes.
program_override string   Replaces the $PROGRAM part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program_override("kernel") option in the source containing /proc/kmsg. NOTE: This option replaces the deprecated log_prefix() option.
recursive yes or no no When enabled, syslog-ng PE monitors every subdirectory of the directory set in the path of the file parameter, and reads log messages from files with the set filename. The recursive option can be used together with wildcards in the filename.
time_zone() timezone in +/-HH:MM format   The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

Table 8.1. Options of the file() sources


[Example] Example 8.2. Using the file() driver 
source s_file { file("/var/log/messages"); };
[Example] Example 8.3. Tailing files

The following source checks the access.log file every second for new messages.

source s_tail { file("/var/log/apache/access.log" 
                    follow_freq(1) flags(no-parse)); };
[Example] Example 8.4. Using wildcards in the filename

The following example monitors every file with the .log extension in the /var/application directory for log messages. Note that only syslog-ng PE supports wildcards in the file and pathnames.

source s_file { file("/var/application/*.log" follow_freq(1));};
[Example] Example 8.5. Monitoring multiple directories

The following example reads files having the .log extension from the /var/application/ directory and its subdirectories. Note that only syslog-ng PE supports recursive directory handling and wildcards in the file and pathnames.

source s_file_subdirectories { file("/var/application/*.log"
                    recursive(yes) 
                    follow_freq(1) 
                    log_fetch_limit(100)
                    );};
Prev  Up  Next
8.1.1. internal()  Home | Contents  8.1.3. pipe()

© 2007-2010 BalaBit IT Security
Please send your comments or documentation bugs to: documentation@balabit.com