Starting with version 3.10, syslog-ng can collect messages from multiple text files. You do not have to specify file names one by one, just use a wildcard to select which files to read. This is especially useful when you do not know the file names by the time syslog-ng is started. This is often the case with web servers with multiple virtual hosts. From this blog you can learn how to get started using the wildcard-file() source.

## Before you begin

Before configuring syslog-ng, you should have a web server – or any other software writing multiple log files – already up and running. In my example I use Apache HTTPD access logs, but you should be able to adopt the basic example to any software easily just by changing file names.

## Configuring syslog-ng

The following configuration reads any file ending with “log” in its name from the /var/log/apache2 directory and writes all messages in JSON format into a single file. You should append these configuration snippets to your syslog-ng.conf or in a separate .conf file under /etc/syslog-ng/conf.d/ if supported by your Linux distribution.

First, define a wildcard-file() source. There are two mandatory parameters:

• base-dir() configures the directory where syslog-ng looks for log files to read. In this case, it is the /var/log/apache2 directory.
• filename-pattern() accepts a simple glob pattern which defines files to search for. A “*” represents zero or more characters, while a “?” a single character. In this case, it is any file name that ends with “log”.

The no-parse flag is necessary in this example, because by default syslog-ng parses messages using the syslog parser, but Apache HTTPD uses its own format for logging. For a complete list of wildcard-file() options check the documentation at https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.10-guides/en/syslog-ng-ose-v3.10-guide-admin/html-single/index.html#configuring-sources-wildcard-file.

source s_apache2 {
wildcard-file(
base-dir("/var/log/apache2/")
filename-pattern("*log")
flags(no-parse)
);
};

Next, define a destination. Here, I’m using a JSON template, so that the different fields of the message are easy to distinguish. The “FILE_NAME” macro contains the file name with the full path name. The “MESSAGE” macro contains the whole message as it is read from the log files.

destination d_apache2 {
file(
"/var/log/web"
template("\$(format_json --key FILE_NAME --key MESSAGE)\n\n")
);
};

Finally, define a log statement that connects the source and destination together:

log { source(s_apache2); destination(d_apache2); };

If you have configured file names correctly, you should see similar entries in /var/log/web (or the destination you have configured):

{"MESSAGE":"172.16.146.1 - - [14/Jun/2017:17:17:16 +0200] \"GET / HTTP/1.0\" 200 4 \"-\" \"w3m/0.5.3+git20170102\"","FILE_NAME":"/var/log/apache2/test_log"}

In the MESSAGE field you see a message in the Apache combined log format. If you use VirtualHost names in your file names, you can use this information to identify which log message belongs to which website.

## What is next

Logging as a service (LaaS) providers often recommend their agents to be installed next to syslog(-ng) just to cover this situation. Installing additional software is not necessary any more to be able to forward messages from a directory of log files. Also, using LaaS providers from syslog-ng was never easier thanks to the syslog-ng configuration library (SCL), which hides away the complexity of setting up these destinations.

Try it yourself or check my blog next week where I will add a parser and a LaaS provider to the configuration.

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a long list of possibilities, check our contact page at https://syslog-ng.org/contact-us/. On Twitter, I am available as @PCzanik.