Do you want to simplify parsing your log messages? Try the new “application adapter” and “enterprise-wide message model” frameworks in syslog-ng: you can automatically parse log messages and forward the results to another syslog-ng instance. Optionally, you can also include the original, raw message that you can forward unmodified to a SIEM system for further analysis.
Many organizations store logs without parsing or analyzing the messages, because they are only required to collect the logs. Doing anything further with logs is an effort low on their priority list – except when a crash or breach occurs. The new “application adapter” and “enterprise-wide message model” frameworks of syslog-ng make it easier for you to get started with message parsing, and allow you to forward the results. They provide:
- A set of example adapters for sudo, iptables and other log messages that create name-value pairs from the important information in the messages.
- A new syslog-ng() destination which forwards every name-value pair together with the log message.
- A new default-network-drivers() source that is listening on every standard syslog ports, and automatically parses the incoming messages. The system() source was also changed to parse locally generated messages.
- A new flag called “store-raw-message” that allows you to forward the original message unmodified, just as syslog-ng has received it
With these changes, syslog-ng becomes capable of turning the incoming unstructured syslog messages into a set of name-value pairs, making structured log processing and searching possible, and also helping you to create dashboards easier (for example, in Kibana or Splunk).
While these features are work in progress and might change considerably in future releases, we appreciate early testing and feedback. With the help of your comments we could cover not just our internal use cases, but also take into account the needs of the wider community.
Before you begin
Note, that both of these features are still work in progress. Right now they are not available in the main source code, only in the https://github.com/balabit/syslog-ng/pull/1689 pull request, where you can read the comments and take a look at the code. For testing, you can also download RPM packages for openSUSE, SLES and RHEL (& compatibles) from my git snapshot repositories.
You should have at least two (virtual) machines available for testing: one acting as a client and an other one acting as a server.
Before showing you configurations for a test environment, let me introduce you to the major components of application adapters and the enterprise-wide message model in more depth.
Application adapters
Application adapters enable automatic parsing for a variety of log messages. Right now there are adapters for sudo, iptables, and some of the Cisco log formats. As a result of parsing, syslog-ng creates name-value pairs that you can use later on for filtering. You can also store them together with the message for easier querying. Syslog-ng adds a prefix to the names of parsed values: a dot and the application name. For example for “sudo” logs the prefix is “.sudo.”.
This is the information forwarded about a sudo event. As you can see, the JSON template replaces the leading dot with an undersore:
{
"_sudo": {
"USER": "root",
"TTY": "pts/0",
"SUBJECT": "czanik",
"PWD": "/home/czanik",
"COMMAND": "/bin/ls /root"
},
"SOURCE": "s_dnd",
"PROGRAM": "sudo",
"PRIORITY": "notice",
"PID": "8740",
"MESSAGE": " czanik : TTY=pts/0 ; PWD=/home/czanik ; USER=root ; COMMAND=/bin/ls /root",
"ISODATE": "2017-11-23T06:05:57-05:00",
"HOST_FROM": "localhost",
"HOST": "localhost.localdomain",
"FACILITY": "authpriv"
}
Before the regular syslog fields you can see values parsed from the message by syslog-ng under the “sudo” hierarchy.
After you have installed syslog-ng, you can find the application adapters bundled with syslog-ng under the “scl” directory, which you can normally find under /usr/share/syslog-ng/include/scl in your filesystem. Taking one of these as an example, you can create such an adapter yourself for the logs of other applications. For example here is the content of /usr/share/syslog-ng/include/scl/sudo/sudo.conf (minus the copyright from the beginning of the file).
block parser sudo-parser(prefix('.sudo.')) {
kv-parser(prefix(`prefix`) pair-separator(';') extract-stray-words-into('0'));
csv-parser(columns("`prefix`SUBJECT") template("$(list-head $0)") delimiters(' '));
};
application sudo[syslog] {
filter { program("sudo" type(string)); };
parser { sudo-parser(); };
};
Enterprise-wide message model
Until now, you could parse log messages or enrich them with external data on any component of your logging infrastructure that was running syslog-ng: a client, relay, or the server. However, forwarding the parsed values, or any other external information to the next component of the infrastructure was difficult and inconvenient, therefore parsing happened typically on the server. But now, the new transport mechanism of syslog-ng (dubbed enterprise-wide message model) allows you to deliver structured messages from the initial receiving syslog-ng component right up to the central log server, through any number of hops. It does not matter if you parse the messages on the client, on a relay, or on the central server, their structured results will be available where you store the messages. Optionally, you can also forward the original raw message as the first syslog-ng component in your infrastructure has received it, which is important if you want to forward a message for example to a SIEM system. To make use of the enterprise-wide message model, you have to use the syslog-ng() destination on the sender side, and the default-network-drivers() source on the receiver side.
The syslog-ng() destination
The syslog-ng() destination uses a special format to forward messages and the name-value pairs related to the message to other syslog-ng instances. The program name in the message is set to @syslog-ng, and the rest of the information is embedded in JSON format.
The default-network-drivers() source
The default-network-drivers() source is listening on all standard-syslog related ports, and automatically parses messages using the available application adapters. It supports TCP and UDP connections using the legacy and the new syslog protocol. Encrypted connections are also supported, but that needs additional configuration. It also automatically recognizes and parses messages sent using the syslog-ng() destination.
Note that you might need to update your SELinux and firewall rules to be able to use this feature.
The store-raw-message flag
One simple use-case for this mechanism is the transfer of the original RAW message as received from the client. With the new “store-raw-message” flag on inputs, syslog-ng saves the original message as received from the client in a name-value pair called ${RAWMSG}. You can forward this raw message to another syslog-ng component using the syslog-ng() destination + default-network-drivers() source pair, all the way to your central syslog-ng server, which can forward it to a SIEM system, in its original form, ensuring that the SIEM can process it.
Setting up a test environment
If you really want you can test the new features using a single instance of syslog-ng, but I recommend using two (virtual) machines for a more realistic test. My example configurations assume that you have two machines, a client and a server. They give you plenty of room for testing, so even if you do not have an Elasticsearch server ready, you can see how the features work.
For my tests I installed syslog-ng on CentOS 7 from the RPM packages in my experimental git snapshot repository. For the tests I simply disabled SELinux and firewalld. Of course while this is OK for a test environment, you should never do that in production. Check my blog if you need help running syslog-ng with SELinux and firewalld enabled.
Configuring the client
You can append the configuration snippet below to the end of your current syslog-ng configuration, or create a new .conf file under /etc/syslog-ng/conf.d/ if this possibility is enabled in your distribution.
source s_net {
tcp(ip("0.0.0.0") port("514") flags(store-raw-message));
};
destination d_kv {
file("/var/log/messages-kv.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3));
};
log { source(s_sys); destination(d_kv); };
destination d_ewmm {
syslog-ng(server("172.16.146.131"));
};
log { source(s_sys); source(s_net); destination(d_ewmm); };
In this configuration:
- The s_net source collects syslog messages over a TCP port and also stores the original message to RAWMSG thanks to flags(store-raw-message)
- The d_kv destination writes the parsed messages into a local file
- The first log statement connects local system logs to the above destination. The CentOS package uses s_sys for local system logs and as it utilizes the system() source messages are parsed automatically.
- The d_ewmm destination sends log messages to another syslog-ng instance specified in the server() parameter.
- The second log statement connects local logs and the above network source to another syslog-ng instance using the enterprise-wide message model.
Configuring the server
You can append the configuration snippet below to the end of your current syslog-ng configuration or create a new .conf file under /etc/syslog-ng/conf.d/ if this possibility is enabled in your distribution.
# network source opening default syslog ports
source s_dnd {
default-network-drivers();
};
# plain text file destination
destination d_fromnet {
file("/var/log/fromnet");
};
# saving from network source to plain text file
log {source(s_dnd); destination(d_fromnet); };
# text file destination with all value pairs welf formatted
destination d_kv {
file("/var/log/fromnet-kv.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3));
};
# saving from network source to text file together with value pairs
log { source(s_dnd); destination(d_kv); };
# Elasticsearch destination
destination d_elastic {
elasticsearch2 (
cluster("elasticsearch")
client_mode("http")
index("syslog-${YEAR}.${MONTH}.${DAY}")
time-zone(UTC)
type("syslog")
flush-limit(1)
server("172.16.146.131")
template("$(format-json --rekey .* --shift 1 --scope rfc5424 --scope dot-nv-pairs --scope nv-pairs --exclude DATE --key ISODATE)")
persist-name(elasticsearch-syslog)
)
};
# template test
destination d_templatetest {
file("/var/log/templatetest"
template("$(format-json --rekey .* --shift 1 --scope rfc5424 --scope dot-nv-pairs --scope nv-pairs --exclude DATE --key ISODATE)\n\n")
);
};
# network destination sending original raw message
destination d_siem {
tcp("172.16.146.130" port("514") template("${RAWMSG}\n"));
};
# saving from network source to Elasticsearch
log {
source(s_dnd);
destination(d_elastic);
destination(d_templatetest);
# destination(d_siem);
};
In this configuration:
- The s_dnd source utilizes the new default-network-drivers source. Encryption does not work, as it is not configured here.
- The d_fromnet destination is a regular plain text file destination followed by a log statement connecting it to the s_dnd source.
- The d_kv destination includes all the value pairs welf formatted followed by a log statement connecting it to the s_dnd source.
- The d_elasticsearch destination sends the message together with all related value pairs to an Elasticsearch database. Note, that the template includes “–rekey .* –shift 1” as field names starting with an underscore are problematic in Elasticsearch. This options removes the leading dot from field names.
- The d_templatetest destination uses the same template as the Elasticsearch destination, except the additional line feeds at the end. This can be used to fine tune your template or you can use it if you do not have Elasticsearch in your environment.
- The d_siem destination sends the original raw message to a TCP destination. Note the template: it uses the RAWMSG macro which exists only if the store-raw-message flag is enabled.
- Finally, there is a log statement which connects the s_dnd source both to Elasticsearch and the test destination.
Testing
Once everything is configured and syslog-ng is reloaded to take configurations into effect, you are ready for testing. We send two test messages. The first one is running sudo as a user and checking traces in the logs. The other one is sending a simple message to the network source on the client so we can check raw message forwarding.
Note that if you use journald for logging, as for example CentOS 7 does, you will see tons of additional name-value pairs coming from the journald driver of syslog-ng.
Testing with sudo
Use sudo for something on the client machine and check the logs. In my example I executed “sudo ls /root” as user czanik.
The /var/log/secure file on the client and the /var/log/fromnet file on the server should contain a similar line:
Nov 24 07:03:31 localhost.localdomain sudo[12766]: czanik : TTY=pts/0 ; PWD=/home/czanik ; USER=root ; COMMAND=/bin/ls /root
The /var/log/messages-kv.log file on the client and the /var/log/fromnet-kv.log on the server should contain a similar line. Note the field names starting with .sudo. and the many name-value pairs coming from journald.
2017-11-24T07:03:31.611-05:00 localhost.localdomain .journald.MESSAGE=" czanik : TTY=pts/0 ; PWD=/home/czanik ; USER=root ; COMMAND=/bin/ls /root" .journald.PRIORITY=5 .journald.SYSLOG_FACILITY=10 .journald.SYSLOG_IDENTIFIER=sudo .journald._AUDIT_LOGINUID=0 .journald._AUDIT_SESSION=2 .journald._BOOT_ID=7091940ba2d14a37ac1d652bb048a06f .journald._CAP_EFFECTIVE=1fffffffff .journald._CMDLINE="sudo ls /root" .journald._COMM=sudo .journald._EXE=/usr/bin/sudo .journald._GID=1000 .journald._HOSTNAME=localhost.localdomain .journald._MACHINE_ID=47cc66fe04ad4698a6bee3ef8d6ac86f .journald._PID=12766 .journald._SELINUX_CONTEXT=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 .journald._SOURCE_REALTIME_TIMESTAMP=1511525011608545 .journald._SYSTEMD_CGROUP=/user.slice/user-0.slice/session-2.scope .journald._SYSTEMD_OWNER_UID=0 .journald._SYSTEMD_SESSION=2 .journald._SYSTEMD_SLICE=user-0.slice .journald._SYSTEMD_UNIT=session-2.scope .journald._TRANSPORT=syslog .journald._UID=0 .sudo.COMMAND="/bin/ls /root" .sudo.PWD=/home/czanik .sudo.SUBJECT=czanik .sudo.TTY=pts/0 .sudo.USER=root HOST=localhost.localdomain HOST_FROM=localhost MESSAGE=" czanik : TTY=pts/0 ; PWD=/home/czanik ; USER=root ; COMMAND=/bin/ls /root" PID=12766 PROGRAM=sudo SOURCE=s_dnd
The /var/log/templatetest file should contain a similar line in JSON format:
{"sudo":{"USER":"root","TTY":"pts/0","SUBJECT":"czanik","PWD":"/home/czanik","COMMAND":"/bin/ls /root"},"journald":{"_UID":"0","_TRANSPORT":"syslog","_SYSTEMD_UNIT":"session-2.scope","_SYSTEMD_SLICE":"user-0.slice","_SYSTEMD_SESSION":"2","_SYSTEMD_OWNER_UID":"0","_SYSTEMD_CGROUP":"/user.slice/user-0.slice/session-2.scope","_SOURCE_REALTIME_TIMESTAMP":"1511525011608545","_SELINUX_CONTEXT":"unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023","_PID":"12766","_MACHINE_ID":"47cc66fe04ad4698a6bee3ef8d6ac86f","_HOSTNAME":"localhost.localdomain","_GID":"1000","_EXE":"/usr/bin/sudo","_COMM":"sudo","_CMDLINE":"sudo ls /root","_CAP_EFFECTIVE":"1fffffffff","_BOOT_ID":"7091940ba2d14a37ac1d652bb048a06f","_AUDIT_SESSION":"2","_AUDIT_LOGINUID":"0","SYSLOG_IDENTIFIER":"sudo","SYSLOG_FACILITY":"10","PRIORITY":"5","MESSAGE":" czanik : TTY=pts/0 ; PWD=/home/czanik ; USER=root ; COMMAND=/bin/ls /root"},"SOURCE":"s_dnd","PROGRAM":"sudo","PRIORITY":"notice","PID":"12766","MESSAGE":" czanik : TTY=pts/0 ; PWD=/home/czanik ; USER=root ; COMMAND=/bin/ls /root","ISODATE":"2017-11-24T07:03:31-05:00","HOST_FROM":"localhost","HOST":"localhost.localdomain","FACILITY":"authpriv"}
And finally you should see the same data also in Elasticsearch if you enabled it.
Testing raw message forwarding
To test raw message forwarding you should send a test message to the network source on the client. Use a similar command line:
logger -T -n 127.0.0.1 -P 514 -i This is a test
Using the above configurations the message is sent directly to the server using the enterprise-wide message model.
The /var/log/fromnet file should contain a similar line:
Nov 24 07:35:36 127.0.0.1 root[13629]: This is a test
The /var/log/templatetest file is more interesting, as it also contains the name-value pairs, including RAWMSG. As you can see it is completely untouched and includes all fields of a syslog message including the <> part at the beginning.
{"SOURCE":"s_dnd","RAWMSG":"<5>Nov 24 07:35:36 root[13629]: This is a test","PROGRAM":"root","PRIORITY":"notice","PID":"13629","MESSAGE":"This is a test","LEGACY_MSGHDR":"root[13629]: ","ISODATE":"2017-11-24T07:35:36-05:00","HOST_FROM":"127.0.0.1","HOST":"127.0.0.1","FACILITY":"kern"}