Chapter 7. Sending and storing log messages — destinations and destination drivers

7.1. amqp: Publishing messages using AMQP
7.1.1. amqp() destination options
7.2. elasticsearch: Sending messages directly to Elasticsearch version 1.x
7.2.1. Prerequisites
7.2.2. How syslog-ng OSE interacts with Elasticsearch
7.2.3. Client modes
7.2.4. Elasticsearch destination options
7.3. elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher
7.3.1. Prerequisites
7.3.2. How syslog-ng OSE interacts with Elasticsearch
7.3.3. Client modes
7.3.4. Elasticsearch X-Pack (Shield) and syslog-ng OSE
7.3.5. Search Guard and syslog-ng OSE
7.3.6. Elasticsearch2 destination options
7.4. file: Storing messages in plain-text files
7.4.1. file() destination options
7.5. graphite: Sending metrics to Graphite
7.5.1. graphite() destination options
7.6. hdfs: Storing messages on the Hadoop Distributed File System (HDFS)
7.6.1. Prerequisites
7.6.2. How syslog-ng OSE interacts with HDFS
7.6.3. Storing messages with MapR-FS
7.6.4. Kerberos authentication with syslog-ng hdfs() destination
7.6.5. HDFS destination options
7.7. Posting messages over HTTP
7.7.1. HTTP destination options
7.8. http: Posting messages over HTTP without Java
7.8.1. HTTP destination options
7.9. kafka: Publishing messages to Apache Kafka
7.9.1. Prerequisites
7.9.2. How syslog-ng OSE interacts with Apache Kafka
7.9.3. Kafka destination options
7.10. loggly: Using Loggly
7.10.1. loggly() destination options
7.11. logmatic: Using Logmatic.io
7.11.1. logmatic() destination options
7.12. mongodb: Storing messages in a MongoDB database
7.12.1. How syslog-ng OSE connects the MongoDB server
7.12.2. mongodb() destination options
7.13. network: Sending messages to a remote log server using the RFC3164 protocol (network() driver)
7.13.1. network() destination options
7.14. osquery: Sending log messages to osquery's syslog table
7.14.1. osquery() destination options
7.15. pipe: Sending messages to named pipes
7.15.1. pipe() destination options
7.16. program: Sending messages to external applications
7.16.1. program() destination options
7.17. pseudofile()
7.17.1. pseudofile() destination options
7.18. redis: Storing name-value pairs in Redis
7.18.1. redis() destination options
7.19. riemann: Monitoring your data with Riemann
7.19.1. riemann() destination options
7.20. smtp: Generating SMTP messages (e-mail) from logs
7.20.1. smtp() destination options
7.21. Splunk: Sending log messages to Splunk
7.22. sql: Storing messages in an SQL database
7.22.1. Using the sql() driver with an Oracle database
7.22.2. Using the sql() driver with a Microsoft SQL database
7.22.3. The way syslog-ng interacts with the database
7.22.4. sql() destination options
7.23. stomp: Publishing messages using STOMP
7.23.1. stomp() destination options
7.24. syslog: Sending messages to a remote logserver using the IETF-syslog protocol
7.24.1. syslog() destination options
7.25. tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers)
7.25.1. tcp(), tcp6(), udp(), and udp6() destination options
7.26. unix-stream, unix-dgram: Sending messages to UNIX domain sockets
7.26.1. unix-stream() and unix-dgram() destination options
7.27. usertty: Sending messages to a user terminal — usertty() destination
7.28. Write your own custom destination in Java or Python

A destination is where a log message is sent if the filtering rules match. Similarly to sources, destinations consist of one or more drivers, each defining where and how messages are sent.

Tip

If no drivers are defined for a destination, all messages sent to the destination are discarded. This is equivalent to omitting the destination from the log statement.

To define a destination, add a destination statement to the syslog-ng configuration file using the following syntax:

destination <identifier> {
            destination-driver(params); destination-driver(params); ... };
Example 7.1. A simple destination statement

The following destination statement sends messages to the TCP port 1999 of the 10.1.2.3 host.

destination d_demo_tcp { network("10.1.2.3" port(1999)); };

If name resolution is configured, you can use the hostname of the target server as well.

destination d_tcp { network("target_host" port(1999)); };
Warning
  • Do not define the same drivers with the same parameters more than once, because it will cause problems. For example, do not open the same file in multiple destinations.

  • Do not use the same destination in different log paths, because it can cause problems with most destination types. Instead, use filters and log paths to avoid such situations.

  • Sources and destinations are initialized only when they are used in a log statement. For example, syslog-ng OSE starts listening on a port or starts polling a file only if the source is used in a log statement. For details on creating log statements, see Chapter 8, Routing messages: log paths, flags, and filters.

The following table lists the destination drivers available in syslog-ng OSE. If these destinations do not satisfy your needs, you can extend syslog-ng OSE and write your own destination, for example, in C, Java, or Python. For details, see Section 7.28, Write your own custom destination in Java or Python.

NameDescription
amqp() Publishes messages using the AMQP (Advanced Message Queuing Protocol).
elasticsearch and elasticsearch2 Sends messages to an Elasticsearch server. The elasticsearch2 driver supports Elasticsearch version 2 and newer.
file() Writes messages to the specified file.
graphite() Sends metrics to a Graphite server to store numeric time-series data.
hdfs() Sends messages into a file on a Hadoop Distributed File System (HDFS) node.
http()Sends messages over the HTTP protocol. There are two different implementations of this driver: a Java-based http driver, and an http driver without Java.
kafka() Publishes log messages to the Apache Kafka message bus, where subscribers can access them.
loggly() Sends log messages to the Loggly Logging-as-a-Service provider.
logmatic() Sends log messages to the Logmatic.io Logging-as-a-Service provider.
mongodb() Sends messages to a MongoDB database.
network() Sends messages to a remote host using the BSD-syslog protocol over IPv4 and IPv6. Supports the TCP, UDP, and TLS network protocols.
pipe() Writes messages to the specified named pipe.
program() Forks and launches the specified program, and sends messages to its standard input.
redis() Sends messages as name-value pairs to a Redis key-value store.
riemann() Sends metrics or events to a Riemann monitoring system.
smtp() Sends e-mail messages to the specified recipients.
sql() Sends messages into an SQL database. In addition to the standard syslog-ng packages, the sql() destination requires database-specific packages to be installed. Refer to the section appropriate for your platform in Chapter 3, Installing syslog-ng.
stomp() Sends messages to a STOMP server.
syslog() Sends messages to the specified remote host using the IETF-syslog protocol. The IETF standard supports message transport using the UDP, TCP, and TLS networking protocols.
unix-dgram() Sends messages to the specified unix socket in SOCK_DGRAM style (BSD).
unix-stream() Sends messages to the specified unix socket in SOCK_STREAM style (Linux).
usertty() Sends messages to the terminal of the specified user, if the user is logged in.

Table 7.1. Destination drivers available in syslog-ng