The syslog-ng Open Source Edition 3.13 Administrator Guide

This guide is published under the Creative Commons Attribution-Noncommercial-No Derivative Works (by-nc-nd) 3.0 license. See Appendix C, Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License for details. The latest version is always available at https://www.balabit.com/support/documentation.

Some rights reserved.

This documentation and the product it describes are considered protected by copyright according to the applicable laws.

AIX™, AIX 5L™, AS/400™, BladeCenter™, eServer™, IBM™, the IBM™ logo, IBM System i™, IBM System i5™, IBM System x™, iSeries™, i5/OS™, Netfinity™, NetServer™, OpenPower™, OS/400™, PartnerWorld™, POWER™, ServerGuide™, ServerProven™, and xSeries™ are trademarks or registered trademarks of International Business Machines.

Alliance Log Agent for System i™ is a registered trademark of Patrick Townsend & Associates, Inc.

The Balabit™ name and the Balabit™ logo are registered trademarks of Balabit SA.

Debian™ is a registered trademark of Software in the Public Interest Inc.

Hadoop™ and the Hadoop elephant logo are trademarks of the Apache Software Foundation.

Linux™ is a registered trademark of Linus Torvalds.

MapR™, is a trademark of MapR Technologies, Inc.

Elasticsearch™ and Kibana™ is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.

Apache Kafka and the Apache Kafka Logo are trademarks of the Apache Software Foundation.

MySQL™ is a registered trademark of Oracle and/or its affiliates.

Oracle™, JD Edwards™, PeopleSoft™, and Siebel™ are registered trademarks of Oracle Corporation and/or its affiliates.

Red Hat™, Inc., Red HatEnterprise Linux™ and Red HatLinux™ are trademarks of Red Hat, Inc.

SUSE™ is a trademark of SUSE AG, a Novell business.

Solaris™ is a registered trademark of Oracle and/or its affiliates.

Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries.

The syslog-ng™ name and the syslog-ng™ logo are registered trademarks of Balabit.

Windows™ 95, 98, ME, 2000, XP, Server 2003, Vista, Server 2008, 7, 8, and Server 2012 are registered trademarks of Microsoft Corporation.

All other product names mentioned herein are the trademarks of their respective owners.

DISCLAIMER. Balabit is not responsible for any third-party websites mentioned in this document. Balabit does not endorse and is not responsible or liable for any content, advertising, products, or other material on or available from such sites or resources. Balabit will not be responsible or liable for any damage or loss caused or alleged to be caused by or in connection with use of or reliance on any such content, goods, or services that are available on or through any such sites or resources.

This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (https://www.openssl.org/). This product includes cryptographic software written by Eric Young (eay@cryptsoft.com)

This product includes open source software components. For details on the licenses and availability of these software components, see Appendix B, Open source licenses.

January 09, 2018


Table of Contents

Preface
1. Summary of contents
2. Target audience and prerequisites
3. Products covered in this guide
4. Typographical conventions
5. Contact and support information
5.1. Sales contact
5.2. Support contact
5.3. Training
6. About this document
6.1. Summary of changes
6.2. Feedback
6.3. Acknowledgments
1. Introduction to syslog-ng
1.1. What syslog-ng is
1.2. What syslog-ng is not
1.3. Why is syslog-ng needed?
1.4. What is new in syslog-ng Open Source Edition 3.13?
1.5. Who uses syslog-ng?
1.6. Supported platforms
2. The concepts of syslog-ng
2.1. The philosophy of syslog-ng
2.2. Logging with syslog-ng
2.2.1. The route of a log message in syslog-ng
2.3. Modes of operation
2.3.1. Client mode
2.3.2. Relay mode
2.3.3. Server mode
2.4. Global objects
2.5. Timezones and daylight saving
2.5.1. How syslog-ng OSE assigns timezone to the message
2.5.2. A note on timezones and timestamps
2.6. The license of syslog-ng OSE
2.7. High availability support
2.8. The structure of a log message
2.8.1. BSD-syslog or legacy-syslog messages
2.8.2. IETF-syslog messages
2.9. Message representation in syslog-ng OSE
2.10. Structuring macros, metadata, and other value-pairs
2.10.1. Specifying data types in value-pairs
2.11. Things to consider when forwarding messages between syslog-ng OSE hosts
2.12. Commercial version of syslog-ng
3. Installing syslog-ng
3.1. Compiling syslog-ng from source
3.2. Compiling options of syslog-ng OSE
3.3. Uninstalling syslog-ng OSE
3.4. Configuring Microsoft SQL Server to accept logs from syslog-ng
4. The syslog-ng OSE quick-start guide
4.1. Configuring syslog-ng on client hosts
4.2. Configuring syslog-ng on server hosts
4.3. Configuring syslog-ng relays
4.3.1. Configuring syslog-ng on relay hosts
4.3.2. How relaying log messages works
5. The syslog-ng OSE configuration file
5.1. Notes about the configuration syntax
5.2. Defining configuration objects inline
5.3. Using channels in configuration objects
5.4. Global and environmental variables
5.5. Modules in syslog-ng OSE
5.5.1. Loading modules
5.6. Managing complex syslog-ng configurations
5.6.1. Including configuration files
5.6.2. Reusing configuration blocks
5.6.3. Generating configuration blocks from a script
6. Collecting log messages — sources and source drivers
6.1. How sources work
6.2. internal: Collecting internal messages
6.2.1. internal() source options
6.3. file: Collecting messages from text files
6.3.1. Notes on reading kernel messages
6.3.2. file() source options
6.4. wildcard-file: Collecting messages from multiple text files
6.4.1. wildcard-file() source options
6.5. network: Collecting messages using the RFC3164 protocol (network() driver)
6.5.1. network() source options
6.6. nodejs: Receiving JSON messages from nodejs applications
6.6.1. nodejs() source options
6.7. mbox: Converting local e-mail messages to log messages
6.8. osquery: Collect and parse osquery result logs
6.8.1. osquery() source options
6.9. pipe: Collecting messages from named pipes
6.9.1. pipe() source options
6.10. pacct: Collecting process accounting logs on Linux
6.10.1. pacct() options
6.11. program: Receiving messages from external applications
6.11.1. program() source options
6.12. snmptrap: Read Net-SNMP traps
6.12.1. snmptrap() source options
6.13. sun-streams: Collecting messages on Sun Solaris
6.13.1. sun-streams() source options
6.14. syslog: Collecting messages using the IETF syslog protocol (syslog() driver)
6.14.1. syslog() source options
6.15. system: Collecting the system-specific log messages of a platform
6.16. systemd-journal: Collecting messages from the systemd-journal system log storage
6.16.1. systemd-journal() source options
6.17. systemd-syslog: Collecting systemd messages using a socket
6.18. tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol
6.18.1. tcp(), tcp6(), udp() and udp6() source options — OBSOLETE
6.19. unix-stream, unix-dgram: Collecting messages from UNIX domain sockets
6.19.1. UNIX credentials and other metadata
6.19.2. unix-stream() and unix-dgram() source options
7. Sending and storing log messages — destinations and destination drivers
7.1. amqp: Publishing messages using AMQP
7.1.1. amqp() destination options
7.2. elasticsearch: Sending messages directly to Elasticsearch version 1.x
7.2.1. Prerequisites
7.2.2. How syslog-ng OSE interacts with Elasticsearch
7.2.3. Client modes
7.2.4. Elasticsearch destination options
7.3. elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher
7.3.1. Prerequisites
7.3.2. How syslog-ng OSE interacts with Elasticsearch
7.3.3. Client modes
7.3.4. Elasticsearch X-Pack (Shield) and syslog-ng OSE
7.3.5. Search Guard and syslog-ng OSE
7.3.6. Elasticsearch2 destination options
7.4. file: Storing messages in plain-text files
7.4.1. file() destination options
7.5. graphite: Sending metrics to Graphite
7.5.1. graphite() destination options
7.6. hdfs: Storing messages on the Hadoop Distributed File System (HDFS)
7.6.1. Prerequisites
7.6.2. How syslog-ng OSE interacts with HDFS
7.6.3. Storing messages with MapR-FS
7.6.4. Kerberos authentication with syslog-ng hdfs() destination
7.6.5. HDFS destination options
7.7. Posting messages over HTTP
7.7.1. HTTP destination options
7.8. http: Posting messages over HTTP without Java
7.8.1. HTTP destination options
7.9. kafka: Publishing messages to Apache Kafka
7.9.1. Prerequisites
7.9.2. How syslog-ng OSE interacts with Apache Kafka
7.9.3. Kafka destination options
7.10. loggly: Using Loggly
7.10.1. loggly() destination options
7.11. logmatic: Using Logmatic.io
7.11.1. logmatic() destination options
7.12. mongodb: Storing messages in a MongoDB database
7.12.1. How syslog-ng OSE connects the MongoDB server
7.12.2. mongodb() destination options
7.13. network: Sending messages to a remote log server using the RFC3164 protocol (network() driver)
7.13.1. network() destination options
7.14. osquery: Sending log messages to osquery's syslog table
7.14.1. osquery() destination options
7.15. pipe: Sending messages to named pipes
7.15.1. pipe() destination options
7.16. program: Sending messages to external applications
7.16.1. program() destination options
7.17. pseudofile()
7.17.1. pseudofile() destination options
7.18. redis: Storing name-value pairs in Redis
7.18.1. redis() destination options
7.19. riemann: Monitoring your data with Riemann
7.19.1. riemann() destination options
7.20. smtp: Generating SMTP messages (e-mail) from logs
7.20.1. smtp() destination options
7.21. Splunk: Sending log messages to Splunk
7.22. sql: Storing messages in an SQL database
7.22.1. Using the sql() driver with an Oracle database
7.22.2. Using the sql() driver with a Microsoft SQL database
7.22.3. The way syslog-ng interacts with the database
7.22.4. sql() destination options
7.23. stomp: Publishing messages using STOMP
7.23.1. stomp() destination options
7.24. syslog: Sending messages to a remote logserver using the IETF-syslog protocol
7.24.1. syslog() destination options
7.25. tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers)
7.25.1. tcp(), tcp6(), udp(), and udp6() destination options
7.26. unix-stream, unix-dgram: Sending messages to UNIX domain sockets
7.26.1. unix-stream() and unix-dgram() destination options
7.27. usertty: Sending messages to a user terminal — usertty() destination
7.28. Write your own custom destination in Java or Python
8. Routing messages: log paths, flags, and filters
8.1. Log paths
8.1.1. Embedded log statements
8.1.2. Junctions and channels
8.1.3. Log path flags
8.2. Managing incoming and outgoing messages with flow-control
8.2.1. Flow-control and multiple destinations
8.2.2. Configuring flow-control
8.3. Using disk-based and memory buffering
8.3.1. Enabling reliable disk-based buffering
8.3.2. Enabling normal disk-based buffering
8.3.3. Enabling memory buffering
8.3.4. About disk queue files
8.4. Filters
8.4.1. Using filters
8.4.2. Combining filters with boolean operators
8.4.3. Comparing macro values in filters
8.4.4. Using wildcards, special characters, and regular expressions in filters
8.4.5. Tagging messages
8.4.6. Filter functions
8.5. Dropping messages
9. Global options of syslog-ng OSE
9.1. Configuring global syslog-ng options
9.2. Global options
10. TLS-encrypted message transfer
10.1. Secure logging using TLS
10.2. Encrypting log messages with TLS
10.2.1. Configuring TLS on the syslog-ng clients
10.2.2. Configuring TLS on the syslog-ng server
10.3. Mutual authentication using TLS
10.3.1. Configuring TLS on the syslog-ng clients
10.3.2. Configuring TLS on the syslog-ng server
10.4. TLS options
11. Manipulating messages
11.1. Customizing message format using macros and templates
11.1.1. Formatting messages, filenames, directories, and tablenames
11.1.2. Templates and macros
11.1.3. Date-related macros
11.1.4. Hard vs. soft macros
11.1.5. Macros of syslog-ng OSE
11.1.6. Using template functions
11.1.7. Template functions of syslog-ng OSE
11.1.8. Modifying the on-the-wire message format
11.2. Modifying messages using rewrite rules
11.2.1. Replacing message parts
11.2.2. Setting message fields to specific values
11.2.3. Unsetting message fields
11.2.4. Creating custom SDATA fields
11.2.5. Setting multiple message fields to specific values
11.2.6. map-value-pairs: Rename value-pairs to normalize logs
11.2.7. Conditional rewrites
11.2.8. Adding and deleting tags
11.2.9. Anonymizing credit card numbers
11.3. Regular expressions
11.3.1. Types and options of regular expressions
11.3.2. Optimizing regular expressions
12. Parsers and segmenting structured messages
12.1. Parsing syslog messages
12.1.1. Options of syslog-parser parsers
12.2. Parsing messages with comma-separated and similar values
12.2.1. Options of CSV parsers
12.3. Parsing key=value pairs
12.3.1. Options of key=value parsers
12.4. The JSON parser
12.4.1. Options of JSON parsers
12.5. The XML parser
12.5.1. Options of XML parsers
12.6. Parsing dates and timestamps
12.6.1. Options of date-parser() parsers
12.7. The Apache Access Log Parser
12.7.1. Options of apache-accesslog-parser() parsers
12.8. The Cisco Parser
12.9. The Linux Audit Parser
12.9.1. Options of linux-audit-parser() parsers
12.10. The Python Parser
13. Processing message content with a pattern database
13.1. Classifying log messages
13.1.1. The structure of the pattern database
13.1.2. How pattern matching works
13.1.3. Artificial ignorance
13.2. Using pattern databases
13.2.1. Using parser results in filters and templates
13.2.2. Downloading sample pattern databases
13.3. Correlating log messages using pattern databases
13.3.1. Referencing earlier messages of the context
13.4. Triggering actions for identified messages
13.4.1. Conditional actions
13.4.2. External actions
13.4.3. Actions and message correlation
13.5. Creating pattern databases
13.5.1. Using pattern parsers
13.5.2. What's new in the syslog-ng pattern database format V5
13.5.3. The syslog-ng pattern database format
14. Correlating log messages
14.1. Correlating messages using the grouping-by() parser
14.1.1. Referencing earlier messages of the context
14.1.2. Options of grouping-by parsers
15. Enriching log messages with external data
15.1. Adding metadata from an external file
15.1.1. Options add-contextual-data()
15.2. Looking up GeoIP data from IP addresses (DEPRECATED)
15.2.1. Options of geoip parsers
15.3. Looking up GeoIP2 data from IP addresses
15.3.1. Referring to parts of the message as a macro
15.3.2. Using the GeoIP2 parser
15.3.3. Transferring your logs to Elasticsearch using GeoIP2
15.3.4. Options of geoip2 parsers
16. Statistics of syslog-ng
17. Multithreading and scaling in syslog-ng OSE
17.1. Multithreading concepts of syslog-ng OSE
17.2. Configuring multithreading
17.3. Optimizing multithreaded performance
18. Troubleshooting syslog-ng
18.1. Possible causes of losing log messages
18.2. Creating syslog-ng core files
18.3. Collecting debugging information with strace, truss, or tusc
18.4. Running a failure script
18.5. Stopping syslog-ng
18.6. Reporting bugs and finding help
18.7. Recover data from orphaned diskbuffer files
19. Best practices and examples
19.1. General recommendations
19.2. Handling large message load
19.3. Using name resolution in syslog-ng
19.3.1. Resolving hostnames locally
19.4. Collecting logs from chroot
19.5. Configuring log rotation
A. The syslog-ng manual pages
dqtool — Display the contents of a disk-buffer file created with syslog-ng Open Source Edition
loggen — Generate syslog messages at a specified rate
pdbtool — An application to test and convert syslog-ng pattern database rules
syslog-ng-debun — syslog-ng DEBUg buNdle generator
syslog-ng — syslog-ng system logger application
syslog-ng.conf — syslog-ng configuration file
syslog-ng-ctl — Display message statistics and enable verbose, debug and trace modes in syslog-ng Open Source Edition
B. Open source licenses
B.1. GNU General Public License
B.1.1. Preamble
B.1.2. TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
B.1.3. How to Apply These Terms to Your New Programs
B.2. GNU Lesser General Public License
B.2.1. Preamble
B.2.2. TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
B.2.3. How to Apply These Terms to Your New Libraries
B.3. License attributions
C. Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License
Glossary
Index
List of syslog-ng OSE parameters