Other formats |

The syslog-ng Open Source Edition 3.9 Administrator Guide

This guide is published under the Creative Commons Attribution-Noncommercial-No Derivative Works (by-nc-nd) 3.0 license. See Appendix D, Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License for details. The latest version is always available at https://www.balabit.com/support/documentation.

Some rights reserved.

This documentation and the product it describes are considered protected by copyright according to the applicable laws.

This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (https://www.openssl.org/). This product includes cryptographic software written by Eric Young (eay@cryptsoft.com)

AIX™, AIX 5L™, AS/400™, BladeCenter™, eServer™, IBM™, the IBM™ logo, IBM System i™, IBM System i5™, IBM System x™, iSeries™, i5/OS™, Netfinity™, NetServer™, OpenPower™, OS/400™, PartnerWorld™, POWER™, ServerGuide™, ServerProven™, and xSeries™ are trademarks or registered trademarks of International Business Machines.

Alliance Log Agent for System i™ is a registered trademark of Patrick Townsend & Associates, Inc.

The Balabit™ name and the Balabit™ logo are registered trademarks of Balabit SA.

Debian™ is a registered trademark of Software in the Public Interest Inc.

Hadoop™ and the Hadoop elephant logo are trademarks of the Apache Software Foundation.

Linux™ is a registered trademark of Linus Torvalds.

MapR™, is a trademark of MapR Technologies, Inc.

Elasticsearch™ and Kibana™ is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.

Apache Kafka and the Apache Kafka Logo are trademarks of the Apache Software Foundation.

MySQL™ is a registered trademark of Oracle and/or its affiliates.

Oracle™, JD Edwards™, PeopleSoft™, and Siebel™ are registered trademarks of Oracle Corporation and/or its affiliates.

Red Hat™, Inc., Red HatEnterprise Linux™ and Red HatLinux™ are trademarks of Red Hat, Inc.

SUSE™ is a trademark of SUSE AG, a Novell business.

Solaris™ is a registered trademark of Oracle and/or its affiliates.

The syslog-ng™ name and the syslog-ng™ logo are registered trademarks of Balabit.

Windows™ 95, 98, ME, 2000, XP, Server 2003, Vista, Server 2008, 7, 8, and Server 2012 are registered trademarks of Microsoft Corporation.

All other product names mentioned herein are the trademarks of their respective owners.

DISCLAIMER. Balabit is not responsible for any third-party websites mentioned in this document. Balabit does not endorse and is not responsible or liable for any content, advertising, products, or other material on or available from such sites or resources. Balabit will not be responsible or liable for any damage or loss caused or alleged to be caused by or in connection with use of or reliance on any such content, goods, or services that are available on or through any such sites or resources.

February 17, 2017

Table of Contents

1. Summary of contents
2. Target audience and prerequisites
3. Products covered in this guide
4. Typographical conventions
5. Contact and support information
5.1. Sales contact
5.2. Support contact
5.3. Training
6. About this document
6.1. Summary of changes
6.2. Feedback
6.3. Acknowledgments
1. Introduction to syslog-ng
1.1. What syslog-ng is
1.2. What syslog-ng is not
1.3. Why is syslog-ng needed?
1.4. What is new in syslog-ng Open Source Edition 3.9?
1.5. Who uses syslog-ng?
1.6. Supported platforms
2. The concepts of syslog-ng
2.1. The philosophy of syslog-ng
2.2. Logging with syslog-ng
2.2.1. The route of a log message in syslog-ng
2.3. Modes of operation
2.3.1. Client mode
2.3.2. Relay mode
2.3.3. Server mode
2.4. Global objects
2.5. Timezones and daylight saving
2.5.1. How syslog-ng OSE assigns timezone to the message
2.5.2. A note on timezones and timestamps
2.6. The license of syslog-ng OSE
2.7. High availability support
2.8. The structure of a log message
2.8.1. BSD-syslog or legacy-syslog messages
2.8.2. IETF-syslog messages
2.9. Message representation in syslog-ng OSE
2.10. Structuring macros, metadata, and other value-pairs
2.10.1. Specifying data types in value-pairs
2.11. Things to consider when forwarding messages between syslog-ng OSE hosts
3. Installing syslog-ng
3.1. Compiling syslog-ng from source
3.2. Compiling options of syslog-ng OSE
3.3. Uninstalling syslog-ng OSE
3.4. Configuring Microsoft SQL Server to accept logs from syslog-ng
4. The syslog-ng OSE quick-start guide
4.1. Configuring syslog-ng on client hosts
4.2. Configuring syslog-ng on server hosts
4.3. Configuring syslog-ng relays
4.3.1. Configuring syslog-ng on relay hosts
4.3.2. How relaying log messages works
5. The syslog-ng OSE configuration file
5.1. Notes about the configuration syntax
5.2. Defining configuration objects inline
5.3. Using channels in configuration objects
5.4. Global and environmental variables
5.5. Modules in syslog-ng OSE
5.5.1. Loading modules
5.6. Managing complex syslog-ng configurations
5.6.1. Including configuration files
5.6.2. Reusing configuration blocks
5.6.3. Generating configuration blocks from a script
6. Collecting log messages — sources and source drivers
6.1. How sources work
6.2. Collecting internal messages
6.2.1. internal() source options
6.3. Collecting messages from text files
6.3.1. Notes on reading kernel messages
6.3.2. file() source options
6.4. Collecting messages using the RFC3164 protocol (network() driver)
6.4.1. network() source options
6.5. Receiving JSON messages from nodejs applications
6.5.1. nodejs() source options
6.6. Converting local e-mail messages to log messages
6.7. Collecting messages from named pipes
6.7.1. pipe() source options
6.8. Collecting process accounting logs on Linux
6.8.1. pacct() options
6.9. Receiving messages from external applications
6.9.1. program() source options
6.10. Collecting messages on Sun Solaris
6.10.1. sun-streams() source options
6.11. Collecting messages using the IETF syslog protocol (syslog() driver)
6.11.1. syslog() source options
6.12. Collecting the system-specific log messages of a platform
6.13. Collecting messages from the systemd-journal system log storage
6.13.1. systemd-journal() source options
6.14. Collecting systemd messages using a socket
6.15. Collecting messages from remote hosts using the BSD syslog protocol
6.15.1. tcp(), tcp6(), udp() and udp6() source options — OBSOLETE
6.16. Collecting messages from UNIX domain sockets
6.16.1. UNIX credentials and other metadata
6.16.2. unix-stream() and unix-dgram() source options
7. Sending and storing log messages — destinations and destination drivers
7.1. Publishing messages using AMQP
7.1.1. amqp() destination options
7.2. Sending messages directly to Elasticsearch version 1.x
7.2.1. Prerequisites
7.2.2. How syslog-ng OSE interacts with Elasticsearch
7.2.3. Client modes
7.2.4. Elasticsearch destination options
7.3. Sending messages directly to Elasticsearch version 2.0 or higher
7.3.1. Prerequisites
7.3.2. How syslog-ng OSE interacts with Elasticsearch
7.3.3. Client modes
7.3.4. Elasticsearch X-Pack (Shield) and syslog-ng OSE
7.3.5. Search Guard and syslog-ng OSE
7.3.6. Elasticsearch destination options
7.4. Storing messages in plain-text files
7.4.1. file() destination options
7.5. Sending metrics to Graphite
7.5.1. graphite() destination options
7.6. Storing messages on the Hadoop Distributed File System (HDFS)
7.6.1. Prerequisites
7.6.2. How syslog-ng OSE interacts with HDFS
7.6.3. Storing messages with MapR-FS
7.6.4. HDSF destination options
7.7. Posting messages over HTTP
7.7.1. HTTP destination options
7.8. Posting messages over HTTP without Java
7.8.1. HTTP destination options
7.9. Publishing messages to Apache Kafka
7.9.1. Prerequisites
7.9.2. How syslog-ng OSE interacts with Apache Kafka
7.9.3. Kafka destination options
7.10. Using Loggly
7.10.1. loggly() destination options
7.11. Using Logmatic.io
7.11.1. logmatic() destination options
7.12. Storing messages in a MongoDB database
7.12.1. How syslog-ng OSE connects the MongoDB server
7.12.2. mongodb() destination options
7.13. Sending messages to a remote log server using the RFC3164 protocol (network() driver)
7.13.1. network() destination options
7.14. Sending messages to named pipes
7.14.1. pipe() destination options
7.15. Sending messages to external applications
7.15.1. program() destination options
7.16. pseudofile()
7.16.1. pseudofile() destination options
7.17. Storing name-value pairs in Redis
7.17.1. redis() destination options
7.18. Monitoring your data with Riemann
7.18.1. riemann() destination options
7.19. Generating SMTP messages (e-mail) from logs
7.19.1. smtp() destination options
7.20. Storing messages in an SQL database
7.20.1. Using the sql() driver with an Oracle database
7.20.2. Using the sql() driver with a Microsoft SQL database
7.20.3. The way syslog-ng interacts with the database
7.20.4. sql() destination options
7.21. Publishing messages using STOMP
7.21.1. stomp() destination options
7.22. Sending messages to a remote logserver using the IETF-syslog protocol
7.22.1. syslog() destination options
7.23. Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers)
7.23.1. tcp(), tcp6(), udp(), and udp6() destination options
7.24. Sending messages to UNIX domain sockets
7.24.1. unix-stream() and unix-dgram() destination options
7.25. Sending messages to a user terminal — usertty() destination
7.26. Write your own custom destination in Java or Python
8. Routing messages: log paths and filters
8.1. Log paths
8.1.1. Embedded log statements
8.1.2. Junctions and channels
8.1.3. Log path flags
8.2. Managing incoming and outgoing messages with flow-control
8.2.1. Flow-control and multiple destinations
8.2.2. Configuring flow-control
8.3. Using disk-based and memory buffering
8.3.1. Enabling reliable disk-based buffering
8.3.2. Enabling normal disk-based buffering
8.3.3. Enabling memory buffering
8.4. Filters
8.4.1. Using filters
8.4.2. Combining filters with boolean operators
8.4.3. Comparing macro values in filters
8.4.4. Using wildcards, special characters, and regular expressions in filters
8.4.5. Tagging messages
8.4.6. Filter functions
8.5. Dropping messages
9. Global options of syslog-ng OSE
9.1. Configuring global syslog-ng options
9.2. Global options
10. TLS-encrypted message transfer
10.1. Secure logging using TLS
10.2. Encrypting log messages with TLS
10.2.1. Configuring TLS on the syslog-ng clients
10.2.2. Configuring TLS on the syslog-ng server
10.3. Mutual authentication using TLS
10.3.1. Configuring TLS on the syslog-ng clients
10.3.2. Configuring TLS on the syslog-ng server
10.4. TLS options
11. Manipulating messages
11.1. Customizing message format
11.1.1. Formatting messages, filenames, directories, and tablenames
11.1.2. Templates and macros
11.1.3. Date-related macros
11.1.4. Hard vs. soft macros
11.1.5. Macros of syslog-ng OSE
11.1.6. Using template functions
11.1.7. Template functions of syslog-ng OSE
11.1.8. Modifying the on-the-wire message format
11.2. Modifying messages
11.2.1. Replacing message parts
11.2.2. Setting message fields to specific values
11.2.3. Unsetting message fields
11.2.4. Creating custom SDATA fields
11.2.5. Setting multiple message fields to specific values
11.2.6. Conditional rewrites
11.2.7. Adding and deleting tags
11.2.8. Anonymizing credit card numbers
11.3. Regular expressions
11.3.1. Types and options of regular expressions
11.3.2. Optimizing regular expressions
12. Parsing and segmenting structured messages
12.1. Parsing syslog messages
12.1.1. Options of syslog-parser parsers
12.2. Parsing messages with comma-separated and similar values
12.2.1. Options of CSV parsers
12.3. Parsing key=value pairs
12.3.1. Options of key=value parsers
12.4. The JSON parser
12.4.1. Options of JSON parsers
12.5. Parsing dates and timestamps
12.5.1. Options of date-parser() parsers
12.6. The Apache Access Log Parser
12.6.1. Options of apache-accesslog-parser() parsers
12.7. The Linux Audit Parser
12.7.1. Options of linux-audit-parser() parsers
13. Processing message content with a pattern database
13.1. Classifying log messages
13.1.1. The structure of the pattern database
13.1.2. How pattern matching works
13.1.3. Artificial ignorance
13.2. Using pattern databases
13.2.1. Using parser results in filters and templates
13.2.2. Downloading sample pattern databases
13.3. Correlating log messages using pattern databases
13.3.1. Referencing earlier messages of the context
13.4. Triggering actions for identified messages
13.4.1. Conditional actions
13.4.2. External actions
13.4.3. Actions and message correlation
13.5. Creating pattern databases
13.5.1. Using pattern parsers
13.5.2. What's new in the syslog-ng pattern database format V5
13.5.3. The syslog-ng pattern database format
14. Correlating log messages
14.1. Correlating messages using the grouping-by() parser
14.1.1. Referencing earlier messages of the context
14.1.2. Options of grouping-by parsers
15. Enriching log messages with external data
15.1. Adding metadata from an external file
15.1.1. Options add-contextual-data()
15.2. Looking up GeoIP data from IP addresses
15.2.1. Options of geoip parsers
16. Statistics of syslog-ng
17. Multithreading and scaling in syslog-ng OSE
17.1. Multithreading concepts of syslog-ng OSE
17.2. Configuring multithreading
17.3. Optimizing multithreaded performance
18. Troubleshooting syslog-ng
18.1. Possible causes of losing log messages
18.2. Creating syslog-ng core files
18.3. Collecting debugging information with strace, truss, or tusc
18.4. Running a failure script
18.5. Stopping syslog-ng
18.6. Reporting bugs and finding help
19. Best practices and examples
19.1. General recommendations
19.2. Handling large message load
19.3. Using name resolution in syslog-ng
19.3.1. Resolving hostnames locally
19.4. Collecting logs from chroot
19.5. Configuring log rotation
A. The syslog-ng manual pages
dqtool — Display the contents of a disk-buffer file created with syslog-ng Open Source Edition
loggen — Generate syslog messages at a specified rate
pdbtool — An application to test and convert syslog-ng pattern database rules
syslog-debun — syslog-ng DEBUg buNdle generator
syslog-ng — syslog-ng system logger application
syslog-ng.conf — syslog-ng configuration file
syslog-ng-ctl — Display message statistics and enable verbose, debug and trace modes in syslog-ng Open Source Edition
B. GNU General Public License
B.1. Preamble
B.2.1. Section 0
B.2.2. Section 1
B.2.3. Section 2
B.2.4. Section 3
B.2.5. Section 4
B.2.6. Section 5
B.2.7. Section 6
B.2.8. Section 7
B.2.9. Section 8
B.2.10. Section 9
B.2.11. Section 10
B.2.12. NO WARRANTY Section 11
B.2.13. Section 12
B.3. How to Apply These Terms to Your New Programs
C. GNU Lesser General Public License
C.1. Preamble
C.2.1. Section 0
C.2.2. Section 1
C.2.3. Section 2
C.2.4. Section 3
C.2.5. Section 4
C.2.6. Section 5
C.2.7. Section 6
C.2.8. Section 7
C.2.9. Section 8
C.2.10. Section 9
C.2.11. Section 10
C.2.12. Section 11
C.2.13. Section 12
C.2.14. Section 13
C.2.15. Section 14
C.2.16. NO WARRANTY Section 15
C.2.17. Section 16
C.3. How to Apply These Terms to Your New Libraries
D. Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License
List of syslog-ng OSE parameters

List of Examples

2.1. Using type-hinting
2.2. Using the value-pairs() option
2.3. Using the rekey() option
4.1. The default configuration file of syslog-ng OSE
4.2. A simple configuration for clients
4.3. A simple configuration for servers
4.4. A simple configuration for relays
5.1. A simple configuration file
5.2. Using required and optional parameters
5.3. Using inline definitions
5.4. Using channels
5.5. Using global variables
5.6. Reusing configuration blocks
5.7. Defining blocks with multiple elements
5.8. Passing arguments to blocks
5.9. Using arguments in blocks
6.1. A simple source statement
6.2. A source statement using two source drivers
6.3. Setting default priority and facility
6.4. Source statement on a Linux based operating system
6.5. Using the internal() driver
6.6. Using the file() driver
6.7. Tailing files
6.8. Processing indented multi-line messages
6.9. Processing Tomcat logs
6.10. Using the network() driver
6.11. Initial window size of a connection
6.12. Processing indented multi-line messages
6.13. Processing Tomcat logs
6.14. Using the nodejs() driver
6.15. Using the mbox() driver
6.16. Using the pipe() driver
6.17. Initial window size of a connection
6.18. Processing indented multi-line messages
6.19. Processing Tomcat logs
6.20. Using the program() driver
6.21. Initial window size of a connection
6.22. Using the sun-streams() driver
6.23. Initial window size of a connection
6.24. Using the syslog() driver
6.25. Initial window size of a connection
6.26. Processing indented multi-line messages
6.27. Processing Tomcat logs
6.28. Sending all fields through syslog protocol using the systemd-journal() driver
6.29. Filtering for a specific field using the systemd-journal() driver
6.30. Sending all fields in value-pairs using the systemd-journal() driver
6.31. Using the systemd-syslog() driver
6.32. Using the unix-stream() and unix-dgram() drivers
6.33. Initial window size of a connection
7.1. A simple destination statement
7.2. Using the amqp() driver
7.3. Examples for using disk-buffer()
7.4. Sending log data to Elasticsearch version 1.x
7.5. Example for the .yml file
7.6. Examples for using disk-buffer()
7.7. Sending log data to Elasticsearch version 2.x and above
7.8. Sending log data to Elasticsearch using the HTTP REST API
7.9. Examples for using disk-buffer()
7.10. Using the file() driver
7.11. Using the file() driver with macros in the file name and a template for the message
7.12. Examples for using disk-buffer()
7.13. Using the graphite() driver
7.14. Storing logfiles on HDFS
7.15. Storing logfiles with MapR-FS
7.16. Examples for using disk-buffer()
7.17. Sending log data to a web service
7.18. Sending log data to a web service
7.19. Examples for using disk-buffer()
7.20. Sending log data to Apache Kafka
7.21. Using the loggly() driver
7.22. Using the logmatic() driver
7.23. Using the mongodb() driver
7.24. Examples for using disk-buffer()
7.25. Using the network() driver
7.26. Examples for using disk-buffer()
7.27. Using the pipe() driver
7.28. Using the program() destination driver
7.29. Examples for using disk-buffer()
7.30. Using the redis() driver
7.31. Examples for using disk-buffer()
7.32. Using the riemann() driver
7.33. Examples for using disk-buffer()
7.34. Using the smtp() driver
7.35. Simple e-mail alerting with the smtp() driver
7.36. Examples for using disk-buffer()
7.37. Using the sql() driver
7.38. Using the sql() driver with an Oracle database
7.39. Using the sql() driver with an MSSQL database
7.40. Examples for using disk-buffer()
7.41. Setting flags for SQL destinations
7.42. Using SQL NULL values
7.43. Value: default
7.44. Using the stomp() driver
7.45. Examples for using disk-buffer()
7.46. Using the syslog() driver
7.47. Examples for using disk-buffer()
7.48. Using the unix-stream() driver
7.49. Examples for using disk-buffer()
7.50. Using the usertty() driver
8.1. A simple log statement
8.2. Using embedded log paths
8.3. Using junctions
8.4. Using log path flags
8.5. Soft flow-control
8.6. Hard flow-control
8.7. Sizing parameters for flow-control
8.8. Example for using reliable disk-based buffering
8.9. Example for using normal disk-based buffering
8.10. Example for using memory buffering
8.11. A simple filter statement
8.12. Comparing macro values in filters
8.13. Filtering with widcards
8.14. Selecting messages using the in-list filter
8.15. Adding tags and filtering messages with tags
8.16. Skipping messages
9.1. Using global options
10.1. A destination statement using TLS
10.2. A source statement using TLS
10.3. Disabling mutual authentication
10.4. A destination statement using mutual authentication
10.5. A source statement using TLS
10.6. Using ssl-options
11.1. Using templates and macros
11.2. Using SDATA macros
11.3. Using custom template functions
11.4. Using the format-cef-extension template function
11.5. Using the format-json template function
11.6. Using the format-welf() template function
11.7. Using the graphite-output template function
11.8. Using the grep template function
11.9. Using the $(hash) template function
11.10. Anonymizing IP addresses
11.11. Using pattern databases and the if template function
11.12. Using the indent-multi-line template function
11.13. Using the padding template function
11.14. Writing template functions in Python
11.15. Using the sanitize template function
11.16. Using the substr template function
11.17. Using Universally Unique Identifiers
11.18. Using substitution rules
11.19. Anonymizing IP addresses
11.20. Setting message fields to a particular value
11.21. Unsetting a message field
11.22. Rewriting custom SDATA fields
11.23. Using groupset rewrite rules
11.24. Using conditional rewriting
11.25. Using Posix regular expressions
11.26. Using PCRE regular expressions
11.27. Optimizing regular expressions in filters
12.1. Using junctions
12.2. Segmenting hostnames separated with a dash
12.3. Parsing Apache log files
12.4. Segmenting a part of a message
12.5. Adding the end of the message to the last column
12.6. Using a key=value parser
12.7. Using a JSON parser
12.8. Convert logstash eventlog format v0 to v1
12.9. Using the marker option in JSON parser
12.10. Using the date-parser()
12.11. Using the apache-accesslog-parser parser
12.12. Using the linux-audit-parser() parser
13.1. Defining pattern databases
13.2. Using classification results
13.3. Using classification results for filtering messages
13.4. Using pattern parsers as macros
13.5. How syslog-ng OSE calculates context-timeout
13.6. Using message correlation
13.7. Referencing values from an earlier message
13.8. Using the grep template function
13.9. Sending triggered messages to the internal() source
13.10. Generating messages for pattern database matches
13.11. Generating messages with inherited values
13.12. Creating a new context from an action
13.13. Actions based on the number of messages
13.14. Sending triggered messages to external applications
13.15. Referencing values from an earlier message
13.16. Using the inherit-properties option
13.17. Sending alert when a client disappears
13.18. Pattern parser syntax
13.19. Using the STRING and ESTRING parsers
13.20. A pattern database containing a single rule
13.21. Generating messages for pattern database matches
13.22. Generating messages with inherited values
13.23. Generating messages for pattern database matches
13.24. Generating messages with inherited values
14.1. How syslog-ng OSE calculates context-timeout
14.2. Correlating Linux Audit logs
14.3. Referencing values from an earlier message
14.4. Using the grep template function
14.5. Sending triggered messages to the internal() source
15.1. Adding metadata from a CSV file
15.2. Using the GeoIP parser
17.1. Enabling multithreading
19.1. File destination for log rotation
19.2. Command for cron for log rotation
A.1. Using required and optional parameters
A.2. Using global options