Other formats |

The syslog-ng Open Source Edition 3.6 Administrator Guide

This guide is published under the Creative Commons Attribution-Noncommercial-No Derivative Works (by-nc-nd) 3.0 license. See Appendix D, Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License for details. The latest version is always available at https://www.balabit.com/support/documentation.

Some rights reserved.

This documentation and the product it describes are considered protected by copyright according to the applicable laws.

This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (https://www.openssl.org/). This product includes cryptographic software written by Eric Young (eay@cryptsoft.com)

AIX™, AIX 5L™, AS/400™, BladeCenter™, eServer™, IBM™, the IBM™ logo, IBM System i™, IBM System i5™, IBM System x™, iSeries™, i5/OS™, Netfinity™, NetServer™, OpenPower™, OS/400™, PartnerWorld™, POWER™, ServerGuide™, ServerProven™, and xSeries™ are trademarks or registered trademarks of International Business Machines.

Alliance Log Agent for System i™ is a registered trademark of Patrick Townsend & Associates, Inc.

The Balabit™ name and the Balabit™ logo are registered trademarks of Balabit SA.

Debian™ is a registered trademark of Software in the Public Interest Inc.

Hadoop™ and the Hadoop elephant logo are trademarks of the Apache Software Foundation.

Linux™ is a registered trademark of Linus Torvalds.

MapR™, is a trademark of MapR Technologies, Inc.

Elasticsearch™ and Kibana™ is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.

Apache Kafka and the Apache Kafka Logo are trademarks of the Apache Software Foundation.

MySQL™ is a registered trademark of Oracle and/or its affiliates.

Oracle™, JD Edwards™, PeopleSoft™, and Siebel™ are registered trademarks of Oracle Corporation and/or its affiliates.

Red Hat™, Inc., Red HatEnterprise Linux™ and Red HatLinux™ are trademarks of Red Hat, Inc.

SUSE™ is a trademark of SUSE AG, a Novell business.

Solaris™ is a registered trademark of Oracle and/or its affiliates.

The syslog-ng™ name and the syslog-ng™ logo are registered trademarks of Balabit.

Windows™ 95, 98, ME, 2000, XP, Server 2003, Vista, Server 2008, 7, 8, and Server 2012 are registered trademarks of Microsoft Corporation.

All other product names mentioned herein are the trademarks of their respective owners.


Balabit is not responsible for any third-party websites mentioned in this document. Balabit does not endorse and is not responsible or liable for any content, advertising, products, or other material on or available from such sites or resources. Balabit will not be responsible or liable for any damage or loss caused or alleged to be caused by or in connection with use of or reliance on any such content, goods, or services that are available on or through any such sites or resources.

November 30, 2016

This manual is the primary documentation of the syslog-ng Open Source Edition 3.6 application.

Table of Contents

1. Summary of contents
2. Target audience and prerequisites
3. Products covered in this guide
4. Typographical conventions
5. Contact and support information
5.1. Sales contact
5.2. Support contact
5.3. Training
6. About this document
6.1. Summary of changes
6.2. Feedback
6.3. Acknowledgments
1. Introduction to syslog-ng
1.1. What syslog-ng is
1.2. What syslog-ng is not
1.3. Why is syslog-ng needed?
1.4. What is new in syslog-ng Open Source Edition 3.6?
1.5. Who uses syslog-ng?
1.6. Supported platforms
2. The concepts of syslog-ng
2.1. The philosophy of syslog-ng
2.2. Logging with syslog-ng
2.2.1. The route of a log message in syslog-ng
2.3. Modes of operation
2.3.1. Client mode
2.3.2. Relay mode
2.3.3. Server mode
2.4. Global objects
2.5. Timezones and daylight saving
2.5.1. How syslog-ng OSE assigns timezone to the message
2.5.2. A note on timezones and timestamps
2.6. The license of syslog-ng OSE
2.7. High availability support
2.8. The structure of a log message
2.8.1. BSD-syslog or legacy-syslog messages
2.8.2. IETF-syslog messages
2.9. Message representation in syslog-ng OSE
2.10. Structuring macros, metadata, and other value-pairs
2.10.1. Specifying data types in value-pairs
3. Installing syslog-ng
3.1. Compiling syslog-ng from source
3.2. Uninstalling syslog-ng OSE
3.3. Configuring Microsoft SQL Server to accept logs from syslog-ng
4. The syslog-ng OSE quick-start guide
4.1. Configuring syslog-ng on client hosts
4.2. Configuring syslog-ng on server hosts
4.3. Configuring syslog-ng relays
4.3.1. Configuring syslog-ng on relay hosts
4.3.2. How relaying log messages works
5. The syslog-ng OSE configuration file
5.1. Location of the syslog-ng configuration file
5.2. The configuration syntax in detail
5.3. Notes about the configuration syntax
5.4. Defining configuration objects inline
5.5. Using channels in configuration objects
5.6. Global and environmental variables
5.7. Modules in syslog-ng OSE
5.7.1. Loading modules
5.8. Managing complex syslog-ng configurations
5.8.1. Including configuration files
5.8.2. Reusing configuration blocks
5.8.3. Generating configuration blocks from a script
6. Collecting log messages — sources and source drivers
6.1. How sources work
6.2. Collecting internal messages
6.2.1. internal() source options
6.3. Collecting messages from text files
6.3.1. Notes on reading kernel messages
6.3.2. file() source options
6.4. Collecting messages using the RFC3164 protocol (network() driver)
6.4.1. network() source options
6.5. Receiving JSON messages from nodejs applications
6.5.1. nodejs() source options
6.6. Collecting messages from named pipes
6.6.1. pipe() source options
6.7. Collecting process accounting logs on Linux
6.7.1. pacct() options
6.8. Receiving messages from external applications
6.8.1. program() source options
6.9. Collecting messages on Sun Solaris
6.9.1. sun-streams() source options
6.10. Collecting messages using the IETF syslog protocol (syslog() driver)
6.10.1. syslog() source options
6.11. Collecting the system-specific log messages of a platform
6.12. Collecting messages from the systemd-journal system log storage
6.12.1. systemd-journal() source options
6.13. Collecting systemd messages using a socket
6.14. Collecting messages from remote hosts using the BSD syslog protocol
6.14.1. tcp(), tcp6(), udp() and udp6() source options — OBSOLETE
6.15. Collecting messages from UNIX domain sockets
6.15.1. UNIX credentials and other metadata
6.15.2. unix-stream() and unix-dgram() source options
7. Sending and storing log messages — destinations and destination drivers
7.1. Publishing messages using AMQP
7.1.1. amqp() destination options
7.2. Storing messages in plain-text files
7.2.1. file() destination options
7.3. Sending metrics to Graphite
7.3.1. graphite() destination options
7.4. Storing messages in a MongoDB database
7.4.1. How syslog-ng OSE connects the MongoDB server
7.4.2. mongodb() destination options
7.5. Sending messages to a remote log server using the RFC3164 protocol (network() driver)
7.5.1. network() destination options
7.6. Sending messages to named pipes
7.6.1. pipe() destination options
7.7. Sending messages to external applications
7.7.1. program() destination options
7.8. pseudofile()
7.8.1. pseudofile() destination options
7.9. Storing name-value pairs in Redis
7.9.1. redis() destination options
7.10. Monitoring your data with Riemann
7.10.1. riemann() destination options
7.11. Generating SMTP messages (e-mail) from logs
7.11.1. smtp() destination options
7.12. Storing messages in an SQL database
7.12.1. Using the sql() driver with an Oracle database
7.12.2. Using the sql() driver with a Microsoft SQL database
7.12.3. The way syslog-ng interacts with the database
7.12.4. sql() destination options
7.13. Publishing messages using STOMP
7.13.1. stomp() destination options
7.14. Sending messages to a remote logserver using the IETF-syslog protocol
7.14.1. syslog() destination options
7.15. Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers)
7.15.1. tcp(), tcp6(), udp(), and udp6() destination options
7.16. Sending messages to UNIX domain sockets
7.16.1. unix-stream() and unix-dgram() destination options
7.17. Sending messages to a user terminal — usertty() destination
8. Routing messages: log paths and filters
8.1. Log paths
8.1.1. Embedded log statements
8.1.2. Junctions and channels
8.1.3. Log path flags
8.2. Managing incoming and outgoing messages with flow-control
8.2.1. Flow-control and multiple destinations
8.2.2. Configuring flow-control
8.3. Filters
8.3.1. Using filters
8.3.2. Combining filters with boolean operators
8.3.3. Comparing macro values in filters
8.3.4. Using wildcards, special characters, and regular expressions in filters
8.3.5. Tagging messages
8.3.6. Filter functions
8.4. Dropping messages
9. Global options of syslog-ng OSE
9.1. Configuring global syslog-ng options
9.2. Global options
10. TLS-encrypted message transfer
10.1. Secure logging using TLS
10.2. Encrypting log messages with TLS
10.2.1. Configuring TLS on the syslog-ng clients
10.2.2. Configuring TLS on the syslog-ng server
10.3. Mutual authentication using TLS
10.3.1. Configuring TLS on the syslog-ng clients
10.3.2. Configuring TLS on the syslog-ng server
10.4. TLS options
11. Manipulating messages
11.1. Customizing message format
11.1.1. Formatting messages, filenames, directories, and tablenames
11.1.2. Templates and macros
11.1.3. Date-related macros
11.1.4. Hard vs. soft macros
11.1.5. Macros of syslog-ng OSE
11.1.6. Using template functions
11.1.7. Template functions of syslog-ng OSE
11.1.8. Modifying the on-the-wire message format
11.2. Modifying messages
11.2.1. Replacing message parts
11.2.2. Setting message fields to specific values
11.2.3. Creating custom SDATA fields
11.2.4. Setting multiple message fields to specific values
11.2.5. Conditional rewrites
11.2.6. Adding and deleting tags
11.2.7. Anonymizing credit card numbers
11.3. Regular expressions
11.3.1. Types and options of regular expressions
11.3.2. Optimizing regular expressions
12. Parsing and segmenting structured messages
12.1. Parsing syslog messages
12.2. Parsing messages
12.2.1. Options of CSV parsers
12.3. The JSON parser
12.3.1. Options of JSON parsers
13. Processing message content with a pattern database
13.1. Classifying log messages
13.1.1. The structure of the pattern database
13.1.2. How pattern matching works
13.1.3. Artificial ignorance
13.2. Using pattern databases
13.2.1. Using parser results in filters and templates
13.2.2. Downloading sample pattern databases
13.3. Correlating log messages
13.3.1. Referencing earlier messages of the context
13.4. Triggering actions for identified messages
13.4.1. Conditional actions
13.4.2. External actions
13.4.3. Actions and message correlation
13.5. Creating pattern databases
13.5.1. Using pattern parsers
13.5.2. What's new in the syslog-ng pattern database format V4
13.5.3. The syslog-ng pattern database format
14. Statistics of syslog-ng
15. Multithreading and scaling in syslog-ng OSE
15.1. Multithreading concepts of syslog-ng OSE
15.2. Configuring multithreading
15.3. Optimizing multithreaded performance
16. Troubleshooting syslog-ng
16.1. Possible causes of losing log messages
16.2. Creating syslog-ng core files
16.3. Collecting debugging information with strace, truss, or tusc
16.4. Running a failure script
16.5. Stopping syslog-ng
17. Best practices and examples
17.1. General recommendations
17.2. Handling large message load
17.3. Using name resolution in syslog-ng
17.3.1. Resolving hostnames locally
17.4. Collecting logs from chroot
17.5. Configuring log rotation
A. The syslog-ng manual pages
loggen — Generate syslog messages at a specified rate
pdbtool — An application to test and convert syslog-ng pattern database rules
syslog-ng — syslog-ng system logger application
syslog-ng.conf — syslog-ng configuration file
syslog-ng-ctl — Display message statistics and enable verbose, debug and trace modes in syslog-ng Open Source Edition
B. GNU General Public License
B.1. Preamble
B.2.1. Section 0
B.2.2. Section 1
B.2.3. Section 2
B.2.4. Section 3
B.2.5. Section 4
B.2.6. Section 5
B.2.7. Section 6
B.2.8. Section 7
B.2.9. Section 8
B.2.10. Section 9
B.2.11. Section 10
B.2.12. NO WARRANTY Section 11
B.2.13. Section 12
B.3. How to Apply These Terms to Your New Programs
C. GNU Lesser General Public License
C.1. Preamble
C.2.1. Section 0
C.2.2. Section 1
C.2.3. Section 2
C.2.4. Section 3
C.2.5. Section 4
C.2.6. Section 5
C.2.7. Section 6
C.2.8. Section 7
C.2.9. Section 8
C.2.10. Section 9
C.2.11. Section 10
C.2.12. Section 11
C.2.13. Section 12
C.2.14. Section 13
C.2.15. Section 14
C.2.16. NO WARRANTY Section 15
C.2.17. Section 16
C.3. How to Apply These Terms to Your New Libraries
D. Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License
List of syslog-ng OSE parameters

List of Examples

2.1. Using type-hinting
2.2. Using the value-pairs() option
2.3. Using the rekey() option
4.1. The default configuration file of syslog-ng OSE
4.2. A simple configuration for clients
4.3. A simple configuration for servers
4.4. A simple configuration for relays
5.1. A simple configuration file
5.2. Using required and optional parameters
5.3. Using inline definitions
5.4. Using channels
5.5. Using global variables
5.6. Reusing configuration blocks
5.7. Defining blocks with multiple elements
5.8. Passing arguments to blocks
5.9. Using arguments in blocks
6.1. A simple source statement
6.2. A source statement using two source drivers
6.3. Setting default priority and facility
6.4. Source statement on a Linux based operating system
6.5. Using the internal() driver
6.6. Using the file() driver
6.7. Tailing files
6.8. Processing indented multi-line messages
6.9. Processing Tomcat logs
6.10. Using the network() driver
6.11. Initial window size of a connection
6.12. Using the nodejs() driver
6.13. Using the pipe() driver
6.14. Initial window size of a connection
6.15. Processing indented multi-line messages
6.16. Processing Tomcat logs
6.17. Using the program() driver
6.18. Initial window size of a connection
6.19. Using the sun-streams() driver
6.20. Initial window size of a connection
6.21. Using the syslog() driver
6.22. Initial window size of a connection
6.23. Sending all fields through syslog protocol using the systemd-journal() driver
6.24. Filtering for a specific field using the systemd-journal() driver
6.25. Sending all fields in value-pairs using the systemd-journal() driver
6.26. Using the systemd-syslog() driver
6.27. Using the unix-stream() and unix-dgram() drivers
6.28. Initial window size of a connection
7.1. A simple destination statement
7.2. Using the amqp() driver
7.3. Using the file() driver
7.4. Using the file() driver with macros in the file name and a template for the message
7.5. Using the graphite() driver
7.6. Using the mongodb() driver
7.7. Using the network() driver
7.8. Using the pipe() driver
7.9. Using the program() destination driver
7.10. Using the redis() driver
7.11. Using the riemann() driver
7.12. Using the smtp() driver
7.13. Simple e-mail alerting with the smtp() driver
7.14. Using the sql() driver
7.15. Using the sql() driver with an Oracle database
7.16. Using the sql() driver with an MSSQL database
7.17. Setting flags for SQL destinations
7.18. Using SQL NULL values
7.19. Value: default
7.20. Using the stomp() driver
7.21. Using the syslog() driver
7.22. Using the unix-stream() driver
7.23. Using the usertty() driver
8.1. A simple log statement
8.2. Using embedded log paths
8.3. Using junctions
8.4. Using log path flags
8.5. Soft flow-control
8.6. Hard flow-control
8.7. Sizing parameters for flow-control
8.8. A simple filter statement
8.9. Comparing macro values in filters
8.10. Filtering with widcards
8.11. Selecting messages using the in-list filter
8.12. Adding tags and filtering messages with tags
8.13. Skipping messages
9.1. Using global options
10.1. A destination statement using TLS
10.2. A source statement using TLS
10.3. Disabling mutual authentication
10.4. A destination statement using mutual authentication
10.5. A source statement using TLS
11.1. Using templates and macros
11.2. Using SDATA macros
11.3. Using the format-json template function
11.4. Using the graphite-output template function
11.5. Using the grep template function
11.6. Using the $(hash) template function
11.7. Anonymizing IP addresses
11.8. Using pattern databases and the if template function
11.9. Using the indent-multi-line template function
11.10. Using the padding template function
11.11. Using the sanitize template function
11.12. Using the substr template function
11.13. Using Universally Unique Identifiers
11.14. Using substitution rules
11.15. Anonymizing IP addresses
11.16. Setting message fields to a particular value
11.17. Rewriting custom SDATA fields
11.18. Using groupset rewrite rules
11.19. Using conditional rewriting
11.20. Using Posix regular expressions
11.21. Using PCRE regular expressions
11.22. Optimizing regular expressions in filters
12.1. Using junctions
12.2. Segmenting hostnames separated with a dash
12.3. Parsing Apache log files
12.4. Segmenting a part of a message
12.5. Adding the end of the message to the last column
12.6. Using a JSON parser
12.7. Using the marker option in JSON parser
13.1. Defining pattern databases
13.2. Using classification results
13.3. Using classification results for filtering messages
13.4. Using pattern parsers as macros
13.5. How syslog-ng OSE calculates context-timeout
13.6. Using message correlation
13.7. Referencing values from an earlier message
13.8. Sending triggered messages to the internal() source
13.9. Generating messages for pattern database matches
13.10. Generating messages with inherited values
13.11. Actions based on the number of messages
13.12. Sending triggered messages to external applications
13.13. Referencing values from an earlier message
13.14. Sending alert when a client disappears
13.15. Pattern parser syntax
13.16. Using the STRING and ESTRING parsers
13.17. A V4 pattern database containing a single rule
15.1. Enabling multithreading
17.1. File destination for log rotation
17.2. Command for cron for log rotation
A.1. Using required and optional parameters
A.2. Using global options