The syslog-ng Open Source Edition 3.4 Administrator Guide

This guide is published under the Creative Commons Attribution-Noncommercial-No Derivative Works (by-nc-nd) 3.0 license. See Appendix D, Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License for details. The latest version is always available at https://www.balabit.com/support/documentation.

Some rights reserved.

This documentation and the product it describes are considered protected by copyright according to the applicable laws.

This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (http://www.openssl.org/). This product includes cryptographic software written by Eric Young (eay@cryptsoft.com)

AIX™, AIX 5L™, AS/400™, BladeCenter™, eServer™, IBM™, the IBM™ logo, IBM System i™, IBM System i5™, IBM System x™, iSeries™, i5/OS™, Netfinity™, NetServer™, OpenPower™, OS/400™, PartnerWorld™, POWER™, ServerGuide™, ServerProven™, and xSeries™ are trademarks or registered trademarks of International Business Machines.

Alliance Log Agent for System i™ is a registered trademark of Patrick Townsend & Associates, Inc.

The Balabit™ name and the Balabit™ logo are registered trademarks of Balabit SA.

Debian™ is a registered trademark of Software in the Public Interest Inc.

Hadoop™ and the Hadoop elephant logo are trademarks of the Apache Software Foundation.

Linux™ is a registered trademark of Linus Torvalds.

MapR™, is a trademark of MapR Technologies, Inc.

Elasticsearch™ and Kibana™ is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.

Apache Kafka and the Apache Kafka Logo are trademarks of the Apache Software Foundation.

MySQL™ is a registered trademark of Oracle and/or its affiliates.

Oracle™, JD Edwards™, PeopleSoft™, and Siebel™ are registered trademarks of Oracle Corporation and/or its affiliates.

Red Hat™, Inc., Red HatEnterprise Linux™ and Red HatLinux™ are trademarks of Red Hat, Inc.

SUSE™ is a trademark of SUSE AG, a Novell business.

Solaris™ is a registered trademark of Oracle and/or its affiliates.

The syslog-ng™ name and the syslog-ng™ logo are registered trademarks of Balabit.

Windows™ 95, 98, ME, 2000, XP, Server 2003, Vista, Server 2008, 7, 8, and Server 2012 are registered trademarks of Microsoft Corporation.

All other product names mentioned herein are the trademarks of their respective owners.

DISCLAIMER

Balabit is not responsible for any third-party websites mentioned in this document. Balabit does not endorse and is not responsible or liable for any content, advertising, products, or other material on or available from such sites or resources. Balabit will not be responsible or liable for any damage or loss caused or alleged to be caused by or in connection with use of or reliance on any such content, goods, or services that are available on or through any such sites or resources.

July 21, 2016

Abstract

This manual is the primary documentation of the syslog-ng Open Source Edition 3.4 application.


Table of Contents

Preface
1. Summary of contents
2. Target audience and prerequisites
3. Products covered in this guide
4. Typographical conventions
5. Contact and support information
5.1. Sales contact
5.2. Support contact
5.3. Training
6. About this document
6.1. Summary of changes
6.2. Feedback
6.3. Acknowledgments
1. Introduction to syslog-ng
1.1. What syslog-ng is
1.2. What syslog-ng is not
1.3. Why is syslog-ng needed?
1.4. What is new in syslog-ng Open Source Edition 3.4?
1.5. Who uses syslog-ng?
1.6. Supported platforms
2. The concepts of syslog-ng
2.1. The philosophy of syslog-ng
2.2. Logging with syslog-ng
2.2.1. The route of a log message in syslog-ng
2.3. Modes of operation
2.3.1. Client mode
2.3.2. Relay mode
2.3.3. Server mode
2.4. Global objects
2.5. Timezones and daylight saving
2.5.1. A note on timezones and timestamps
2.6. The license of syslog-ng OSE
2.7. High availability support
2.8. The structure of a log message
2.8.1. BSD-syslog or legacy-syslog messages
2.8.2. IETF-syslog messages
2.9. Message representation in syslog-ng OSE
2.10. Structuring macros, metadata, and other value-pairs
3. Installing syslog-ng
3.1. Compiling syslog-ng from source
3.2. Uninstalling syslog-ng OSE
3.3. Configuring Microsoft SQL Server to accept logs from syslog-ng
4. The syslog-ng OSE quick-start guide
4.1. Configuring syslog-ng on client hosts
4.2. Configuring syslog-ng on server hosts
4.3. Configuring syslog-ng relays
4.3.1. Configuring syslog-ng on relay hosts
4.3.2. How relaying log messages works
5. The syslog-ng OSE configuration file
5.1. Location of the syslog-ng configuration file
5.2. The configuration syntax in detail
5.3. Notes about the configuration syntax
5.4. Defining configuration objects inline
5.5. Using channels in configuration objects
5.6. Global and environmental variables
5.7. Loading modules
5.7.1. Loading modules
5.8. Managing complex syslog-ng configurations
5.8.1. Including configuration files
5.8.2. Reusing configuration blocks
6. Collecting log messages — sources and source drivers
6.1. How sources work
6.2. Collecting internal messages
6.2.1. internal() source options
6.3. Collecting messages from text files
6.3.1. Notes on reading kernel messages
6.3.2. file() source options
6.4. Collecting messages using the RFC3164 protocol
6.4.1. network() source options
6.5. Collecting messages from named pipes
6.5.1. pipe() source options
6.6. Collecting process accounting logs on Linux
6.6.1. pacct() options
6.7. Receiving messages from external applications
6.7.1. program() source options
6.8. Collecting messages on Sun Solaris
6.8.1. sun-streams() source options
6.9. Collecting messages using the IETF syslog protocol
6.9.1. syslog() source options
6.10. Collecting the system-specific log messages of a platform
6.11. Collecting messages from remote hosts using the BSD syslog protocol
6.11.1. tcp(), tcp6(), udp() and udp6() source options
6.12. Collecting messages from UNIX domain sockets
6.12.1. unix-stream() and unix-dgram() source options
7. Sending and storing log messages — destinations and destination drivers
7.1. Publishing messages using AMQP
7.1.1. amqp() destination options
7.2. Storing messages in plain-text files
7.2.1. file() destination options
7.3. Storing messages in a MongoDB database
7.3.1. mongodb() destination options
7.4. Sending messages to a remote logserver using the RFC3164 protocol
7.4.1. network() destination options
7.5. Sending messages to named pipes
7.5.1. pipe() destination options
7.6. Sending messages to external applications
7.6.1. program() destination options
7.7. Generating SMTP messages (e-mail) from logs
7.7.1. smtp() destination options
7.8. Storing messages in an SQL database
7.8.1. Using the sql() driver with an Oracle database
7.8.2. Using the sql() driver with a Microsoft SQL database
7.8.3. The way syslog-ng interacts with the database
7.8.4. sql() destination options
7.9. Sending messages to a remote logserver using the IETF-syslog protocol
7.9.1. syslog() destination options
7.10. Sending messages to a remote logserver using the legacy BSD-syslog protocol
7.10.1. tcp(), tcp6(), udp(), and udp6() destination options
7.11. Sending messages to UNIX domain sockets
7.11.1. unix-stream() and unix-dgram() destination options
7.12. Sending messages to a user terminal — usertty() destination
8. Routing messages: log paths and filters
8.1. Log paths
8.1.1. Embedded log statements
8.1.2. Junctions and channels
8.1.3. Log path flags
8.2. Managing incoming and outgoing messages with flow-control
8.2.1. Flow-control and multiple destinations
8.2.2. Configuring flow-control
8.3. Filters
8.3.1. Using filters
8.3.2. Combining filters with boolean operators
8.3.3. Comparing macro values in filters
8.3.4. Using wildcards, special characters, and regular expressions in filters
8.3.5. Tagging messages
8.3.6. Filter functions
8.4. Dropping messages
9. Global options of syslog-ng OSE
9.1. Configuring global syslog-ng options
9.2. Global options
10. TLS-encrypted message transfer
10.1. Secure logging using TLS
10.2. Encrypting log messages with TLS
10.2.1. Configuring TLS on the syslog-ng clients
10.2.2. Configuring TLS on the syslog-ng server
10.3. Mutual authentication using TLS
10.3.1. Configuring TLS on the syslog-ng clients
10.3.2. Configuring TLS on the syslog-ng server
10.4. TLS options
11. Manipulating messages
11.1. Customizing message format
11.1.1. Formatting messages, filenames, directories, and tablenames
11.1.2. Templates and macros
11.1.3. Date-related macros
11.1.4. Hard vs. soft macros
11.1.5. Macros of syslog-ng OSE
11.1.6. Using template functions
11.1.7. Template functions of syslog-ng OSE
11.2. Modifying messages
11.2.1. Replacing message parts
11.2.2. Setting message fields to specific values
11.2.3. Creating custom SDATA fields
11.2.4. Conditional rewrites
11.2.5. Adding and deleting tags
11.3. Regular expressions
11.3.1. Types and options of regular expressions
11.3.2. Optimizing regular expressions
12. Parsing and segmenting structured messages
12.1. Parsing syslog messages
12.2. Parsing messages
12.2.1. Options of CSV parsers
12.3. The JSON parser
13. Processing message content with a pattern database
13.1. Classifying log messages
13.1.1. The structure of the pattern database
13.1.2. How pattern matching works
13.1.3. Artificial ignorance
13.2. Using pattern databases
13.2.1. Using parser results in filters and templates
13.2.2. Downloading sample pattern databases
13.3. Correlating log messages
13.3.1. Referencing earlier messages of the context
13.4. Triggering actions for identified messages
13.4.1. Conditional actions
13.4.2. External actions
13.4.3. Actions and message correlation
13.5. Creating pattern databases
13.5.1. Using pattern parsers
13.5.2. What's new in the syslog-ng pattern database format V4
13.5.3. The syslog-ng pattern database format
14. Statistics of syslog-ng
15. Multithreading and scaling in syslog-ng OSE
15.1. Multithreading concepts of syslog-ng OSE
15.2. Configuring multithreading
15.3. Optimizing multithreaded performance
16. Troubleshooting syslog-ng
16.1. Possible causes of losing log messages
16.2. Creating syslog-ng core files
16.3. Collecting debugging information with strace, truss, or tusc
16.4. Running a failure script
16.5. Stopping syslog-ng
17. Best practices and examples
17.1. General recommendations
17.2. Handling lots of parallel connections
17.3. Handling large message load
17.4. Using name resolution in syslog-ng
17.4.1. Resolving hostnames locally
17.5. Collecting logs from chroot
A. The syslog-ng manual pages
loggen — Generate syslog messages at a specified rate
pdbtool — An application to test and convert syslog-ng pattern database rules
syslog-ng — syslog-ng system logger application
syslog-ng.conf — syslog-ng configuration file
syslog-ng-ctl — Display message statistics and enable verbose, debug and trace modes in syslog-ng Open Source Edition
B. GNU General Public License
B.1. Preamble
B.2. TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
B.2.1. Section 0
B.2.2. Section 1
B.2.3. Section 2
B.2.4. Section 3
B.2.5. Section 4
B.2.6. Section 5
B.2.7. Section 6
B.2.8. Section 7
B.2.9. Section 8
B.2.10. Section 9
B.2.11. Section 10
B.2.12. NO WARRANTY Section 11
B.2.13. Section 12
B.3. How to Apply These Terms to Your New Programs
C. GNU Lesser General Public License
C.1. Preamble
C.2. TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
C.2.1. Section 0
C.2.2. Section 1
C.2.3. Section 2
C.2.4. Section 3
C.2.5. Section 4
C.2.6. Section 5
C.2.7. Section 6
C.2.8. Section 7
C.2.9. Section 8
C.2.10. Section 9
C.2.11. Section 10
C.2.12. Section 11
C.2.13. Section 12
C.2.14. Section 13
C.2.15. Section 14
C.2.16. NO WARRANTY Section 15
C.2.17. Section 16
C.3. How to Apply These Terms to Your New Libraries
D. Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License
Glossary
Index
List of syslog-ng OSE parameters

List of Examples

2.1. Using the value-pairs() option
2.2. Using the rekey() option
4.1. The default configuration file of syslog-ng OSE
4.2. A simple configuration for clients
4.3. A simple configuration for servers
4.4. A simple configuration for relays
5.1. A simple configuration file
5.2. Using required and optional parameters
5.3. Using inline definitions
5.4. Using channels
5.5. Using global variables
5.6. Reusing configuration blocks
5.7. Defining blocks with multiple elements
5.8. Passing arguments to blocks
5.9. Using arguments in blocks
6.1. A simple source statement
6.2. A source statement using two source drivers
6.3. Setting default priority and facility
6.4. Source statement on a Linux based operating system
6.5. Using the internal() driver
6.6. Using the file() driver
6.7. Tailing files
6.8. Initial window size of a connection
6.9. Using the network() driver
6.10. Initial window size of a connection
6.11. Using the pipe() driver
6.12. Initial window size of a connection
6.13. Using the program() driver
6.14. Initial window size of a connection
6.15. Using the sun-streams() driver
6.16. Initial window size of a connection
6.17. Using the syslog() driver
6.18. Initial window size of a connection
6.19. Using the udp() and tcp() drivers
6.20. Initial window size of a connection
6.21. Using the unix-stream() and unix-dgram() drivers
6.22. Initial window size of a connection
7.1. A simple destination statement
7.2. Using the amqp() driver
7.3. Using the file() driver
7.4. Using the file() driver with macros in the file name and a template for the message
7.5. Using the mongodb() driver
7.6. Using the network() driver
7.7. Using the pipe() driver
7.8. Using the program() destination driver
7.9. Using the smtp() driver
7.10. Simple e-mail alerting with the smtp() driver
7.11. Using the sql() driver
7.12. Using the sql() driver with an Oracle database
7.13. Using the sql() driver with an MSSQL database
7.14. Setting flags for SQL destinations
7.15. Using SQL NULL values
7.16. Value: default
7.17. Using the syslog() driver
7.18. Using the tcp() driver
7.19. Using the unix-stream() driver
7.20. Using the usertty() driver
8.1. A simple log statement
8.2. Using embedded log paths
8.3. Using junctions
8.4. Using log path flags
8.5. Soft flow-control
8.6. Hard flow-control
8.7. Sizing parameters for flow-control
8.8. A simple filter statement
8.9. Comparing macro values in filters
8.10. Filtering with widcards
8.11. Adding tags and filtering messages with tags
8.12. Skipping messages
9.1. Using global options
10.1. A destination statement using TLS
10.2. A source statement using TLS
10.3. Disabling mutual authentication
10.4. A destination statement using mutual authentication
10.5. A source statement using TLS
11.1. Using templates and macros
11.2. Using SDATA macros
11.3. Using the format-json template function
11.4. Using the grep template function
11.5. Using pattern databases and the if template function
11.6. Using the indent-multi-line template function
11.7. Using the sanitize template function
11.8. Using the substr template function
11.9. Using the $(hash) template function
11.10. Using Universally Unique Identifiers
11.11. Using substitution rules
11.12. Setting message fields to a particular value
11.13. Rewriting custom SDATA fields
11.14. Using conditional rewriting
11.15. Using Posix regular expressions
11.16. Using PCRE regular expressions
11.17. Optimizing regular expressions in filters
12.1. Using junctions
12.2. Segmenting hostnames separated with a dash
12.3. Parsing Apache log files
12.4. Segmenting a part of a message
12.5. Adding the end of the message to the last column
12.6. Using a JSON parser
12.7. Using the marker option in JSON parser
13.1. Defining pattern databases
13.2. Using classification results
13.3. Using classification results for filtering messages
13.4. Using pattern parsers as macros
13.5. How syslog-ng OSE calculates context-timeout
13.6. Using message correlation
13.7. Sending triggered messages to the internal() source
13.8. Generating messages for pattern database matches
13.9. Generating messages with inherited values
13.10. Actions based on the number of messages
13.11. Sending triggered messages to external applications
13.12. Pattern parser syntax
13.13. Using the STRING and ESTRING parsers
13.14. A V4 pattern database containing a single rule
15.1. Enabling multithreading
A.1. Using required and optional parameters
A.2. Using global options

List of Procedures

2.2.1. The route of a log message in syslog-ng
3.1. Compiling syslog-ng from source
3.3. Configuring Microsoft SQL Server to accept logs from syslog-ng
4.1. Configuring syslog-ng on client hosts
4.2. Configuring syslog-ng on server hosts
4.3.1. Configuring syslog-ng on relay hosts
10.2.1. Configuring TLS on the syslog-ng clients
10.2.2. Configuring TLS on the syslog-ng server
10.3.1. Configuring TLS on the syslog-ng clients
10.3.2. Configuring TLS on the syslog-ng server
16.2. Creating syslog-ng core files
17.4.1. Resolving hostnames locally
17.5. Collecting logs from chroot

Preface

Welcome to the syslog-ng Open Source Edition 3.4 Administrator Guide!

This document describes how to configure and manage syslog-ng. Background information for the technology and concepts used by the product is also discussed.

1. Summary of contents

Chapter 1, Introduction to syslog-ng describes the main functionality and purpose of syslog-ng OSE.

Chapter 2, The concepts of syslog-ng discusses the technical concepts and philosophies behind syslog-ng OSE.

Chapter 3, Installing syslog-ng describes how to install syslog-ng OSE on various UNIX-based platforms using the precompiled binaries.

Chapter 4, The syslog-ng OSE quick-start guide provides a briefly explains how to perform the most common log collecting tasks with syslog-ng OSE.

Chapter 5, The syslog-ng OSE configuration file discusses the configuration file format and syntax in detail, and explains how to manage large-scale configurations using included files and reusable configuration snippets.

Chapter 6, Collecting log messages — sources and source drivers explains how to collect and receive log messages from various sources.

Chapter 7, Sending and storing log messages — destinations and destination drivers describes the different methods to store and forward log messages.

Chapter 8, Routing messages: log paths and filters explains how to route and sort log messages, and how to use filters to select specific messages.

Chapter 9, Global options of syslog-ng OSE lists the global options of syslog-ng OSE and explains how to use them.

Chapter 10, TLS-encrypted message transfer shows how to secure and authenticate log transport using TLS encryption.

Chapter 11, Manipulating messages describes how to customize message format using templates and macros, how to rewrite and modify messages, and how to use regular expressions.

Chapter 12, Parsing and segmenting structured messages describes how to segment and process structured messages like comma-separated values.

Chapter 13, Processing message content with a pattern database explains how to identify and process log messages using a pattern database.

Chapter 14, Statistics of syslog-ng details the available statistics that syslog-ng OSE collects about the processed log messages.

Chapter 15, Multithreading and scaling in syslog-ng OSE describes how to configure syslog-ng OSE to use multiple processors, and how to optimize its performance.

Chapter 16, Troubleshooting syslog-ng offers tips to solving problems.

Chapter 17, Best practices and examples gives recommendations to configure special features of syslog-ng.

Appendix A, The syslog-ng manual pages contains the manual pages of the syslog-ng OSE application.

Appendix C, GNU Lesser General Public License includes the text of the LGPLv2.1 license applicable to the core of syslog-ng Open Source Edition.

Appendix B, GNU General Public License includes the text of the GPLv2 license applicable to syslog-ng Open Source Edition.

Appendix D, Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License includes the text of the Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License applicable to The syslog-ng Open Source Edition 3.4 Administrator Guide.

Glossary provides definitions of important terms used in this guide.

List of syslog-ng OSE parameters provides cross-references to the definitions of options, parameters, and macros available in syslog-ng OSE.

The Index provides cross-references to important terms used in this guide.

2. Target audience and prerequisites

This guide is intended for system administrators and consultants responsible for designing and maintaining logging solutions and log centers. It is also useful for IT decision makers looking for a tool to implement centralized logging in heterogeneous environments.

The following skills and knowledge are necessary for a successful syslog-ng administrator:

  • At least basic system administration knowledge.

  • An understanding of networks, TCP/IP protocols, and general network terminology.

  • Working knowledge of the UNIX or Linux operating system.

  • In-depth knowledge of the logging process of various platforms and applications.

  • An understanding of the legacy syslog (BSD-syslog) protocol) and the new syslog (IETF-syslog) protocol) standard.

3. Products covered in this guide

This guide describes the use of the following products:

  • syslog-ng Open Source Edition (syslog-ng OSE) 3.4.1 and later

4. Typographical conventions

Before you start using this guide, it is important to understand the terms and typographical conventions used in the documentation. For more information on specialized terms and abbreviations used in the documentation, see theGlossary at the end of this document.

The following kinds of text formatting and icons identify special information in the document.

Tip

Tips provide best practices and recommendations.

Note

Notes provide additional information on a topic, and emphasize important facts and considerations.

Warning

Warnings mark situations where loss of data or misconfiguration of the device is possible if the instructions are not obeyed.

Command

Commands you have to execute.

Emphasis

Reference items, additional readings.

/path/to/file

File names.

Parameters

Parameter and attribute names.

Label

GUI output messages or dialog labels.

Menu

A submenu or menu item in the menu bar.

Button

Buttons in dialog windows.

5. Contact and support information

This product is developed and maintained by BalaBit IT Security Ltd. We are located in Budapest, Hungary. Our address is:

Balabit SA2 Alíz StreetH-1117BudapestHungaryTel: +36 1 398-6700Fax: +36 1 208-0875
         E-mail: 
         Web: http://balasys.hu/
       

5.1. Sales contact

You can directly contact us with sales related topics at the e-mail address , or leave us your contact information and we call you back.

5.2. Support contact

In case you experience a problem that is not covered in this guide, visit the syslog-ng wiki or post it on syslog-ng mailing list.

To report bugs found in syslog-ng OSE, visit this page.

Precompiled binary packages are available for free from various third-parties. Most of these are listed on the syslog-ng website.

Note

Balabit offers a Premium Edition of syslog-ng and a log management appliance syslog-ng Store Box for which professional support is available. For more information on which syslog-ng product version meets your needs contact our sales department at.

For other products of Balabit visit http://www.balabit.com/.

5.3. Training

Balabit SA holds courses on using its products for new and experienced users. For dates, details, and application forms, visit the https://my.balabit.com/training webpage.

6. About this document

This guide is a work-in-progress document with new versions appearing periodically.

The latest version of this document can be downloaded from the BalaBit website here.

6.1. Summary of changes

This section lists the changes of The syslog-ng Open Source Edition Administrator Guide.

6.1.1. Version 3.3 - 3.4

Changes in product: 

Changes in documentation: 

6.1.2. Version 3.2 - 3.3

Changes in product: 

Changes in documentation: 

6.2. Feedback

Any feedback is greatly appreciated, especially on what else this document should cover. General comments, errors found in the text, and any suggestions about how to improve the documentation is welcome at documentation@balabit.com.

6.3. Acknowledgments

BalaBit would like to express its gratitude to the syslog-ng users and the syslog-ng community for their invaluable help and support, including the community members listed at syslog-ng Community Page.

Chapter 1. Introduction to syslog-ng

This chapter introduces the syslog-ng Open Source Edition application in a non-technical manner, discussing how and why is it useful, and the benefits it offers to an existing IT infrastructure.

1.1. What syslog-ng is

The syslog-ng application is a flexible and highly scalable system logging application that is ideal for creating centralized and trusted logging solutions. The main features of syslog-ng are summarized below.

  • Reliable log transfer: The syslog-ng application enables you to send the log messages of your hosts to remote servers using the latest protocol standards. The logs of different servers can be collected and stored centrally on dedicated log servers. Transferring log messages using the TCP protocol ensures that no messages are lost.

  • Secure logging using TLS: Log messages may contain sensitive information that should not be accessed by third parties. Therefore, syslog-ng supports the Transport Layer Security (TLS) protocol to encrypt the communication. TLS also allows the mutual authentication of the host and the server using X.509 certificates.

  • Direct database access: Storing your log messages in a database allows you to easily search and query the messages and interoperate with log analyzing applications. The syslog-ng application supports the following databases: MSSQL, MySQL, Oracle, PostgreSQL, and SQLite.

  • Heterogeneous environments: The syslog-ng application is the ideal choice to collect logs in massively heterogeneous environments using several different operating systems and hardware platforms, including Linux, Unix, BSD, Sun Solaris, HP-UX, Tru64, and AIX.

  • Filter and classify: The syslog-ng application can sort the incoming log messages based on their content and various parameters like the source host, application, and priority. Directories, files, and database tables can be created dynamically using macros. Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important log messages to the selected destinations.

  • Parse and rewrite: The syslog-ng application can segment log messages to named fields or columns, and also modify the values of these fields.

  • IPv4 and IPv6 support: The syslog-ng application can operate in both IPv4 and IPv6 network environments; it can receive and send messages to both types of networks.

1.2. What syslog-ng is not

The syslog-ng application is not log analysis software. It can filter log messages and select only the ones matching certain criteria. It can even convert the messages and restructure them to a predefined format, or parse the messages and segment them into different fields. But syslog-ng cannot interpret and analyze the meaning behind the messages, or recognize patterns in the occurrence of different messages.

1.3. Why is syslog-ng needed?

Log messages contain information about the events happening on the hosts. Monitoring system events is essential for security and system health monitoring reasons.

The original syslog protocol separates messages based on the priority of the message and the facility sending the message. These two parameters alone are often inadequate to consistently classify messages, as many applications might use the same facility — and the facility itself is not even included in the log message. To make things worse, many log messages contain unimportant information. The syslog-ng application helps you to select only the really interesting messages, and forward them to a central server.

Company policies or other regulations often require log messages to be archived. Storing the important messages in a central location greatly simplifies this process.

For details on how can you use syslog-ng to comply with various regulations, see the Regulatory compliance and system logging whitepaper available here

1.4. What is new in syslog-ng Open Source Edition 3.4?

Version 3.4 of syslog-ng Open Source Edition includes the following main features:

1.5. Who uses syslog-ng?

The syslog-ng application is used worldwide by companies and institutions who collect and manage the logs of several hosts, and want to store them in a centralized, organized way. Using syslog-ng is particularly advantageous for:

  • Internet Service Providers;

  • Financial institutions and companies requiring policy compliance;

  • Server, web, and application hosting companies;

  • Datacenters;

  • Wide area network (WAN) operators;

  • Server farm administrators.

The following is a list of public references — companies who use syslog-ng in their production environment:

1.6. Supported platforms

The syslog-ng Open Source Edition application is highly portable and is known to run on a wide range of hardware architectures (x86, x86_64, SUN Sparc, PowerPC 32 and 64, Alpha) and operating systems, including Linux, BSD, Solaris, IBM AIX, HP-UX, Mac OS X, Cygwin, Tru64, and others.

Chapter 2. The concepts of syslog-ng

This chapter discusses the technical concepts of syslog-ng.

2.1. The philosophy of syslog-ng

Typically, syslog-ng is used to manage log messages and implement centralized logging, where the aim is to collect the log messages of several devices on a single, central log server. The different devices — called syslog-ng clients — all run syslog-ng, and collect the log messages from the various applications, files, and other sources. The clients send all important log messages to the remote syslog-ng server, where the server sorts and stores them.

2.2. Logging with syslog-ng

The syslog-ng application reads incoming messages and forwards them to the selected destinations. The syslog-ng application can receive messages from files, remote hosts, and other sources.

Log messages enter syslog-ng in one of the defined sources, and are sent to one or more destinations.

Sources and destinations are independent objects; log paths define what syslog-ng does with a message, connecting the sources to the destinations. A log path consists of one or more sources and one or more destinations; messages arriving from a source are sent to every destination listed in the log path. A log path defined in syslog-ng is called a log statement.

Optionally, log paths can include filters. Filters are rules that select only certain messages, for example, selecting only messages sent by a specific application. If a log path includes filters, syslog-ng sends only the messages satisfying the filter rules to the destinations set in the log path.

Other optional elements that can appear in log statements are parsers and rewriting rules. Parsers segment messages into different fields to help processing the messages, while rewrite rules modify the messages by adding, replacing, or removing parts of the messages.

2.2.1. Procedure – The route of a log message in syslog-ng

Purpose: 

The following procedure illustrates the route of a log message from its source on the syslog-ng client to its final destination on the central syslog-ng server.

The route of a log message

Figure 2.1. The route of a log message


Steps: 

  1. A device or application sends a log message to a source on the syslog-ng client. For example, an Apache web server running on Linux enters a message into the /var/log/apache file.

  2. The syslog-ng client running on the web server reads the message from its /var/log/apache source.

  3. The syslog-ng client processes the first log statement that includes the /var/log/apache source.

  4. The syslog-ng client performs optional operations (message filtering, parsing, and rewriting) on the message; for example, it compares the message to the filters of the log statement (if any). If the message complies with all filter rules, syslog-ng sends the message to the destinations set in the log statement, for example, to the remote syslog-ng server.

    Warning

    Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.

    Note

    The syslog-ng client sends a message to all matching destinations by default. As a result, a message may be sent to a destination more than once, if the destination is used in multiple log statements. To prevent such situations, use the final flag in the destination statements. For details, see Table 8.1, Log statement flags.

  5. The syslog-ng client processes the next log statement that includes the /var/log/apache source, repeating Steps 3-4.

  6. The message sent by the syslog-ng client arrives from a source set in the syslog-ng server.

  7. The syslog-ng server reads the message from its source and processes the first log statement that includes that source.

  8. The syslog-ng server performs optional operations (message filtering, parsing, and rewriting) on the message; for example, it compares the message to the filters of the log statement (if any). If the message complies with all filter rules, syslog-ng sends the message to the destinations set in the log statement.

    Warning

    Message filtering, parsing, and rewriting is performed in the order that the operations appear in the log statement.

  9. The syslog-ng server processes the next log statement, repeating Steps 7-9.

    Note

    The syslog-ng application can stop reading messages from its sources if the destinations cannot process the sent messages. This feature is called flow-control and is detailed in Section 8.2, Managing incoming and outgoing messages with flow-control.

2.3. Modes of operation

The syslog-ng Open Source Edition application has three typical operation scenarios: Client, Server, and Relay.

2.3.1. Client mode

Client-mode operation

Figure 2.2. Client-mode operation


In client mode, syslog-ng collects the local logs generated by the host and forwards them through a network connection to the central syslog-ng server or to a relay. Clients often also log the messages locally into files.

2.3.2. Relay mode

Relay-mode operation

Figure 2.3. Relay-mode operation


In relay mode, syslog-ng receives logs through the network from syslog-ng clients and forwards them to the central syslog-ng server using a network connection. Relays also log the messages from the relay host into a local file, or forward these messages to the central syslog-ng server.

2.3.3. Server mode

Server-mode operation

Figure 2.4. Server-mode operation


In server mode, syslog-ng acts as a central log-collecting server. It receives messages from syslog-ng clients and relays over the network, and stores them locally in files, or passes them to other applications, for example log analyzers.

2.4. Global objects

The syslog-ng application uses the following objects:

  • Source driver: A communication method used to receive log messages. For example, syslog-ng can receive messages from a remote host via TCP/IP, or read the messages of a local application from a file. For details on source drivers, see Chapter 6, Collecting log messages — sources and source drivers.

  • Source: A named collection of configured source drivers.

  • Destination driver: A communication method used to send log messages. For example, syslog-ng can send messages to a remote host via TCP/IP, or write the messages into a file or database. For details on destination drivers, see Chapter 7, Sending and storing log messages — destinations and destination drivers.

  • Destination: A named collection of configured destination drivers.

  • Filter: An expression to select messages. For example, a simple filter can select the messages received from a specific host. For details, see Section 11.1, Customizing message format.

  • Macro: An identifier that refers to a part of the log message. For example, the ${HOST} macro returns the name of the host that sent the message. Macros are often used in templates and filenames. For details, see Section 11.1, Customizing message format.

  • Parser: Parsers are objects that parse the incoming messages, or parts of a message. For example, the csv-parser() can segment messages into separate columns at a predefined separator character (for example a comma). Every column has a unique name that can be used as a macro. For details, see Chapter 12, Parsing and segmenting structured messages and Chapter 13, Processing message content with a pattern database.

  • Rewrite rule: A rule modifies a part of the message, for example, replaces a string, or sets a field to a specified value. For details, see Section 11.2, Modifying messages.

  • Log paths: A combination of sources, destinations, and other objects like filters, parsers, and rewrite rules. The syslog-ng application sends messages arriving from the sources of the log paths to the defined destinations, and performs filtering, parsing, and rewriting of the messages. Log paths are also called log statements. Log statements can include other (embedded) log statements and junctions to create complex log paths. For details, see Chapter 8, Routing messages: log paths and filters.

  • Template: A template is a set of macros that can be used to restructure log messages or automatically generate file names. For example, a template can add the hostname and the date to the beginning of every log message. For details, see Section 11.1, Customizing message format.

  • Option: Options set global parameters of syslog-ng, like the parameters of name resolution and timezone handling. For details, see Chapter 9, Global options of syslog-ng OSE.

For details on the above objects, see Section 5.2, The configuration syntax in detail.

2.5. Timezones and daylight saving

The syslog-ng application receives the timezone and daylight saving information from the operating system it is installed on. If the operating system handles daylight saving correctly, so does syslog-ng.

The syslog-ng application supports messages originating from different timezones. The original syslog protocol (RFC3164) does not include timezone information, but syslog-ng provides a solution by extending the syslog protocol to include the timezone in the log messages. The syslog-ng application also enables administrators to supply timezone information for legacy devices which do not support the protocol extension.

Timezone information is associated with messages entering syslog-ng is selected using the following algorithm:

  1. The sender application (for example the syslog-ng client) or host specifies the timezone of the messages. If the incoming message includes a timezone it is associated with the message. Otherwise, the local timezone is assumed.

  2. Specify the time_zone() parameter for the source driver that reads the message. This timezone will be associated with the messages only if no timezone is specified within the message itself. Each source defaults to the value of the recv_time_zone() global option. It is not possible to override only the timezone information of the incoming message; but setting the keep-timestamp() option to no allows syslog-ng OSE to replace the full timestamp (timezone included) with the time the message was received.

    Note

    When processing a message that does not contain timezone information, the syslog-ng OSE application will use the timezone and daylight-saving that was effective when the timestamp was generated. For example, the current time is 2011-03-11 (March 11, 2011) in the EU/Budapest timezone. When daylight-saving is active (summertime), the offset is +02:00. When daylight-saving is inactive (wintertime) the timezone offset is +01:00. If the timestamp of an incoming message is 2011-01-01, the timezone associated with the message will be +01:00, but the timestamp will be converted, because 2011-01-01 meant winter time when daylight saving is not active but the current timezone is +02:00.

  3. Specify the timezone in the destination driver using the time_zone() parameter. Each destination driver might have an associated timezone value; syslog-ng converts message timestamps to this timezone before sending the message to its destination (file or network socket). Each destination defaults to the value of the send_time_zone() global option.

    Note

    A message can be sent to multiple destination zones. The syslog-ng application converts the timezone information properly for every individual destination zone.

    Warning

    If syslog-ng OSE sends the message is to the destination using the legacy-syslog protocol (RFC3164) which does not support timezone information in its timestamps, the timezone information cannot be encapsulated into the sent timestamp, so syslog-ng OSE will convert the hour:min values based on the explicitly specified timezone.

  4. If the timezone is not specified, the message is left unchanged.

  5. When macro expansions are used in the destination filenames, the local timezone is used.

2.5.1. A note on timezones and timestamps

If the clients run syslog-ng, then use the ISO timestamp, because it includes timezone information. That way you do not need to adjust the recv_time_zone() parameter of syslog-ng.

If you want syslog-ng to output timestamps in Unix (POSIX) time format, use the S_UNIXTIME and R_UNIXTIME macros. You do not need to change any of the timezone related parameters, because the timestamp information of incoming messages is converted to Unix time internally, and Unix time is a timezone-independent time representation. (Actually, Unix time measures the number of seconds elapsed since midnight of Coordinated Universal Time (UTC) January 1, 1970, but does not count leap seconds.)

2.6. The license of syslog-ng OSE

Starting with version 3.2, the syslog-ng Open Source Edition application is licensed under a combined LGPL+GPL license. The core of syslog-ng OSE is licensed under the GNU Lesser General Public License Version 2.1 license, while the rest of the codebase is licensed under the GNU General Public License Version 2 license.

Note

Practically, the code stored under the lib directory of the source code package is under LGPL, the rest is GPL.

For details about the LGPL and GPL licenses, see Appendix C, GNU Lesser General Public License and Appendix B, GNU General Public License, respectively.

2.7. High availability support

Multiple syslog-ng servers can be run in fail-over mode. The syslog-ng application does not include any internal support for this, as clustering support must be implemented on the operating system level. A tool that can be used to create UNIX clusters is Heartbeat (for details, see this page).

2.8. The structure of a log message

The following sections describe the structure of log messages. Currently there are two standard syslog message formats:

2.8.1. BSD-syslog or legacy-syslog messages

This section describes the format of a syslog message, according to the legacy-syslog or BSD-syslog protocol). A syslog message consists of the following parts:

The total message cannot be longer than 1024 bytes.

The following is a sample syslog message: <133>Feb 25 14:09:07 webserver syslogd: restart. The message corresponds to the following format: <priority>timestamp hostname application: message. The different parts of the message are explained in the following sections.

Note

The syslog-ng application supports longer messages as well. For details, see the log_msg_size() option in Section 9.2, Global options. However, it is not recommended to enable messages larger than the packet size when using UDP destinations.

2.8.1.1. The PRI message part

The PRI part of the syslog message (known as Priority value) represents the Facility and Severity of the message. Facility represents the part of the system sending the message, while severity marks its importance. The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity. The possible facility and severity values are presented below.

Note

Facility codes may slightly vary between different platforms. The syslog-ng application accepts facility codes as numerical values as well.

Numerical Code Facility
0 kernel messages
1 user-level messages
2 mail system
3 system daemons
4 security/authorization messages
5 messages generated internally by syslogd
6 line printer subsystem
7 network news subsystem
8 UUCP subsystem
9 clock daemon
10 security/authorization messages
11 FTP daemon
12 NTP subsystem
13 log audit
14 log alert
15 clock daemon
16-23 locally used facilities (local0-local7)

Table 2.1. syslog Message Facilities


The following table lists the severity values.

Numerical Code Severity
0 Emergency: system is unusable
1 Alert: action must be taken immediately
2 Critical: critical conditions
3 Error: error conditions
4 Warning: warning conditions
5 Notice: normal but significant condition
6 Informational: informational messages
7 Debug: debug-level messages

Table 2.2. syslog Message Severities


2.8.1.2. The HEADER message part

The HEADER part contains a timestamp and the hostname (without the domain name) or the IP address of the device. The timestamp field is the local time in the Mmm dd hh:mm:ss format, where:

  • Mmm is the English abbreviation of the month: Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec.

  • dd is the day of the month on two digits. If the day of the month is less than 10, the first digit is replaced with a space. (For example Aug 7.)

  • hh:mm:ss is the local time. The hour (hh) is represented in a 24-hour format. Valid entries are between 00 and 23, inclusive. The minute (mm) and second (ss) entries are between 00 and 59 inclusive.

Note

The syslog-ng application supports other timestamp formats as well, like ISO, or the PIX extended format. For details, see the ts_format() option in Section 9.2, Global options.

2.8.1.3. The MSG message part

The MSG part contains the name of the program or process that generated the message, and the text of the message itself. The MSG part is usually in the following format: program[pid]: message text.

2.8.2. IETF-syslog messages

This section describes the format of a syslog message, according to the IETF-syslog protocol). A syslog message consists of the following parts:

The following is a sample syslog message:[1]

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

The message corresponds to the following format:

<priority>VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA MSG

In this example, the Facility has the value of 4, severity is 2, so PRI is 34. The VERSION is 1. The message was created on 11 October 2003 at 10:14:15pm UTC, 3 milliseconds into the next second. The message originated from a host that identifies itself as "mymachine.example.com". The APP-NAME is "su" and the PROCID is unknown. The MSGID is "ID47". The MSG is "'su root' failed for lonvick...", encoded in UTF-8. The encoding is defined by theBOM[2]. There is no STRUCTURED-DATA present in the message, this is indicated by "-" in the STRUCTURED-DATA field. The MSG is "'su root' failed for lonvick...".

The HEADER part of the message must be in plain ASCII format, the parameter values of the STRUCTURED-DATA part must be in UTF-8, while the MSG part should be in UTF-8. The different parts of the message are explained in the following sections.

2.8.2.1. The PRI message part

The PRI part of the syslog message (known as Priority value) represents the Facility and Severity of the message. Facility represents the part of the system sending the message, while severity marks its importance. The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity. The possible facility and severity values are presented below.

Note

Facility codes may slightly vary between different platforms. The syslog-ng application accepts facility codes as numerical values as well.

Numerical Code Facility
0 kernel messages
1 user-level messages
2 mail system
3 system daemons
4 security/authorization messages
5 messages generated internally by syslogd
6 line printer subsystem
7 network news subsystem
8 UUCP subsystem
9 clock daemon
10 security/authorization messages
11 FTP daemon
12 NTP subsystem
13 log audit
14 log alert
15 clock daemon
16-23 locally used facilities (local0-local7)

Table 2.3. syslog Message Facilities


The following table lists the severity values.

Numerical Code Severity
0 Emergency: system is unusable
1 Alert: action must be taken immediately
2 Critical: critical conditions
3 Error: error conditions
4 Warning: warning conditions
5 Notice: normal but significant condition
6 Informational: informational messages
7 Debug: debug-level messages

Table 2.4. syslog Message Severities


2.8.2.2. The HEADER message part

The HEADER part contains the following elements:

  • VERSION: Version number of the syslog protocol standard. Currently this can only be 1.

  • ISOTIMESTAMP: The time when the message was generated in the ISO 8601 compatible standard timestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example: 2006-06-13T15:58:00.123+01:00.

  • HOSTNAME: The machine that originally sent the message.

  • APPLICATION: The device or application that generated the message

  • PID: The process name or process ID of the syslog application that sent the message. It is not necessarily the process ID of the application that generated the message.

  • MESSAGEID: The ID number of the message.

Note

The syslog-ng application supports other timestamp formats as well, like ISO, or the PIX extended format. The timestamp used in the IETF-syslog protocol is derived from RFC3339, which is based on ISO8601. For details, see the ts_format() option in Section 9.2, Global options.

The syslog-ng OSE application will truncate the following fields:

  • If APP-NAME is longer than 48 characters it will be truncated to 48 characters.

  • If PROC-ID is longer than 128 characters it will be truncated to 128 characters.

  • If MSGID is longer than 32 characters it will be truncated to 32 characters.

  • If HOSTNAME is longer than 255 characters it will be truncated to 255 characters.

2.8.2.3. The STRUCTURED-DATA message part

The STRUCTURED-DATA message part may contain meta- information about the syslog message, or application-specific information such as traffic counters or IP addresses. STRUCTURED-DATA consists of data blocks enclosed in brackets ([]). Every block includes the ID of the block, and one or more name=value pairs. The syslog-ng application automatically parses the STRUCTURED-DATA part of syslog messages, which can be referenced in macros (for details, see Section 11.1.5, Macros of syslog-ng OSE). An example STRUCTURED-DATA block looks like:

[exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"]

2.8.2.4. The MSG message part

The MSG part contains the text of the message itself. The encoding of the text must be UTF-8 if theBOM[3]character is present in the message. If the message does not contain the BOM character, the encoding is treated as unknown. Usually messages arriving from legacy sources do not include the BOM character. CRLF characters will not be removed from the message.



[1] Source: http://tools.ietf.org/html/rfc5424

[2] The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

[3] The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

2.9. Message representation in syslog-ng OSE

When the syslog-ng OSE application receives a message, it automatically parses the message. The syslog-ng OSE application can automatically parse log messages that conform to the RFC3164 (BSD or legacy-syslog) or the RFC5424 (IETF-syslog) message formats. If syslog-ng OSE cannot parse a message, it results in an error.

Tip

In case you need to relay messages that cannot be parsed without any modifications or changes, use the flags(no-parse) option in the source definition, and a template containing only the ${MSG} macro in the destination definition.

A parsed message has the following parts.

  • Timestamps. Two timestamps are associated with every message: one is the timestamp contained within the message (that is, when the sender sent the message), the other is the time when syslog-ng OSE has actually received the message.

  • Severity. The severity of the message.

  • Facility. The facility that sent the message.

  • Tags. Custom text labels added to the message that are mainly used for filtering. None of the current message transport protocols adds tags to the log messages. Tags can be added to the log message only within syslog-ng OSE. The syslog-ng OSE application automatically adds the id of the source as a tag to the incoming messages. Other tags can be added to the message by the pattern database, or using the tags() option of the source.

  • IP address of the sender. The IP address of the host that sent the message. Note that the IP address of the sender is a hard macro and cannot be modified within syslog-ng OSE but the associated hostname can be modified, for example, using rewrite rules.

  • Hard macros. Hard macros contain data that is directly derived from the log message, for example, the ${MONTH} macro derives its value from the timestamp. The most important consideration with hard macros is that they are read-only, meaning they cannot be modified using rewrite rules or other means.

  • Soft macros. Soft macros (sometimes also called name-value pairs) are either built-in macros automatically generated from the log message (for example, ${HOST}), or custom user-created macros generated by using the syslog-ng pattern database or a CSV-parser. The SDATA fields of RFC5424-formatted log messages become soft macros as well. In contrast with hard macros, soft macros are writable and can be modified within syslog-ng OSE, for example, using rewrite rules.

    Note

    It is also possible to set the value of built-in soft macros using parsers, for example, to set the ${HOST} macro from the message using a column of a CSV-parser.

    The data extracted from the log messages using named pattern parsers in the pattern database are also soft macros.

    Tip

    For the list of hard and soft macros, see Section 11.1.4, Hard vs. soft macros.

2.10. Structuring macros, metadata, and other value-pairs

Available in syslog-ng OSE 3.3 and later.

The syslog-ng OSE application allows you to select and construct name-value pairs from any information already available about the log message, or extracted from the message itself. This structured information can be used directly in a MongoDB database (see the value-pairs() option of the mongodb() destination in Section 7.3, Storing messages in a MongoDB database), or in other destinations using the format-json() template function (see Section format-json for details).

When using value-pairs, there are three ways to specify which information (that is, macros or other name-value pairs) to include in the selection.

  • Select groups of macros using the scope() parameter, and optionally remove certain macros from the group using the exclude() parameter.

  • List specific macros to include using the key() parameter.

  • Define new name-value pairs to include using the pair() parameter.

These parameters are detailed in Section value-pairs().

value-pairs()

Type: parameter list of the value-pairs() option
Default:
empty string

Description:The value-pairs() option allows you to select specific information about a message easily using predefined macro groups. The selected information is represented as name-value pairs and can be used formatted to JSON format, or directly used in a mongodb() destination.

Example 2.1. Using the value-pairs() option

The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message.

value-pairs(
    scope(nv_pairs core syslog all_macros selected_macros everything)
    exclude("R_*")
    exclude("S_*")
    key(".SDATA.meta.sequenceId")
    pair("MSGHDR" "$PROGRAM[$PID]: ")
)

The following example selects the same information as the previous example, but converts it into JSON format.

$(format-json --scope nv_pairs,core,syslog,all_macros,selected_macros,everything \
  --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
  --pair MSGHDR="$PROGRAM[$PID]: ")
Note

Every macro is included in the selection only once, but redundant information may appear if multiple macros include the same information (for example, including several date-related macros in the selection).

The value-pairs() option has the following parameters. The parameters are evaluated in the following order:

  1. scope()

  2. exclude()

  3. key()

  4. pair()

exclude()  
Type: Space-separated list of macros to remove from the selection created using the scope() option.
Default: empty string

Description:This option removes the specified macros from the selection. Use it to remove unneeded macros selected using the scope() parameter.

For example, the following example removes the SDATA macros from the selection.

value-pairs(
        scope(rfc5424 selected_macros)
        exclude(".SDATA*")
    )

The name of the macro to remove can include glob expressions and wildcards (*, ?, [ab], {foo,bar}), as described in Section glob. Regular expressions are not supported.

key()  
Type: A glob expression specifying the macros to be included in selection
Default: empty string

Description:This option selects the specified macros. The The selected macros will be included as MACRONAME = MACROVALUE, that is using key("HOST") will result in HOST = $HOST. Multiple macros can be selected using glob expressions. For details on globs, see Section glob. For example:

value-pairs(
        scope(rfc3164)
        key("HOST"))
pair()  
Type: name value pairs in "<NAME>" "<VALUE>"
Default: empty string

Description:This option defines a new name-value pair to be included in the message. The value part can include macros, templates, and template functions as well. For example:

value-pairs(
        scope(rfc3164)
        pair("TIME" "$HOUR:$MIN")
        pair("MSGHDR" "$PROGRAM[$PID]: "))
rekey()  
Type: <pattern-to-select-names>, <list of transformations>
Default: empty string

Description:This option allows you to manipulate and modify the name of the value-pairs. You can define transformations, which are are applied to the selected name-value pairs. The first parameter of the rekey() option is a glob pattern that selects the name-value pairs to modify. If you omit the pattern, the transformations are applied to every key of the scope. For details on globs, see Section glob.

  • If rekey() is used within a key() option, the name-value pairs specified in the glob of the key() option are transformed.

  • If rekey() is used outside the key() option, every name-value pair of the scope() is transformed.

The following transformations are available:

add-prefix("<my-prefix>")

Adds the specified prefix to every name. For example, rekey( add-prefix("my-prefix."))

replace("<prefix-to-replace>", "<new-prefix>")

Replaces a substring at the beginning of the key with another string. Only prefixes can be replaced. For example, replace(".class", ",patterndb") changes the beginning tag .class to .patterndb

shift("<number>")

Cuts the specified number of characters from the beginning of the name.

Example 2.2. Using the rekey() option

The following sample selects every value-pair that begins with .cee., deletes this prefix by cutting 4 characters from the names, and adds a new prefix (events.).

value-pairs(
    key(".cee.*"
        rekey(
            shift(4)
            add-prefix("events.")
        )
    )
)

The rekey() option can be used with the format-json template-function as well, using the following syntax:

$(format-json --key .cee.* --rekey --shift 4 --add-prefix events.)
scope()  
Type: space-separated list of macro groups to include in selection
Default: empty string

Description:This option selects predefined groups of macros. The following groups are available:

  • nv-pairs: Every soft macro (name-value pair) associated with the message, except the ones that start with a dot (.) character. Macros starting with a dot character are generated within syslog-ng OSE and are not originally part of the message, therefore are not included in this group.

  • dot-nv-pairs: Every soft macro (name-value pair) associated with the message which starts with a dot (.) character. For example, .classifier.rule_id and .sdata.*. Macros starting with a dot character are generated within syslog-ng OSE and are not originally part of the message.

  • all-nv-pairs: Include every soft macro (name-value pair). Equivalent to using both nv-pairs and dot-nv-pairs.

  • rfc3164: The macros that correspond to the RFC3164 (legacy or BSD-syslog) message format: $FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, and $R_DATE. The rfc3164 group also has the following aliases: core, base. Note that the value of $R_DATE will be listed under the DATE key.

  • rfc5424: The macros that correspond to the RFC5424 (IETF-syslog) message format: $FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, $MSGID, and $R_DATE. The rfc5424 group also has the following alias: syslog-proto. Note that the value of $R_DATE will be listed under the DATE key.

    The rfc5424 group does not contain any metadata about the message, only information that was present in the original message. To include the most commonly used metadata (for example, the $SOURCEIP macro), use the selected-macros group instead.

  • all-macros: Include every hard macro. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).

  • selected-macros: Include the macros of the rfc5424 group and the most commonly used metadata about the log message: the $TAGS, $SOURCEIP, and $SEQNUM macros.

  • sdata: The metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with .SDATA.

  • everything: Include every hard and soft macros. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).

For example:

value-pairs(
        scope(rfc3164 selected-macros))

Chapter 3. Installing syslog-ng

This chapter explains how to install syslog-ng Open Source Edition on various platforms.

  • You can install syslog-ng OSE on many platforms using the package manager and official repositories of the platform. For a list of third-party packages available for various Linux, UNIX, and other platforms, see third-party binaries page.

  • For instructions on compiling syslog-ng Open Source Edition from the source code, see Procedure 3.1, Compiling syslog-ng from source.

3.1. Procedure – Compiling syslog-ng from source

Purpose: 

To compile syslog-ng Open Source Edition (OSE) from the source code, complete the following steps. Alternatively, you can use precompiled binary packages on several platforms. For a list of third-party packages available for various Linux, UNIX, and other platforms, see third-party binaries page.

Steps: 

  1. Download the latest version of syslog-ng OSE from the BalaBit website or from GitHub. The source code is available as a tar.gz archive file.

  2. Download the latest version of the EventLog library here.

  3. Install the following packages that are required to compile syslog-ng. These packages are available for most UNIX/Linux systems. Alternatively, you can also download the sources and compile them.

    • A version of the gcc C compiler that properly supports Thread Local Storage (TLS), for example, version 4.5 (at least version. For building recent gcc versions on Solaris, see http://jblopen.com/node/16.

    • The GNU flex lexical analyser generator, available here.

    • The bison parser generator, available here.

    • The development files of the glib library, available here.

  4. If you want to use the spoof-source function of syslog-ng, install the development files of the libnet library, available here.

  5. If you want to send e-mails using the smtp() destination, install the development files of the libesmtp library. This library is not needed if you use the --disable-smtp compile option.

  6. If you want to use the /etc/hosts.deny and /etc/hosts.allow for TCP access, install the development files of the libwrap (also called TCP-wrappers) library, available here.

  7. Uncompress the eventlog archive using the

    $ tar xvfz eventlog-x.x.x.x.tar.gz

    or the

    $ gunzip -c eventlog-x.x.x.x.tar.gz | tar xvf -

    command. A new directory containing the source code of eventlog will be created.

  8. By default, eventlog creates a file used by the syslog-ng configure script in the /usr/local/lib/pkgconfig directory. Issue the following command to add this directory to your PKG_CONFIG_PATH:

    PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH
  9. Enter the new directory and issue the following commands:

    $ ./configure
    $ make
    $ make install
  10. Uncompress the syslog-ng archive using the

    tar xvfz syslog-ng-x.xx.tar.gz

    or the

    unzip -c syslog-ng-x.xx.tar.gz | tar xvf -

    command. A new directory containing the source code of syslog-ng will be created.

  11. Enter the new directory and issue the following commands:

    $ ./configure
    $ make
    $ make install

    These commands will build syslog-ng using its default options.

    Note
    • On Solaris, use gmake (GNU make) instead of make.

    • To build syslog-ng OSE with less verbose output, use the make V=0 command. This results in shorter, less verbose output, making warnings and other anomalies easier to notice. Note that silent-rules support is only available in recent automake versions.

  12. If needed, use the following options to change how syslog-ng is compiled using the following command syntax:

    $ ./configure --compile-time-option-name
    Note

    You can also use --disable options, to explicitly disable a feature and override autodetection. For example, to disable the TCP-wrapper support, use the --disable-tcp-wrapper option.

    Warning

    Starting with syslog-ng Open Source Edition 3.0.2, default linking mode of syslog-ng is dynamic. This means that syslog-ng might not be able to start up if the /usr directory is on NFS. On platforms where syslog-ng is used as a system logger, the --enable-mixed-linking is preferred.

    • --disable-json Disable JSON support. It also disables json-parser, and the format-json template function. Also, it disables JSON support even if the json-c library is installed and detected (see --enable-json).

    • --disable-smtp Disable SMTP support. By default, SMTP support is enabled if the libesmtp library is detected.

    • --enable-amqp Enable the amqp destination (enabled by default). The source of the RabbitMQ client is included in the source code package of syslog-ng OSE. To use an external client instead, use the --with-librabbitmq-client=system compiling option. For details on using this destination, see Section 7.1, Publishing messages using AMQP.

    • --enable-debug Include debug information.

    • --enable-dynamic-linking Compile syslog-ng as a completely dynamic binary. If not specified syslog-ng uses mixed linking (--enable-mixed-linking): it links dynamically to system libraries and statically to everything else.

    • --enable-geoip Enable GEOIP support, required for the geoip template function (enabled automatically if the libgeoip library is detected).

    • --enable-ipv6 Enable IPv6 support.

    • --enable-json Enables JSON support (enabled automatically if the json-c 0.9 or newer library is installed and detected).

    • --enable-linux-caps Enable support for capabilities on Linux. For details, see syslog-ng(8).

    • --enable-mongodb Enable the mongodb destination (enabled by default). The source of the MongoDB client is included in the source code package of syslog-ng OSE. To use an external MongoDB client instead, use the --with-libmongo-client=system compiling option. For details on using this destination, see Section 7.3, Storing messages in a MongoDB database.

    • --enable-pacct Enable using the pacct() driver to collect process-accounting logs on Linux systems.

    • --enable-pcre Enable using PCRE-type regular expressions. Requires the libpcre library package, available here.

    • --enable-spoof-source Enable spoof_source feature (disabled by default).

    • --enable-sql Enables the sql() destination (enabled automatically if the libdbi library is installed and detected).

    • --enable-ssl Enable SSL support, required for encrypted message transfer, as well as template functions that calculate hashes and UUIDs (enabled automatically if the libopenssl library is detected).

    • --enable-sun-door Enable Sun door support even if not detected (autodetected by default).

    • --enable-sun-streams Enable Sun STREAMS support even if not detected (autodetected by default).

    • --enable-systemd Enable systemd support on Linux platforms (autodetected by default) (enabled automatically if the libsystemd-daemon library is detected).

    • --enable-tcp-wrapper Enable using /etc/hosts.deny and /etc/hosts.allow for TCP access (enabled automatically if the libwrap libraries are detected).

    • --with-embedded-crypto If this option is set, the crypto library is linked directly into libsyslog-ng: the sources of libsyslog-ng-crypto will be appended to the libsyslog-ng sources, and -crypto is not built.

    • --with-ivykis Specifies which ivykis implementation to use (default value: internal). The source of ivykis is included in the source code package of syslog-ng OSE and is used by default. To use an external implementation instead, use the --with-ivykis=system compiling option.

    • --with-libmongo-client Specifies which MongoDB client to use (default value: internal). The source of the mongodb client is included in the source code package of syslog-ng OSE and is used by default. To use an external MongoDB client instead, use the --with-libmongo-client=system compiling option. For details on using this destination, see Section 7.3, Storing messages in a MongoDB database.

    • --with-librabbitmq-client Specifies which RabbitMQ client to use (default value: internal). The source of the rabbitmq client is included in the source code package of syslog-ng OSE and is used by default. To use an external client instead, use the --with-librabbitmq-client=system compiling option. For details on using this destination, see Section 7.1, Publishing messages using AMQP.

    • --with-module-dir Specifies a single directory where the syslog-ng OSE Makefile will install the modules.

    • --with-module-path Specifies a colon-separated (:) list of directories, where the syslog-ng OSE binary will search for modules.

    • --with-timezone-dir Specifies the directory where syslog-ng looks for the timezone files to resolve the time_zone() and local_time_zone() options. If not specified, the /opt/syslog-ng/share/zoneinfo/ and /usr/share/zoneinfo/ directories are checked, respectively. Note that HP-UX uses a unique file format (tztab) to describe the timezone information; that format is currently not supported in syslog-ng. As a workaround, copy the zoneinfo files from another, non-HP-UX system to the /opt/syslog-ng/share/zoneinfo/ directory of your HP-UX system.

    • --without-compile-date Removes the compilation date from the binary. For example, as openSUSE checks if recompilation changes the binary to detect if dependent packages need to be rebuilt or not, and including the date changes the binary every time.

3.2. Uninstalling syslog-ng OSE

If you need to uninstall syslog-ng OSE for some reason, you have the following options:

  • If you have installed syslog-ng OSE from a .deb package: Execute the dpkg -r syslog-ng command to remove syslog-ng; or the dpkg -P syslog-ng command to remove syslog-ng OSE and the configuration files as well. Note that removing syslog-ng OSE does not restore the syslog daemon used before syslog-ng.

  • If you have installed syslog-ng OSE from an .rpm package: Execute the rpm -e syslog-ng command to remove syslog-ng OSE. Note that removing syslog-ng OSE does not restore the syslog daemon used before syslog-ng OSE.

3.3. Procedure – Configuring Microsoft SQL Server to accept logs from syslog-ng

Purpose: 

Complete the following steps to configure your Microsoft SQL Server to enable remote logins and accept log messages from syslog-ng.

Steps: 

  1. Start the SQL Server Management Studio application. SelectStart > Programs > Microsoft SQL Server 2005 > SQL Server Management Studio.

  2. Create a new database.

    1. Creating a new MSSQL database 1.

      Figure 3.1. Creating a new MSSQL database 1.


      In the Object Explorer, right-click on theDatabasesentry and selectNew Database.

    2. Creating a new MSSQL database 2.

      Figure 3.2. Creating a new MSSQL database 2.


      Enter the name of the new database (for example syslogng) into theDatabase namefield and clickOK.

  3. Create a new database user and associate it with the new database.

    1. Creating a new MSSQL user 1.

      Figure 3.3. Creating a new MSSQL user 1.


      In the Object Explorer, selectSecurity, right-click on theLoginsentry, then selectNew Login.

    2. Creating a new MSSQL user 2.

      Figure 3.4. Creating a new MSSQL user 2.


      Enter a name (for example syslog-ng) for the user into theLogin namefield.

    3. Select theSQL Server Authenticationoption and enter a password for the user.

    4. In theDefault databasefield, select the database created in Step 2 (for example syslogng).

    5. In theDefault languagefield, select the language of log messages that you want to store in the database, then clickOK.

      Warning

      Incorrect language settings may result in the database converting the messages to a different character-encoding format. That way the log messages may become unreadable, causing information loss.

    6. In the Object Explorer, selectSecurity > Logins, then right-click on the new login created in the previous step, and selectProperties.

    7. Associating database with the new user

      Figure 3.5. Associating database with the new user


      SelectUser Mapping. In theUsers mapped to this loginoption, check the line corresponding to the new login (for example syslogng). In theDatabase role membershipfield, check thedb_ownerandpublicoptions.

  4. Associating database with the new user

    Figure 3.6. Associating database with the new user


    Enable remote logins for SQL users.

    In the Object Explorer right-click on your database server, and selectProperties > Security, and set theServer Authenticationoption toSQL Server and Windows Authentication mode.

Chapter 4. The syslog-ng OSE quick-start guide

This chapter provides a very brief introduction into configuring the syslog-ng OSE application. For details on the format of the configuration file and how to configure sources, destinations, and other features, refer to the subsequent chapters.

4.1. Procedure – Configuring syslog-ng on client hosts

Purpose: 

To configure syslog-ng on a client host, complete the following steps.

Steps: 

  1. Install the syslog-ng application on the host. For details installing syslog-ng on specific operating systems, see Chapter 3, Installing syslog-ng.

  2. Configure the local sources to collect the log messages of the host. Starting with version 3.2, syslog-ng OSE automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see Section 6.10, Collecting the system-specific log messages of a platform.

    Add sources to collect the messages from your log files. File sources look like this:

    source s_myfilesource {
            file("/var/log/myapplication.log" follow_freq(1)); };

    Name every source uniquely. For details on configuring file sources, see Section 6.3, Collecting messages from text files.

    Tip

    Many applications send log messages to logfiles by default (for example, the Roundcube webmail client, or the ProFTPD FTP server), but can be configured to send them to syslog instead. If possible, it is recommended to reconfigure the application that way.

    Note

    The default configuration file of syslog-ng OSE collects platform-specific log messages and the intenal log messages of syslog-ng OSE.

    source s_local {
            system();
            internal();
    };
  3. Create a network destination that points directly to the syslog-ng server, or to a local relay. The network destination greatly depends on the protocol that your logserver or relay accepts messages. Many systems still use the legacy BSD-syslog protocol (RFC3162) over the unreliable UDP transport:

    destination d_network { udp("10.1.2.3"); };

    However, if possible, use the much more reliable IETF-syslog protocol over TCP transport:

    destination d_network {syslog(ip("10.1.2.3") transport("tcp")); };
  4. Create a log statement connecting the local sources to the syslog-ng server or relay. For example:

    log {
            source(s_local); destination(d_network); };
  5. If the logs will also be stored locally on the host, create local file destinations.

    Note

    The default configuration of syslog-ng OSE places the collected messages into the /var/log/messages file:

    destination d_local {
        file("/var/log/messages"); };
  6. Create a log statement connecting the local sources to the file destination.

    Note

    The default configuration of syslog-ng OSE has only one log statement:

    log {
        source(s_local); destination(d_local);
  7. Set filters, macros and other features and options (for example TLS encryption) as necessary.

Example 4.1. The default configuration file of syslog-ng OSE

The following is the default configuration file of syslog-ng OSE 3.2. It collects local log messages and the log messages of syslog-ng OSE and forwards them to a logserver using the IETF-syslog protocol.

@version: 3.4
@include "scl.conf"
source s_local { system(); internal(); };
destination d_local {
            file("/var/log/messages"); };
log { source(s_local); destination(d_local); };
Example 4.2. A simple configuration for clients

The following is a simple configuration file that collects local log messages to the /var/log/messages file.

@version: 3.4
@include "scl.conf"
source s_local { system(); internal(); };
destination d_syslog_tcp {
             syslog("192.168.1.1" transport("tcp") port(2010)); };
log { source(s_local);destination(d_syslog_tcp); };

4.2. Procedure – Configuring syslog-ng on server hosts

Purpose: 

To configure syslog-ng on a server host, complete the following steps.

Steps: 

  1. Install the syslog-ng application on the host. For details installing syslog-ng on specific operating systems, see Chapter 3, Installing syslog-ng.

  2. Starting with version 3.2, syslog-ng OSE automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see Section 6.10, Collecting the system-specific log messages of a platform.

  3. Configure the network sources that collect the log messages sent by the clients and relays. How the network sources should be configured depends also on the capabilities of your client hosts: many older networking devices support only the legacy BSD-syslog protocol (RFC3164) using UDP transport:

    source s_network { syslog(ip(10.1.2.3) transport("udp")); };

    However, if possible, use the much more reliable TCP transport:

    source s_network { syslog(ip(10.1.2.3) transport("tcp")); };

    For other options, see Section 6.9, Collecting messages using the IETF syslog protocol and Section 6.11, Collecting messages from remote hosts using the BSD syslog protocol.

    Note

    Starting with syslog-ng OSE version 3.2, the syslog() source driver can handle both BSD-syslog (RFC 3164) and IETF-syslog (RFC 5424-26) messages.

  4. Create local destinations that will store the log messages, for example file- or program destinations. The default configuration of syslog-ng OSE places the collected messages into the /var/log/messages file:

    destination d_local {
        file("/var/log/messages"); };

    If you want to create separate logfiles for every client host, use the ${HOST} macro when specifying the filename, for example:

    destination d_local {
        file("/var/log/messages_${HOST}"); };

    For details on further macros and how to use them, see Chapter 11, Manipulating messages.

  5. Create a log statement connecting the sources to the local destinations.

    log {
            source(s_local); source(s_network); destination(d_local);
  6. Set filters, options (for example TLS encryption) and other advanced features as necessary.

    Note

    By default, the syslog-ng server will treat the relayed messages as if they were created by the relay host, not the host that originally sent them to the relay. In order to use the original hostname on the syslog-ng server, use the keep_hostname(yes) option both on the syslog-ng relay and the syslog-ng server. This option can be set individually for every source if needed.

    If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay to do the name resolution.

Example 4.3. A simple configuration for servers

The following is a simple configuration file for syslog-ng Open Source Edition that collects incoming log messages and stores them in a text file.

@version: 3.4
@include "scl.conf"
    options {
        time_reap(30);
        mark_freq(10);
        keep_hostname(yes);
        };
    source s_local { system(); internal(); };
    source s_network {
        syslog(transport(tcp));
        };
    destination d_logs {
        file(
            "/var/log/syslog-ng/logs.txt"
            owner("root")
            group("root")
            perm(0777)
            ); };
    log { source(s_local); source(s_network); destination(d_logs); };

4.3. Configuring syslog-ng relays

This section describes how to configure syslog-ng OSE as a relay.

4.3.1. Procedure – Configuring syslog-ng on relay hosts

Purpose: 

To configure syslog-ng on a relay host, complete the following steps:

Steps: 

  1. Install the syslog-ng application on the host. For details installing syslog-ng on specific operating systems, see Chapter 3, Installing syslog-ng.

  2. Configure the network sources that collect the log messages sent by the clients.

  3. Create a network destination that points to the syslog-ng server.

  4. Create a log statement connecting the network sources to the syslog-ng server.

  5. Configure the local sources that collect the log messages of the relay host.

  6. Create a log statement connecting the local sources to the syslog-ng server.

  7. Enable the keep_hostname() and disable the chain_hostnames() options.

    Note

    It is recommended to use these options on your syslog-ng OSE server as well.

  8. Set filters and options (for example TLS encryption) as necessary.

    Note

    By default, the syslog-ng server will treat the relayed messages as if they were created by the relay host, not the host that originally sent them to the relay. In order to use the original hostname on the syslog-ng server, use the keep_hostname(yes) option both on the syslog-ng relay and the syslog-ng server. This option can be set individually for every source if needed.

    If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay to do the name resolution.

Example 4.4. A simple configuration for relays

The following is a simple configuration file that collects local and incoming log messages and forwards them to a logserver using the IETF-syslog protocol.

@version: @techversion;
@include "scl.conf"
    options {
        time_reap(30);
        mark_freq(10);
        keep_hostname(yes);
        chain_hostnames(no);
        };
    source s_local { system(); internal(); };
    source s_network {
        syslog(transport(tcp));
        };
    destination d_syslog_tcp {
        syslog("192.168.1.5" transport("tcp") port(2010));
    };

    log { source(s_local); source(s_network);
          destination(d_syslog_tcp);
        };

4.3.2. How relaying log messages works

Depending on your exact needs about relaying log messages, there are many scenarios and syslog-ng OSE options that influence how the log message will look like on the logserver. Some of the most common cases are summarized in the following example[4].

Consider the following example: client-host > syslog-ng-relay > syslog-ng-server, where the IP address of client-host is 192.168.1.2. The client-host device sends a syslog message to syslog-ng-relay. Depending on the settings of syslog-ng-relay, the following can happen.

  • By default, the keep_hostname() option is disabled, so syslog-ng-relay writes the IP address of the sender host (in this case, 192.168.1.2) to the HOST field of the syslog message, discarding any IP address or hostname that was originally in the message.

  • If the keep_hostname() option is enabled on syslog-ng-relay, but name resolution is disabled (the use_dns() option is set to no), syslog-ng-relay uses the HOST field of the message as-is, which is probably 192.168.1.2.

  • To resolve the 192.168.1.2 IP address to a hostname on syslog-ng-relay using a DNS server, use the keep_hostname(no) and use_dns(yes) options. If the DNS server is properly configured and reverse DNS lookup is available for the 192.168.1.2 address, syslog-ng OSE will rewrite the HOST field of the log message to client-host.

    Note

    It is also possible to resolve IP addresses locally, without relying on the DNS server. For details on local name resolution, see Procedure 17.4.1, Resolving hostnames locally.

  • The above points apply to the syslog-ng OSE server (syslog-ng-server) as well, so if syslog-ng-relay is configured properly, use the keep_hostname(yes) option on syslog-ng-server to retain the proper HOST field. Setting on keep_hostname(no) on syslog-ng-server would result in syslog-ng OSE rewriting the HOST field to the address of the host that sent the message to syslog-ng-server, which is syslog-ng-relay in this case.

  • If you cannot or do not want to resolve the 192.168.1.2 IP address on syslog-ng-relay, but want to store your log messages on syslog-ng-server using the IP address of the original host (that is, client-host), you can enable the spoof_source() option on syslog-ng-relay. However, spoof_source() works only under the following conditions:

    • The syslog-ng OSE binary has been compiled with the --enable-spoof-source option.

    • The log messages are sent using the highly unreliable UDP transport protocol. (Extremely unrecommended.)



[4] Many thanks to Lance Laursen for his excellent post about this topic on the syslog-ng mailing list.

Chapter 5. The syslog-ng OSE configuration file

5.1. Location of the syslog-ng configuration file

The syslog-ng application is configured by editing the syslog-ng.conf file. Use any regular text editor application to modify the file. The location of the configuration file depends on how you installed syslog-ng OSE. Native packages of a platform (like the ones downloaded from Linux repositories) typically place the configuration file under the /etc/syslog-ng/ directory.

5.2. The configuration syntax in detail

Every syslog-ng configuration file must begin with a line containing the version information of syslog-ng. For syslog-ng version 3.4, this line looks like:

@version: 3.4

Versioning the configuration file was introduced in syslog-ng 3.0. If the configuration file does not contain the version information, syslog-ng assumes that the file is for syslog-ng version 2.x. In this case it interprets the configuration and sends warnings about the parts of the configuration that should be updated. Version 3.0 and later will correctly operate with configuration files of version 2.x, but the default values of certain parameters have changed since 3.0.

Example 5.1. A simple configuration file

The following is a very simple configuration file for syslog-ng: it collects the internal messages of syslog-ng and the messages from /dev/log into the /var/log/messages_syslog-ng.log file.

@version: 3.4

source s_local { unix-stream("/dev/log"); internal(); };

destination d_file {file("/var/log/messages_syslog-ng.log"); };

log { source(s_local); destination(d_file); };
  • The main body of the configuration file consists of object definitions: sources, destinations, logpaths define which log message are received and where they are sent. All identifiers, option names and attributes, and any other strings used in the syslog-ng configuration file are case sensitive. Objects must be defined before they are referenced in another statement. Object definitions (also called statements) have the following syntax:

    object_type object_id {<options>};
    • Type of the object: One of source, destination, log, filter, parser, rewrite rule, or template.

    • Identifier of the object: A unique name identifying the object. When using a reserved word as an identifier, enclose the identifier in quotation marks.

      Tip

      Use identifiers that refer to the type of the object they identify. For example, prefix source objects with s_, destinations with d_, and so on.

      Note

      Repeating a definition of an object (that is, defining the same object with the same id more than once) is not allowed, unless you use the @define allow-config-dups 1 definition in the configuration file.

    • Parameters: The parameters of the object, enclosed in braces {parameters}.

    • Semicolon: Object definitions end with a semicolon (;).

    For example, the following line defines a source and calls it s_internal.

    source s_internal { internal(); };

    The object can be later referenced in other statements using its ID, for example, the previous source is used as a parameter of the following log statement:

    log { source(s_internal); destination(d_file); };
  • The parameters and options within a statement are similar to function calls of the C programming language: the name of the option followed by a list of its parameters enclosed within brackets and terminated with a semicolon.

    option(parameter1, parameter2); option2(parameter1, parameter2);

    For example, the file() driver in the following source statement has three options: the filename (/var/log/apache/access.log), follow_freq(), and flags(). The follow_freq() option also has a parameter, while the flags() option has two parameters.

    source s_tail { file("/var/log/apache/access.log"
        follow_freq(1) flags(no-parse, validate-utf8)); };

    Objects may have required and optional parameters. Required parameters are positional, meaning that they must be specified in a defined order. Optional parameters can be specified in any order using the option(value) format. If a parameter (optional or required) is not specified, its default value is used. The parameters and their default values are listed in the reference section of the particular object.

    Example 5.2. Using required and optional parameters

    The unix-stream() source driver has a single required argument: the name of the socket to listen on. Optional parameters follow the socket name in any order, so the following source definitions have the same effect:

    source s_demo_stream1 {
            unix-stream("/dev/log" max-connections(10) group(log)); };
    source s_demo_stream2 {
            unix-stream("/dev/log" group(log) max-connections(10)); };
  • Some options are global options, or can be set globally, for example, whether syslog-ng OSE should use DNS resolution to resolve IP addresses. Global options are detailed in Chapter 9, Global options of syslog-ng OSE.

    options { use_dns(no); };
  • All identifiers, attributes, and any other strings used in the syslog-ng configuration file are case sensitive.

  • Objects can be used before definition.

  • Objects can be defined inline as well. This is useful if you use the object only once (for example, a filter). For details, see Section 5.4, Defining configuration objects inline.

  • To add comments to the configuration file, start a line with # and write your comments. These lines are ignored by syslog-ng.

    # Comment: This is a stream source
    source s_demo_stream {
            unix-stream("/dev/log" max-connections(10) group(log)); };
Tip

Before activating a new configuration, check that your configuration file is syntactically correct using the syslog-ng --syntax-only command.

To activate the configuration, reload the configuration of syslog-ng using the /etc/init.d/syslog-ng reload command.

5.3. Notes about the configuration syntax

When you are editing the syslog-ng configuration file, note the following points:

  • The configuration file can contain a maximum of 6665 source / destination / log elements.

  • When writing the names of options and parameters (or other reserved words), the hyphen (-) and underscore (_) characters are equivalent, for example max-connections(10) and max_connections(10) are both correct.

  • Number can be prefixed with + or - to indicate positive or negative values. Numbers beginning with zero (0) or 0x are treated as octal or hexadecimal numbers, respectively.

  • You can use commas (,) to separate options or other parameters for readability; syslog-ng completely ignores them. The following declarations are equivalent:

    source s_demo_stream {
            unix-stream("/dev/log" max-connections(10) group(log)); };
    source s_demo_stream {
            unix-stream("/dev/log", max-connections(10), group(log)); };
  • When enclosing object IDs (for example the name of a destination) between double-quotes ("mydestination"), the ID can include whitespace as well, for example:

    source "s demo stream" {
            unix-stream("/dev/log" max-connections(10) group(log)); };
  • For notes on using regular expressions, see Section 11.3, Regular expressions.

5.4. Defining configuration objects inline

Starting with syslog-ng OSE 3.4, you can define configuration objects inline, where they are actually used, without having to define them in a separate placement. This is useful if you need an object only once, for example, a filter or a rewrite rule. Every object can be defined inline: sources, destinations, filters, parsers, rewrite rules, and so on.

To define an object inline, use braces instead of parentheses. That is, instead of <object-type> (<object-id>);, you use <object-type> {<object-definition>};

Example 5.3. Using inline definitions

The following two configuration examples are equivalent. The first one uses traditional statements, while the second uses inline definitions.

source s_local {
    system();
    internal();
};
destination d_local {
    file("/var/log/messages");
};
log {
    source(s_local);
    destination(d_local);
};
log {
    source {
        system();
        internal();
    };
    destination {
        file("/var/log/messages");s
    };
};

5.5. Using channels in configuration objects

Starting with syslog-ng OSE 3.4, every configuration object is a log expression. Every configuration object is essentially a configuration block, and can include multiple objects. To reference the block, only the top-level object must be referenced. That way you can use embedded log statements, junctions and in-line object definitions within source, destination, filter, rewrite and parser definitions. For example, a source can include a rewrite rule to modify the messages received by the source, and that combination can be used as a simple source in a log statement. This feature allows you to preprocess the log messages very close to the source itself.

To embed multiple objects into a configuration object, use the following syntax. Note that you must enclose the configuration block between braces instead of parenthesis.

<type-of-top-level-object> <name-of-top-level-object> {
    channel {
        <configuration-objects>
    }
}
Example 5.4. Using channels

For example, to process a log file in a specific way, you can define the required processing rules (parsers and rewrite expressions) and combine them in a single object:

source s_apache {
    channel {
        source { file("/var/log/apache/error.log"); };
        parser(p_apache_parser); };
  };
};

log { source(s_apache); ... };

The s_apache source uses a file source (the error log of an Apache webserver) and references a specific parser to process the messages of the error log. The log statement references only the s_apache source, and any other object in the log statement can already use the results of the p_apache_parserparser.

Note

You must start the object definition with a channel even if you will use a junction, for example:

parser demo-parser() {
    channel {
        junction {
            channel { ... }
            channel { ... }
        }
    }
}

If you want to embed configuration objects into sources or destinations, always use channels, otherwise the source or destination will not behave as expected. For example, the following configuration is good:

source s_filtered_hosts {
    pipe("/dev/pipe");
    syslog(ip(192.168.0.1) transport("tcp"));
    syslog(ip(127.0.0.1) transport("tcp"));
    filter (netmask(10.0.0.0/16));
}

5.6. Global and environmental variables

Starting with syslog-ng OSE version 3.2, it is possible to define global variables in the configuration file. Global variables are actually name-value pairs; when syslog-ng processes the configuration file during startup, it automatically replaces `name` with value. To define a global variable, use the following syntax:

@define name "value"

The value can be any string, but special characters must be escaped.To use the variable, insert the name of the variable enclosed between backticks (`, similarly to using variables in Linux or UNIX shells) anywhere in the configuration file.

The value of the global variable can be also specified using the following methods:

  • Without any quotes, as long as the value does not contain any spaces or special characters. In other word, it contains only the following characters: a-zA-Z0-9_..

  • Between apostrophes, in case the value does not contain apostrophes.

  • Between double quotes, in which case special characters must be escaped using backslashes (\).

Tip

The environmental variables of the host are automatically imported and can be used as global variables.

Example 5.5. Using global variables

For example, if an application is creating multiple log files in a directory, you can store the path in a global variable, and use it in your source definitions.

@define mypath "/opt/myapp/logs"
        source s_myapp_1 { file("`mypath`/access.log" follow_freq(1)); };
        source s_myapp_2 { file("`mypath`/error.log" follow_freq(1)); };
        source s_myapp_3 { file("`mypath`/debug.log" follow_freq(1)); };

The syslog-ng OSE application will interpret this as:

@define mypath "/opt/myapp/logs"
        source s_myapp_1 { file("/opt/myapp/logs/access.log" follow_freq(1)); };
        source s_myapp_2 { file("/opt/myapp/logs/error.log" follow_freq(1)); };
        source s_myapp_3 { file("/opt/myapp/logs/debug.log" follow_freq(1)); };

5.7. Loading modules

Starting with syslog-ng Open Source Edition version 3.3, syslog-ng OSE became modular to increase its flexibility and also to simplify the development of additional modules. Most of the functionality of syslog-ng OSE has been moved to separate modules. That way it becomes also possible to finetune the resource requirements of syslog-ng OSE for example, by loading only the modules that are actually used in the configuration, or simply omitting modules that are not used but require large amount of memory.

Each module contains one or more plugins, which add some functionality to syslog-ng OSE, for example, a destination or a source driver.

  • To display the list of available modules, execute the syslog-ng --version command.

  • To the description of the available modules, execute the syslog-ng --module-registry command.

  • To customize which modules are loaded automatically when syslog-ng OSE is started, use the --default-modules command-line option of syslog-ng OSE.

  • To request loading a module from the syslog-ng OSE configuration file, see Section 5.7.1, Loading modules.

For details on the command-line parameters of syslog-ng OSE mentioned in the previous list, see the syslog-ng OSE man page at syslog-ng(8).

5.7.1. Loading modules

The syslog-ng Open Source Edition application loads every available module during startup.

To load a module that is not loaded automatically, include the following statement in the syslog-ng OSE configuration file:

@module <module-name>

Note the following points about the @module statement:

  • The @module statement is a top-level statement, that is, it cannot be nested into any other statement. Usually it is used immediately after the @version statement.

  • Every @module statement loads a single module: loading multiple modules requires a separate @module statement for every module.

  • In the configuration file, the @module statement of a module must be earlier than the module is used.

Note

To disable loading every module automatically, set the autoload-compiled-modules global variable to 0 in your configuration file:

@define autoload-compiled-modules 0

Note that in this case, you have to explicitly load the modules you want to use.

5.8. Managing complex syslog-ng configurations

The following sections describe some methods that can be useful to simplify the management of large-scale syslog-ng installations.

5.8.1. Including configuration files

The syslog-ng application supports including external files in its configuration file, so parts of its configuration can be managed separately. To include the contents of a file in the syslog-ng configuration, use the following syntax:

include "<filename>";

This imports the entire file into the configuration of syslog-ng OSE, at the location of the include statement. The <filename> can be one of the following:

  • A file name, optionally with full path. The filename (not the path) can include UNIX-style wildcard characters (*, ?). When using wildcard characters, syslog-ng OSE will include every matching file. For details on using wildcard characters, see Section glob.

  • A directory. When including a directory, syslog-ng OSE will try to include every file from the directory in alphabetic order, except files beginning with a ~ (tilde) or a . (dot) character. Including a directory is not recursive.

When including configuration files, consider the following points:

  • Defining an object twice is not allowed, unless you use the @define allow-config-dups 1 definition in the configuration file. If an object is defined twice (for example the original syslog-ng configuration file and the file imported into this configuration file both define the same option, source, or other object), then the object that is defined later in the configuration file will be effective. For example, if you set a global option at the beginning of the configuration file, and later include a file that defines the same option with a different value, then the option defined in the imported file will be used.

  • Files can be embedded into each other: the included files can contain include statements as well, up to a maximum depth of 15 levels.

  • You cannot include complete configuration files into each other, only configuration snippets can be included. This means that the included file cannot have a @version statement.

  • Include statements can only be used at top level of the configuration file. For example, the following is correct:

    @version: 3.4
    include "example.conf";

    But the following is not:

    source s_example {
        include "example.conf"
         };
Warning

The syslog-ng application will not start if it cannot find a file that is to be included in its configuration. Always double-check the filenames, paths, and access rights when including configuration files, and use the --syntax-only command-line option to check your configuration.

5.8.2. Reusing configuration blocks

Starting with syslog-ng OSE 3.2, parts of a configuration file can be easily reused, you have to define the block (for example, a source) once, and reference it later. Any syslog-ng object can be a block. Use the following syntax to define a block:

block type name() {<contents of the block>}

Type must be one of the following: destination, filter, log, parser, rewrite, root, source. The root blocks can be used in the "root" context of the configuration file, that is, outside any other statements.

Blocks may be nested into each other, so for example an SCL may be built from other blocks. Blocks are somewhat similar to C++ templates.

The type and name combination of each block must be unique, that is, two blocks can have the same name if their type is different.

To use a block in your configuration file, you have to do two things:

  • Include the file defining the block in the syslog-ng.conf file — or a file already included into syslog-ng.conf.

  • Reference the name of the block in your configuration file. This will insert the block into your configuration. For example, to use a block called myblock, include the following line in your configuration:

    myblock()

    Blocks may have parameters, but even if they do not, the reference must include opening and closing parentheses like in the previous example.

The contents of the block will be inserted into the configuration when syslog-ng OSE is started or reloaded.

Example 5.6. Reusing configuration blocks

Suppose you are running an application on your hosts that logs into the /opt/var/myapplication.log file. Create a file (for example, myblocks.conf) that stores a source describing this file and how it should be read:

block source myappsource() {
                file("/opt/var/myapplication.log" follow_freq(1) default-facility(syslog)); };

Include this file in your main syslog-ng configuration file,reference the block, and use it in a logpath:

@version: 3.4
include "<correct/path>/myblocks.conf";
source s_myappsource { myappsource(); }
...
log { source(s_myappsource); destination(…); };

To define a block that defines more than one object, use root as the type of the block, and reference the block from the main part of the syslog-ng OSE configuration file.

Example 5.7. Defining blocks with multiple elements

The following example defines a source, a destination, and a log path to connect them.

block root mylogs() {
        source s_file { file("/var/log/mylogs.log" follow_freq(1)); };
        destination d_local { file("/var/log/messages"); };
        log { source(s_file); destination(d_local); };
        };
Tip

Since the block is inserted into the syslog-ng OSE configuration when syslog-ng OSE is started, the block can be generated dynamically using an external script if needed. This is useful when you are running syslog-ng OSE on different hosts and you want to keep the main configuration identical.

If you want to reuse more than a single configuration object, for example, a logpath and the definitions of its sources and destinations, use the include feature to reuse the entire snippet. For details, see Section 5.8.1, Including configuration files.

5.8.2.1. Passing arguments to configuration blocks

Configuration blocks can receive arguments as well. The parameters the block can receive must be specified when the block is defined, using the following syntax:

block type block_name(argument1(<default-value-of-the-argument>) argument2(<default-value-of-the-argument>) argument3())

If an argument does not have a default value, use empty parentheses after the name of the argument. To refer the value of the argument in the block, use the name of the argument between backticks (for example, `argument1`).

Example 5.8. Passing arguments to blocks

The following sample defines a file source block, which can receive the name of the file as a parameter. If no parameter is set, it reads messages from the /var/log/messages file.

block source s_logfile (filename("messages")) {
  file("/var/log/`filename`" );
};

source s_example {
  s_logfile(filename("logfile.log"));
};
Example 5.9. Using arguments in blocks

The following example is the code of the pacct() source driver, which is actually a block that can optionally receive two arguments.

block source pacct(file("/var/log/account/pacct") follow-freq(1)) {
@module pacctformat
        file("`file`" follow-freq(`follow-freq`) format("pacct") tags(".pacct"));
};

Chapter 6. Collecting log messages — sources and source drivers

6.1. How sources work

A source is where syslog-ng receives log messages. Sources consist of one or more drivers, each defining where and how messages are received.

To define a source, add a source statement to the syslog-ng configuration file using the following syntax:

source <identifier> { source-driver(params); source-driver(params); ... };
Example 6.1. A simple source statement

The following source statement receives messages on the TCP port 1999 of the interface having the 10.1.2.3 IP address.

source s_demo_tcp { tcp(ip(10.1.2.3) port(1999)); };
Example 6.2. A source statement using two source drivers

The following source statement receives messages on the 1999 TCP port and the 1999 UDP port of the interface having the 10.1.2.3 IP address.

source s_demo_two_drivers {
           tcp(ip(10.1.2.3) port(1999));
           udp(ip(10.1.2.3) port(1999)); };
Example 6.3. Setting default priority and facility

If the message received by the source does not have a proper syslog header, you can use the default-facility() and default-priority() options to set the facility and priority of the messages. Note that these values are applied only to messages that do not set these parameters in their header.

source headerless_messages { udp(default-facility(syslog) default-priority(emerg)); };

Define a source only once. The same source can be used in several log paths. Duplicating sources causes syslog-ng to open the source (TCP/IP port, file, and so on) more than once, which might cause problems. For example, include the /dev/log file source only in one source statement, and use this statement in more than one log path if needed.

Note

Sources and destinations are initialized only when they are used in a log statement. For example, syslog-ng OSE starts listening on a port or starts polling a file only if the source is used in a log statement. For details on creating log statements, see Chapter 8, Routing messages: log paths and filters.

To collect log messages on a specific platform, it is important to know how the native syslogd communicates on that platform. The following table summarizes the operation methods of syslogd on some of the tested platforms:

Platform Method
Linux A SOCK_STREAM unix socket named /dev/log; some of the distributions switched over to using SOCK_DGRAM, though applications still work with either method.
BSD flavors A SOCK_DGRAM unix socket named /var/run/log.
Solaris (2.5 or below) An SVR4 style STREAMS device named /dev/log.
Solaris (2.6 or above) In addition to the STREAMS device used in earlier versions, 2.6 uses a new multithreaded IPC method called door. By default the door used by syslogd is /etc/.syslog_door.
HP-UX 11 or later HP-UX uses a named pipe called /dev/log that is padded to 2048 bytes, for example source s_hp-ux {pipe ("/dev/log" pad_size(2048)}.
AIX 5.2 and 5.3 A SOCK_STREAM or SOCK_DGRAM unix socket called /dev/log.

Table 6.1. Communication methods used between the applications and syslogd


Each possible communication mechanism has a corresponding source driver in syslog-ng. For example, to open a unix socket with SOCK_DGRAM style communication use the driver unix-dgram. The same socket using the SOCK_STREAM style — as used under Linux — is called unix-stream.

Example 6.4. Source statement on a Linux based operating system

The following source statement collects the following log messages:

  • internal(): Messages generated by syslog-ng.

  • udp(ip(0.0.0.0) port(514)): Messages arriving to the 514/UDP port of any interface of the host.

  • unix-stream("/dev/log");: Messages arriving to the /dev/log socket.

source s_demo {
           internal();
           udp(ip(0.0.0.0) port(514));
           unix-stream("/dev/log"); };

The following table lists the source drivers available in syslog-ng.

Name Description
internal() Messages generated internally in syslog-ng.
file() Opens the specified file and reads messages.
pacct() Reads messages from the process accounting logs on Linux.
pipe() Opens the specified named pipe and reads messages.
program() Opens the specified application and reads messages from its standard output.
sun-stream(), sun-streams() Opens the specified STREAMS device on Solaris systems and reads incoming messages.
syslog() Listens for incoming messages using the new IETF-standard syslog protocol.
system() Automatically detects which platform syslog-ng OSE is running on, and collects the native log messages of that platform.
tcp(), tcp6() Listens on the specified TCP port for incoming messages using the BSD-syslog protocol over IPv4 and IPv6 networks, respectively.
udp(), udp6() Listens on the specified UDP port for incoming messages using the BSD-syslog protocol over IPv4 and IPv6 networks, respectively.
unix-dgram() Opens the specified unix socket in SOCK_DGRAM mode and listens for incoming messages.
unix-stream() Opens the specified unix socket in SOCK_STREAM mode and listens for incoming messages.

Table 6.2. Source drivers available in syslog-ng


6.2. Collecting internal messages

All messages generated internally by syslog-ng use this special source. To collect warnings, errors and notices from syslog-ng itself, include this source in one of your source statements.

internal()

The syslog-ng application will issue a warning upon startup if none of the defined log paths reference this driver.

Example 6.5. Using the internal() driver
source s_local { internal(); };

The syslog-ng OSE application sends the following message types from the internal() source: 

  • fatal: Priority value: critical (2); Facility value: syslog (5)

  • error: Priority value: error (3); Facility value: syslog (5)

  • warning: Priority value: warning (4); Facility value: syslog (5)

  • notice: Priority value: notice (5); Facility value: syslog (5)

  • info: Priority value: info (6); Facility value: syslog (5)

6.2.1. internal() source options

The internal() driver has the following options:

tags()

Type: string
Default:  

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example tags("dmz", "router"). This option is available only in syslog-ng 3.1 and later.

6.3. Collecting messages from text files

Collects log messages from plain-text files, for example from the logfiles of an Apache webserver.

The syslog-ng application notices if a file is renamed or replaced with a new file, so it can correctly follow the file even if logrotation is used. When syslog-ng is restarted, it records the position of the last sent log message in the /opt/syslog-ng/var/syslog-ng.persist file, and continues to send messages from this position after the restart.

The file driver has a single required parameter specifying the file to open. For the list of available optional parameters, see Section 6.3.2, file() source options.

Declaration:
            file(filename);
Example 6.6. Using the file() driver
source s_file { file("/var/log/messages"); };
Example 6.7. Tailing files

The following source checks the access.log file every second for new messages.

source s_tail { file("/var/log/apache/access.log"
            follow_freq(1) flags(no-parse)); };
Note

If the message does not have a proper syslog header, syslog-ng treats messages received from files as sent by the kern facility. Use the default-facility and default-priority options in the source definition to assign a different facility if needed.

6.3.1. Notes on reading kernel messages

Note the following points when reading kernel messages on various platforms.

  • The kernel usually sends log messages to a special file (/dev/kmsg on BSDs, /proc/kmsg on Linux). The file() driver reads log messages from such files. The syslog-ng application can periodically check the file for new log messages if the follow_freq() option is set.

  • On Linux, the klogd daemon can be used in addition to syslog-ng to read kernel messages and forward them to syslog-ng. klogd used to preprocess kernel messages to resolve symbols and so on, but as this is deprecated by ksymoops there is really no point in running both klogd and syslog-ng in parallel. Also note that running two processes reading /proc/kmsg at the same time might result in dead-locks.

  • When using syslog-ng to read messages from the /proc/kmsg file, syslog-ng automatically disables the follow_freq() parameter to avoid blocking the file.

  • To read the kernel messages on HP-UX platforms, use the following options in the source statement:

    file("/dev/klog" program_override("kernel") flags(kernel) follow_freq(0));

6.3.2. file() source options

The file() driver has the following options:

default-facility()

Type: facility string
Default: kern

Description:This parameter assigns a facility value to the messages received from the file source, if the message does not specify one.

default-priority()

Type: priority string
Default:  

Description:This parameter assigns an emergency level to the messages received from the file source, if the message does not specify one.

file()

Type: filename with path
Default:  

Description:The file to read messages from.

encoding()

Type: string
Default:  

Description:Specifies the characterset (encoding, for example UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command.

flags()

Type: empty-lines, expect-hostname, kernel, no-multi-line, no-parse, store-legacy-msghdr, syslog-protocol, validate-utf8
Default: empty set

Description:Specifies the log parsing options of the source.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ng OSE removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, syslog-ng OSE will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way syslog-ng OSE assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

    source s_dell { udp(port(2000) flags(no-hostname)); };
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the syslog, udp, unix-dgram drivers support multi-line messages; other drivers, for example, the tcp driver does not.

  • no-parse: By default, syslog-ng OSE parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The syslog-ng OSE application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MSG part of the syslog message. This flag is useful for parsing messages not complying to the syslog format.

  • dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the log message. This is useful of the original format of a non-syslog-compliant message must be retained (syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages formatted according to the new IETF syslog standard (for details, see Section 2.8.2, IETF-syslog messages). If theBOM[5]character is missing, but the message is otherwise UTF-8 compliant, syslog-ng automatically adds the BOM character to the message.

follow_freq()

Type: number
Default: 1

Description:Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, syslog-ng will not attempt to use poll() on the file, but checks whether the file changed every time the follow_freq() interval (in seconds) has elapsed. Floating-point numbers (for example 1.5) can be used as well.

keep_timestamp()

Type: yes or no
Default: yes

Description:Specifies whether syslog-ng should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log_fetch_limit()

Type: number
Default: 10

Description:The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log_fetch_limit() is too high.

log_iw_size()

Type: number
Default: 1000

Description:The size of the initial window, this value is used during flow control. If the max-connections() option is set, the log_iw_size() will be divided by the number of connections, otherwise log_iw_size() is divided by 10 (the default value of the max-connections() option). The resulting number is the initial window size of each connection.

Example 6.8. Initial window size of a connection

If log_iw_size(1000) and max-connections(10), then each connection will have an initial window size of 100.

log_msg_size()

Type: number
Default: Use the global log_msg_size() option, which defaults to 8192.

Description:Specifies the maximum length of incoming log messages. Uses the value of the global option if not specified.

log_prefix() (DEPRECATED)

Type: string
Default:  

Description:A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux. NOTE: This option is deprecated. Use program_override() instead.

optional()

Type: yes or no
Default:  

Description:Instruct syslog-ng to ignore the error if a specific source cannot be initialized. No other attempts to initialize the source will be made until the configuration is reloaded. This option currently applies to the pipe(), unix-dgram, and unix-stream drivers.

pad_size()

Type: number
Default: 0

Description:Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). The syslog-ng OSE application will pad reads from the associated device to the number of bytes set in pad_size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad_size was given and the incoming message does not fit into pad_size, syslog-ng will not read anymore from this pipe and displays the following error message:

Padding was set, and couldn't read enough bytes

program_override()

Type: string
Default:  

Description:Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program_override("kernel") option in the source containing /proc/kmsg. NOTE: This option replaces the deprecated log_prefix() option.

tags()

Type: string
Default:  

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example tags("dmz", "router"). This option is available only in syslog-ng 3.1 and later.

time_zone()

Type: timezone in +/-HH:MM format
Default:  

Description:The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.



[5] The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

6.4. Collecting messages using the RFC3164 protocol

The network() destination driver can receive syslog messages conforming to RFC3164 from the network using the TCP, TLS, and UDP networking protocols.

Example 6.9. Using the network() driver

TCP source listening on the localhost on port 2222 without using the network() driver.

source s_tcp6 {
        tcp6(
            ip("::1")
            port(2222)
        );
};

TCP source listening on the localhost on port 2222 using the network() driver.

source s_network6 {
        network(
            ip("::1")
            transport("tcp")
            port(2222)
            ip-protocol(6)
        );
};
Note

For details on the tcp(), tcp6(), udp(), udp6() drivers, see Section 6.11, Collecting messages from remote hosts using the BSD syslog protocol.

6.4.1. network() source options

The network() driver has the following options.

flags()

Type: empty-lines, expect-hostname, kernel, no-multi-line, no-parse, store-legacy-msghdr, syslog-protocol, validate-utf8
Default: empty set

Description:Specifies the log parsing options of the source.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ng OSE removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, syslog-ng OSE will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way syslog-ng OSE assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

    source s_dell { udp(port(2000) flags(no-hostname)); };
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the syslog, udp, unix-dgram drivers support multi-line messages; other drivers, for example, the tcp driver does not.

  • no-parse: By default, syslog-ng OSE parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The syslog-ng OSE application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MSG part of the syslog message. This flag is useful for parsing messages not complying to the syslog format.

  • dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the log message. This is useful of the original format of a non-syslog-compliant message must be retained (syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages formatted according to the new IETF syslog standard (for details, see Section 2.8.2, IETF-syslog messages). If theBOM[6]character is missing, but the message is otherwise UTF-8 compliant, syslog-ng automatically adds the BOM character to the message.

host_override()

Type: string
Default:  

Description:Replaces the ${HOST} part of the message with the parameter string.

ip() or localip()

Type: string
Default: 0.0.0.0

Description:The IP address to bind to. Note that this is not the address where messages are accepted from.

ip-protocol()

Type: number
Default: 4

Description:Determines the internet protocol version of the given driver (network() or syslog()). The possible values are 4 and 6, corresponding to IPv4 and IPv6. The default value is 4.

ip_tos()

Type: number
Default: 0

Description:Specifies the Type-of-Service value of outgoing packets.

ip_ttl()

Type: number
Default: 0

Description:Specifies the Time-To-Live value of outgoing packets.

keep-alive()

Type: yes or no
Default: yes

Description:Specifies whether connections to sources should be closed when syslog-ng is forced to reload its configuration (upon the receipt of a SIGHUP signal). Note that this applies to the server (source) side of the syslog-ng connections, client-side (destination) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the destination.

keep_hostname()

Type: yes or no
Default: no

Description:Enable or disable hostname rewriting.

  • If enabled (keep_hostname(yes)), syslog-ng OSE assumes that the incoming log message was sent by the host specified in the HOST field of the message.

  • If disabled (keep_hostname(no)), syslog-ng OSE rewrites the HOST field of the message, either to the IP address (if the use_dns() parameter is set to no), or to the hostname (if the use_dns() parameter is set to yes and the IP address can be resolved to a hostname) of the host sending the message to syslog-ng OSE. For details on using name resolution in syslog-ng OSE, see Section 17.4, Using name resolution in syslog-ng.

Note

If the log message does not contain a hostname in its HOST field, syslog-ng OSE automatically adds a hostname to the message.

  • For messages received from the network, this hostname is the address of the host that sent the message (this means the address of the last hop if the message was transferred via a relay).

  • For messages received from the local host, syslog-ng OSE adds the name of the host.

This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

Note

When relaying messages, enable this option on the syslog-ng OSE server and also on every relay, otherwise syslog-ng OSE will treat incoming messages as if they were sent by the last relay.

keep_timestamp()

Type: yes or no
Default: yes

Description:Specifies whether syslog-ng should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log_fetch_limit()

Type: number
Default: 10

Description:The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log_fetch_limit() is too high.

log_iw_size()

Type: number
Default: 1000

Description:The size of the initial window, this value is used during flow control. If the max-connections() option is set, the log_iw_size() will be divided by the number of connections, otherwise log_iw_size() is divided by 10 (the default value of the max-connections() option). The resulting number is the initial window size of each connection.

Example 6.10. Initial window size of a connection

If log_iw_size(1000) and max-connections(10), then each connection will have an initial window size of 100.

log_msg_size()

Type: number
Default: Use the global log_msg_size() option, which defaults to 8192.

Description:Specifies the maximum length of incoming log messages. Uses the value of the global option if not specified.

log_prefix() (DEPRECATED)

Type: string
Default:  

Description:A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux. NOTE: This option is deprecated. Use program_override() instead.

max-connections()

Type: number
Default: 10

Description:Specifies the maximum number of simultaneous connections.

pad_size()

Type: number
Default: 0

Description:Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). The syslog-ng OSE application will pad reads from the associated device to the number of bytes set in pad_size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad_size was given and the incoming message does not fit into pad_size, syslog-ng will not read anymore from this pipe and displays the following error message:

Padding was set, and couldn't read enough bytes

port() or localport()

Type: number
Default: 601

Description:The port number to bind to.

program_override()

Type: string
Default:  

Description:Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program_override("kernel") option in the source containing /proc/kmsg. NOTE: This option replaces the deprecated log_prefix() option.

so_broadcast()

Type: yes or no
Default: no

Description:This option controls the SO_BROADCAST socket option required to make syslog-ng send messages to a broadcast address. For details, see the socket(7) manual page.

so_keepalive()

Type: yes or no
Default: no

Description:Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

so_rcvbuf()

Type: number
Default: 0

Description:Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.

Warning

When receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (that is, the syslog-ng OSE server or relay receiving the messages). Note that on certain platforms, for example, on Red Hat Enterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so_rcvbuf() option of the source is increased. In such cases, you will need to increase the net.core.rmem_max parameter of the host (for example, to 1024000), but do not modify net.core.rmem_default parameter.

As a general rule, increase the so_rcvbuf() so that the buffer size in kilobytes is higher than the rate of incoming messages per second. For example, to receive 2000 messages per second, set the so_rcvbuf() at least to 2 097 152 bytes.

so_sndbuf()

Type: number
Default: 0

Description:Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

tags()

Type: string
Default:  

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example tags("dmz", "router"). This option is available only in syslog-ng 3.1 and later.

tcp-keep-alive()

Type: yes or no
Default: no

Description:This is an obsolete alias of the so_keepalive() option.

time_zone()

Type: timezone in +/-HH:MM format
Default:  

Description:The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

transport()

Type: udp, tcp, or tls
Default: tcp

Description:Specifies the protocol used to receive messages from the source.

Warning

When receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (that is, the syslog-ng OSE server or relay receiving the messages). Note that on certain platforms, for example, on Red Hat Enterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so_rcvbuf() option of the source is increased. In such cases, you will need to increase the net.core.rmem_max parameter of the host (for example, to 1024000), but do not modify net.core.rmem_default parameter.

As a general rule, increase the so_rcvbuf() so that the buffer size in kilobytes is higher than the rate of incoming messages per second. For example, to receive 2000 messages per second, set the so_rcvbuf() at least to 2 097 152 bytes.

tls()

Type: tls options
Default: n/a

Description:This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see Section 10.4, TLS options.

use_dns()

Type: yes, no, persist_only
Default: yes

Description:Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locally from file (for example from /etc/hosts). The syslog-ng OSE application blocks on DNS queries, so enabling DNS may lead to a Denial of Service attack. To prevent DoS, protect your syslog-ng network endpoint with firewall rules, and make sure that all hosts which may get to syslog-ng are resolvable. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

use_fqdn()

Type: yes or no
Default: no

Description:Add Fully Qualified Domain Name instead of short hostname. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.



[6] The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

6.5. Collecting messages from named pipes

The pipe driver opens a named pipe with the specified name and listens for messages. It is used as the native message delivery protocol on HP-UX.

The pipe driver has a single required parameter, specifying the filename of the pipe to open. For the list of available optional parameters, see Section 6.5.1, pipe() source options.

Declaration:
            pipe(filename);
Note

As of syslog-ng Open Source Edition 3.0.2, pipes are created automatically. In earlier versions, you had to create the pipe using the mkfifo(1) command.

Pipe is very similar to the file() driver, but there are a few differences, for example pipe() opens its argument in read-write mode, therefore it is not recommended to be used on special files like /proc/kmsg.

Warning
  • It is not recommended to use pipe() on anything else than real pipes.

  • By default, syslog-ng OSE uses the flags(no-hostname) option for pipes, meaning that syslog-ng OSE assumes that the log messages received from the pipe do not contain the hostname field. If your messages do contain the hostname field, use flags(expect-hostname). For details, see Section flags().

Example 6.11. Using the pipe() driver
source s_pipe { pipe("/dev/pipe" pad_size(2048)); };

6.5.1. pipe() source options

The pipe driver has the following options:

flags()

Type: empty-lines, expect-hostname, kernel, no-multi-line, no-parse, store-legacy-msghdr, syslog-protocol, validate-utf8
Default: empty set

Description:Specifies the log parsing options of the source.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ng OSE removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, syslog-ng OSE will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way syslog-ng OSE assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

    source s_dell { udp(port(2000) flags(no-hostname)); };
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the syslog, udp, unix-dgram drivers support multi-line messages; other drivers, for example, the tcp driver does not.

  • no-parse: By default, syslog-ng OSE parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The syslog-ng OSE application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MSG part of the syslog message. This flag is useful for parsing messages not complying to the syslog format.

  • dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the log message. This is useful of the original format of a non-syslog-compliant message must be retained (syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages formatted according to the new IETF syslog standard (for details, see Section 2.8.2, IETF-syslog messages). If theBOM[7]character is missing, but the message is otherwise UTF-8 compliant, syslog-ng automatically adds the BOM character to the message.

follow_freq()

Type: number
Default: 1

Description:Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, syslog-ng will not attempt to use poll() on the file, but checks whether the file changed every time the follow_freq() interval (in seconds) has elapsed. Floating-point numbers (for example 1.5) can be used as well.

keep_timestamp()

Type: yes or no
Default: yes

Description:Specifies whether syslog-ng should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log_fetch_limit()

Type: number
Default: 10

Description:The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log_fetch_limit() is too high.

log_iw_size()

Type: number
Default: 1000

Description:The size of the initial window, this value is used during flow control. If the max-connections() option is set, the log_iw_size() will be divided by the number of connections, otherwise log_iw_size() is divided by 10 (the default value of the max-connections() option). The resulting number is the initial window size of each connection.

Example 6.12. Initial window size of a connection

If log_iw_size(1000) and max-connections(10), then each connection will have an initial window size of 100.

log_msg_size()

Type: number
Default: Use the global log_msg_size() option, which defaults to 8192.

Description:Specifies the maximum length of incoming log messages. Uses the value of the global option if not specified.

log_prefix() (DEPRECATED)

Type: string
Default:  

Description:A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux. NOTE: This option is deprecated. Use program_override() instead.

optional()

Type: yes or no
Default:  

Description:Instruct syslog-ng to ignore the error if a specific source cannot be initialized. No other attempts to initialize the source will be made until the configuration is reloaded. This option currently applies to the pipe(), unix-dgram, and unix-stream drivers.

pad_size()

Type: number
Default: 0

Description:Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). The syslog-ng OSE application will pad reads from the associated device to the number of bytes set in pad_size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad_size was given and the incoming message does not fit into pad_size, syslog-ng will not read anymore from this pipe and displays the following error message:

Padding was set, and couldn't read enough bytes

pipe()

Type: filename with path
Default:  

Description:The filename of the pipe to read messages from.

program_override()

Type: string
Default:  

Description:Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program_override("kernel") option in the source containing /proc/kmsg. NOTE: This option replaces the deprecated log_prefix() option.

tags()

Type: string
Default:  

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example tags("dmz", "router"). This option is available only in syslog-ng 3.1 and later.

time_zone()

Type: timezone in +/-HH:MM format
Default:  

Description:The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.



[7] The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

6.6. Collecting process accounting logs on Linux

Starting with version 3.2, syslog-ng OSE can collect process accounting logs on Linux systems.Process accounting is the method of recording and summarizing commands executed on Linux, for example, the commands being run, the user who executed the command, CPU time used by the process, exit code, and so on. When process accounting (also called pacct) is enabled on a system, the kernel writes accounting records to the /var/log/account/pacct file (might vary between different Linux distributions).

To use the pacct() driver, the following conditions must be met:

  • The syslog-ng OSE application must be compiled with the --enable-pacct option. Execute the syslog-ng -V command to check if your binary supports process accounting.

  • The pacctformat plugin must be loaded. By default, syslog-ng OSE automatically loads the available modules.

  • The scl.conf file must be included in your syslog-ng configuration:

    @include "scl.conf"
  • Process accounting must be running on the host. You can enable it with the accton command.

The pacct() driver parses the fields of the accounting logs and transforms them into name-value pairs. The fields are defined in the manual page of the accounting log file (man acct), syslog-ng OSE prepends every field with the .pacct. prefix. For example, the ac_uid field that contains the id of the user who started the process will be available under the $.pacct.ac_uid name. These can be used as macros in templates, in filters to select specific messages, and so on.

To use the pacct() driver, use the following syntax.

@version: @techversion;
@include "scl.conf"
source s_pacct { pacct(); };
...
log { source(s_pacct); destination(...); };

6.6.1. pacct() options

The pacct() driver has the following options:

file

Type: filename with path
Default: /var/log/account/pacct

Description:The file where the process accounting logs are stored — syslog-ng OSE reads accounting messages from this file.

follow_freq()

Type: number
Default: 1

Description:Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, syslog-ng will not attempt to use poll() on the file, but checks whether the file changed every time the follow_freq() interval (in seconds) has elapsed. Floating-point numbers (for example 1.5) can be used as well.

6.7. Receiving messages from external applications

The program driver starts an external application and reads messages from the standard output (stdout) of the application. It is mainly useful to receive log messages from daemons that accept incoming messages and convert them to log messages.

The program driver has a single required parameter, specifying the name of the application to start.

Declaration:
program(filename);
Example 6.13. Using the program() driver
source s_program { program("/etc/init.d/mydaemon"); };
Note

The program is restarted automatically if it exits.

6.7.1. program() source options

The program driver has the following options:

flags()

Type: empty-lines, expect-hostname, kernel, no-multi-line, no-parse, store-legacy-msghdr, syslog-protocol, validate-utf8
Default: empty set

Description:Specifies the log parsing options of the source.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ng OSE removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, syslog-ng OSE will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way syslog-ng OSE assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

    source s_dell { udp(port(2000) flags(no-hostname)); };
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the syslog, udp, unix-dgram drivers support multi-line messages; other drivers, for example, the tcp driver does not.

  • no-parse: By default, syslog-ng OSE parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The syslog-ng OSE application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MSG part of the syslog message. This flag is useful for parsing messages not complying to the syslog format.

  • dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the log message. This is useful of the original format of a non-syslog-compliant message must be retained (syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages formatted according to the new IETF syslog standard (for details, see Section 2.8.2, IETF-syslog messages). If theBOM[8]character is missing, but the message is otherwise UTF-8 compliant, syslog-ng automatically adds the BOM character to the message.

follow_freq()

Type: number
Default: 1

Description:Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, syslog-ng will not attempt to use poll() on the file, but checks whether the file changed every time the follow_freq() interval (in seconds) has elapsed. Floating-point numbers (for example 1.5) can be used as well.

keep_timestamp()

Type: yes or no
Default: yes

Description:Specifies whether syslog-ng should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log_fetch_limit()

Type: number
Default: 10

Description:The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log_fetch_limit() is too high.

log_iw_size()

Type: number
Default: 1000

Description:The size of the initial window, this value is used during flow control. If the max-connections() option is set, the log_iw_size() will be divided by the number of connections, otherwise log_iw_size() is divided by 10 (the default value of the max-connections() option). The resulting number is the initial window size of each connection.

Example 6.14. Initial window size of a connection

If log_iw_size(1000) and max-connections(10), then each connection will have an initial window size of 100.

log_msg_size()

Type: number
Default: Use the global log_msg_size() option, which defaults to 8192.

Description:Specifies the maximum length of incoming log messages. Uses the value of the global option if not specified.

log_prefix() (DEPRECATED)

Type: string
Default:  

Description:A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux. NOTE: This option is deprecated. Use program_override() instead.

optional()

Type: yes or no
Default:  

Description:Instruct syslog-ng to ignore the error if a specific source cannot be initialized. No other attempts to initialize the source will be made until the configuration is reloaded. This option currently applies to the pipe(), unix-dgram, and unix-stream drivers.

pad_size()

Type: number
Default: 0

Description:Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). The syslog-ng OSE application will pad reads from the associated device to the number of bytes set in pad_size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad_size was given and the incoming message does not fit into pad_size, syslog-ng will not read anymore from this pipe and displays the following error message:

Padding was set, and couldn't read enough bytes

program

Type: filename with path
Default:  

Description:The name of the application to start and read messages from.

program_override()

Type: string
Default:  

Description:Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program_override("kernel") option in the source containing /proc/kmsg. NOTE: This option replaces the deprecated log_prefix() option.

tags()

Type: string
Default:  

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example tags("dmz", "router"). This option is available only in syslog-ng 3.1 and later.

time_zone()

Type: timezone in +/-HH:MM format
Default:  

Description:The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.



[8] The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

6.8. Collecting messages on Sun Solaris

Solaris uses its STREAMS framework to send messages to the syslogd process. Solaris 2.5.1 and above uses an IPC called door in addition to STREAMS, to confirm the delivery of a message. The syslog-ng application supports the IPC mechanism via the door() option (see below).

Note

The sun-streams() driver must be enabled when the syslog-ng application is compiled (see ./configure --help).

The sun-streams() driver has a single required argument specifying the STREAMS device to open, and the door() option. For the list of available optional parameters, see Section 6.8.1, sun-streams() source options.

Declaration:
    sun-streams(name_of_the_streams_device door(filename_of_the_door));
Example 6.15. Using the sun-streams() driver
source s_stream { sun-streams("/dev/log" door("/etc/.syslog_door")); };

6.8.1. sun-streams() source options

The sun-streams() driver has the following options.

door()

Type: string
Default: none

Description:Specifies the filename of a door to open, needed on Solaris above 2.5.1.

flags()

Type: empty-lines, expect-hostname, kernel, no-multi-line, no-parse, store-legacy-msghdr, syslog-protocol, validate-utf8
Default: empty set

Description:Specifies the log parsing options of the source.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ng OSE removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, syslog-ng OSE will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way syslog-ng OSE assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

    source s_dell { udp(port(2000) flags(no-hostname)); };
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the syslog, udp, unix-dgram drivers support multi-line messages; other drivers, for example, the tcp driver does not.

  • no-parse: By default, syslog-ng OSE parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The syslog-ng OSE application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MSG part of the syslog message. This flag is useful for parsing messages not complying to the syslog format.

  • dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the log message. This is useful of the original format of a non-syslog-compliant message must be retained (syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages formatted according to the new IETF syslog standard (for details, see Section 2.8.2, IETF-syslog messages). If theBOM[9]character is missing, but the message is otherwise UTF-8 compliant, syslog-ng automatically adds the BOM character to the message.

follow_freq()

Type: number
Default: 1

Description:Indicates that the source should be checked periodically. This is useful for files which always indicate readability, even though no new lines were appended. If this value is higher than zero, syslog-ng will not attempt to use poll() on the file, but checks whether the file changed every time the follow_freq() interval (in seconds) has elapsed. Floating-point numbers (for example 1.5) can be used as well.

keep_timestamp()

Type: yes or no
Default: yes

Description:Specifies whether syslog-ng should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log_fetch_limit()

Type: number
Default: 10

Description:The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log_fetch_limit() is too high.

log_iw_size()

Type: number
Default: 1000

Description:The size of the initial window, this value is used during flow control. If the max-connections() option is set, the log_iw_size() will be divided by the number of connections, otherwise log_iw_size() is divided by 10 (the default value of the max-connections() option). The resulting number is the initial window size of each connection.

Example 6.16. Initial window size of a connection

If log_iw_size(1000) and max-connections(10), then each connection will have an initial window size of 100.

log_msg_size()

Type: number
Default: Use the global log_msg_size() option, which defaults to 8192.

Description:Specifies the maximum length of incoming log messages. Uses the value of the global option if not specified.

log_prefix() (DEPRECATED)

Type: string
Default:  

Description:A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux. NOTE: This option is deprecated. Use program_override() instead.

optional()

Type: yes or no
Default:  

Description:Instruct syslog-ng to ignore the error if a specific source cannot be initialized. No other attempts to initialize the source will be made until the configuration is reloaded. This option currently applies to the pipe(), unix-dgram, and unix-stream drivers.

pad_size()

Type: number
Default: 0

Description:Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). The syslog-ng OSE application will pad reads from the associated device to the number of bytes set in pad_size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad_size was given and the incoming message does not fit into pad_size, syslog-ng will not read anymore from this pipe and displays the following error message:

Padding was set, and couldn't read enough bytes

program_override()

Type: string
Default:  

Description:Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program_override("kernel") option in the source containing /proc/kmsg. NOTE: This option replaces the deprecated log_prefix() option.

tags()

Type: string
Default:  

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example tags("dmz", "router"). This option is available only in syslog-ng 3.1 and later.

time_zone()

Type: timezone in +/-HH:MM format
Default:  

Description:The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.



[9] The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

6.9. Collecting messages using the IETF syslog protocol

The syslog() driver can receive messages from the network using the standard IETF-syslog protocol (as described in RFC5424-26). UDP, TCP, and TLS-encrypted TCP can all be used to transport the messages.

Note

The syslog() driver can also receive BSD-syslog-formatted messages (described in RFC 3164, see Section 2.8.1, BSD-syslog or legacy-syslog messages) if they are sent using the IETF-syslog protocol.

In syslog-ng OSE versions 3.1 and earlier, the syslog() driver could handle only messages in the IETF-syslog (RFC 5424-26) format.

For the list of available optional parameters, see Section 6.9.1, syslog() source options.

Declaration:
        syslog(ip() port() transport() options());
Example 6.17. Using the syslog() driver

TCP source listening on the localhost on port 1999.

source s_syslog { syslog(ip(127.0.0.1) port(1999) transport("tcp")); };

UDP source with defaults.

source s_udp { syslog( transport("udp")); };

Encrypted source where the client is also authenticated. For details on the encryption settings, see Section 10.4, TLS options.

source s_syslog_tls{ syslog(
    ip(10.100.20.40)
    transport("tls")
    tls(
    peer-verify(required-trusted)
    ca_dir('/opt/syslog-ng/etc/syslog-ng/keys/ca.d/')
    key_file('/opt/syslog-ng/etc/syslog-ng/keys/server_privatekey.pem')
    cert_file('/opt/syslog-ng/etc/syslog-ng/keys/server_certificate.pem')
    )
    );};
Warning

When receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (that is, the syslog-ng OSE server or relay receiving the messages). Note that on certain platforms, for example, on Red Hat Enterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so_rcvbuf() option of the source is increased. In such cases, you will need to increase the net.core.rmem_max parameter of the host (for example, to 1024000), but do not modify net.core.rmem_default parameter.

As a general rule, increase the so_rcvbuf() so that the buffer size in kilobytes is higher than the rate of incoming messages per second. For example, to receive 2000 messages per second, set the so_rcvbuf() at least to 2 097 152 bytes.

6.9.1. syslog() source options

The syslog() driver has the following options.

flags()

Type: empty-lines, expect-hostname, kernel, no-multi-line, no-parse, store-legacy-msghdr, syslog-protocol, validate-utf8
Default: empty set

Description:Specifies the log parsing options of the source.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ng OSE removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, syslog-ng OSE will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way syslog-ng OSE assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

    source s_dell { udp(port(2000) flags(no-hostname)); };
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the syslog, udp, unix-dgram drivers support multi-line messages; other drivers, for example, the tcp driver does not.

  • no-parse: By default, syslog-ng OSE parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The syslog-ng OSE application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MSG part of the syslog message. This flag is useful for parsing messages not complying to the syslog format.

  • dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the log message. This is useful of the original format of a non-syslog-compliant message must be retained (syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages formatted according to the new IETF syslog standard (for details, see Section 2.8.2, IETF-syslog messages). If theBOM[10]character is missing, but the message is otherwise UTF-8 compliant, syslog-ng automatically adds the BOM character to the message.

host_override()

Type: string
Default:  

Description:Replaces the ${HOST} part of the message with the parameter string.

ip() or localip()

Type: string
Default: 0.0.0.0

Description:The IP address to bind to. Note that this is not the address where messages are accepted from.

ip-protocol()

Type: number
Default: 4

Description:Determines the internet protocol version of the given driver (network() or syslog()). The possible values are 4 and 6, corresponding to IPv4 and IPv6. The default value is 4.

ip_tos()

Type: number
Default: 0

Description:Specifies the Type-of-Service value of outgoing packets.

ip_ttl()

Type: number
Default: 0

Description:Specifies the Time-To-Live value of outgoing packets.

keep-alive()

Type: yes or no
Default: yes

Description:Specifies whether connections to sources should be closed when syslog-ng is forced to reload its configuration (upon the receipt of a SIGHUP signal). Note that this applies to the server (source) side of the syslog-ng connections, client-side (destination) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the destination.

keep_hostname()

Type: yes or no
Default: no

Description:Enable or disable hostname rewriting.

  • If enabled (keep_hostname(yes)), syslog-ng OSE assumes that the incoming log message was sent by the host specified in the HOST field of the message.

  • If disabled (keep_hostname(no)), syslog-ng OSE rewrites the HOST field of the message, either to the IP address (if the use_dns() parameter is set to no), or to the hostname (if the use_dns() parameter is set to yes and the IP address can be resolved to a hostname) of the host sending the message to syslog-ng OSE. For details on using name resolution in syslog-ng OSE, see Section 17.4, Using name resolution in syslog-ng.

Note

If the log message does not contain a hostname in its HOST field, syslog-ng OSE automatically adds a hostname to the message.

  • For messages received from the network, this hostname is the address of the host that sent the message (this means the address of the last hop if the message was transferred via a relay).

  • For messages received from the local host, syslog-ng OSE adds the name of the host.

This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

Note

When relaying messages, enable this option on the syslog-ng OSE server and also on every relay, otherwise syslog-ng OSE will treat incoming messages as if they were sent by the last relay.

keep_timestamp()

Type: yes or no
Default: yes

Description:Specifies whether syslog-ng should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log_fetch_limit()

Type: number
Default: 10

Description:The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log_fetch_limit() is too high.

log_iw_size()

Type: number
Default: 1000

Description:The size of the initial window, this value is used during flow control. If the max-connections() option is set, the log_iw_size() will be divided by the number of connections, otherwise log_iw_size() is divided by 10 (the default value of the max-connections() option). The resulting number is the initial window size of each connection.

Example 6.18. Initial window size of a connection

If log_iw_size(1000) and max-connections(10), then each connection will have an initial window size of 100.

log_msg_size()

Type: number
Default: Use the global log_msg_size() option, which defaults to 8192.

Description:Specifies the maximum length of incoming log messages. Uses the value of the global option if not specified.

log_prefix() (DEPRECATED)

Type: string
Default:  

Description:A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux. NOTE: This option is deprecated. Use program_override() instead.

max-connections()

Type: number
Default: 10

Description:Specifies the maximum number of simultaneous connections.

pad_size()

Type: number
Default: 0

Description:Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). The syslog-ng OSE application will pad reads from the associated device to the number of bytes set in pad_size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad_size was given and the incoming message does not fit into pad_size, syslog-ng will not read anymore from this pipe and displays the following error message:

Padding was set, and couldn't read enough bytes

port() or localport()

Type: number
Default: 601

Description:The port number to bind to.

program_override()

Type: string
Default:  

Description:Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program_override("kernel") option in the source containing /proc/kmsg. NOTE: This option replaces the deprecated log_prefix() option.

so_broadcast()

Type: yes or no
Default: no

Description:This option controls the SO_BROADCAST socket option required to make syslog-ng send messages to a broadcast address. For details, see the socket(7) manual page.

so_keepalive()

Type: yes or no
Default: no

Description:Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

so_rcvbuf()

Type: number
Default: 0

Description:Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.

Warning

When receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (that is, the syslog-ng OSE server or relay receiving the messages). Note that on certain platforms, for example, on Red Hat Enterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so_rcvbuf() option of the source is increased. In such cases, you will need to increase the net.core.rmem_max parameter of the host (for example, to 1024000), but do not modify net.core.rmem_default parameter.

As a general rule, increase the so_rcvbuf() so that the buffer size in kilobytes is higher than the rate of incoming messages per second. For example, to receive 2000 messages per second, set the so_rcvbuf() at least to 2 097 152 bytes.

so_sndbuf()

Type: number
Default: 0

Description:Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

tags()

Type: string
Default:  

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example tags("dmz", "router"). This option is available only in syslog-ng 3.1 and later.

tcp-keep-alive()

Type: yes or no
Default: no

Description:This is an obsolete alias of the so_keepalive() option.

time_zone()

Type: timezone in +/-HH:MM format
Default:  

Description:The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

transport()

Type: udp, tcp, or tls
Default: tcp

Description:Specifies the protocol used to receive messages from the source.

Warning

When receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (that is, the syslog-ng OSE server or relay receiving the messages). Note that on certain platforms, for example, on Red Hat Enterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so_rcvbuf() option of the source is increased. In such cases, you will need to increase the net.core.rmem_max parameter of the host (for example, to 1024000), but do not modify net.core.rmem_default parameter.

As a general rule, increase the so_rcvbuf() so that the buffer size in kilobytes is higher than the rate of incoming messages per second. For example, to receive 2000 messages per second, set the so_rcvbuf() at least to 2 097 152 bytes.

tls()

Type: tls options
Default: n/a

Description:This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see Section 10.4, TLS options.

use_dns()

Type: yes, no, persist_only
Default: yes

Description:Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locally from file (for example from /etc/hosts). The syslog-ng OSE application blocks on DNS queries, so enabling DNS may lead to a Denial of Service attack. To prevent DoS, protect your syslog-ng network endpoint with firewall rules, and make sure that all hosts which may get to syslog-ng are resolvable. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

use_fqdn()

Type: yes or no
Default: no

Description:Add Fully Qualified Domain Name instead of short hostname. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.



[10] The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

6.10. Collecting the system-specific log messages of a platform

Starting with version 3.2, syslog-ng OSE can automatically collect the system-specific log messages of the host on a number of platforms using the system() driver. If the system() driver is included in the syslog-ng OSE configuration file, syslog-ng OSE automatically adds the following sources to the syslog-ng OSE configuration.

Note

syslog-ng OSE versions 3.2-3.3 used an external script to generate the system() source, but this was problematic in certain situations, for example, when the host used a strict AppArmor profile. Therefore, the system() source is now generated internally in syslog-ng OSE.

The system() driver is also used in the default configuration file of syslog-ng OSE. For details on the default configuration file, see Example 4.1, The default configuration file of syslog-ng OSE.

Warning

If syslog-ng OSE does not recognize the platform it is installed on, it does not add any sources.

Platform Message source
AIX and Tru64
unix-dgram("/dev/log");
FreeBSD
unix-dgram("/var/run/log");
unix-dgram("/var/run/logpriv" perm(0600));
file("/dev/klog" follow-freq(0) program-override("kernel") flags(no-parse));

For FreeBSD versions earlier than 9.1, follow-freq(1) is used.

GNU/kFreeBSD
unix-dgram("/var/run/log");
file("/dev/klog" follow-freq(0) program-override("kernel"));
HP-UX
pipe("/dev/log" pad_size(2048));
Linux
unix-dgram("/dev/log");
file("/proc/kmsg" program-override("kernel") flags(kernel));

Note that on Linux, the so_rcvbuf option of the system() source is automatically set to 8192.

If the host is running under systemd, syslog-ng OSE reads the /run/systemd/journal/syslog socket instead of /dev/log.

Solaris 8
sun-streams("/dev/log");
Solaris 9
sun-streams("/dev/log" door("/etc/.syslog_door"));
Solaris 10
sun-streams("/dev/log" door("/var/run/syslog_door"));

Table 6.3. Sources automatically added by syslog-ng Open Source Edition


6.11. Collecting messages from remote hosts using the BSD syslog protocol

Note

The tcp(), tcp6(), udp(), and udp6() drivers will be deprecated in later versions, use the network() driver instead.

The tcp(), tcp6(), udp(), udp6() drivers can receive syslog messages conforming to RFC3164 from the network using the TCP and UDP networking protocols. The tcp6() and udp6() drivers use the IPv6 network protocol, while tcp() and udp() use IPv4.

UDP is a simple datagram oriented protocol, which provides "best effort service" to transfer messages between hosts. It may lose messages, and no attempt is made at the protocol level to retransmit such lost messages. The BSD-syslog protocol traditionally uses UDP.

TCP provides connection-oriented service, which basically means that the path of the messages is flow-controlled. Along this path, each message is acknowledged, and retransmission is done for lost packets. Generally it is safer to use TCP, because lost connections can be detected, and no messages get lost, assuming that the TCP connection does not break. When a TCP connection is broken the 'in-transit' messages that were sent by syslog-ng but not yet received on the other side are lost. (Basically these messages are still sitting in the socket buffer of the sending host and syslog-ng has no information about the fate of these messages).

The tcp() and udp() drivers do not have any required parameters. By default they bind to the 0.0.0.0:514 address, which means that syslog-ng will listen on all available interfaces, port 514. To limit accepted connections to only one interface, use the localip() parameter as described below. For the list of available optional parameters, see Section 6.11.1, tcp(), tcp6(), udp() and udp6() source options.

Declaration:
            tcp([options]);
            udp([options]);
Note

The TCP port 514 is reserved for use with rshell, so select a different port if syslog-ng and rshell is used at the same time.

If you specify a multicast bind address to udp() and udp6(), syslog-ng will automatically join the necessary multicast group. TCP does not support multicasting.

The syslog-ng application supports TLS (Transport Layer Security, also known as SSL) for the tcp() and tcp6() drivers. For details, see the TLS-specific options below and Section 10.2, Encrypting log messages with TLS. For the list of available optional parameters, see Section 6.11.1, tcp(), tcp6(), udp() and udp6() source options.

Tip

The syslog() driver also supports TLS-encrypted connections.

Example 6.19. Using the udp() and tcp() drivers

A simple udp() source with default settings.

source s_udp { udp(); };# An UDP source with default settings.

A TCP source listening on the localhost interface, with a limited number of connections allowed.

source s_tcp { tcp(ip(127.0.0.1) port(1999) max-connections(10)); };

A TCP source listening on a TLS-encrypted channel.

source s_tcp { tcp(ip(127.0.0.1) port(1999)
           tls(peer-verify('required-trusted')
               key_file('/opt/syslog-ng/etc/syslog-ng/syslog-ng.key')
           cert_file('/opt/syslog-ng/etc/syslog-ng/syslog-ng.crt')));
};

A TCP source listening for messages using the IETF-syslog message format. Note that for transferring IETF-syslog messages, generally you are recommended to use the syslog() driver on both the client and the server, as it uses both the IETF-syslog message format and the protocol. For details, see Section 6.9, Collecting messages using the IETF syslog protocol.

source s_tcp_syslog { tcp(ip(127.0.0.1) port(1999) flags(syslog-protocol) ); };

6.11.1. tcp(), tcp6(), udp() and udp6() source options

The tcp(), tcp6(), udp(), udp6() drivers can receive messages conforming to RFC3164 from the network using the TCP and UDP networking protocols.

The following options are valid for tcp(), tcp6(), udp(), and udp6() drivers:

Warning

When receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (that is, the syslog-ng OSE server or relay receiving the messages). Note that on certain platforms, for example, on Red Hat Enterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so_rcvbuf() option of the source is increased. In such cases, you will need to increase the net.core.rmem_max parameter of the host (for example, to 1024000), but do not modify net.core.rmem_default parameter.

As a general rule, increase the so_rcvbuf() so that the buffer size in kilobytes is higher than the rate of incoming messages per second. For example, to receive 2000 messages per second, set the so_rcvbuf() at least to 2 097 152 bytes.

encoding()

Type: string
Default:  

Description:Specifies the characterset (encoding, for example UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command.

flags()

Type: empty-lines, expect-hostname, kernel, no-multi-line, no-parse, store-legacy-msghdr, syslog-protocol, validate-utf8
Default: empty set

Description:Specifies the log parsing options of the source.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ng OSE removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, syslog-ng OSE will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way syslog-ng OSE assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

    source s_dell { udp(port(2000) flags(no-hostname)); };
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the syslog, udp, unix-dgram drivers support multi-line messages; other drivers, for example, the tcp driver does not.

  • no-parse: By default, syslog-ng OSE parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The syslog-ng OSE application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MSG part of the syslog message. This flag is useful for parsing messages not complying to the syslog format.

  • dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the log message. This is useful of the original format of a non-syslog-compliant message must be retained (syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages formatted according to the new IETF syslog standard (for details, see Section 2.8.2, IETF-syslog messages). If theBOM[11]character is missing, but the message is otherwise UTF-8 compliant, syslog-ng automatically adds the BOM character to the message.

host_override()

Type: string
Default:  

Description:Replaces the ${HOST} part of the message with the parameter string.

ip() or localip()

Type: string
Default: 0.0.0.0

Description:The IP address to bind to. Note that this is not the address where messages are accepted from.

keep-alive()

Type: yes or no
Default: yes

Description:Specifies whether connections to sources should be closed when syslog-ng is forced to reload its configuration (upon the receipt of a SIGHUP signal). Note that this applies to the server (source) side of the syslog-ng connections, client-side (destination) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the destination.

keep_hostname()

Type: yes or no
Default: no

Description:Enable or disable hostname rewriting.

  • If enabled (keep_hostname(yes)), syslog-ng OSE assumes that the incoming log message was sent by the host specified in the HOST field of the message.

  • If disabled (keep_hostname(no)), syslog-ng OSE rewrites the HOST field of the message, either to the IP address (if the use_dns() parameter is set to no), or to the hostname (if the use_dns() parameter is set to yes and the IP address can be resolved to a hostname) of the host sending the message to syslog-ng OSE. For details on using name resolution in syslog-ng OSE, see Section 17.4, Using name resolution in syslog-ng.

Note

If the log message does not contain a hostname in its HOST field, syslog-ng OSE automatically adds a hostname to the message.

  • For messages received from the network, this hostname is the address of the host that sent the message (this means the address of the last hop if the message was transferred via a relay).

  • For messages received from the local host, syslog-ng OSE adds the name of the host.

This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

Note

When relaying messages, enable this option on the syslog-ng OSE server and also on every relay, otherwise syslog-ng OSE will treat incoming messages as if they were sent by the last relay.

keep_timestamp()

Type: yes or no
Default: yes

Description:Specifies whether syslog-ng should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log_fetch_limit()

Type: number
Default: 10

Description:The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log_fetch_limit() is too high.

log_iw_size()

Type: number
Default: 1000

Description:The size of the initial window, this value is used during flow control. If the max-connections() option is set, the log_iw_size() will be divided by the number of connections, otherwise log_iw_size() is divided by 10 (the default value of the max-connections() option). The resulting number is the initial window size of each connection.

Example 6.20. Initial window size of a connection

If log_iw_size(1000) and max-connections(10), then each connection will have an initial window size of 100.

log_msg_size()

Type: number
Default: Use the global log_msg_size() option, which defaults to 8192.

Description:Specifies the maximum length of incoming log messages. Uses the value of the global option if not specified.

log_prefix() (DEPRECATED)

Type: string
Default:  

Description:A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux. NOTE: This option is deprecated. Use program_override() instead.

max-connections()

Type: number
Default: 10

Description:Specifies the maximum number of simultaneous connections.

Note that the udp() and udp6() drivers do not support this option.

pad_size()

Type: number
Default: 0

Description:Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). The syslog-ng OSE application will pad reads from the associated device to the number of bytes set in pad_size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad_size was given and the incoming message does not fit into pad_size, syslog-ng will not read anymore from this pipe and displays the following error message:

Padding was set, and couldn't read enough bytes

port() or localport()

Type: number
Default: 514

Description:The port number to bind to.

program_override()

Type: string
Default:  

Description:Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program_override("kernel") option in the source containing /proc/kmsg. NOTE: This option replaces the deprecated log_prefix() option.

so_keepalive()

Type: yes or no
Default: no

Description:Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

so_rcvbuf()

Type: number
Default: 0

Description:Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.

Warning

When receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (that is, the syslog-ng OSE server or relay receiving the messages). Note that on certain platforms, for example, on Red Hat Enterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so_rcvbuf() option of the source is increased. In such cases, you will need to increase the net.core.rmem_max parameter of the host (for example, to 1024000), but do not modify net.core.rmem_default parameter.

As a general rule, increase the so_rcvbuf() so that the buffer size in kilobytes is higher than the rate of incoming messages per second. For example, to receive 2000 messages per second, set the so_rcvbuf() at least to 2 097 152 bytes.

tcp-keep-alive()

Type: yes or no
Default: no

Description:This is an obsolete alias of the so_keepalive() option.

tcp-keepalive-intvl()

Type: number [seconds]
Default: 0

Description:Specifies the interval (number of seconds) between subsequential keepalive probes, regardless of the traffic exchanged in the connection. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_intvl. The default value is 0, which means using the kernel default.

Warning

The tcp-keepalive-time(), tcp-keepalive-probes(), and tcp-keepalive-intvl() options only work on platforms which support the TCP_KEEPCNT, TCP_KEEPIDLE,and TCP_KEEPINTVL setsockopts. Currently, this is Linux.

A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() * tcp-keepalive-probes() seconds.

Available in syslog-ng OSE version 3.4 and later.

tcp-keepalive-probes()

Type: number
Default: 0

Description:Specifies the number of unacknowledged probes to send before considering the connection dead. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_probes. The default value is 0, which means using the kernel default.

Warning

The tcp-keepalive-time(), tcp-keepalive-probes(), and tcp-keepalive-intvl() options only work on platforms which support the TCP_KEEPCNT, TCP_KEEPIDLE,and TCP_KEEPINTVL setsockopts. Currently, this is Linux.

A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() * tcp-keepalive-probes() seconds.

Available in syslog-ng OSE version 3.4 and later.

tcp-keepalive-time()

Type: number [seconds]
Default: 0

Description:Specifies the interval (in seconds) between the last data packet sent and the first keepalive probe. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_time. The default value is 0, which means using the kernel default.

Warning

The tcp-keepalive-time(), tcp-keepalive-probes(), and tcp-keepalive-intvl() options only work on platforms which support the TCP_KEEPCNT, TCP_KEEPIDLE,and TCP_KEEPINTVL setsockopts. Currently, this is Linux.

A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() * tcp-keepalive-probes() seconds.

Available in syslog-ng OSE version 3.4 and later.

tags()

Type: string
Default:  

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example tags("dmz", "router"). This option is available only in syslog-ng 3.1 and later.

time_zone()

Type: timezone in +/-HH:MM format
Default:  

Description:The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.

tls()

Type: tls options
Default: n/a

Description:This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see Section 10.4, TLS options.

use_dns()

Type: yes, no, persist_only
Default: yes

Description:Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locally from file (for example from /etc/hosts). The syslog-ng OSE application blocks on DNS queries, so enabling DNS may lead to a Denial of Service attack. To prevent DoS, protect your syslog-ng network endpoint with firewall rules, and make sure that all hosts which may get to syslog-ng are resolvable. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

use_fqdn()

Type: yes or no
Default: no

Description:Add Fully Qualified Domain Name instead of short hostname. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.



[11] The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

6.12. Collecting messages from UNIX domain sockets

The unix-stream() and unix-dgram() drivers open an AF_UNIX socket and start listening on it for messages. The unix-stream() driver is primarily used on Linux and uses SOCK_STREAM semantics (connection oriented, no messages are lost); while unix-dgram() is used on BSDs and uses SOCK_DGRAM semantics: this may result in lost local messages if the system is overloaded.

To avoid denial of service attacks when using connection-oriented protocols, the number of simultaneously accepted connections should be limited. This can be achieved using the max-connections() parameter. The default value of this parameter is quite strict, you might have to increase it on a busy system.

Both unix-stream and unix-dgram have a single required argument that specifies the filename of the socket to create. For the list of available optional parameters, see Section 6.12.1, unix-stream() and unix-dgram() source options

Declaration:
            unix-stream(filename [options]);
            unix-dgram(filename [options]);
Note

syslogd on Linux originally used SOCK_STREAM sockets, but some distributions switched to SOCK_DGRAM around 1999 to fix a possible DoS problem. On Linux you can choose to use whichever driver you like as syslog clients automatically detect the socket type being used.

Example 6.21. Using the unix-stream() and unix-dgram() drivers
source s_stream { unix-stream("/dev/log" max-connections(10)); };
source s_dgram { unix-dgram("/var/run/log"); };

6.12.1. unix-stream() and unix-dgram() source options

These two drivers behave similarly: they open an AF_UNIX socket and start listening on it for messages. The following options can be specified for these divers:

encoding()

Type: string
Default:  

Description:Specifies the characterset (encoding, for example UTF-8) of messages using the legacy BSD-syslog protocol. To list the available character sets on a host, execute the iconv -l command.

flags()

Type: empty-lines, expect-hostname, kernel, no-multi-line, no-parse, store-legacy-msghdr, syslog-protocol, validate-utf8
Default: empty set

Description:Specifies the log parsing options of the source.

  • empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ng OSE removes empty lines automatically.

  • expect-hostname: If the expect-hostname flag is enabled, syslog-ng OSE will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.

  • kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.

  • no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way syslog-ng OSE assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:

    source s_dell { udp(port(2000) flags(no-hostname)); };
  • no-multi-line: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the syslog, udp, unix-dgram drivers support multi-line messages; other drivers, for example, the tcp driver does not.

  • no-parse: By default, syslog-ng OSE parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The syslog-ng OSE application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MSG part of the syslog message. This flag is useful for parsing messages not complying to the syslog format.

  • dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the log message. This is useful of the original format of a non-syslog-compliant message must be retained (syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.

  • syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.

  • validate-utf8: The validate-utf8 flag enables encoding-verification for messages formatted according to the new IETF syslog standard (for details, see Section 2.8.2, IETF-syslog messages). If theBOM[12]character is missing, but the message is otherwise UTF-8 compliant, syslog-ng automatically adds the BOM character to the message.

group()

Type: string
Default: root

Description:Set the gid of the socket.

host_override()

Type: string
Default:  

Description:Replaces the ${HOST} part of the message with the parameter string.

keep-alive()

Type: yes or no
Default: yes

Description:Selects whether to keep connections open when syslog-ng is restarted; cannot be used with unix-dgram().

keep_timestamp()

Type: yes or no
Default: yes

Description:Specifies whether syslog-ng should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

log_fetch_limit()

Type: number
Default: 10

Description:The maximum number of messages fetched from a source during a single poll loop. The destination queues might fill up before flow-control could stop reading if log_fetch_limit() is too high.

log_iw_size()

Type: number
Default: 1000

Description:The size of the initial window, this value is used during flow control. If the max-connections() option is set, the log_iw_size() will be divided by the number of connections, otherwise log_iw_size() is divided by 10 (the default value of the max-connections() option). The resulting number is the initial window size of each connection.

Example 6.22. Initial window size of a connection

If log_iw_size(1000) and max-connections(10), then each connection will have an initial window size of 100.

log_msg_size()

Type: number
Default: Use the global log_msg_size() option, which defaults to 8192.

Description:Specifies the maximum length of incoming log messages. Uses the value of the global option if not specified.

log_prefix() (DEPRECATED)

Type: string
Default:  

Description:A string added to the beginning of every log message. It can be used to add an arbitrary string to any log source, though it is most commonly used for adding kernel: to the kernel messages on Linux. NOTE: This option is deprecated. Use program_override() instead.

max-connections()

Type: number
Default: 256

Description:Limits the number of simultaneously open connections. Cannot be used with unix-dgram().

optional()

Type: yes or no
Default:  

Description:Instruct syslog-ng to ignore the error if a specific source cannot be initialized. No other attempts to initialize the source will be made until the configuration is reloaded. This option currently applies to the pipe(), unix-dgram, and unix-stream drivers.

owner()

Type: string
Default: root

Description:Set the uid of the socket.

pad_size()

Type: number
Default: 0

Description:Specifies input padding. Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). The syslog-ng OSE application will pad reads from the associated device to the number of bytes set in pad_size(). Mostly used on HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad_size was given and the incoming message does not fit into pad_size, syslog-ng will not read anymore from this pipe and displays the following error message:

Padding was set, and couldn't read enough bytes

perm()

Type: number
Default: 0666

Description:Set the permission mask. For octal numbers prefix the number with '0', for example: use 0755 for rwxr-xr-x.

program_override()

Type: string
Default:  

Description:Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program_override("kernel") option in the source containing /proc/kmsg. NOTE: This option replaces the deprecated log_prefix() option.

so_keepalive()

Type: yes or no
Default: no

Description:Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

so_rcvbuf()

Type: number
Default: 0

Description:Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.

Warning

When receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (that is, the syslog-ng OSE server or relay receiving the messages). Note that on certain platforms, for example, on Red Hat Enterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so_rcvbuf() option of the source is increased. In such cases, you will need to increase the net.core.rmem_max parameter of the host (for example, to 1024000), but do not modify net.core.rmem_default parameter.

As a general rule, increase the so_rcvbuf() so that the buffer size in kilobytes is higher than the rate of incoming messages per second. For example, to receive 2000 messages per second, set the so_rcvbuf() at least to 2 097 152 bytes.

tags()

Type: string
Default:  

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosed between double quotes. When adding multiple tags, separate them with comma, for example tags("dmz", "router"). This option is available only in syslog-ng 3.1 and later.

time_zone()

Type: timezone in +/-HH:MM format
Default:  

Description:The default timezone for messages read from the source. Applies only if no timezone is specified within the message itself.



[12] The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

Chapter 7. Sending and storing log messages — destinations and destination drivers

A destination is where a log message is sent if the filtering rules match. Similarly to sources, destinations consist of one or more drivers, each defining where and how messages are sent.

Tip

If no drivers are defined for a destination, all messages sent to the destination are discarded. This is equivalent to omitting the destination from the log statement.

To define a destination, add a destination statement to the syslog-ng configuration file using the following syntax:

destination <identifier> {
            destination-driver(params); destination-driver(params); ... };
Example 7.1. A simple destination statement

The following destination statement sends messages to the TCP port 1999 of the 10.1.2.3 host.

destination d_demo_tcp { tcp("10.1.2.3" port(1999)); };

If name resolution is configured, the hostname of the target server can be used as well.

destination d_tcp { tcp("target_host" port(1999); localport(999)); };
Note

Sources and destinations are initialized only when they are used in a log statement. For example, syslog-ng OSE connects a remote destination only if the destination is used in a log statement. For details on creating log statements, see Chapter 8, Routing messages: log paths and filters.

The following table lists the destination drivers available in syslog-ng.

Name Description
amqp() Publishes messages using the AMQP (Advanced Message Queuing Protocol).
file() Writes messages to the specified file.
pipe() Writes messages to the specified named pipe.
program() Forks and launches the specified program, and sends messages to its standard input.
smtp() Sends e-mail messages to the specified recipients.
sql() Sends messages into an SQL database. In addition to the standard syslog-ng packages, the sql() destination requires database-specific packages to be installed. Refer to the section appropriate for your platform in Chapter 3, Installing syslog-ng.
syslog() Sends messages to the specified remote host using the IETF-syslog protocol. The IETF standard supports message transport using the UDP, TCP, and TLS networking protocols.
tcp() and tcp6() Sends messages to the specified TCP port of a remote host using the BSD-syslog protocol over IPv4 and IPv6, respectively.
udp() and udp6() Sends messages to the specified UDP port of a remote host using the BSD-syslog protocol over IPv4 and IPv6, respectively.
unix-dgram() Sends messages to the specified unix socket in SOCK_DGRAM style (BSD).
unix-stream() Sends messages to the specified unix socket in SOCK_STREAM style (Linux).
usertty() Sends messages to the terminal of the specified user, if the user is logged in.

Table 7.1. Destination drivers available in syslog-ng


7.1. Publishing messages using AMQP

The amqp() driver publishes messages using the AMQP (Advanced Message Queuing Protocol). syslog-ng OSE supports AMQP versions 0.9.1 and 1.0. The syslog-ng OSE amqp() driver supports persistence, and every available exchange types.

The name-value pairs selected with the value-pairs() option will be sent as AMQP headers, while the body of the AMQP message is empty by default (but you can add custom content using the body() option). Publishing the name-value pairs as headers makes it possible to use the Headers exchange-type and subscribe only to interesting log streams. This solution is more flexible than using the routing_key() option.

For the list of available parameters, see Section 7.1.1, amqp() destination options.

Declaration:
amqp( host("<amqp-server-address>") )
Example 7.2. Using the amqp() driver

The following example shows the default values of the available options.

destination d_amqp {
    amqp(
        vhost("/")
        host("127.0.0.1")
        port(5672)
        exchange("syslog")
        exchange_type("header")
        routing_key("")
        body("")
        persistent(yes)
        value-pairs(
            scope("selected-macros" "nv-pairs" "sdata")
        )
    );
};

7.1.1. amqp() destination options

The amqp() driver publishes messages using the AMQP (Advanced Message Queuing Protocol).

The amqp() destination has the following options:

body()

Type: string
Default: empty string

Description:The body of the AMQP message. You can also use macros and templates.

exchange()

Type: string
Default: syslog

Description:The name of the AMQP exchange where syslog-ng OSE sends the message. Exchanges take a message and route it into zero or more queues.

exchange-declare()

Type: yes|no
Default: no

Description:By default, syslog-ng OSE does not create non-existing exchanges. Use the exchange-declare(yes) option to automatically create exchanges.

exchange-type()

Type: direct|fanout|topic|headers
Default: header

Description:The type of the AMQP exchange.

host()

Type: hostname or IP address
Default: 127.0.0.1

Description:The hostname or IP address of the AMQP server.

password()

Type: string
Default: n/a

Description:The password used to authenticate on the AMQP server.

persistent()

Type: yes|no
Default: yes

Description:If this option is enabled, the AMQP server or broker will store the messages on its hard disk. That way, the messages will be retained if the AMQP server is restarted, if the message queue is set to be durable on the AMQP server.

port()

Type: number
Default: 5672

Description:The port number of the AMQP server.

routing-key()

Type: string
Default: empty string

Description:Specifies a routing key for the exchange. The routing key selects certain messages published to an exchange to be routed to the bound queue. In other words, the routing key acts like a filter. The routing key can include macros and templates.

username()

Type: string
Default: empty string

Description:The username used to authenticate on the AMQP server.

value-pairs()

Type: parameter list of the value-pairs() option
Default:
scope("selected-macros" "nv-pairs")
  exclude("R_*")
  exclude("S_*")
  exclude("HOST_FROM")
  exclude("LEGACY_MSGHDR")
  exclude("MSG")
  exclude("SDATA")

Description:The value-pairs() option creates structured name-value pairs from the data and metadata of the log message. For details on using value-pairs(), see Section 2.10, Structuring macros, metadata, and other value-pairs.

Note

Empty keys are not logged.

vhost()

Type: string
Default: /

Description:The name of the AMQP virtual host to send the messages to.

7.2. Storing messages in plain-text files

The file driver is one of the most important destination drivers in syslog-ng. It allows to output messages to the specified text file, or to a set of files.

The destination filename may include macros which get expanded when the message is written, thus a simple file() driver may create several files: for example, syslog-ng OSE can store the messages of client hosts in a separate file for each host. For more information on available macros see Section 11.1.5, Macros of syslog-ng OSE.

If the expanded filename refers to a directory which does not exist, it will be created depending on the create_dirs() setting (both global and a per destination option).

The file() has a single required parameter that specifies the filename that stores the log messages. For the list of available optional parameters, see Section 7.2.1, file() destination options.

Declaration:
            file(filename options());
Example 7.3. Using the file() driver
destination d_file { file("/var/log/messages" ); };
Example 7.4. Using the file() driver with macros in the file name and a template for the message
destination d_file {
        file("/var/log/${YEAR}.${MONTH}.${DAY}/messages"
             template("${HOUR}:${MIN}:${SEC} ${TZ} ${HOST} [${LEVEL}] ${MSG} ${MSG}\n")
             template-escape(no));
};
Note

When using this destination, update the configuration of your log rotation program to rotate these files. Otherwise, the log files can become very large.

Also, after rotating the log files, reload syslog-ng OSE using the syslog-ng-ctl reload command, or use another method to send a SIGHUP to syslog-ng OSE.

Warning

Since the state of each created file must be tracked by syslog-ng, it consumes some memory for each file. If no new messages are written to a file within 60 seconds (controlled by the time-reap() global option), it is closed, and its state is freed.

Exploiting this, a DoS attack can be mounted against the system. If the number of possible destination files and its needed memory is more than the amount available on the syslog-ng server.

The most suspicious macro is ${PROGRAM}, where the number of possible variations is rather high. Do not use the ${PROGRAM} macro in insecure environments.

7.2.1. file() destination options

The file() driver outputs messages to the specified text file, or to a set of files. The file() destination has the following options:

Warning

When creating several thousands separate log files, syslog-ng might not be able to open the required number of files. This might happen for example when using the ${HOST} macro in the filename while receiving messages from a large number of hosts. To overcome this problem, adjust the --fd-limit command-line parameter of syslog-ng or the global ulimit parameter of your host. For setting the --fd-limit command-line parameter of syslog-ng see the syslog-ng(8) manual page. For setting the ulimit parameter of the host, see the documentation of your operating system.

create_dirs()

Type: yes or no
Default: no

Description:Enable creating non-existing directories.

dir_group()

Type: string
Default: Use the global settings

Description:The group of the directories created by syslog-ng. To preserve the original properties of an existing directory, use the option without specifying an attribute: dir_group().

dir_owner()

Type: string
Default: Use the global settings

Description:The owner of the directories created by syslog-ng. To preserve the original properties of an existing directory, use the option without specifying an attribute: dir_owner().

dir_perm()

Type: number
Default: Use the global settings

Description:The permission mask of directories created by syslog-ng. Log directories are only created if a file after macro expansion refers to a non-existing directory, and directory creation is enabled (see also the create_dirs() option). For octal numbers prefix the number with 0, for example use 0755 for rwxr-xr-x.

To preserve the original properties of an existing directory, use the option without specifying an attribute: dir_perm(). Note that when creating a new directory without specifying attributes for dir_perm(), the default permission of the directories is masked with the umask of the parent process (typically 0022).

flags()

Type: no_multi_line, syslog-protocol
Default: empty set

Description:Flags influence the behavior of the destination driver.

  • no-multi-line: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line.

  • syslog-protocol: The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver, and that the syslog driver automatically adds the frame header to the messages.

flush_lines()

Type: number
Default: Use global setting.

Description:Specifies how many lines are sent to a destination at a time. The syslog-ng OSE application waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the destination, but also increases message latency. The latency can be limited by the use of the flush-timeout option.

flush_timeout() (DEPRECATED)

Type: time in milliseconds
Default: Use global setting.

Description:This is a deprecated option. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. For details, see the flush_lines option.

frac_digits()

Type: number
Default: Value of the global option (which defaults to 0)

Description:The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.

fsync()

Type: yes or no
Default: no

Description:Forces an fsync() call on the destination fd after each write. Note: enabling this option may seriously degrade performance.

group()

Type: string
Default: Use the global settings

Description:Set the group of the created file to the one specified. To preserve the original properties of an existing file, use the option without specifying an attribute: group().

local_time_zone()

Type: name of the timezone or the timezone offset
Default: The local timezone.

Description:Sets the timezone used when expanding filename and tablename templates. The timezone can be specified as using the name of the (for example time_zone("Europe/Budapest")), or as the timezone offset (for example +01:00). The valid timezone names are listed under the /usr/share/zoneinfo directory.

log_fifo_size()

Type: number
Default: Use global setting.

Description:The number of messages that the output queue can store.

mark_mode()

Accepted values: internal | dst-idle | host-idle | periodical | none | global
Default:

internal for pipe, program drivers

none for file, unix-dgram, unix-stream drivers

global for syslog, tcp, udp destinations

host-idle for global option

Description:The mark-mode() option can be set for the following destination drivers: file(), program(), unix_dgram(), unix_stream(), udp(), udp6(), tcp(), tcp6(), pipe(), syslog() and in global option.

  • internal: When internal mark mode is selected, internal source should be placed in the log path as this mode does not generate mark by itself at the destination. This mode only yelds the mark messages from internal source. This is the mode as syslog-ng OSE 3.3 worked. MARK will be generated by internal source if there was NO traffic on local sources:

    file(), pipe(), unix-stream(), unix-dgram(), program()

  • dst-idle: Sends mark signal if there was NO traffic on destination drivers. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • host-idle: Sends mark signal if there was NO local message on destination drivers. For example mark is generated even if messages were received from tcp. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • periodical: Sends mark signal perodically, regardless of traffic on destination driver. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to the drivers where none is the default value, then none will be used.

  • global: Destination driver uses the global mark-mode setting. The syslog-ng interprets syntax error if the global mark-mode is global.

Note

In case of dst-idle, host-idle and periodical; MARK message will not be written in the destination, if it is not open yet.

Available in syslog-ng OSE 3.4 and later.

overwrite_if_older()

Type: number
Default: 0

Description:If set to a value higher than 0, syslog-ng checks when the file was last modified before starting to write into the file. If the file is older than the specified amount of time (in seconds), then syslog-ng removes the existing file and opens a new file with the same name. In combination with for example the ${WEEKDAY} macro, this can be used for simple log rotation, in case not all history has to be kept. (Note that in this weekly log rotation example if its Monday 00:01, then the file from last Monday is not seven days old, because it was probably last modified shortly before 23:59 last Monday, so it is actually not even six days old. So in this case, set the overwrite_if_older() parameter to a-bit-less-than-six-days, for example, to 518000 seconds.

owner()

Type: string
Default: Use the global settings

Description:Set the owner of the created file to the one specified. To preserve the original properties of an existing file, use the option without specifying an attribute: owner().

pad_size()

Type: number
Default: 0

Description:If set, syslog-ng OSE will pad output messages to the specified size (in bytes). Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes).

Warning

Hazard of data loss! If the size of the incoming message is larger than the previously set pad_size value, syslog-ng will truncate the message to the specified size. Therefore, all message content above that size will be lost.

perm()

Type: number
Default: Use the global settings

Description:The permission mask of the file if it is created by syslog-ng. For octal numbers prefix the number with 0, for example use 0755 for rwxr-xr-x.

To preserve the original properties of an existing file, use the option without specifying an attribute: perm().

suppress()

Type: seconds
Default: 0 (disabled)

Description:If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.

template()

Type: string
Default: A format conforming to the default logfile format.

Description:Specifies a template defining the logformat to be used in the destination. Macros are described in Section 11.1.5, Macros of syslog-ng OSE. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.

template_escape()

Type: yes or no
Default: no

Description:Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.

throttle()

Type: number
Default: 0

Description:Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using large enough buffers as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.

time_zone()

Type: timezone in +/-HH:MM format
Default: unspecified

Description:Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.

ts_format()

Type: rfc3164, bsd, rfc3339, iso
Default: Use the global option (which defaults to rfc3164)

Description:Override the global timestamp format (set in the global ts_format() parameter) for the specific destination. For details, see Section ts_format().

7.3. Storing messages in a MongoDB database

The mongodb() driver sends messages to a MongoDB database. MongoDB is a schema-free, document-oriented database. For the list of available optional parameters, see Section 7.3.1, mongodb() destination options.

Declaration:
                mongodb(parameters);

The mongodb() driver does not support creating indexes, as that can be a very complex operation in MongoDB. If needed, the administrator of the MongoDB database must ensure that indexes are created on the collections.

Example 7.5. Using the mongodb() driver

The following example creates a mongodb() destination using only default values.

destination d_mongodb {
    mongodb();
};

The following example displays the default values, and is equivalent with the previous example.

destination d_mongodb {
    mongodb(
        servers("localhost:27017")
        database("syslog")
        collection("messages")
        value-pairs(
            scope("selected-macros" "nv-pairs")
            exclude("R_*")
            exclude("S_*")
            exclude("HOST_FROM")
            exclude("LEGACY_MSGHDR")
            exclude("MSG")
            exclude("SDATA")
        )
    );
};

7.3.1. mongodb() destination options

The mongodb() driver sends messages to a MongoDB database. MongoDB is a schema-free, document-oriented database.

The mongodb() destination has the following options:

collection()

Type: string
Default: messages

Description:The name of the MongoDB collection where the log messages are stored (collections are similar to SQL tables). Note that the name of the collection must not start with a dollar sign ($), and that it may contain dot (.) characters.

Warning

Hazard of data loss! The syslog-ng OSE application does not verify that the specified collection name does not contain invalid characters. If you specify a collection with an invalid name, the log messages sent to the MongoDB database will be irrevocably lost without any warning.

database()

Type: string
Default: syslog

Description:The name of the MongoDB database where the log messages are stored. Note that the name of the database must not start with a dollar sign ($) and it cannot contain dot (.) characters.

Warning

Hazard of data loss! The syslog-ng OSE application does not verify that the specified database name does not contain invalid characters. If you specify a database with an invalid name, the log messages sent to the MongoDB database will be irrevocably lost without any warning.

frac_digits()

Type: number
Default: Value of the global option (which defaults to 0)

Description:The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.

host() (DEPRECATED)

Type: hostname or IP address
Default: localhost

OBSOLETE: use the servers() option instead.

Description:Hostname or IP address of the database server. When specifying an IP address, IPv4 (for example, 192.168.0.1) or IPv6 (for example, [::1]) can be used as well.

Note

If you specify host="localhost" (or use the default), syslog-ng OSE will use a socket to connect to the local database server. Use host="127.0.0.1" to force TCP communication between syslog-ng OSE and the local database server.

log_fifo_size()

Type: number
Default: Use global setting.

Description:The number of messages that the output queue can store.

port() (DEPRECATED)

Type: number
Default: 27017

OBSOLETE: use the servers() option instead.

Description:The port number to connect to.

path()

Type: string
Default: empty

Description:If the path() option is set, syslog-ng OSE will connect to the database using specified socket. Note that you cannot set the path() and the servers() options at the same time.

safe-mode()

Type: yes or no
Default: no

Description:If safe-mode() is enabled, syslog-ng OSE performs an extra check after each insert to verify that the insert succeeded. The insert is successful only if this second check is successful. Note that enabling this option reduces the performance of the driver.

servers()

Type: list of hostname:port pairs
Default: 127.0.0.1:27017

Description:Specifies the hostname or IP address and the port number of the database server. When specifying an IP address, IPv4 (for example, 192.168.0.1) or IPv6 (for example, [::1]) can be used as well.

To send the messages to a MongoDB replicaset, specify the addresses of the database servers as a comma-separated list, for example: servers(192.168.1.1:27017,192.168.3.3:27017)

Note

If you specify servers("localhost") (or use the default), syslog-ng OSE will use a socket to connect to the local database server. Use servers("127.0.0.1") to force TCP communication between syslog-ng OSE and the local database server.

value-pairs()

Type: parameter list of the value-pairs() option
Default:
scope("selected-macros" "nv-pairs")
  exclude("R_*")
  exclude("S_*")
  exclude("HOST_FROM")
  exclude("LEGACY_MSGHDR")
  exclude("MSG")
  exclude("SDATA")

Description:The value-pairs() option creates structured name-value pairs from the data and metadata of the log message. For details on using value-pairs(), see Section 2.10, Structuring macros, metadata, and other value-pairs.

Note

Empty keys are not logged.

7.4. Sending messages to a remote logserver using the RFC3164 protocol

The network() destination driver can send syslog messages conforming to RFC3164 from the network using the TCP, TLS, and UDP networking protocols.

Example 7.6. Using the network() driver

TCP source listening on the localhost on port 2222 without using the network() driver.

destination d_tcp6 {
        tcp6(
            ip("::1")
            port(2222)
        );
        };

TCP source listening on the localhost on port 2222 using the network() driver.

destination d_network6 {
        network(
            ip("::1")
            port(2222)
            transport(tcp)
            ip-protocol(6)
        );
};
Note

For details on the tcp(), tcp6(), udp(), udp6() drivers, see Section 7.10, Sending messages to a remote logserver using the legacy BSD-syslog protocol.

7.4.1. network() destination options

The network() driver sends messages to a remote host (for example a syslog-ng server or relay) on the local intranet or internet using the RFC3164 syslog protocol (for details about the protocol, see Section 2.8.1, BSD-syslog or legacy-syslog messages). The network() driver supports sending messages using the UDP, TCP, or the encrypted TLS networking protocols.

These destinations have the following options:

flags()

Type: no_multi_line, syslog-protocol
Default: empty set

Description:Flags influence the behavior of the destination driver.

  • no-multi-line: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line.

  • syslog-protocol: The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver, and that the syslog driver automatically adds the frame header to the messages.

flush_lines()

Type: number
Default: Use global setting.

Description:Specifies how many lines are sent to a destination at a time. The syslog-ng OSE application waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the destination, but also increases message latency. The latency can be limited by the use of the flush-timeout option.

flush_timeout() (DEPRECATED)

Type: time in milliseconds
Default: Use global setting.

Description:This is a deprecated option. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. For details, see the flush_lines option.

frac_digits()

Type: number
Default: Value of the global option (which defaults to 0)

Description:The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.

ip_tos()

Type: number
Default: 0

Description:Specifies the Type-of-Service value of outgoing packets.

ip_ttl()

Type: number
Default: 0

Description:Specifies the Time-To-Live value of outgoing packets.

keep-alive()

Type: yes or no
Default: yes

Description:Specifies whether connections to destinations should be closed when syslog-ng is reloaded. Note that this applies to the client (destination) side of the syslog-ng connections, server-side (source) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the source.

localip()

Type: string
Default: 0.0.0.0

Description:The IP address to bind to before connecting to target.

localport()

Type: number
Default: 0

Description:The port number to bind to. Messages are sent from this port.

log_fifo_size()

Type: number
Default: Use global setting.

Description:The number of messages that the output queue can store.

mark_freq()

Accepted values: number
Default: 1200

Description:An alias for the obsolete mark() option, retained for compatibility with syslog-ng version 1.6.x. The number of seconds between two MARK messages. MARK messages are generated when there was no message traffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. The mark-freq can be set for global option and/or every MARK capable destination driver if mark-mode is periodical or dst-idle or host-idle. If mark-freq is not defined in the destination, then the mark-freq will be inherited from the global options. If the destination uses internal mark-mode, then the global mark-freq will be valid (does not matter what mark-freq set in the destination side).

mark_mode()

Accepted values: internal | dst-idle | host-idle | periodical | none | global
Default:

internal for pipe, program drivers

none for file, unix-dgram, unix-stream drivers

global for syslog, tcp, udp destinations

host-idle for global option

Description:The mark-mode() option can be set for the following destination drivers: file(), program(), unix_dgram(), unix_stream(), udp(), udp6(), tcp(), tcp6(), pipe(), syslog() and in global option.

  • internal: When internal mark mode is selected, internal source should be placed in the log path as this mode does not generate mark by itself at the destination. This mode only yelds the mark messages from internal source. This is the mode as syslog-ng OSE 3.3 worked. MARK will be generated by internal source if there was NO traffic on local sources:

    file(), pipe(), unix-stream(), unix-dgram(), program()

  • dst-idle: Sends mark signal if there was NO traffic on destination drivers. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • host-idle: Sends mark signal if there was NO local message on destination drivers. For example mark is generated even if messages were received from tcp. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • periodical: Sends mark signal perodically, regardless of traffic on destination driver. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to the drivers where none is the default value, then none will be used.

  • global: Destination driver uses the global mark-mode setting. The syslog-ng interprets syntax error if the global mark-mode is global.

Note

In case of dst-idle, host-idle and periodical; MARK message will not be written in the destination, if it is not open yet.

Available in syslog-ng OSE 3.4 and later.

port() or destport()

Type: number
Default: 601

Description:The port number to connect to. Note that the default port numbers used by syslog-ng do not comply with the latest RFC which was published after the release of syslog-ng 3.0.2, therefore the default port numbers will change in the future releases.

so_broadcast()

Type: yes or no
Default: no

Description:This option controls the SO_BROADCAST socket option required to make syslog-ng send messages to a broadcast address. For details, see the socket(7) manual page.

so_keepalive()

Type: yes or no
Default: no

Description:Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

so_rcvbuf()

Type: number
Default: 0

Description:Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.

so_sndbuf()

Type: number
Default: 0

Description:Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

spoof_source()

Type: yes or no
Default: no

Description:Enables source address spoofing. This means that the host running syslog-ng generates UDP packets with the source IP address matching the original sender of the message. It is useful when you want to perform some kind of preprocessing via syslog-ng then forward messages to your central log management solution with the source address of the original sender. This option only works for UDP destinations though the original message can be received by TCP as well. This option is only available if syslog-ng was compiled using the --enable-spoof-source configuration option.

suppress()

Type: seconds
Default: 0 (disabled)

Description:If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.

tcp-keepalive-intvl()

Type: number [seconds]
Default: 0

Description:Specifies the interval (number of seconds) between subsequential keepalive probes, regardless of the traffic exchanged in the connection. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_intvl. The default value is 0, which means using the kernel default.

Warning

The tcp-keepalive-time(), tcp-keepalive-probes(), and tcp-keepalive-intvl() options only work on platforms which support the TCP_KEEPCNT, TCP_KEEPIDLE,and TCP_KEEPINTVL setsockopts. Currently, this is Linux.

A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() * tcp-keepalive-probes() seconds.

Available in syslog-ng OSE version 3.4 and later.

tcp-keepalive-probes()

Type: number
Default: 0

Description:Specifies the number of unacknowledged probes to send before considering the connection dead. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_probes. The default value is 0, which means using the kernel default.

Warning

The tcp-keepalive-time(), tcp-keepalive-probes(), and tcp-keepalive-intvl() options only work on platforms which support the TCP_KEEPCNT, TCP_KEEPIDLE,and TCP_KEEPINTVL setsockopts. Currently, this is Linux.

A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() * tcp-keepalive-probes() seconds.

Available in syslog-ng OSE version 3.4 and later.

tcp-keepalive-time()

Type: number [seconds]
Default: 0

Description:Specifies the interval (in seconds) between the last data packet sent and the first keepalive probe. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_time. The default value is 0, which means using the kernel default.

Warning

The tcp-keepalive-time(), tcp-keepalive-probes(), and tcp-keepalive-intvl() options only work on platforms which support the TCP_KEEPCNT, TCP_KEEPIDLE,and TCP_KEEPINTVL setsockopts. Currently, this is Linux.

A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() * tcp-keepalive-probes() seconds.

Available in syslog-ng OSE version 3.4 and later.

template()

Type: string
Default: A format conforming to the default logfile format.

Description:Specifies a template defining the logformat to be used in the destination. Macros are described in Section 11.1.5, Macros of syslog-ng OSE. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.

template_escape()

Type: yes or no
Default: no

Description:Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.

throttle()

Type: number
Default: 0

Description:Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using large enough buffers as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.

time_zone()

Type: timezone in +/-HH:MM format
Default: unspecified

Description:Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.

tls()

Type: tls options
Default: n/a

Description:This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see Section 10.4, TLS options.

transport()

Type: udp, tcp, or tls
Default: tcp

Description:Specifies the protocol used to send messages to the destination server.

ts_format()

Type: rfc3164, bsd, rfc3339, iso
Default: Use the global option (which defaults to rfc3164)

Description:Override the global timestamp format (set in the global ts_format() parameter) for the specific destination. For details, see Section ts_format().

7.5. Sending messages to named pipes

The pipe() driver sends messages to a named pipe like /dev/xconsole.

The pipe driver has a single required parameter, specifying the filename of the pipe to open. The filename can include macros. For the list of available optional parameters, see Section 7.5.1, pipe() destination options.

Declaration:
            pipe(filename);
Warning

Starting with syslog-ng OSE 3.0.2, pipes are created automatically. In earlier versions, you had to create the pipe using the mkfifo(1) command.

Example 7.7. Using the pipe() driver
destination d_pipe { pipe("/dev/xconsole"); };

7.5.1. pipe() destination options

This driver sends messages to a named pipe like /dev/xconsole.

The pipe() destination has the following options:

flags()

Type: no_multi_line, syslog-protocol
Default: empty set

Description:Flags influence the behavior of the destination driver.

  • no-multi-line: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line.

  • syslog-protocol: The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver, and that the syslog driver automatically adds the frame header to the messages.

flush_lines()

Type: number
Default: Use global setting.

Description:Specifies how many lines are sent to a destination at a time. The syslog-ng OSE application waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the destination, but also increases message latency. The latency can be limited by the use of the flush-timeout option.

flush_timeout() (DEPRECATED)

Type: time in milliseconds
Default: Use global setting.

Description:This is a deprecated option. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. For details, see the flush_lines option.

frac_digits()

Type: number
Default: Value of the global option (which defaults to 0)

Description:The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.

group()

Type: string
Default: Use the global settings

Description:Set the group of the created file to the one specified. To preserve the original properties of an existing file, use the option without specifying an attribute: group().

log_fifo_size()

Type: number
Default: Use global setting.

Description:The number of messages that the output queue can store.

mark_mode()

Accepted values: internal | dst-idle | host-idle | periodical | none | global
Default:

internal for pipe, program drivers

none for file, unix-dgram, unix-stream drivers

global for syslog, tcp, udp destinations

host-idle for global option

Description:The mark-mode() option can be set for the following destination drivers: file(), program(), unix_dgram(), unix_stream(), udp(), udp6(), tcp(), tcp6(), pipe(), syslog() and in global option.

  • internal: When internal mark mode is selected, internal source should be placed in the log path as this mode does not generate mark by itself at the destination. This mode only yelds the mark messages from internal source. This is the mode as syslog-ng OSE 3.3 worked. MARK will be generated by internal source if there was NO traffic on local sources:

    file(), pipe(), unix-stream(), unix-dgram(), program()

  • dst-idle: Sends mark signal if there was NO traffic on destination drivers. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • host-idle: Sends mark signal if there was NO local message on destination drivers. For example mark is generated even if messages were received from tcp. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • periodical: Sends mark signal perodically, regardless of traffic on destination driver. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to the drivers where none is the default value, then none will be used.

  • global: Destination driver uses the global mark-mode setting. The syslog-ng interprets syntax error if the global mark-mode is global.

Note

In case of dst-idle, host-idle and periodical; MARK message will not be written in the destination, if it is not open yet.

Available in syslog-ng OSE 3.4 and later.

owner()

Type: string
Default: Use the global settings

Description:Set the owner of the created file to the one specified. To preserve the original properties of an existing file, use the option without specifying an attribute: owner().

pad_size()

Type: number
Default: 0

Description:If set, syslog-ng OSE will pad output messages to the specified size (in bytes). Some operating systems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes).

Warning

Hazard of data loss! If the size of the incoming message is larger than the previously set pad_size value, syslog-ng will truncate the message to the specified size. Therefore, all message content above that size will be lost.

perm()

Type: number
Default: 0600

Description:The permission mask of the pipe. For octal numbers prefix the number with '0', for example: use 0755 for rwxr-xr-x.

suppress()

Type: seconds
Default: 0 (disabled)

Description:If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.

template()

Type: string
Default: A format conforming to the default logfile format.

Description:Specifies a template defining the logformat to be used in the destination. Macros are described in Section 11.1.5, Macros of syslog-ng OSE. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.

template_escape()

Type: yes or no
Default: no

Description:Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.

throttle()

Type: number
Default: 0

Description:Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using large enough buffers as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.

time_zone()

Type: timezone in +/-HH:MM format
Default: unspecified

Description:Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.

ts_format()

Type: rfc3164, bsd, rfc3339, iso
Default: Use the global option (which defaults to rfc3164)

Description:Override the global timestamp format (set in the global ts_format() parameter) for the specific destination. For details, see Section ts_format().

7.6. Sending messages to external applications

The program() driver starts an external application or script and sends the log messages to its standard input (stdin).

The program() driver has a single required parameter, specifying a program name to start. The program is executed with the help of the current shell, so the command may include both file patterns and I/O redirections. For the list of available optional parameters, see Section 7.6.1, program() destination options.

Declaration:
            program(command_to_run);
Note

The syslog-ng OSE application executes program destinations through the standard system shell. If the system shell is not bash and you experience problems with the program destination, try changing the /bin/sh link to /bin/bash.

Note
  • The syslog-ng OSE application automatically restarts the external program if it exits for reliability reasons. However it is not recommended to launch programs for single messages, because if the message rate is high, launching several instances of an application might overload the system, resulting in Denial of Service.

  • Certain external applications buffer the log messages, which might cause unexpected latency and other problems. For example, if you send the log messages to an external Perl script, Perl uses a line buffer for terminal output and block buffer otherwise. You might want to disable buffering in the external application.

Warning

The syslog-ng OSE application must be able to start and restart the external program, and have the necessary permissions to do so. For example, if your host is running AppArmor, you might have to modify your AppArmor configuration to enable syslog-ng OSE to execute external applications.

Note that the message format does not include the priority and facility values by default. To add these values, specify a template for the program destination, as shown in the following example.

Example 7.8. Using the program() destination driver
destination d_prog { program("/bin/script" template("<${PRI}>${DATE} ${HOST} ${MSG}\n") ); };

7.6.1. program() destination options

This driver starts an external application or script and sends the log messages to its standard input (stdin).

The program() destination has the following options:

flags()

Type: no_multi_line, syslog-protocol
Default: empty set

Description:Flags influence the behavior of the destination driver.

  • no-multi-line: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line.

  • syslog-protocol: The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver, and that the syslog driver automatically adds the frame header to the messages.

flush_lines()

Type: number
Default: Use global setting.

Description:Specifies how many lines are sent to a destination at a time. The syslog-ng OSE application waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the destination, but also increases message latency. The latency can be limited by the use of the flush-timeout option.

flush_timeout() (DEPRECATED)

Type: time in milliseconds
Default: Use global setting.

Description:This is a deprecated option. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. For details, see the flush_lines option.

frac_digits()

Type: number
Default: Value of the global option (which defaults to 0)

Description:The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.

log_fifo_size()

Type: number
Default: Use global setting.

Description:The number of messages that the output queue can store.

mark_mode()

Accepted values: internal | dst-idle | host-idle | periodical | none | global
Default:

internal for pipe, program drivers

none for file, unix-dgram, unix-stream drivers

global for syslog, tcp, udp destinations

host-idle for global option

Description:The mark-mode() option can be set for the following destination drivers: file(), program(), unix_dgram(), unix_stream(), udp(), udp6(), tcp(), tcp6(), pipe(), syslog() and in global option.

  • internal: When internal mark mode is selected, internal source should be placed in the log path as this mode does not generate mark by itself at the destination. This mode only yelds the mark messages from internal source. This is the mode as syslog-ng OSE 3.3 worked. MARK will be generated by internal source if there was NO traffic on local sources:

    file(), pipe(), unix-stream(), unix-dgram(), program()

  • dst-idle: Sends mark signal if there was NO traffic on destination drivers. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • host-idle: Sends mark signal if there was NO local message on destination drivers. For example mark is generated even if messages were received from tcp. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • periodical: Sends mark signal perodically, regardless of traffic on destination driver. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to the drivers where none is the default value, then none will be used.

  • global: Destination driver uses the global mark-mode setting. The syslog-ng interprets syntax error if the global mark-mode is global.

Note

In case of dst-idle, host-idle and periodical; MARK message will not be written in the destination, if it is not open yet.

Available in syslog-ng OSE 3.4 and later.

suppress()

Type: seconds
Default: 0 (disabled)

Description:If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.

template()

Type: string
Default: A format conforming to the default logfile format.

Description:Specifies a template defining the logformat to be used in the destination. Macros are described in Section 11.1.5, Macros of syslog-ng OSE. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.

template_escape()

Type: yes or no
Default: no

Description:Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.

throttle()

Type: number
Default: 0

Description:Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using large enough buffers as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.

time_zone()

Type: timezone in +/-HH:MM format
Default: unspecified

Description:Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.

ts_format()

Type: rfc3164, bsd, rfc3339, iso
Default: Use the global option (which defaults to rfc3164)

Description:Override the global timestamp format (set in the global ts_format() parameter) for the specific destination. For details, see Section ts_format().

7.7. Generating SMTP messages (e-mail) from logs

The smtp() driver sends e-mail messages triggered by log messages. The smtp() driver uses SMTP, without needing external applications. You can customize the main fields of the e-mail, add extra headers, send the e-mail to multiple recipients, and so on.

The subject(), body(), and header() fields may include macros which get expanded in the e-mail. For more information on available macros see Section 11.1.5, Macros of syslog-ng OSE.

The smtp() driver has the following required parameters: host(), port(), from(), to(), subject(), and body(). For the list of available optional parameters, see Section 7.7.1, smtp() destination options.

Note

The smtp() destination driver is available only in syslog-ng OSE 3.4 and later.

Declaration:
        smtp(host() port() from() to() subject() body() options());
Example 7.9. Using the smtp() driver

The following example defines an smtp() destination using only the required parameters.

destination d_smtp {
    smtp(
        host("localhost")
        port(25)
        from("syslog-ng alert service" "noreply@example.com")
        to("Admin #1" "admin1@example.com")
        subject("[ALERT] Important log message of $LEVEL condition received from $HOST/$PROGRAM!")
        body("Hi!\nThe syslog-ng alerting service detected the following important log message:\n $MSG\n-- \nsyslog-ng\n")
    );
};

The following example sets some optional parameters as well.

destination d_smtp {
    smtp(
        host("localhost")
        port(25)
        from("syslog-ng alert service" "noreply@example.com")
        to("Admin #1" "admin1@example.com")
        to("Admin #2" "admin2@example.com")
        cc("Admin BOSS" "admin.boss@example.com")
        bcc("Blind CC" "blindcc@example.com")
        subject("[ALERT] Important log message of $LEVEL condition received from $HOST/$PROGRAM!")
        body("Hi!\nThe syslog-ng alerting service detected the following important log message:\n $MSG\n-- \nsyslog-ng\n")
        header("X-Program", "$PROGRAM")
        );
};
Example 7.10. Simple e-mail alerting with the smtp() driver

The following example sends an e-mail alert if the eth0 network interface of the host is down.

filter f_linkdown {
    match("eth0: link down" value("MESSAGE"));
};
destination d_alert {
    smtp(
        host("localhost") port(25)
        from("syslog-ng alert service" "syslog@localhost")
        reply_to("Admins" "root@localhost")
        to("Ennekem" "me@localhost")
        subject("[SYSLOG ALERT]: eth0 link down")
        body("Syslog received an alert:\n$MSG")
        );
};

log {
    source(s_local);
    filter(f_linkdown);
    destination(d_alert);
};

7.7.1. smtp() destination options

The smtp() sends e-mail messages using SMTP, without needing external applications. The smtp() destination has the following options:

body()

Type: string
Default: n/a

Description:The BODY field of the e-mail. You can also use macros in the string. Use \n to start a new line. For example:

body("syslog-ng OSE received the following alert from $HOST:\n$MSG")

bcc()

Type: string
Default: n/a

Description:The BCC recipient of the e-mail (contents of the BCC field). You can specify the e-mail address, or the name and the e-mail address. Set the bcc() option multiple times to send the e-mail to multiple recipients. For example:

bcc("admin@example.com")

or

bcc("Admin" "admin@example.com")

or

bcc("Admin" "admin@example.com")
bcc("Admin2" "admin2@example.com")

cc()

Type: string
Default: n/a

Description:The CC recipient of the e-mail (contents of the CC field). You can specify the e-mail address, or the name and the e-mail address. Set the cc() option multiple times to send the e-mail to multiple recipients. For example:

cc("admin@example.com")

or

cc("Admin" "admin@example.com")

or

cc("Admin" "admin@example.com")
cc("Admin2" "admin2@example.com")

from()

Type: string
Default: n/a

Description:The sender of the e-mail (contents of the FROM field). You can specify the e-mail address, or the name and the e-mail address. For example:

from("admin@example.com")

or

from("Admin" "admin@example.com")

If you specify the from() option multiple times, the last value will be used. Instead of the from() option, you can also use sender(), which is just an alias of the from() option.

header()

Type: string
Default: n/a

Description:Adds an extra header to the e-mail with the specified name and content. The first parameter sets the name of the header, the second one its value. The value of the header can contain macros. Set the header() option multiple times to add multiple headers. For example:

header("X-Program", "$PROGRAM")

When using the header option, note the following points:

  • Do not use the header() option to set the values of headers that have dedicated options. Use it only to add extra headers.

  • If you set the same custom header multiple times, only the first will be added to the e-mail, other occurrences will be ignored.

  • It is not possible to set the DATE header.

host()

Type: hostname or IP address
Default: n/a

Description:Hostname or IP address of the SMTP server.

Note

If you specify host="localhost", syslog-ng OSE will use a socket to connect to the local SMTP server. Use host="127.0.0.1" to force TCP communication between syslog-ng OSE and the local SMTP server.

log_fifo_size()

Type: number
Default: Use global setting.

Description:The number of messages that the output queue can store.

port()

Type: number
Default: 25

Description:The port number of the SMTP server.

reply-to()

Type: string
Default: n/a

Description:Replies of the recipient will be sent to this address (contents of the REPLY-TO field). You can specify the e-mail address, or the name and the e-mail address. Set the reply-to() option multiple times to send the e-mail to multiple recipients. For example:

reply-to("admin@example.com")

or

reply-to("Admin" "admin@example.com")

or

reply-to("Admin" "admin@example.com")
reply-to("Admin2" "admin2@example.com")

subject()

Type: string
Default: n/a

Description:The SUBJECT field of the e-mail. You can also use macros. For example:

subject("[SYSLOG ALERT]: Critical error message received from $HOST")

If you specify the subject() option multiple times, the last value will be used.

to()

Type: string
Default: localhost

Description:The recipient of the e-mail (contents of the TO field). You can specify the e-mail address, or the name and the e-mail address. Set the to() option multiple times to send the e-mail to multiple recipients. For example:

to("admin@example.com")

or

to("Admin" "admin@example.com")

or

to("Admin" "admin@example.com")
to("Admin2" "admin2@example.com")

7.8. Storing messages in an SQL database

The sql() driver sends messages into an SQL database. Currently the Microsoft SQL (MSSQL), MySQL, Oracle, PostgreSQL, and SQLite databases are supported.

Declaration:
    sql(database_type host_parameters database_parameters [options]);

The sql() driver has the following required parameters: type(), database(), table, columns(), and values.

Warning

The syslog-ng application requires read and write access to the SQL table, otherwise it cannot verify that the destination table exists.

Currently the syslog-ng application has default schemas for the different databases and uses these defaults if the database schema (for example columns and column types) is not defined in the configuration file. However, these schemas will be deprecated and specifying the exact database schema will be required in later versions of syslog-ng.

The table and value parameters can include macros to create tables and columns dynamically (for details, see Section 11.1.5, Macros of syslog-ng OSE).

Warning

When using macros in table names, note that some databases limit the maximum allowed length of table names. Consult the documentation of the database for details.

Inserting the records into the database is performed by a separate thread. The syslog-ng application automatically performs the escaping required to insert the messages into the database.

Example 7.11. Using the sql() driver

The following example sends the log messages into a PostgreSQL database running on the logserver host. The messages are inserted into the logs database, the name of the table includes the exact date and the name of the host sending the messages. The syslog-ng application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges.

destination d_sql {
  sql(type(pgsql)
  host("logserver") username("syslog-ng") password("password")
  database("logs")
  table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}")
  columns("datetime", "host", "program", "pid", "message")
  values("{$R_DATE}", "${HOST}", "${PROGRAM}", "${PID}", "${MSGONLY}")
  indexes("datetime", "host", "program", "pid", "message"));
  };

The following example specifies the type of the database columns as well:

destination d_sql {
  sql(type(pgsql)
  host("logserver") username("syslog-ng") password("password")
  database("logs")
  table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}")
  columns("datetime varchar(16)", "host varchar(32)", "program  varchar(20)", "pid varchar(8)", "message  varchar(200)")
  values("${R_DATE}", "${HOST}", "${PROGRAM}", "${PID}", "${MSGONLY}")
  indexes("datetime", "host", "program", "pid", "message"));
};

7.8.1. Using the sql() driver with an Oracle database

The Oracle sql destination has some special aspects that are important to note.

  • The hostname of the database server is set in the tnsnames.ora file, not in the host parameter of the sql() destination.

    If the tnsnames.ora file is not located in the /etc directory (or in the /var/opt/oracle directory on Solaris), set the following Oracle-related environment variables, so syslog-ng OSE will find the file: ORACLE_BASE, ORACLE_HOME, and ORACLE_SID. For details, see the documentation of the Oracle Instant Client.

  • You cannot use the same database() settings in more than one destination, because the database() option of the SQL driver is just a reference to the connection string of the tnsnames.ora file. To overcome this problem, you can duplicate the connections in the tnsnames.ora file under a different name, and use a different table in each Oracle destination in syslog-ng OSE.

  • As certain database versions limit the maximum length of table names, macros in the table names should be used with care.

  • In the current version of syslog-ng OSE, the types of database columns must be explicitly set for the Oracle destination. The column used to store the text part of the syslog messages should be able to store messages as long as the longest message permitted by syslog-ng, therefore it is usually recommended to use the varchar2 or clob column type. (The maximum length of the messages can be set using the log_msg_size() option.) For details, see the following example.

  • The Oracle Instant Client used by syslog-ng OSE supports only the following character sets:

    • Single-byte character sets: US7ASCII, WE8DEC, WE8MSWIN1252, and WE8ISO8859P1

    • Unicode character sets: UTF8, AL16UTF16, and AL32UTF8

Example 7.12. Using the sql() driver with an Oracle database

The following example sends the log messages into an Oracle database running on the logserver host, which must be set in the /etc/tnsnames.ora file. The messages are inserted into the LOGS database, the name of the table includes the exact date when the messages were sent. The syslog-ng application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges.

destination d_sql {
  sql(type(oracle)
  username("syslog-ng") password("password")
  database("LOGS")
  table("msgs_${R_YEAR}${R_MONTH}${R_DAY}")
  columns("datetime varchar(16)", "host varchar(32)", "program varchar(32)", "pid varchar(8)", "message varchar2")
  values("${R_DATE}", "${HOST}", "${PROGRAM}", "${PID}", "${MSGONLY}")
  indexes("datetime", "host", "program", "pid", "message"));
  };

The Oracle Instant Client retrieves the address of the database server from the /etc/tnsnames.ora file. Edit or create this file as needed for your configuration. A sample is provided below.

LOGS =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)
(HOST = logserver)
(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = EXAMPLE.SERVICE)
)
)

7.8.2. Using the sql() driver with a Microsoft SQL database

The mssql database driver can access Microsoft SQL (MSSQL) destinations. This driver has some special aspects that are important to note.

  • The date format used by the MSSQL database must be explicitly set in the /etc/locales.conf file of the syslog-ng server. For details, see the following example.

  • As certain database versions limit the maximum length of table names, macros in the table names should be used with care.

  • In the current version of syslog-ng OSE, the types of database columns must be explicitly set for the MSSQL destination.

    Warning

    The following column types cannot be used in MSSQL destinations: nchar, nvarchar, ntext, and xml.

  • The column used to store the text part of the syslog messages should be able to store messages as long as the longest message permitted by syslog-ng. The varchar column type can store maximum 4096 bytes-long messages. The maximum length of the messages can be set using the log_msg_size() option. For details, see the following example.

  • Remote access for SQL users must be explicitly enabled on the Microsoft Windows host running the Microsoft SQL Server. For details, see Procedure 3.3, Configuring Microsoft SQL Server to accept logs from syslog-ng.

Example 7.13. Using the sql() driver with an MSSQL database

The following example sends the log messages into an MSSQL database running on the logserver host. The messages are inserted into the syslogng database, the name of the table includes the exact date when the messages were sent. The syslog-ng application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges.

destination d_mssql {
sql(type(mssql) host("logserver") port("1433")
  username("syslogng") password("syslogng") database("syslogng")
  table("msgs_${R_YEAR}${R_MONTH}${R_DAY}")columns("datetime varchar(16)", "host varchar(32)",
  "program varchar(32)", "pid varchar(8)", "message varchar(4096)")
  values("${R_DATE}", "${HOST}", "${PROGRAM}", "${PID}", "${MSGONLY}")
  indexes("datetime", "host", "program", "pid"));
  };

The date format used by the MSSQL database must be explicitly set in the /etc/locales.conf file of the syslog-ng server. Edit or create this file as needed for your configuration. A sample is provided below.

[default]
date = "%Y-%m-%d %H:%M:%S"

7.8.3. The way syslog-ng interacts with the database

Used SQL operations by syslog-ng. 

Create table:

  • If the given table does not exist, syslog-ng tries to create it with the given column types.

  • The syslog-ng OSE application automatically creates the required tables and columns, if the user account used to connect to the database has the required privileges.

  • If syslog-ng cannot create or alter a table, it tries to do it again when reach the next time_reopen.

Alter table:

  • If the table structure is different from given structure in an existing table, syslog-ng tries to add columns in this table but never will delete or modify an existing column.

  • If syslog-ng OSE cannot create or alter a table, it tries to do it again when reach the next time_reopen.

  • The syslog-ng OSE application requires read and write access to the SQL table, otherwise it cannot verify that the destination table exists.

Insert table:

  • Insert new records in a table.

  • Inserting the records into the database is performed by a separate thread.

  • The syslog-ng OSE application automatically performs the escaping required to insert the messages into the database.

  • If insert returns with error, syslog-ng tries to insert the message +two times by default, then drops it. Retrying time is the value of time_reopen().

Encoding. 

The syslog-ng OSE application uses UTF-8 by default when writes logs into database.

Start/stop and reload. 

Start:

  • The syslog-ng OSE application will connect to database automatically after starting regardless existing incoming messages.

Stop:

  • The syslog-ng OSE application will close the connection to database before shutting down.

Possibility of losing logs:

  • The syslog-ng OSE application cannot lose logs during shutting down if disk buffer was given and it is not full yet.

  • The syslog-ng OSE application cannot lose logs during shutting down if disk buffer was not given.

Reload:

  • The syslog-ng OSE application will close the connection to database if it received SIGHUP signal (reload).

  • It will reconnect to the database when it tries to send a new message to this database again.

Macros: 

The value of ${SEQNUM} macro will be overrided by sql driver regardless of local or relayed incoming message.

It will be grown continously.

7.8.3.1. MySQL-specific interaction methods

To specify the socket to use, set and export the MYSQL_UNIX_PORT environment variable, for example MYSQL_UNIX_PORT=/var/lib/mysql/mysql.sock; export MYSQL_UNIX_PORT.

7.8.3.2. MsSQL-specific interaction methods

In SQL Server 2005 this restriction is lifted - kind of. The total length of all key columns in an index cannot exceed 900 bytes.

If you are using null() in your configuration, be sure that the columns allow NULL to insert. Give the column as the following example: "datetime varchar(16) NULL".

The date format used by the MSSQL database must be explicitly set in the /etc/locales.conf file of the syslog-ng server. [default] date = "%Y-%m-%d %H:%M:%S".

7.8.4. sql() destination options

This driver sends messages into an SQL database. The sql() destination has the following options:

columns()

Type: string list
Default: "date", "facility", "level", "host", "program", "pid", "message"

Description:Name of the columns storing the data in fieldname [dbtype] format. The [dbtype] parameter is optional, and specifies the type of the field. By default, syslog-ng creates text columns. Note that not every database engine can index text fields.

Warning

The following column types cannot be used in MSSQL destinations: nchar, nvarchar, ntext, and xml.

database()

Type: string
Default: logs

Description:Name of the database that stores the logs. Macros cannot be used in database name. Also, when using an Oracle database, you cannot use the same database() settings in more than one destination.

dbd-option()

Type: string
Default: empty string

Description:Specify database options that are set whenever syslog-ng OSE connects to the database server. Consult the documentation of your database server for details on the available options. Syntax:

dbd-option(OPTION_NAME VALUE)

OPTION_NAME is always a string, VALUE is a string or a number. For example:

dbd-option("null.sleep.connect" 1)
dbd-option("null.sleep.query" 5)

flags()

Type: list of flags
Default: empty string

Description:Flags related to the sql() destination.

  • dont-create-tables: Enable this flag to prevent syslog-ng OSE from creating non-existing database tables automatically. The syslog-ng OSE application typically has to create tables if you use macros in the table names. Available in syslog-ng OSE version 3.2 and later.

  • explicit-commits: By default, syslog-ng OSE commits every log message to the target database individually. When the explicit-commits option is enabled, messages are committed in batches. This improves the performance, but results in some latency, as the messages are not immediately sent to the database. The size and frequency of batched commits can be set using the flush_lines and flush_timeout parameters. The explicit-commits option is available in syslog-ng OSE version 3.2 and later.

Example 7.14. Setting flags for SQL destinations

The following example sets the dont-create-tables and explicit-commits flags for an sql() destination.

flags(dont-create-tables,explicit-commits)

flush_lines()

Type: number
Default: Use global setting.

Description:Specifies how many lines are sent to a destination at a time. The syslog-ng OSE application waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the destination, but also increases message latency. The latency can be limited by the use of the flush-timeout option.

flush_timeout() (DEPRECATED)

Type: time in milliseconds
Default: Use global setting.

Description:This is a deprecated option. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. For details, see the flush_lines option.

frac_digits()

Type: number
Default: Value of the global option (which defaults to 0)

Description:The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.

host()

Type: hostname or IP address
Default: n/a

Description:Hostname of the database server. Note that Oracle destinations do not use this parameter, but retrieve the hostname from the /etc/tnsnames.ora file.

Note

If you specify host="localhost", syslog-ng will use a socket to connect to the local database server. Use host="127.0.0.1" to force TCP communication between syslog-ng and the local database server.

To specify the socket to use, set and export the MYSQL_UNIX_PORT environment variable, for example MYSQL_UNIX_PORT=/var/lib/mysql/mysql.sock; export MYSQL_UNIX_PORT.

indexes()

Type: string list
Default: "date", "facility", "host", "program"

Description:The list of columns that are indexed by the database to speed up searching. To disable indexing for the destination, include the empty indexes() parameter in the destination, simply omitting the indexes parameter will cause syslog-ng to request indexing on the default columns.

The syslog-ng OSE application will create the name of indexes automaticaly with the following method:

  • In case of MsSQL, PostgreSQL, MySQL or SQLite or (Oracle but tablename < 30 characters): {table}_{column}_idx.

  • In case of Oracle and tablename > 30 characters: md5sum of {table}_{column}-1 and the first character will be replaced by "i" character and the md5sum will be truncated to 30 characters.

local_time_zone()

Type: name of the timezone or the timezone offset
Default: The local timezone.

Description:Sets the timezone used when expanding filename and tablename templates. The timezone can be specified as using the name of the (for example time_zone("Europe/Budapest")), or as the timezone offset (for example +01:00). The valid timezone names are listed under the /usr/share/zoneinfo directory.

log_fifo_size()

Type: number
Default: Use global setting.

Description:The number of messages that the output queue can store.

null()

Type: string
Default:  

Description:If the content of a column matches the string specified in the null() parameter, the contents of the column will be replaced with an SQL NULL value. If unset (by default), the option does not match on any string. For details, see the Example 7.15, Using SQL NULL values.

Example 7.15. Using SQL NULL values

The null() parameter of the SQL driver can be used to replace the contents of a column with a special SQL NULL value. To replace every column that contains an empty string with NULL, use the null("") option, for example

destination d_sql {
                    sql(type(pgsql)
                    host("logserver") username("syslog-ng") password("password")
                    database("logs")
                    table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}")
                    columns("datetime", "host", "program", "pid", "message")
                    values("${R_DATE}", "${HOST}", "${PROGRAM}", "${PID}", "${MSGONLY}")
                    indexes("datetime", "host", "program", "pid", "message")
                    null(""));
                    };

To replace only a specific column (for example pid) if it is empty, assign a default value to the column, and use this default value in the null() parameter:

destination d_sql {
                    sql(type(pgsql)
                    host("logserver") username("syslog-ng") password("password")
                    database("logs")
                    table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}")
                    columns("datetime", "host", "program", "pid", "message")
                    values("${R_DATE}", "${HOST}", "${PROGRAM}", "${PID:-@@NULL@@}", "${MSGONLY}")
                    indexes("datetime", "host", "program", "pid", "message")
                    null("@@NULL@@"));
                    };

Ensure that the default value you use does not appear in the actual log messages, because other occurrences of this string will be replaced with NULL as well.

password()

Type: string
Default: n/a

Description:Password of the database user.

port()

Type: number
Default: 1433 TCP for MSSQL, 3306 TCP for MySQL, 1521 for Oracle, and 5432 TCP for PostgreSQL

Description:The port number to connect to.

retry_sql_inserts

Type: number
Default: 3

Description:The number of insertion attempts. If syslog-ng OSE could not insert a message into the database, it will repeat the attempt until the number of attempts reaches retry_sql_inserts, then drops the line. For example, syslog-ng OSE will try to insert a message maximum three times by default (once for first insertion and twice if the first insertion was failed).

session-statements()

Type: comma-separated list of SQL statements
Default: empty string

Description:Specifies one or more SQL-like statement which is executed after syslog-ng OSE has successfully connected to the database. For example:

session-statements("SET COLLATION_CONNECTION='utf8_general_ci'")
Warning

The syslog-ng OSE application does not validate or limit the contents of customized queries. Consequently, queries performed with a user with write-access can potentially modify or even harm the database. Use customized queries with care, and only for your own responsibility.

table()

Type: string
Default: messages

Description:Name of the database table to use (can include macros). When using macros, note that some databases limit the length of table names.

time_zone()

Type: timezone in +/-HH:MM format
Default: unspecified

Description:Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.

type()

Type: mssql, mysql, oracle, pgsql, or sqlite3
Default: mysql

Description:Specifies the type of the database, that is, the DBI database driver to use. Use the mssql option to send logs to an MSSQL database. For details, see the examples of the databases on the following sections.

username()

Type: string
Default: n/a

Description:Name of the database user.

values()

Type: string list
Default: "${R_YEAR}-${R_MONTH}-${R_DAY}, ${R_HOUR}:${R_MIN}:${R_SEC}", "${FACILITY}", "${LEVEL}", "${HOST}", "${PROGRAM}", "${PID}", "${MSGONLY}"

Description:The parts of the message to store in the fields specified in the columns parameter.

It is possible to give a special value calling: default (without quotation marks).It means that the value will be used that is the default of the column type of this value.

Example 7.16. Value: default
columns("date datetime", "host varchar(32)", "row_id serial")
	values("${R_DATE}", "${HOST}", default)

7.9. Sending messages to a remote logserver using the IETF-syslog protocol

The syslog() driver sends messages to a remote host (for example a syslog-ng server or relay) on the local intranet or internet using the new standard syslog protocol developed by IETF (for details about the new protocol, see Section 2.8.2, IETF-syslog messages). The protocol supports sending messages using the UDP, TCP, or the encrypted TLS networking protocols.

The required arguments of the driver are the address of the destination host (where messages should be sent). The transport method (networking protocol) is optional, syslog-ng uses the TCP protocol by default. For the list of available optional parameters, see Section 7.9.1, syslog() destination options.

Declaration:
            syslog(host transport [options]);
Note

Note that the syslog destination driver has required parameters, while the source driver defaults to the local bind address, and every parameter is optional.

The udp transport method automatically sends multicast packets if a multicast destination address is specified. The tcp and tls methods do not support multicasting.

Note

The default ports for the different transport protocols are as follows: UDP — 514; TLS — 6514.

Example 7.17. Using the syslog() driver
destination d_tcp { syslog("10.1.2.3" transport("tcp") port(1999) localport(999)); };

If name resolution is configured, the hostname of the target server can be used as well.

destination d_tcp { syslog("target_host" transport("tcp") port(1999) localport(999)); };

Send the log messages using TLS encryption and use mutual authentication. For details on the encryption and authentication options, see Section 10.4, TLS options.

destination d_syslog_tls{
            syslog("10.100.20.40"
            transport("tls")
            port(6514)
            tls(peer-verify(required-trusted)
            ca_dir('/opt/syslog-ng/etc/syslog-ng/keys/ca.d/')
            key_file('/opt/syslog-ng/etc/syslog-ng/keys/client_key.pem')
            cert_file('/opt/syslog-ng/etc/syslog-ng/keys/client_certificate.pem'))
            );};

7.9.1. syslog() destination options

The syslog() driver sends messages to a remote host (for example a syslog-ng server or relay) on the local intranet or internet using the RFC5424 syslog protocol developed by IETF (for details about the protocol, see Section 2.8.2, IETF-syslog messages). The protocol supports sending messages using the UDP, TCP, or the encrypted TLS networking protocols.

These destinations have the following options:

flags()

Type: no_multi_line, syslog-protocol
Default: empty set

Description:Flags influence the behavior of the destination driver.

  • no-multi-line: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line.

  • syslog-protocol: The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver, and that the syslog driver automatically adds the frame header to the messages.

flush_lines()

Type: number
Default: Use global setting.

Description:Specifies how many lines are sent to a destination at a time. The syslog-ng OSE application waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the destination, but also increases message latency. The latency can be limited by the use of the flush-timeout option.

flush_timeout() (DEPRECATED)

Type: time in milliseconds
Default: Use global setting.

Description:This is a deprecated option. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. For details, see the flush_lines option.

frac_digits()

Type: number
Default: Value of the global option (which defaults to 0)

Description:The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.

ip_tos()

Type: number
Default: 0

Description:Specifies the Type-of-Service value of outgoing packets.

ip_ttl()

Type: number
Default: 0

Description:Specifies the Time-To-Live value of outgoing packets.

keep-alive()

Type: yes or no
Default: yes

Description:Specifies whether connections to destinations should be closed when syslog-ng is reloaded. Note that this applies to the client (destination) side of the syslog-ng connections, server-side (source) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the source.

localip()

Type: string
Default: 0.0.0.0

Description:The IP address to bind to before connecting to target.

localport()

Type: number
Default: 0

Description:The port number to bind to. Messages are sent from this port.

log_fifo_size()

Type: number
Default: Use global setting.

Description:The number of messages that the output queue can store.

mark_freq()

Accepted values: number
Default: 1200

Description:An alias for the obsolete mark() option, retained for compatibility with syslog-ng version 1.6.x. The number of seconds between two MARK messages. MARK messages are generated when there was no message traffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. The mark-freq can be set for global option and/or every MARK capable destination driver if mark-mode is periodical or dst-idle or host-idle. If mark-freq is not defined in the destination, then the mark-freq will be inherited from the global options. If the destination uses internal mark-mode, then the global mark-freq will be valid (does not matter what mark-freq set in the destination side).

mark_mode()

Accepted values: internal | dst-idle | host-idle | periodical | none | global
Default:

internal for pipe, program drivers

none for file, unix-dgram, unix-stream drivers

global for syslog, tcp, udp destinations

host-idle for global option

Description:The mark-mode() option can be set for the following destination drivers: file(), program(), unix_dgram(), unix_stream(), udp(), udp6(), tcp(), tcp6(), pipe(), syslog() and in global option.

  • internal: When internal mark mode is selected, internal source should be placed in the log path as this mode does not generate mark by itself at the destination. This mode only yelds the mark messages from internal source. This is the mode as syslog-ng OSE 3.3 worked. MARK will be generated by internal source if there was NO traffic on local sources:

    file(), pipe(), unix-stream(), unix-dgram(), program()

  • dst-idle: Sends mark signal if there was NO traffic on destination drivers. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • host-idle: Sends mark signal if there was NO local message on destination drivers. For example mark is generated even if messages were received from tcp. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • periodical: Sends mark signal perodically, regardless of traffic on destination driver. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to the drivers where none is the default value, then none will be used.

  • global: Destination driver uses the global mark-mode setting. The syslog-ng interprets syntax error if the global mark-mode is global.

Note

In case of dst-idle, host-idle and periodical; MARK message will not be written in the destination, if it is not open yet.

Available in syslog-ng OSE 3.4 and later.

port() or destport()

Type: number
Default: 601

Description:The port number to connect to. Note that the default port numbers used by syslog-ng do not comply with the latest RFC which was published after the release of syslog-ng 3.0.2, therefore the default port numbers will change in the future releases.

so_broadcast()

Type: yes or no
Default: no

Description:This option controls the SO_BROADCAST socket option required to make syslog-ng send messages to a broadcast address. For details, see the socket(7) manual page.

so_keepalive()

Type: yes or no
Default: no

Description:Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

so_rcvbuf()

Type: number
Default: 0

Description:Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.

so_sndbuf()

Type: number
Default: 0

Description:Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

spoof_source()

Type: yes or no
Default: no

Description:Enables source address spoofing. This means that the host running syslog-ng generates UDP packets with the source IP address matching the original sender of the message. It is useful when you want to perform some kind of preprocessing via syslog-ng then forward messages to your central log management solution with the source address of the original sender. This option only works for UDP destinations though the original message can be received by TCP as well. This option is only available if syslog-ng was compiled using the --enable-spoof-source configuration option.

suppress()

Type: seconds
Default: 0 (disabled)

Description:If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.

tcp-keepalive-intvl()

Type: number [seconds]
Default: 0

Description:Specifies the interval (number of seconds) between subsequential keepalive probes, regardless of the traffic exchanged in the connection. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_intvl. The default value is 0, which means using the kernel default.

Warning

The tcp-keepalive-time(), tcp-keepalive-probes(), and tcp-keepalive-intvl() options only work on platforms which support the TCP_KEEPCNT, TCP_KEEPIDLE,and TCP_KEEPINTVL setsockopts. Currently, this is Linux.

A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() * tcp-keepalive-probes() seconds.

Available in syslog-ng OSE version 3.4 and later.

tcp-keepalive-probes()

Type: number
Default: 0

Description:Specifies the number of unacknowledged probes to send before considering the connection dead. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_probes. The default value is 0, which means using the kernel default.

Warning

The tcp-keepalive-time(), tcp-keepalive-probes(), and tcp-keepalive-intvl() options only work on platforms which support the TCP_KEEPCNT, TCP_KEEPIDLE,and TCP_KEEPINTVL setsockopts. Currently, this is Linux.

A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() * tcp-keepalive-probes() seconds.

Available in syslog-ng OSE version 3.4 and later.

tcp-keepalive-time()

Type: number [seconds]
Default: 0

Description:Specifies the interval (in seconds) between the last data packet sent and the first keepalive probe. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_time. The default value is 0, which means using the kernel default.

Warning

The tcp-keepalive-time(), tcp-keepalive-probes(), and tcp-keepalive-intvl() options only work on platforms which support the TCP_KEEPCNT, TCP_KEEPIDLE,and TCP_KEEPINTVL setsockopts. Currently, this is Linux.

A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() * tcp-keepalive-probes() seconds.

Available in syslog-ng OSE version 3.4 and later.

template()

Type: string
Default: A format conforming to the default logfile format.

Description:Specifies a template defining the logformat to be used in the destination. Macros are described in Section 11.1.5, Macros of syslog-ng OSE. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.

template_escape()

Type: yes or no
Default: no

Description:Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.

throttle()

Type: number
Default: 0

Description:Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using large enough buffers as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.

time_zone()

Type: timezone in +/-HH:MM format
Default: unspecified

Description:Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.

tls()

Type: tls options
Default: n/a

Description:This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see Section 10.4, TLS options.

transport()

Type: udp, tcp, or tls
Default: tcp

Description:Specifies the protocol used to send messages to the destination server.

ts_format()

Type: rfc3164, bsd, rfc3339, iso
Default: Use the global option (which defaults to rfc3164)

Description:Override the global timestamp format (set in the global ts_format() parameter) for the specific destination. For details, see Section ts_format().

7.10. Sending messages to a remote logserver using the legacy BSD-syslog protocol

Note

The tcp(), tcp6(), udp(), and udp6() drivers will be deprecated in later versions, use the network() driver instead.

The tcp(), tcp6(), udp(), and udp6() drivers send messages to another host (for example a syslog-ng server or relay) on the local intranet or internet using the UDP or TCP protocol. The tcp6() and udp6() drivers use the IPv6 network protocol.

All four drivers have a single required parameter specifying the destination host address, where messages should be sent. For the list of available optional parameters, see Section 7.10.1, tcp(), tcp6(), udp(), and udp6() destination options.

The udp() and udp6() drivers automatically send multicast packets if a multicast destination address is specified. The tcp() and tcp6() drivers do not support multicasting.

Declaration:
            tcp(host [options]);
            udp(host [options]);
            tcp6(host [options]);
            udp6(host [options]);
Example 7.18. Using the tcp() driver
destination d_tcp { tcp("10.1.2.3" port(1999) localport(999)); };

If name resolution is configured, the hostname of the target server can be used as well.

destination d_tcp { tcp("target_host" port(1999) localport(999)); };

To send messages using the IETF-syslog message format without using the IETF-syslog protocol, enable the syslog-protocol flag:

destination d_tcp { tcp("10.1.2.3" port(1999) flags(syslog-protocol) ); };

(For details on how to use the IETF-syslog protocol, see Section 7.9.1, syslog() destination options.)

Using an IPv6 address:

tcp6("fd00:abcd:4321:2:20c:29ff:fea8:9671" port(514));

7.10.1. tcp(), tcp6(), udp(), and udp6() destination options

This driver sends messages to another host on the local intranet or internet according to RFC3164 using the UDP or TCP protocol. The tcp6() and udp6() drivers use the IPv6 network protocol.

Note

When using IPv6 destination addresses, always enclose the address between double-quotes:

tcp6("fd00:abcd:4321:2:20c:29ff:fea8:9671" port(514));

These destinations have the following options:

flags()

Type: no_multi_line, syslog-protocol
Default: empty set

Description:Flags influence the behavior of the destination driver.

  • no-multi-line: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line.

  • syslog-protocol: The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver, and that the syslog driver automatically adds the frame header to the messages.

flush_lines()

Type: number
Default: Use global setting.

Description:Specifies how many lines are sent to a destination at a time. The syslog-ng OSE application waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the destination, but also increases message latency. The latency can be limited by the use of the flush-timeout option.

flush_timeout() (DEPRECATED)

Type: time in milliseconds
Default: Use global setting.

Description:This is a deprecated option. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. For details, see the flush_lines option.

frac_digits()

Type: number
Default: Value of the global option (which defaults to 0)

Description:The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.

ip_tos()

Type: number
Default: 0

Description:Specifies the Type-of-Service value of outgoing packets.

ip_ttl()

Type: number
Default: 0

Description:Specifies the Time-To-Live value of outgoing packets.

keep-alive()

Type: yes or no
Default: yes

Description:Specifies whether connections to destinations should be closed when syslog-ng is reloaded. Note that this applies to the client (destination) side of the syslog-ng connections, server-side (source) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the source.

localip()

Type: string
Default: 0.0.0.0

Description:The IP address to bind to before connecting to target.

localport()

Type: number
Default: 0

Description:The port number to bind to. Messages are sent from this port.

mark_mode()

Accepted values: internal | dst-idle | host-idle | periodical | none | global
Default:

internal for pipe, program drivers

none for file, unix-dgram, unix-stream drivers

global for syslog, tcp, udp destinations

host-idle for global option

Description:The mark-mode() option can be set for the following destination drivers: file(), program(), unix_dgram(), unix_stream(), udp(), udp6(), tcp(), tcp6(), pipe(), syslog() and in global option.

  • internal: When internal mark mode is selected, internal source should be placed in the log path as this mode does not generate mark by itself at the destination. This mode only yelds the mark messages from internal source. This is the mode as syslog-ng OSE 3.3 worked. MARK will be generated by internal source if there was NO traffic on local sources:

    file(), pipe(), unix-stream(), unix-dgram(), program()

  • dst-idle: Sends mark signal if there was NO traffic on destination drivers. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • host-idle: Sends mark signal if there was NO local message on destination drivers. For example mark is generated even if messages were received from tcp. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • periodical: Sends mark signal perodically, regardless of traffic on destination driver. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to the drivers where none is the default value, then none will be used.

  • global: Destination driver uses the global mark-mode setting. The syslog-ng interprets syntax error if the global mark-mode is global.

Note

In case of dst-idle, host-idle and periodical; MARK message will not be written in the destination, if it is not open yet.

Available in syslog-ng OSE 3.4 and later.

port() or destport()

Type: number
Default: 514

Description:The port number to connect to. Note that the default port numbers used by syslog-ng do not comply with the latest RFC which was published after the release of syslog-ng 3.0.2, therefore the default port numbers will change in the future releases.

Note

The TCP port 514 is reserved for use with rshell, so select a different port if syslog-ng and rshell is used at the same time.

so_broadcast()

Type: yes or no
Default: no

Description:This option controls the SO_BROADCAST socket option required to make syslog-ng send messages to a broadcast address. For details, see the socket(7) manual page.

so_keepalive()

Type: yes or no
Default: no

Description:Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

so_rcvbuf()

Type: number
Default: 0

Description:Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.

so_sndbuf()

Type: number
Default: 0

Description:Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

spoof_source()

Type: yes or no
Default: no

Description:Enables source address spoofing. This means that the host running syslog-ng generates UDP packets with the source IP address matching the original sender of the message. It is useful when you want to perform some kind of preprocessing via syslog-ng then forward messages to your central log management solution with the source address of the original sender. This option only works for UDP destinations though the original message can be received by TCP as well. This option is only available if syslog-ng was compiled using the --enable-spoof-source configuration option.

suppress()

Type: seconds
Default: 0 (disabled)

Description:If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.

tcp-keepalive-intvl()

Type: number [seconds]
Default: 0

Description:Specifies the interval (number of seconds) between subsequential keepalive probes, regardless of the traffic exchanged in the connection. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_intvl. The default value is 0, which means using the kernel default.

Warning

The tcp-keepalive-time(), tcp-keepalive-probes(), and tcp-keepalive-intvl() options only work on platforms which support the TCP_KEEPCNT, TCP_KEEPIDLE,and TCP_KEEPINTVL setsockopts. Currently, this is Linux.

A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() * tcp-keepalive-probes() seconds.

Available in syslog-ng OSE version 3.4 and later.

tcp-keepalive-probes()

Type: number
Default: 0

Description:Specifies the number of unacknowledged probes to send before considering the connection dead. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_probes. The default value is 0, which means using the kernel default.

Warning

The tcp-keepalive-time(), tcp-keepalive-probes(), and tcp-keepalive-intvl() options only work on platforms which support the TCP_KEEPCNT, TCP_KEEPIDLE,and TCP_KEEPINTVL setsockopts. Currently, this is Linux.

A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() * tcp-keepalive-probes() seconds.

Available in syslog-ng OSE version 3.4 and later.

tcp-keepalive-time()

Type: number [seconds]
Default: 0

Description:Specifies the interval (in seconds) between the last data packet sent and the first keepalive probe. This option is equivalent to /proc/sys/net/ipv4/tcp_keepalive_time. The default value is 0, which means using the kernel default.

Warning

The tcp-keepalive-time(), tcp-keepalive-probes(), and tcp-keepalive-intvl() options only work on platforms which support the TCP_KEEPCNT, TCP_KEEPIDLE,and TCP_KEEPINTVL setsockopts. Currently, this is Linux.

A connection that has no traffic is closed after tcp-keepalive-time() + tcp-keepalive-intvl() * tcp-keepalive-probes() seconds.

Available in syslog-ng OSE version 3.4 and later.

template()

Type: string
Default: A format conforming to the default logfile format.

Description:Specifies a template defining the logformat to be used in the destination. Macros are described in Section 11.1.5, Macros of syslog-ng OSE. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.

template_escape()

Type: yes or no
Default: no

Description:Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.

throttle()

Type: number
Default: 0

Description:Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using large enough buffers as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.

time_zone()

Type: timezone in +/-HH:MM format
Default: unspecified

Description:Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.

tls()

Type: tls options
Default: n/a

Description:This option sets various options related to TLS encryption, for example, key/certificate files and trusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see Section 10.4, TLS options.

ts_format()

Type: rfc3164, bsd, rfc3339, iso
Default: Use the global option (which defaults to rfc3164)

Description:Override the global timestamp format (set in the global ts_format() parameter) for the specific destination. For details, see Section ts_format().

7.11. Sending messages to UNIX domain sockets

The unix-stream() and unix-dgram() drivers send messages to a UNIX domain socket in either SOCK_STREAM or SOCK_DGRAM mode.

Both drivers have a single required argument specifying the name of the socket to connect to. For the list of available optional parameters, see Section 7.11.1, unix-stream() and unix-dgram() destination options.

Declaration:
            unix-stream(filename [options]);
            unix-dgram(filename [options]);
Example 7.19. Using the unix-stream() driver
destination d_unix_stream { unix-stream("/var/run/logs"); };

7.11.1. unix-stream() and unix-dgram() destination options

These drivers send messages to a unix socket in either SOCK_STREAM or SOCK_DGRAM mode. The unix-stream() and unix-dgram() destinations have the following options:

flags()

Type: no_multi_line, syslog-protocol
Default: empty set

Description:Flags influence the behavior of the destination driver.

  • no-multi-line: The no-multi-line flag disables line-breaking in the messages; the entire message is converted to a single line.

  • syslog-protocol: The syslog-protocol flag instructs the driver to format the messages according to the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag is enabled, macros used for the message have effect only for the text of the message, the message header is formatted to the new standard. Note that this flag is not needed for the syslog driver, and that the syslog driver automatically adds the frame header to the messages.

flush_lines()

Type: number
Default: Use global setting.

Description:Specifies how many lines are sent to a destination at a time. The syslog-ng OSE application waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the destination, but also increases message latency. The latency can be limited by the use of the flush-timeout option.

flush_timeout() (DEPRECATED)

Type: time in milliseconds
Default: Use global setting.

Description:This is a deprecated option. Specifies the time syslog-ng waits for lines to accumulate in its output buffer. For details, see the flush_lines option.

frac_digits()

Type: number
Default: Value of the global option (which defaults to 0)

Description:The syslog-ng application can store fractions of a second in the timestamps according to the ISO8601 format. The frac_digits() parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601 timestamps as well.

log_fifo_size()

Type: number
Default: Use global setting.

Description:The number of messages that the output queue can store.

keep-alive()

Type: yes or no
Default: yes

Description:Specifies whether connections to destinations should be closed when syslog-ng is reloaded. Note that this applies to the client (destination) side of the syslog-ng connections, server-side (source) connections are always reopened after receiving a HUP signal unless the keep-alive option is enabled for the source.

so_broadcast()

Type: yes or no
Default: no

Description:This option controls the SO_BROADCAST socket option required to make syslog-ng send messages to a broadcast address. For details, see the socket(7) manual page.

so_keepalive()

Type: yes or no
Default: no

Description:Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-stream sockets. For details, see the socket(7) manual page.

mark_mode()

Accepted values: internal | dst-idle | host-idle | periodical | none | global
Default:

internal for pipe, program drivers

none for file, unix-dgram, unix-stream drivers

global for syslog, tcp, udp destinations

host-idle for global option

Description:The mark-mode() option can be set for the following destination drivers: file(), program(), unix_dgram(), unix_stream(), udp(), udp6(), tcp(), tcp6(), pipe(), syslog() and in global option.

  • internal: When internal mark mode is selected, internal source should be placed in the log path as this mode does not generate mark by itself at the destination. This mode only yelds the mark messages from internal source. This is the mode as syslog-ng OSE 3.3 worked. MARK will be generated by internal source if there was NO traffic on local sources:

    file(), pipe(), unix-stream(), unix-dgram(), program()

  • dst-idle: Sends mark signal if there was NO traffic on destination drivers. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • host-idle: Sends mark signal if there was NO local message on destination drivers. For example mark is generated even if messages were received from tcp. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • periodical: Sends mark signal perodically, regardless of traffic on destination driver. Mark signal from internal source will be dropped.

    MARK signal can be sent by the following destination drivers: tcp(), udp(), syslog(), program(), file(), pipe(), unix-stream(), unix-dgram().

  • none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to the drivers where none is the default value, then none will be used.

  • global: Destination driver uses the global mark-mode setting. The syslog-ng interprets syntax error if the global mark-mode is global.

Note

In case of dst-idle, host-idle and periodical; MARK message will not be written in the destination, if it is not open yet.

Available in syslog-ng OSE 3.4 and later.

so_rcvbuf()

Type: number
Default: 0

Description:Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manual page.

so_sndbuf()

Type: number
Default: 0

Description:Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

suppress()

Type: seconds
Default: 0 (disabled)

Description:If several identical log messages would be sent to the destination without any other messages between the identical messages (for example, an application repeated an error message ten times), syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.

template()

Type: string
Default: A format conforming to the default logfile format.

Description:Specifies a template defining the logformat to be used in the destination. Macros are described in Section 11.1.5, Macros of syslog-ng OSE. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure the receiver can cope with the custom format defined.

template_escape()

Type: yes or no
Default: no

Description:Turns on escaping for the ', ", and backspace characters in templated output files. This is useful for generating SQL statements and quoting string contents so that parts of the log message are not interpreted as commands to the SQL server.

throttle()

Type: number
Default: 0

Description:Sets the maximum number of messages sent to the destination per second. Use this output-rate-limiting functionality only when using large enough buffers as well to avoid the risk of losing messages. Specifying 0 or a lower value sets the output limit to unlimited.

time_zone()

Type: timezone in +/-HH:MM format
Default: unspecified

Description:Convert timestamps to the timezone specified by this option. If this option is not set then the original timezone information in the message is used.

ts_format()

Type: rfc3164, bsd, rfc3339, iso
Default: Use the global option (which defaults to rfc3164)

Description:Override the global timestamp format (set in the global ts_format() parameter) for the specific destination. For details, see Section ts_format().

7.12. Sending messages to a user terminal — usertty() destination

This driver writes messages to the terminal of a logged-in user.

The usertty() driver has a single required argument, specifying a username who should receive a copy of matching messages. Use the asterisk * to specify every user currently logged in to the system.

Declaration:
usertty(username);

The usertty() does not have any further options nor does it support templates.

Example 7.20. Using the usertty() driver
destination d_usertty { usertty("root"); };

Chapter 8. Routing messages: log paths and filters

8.1. Log paths

Log paths determine what happens with the incoming log messages. Messages coming from the sources listed in the log statement and matching all the filters are sent to the listed destinations.

To define a log path, add a log statement to the syslog-ng configuration file using the following syntax:

log {
    source(s1); source(s2); ...
    optional_element(filter1|parser1|rewrite1);...
    optional_element(filter2|parser2|rewrite2);...
    destination(d1); destination(d2); ...
    flags(flag1[, flag2...]);
    };
Warning

Log statements are processed in the order they appear in the configuration file, thus the order of log paths may influence what happens to a message, especially when using filters and log flags.

Example 8.1. A simple log statement

The following log statement sends all messages arriving to the localhost to a remote server.

source s_localhost { tcp(ip(127.0.0.1) port(1999) ); };
destination d_tcp { tcp("10.1.2.3" port(1999); localport(999)); };
log { source(s_localhost); destination(d_tcp); };

All matching log statements are processed by default, and the messages are sent to every matching destination by default. So a single log message might be sent to the same destination several times, provided the destination is listed in several log statements, and it can be also sent to several different destinations.

This default behavior can be changed using the flags() parameter. Flags apply to individual log paths; they are not global options. The following flags available in syslog-ng:

  • final: Do not send the messages processed by this log path to any further destination.

  • fallback: Process messages that were not processed by other log paths.

  • catchall: Process every message, regardless of its source or if it was already processed by other log paths.

  • flow-control: Stop reading messages from the source if the destination cannot accept them. For details, see Section 8.2, Managing incoming and outgoing messages with flow-control.

For details on the individual flags, see Section 8.1.3, Log path flags. The effect and use of the flow-control flag is detailed in Section 8.2, Managing incoming and outgoing messages with flow-control.

8.1.1. Embedded log statements

Starting from version 3.0, syslog-ng can handle embedded log statements (also called log pipes). Embedded log statements are useful for creating complex, multi-level log paths with several destinations and use filters, parsers, and rewrite rules.

For example, if you want to filter your incoming messages based on the facility parameter, and then use further filters to send messages arriving from different hosts to different destinations, you would use embedded log statements.

Embedded log statement

Figure 8.1. Embedded log statement


Embedded log statements include sources — and usually filters, parsers, rewrite rules, or destinations — and other log statements that can include filters, parsers, rewrite rules, and destinations. The following rules apply to embedded log statements:

  • Only the beginning (also called top-level) log statement can include sources.

  • Embedded log statements can include multiple log statements on the same level (that is, a top-level log statement can include two or more log statements).

  • Embedded log statements can include several levels of log statements (that is, a top-level log statement can include a log statement that includes another log statement, and so on).

  • After an embedded log statement, you can write either another log statement, or the flags() option of the original log statement. You cannot use filters or other configuration objects.

  • Embedded log statements that are on the same level receive the same messages from the higher-level log statement. For example, if the top-level log statement includes a filter, the lower-level log statements receive only the messages that pass the filter.

Embedded log statements

Figure 8.2. Embedded log statements


Embedded log filters can be used to optimize the processing of log messages, for example, to re-use the results of filtering and rewriting operations.

8.1.1.1. Using embedded log statements

Embedded log statements (for details, see Section 8.1.1, Embedded log statements) re-use the results of processing messages (for example the results of filtering or rewriting) to create complex log paths. Embedded log statements use the same syntax as regular log statements, but they cannot contain additional sources. To define embedded log statements, use the following syntax:

log {
    source(s1); source(s2); ...

    optional_element(filter1|parser1|rewrite1);
    optional_element(filter2|parser2|rewrite2);...

    destination(d1); destination(d2); ...

    #embedded log statement
    log
        {
        optional_element(filter1|parser1|rewrite1);
        optional_element(filter2|parser2|rewrite2);
        ...
        destination(d1); destination(d2); ...
        #another embedded log statement
        log
        {
            optional_element(filter1|parser1|rewrite1);
            optional_element(filter2|parser2|rewrite2);
            ...
            destination(d1); destination(d2); ...};
        };
    #set flags after the embedded log statements
    flags(flag1[, flag2...]);
        };
Example 8.2. Using embedded log paths

The following log path sends every message to the configured destinations: both the d_file1 and the d_file2 destinations receive every message of the source.

log { source(s_localhost); destination(d_file1); destination(d_file2); };

The next example is equivalent with the one above, but uses an embedded log statement.

log { source(s_localhost); destination(d_file1);
                    log {destination(d_file2); };
};

The following example uses two filters:

  • messages coming from the host 192.168.1.1 are sent to the d_file1 destination; and

  • messages coming from the host 192.168.1.1 and containing the string example are sent to the d_file2 destination.

log { source(s_localhost); host(192.168.1.); destination(d_file1);
                    log {message("example"); destination(d_file2); };
};

The following example collects logs from multiple source groups and uses the source() filter in the embedded log statement to select messages of the s_network source group.

log { source(s_localhost); source(s_network); destination(d_file1);
                    log {source(s_network); destination(d_file2); };
                    };

8.1.2. Junctions and channels

Junctions make it possible to send the messages to different channels, process the messages differently on each channel, and then join every channel together again. You can define any number of channels in a junction: every channel receives a copy of every message that reaches the junction. Every channel can process the messages differently, and at the end of the junction, the processed messages of every channel return to the junction again, where further processing is possible.

A junction includes one or more channels. A channel usually includes at least one filter, though that is not enforced. Otherwise, channels are identical to log statements, and can include any kind of objects, for example, parsers, rewrite rules, destinations, and so on. (For details on using channels, as well as on using channels outside junctions, see Section 5.5, Using channels in configuration objects.)

Note

Certain parsers can also act as filters:

  • The JSON parser automatically discards messages that are not valid JSON messages.

  • The csv-parser() discards invalid messages if the flags(drop-invalid) option is set.

You can also use log-path flags in the channels of the junction. Within the junction, a message is processed by every channel, in the order the channels appear in the configuration file. Typically if your channels have filters, you also set the flags(final) option for the channel. However, note that the log-path flags of the channel apply only within the junction, for example, if you set the final flag for a channel, then the subsequent channels of the junction will not receive the message, but this does not affect any other log path or junction of the configuration. The only exception is the flow-control flag: if you enable flow-control in a junction, it affects the entire log path. For details on log-path flags, see Section 8.1.3, Log path flags.

junction {
    channel { <other-syslog-ng-objects> <log-path-flags>}
    channel { <other-syslog-ng-objects> <log-path-flags>}
    ...
    }
Example 8.3. Using junctions

For example, suppose that you have a single network source that receives log messages from different devices, and some devices send messages that are not RFC-compliant (some routers are notorious for that). To solve this problem in earlier versions of syslog-ng OSE, you had to create two different network sources using different IP addresses or ports: one that received the RFC-compliant messages, and one that received the improperly formatted messages (for example, using the flags(no-parse) option). Using junctions this becomes much more simple: you can use a single network source to receive every message, then use a junction and two channels. The first channel processes the RFC-compliant messages, the second everything else. At the end, every message is stored in a single file. The filters used in the example can be host() filters (if you have a list of the IP addresses of the devices sending non-compliant messages), but that depends on your environment.

log {
    source s_network { syslog(ip(10.1.2.3) transport("tcp") flags(no-parse)); };
    junction {
        channel { filter(f_compliant_hosts); parser { syslog-parser(); }; };
        channel { filter(f_noncompliant_hosts); };
    };
    destination { file("/var/log/messages"); };
};

Since every channel receives every message that reaches the junction, use the flags(final) option in the channels to avoid the unnecessary processing the messages multiple times:

log {
    source s_network { syslog(ip(10.1.2.3) transport("tcp") flags(no-parse)); };
    junction {
        channel { filter(f_compliant_hosts); parser { syslog-parser(); }; flags(final); };
        channel { filter(f_noncompliant_hosts); flags(final); };
    };
    destination { file("/var/log/messages"); };
};
Note

Junctions differ from embedded log statements, because embedded log statements are like branches: they split the flow of messages into separate paths, and the different paths do not meet again. Messages processed on different embedded log statements cannot be combined together for further processing. However, junctions split the messages to channels, then combine the channels together.

8.1.3. Log path flags

Flags influence the behavior of syslog-ng, and the way it processes messages. The following flags may be used in the log paths, as described in Section 8.1, Log paths.

Flag Description
catchall This flag means that the source of the message is ignored, only the filters are taken into account when matching messages. A log statement using the catchall flag processes every message that arrives to any of the defined sources.
fallback This flag makes a log statement 'fallback'. Fallback log statements process messages that were not processed by other, 'non-fallback' log statements.
final This flag means that the processing of log messages processed by the log statement ends here, other log statements appearing later in the configuration file will not process the messages processed by the log statement labeled as 'final'. Note that this does not necessarily mean that matching messages will be stored only once, as there can be matching log statements processed prior the current one.
flow-control Enables flow-control to the log path, meaning that syslog-ng will stop reading messages from the sources of this log statement if the destinations are not able to process the messages at the required speed. If disabled, syslog-ng will drop messages if the destination queues are full. If enabled, syslog-ng will only drop messages if the destination queues/window sizes are improperly sized. For details, see Section 8.2, Managing incoming and outgoing messages with flow-control.

Table 8.1. Log statement flags


Warning

The final, fallback, and catchall flags apply only for the top-level log paths, they have no effect on embedded log paths.

Example 8.4. Using log path flags

Let's suppose that you have two hosts (myhost_A and myhost_B) that run two applications each (application_A and application_B), and you collect the log messages to a central syslog-ng server. On the server, you create two log paths:

  • one that processes only the messages sent by myhost_A; and

  • one that processes only the messages sent by application_A.

This means that messages sent by application_A running on myhost_A will be processed by both log paths, and the messages of application_B running on myhost_B will not be processed at all.

  • If you add the final flag to the first log path, then only this log path will process the messages of myhost_A, so the second log path will receive only the messages of application_A running on myhost_B.

  • If you create a third log path that includes the fallback flag, it will process the messages not processed by the first two log paths, in this case, the messages of application_B running on myhost_B.

  • Adding a fourth log path with the catchall flag would process every message received by the syslog-ng server.

    log { source(s_localhost); destination(d_file); flags(catchall); };

8.2. Managing incoming and outgoing messages with flow-control

This section describes the internal message-processing model of syslog-ng, as well as the flow-control feature that can prevent message losses. To use flow-control, the flow-control flag must be enabled for the particular log path.

The syslog-ng application monitors (polls) the sources defined in its configuration file, periodically checking each source for messages. When a log message is found in one of the sources, syslog-ng polls every source and reads the available messages. These messages are processed and put into the output buffer of syslog-ng (also called fifo). From the output buffer, the operating system sends the messages to the appropriate destinations.

In large-traffic environments many messages can arrive during a single poll loop, therefore syslog-ng reads only a fixed number of messages from each source. The log_fetch_limit() option specifies the number of messages read during a poll loop from a single source.

Managing log messages in syslog-ng

Figure 8.3. Managing log messages in syslog-ng


Note

The log_fetch_limit() parameter can be set as a global option, or for every source individually.

Every destination has its own output buffer. The output buffer is needed because the destination might not be able to accept all messages immediately. The log_fifo_size() parameter sets the size of the output buffer. The output buffer must be larger than the log_fetch_limit() of the sources, to ensure that every message read during the poll loop fits into the output buffer. If the log path sends messages to a destination from multiple sources, the output buffer must be large enough to store the incoming messages of every source.

TCP and unix-stream sources can receive the logs from several incoming connections (for example many different clients or applications). For such sources, syslog-ng reads messages from every connection, thus the log_fetch_limit() parameter applies individually to every connection of the source.

Managing log messages of TCP sources in syslog-ng

Figure 8.4. Managing log messages of TCP sources in syslog-ng


The flow-control of syslog-ng introduces a control window to the source that tracks how many messages can syslog-ng accept from the source. Every message that syslog-ng reads from the source lowers the window size by one; every message that syslog-ng successfully sends from the output buffer increases the window size by one. If the window is full (that is, its size decreases to zero), syslog-ng stops reading messages from the source. The initial size of the control window is by default 1000: the log_fifo_size() must be larger than this value in order for flow-control to have any effect. If a source accepts messages from multiple connections, all messages use the same control window.

Note

Starting with syslog-ng OSE version 3.3, if the source can handle multiple connections (for example, tcp()), the size of the control window is divided by the value of the max_connections() parameter and this smaller control window is applied to each connection of the source.

When flow-control is used, every source has its own control window. As a worst-case situation, the output buffer of the destination must be set to accommodate all messages of every control window, that is, the log_fifo_size() of the destination must be greater than number_of_sources*log_iw_size(). This applies to every source that sends logs to the particular destination. Thus if two sources having several connections and heavy traffic send logs to the same destination, the control window of both sources must fit into the output buffer of the destination. Otherwise, syslog-ng does not activate the flow-control, and messages may be lost.

The syslog-ng application handles outgoing messages the following way:

Handling outgoing messages in syslog-ng OSE

Figure 8.5. Handling outgoing messages in syslog-ng OSE


  • Output queue: Messages from the output queue are sent to the target syslog-ng server. The syslog-ng application puts the outgoing messages directly into the output queue, unless the output queue is full. The output queue can hold 64 messages, this is a fixed value and cannot be modified.

  • Overflow queue: If the output queue is full, syslog-ng puts the outgoing messages into the overflow queue of the destination. (The overflow queue is identical to the output buffer used by other destinations.) The log_fifo_size() parameter specifies the number of messages stored in the overflow queue. For details on sizing the log_fifo_size() parameter, see Section 8.2, Managing incoming and outgoing messages with flow-control.

There are two types of flow-control: Hard flow-control and soft flow-control.

  • Soft flow-control: In case of soft flow-control there is no message lost if the destination can accept messages, but it is possible to lose messages if it cannot accept messages (for example non-writeable file destination, or the disk becomes full), and all buffers are full. Soft flow-control cannot be configured, it is automatically available for file destinations.

    Example 8.5. Soft flow-control

    source s_file {file("/tmp/input_file.log");};
    destination d_file {file("/tmp/output_file.log");};
    destination d_tcp { tcp("127.0.0.1" port(2222) log_fifo_size(1000)); };
    log{ source(s_file); destination(d_file); destination(d_tcp);};
    

    Warning

    Hazard of data loss! For destinations other than file, soft flow-control is not available. Thus, it is possible to lose log messages on those destinations. To avoid data loss on those destinations, use hard flow-control.

  • Hard flow-control: In case of hard flow-control there is no message lost. To use hard flow-control, enable the flow_control flag in the logpath. Hard flow-control is available for all destinations.

    Example 8.6. Hard flow-control

    source s_file {file("/tmp/input_file.log");};
    destination d_file {file("/tmp/output_file.log");};
    destination d_tcp { tcp("127.0.0.1" port(2222) log_fifo_size(1000)); };
    log{ source(s_file); destination(d_file); destination(d_tcp) flags(flow_control);};
    

8.2.1. Flow-control and multiple destinations

Using flow-control on a source has an important side-effect if the messages of the source are sent to multiple destinations. If flow-control is in use and one of the destinations cannot accept the messages, the other destinations do not receive any messages either, because syslog-ng stops reading the source. For example, if messages from a source are sent to a remote server and also stored locally in a file, and the network connection to the server becomes unavailable, neither the remote server nor the local file will receive any messages.

Note

Creating separate log paths for the destinations that use the same flow-controlled source does not avoid the problem.

8.2.2. Configuring flow-control

For details on how flow-control works, see Section 8.2, Managing incoming and outgoing messages with flow-control. The summary of the main points is as follows:

  • The syslog-ng application normally reads a maximum of log_fetch_limit() number of messages from a source.

  • From TCP and unix-stream sources, syslog-ng reads a maximum of log_fetch_limit() from every connection of the source. The number of connections to the source is set using the max_connections() parameter.

  • Every destination has an output buffer (log_fifo_size()).

  • Flow-control uses a control window to determine if there is free space in the output buffer for new messages. Every source has its own control window; log_iw_size() parameter sets the size of the control window.

  • When a source accepts multiple connections, the size of the control window is divided by the value of the max_connections() parameter and this smaller control window is applied to each connection of the source.

  • The output buffer must be larger than the control window of every source that logs to the destination.

  • If the control window is full, syslog-ng stops reading messages from the source until some messages are successfully sent to the destination.

  • If the output buffer becomes full, and flow-control is not used, messages may be lost.

Warning

If you modify the max_connections() or the log_fetch_limit() parameter, do not forget to adjust the log_iw_size() and log_fifo_size() parameters accordingly.

Example 8.7. Sizing parameters for flow-control

Suppose that syslog-ng has a source that must accept up to 300 parallel connections. Such situation can arise when a network source receives connections from many clients, or if many applications log to the same socket. Therefore, set the max_connections() parameter of the source to 300. However, the log_fetch_limit() (default value: 10) parameter applies to every connection of the source individually, while the log_iw_size() (default value: 1000) parameter applies to the source. In a worst-case scenario, the destination does not accept any messages, while all 300 connections send at least log_fetch_limit() number of messages to the source during every poll loop. Therefore, the control window must accommodate at least max_connections()*log_fetch_limit() messages to be able to read every incoming message of a poll loop. In the current example this means that (log_iw_size() should be greater than 300*10=3000. If the control window is smaller than this value, the control window might fill up with messages from the first connections — causing syslog-ng to read only one message of the last connections in every poll loop.

The output buffer of the destination must accommodate at least log_iw_size() messages, but use a greater value: in the current example 3000*10=30000 messages. That way all incoming messages of ten poll loops fit in the output buffer. If the output buffer is full, syslog-ng does not read any messages from the source until some messages are successfully sent to the destination.

source s_localhost {
            tcp(ip(127.0.0.1) port(1999) max-connections(300)); };
destination d_tcp {
            tcp("10.1.2.3" port(1999); localport(999)); log_fifo_size(30000); };
log { source(s_localhost); destination(d_tcp); flags(flow-control); };

If other sources send messages to this destination, than the output buffer must be further increased. For example, if a network host with maximum 100 connections also logs into the destination, than increase the log_fifo_size() by 10000.

source s_localhost {
            tcp(ip(127.0.0.1) port(1999) max-connections(300)); };
source s_tcp {
            tcp(ip(192.168.1.5) port(1999) max-connections(100)); };
destination d_tcp {
            tcp("10.1.2.3" port(1999); localport(999)); log_fifo_size(40000); };
log { source(s_localhost); destination(d_tcp); flags(flow-control); };

For details, see also Section 17.2, Handling lots of parallel connections.

8.3. Filters

The following sections describe how to select and filter log messages.

8.3.1. Using filters

Filters perform log routing within syslog-ng: a message passes the filter if the filter expression is true for the particular message. If a log statement includes filters, the messages are sent to the destinations only if they pass all filters of the log path. For example, a filter can select only the messages originating from a particular host. Complex filters can be created using filter functions and logical boolean expressions.

To define a filter, add a filter statement to the syslog-ng configuration file using the following syntax:

filter <identifier> { <filter_type>("<filter_expression>"); };
Example 8.8. A simple filter statement

The following filter statement selects the messages that contain the word deny and come from the host example.

filter demo_filter { host("example") and match("deny" value("MESSAGE")) };

For the filter to have effect, include it in a log statement:

log demo_filteredlog {
        source(s1);
        filter(demo_filter);
        destination(d1);};

8.3.2. Combining filters with boolean operators

When a log statement includes multiple filter statements, syslog-ng sends a message to the destination only if all filters are true for the message. In other words, the filters are connected with the logical AND operator. In the following example, no message arrives to the destination, because the filters are exclusive (the hostname of a client cannot be example1 and example2 at the same time):

filter demo_filter1 { host("example1"); };
filter demo_filter2 { host("example2"); };
log demo_filteredlog {
    source(s1); source(s2);
    filter(demo_filter1); filter(demo_filter2);
    destination(d1); destination(d2); };

To select the messages that come from either host example1 or example2, use a single filter expression:

filter demo_filter { host("example1") or host("example2"); };
log demo_filteredlog {
    source(s1); source(s2);
    filter(demo_filter);
    destination(d1); destination(d2); };

Use the not operator to invert filters, for example, to select the messages that were not sent by host example1:

filter demo_filter { not host("example1"); };

However, to select the messages that were not sent by host example1 or example2, you have to use the and operator (that's how boolean logic works):

filter demo_filter { not host("example1") and not host("example2"); };

Alternatively, you can use parentheses to avoid this confusion:

filter demo_filter { not (host("example1") or host("example2")); };

For a complete description on filter functions, see Section 8.3.6, Filter functions.

The following filter statement selects the messages that contain the word deny and come from the host example.

filter demo_filter { host("example") and match("deny" value("MESSAGE")); };

The value() parameter of the match function limits the scope of the function to the text part of the message (that is, the part returned by the ${MESSAGE} macro). For details on using the match() filter function, see Section match().

Tip

Filters are often used together with log path flags. For details, see Section 8.1.3, Log path flags.

8.3.3. Comparing macro values in filters

Starting with syslog-ng OSE version 3.2, it is also possible to compare macro values and templates as numerical and string values. String comparison is alphabetical: it determines if a string is alphabetically greater or equal to another string. Use the following syntax to compare macro values or templates. For details on macros and templates, see Section 11.1, Customizing message format.

filter <filter-id>
        {"<macro-or-template>" operator "<value-or-macro-or-template>"};
Example 8.9. Comparing macro values in filters

The following expression selects log messages containing a PID (that is, ${PID} macro is not empty):

filter f_pid {"${PID}" !=""};

The following expression accomplishes the same, but uses a template as the left argument of the operator and compares the values as strings:

filter f_pid {"${HOST}${PID}" eq "${HOST}"};

The following example selects messages with priority level 4 or higher.

filter f_level {"${LEVEL_NUM}" > "5"};

Note that:

  • The macro or template must be enclosed in double-quotes.

  • The $ character must be used before macros.

  • Using comparator operators can be equivalent to using filter functions, but is somewhat slower. For example, using "${HOST}" eq "myhost" is equivalent to using host("myhost" type(string)).

  • You can use any macro in the expression, including user-defined macros from parsers and results of pattern database classifications.

  • The results of filter functions are boolean values, so they cannot be compared to other values.

  • You can use boolean operators to combine comparison expressions.

The following operators are available:

Numerical operator String operator Meaning
== eq Equals
!= ne Not equal to
> gt Greater than
< lt Less than
>= ge Greater than or equal
=< le Less than or equal

Table 8.2. Numerical and string comparison operators


8.3.4. Using wildcards, special characters, and regular expressions in filters

The host(), match(), and program() filter functions accept regular expressions as parameters. The exact type of the regular expression to use can be specified with the type() option. By default, syslog-ng OSE uses POSIX regular expressions.

In regular expressions, the asterisk (*) character means 0, 1 or any number of the previous expression. For example, in the f*ilter expression the asterisk means 0 or more f letters. This expression matches for the following strings: ilter, filter, ffilter, and so on. To achieve the wildcard functionality commonly represented by the asterisk character in other applications, use .* in your expressions, for example f.*ilter.

Alternatively, if you do not need regular expressions, only wildcards, use type(glob) in your filter:

Example 8.10. Filtering with widcards

The following filter matches on hostnames starting with the myhost string, for example, on myhost-1, myhost-2, and so on.

filter f_wildcard {host("myhost*" type(glob));};

For details on using regular expressions in syslog-ng OSE, see Section 8.3.4, Using wildcards, special characters, and regular expressions in filters.

To filter for special control characters like the carriage return (CR), use the \r escape prefix in syslog-ng OSE version 3.0 and 3.1. In syslog-ng OSE 3.2 and later, you can also use the \x escape prefix and the ASCII code of the character. For example, to filter on carriage returns, use the following filter:

filter f_carriage_return {match("\x0d" value ("MESSAGE"));};

8.3.5. Tagging messages

Starting with syslog-ng 3.1, it is also possible to label the messages with custom tags. Tags are simple labels, identified by their names, which must be unique. Currently syslog-ng can tag a message at two different places:

When syslog-ng receives a message, it automatically adds the .source.<id_of_the_source_statement> tag to the message. Use the tags() option of the source to add custom tags, and the tags() option of the filters to select only specific messages.

Note
  • Tagging messages and also filtering on the tags is very fast, much faster then other types of filters.

  • Tags are available locally, that is, if you add tags to a message on the client, these tags will not be available on the server.

  • To include the tags in the message, use the ${TAGS} macro in a template. Alternatively, if you are using the IETF-syslog message format, you can include the ${TAGS} macro in the .SDATA.meta part of the message. Note that the ${TAGS} macro is available only in syslog-ng OSE 3.1.1 and later.

For an example on tagging, see Example 8.11, Adding tags and filtering messages with tags.

8.3.6. Filter functions

The following functions may be used in the filter statement, as described in Section 8.3, Filters.

Name Description
facility() Filter messages based on the sending facility.
filter() Call another filter function.
host() Filter messages based on the sending host.
level() or priority() Filter messages based on their priority.
match() Use a regular expression to filter messages based on a specified header or content field.
message() Use a regular expression to filter messages based their content.
netmask() Filter messages based on the IP address of the sending host.
program() Filter messages based on the sending application.
source() Select messages of the specified syslog-ng OSE source statement.
tags() Select messages having the specified tag.

Table 8.3. Filter functions available in syslog-ng OSE


facility()

Synopsis: facility(<facility-name>) or facility(<facility-code>) or facility(<facility-name>..<facility-name>)

Description:Match messages having one of the listed facility code.

The facility() filter accepts both the name and the numerical code of the facility or the importance level. Facility codes 0-23 are predefined and can be referenced by their usual name. Facility codes above 24 are not defined.

You can use the facility filter the following ways:

  • Use a single facility name, for example, facility(user)

  • Use a single facility code, for example, facility(1)

  • Use a facility range (works only with facility names), for example, facility(local0..local5)

The syslog-ng application recognizes the following facilities: (Note that some of these facilities are available only on specific platforms.)

Numerical Code Facility name Facility
0 kern kernel messages
1 user user-level messages
2 mail mail system
3 daemon system daemons
4 auth security/authorization messages
5 syslog messages generated internally by syslogd
6 lpr line printer subsystem
7 news network news subsystem
8 uucp UUCP subsystem
9 cron clock daemon
10 authpriv security/authorization messages
11 ftp FTP daemon
12 ntp NTP subsystem
13 security log audit
14 console log alert
15 solaris-cron clock daemon
16-23 local0..local7 locally used facilities (local0-local7)

Table 8.4. syslog Message Facilities recognized by the facility() filter


filter()

Synopsis: filter(filtername)

Description:Call another filter rule and evaluate its value.

host()

Synopsis: host(regexp)

Description:Match messages by using a regular expression against the hostname field of log messages.

level() or priority()

Synopsis: level(<priority-level>) or level(<priority-level>..<priority-level>)

Description:The level() filter selects messages corresponding to a single importance level, or a level-range. To select messages of a specific level, use the name of the level as a filter parameter, for example use the following to select warning messages:

level(warning)

To select a range of levels, include the beginning and the ending level in the filter, separated with two dots (..). For example, to select every message of error or higher level, use the following filter:

level(err..emerg)

The level() filter accepts the following levels: emerg, alert, crit, err, warning, notice, info, debug.

match()

Synopsis: match(regexp)

Description:Match a regular expression to the headers and the message itself (that is, the values returned by the MSGHDR and MSG macros). Note that in syslog-ng version 2.1 and earlier, the match() filter was applied only to the text of the message, excluding the headers. This functionality has been moved to the message() filter.

To limit the scope of the match to a specific part of the message (identified with a macro), use the match(regexp value("MACRO")) syntax. Do not include the $ sign in the parameter of the value() option.

The value() parameter accepts both built-in macros and user-defined ones created with a parser or using a pattern database. For details on macros and parsers, see Section 11.1.2, Templates and macros, Section 12.2, Parsing messages, and Section 13.2.1, Using parser results in filters and templates.

message()

Synopsis: message(regexp)

Description:Match a regular expression to the text of the log message, excluding the headers (that is, the value returned by the MSG macros). Note that in syslog-ng version 2.1 and earlier, this functionality was performed by the match() filter.

netmask()

Synopsis: netmask(ip/mask)

Description:Select only messages sent by a host whose IP address belongs to the specified IP subnet. Note that this filter checks the IP address of the last-hop relay (the host that actually sent the message to syslog-ng), not the contents of the HOST field of the message.

program()

Synopsis: program(regexp)

Description:Match messages by using a regular expression against the program name field of log messages.

source()

Synopsis: source id

Description:Select messages of a source statement. This filter can be used in embedded log statements if the parent statement contains multiple source groups — only messages originating from the selected source group are sent to the destination of the embedded log statement.

tags()

Synopsis: tag

Description:Select messages labeled with the specified tag. Every message automatically has the tag of its source in .source.<id_of_the_source_statement> format. This option is available only in syslog-ng 3.1 and later.

Example 8.11. Adding tags and filtering messages with tags
source s_tcp {
    tcp(ip(192.168.1.1) port(1514) tags("tcp", "router"));
    };

Use the tags() option of the filters to select only specific messages:

filter f_tcp {
       tags(".source.s_tcp");
    };

    filter f_router {
       tags("router");
    };
Note

Starting with version 3.2, syslog-ng OSE automatically adds the class of the message as a tag using the .classifier.<message-class> format. For example, messages classified as "system" receive the .classifier.system tag. Use the tags() filter function to select messages of a specific class.

filter f_tag_filter {tags(".classifier.system");};

8.4. Dropping messages

To skip the processing of a message without sending it to a destination, create a log statement with the appropriate filters, but do not include any destination in the statement, and use the final flag.

Example 8.12. Skipping messages

The following log statement drops all debug level messages without any further processing.

filter demo_debugfilter { level(debug); };
log { source(s_all); filter(demo_debugfilter); flags(final); };

Chapter 9. Global options of syslog-ng OSE

9.1. Configuring global syslog-ng options

The syslog-ng application has a number of global options governing DNS usage, the timestamp format used, and other general points. Each option may have parameters, similarly to driver specifications. To set global options, add an option statement to the syslog-ng configuration file using the following syntax:

options { option1(params); option2(params); ... };
Example 9.1. Using global options

To disable domain name resolving, add the following line to the syslog-ng configuration file:

options { use_dns(no); };

For a detailed list of the available options, see Section 9.2, Global options. For important global options and recommendations on their use, see Chapter 17, Best practices and examples.

9.2. Global options

The following options can be specified in the options statement, as described in Section 9.1, Configuring global syslog-ng options.

bad_hostname()

Accepted values: regular expression
Default: no

Description:A regexp containing hostnames which should not be handled as hostnames.

chain_hostnames()

Accepted values: yes | no
Default: no

Description:Enable or disable the chained hostname format. If the log message is forwarded to the logserver via a relay, and the chain_hostnames() option is enabled, the relay adds its own hostname to the hostname of the client, separated with a