Table of Contents

Preface

1. Summary of contents

2. Target audience and prerequisites

3. Products covered in this guide

4. Typographical conventions

5. Contact and support information

5.1. Sales contact

5.2. Support contact

5.3. Training

6. About this document

6.1. Summary of changes

6.2. Feedback

6.3. Acknowledgments

1. Introduction to syslog-ng

1.1. What syslog-ng is

1.2. What syslog-ng is not

1.3. Why is syslog-ng needed?

1.4. What is new in syslog-ng Open Source Edition 3.3?

1.5. Who uses syslog-ng?

1.6. Supported platforms

2. The concepts of syslog-ng

2.1. The philosophy of syslog-ng

2.2. Logging with syslog-ng

2.2.1. The route of a log message in syslog-ng

2.3. Modes of operation

2.3.1. Client mode

2.3.2. Relay mode

2.3.3. Server mode

2.4. Global objects

2.5. Timezones and daylight saving

2.5.1. A note on timezones and timestamps

2.6. Stable and feature releases of syslog-ng OSE

2.7. The license of syslog-ng OSE

2.8. High availability support

2.9. The structure of a log message

2.9.1. BSD-syslog or legacy-syslog messages

2.9.2. IETF-syslog messages

2.9.3. Message representation in syslog-ng OSE

2.9.4. Structuring macros, metadata, and other value-pairs

3. Installing syslog-ng

3.1. Compiling syslog-ng from source

3.2. Uninstalling syslog-ng

3.3. Configuring Microsoft SQL Server to accept logs from syslog-ng

4. The syslog-ng OSE quick-start guide

4.1. Configuring syslog-ng on client hosts

4.2. Configuring syslog-ng on server hosts

4.3. Configuring syslog-ng relays

4.3.1. Configuring syslog-ng on relay hosts

4.3.2. How relaying log messages works

5. The syslog-ng OSE configuration file

5.1. The syslog-ng configuration file

5.1.1. The configuration syntax in detail

5.1.2. Notes about the configuration syntax

5.2. Global and environmental variables

5.3. Loading modules

5.3.1. Loading modules

5.4. Managing complex syslog-ng configurations

5.4.1. Including configuration files

5.4.2. Reusing configuration blocks

6. Collecting log messages — sources and source drivers

6.1. How sources work

6.2. Collecting internal messages

6.3. Collecting messages from text files

6.3.1. file() source options

6.4. Collecting messages from named pipes

6.4.1. pipe() source options

6.5. Collecting process accounting logs on Linux

6.5.1. pacct() options

6.6. Receiving messages from external applications

6.6.1. program() source options

6.7. Collecting messages on Sun Solaris

6.7.1. sun-streams() source options

6.8. Collecting messages using the IETF syslog protocol

6.8.1. syslog() source options

6.9. Collecting the system-specific log messages of a platform

6.10. Collecting messages from remote hosts using the BSD syslog protocol

6.10.1. tcp(), tcp6(), udp() and udp6() source options

6.11. Collecting messages from UNIX domain sockets

6.11.1. unix-stream() and unix-dgram() source options

7. Sending and storing log messages — destinations and destination drivers

7.1. Storing messages in plain-text files

7.1.1. file() destination options

7.2. Storing messages in a MongoDB database

7.2.1. mongodb() destination options

7.3. Sending messages to named pipes

7.3.1. pipe() destination options

7.4. Sending messages to external applications

7.4.1. program() destination options

7.5. Storing messages in an SQL database

7.5.1. Using the sql() driver with an Oracle database

7.5.2. Using the sql() driver with a Microsoft SQL database

7.5.3. The way syslog-ng interacts with the database

7.5.4. sql() destination options

7.6. Sending messages to a remote logserver using the IETF-syslog protocol

7.6.1. syslog() destination options

7.7. Sending messages to a remote logserver using the legacy BSD-syslog protocol

7.7.1. tcp(), tcp6(), udp(), and udp6() destination options

7.8. Sending messages to UNIX domain sockets

7.8.1. unix-stream() and unix-dgram() destination options

7.9. Sending messages to a user terminal — usertty() destination

8. Routing messages: log paths and filters

8.1. Log paths

8.1.1. Embedded log statements

8.1.2. Log path flags

8.2. Managing incoming and outgoing messages with flow-control

8.2.1. Flow-control and multiple destinations

8.2.2. Configuring flow-control

8.3. Filters

8.3.1. Using filters

8.3.2. Combining filters with boolean operators

8.3.3. Comparing macro values in filters

8.3.4. Using wildcards, special characters, and regular expressions in filters

8.3.5. Tagging messages

8.3.6. Filter functions

8.4. Dropping messages

9. Global options of syslog-ng OSE

9.1. Configuring global syslog-ng options

9.2. Global options

10. TLS-encrypted message tranfer

10.1. Secure logging using TLS

10.2. Encrypting log messages with TLS

10.2.1. Configuring TLS on the syslog-ng clients

10.2.2. Configuring TLS on the syslog-ng server

10.3. Mutual authentication using TLS

10.3.1. Configuring TLS on the syslog-ng clients

10.3.2. Configuring TLS on the syslog-ng server

10.4. TLS options

11. Manipulating messages

11.1. Customizing message format

11.1.1. Formatting messages, filenames, directories, and tablenames

11.1.2. Templates and macros

11.1.3. Hard vs. soft macros

11.1.4. Macros of syslog-ng OSE

11.1.5. Using template functions

11.1.6. Template functions of syslog-ng OSE

11.2. Modifying messages

11.2.1. Conditional rewrites

11.3. Regular expressions

11.3.1. Types and options of regular expressions

11.3.2. Optimizing regular expressions

12. Parsing and segmenting structured messages

12.1. Parsing messages

12.2. Options of CSV parsers

13. Processing message content with a pattern database

13.1. Classifying log messages

13.1.1. The structure of the pattern database

13.1.2. How pattern matching works

13.1.3. Artificial ignorance

13.2. Using pattern databases

13.2.1. Using parser results in filters and templates

13.2.2. Downloading sample pattern databases

13.3. Correlating log messages

13.3.1. Referencing earlier messages of the context

13.4. Triggering actions for identified messages

13.5. Creating pattern databases

13.5.1. Using pattern parsers

13.5.2. What's new in the syslog-ng pattern database format V4

13.5.3. The syslog-ng pattern database format

14. Statistics of syslog-ng

15. Multithreading and scaling in syslog-ng OSE

15.1. Multithreading concepts of syslog-ng OSE

15.2. Configuring multithreading

15.3. Optimizing multithreaded performance

16. Troubleshooting syslog-ng

16.1. Possible causes of losing log messages

16.2. Creating syslog-ng core files

16.3. Collecting debugging information with strace, truss, or tusc

16.4. Running a failure script

16.5. Stopping syslog-ng

17. Best practices and examples

17.1. General recommendations

17.2. Handling lots of parallel connections

17.3. Handling large message load

17.4. Using name resolution in syslog-ng

17.4.1. Resolving hostnames locally

17.5. Collecting logs from chroot

Appendix 1. The syslog-ng manual pages

loggen — Generate syslog messages at a specified rate

pdbtool — An application to test and convert syslog-ng pattern database rules

syslog-ng — syslog-ng system logger application

syslog-ng.conf — syslog-ng configuration file

syslog-ng-ctl — Display message statistics and enable verbose, debug and trace modes in syslog-ng Open Source Edition

Appendix 2. GNU Lesser General Public License

2.1. Preamble

2.2. TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

2.2.1. Section 0

2.2.2. Section 1

2.2.3. Section 2

2.2.4. Section 3

2.2.5. Section 4

2.2.6. Section 5

2.2.7. Section 6

2.2.8. Section 7

2.2.9. Section 8

2.2.10. Section 9

2.2.11. Section 10

2.2.12. Section 11

2.2.13. Section 12

2.2.14. Section 13

2.2.15. Section 14

2.2.16. NO WARRANTY Section 15

2.2.17. Section 16

2.3. How to Apply These Terms to Your New Libraries

Appendix 3. GNU General Public License

3.1. Preamble

3.2. TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

3.2.1. Section 0

3.2.2. Section 1

3.2.3. Section 2

3.2.4. Section 3

3.2.5. Section 4

3.2.6. Section 5

3.2.7. Section 6

3.2.8. Section 7

3.2.9. Section 8

3.2.10. Section 9

3.2.11. Section 10

3.2.12. NO WARRANTY Section 11

3.2.13. Section 12

3.3. How to Apply These Terms to Your New Programs

Appendix 4. Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License

Glossary

List of syslog-ng OSE parameters

Index