Syslog-ng Premium Edition can send and receive log messages in a reliable way over the TCP transport layer using the Reliable Log Transfer Protocol™ (RLTP™). RLTP™ is a new transport protocol that prevents message loss during connection breaks. It detects the last received message on the receiving end and then starts resending messages from that point, ensuring messages are not duplicated at the receiving end in case of a connection break.
Syslog-ng stores messages on the local hard disk if the central log server or the network connection becomes unavailable. The syslog-ng application automatically sends the stored messages to the server when the connection is re-established, in the same order the messages were received. The disk buffer is persistent - no messages are lost even if syslog-ng is restarted.
Flow-control uses a control window to determine if there is free space in the output buffer of syslog-ng for new messages. If the output buffer is full and the destination cannot accept new messages for some reason, for example it's overloaded or the network connection has become unavailable. In such cases, syslog-ng stops reading messages from the source until some messages have been successfully sent to the destination.
Log messages may contain sensitive information that should not be accessed by third parties. Therefore, syslog-ng Premium Edition uses the Transport Layer Security (TLS) protocol to encrypt the communication. TLS also allows the mutual authentication of the host and the server using X.509 certificates.
The Premium Edition of syslog-ng can store log messages securely in encrypted, compressed, indexed and timestamped binary files, so any sensitive data is available only for authorized personnel who have the appropriate encryption key. Timestamps can be requested from external Timestamping Authorities.
The syslog-ng application is optimized for performance, and can handle an enormous amount of messages. Depending on its exact configuration, it can process over half a million messages per second in real-time, and over 24 GB of raw logs per hour on standard server hardware.
With the syslog-ng client-relay architecture, IT organizations can collect log messages from more than 10,000 log sources across a geographically distributed environment on one central log server.
Syslog-ng Premium Edition allows you to granularly select which statistics of syslog-ng PE you want to monitor. The statistics are available as structured name-value pairs, so you can format the output similarly to other log messages. That way, you can easily convert the statistics and metrics, for example, into JSON or WELF format, and send the results into your enterprise monitoring solution (for example, IBM Tivoli Netcool, Riemann, Redis, or Graphite).
Syslog-ng Premium Edition can natively collect and process log messages from SQL databases, enabling users to easily manage log messages from a wide variety of enterprise software and custom applications. The syslog-ng Agent for Windows is an event log collector and forwarder application for Microsoft Windows platforms.
Some applications use many different log files, and sometimes these files are not even located in the same folder. Automatically generated file and folder names are also often a problem. To solve these issues, the filenames and paths specifying the log files read by syslog-ng can include wildcards, and syslog-ng can automatically scan entire subfolder trees for the specified files. The syslog-ng Premium Edition application is also able to process multi-line log messages, for example, Apache Tomcat messages.
Many large organizations need to send their logs to multiple log analysis tools. Different groups, including IT operations, IT security and corporate risk and governance, need access to the same log data but have different log analysis goals and tools. Most log analysis and SIEM solutions can receive syslog messages. The syslog-ng application can send logs directly to SQL databases, MongoDB and Hadoop Distributed File System (HDFS) nodes, or use the Standard Network Management Protocol (SNMP) and Simple Mail Transfer Protocol (SMTP) for other destinations.
The syslog-ng application can sort the incoming log messages based on their content and various parameters like the source host, application, and priority. Directories, files, and database tables can be created dynamically using macros. Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important log messages to the selected destinations.
The Python Log Parser allows you to write your own parsers in Python. Practically, that way you can process the log message (or parts of the log message) any way you need. You can also write your own template functions in Python.
Syslog-ng can use an external database file to append custom name-value pairs to incoming logs, thus extending, enriching, and complementing the data found in the log message. You can also correlate and aggregate information from log messages using a few simple filters that are similar to SQL GROUPBY statements.
The syslog-ng application can compare the contents of the log messages to a database of predefined message patterns.
By comparing log messages to known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can then be used to classify the type of the event described in the log message. The message classes can be customized, and for example can label the messages as user login, application crash, file transfer, etc. events.
In addition to classifying messages, you can also add different tags which can be used later for filtering messages. For example, to collect messages tagged as user_login to a separate file or to perform conditional post processing on the tagged messages.
Syslog-ng also makes real time event correlation possible. This can be useful in many different situations, for example important data for a single event is often scattered into multiple syslog messages. Also login and logout events are often logged far away from each other, even in different log files, making log analysis difficult. Using correlation these can be collected into a single new message.