From this blog you will learn how to compile a simple configuration for Suricata on the Turris Omnia router and how to configure syslog-ng to forward its log messages to a central log collector. In the second part of this blog, you will learn why and how to parse the JSON-based log messages from Suricata and a few tips on what you can do with these messages, for example sending alerts or sending logs for visualization to Elasticsearch and Kibana. Although this is a specific example, you can adopt it with minimal changes to your own environment.
What is Suricata
Suricata is a free and open source Intrusion Detection System (IDS). It can inspect your network traffic, detect several types of sophisticated attacks and alert you about problems. It can also create a log file about any network connection helping you in network forensics, and save log messages in a nicely structured JSON format.
Suricata is also available on the Turris Omnia router. All you have to do is install the package and enable it using the web interface of the router.
Configuring Suricata and syslog-ng on the Turris Omnia
Installing and configuring Suricata
As a first step, install and configure Suricata. It comes with a ready-to-use configuration and should work even if you do not change anything.
The main configuration is available in the following file: /etc/surricata/suricata.yaml.
A comment in the configuration claims that using a more specific address range for the local network is better for alert accuracy and performance. I have changed the HOME_NET variable to my local /24 network and also disabled stats logging, as it is a quickly growing file and I never looked at it.
There are four log files created by Suricata under the /var/log/suricata directory:
- suricata.log: startup messages of Suricata
- stats.log: regular statistics about your network traffic
- fast.log: suspicious activity found by Suricata
- eve.json: the traffic of your local network in JSON messages, and the alerts sent to fast.log in JSON format
The syslog-ng application on Turris Omnia
The syslog-ng version on the Turris Omnia is quite old. It is perfectly OK for the tasks it is configured for on the Turris, but lacks many interesting features (parsing JSON and other message formats, encryption, and so on). What it means for us is that we can only use it to forward logs to another location for further processing. Even then, we have to make sure that messages are forwarded over a trusted network (because the lack of encryption) or that there is a VPN connection to the target server.
To configure syslog-ng on the Turris, create a new file under the /etc/syslog-ng.d directory. I have named it suricata.conf. Edit the file so that it has a similar content:
Configuring syslog-ng on Turris Omnia
IP and port: Replace the IP addresses and port numbers to match your environment. Use separate ports (in this example: TCP: 514, Suricata: 5514) to make it easier to configure the central syslog-ng server.
program-override(): This option is there to make sure that you can search for Suricata based on the application name in your log messages.
flags(no-parse): Required, because otherwise syslog-ng would attempt to interpret the file as syslog messages and would lose part of the JSON message along the way. It would make the result unusable on the receiving end.
source s_suricata {
file("/var/log/suricata/eve.json"
program-override("suricata")
flags(no-parse));
};
destination d_tcp{
tcp("192.168.1.184" port(514));
};
destination d_suricata{
tcp("192.168.1.184" port(5514));
};
log {
source(src);
source(kernel);
destination(d_tcp);
};
log {
source(s_suricata);
destination(d_suricata);
};
If you only want to forward the Surricata logs, but you do not want to collect the rest of the logs from the Turris Omnia router, delete the first TCP destination and log statement:
destination d_tcp{tcp("192.168.1.184" port(514));};
log {source(src); source(kernel); destination(d_tcp); };
However, it is recommended that if you already use a central syslog-ng log collection server, you also collect the system logs from the Turris Omnia router too.
Configuring the central syslog-ng server
Create a configuration file on the central syslog-ng server. This will help you test whether the log transfer is working properly.
You will now define:
- two network sources
- two file destinations
- and write all incoming logs to files unmodified
Because your central syslog-ng server might reside on very diverse platforms, I cannot recommend a name or location for your configuration file.
Configuring syslog-ng on the central syslog-ng server
IP and port: Replace the IP addresses and port numbers to match your environment.
source s_net {
tcp(ip("0.0.0.0")
port(514));
};
source s_suricata {
tcp(ip("0.0.0.0")
port(5514));
};
destination d_turris {
file("/var/log/turris");
};
destination d_suricata {
file("/var/log/suricata");
};
log {
source(s_net);
destination(d_turris);
};
log {
source(s_suricata);
destination(d_suricata);
};
Testing the central syslog-ng server configuration:
Save the configuration file and reload syslog-ng so that the configuration changes take effect.
Use the tail command to check whether data has arrived to your central syslog-ng server. If you cannot see any messages in the files, try the following:
Reload syslog-ng on the Turris, so that it recognizes quicker that your central server is ready to receive messages.
Check if the configured ports are open on your firewall.
If you use SELinux, check /var/log/audit/audit.log if it blocks syslog-ng in any way.
Processing Suricata logs with syslog-ng
Up until now, the configuration files have also included the system logs of Turris. From now on we will only focus on Suricata logs.
Sending logs to Loggly or other LaaS
Loggly and many other Logging as a service (LaaS) providers can parse JSON-based log messages automatically. If you have such an account, you can forward your log messages for further analysis there. You can find more information about sending messages to Loggly at https://www.loggly.com/blog/sending-json-format-logs-syslog-ng/. You do not even have to use the JSON template function, because the message is already in JSON format. Make sure that you enable TLS encryption, because you will be sending logs over the Internet.
Starting with version 3.8 of syslog-ng you can take advantage of disk-based buffering. This means that even if your network connection to Loggly is down, messages are not lost, because they can wait in the buffer until connection is restored.
Storing logs in Elasticsearch
Nowadays, one of the most popular destinations is Elasticsearch destination. You can visualize these logs in Kibana.
In this post, I only include the relevant parts of the configuration. The HTTP client mode I use in the example works fromversion 3.8 of syslog-ng. The advantage of this is that it works independent of the Elasticsearch server version and the required JAR files are bundled together with syslog-ng Java modules.
You can read about why and how to get started with Elasticsearch and Kibana in my blog “Logging to Elasticsearch made simple with syslog-ng”.
Steps:
Step 1.
First, define a Suricata source so that you can receive the log messages from the Turris box. Make sure that you use the same port number as on the sender side:
source s_suricata {
tcp(ip("0.0.0.0") port(5514));
};
Step 2.
Next, define a parser for JSON-based messages. You will use the “suricata” prefix, because all messages will come from Suricata. This way name-value-pairs from Suricata do not collide with syslog-ng’s own.
You might ask, why to parse messages if Elasticsearch expects JSON-formatted messages. Parsing allows you to use the content of different fields within syslog-ng. You can filter out some of the network events from the stream or send an email alert when Suricata finds something suspicious.
parser p_json {
json-parser (prefix("suricata."));
};
Step 3.
Now define the Elasticsearch destination itself. It has a few changes compared to the default settings.
client-mode(): Configure it to use HTTP
flush-limit(): Set it to 1 to make sure that you see new messages in Kibana as soon as they are received by syslog-ng, and also, to prevent lag. The default value, 5000, is good for situations where you need high performance, but for Turris, where there are no new messages regularly, it can create quite some lag.
template(): Because in the previous step, you have included the parsed Suricata fields with the “suricata” prefix, you can safely exclude the original JSON string (format-json) in the MESSAGE macro (–exclude MESSAGE). The rfc5424 scope (–scope rfc5424) makes sure that regular syslog fields are included, and therefore messages from Suricata can easily be searched together with the rest of the log messages.
destination d_elastic {
elasticsearch2 (
cluster("syslog-ng")
client-mode("http")
index("syslog-ng")
type("test")
flush-limit("1")
template("$(format-json --scope rfc5424 --scope nv-pairs --exclude MESSAGE --exclude DATE --key ISODATE)")
)
};
Step 4.
In the last step, connect the previous three configuration snippets in a single log path.
log {
source(s_suricata);
parser(p_json);
destination(d_elastic);
};
Restart syslog-ng. After the restart process, you should see incoming Suricata logs in Kibana.
Filtering messages
If you want to receive an email on each alert by Suricata or just simply collect them to a separate file, use filters. In the following example, you will add a filter and a destination to the previous syslog-ng configuration file and modify the log path to include them. This way, all logs are still stored to Elasticsearch, but alerts are also saved to a file:
filter f_alert {
"${suricata.event_type}" eq "alert"
};
destination d_alert {
file("/var/log/alert");
};
log {
source(s_suricata);
parser(p_json);
destination(d_json);
destination(d_elastic);
filter(f_alert);
destination(d_alert);
};
You can use similar filters if you want to send only specific log types to Graphite for visualization, or to Riemann for alerting.
Summary
This was just a quick introduction to Suricata and syslog-ng. The examples are based on Suricata running on the Turris Omnia router, but you can adopt them with minimal changes to your own environment.
These examples are good to whet your appetite, but you should read the official documentation if you plan to use it in production, because it requires a more in-depth knowledge of syslog-ng. Save the following references:
-
- Suricata documentation: https://suricata-ids.org/docs/
- The syslog-ng application documentation:https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide
- Turris Omnia: https://www.turris.cz/doc/en/start
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us.