News

We have moved forward again. From 2010 we issue CPE certificates about our trainings. if you need that, just indicate on the traning entry-form. The certificate do not contain CPE points, but hours, that helps everybody counting the points. It can be different for every profession. In addition CPE points are maximized if the training is product specific. The certified hours are 6 netto hours a day, therefore we issue 6 hours for SSB, 12 hours for syslog-ng PE and 6 hours for SCB. Fur further info wisit ISACA website.

It was in November 2007 when the initial commit of X Control Box was pushed to our VC server -- the idea was to create a framework based on the current Shell Control Box codebase that can act as a base for all our future appliance-like products. We took the chance to rewrite everything that needed to be rewritten: we added proper widgeting for our interface elements, cleared up the CSS code and the browser-server communication, the search interfaces, the whole permission system and added detailed changelogging for everything that can be changed on the UI. The resulting codebased proved to be usable: it took only a couple of weeks to create the initial pre-alpha version of syslog-ng Store Box, and ever since then we've really benefited from having a well-usable common framework into which 2630 commits have added 300k+ lines of code and templates by now.

The video below shows this progress from the beginning to the current state using the amazing visualization tool Gource. Make sure you click over to YouTube and watch it in high quality, it really worth it!

Since the last post, I could hack a couple of hours on the plugins branch, which now compiles. The plugin framework is capable for supporting a quite important core functionality: all socket like sources/destinations are now found in an external plugin called "afsocket".

The reason I've started with afsocket is to make syslog-ng a bit less dependant on OpenSSL. A couple of distributions didn't include syslog-ng 3.0 in their current releases, because it uses OpenSSL from /usr, while syslog-ng should remain in the root directory.

By separating afsocket from the syslog-ng core, I can compile afsocket with and without TLS support, which can be put into separate packages. Thus syslog-ng can operate without OpenSSL.

And the same plugin framework will enable me to create a wide variety of plugins. My ideas:

• Plugins for all syslog-ng components (source, destination, filter, rewrite, parser)
• Python scriptability (a simple correllation engine in Python?)
  
• macro transformation functions, for example: $(stripslashes $macro), usable anywhere in templates and stripslashes a plugin that is invoked whenever such an expansion occurs
• Hooks for transforming the log message as it enters syslog-ng (to fix parsing errors for example),

Do you have other ideas? Please post them as comments or as emails to the mailing list.

Again, this functionality is experimental, and I'm still going to rebase the current code and will probably be integrated to syslog-ng 3.2. I got to release 3.1 final first though. :)

Things have been a little rough last couple of months, that's why I haven't posted here. I'm in a rush right now as well, but I just wanted to let you know that I have started working on modularizing syslog-ng.

It is only a preliminary prototype, and as of now it doesn't compile, but the way it's going to work is already visible: each plugin will have its own plugin and with some trickery the large syslog-ng.conf parser will call out to the plugin parser. The user will recognize such a plugin as an integral part of syslog-ng.

E.g. this is a sample configuration file:

@version: 3.0
@module: dummy

...

destination d_dummy { dummy(dummy_opt(yes)); };

...

See the dummy plugin code in my git repository, in the "plugins" branch. Please note that that branch is going to be rebased a couple of times yet, I've released it in the spirit of "release early, release often".

I hope to get some of the recent contributions into plugins, instead of bloating the core syslog-ng code. For example output colorization. I'm also thinking about adding built-in scripting support via Python.

I have just found a bug in syslog-ng OSE 3.1beta2. When you use keep_hostname(yes) in the options, then at those sources where this field is empty (e.g. messages arriving on /dev/log), it eliminates the hostname, somehow this way:

myuser@myhost:~$ logger hello
myuser@myhost:~$ grep hello /var/log/syslog
myuser@myhost:~$ 2010-03-05T14:05:33+01:00 myuser[]: hello

After the patch:

myuser@myhost:~$ logger hello
myuser@myhost:~$ grep hello /var/log/syslog
myuser@myhost:~$ 2010-03-05T14:05:33+01:00 myhost myuser[]: hello

Not a big bug, but was very embrassing. The patch is in Bazsi's git tree out there. Thank you Bazsi!

I wish you a pleasant weekend!

I have finished the first online training. We used webex and phone conference. I must tell it was very tiring, but fun. Hopefully there will be more and more training whit this green way of teaching.

Let's imagine a webserver with many virtual host. It is jailed to a chroot, but it could be on a virtual machine (even a real one). I do want to collect the logs with the following requirements:

• No messages available by apache
• No need to handle hundres of sources. Transfer every log on one channel (okay let's use two;-)
• No need to rotate and support archive
• Use secure protocoll, but I guess I do not have to even mention;-)

Les's see how to start. The first step is force apache to log to a named pipe. Not a big trick, just create the pipes. Use mknod:

mknod /var/log/apache/access.log p
mknod /var/log/apache/error.log p

Now set all the virtualhost to use them:

ErrorLog /var/log/apache2/error.log
LogLevel debug
CustomLog /var/log/apache2/access.log combined

Important, that all the virtual hosts will use these nodes, therefor logs are not stored in the apache chroot any more. Now we have to read messages some way. The only small problem is apache uses Common Log Format, which is fare from any standard syslog format. Fortunateley it is possible to modify it in apache conf. The original looks this:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Let's change it to comply to syslog RFC mine looks this:

LogFormat "<123>Jan 12 12:12:12 %v apache[666]: %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Do not care about strange PRI field and the fixed timestamp neither the funny PID. I only used it because google indexing;-) Which really count is hostname, which continas the name of the virtual host (%v = virutal host) and the original combined message is delivered at the MESSAGE field ($MSGONLY). So now we are ready with the apache side, let's focus on the syslog-ng side. The first step is reading the two pipes:

source s_apache_access {
pipe("/var/log/apache2/access.log);
};

On the destination (writin side) we simple sore in different files by hostname field:

destination d_apache_access {
file("/var/log/apache2/$FULLHOST" template("$MSGONLYn") template-escape(no) owner("root") group("adm") perm(0640));

file("/var/log/archive/$R_YEAR/apache/$R_MONTH/$FULLHOST.$R_DAY" template("$MSGONLYn") template-escape(no) owner("root") group("adm") perm(0640) create_dirs(yes) dir_owner("root") dir_group("adm"));
};

I like solving archiving under the same time, therefore the second destination does it. Now I have to wire the client and the server side:

destination d_logserver_net {
tcp("1.2.3.4" port(514)
tls(ca_dir("/opt/syslog-ng/etc/syslog-ng/ca.d")
key_file("/opt/syslog-ng/etc/syslog-ng/key.d/client.key")
cert_file("/opt/syslog-ng/etc/syslog-ng/cert.d/client_cert.pem")));
};

log {
source(s_apache_access);
destination(d_logserver_net);
};

On the server side we receive the messages:

source s_apache_net {
tcp(ip(0.0.0.0) port(1999)
tls( key_file("/opt/syslog-ng/etc/syslog-ng/key.d/syslog-ng.key")
cert_file("/opt/syslog-ng/etc/syslog-ng/cert.d/syslog-ng.cert")
ca_dir("/opt/syslog-ng/etc/syslog-ng/ca.d")) );
};

log {
source(s_apache);
destination(d_apache_access);
};

We are ready. It was not very difficult, was it?