The syslog-ng Agent for IBM System i Administrator Guide

UNIX and Linux systems have long supported the collection of system and application logs under the general name of “syslogs”. Many system logs in these environments are text files that are created by various applications and services. System logs collect important information about the health of the server, and about the security environment. They are important sources of information about potential or real security threats. The format of system logs can be very informal and include just informational text. However, there is a formal standard for system logs in RFC 3164. When logs are collected by the syslog-ng Agent application they are automatically placed in the RFC 3164 syslog format.

Some Security Information Management (SIM) products use a special log format called the Common Event Format (CEF). syslog-ng Agent for IBM System i supports the CEF format for customers who want to send information directly to a log monitoring systems in CEF format.

The system audit journal QAUDJRN can be configured to capture a large number of security and system management events. The syslog-ng Agent solution can capture the entries in the QAUDJRN journal in real time, format them for the Syslog or CEF format, and send them to a syslogd or Syslog-ng server. The syslog server can be running on the System i or any remote server. The IBM System i Security Reference Manual provides information on how to change system values, user profiles, and objects to capture various system change and security events.

The System i QHST message log collects important job, user, and security information. syslog-ng Agent can be configured to capture QHST entries and convert them to the security audit journal (QAUDJRN) for a permanent record of system activity. When this option is enabled, syslog-ng Agent works as a background subsystem job to collect the logs in order to avoid losing log messages to du automated system clean, user deletions, etc.

The system operator message queue QSYSOPR receives application and system messages. The syslog-ng Agent software can capture the messages in the QSYSOPR message queue, format them to the syslog or CEF standard, and send them to a syslog server on the same System i or to a syslog server on a remote system.

syslog-ng Agent provides an easy way to view user profile authorities and change user auditing. In addition to viewing a user's explicit authorities on the user profile, syslog-ng Agent will inspect any group profile or supplemental group profile to determine any authorities that are inherited from these users. You can then view the authorities and change the audit setting for each user. syslog-ng Agent recommends an audit level for each user and you can change the audit setting.

syslog-ng Agent provides command and API interfaces to allow a user program to create Syslog or CEF event records. User applications can provide simple text values for messages and specify the priority and facility ID for the message. Any user application can be enabled for syslog or CEF application messages.

syslog-ng Agent gives you the ability to filter the system audit journal (QAUDJRN) entries that you want to send to a central log server. This can reduce the amount of network traffic and the type of events you transmit to the server. syslog-ng Agent provides a complete list of log system security audit journal event types and the security administrator can edit the events to suppress transmission to the central log server.

Many System i customers use open source solutions such as the Apache web server, OpenSSH, Perl, or MySQL on the System i. These open source applications support normal logging functions. When enabled these applications send logs to the IFS directory /var/log. You can automatically send these logs to a Syslogd or Syslog-ng server using the Balabit Syslog-ng Open Source Edition or Premium Edition product. See the separate documentation on the Syslog-ng product.

Patrick Townsend & Associates has partnered with Balabit IT Security to bring the Syslog-ng product to the System i platform. This application can be installed and run as a service directly on the System i platform. The application runs in the Portable Application Solutions Environment (PASE). When installed on the System i the Syslog-ng server can be configured to send any logs in the IFS file system.

Syslog-ng is the next generation of server processing for system log management. Syslog-ng is available in an open source version and in a premium edition that provides additional security and management functions (see web site http://www.balabit.com/). Syslog-ng is shipped with most Linux distributions and is available for a wide variety of platforms. Syslog-ng can act in one of three roles:

By consolidating logs on a central server Syslog-ng enables intelligent evaluation of system information in one location. See the separate documentation on the Syslog-ng software solutions.

Security information management systems attempt to monitor the health and security of all of the significant server systems in an Enterprise. Typically this means consolidating syslog and other information from servers in one repository, and then performing real-time analysis, reporting, and alerts of potential problems. A number of vendors provide software and appliance solutions for log analysis. These solutions include ArcSight ESM, Symantec SIM, LogLogic, CrossTec, and many others. syslog-ng Agent for IBM System i provides the solution needed to bring the System i platform into a common information management strategy.

The syslog-ng Agent for IBM System i product runs on any version of IBM OS/400 or i5/OS from V5R1 and later.

When you download the syslog-ng Agent product from the web site, you must unzip the product. You software provider will supply a pass phrase for extracting the files. See the Readme.txt file in the download for instructions on how to copy the software to the System i using FTP. Once you transfer the Save file to the System i you will restore the library ALLSYL100.

To upgrade the syslog-ng Agent product you must first end the ALLSYL100 subsystem and rename the library:

Endsbs sbs(allsyl100) option(*immed)
		Rnmobj obj(allsyl100) objtype(*lib) newobj(allsylold)

After renaming the library you can install the new version using the Internet download.

After installing the new version use the Apply Release Upgrade option on the Installation menu to copy your configuration information from the old library to the new version.

You can revert to a previous version of the product by renaming the new library to a save name, and then renaming the old version to the library ALLSYL100. This will restore the previous version.

Before you can configure and use the syslog-ng Agent agent and server you must enter a temporary or permanent license code. Please contact your software supplier to receive the license code. The Open Source Edition of the syslog server does not require a license code to operate.

If you will be sending system audit journal information to Syslog-ng you may wish to delete older journal receivers before starting the process. syslog-ng Agent collects journal entries from the beginning of the current chain of journal receivers. The date of all log entries is the date of the actual journal entry, but there may be a lot of historical information that you do not want to process. You should consider making a permanent backup of system audit journals before deleting them. Use the Work With Journal Attributes (WRKJRNA) command to view the journal receivers for the QAUDJRN journal.

Create the first journal receiver in a user library that is backed up on a regular basis. You can use an existing library, or you can create a library to contain the journals. The library can have public authority, but the journal receivers should exclude public authority. Use a name with a 4 or 5 digit number at the end of the name, and start with sequence number 1. The operating system will automatically increment the sequence number by 1 when creating a new journal receiver.

CRTJRNRCV
      JRNRCV(MYLIB/AUDRCV0001)
      THRESHOLD(100000)
      AUT(*EXCLUDE)
      TEXT(’Auditing Journal Receiver’)

Create the QAUDJRN journal in the QSYS library. You must have authority to the QSYS library and *AUDIT special authority. Use the following parameters for this journal:

CRTJRN
      JRN(QSYS/QAUDJRN)
      JRNRCV(JRNLIB/AUDRCV0001)
      MNGRCV(*SYSTEM)
      DLTRCV(*NO)
      AUT(*EXCLUDE)
      TEXT(’Auditing Journal’)

You specify which types of events are collected in the QAUDJRN journal by specifying the system values for QAUDLVL and QAUDLVL2. You can view a list of all of the auditing values by prompting the values when using the WRKSYSVAL command.

	Note
	In V5R4 of the operating system IBM introduced some group values to make specifying audit types easier. See the notes below.

To collect basic security information you can use the *SECURITY group value like this:

CHGSYSVAL
      SYSVAL(QAUDLVL)
      VALUE(‘*SECURITY’)

An alternative to using the *SECURITY group is to use the Change Security Audit (CHGSECAUD) command to set the basic security values and start auditing. Be careful, this command will create the journal and journal receiver, change the audit levels, and start audit collection. It is recommended that you specify the actual values with the Change System Value command rather than the Change Security Audit command.

If you want to collect more extensive security information you can add the values PGMFAIL, AUTFAIL, PGMADP, SAVRST, OFCSVR, and SERVICE:

CHGSYSVAL
      SYSVAL(QAUDLVL)
      VALUE(‘*SECURITY’ ‘*PGMFAIL’ ‘*AUTFAIL’ ‘*PGMADP’ ‘*SAVRST’
      ‘*OFCSRFV’ ‘*SERVICE’)

Certain data security regulations require that you monitor for system changes. To collect events for system changes you can add the values CREATE, DELETE, OBJAUD, OBJMGT, OPTICAL, and SAVRST:

CHGSYSVAL
      SYSVAL(QAUDLVL)
      VALUE(‘*SECURITY’ ‘*AUTFAIL’ ‘*NETCMN’ ‘*SAVRST’ ‘*OFCSRFV’
      ‘*SERVICE’ ‘*CREATE’ ‘*DELETE’ ‘*OBJAUD’ ‘*OBJMGT’
      ‘*OPTICAL’ ‘*SAVRST’)

There is a limit to the number of values you can specify in the QAUDLVL system value. Use the *AUDLVL2 special value to indicate that additional values are to be found in the QAUDLVL system value.

You should consider auditing the activities of any user with security administrator, all object, or service authority. You can use the syslog-ng Agent configuration option to review user auditing and make changes. Or, you can turn on user auditing with the Change User Audit (CHGUSRAUD) command. Use this command to enable the highest level of monitoring for any IBM provided profile and any profile that you create with special authorities such as *SECADM, *SERVICE, or *ALLOBJ:

CHGUSRAUD
      USRPRF(QSECOFR)
      AUDLVL(*ALL)

Repeat this process for all privileged users.

To audit access to any file containing sensitive data such as credit card numbers, social security numbers, etc., use the Change Object Audit command (CHGOBJAUD). You may also want to audit access to files that contain configuration information. For IFS files use the Change Audit (CHGAUD) command. For QDLS files use the Change DLO Audit (CHGDLOAUD) command.

CHGOBJAUD
      OBJ(MYLIB/MYFILE)
      OBJAUD(*ALL)
CHGAUD
      OBJ(‘/mydirectory/*’)
      OBJAUD(*ALL)
CHGDLOAUD
      FOLDER(MYFOLDER)
      OBJAUD(*all)

	Note
	You must specify the value *OBJAUD in the QAUDLVL or QAUDLVL system value to perform object auditing.

You may have programs that adopt special authority or which perform sensitive functions. A program that adopts the QSECOFR profile authority, or which maintains salary data are examples of this type of program. You can make a list of programs that adopt authority by using the Display Program Adopt (DSPPGMADP) command. Use the CHGOBJAUD command to start auditing:

CHGOBJAUJD
      OBJ(MYLIB/MYPROGRAM)
      OBJAUD(*ALL)

	Note
	You must specify the value *OBJAUD in the QAUDLVL or QAUDLVL system value to perform object auditing.

Use the Work With System Values command to establish the action to take if the system cannot write to the security audit journal. It is unlikely that this will occur, but you should set a value for this event.

CHGSYSVAL
      SYSVAL(QAUDENDACN)
      VALUE(‘*NOTIFY’)

Use the WRKSYSVAL command to establish how often journal records are written to disk. The smaller the value the less the likely is the loss of journal entries, but the more impact journaling has on system performance. For most users the value of *SYS will provide a balance between performance and log collection reliability. For maximum security set the value to 1.

CHGSYSVAL
      SYSVAL(QAUDFRCLVL)
      VALUE(*SYS)
CHGSYSVAL
      SYSVAL(QAUDFRCLVL)
      VALUE(1)

The system value QAUDCTL controls the actual collection of system events to the QAUDJRN security journal. You must change this system value to indicate that event collection should start. Use the CHGSSVAL command to enable security auditing:

CHGSYSVAL
      SYSVAL(QAUDCTL)
      VALUE(‘*AUDLVL’)

After installing the Alliance LogAgent for System i product and entering a license code, you can configure the agent applications. You can display the main menu by adding the ALLSYL100 library to your library list and displaying the main menu SYMAIN:

Addlible allsyl100
		Go symain

The following menu is displayed:

Figure 4.1. 

Select an option to begin work.

Select the option for Configuration, then select the option to Configure syslog-ng Agent. The following panel is displayed:

Figure 4.2. 

In order to send syslog entries to a syslog server you must configure the Alliance TCP client communications. From the Configuration menu take the option to Work With TCP Clients. The following panel is displayed.

Figure 4.3. 

Three sample configurations are displayed. The first sample SYSLOG uses standard TCP sockets to send syslog records to a Syslog-ng server. The second sample SYSLOGD sends log records to a Syslogd server using standard UDP communications. The third sample uses SSL/TLS TCP sockets to send syslog records to a Syslog-ng server using secure SSL communications.

Use option 2 to change a configuration. Use option 3 to copy the configuration to a new definition. Use option 4 to delete a configuration. Use option 6 to print the configuration details.

When you select option 2 to change the TCP client configuration the following panel is displayed:

Figure 4.4. 

Use this option to define user-created QAUDJRN journal entries. When a user application sends an entry to the security journal QAUDJRN a user-defined journal entry type is used. This is a two-character value and is different than the journal entry types that are created by i5/OS. In order to report these events you need to define them with this option and provide text and severity values. You must be QSECOFR to access this option.

Figure 4.5. 

Use this option to start the Edit File (EDTF) command to edit the syslog-ng.conf configuration file in the IFS directory. You can add new sources, destinations, filters and other configuration definitions to the file. Please consult the syslog-ng Administration guide for information on how to configure syslog-ng options.

Use this option to view user profile security and audit settings, and to make changes to use audit options. This option will build a list of user profiles, resolve the group profile and supplemental group profile authorities, and recommend changes to user auditing settings. All users with a high level of security and object privileges should have user auditing enabled. syslog-ng Agent will recommend that users with security administration and all object authorities have the maximum level of user auditing enabled. Users with programmer or system operator privileges will be recommended for a medium amount of user auditing.

Figure 4.6. 

When you use option 5 to view a user the following panel is displayed.

Figure 4.7. 

This is the second panel when displaying user information.

Figure 4.8. 

Use this option to associate the message severities in the QSYSOPR message queue with syslog severity values. Alliance provides a default set of mappings between the severity of a message in QSYSOPR to the severity of the event reported to the syslog server. You can use this option to change these default mappings.

Figure 4.9. 

Use this panel to add or change a mapping between QSYSOPR message severity to syslog facility and severity.

Figure 4.10. 

Use this option to define the starting point for the QAUDJRN extraction and reporting process. After installation syslog-ng Agent will start with the first entry in the first QAUDJRN journal receiver. You may wish to skip some of the older events. You can do this by specifying a starting journal sequence number and journal receiver.

To determine the name of the QAUDJRN journal receiver, use the Work With Journal Attributes (WRKJRNA) command for the journal QAUDJRN. This command will show you the name of the currently attached journal receiver.

Figure 4.11. 

After configuring the global options and a TCP communications client, you must start the Alliance subsystem ALLSYL100 to start collecting logs. On the configuration menu take the option to Start Syslog Subsystem. The following panel is displayed:

Figure 4.12. 

Press Enter to start the subsystem. Depending on the configuration options you have selected, the following jobs will appear in the subsystem:

You can use options on the Configuration menu to view active jobs in the Alliance ALLSYL100 subsystem, and to end the subsystem. You can also end the subsystem ALLSYL100 manually using the End Subsystem (ENDSBS) command with the *IMMED option.

	Note
	The first time you start the Alliance subsystem the audit journal and operator message queue processes will begin collecting information starting from the earliest message. If there is a substantial amount of history in the journal or message queue it may take time for these messages to be sent to the Syslog-ng server.

QSYS/STRSBS SBSD(ALLSYL100/ALLSYL100)
MONMSG MSGID(CPF0000)

The application maintenance option can be used to purge information from the internal diagnostic logs, historical information, and to re-organize any physical files used by the Alliance application. From the main menu take the option for Application Maintenance, then option 1 to run maintenance. The following panel is displayed:

Figure 4.13. 

The View Application Logs option can be used to view internal Alliance application logs such as the TCP or SSL/TSL TCP application log. From the main menu take the option for Inquiry, then option 1 to view the log. The following panel is displayed:

Figure 4.14. 

Use option 5 to view the log or option 6 to print the log. For large logs it is recommended that you use option 6 to print the log and then use the Work With Spooled Files (WRKSPLF) command to view the report.

	Note
	The application logs only contain internal Alliance diagnostic information and do not contain Syslog information that has been collected.

To enable logging in the Apache server, use the Work With Links (WRKLNK) command to edit the httpd_conf file in the /www/(server-name)/conf directory. You will need to add a “LogCycle” directive in order to force the Apache server to create one file. Without this directive the log files will have an appended time stamp and the syslog-ng application will not be able to process them. After updating the configuration file stop and re-start the Apache web server instance with the Start TCP Server (STRTCPSVR) command.

LogCycle Off
CustomLog logs/access_log combined

You must now configure a source to read the file in syslog-ng. Apache logs will generally be placed in the /www/(server-name)/logs directory. Consult the syslog-ng documentation for instructions on how to configure syslog-ng to read the log file.

To enable logging in the OpenSSH server, use the Work With Links (WRKLNK) command to edit the sshd_conf file in the /QopenSys/QIBM/ /ProdData/SC1/OpenSSH/openssh-3.5p1/etc directory like this:

SyslogFacility AUTH
LogLevel INFO

Consult the documentation on the OpenSSH web site (http://www.openssh.org) for other syslog options.

After changing the sshd_config file you must create an empty log file. Sign on as QSECOFR, use the STRQSH shell, and use the following commands:

mkdir /var/adm
touch /var/adm/sshlog

You must now configure a source to read the file in syslog-ng. Consult the syslog-ng documentation for instructions on how to configure syslog-ng to read the log file.

A number of other open systems and proprietary applications can be deployed on the syslog-ng Agent including MySQL, PHP, Perl, and others. Most of these types of applications can be enabled to collect system logs. Please consult the documentation for these servers on the steps to take to start collecting logs. Once logging is active you can configure a “source” statement in Syslog-ng to capture the logs.

When syslog-ng Agent encounters a problem processing a Syslog transaction it may send a message to the system operator message queue. Use the DSPMSG command to view these messages. Many of the messages have second level text. You can use F1 or the HELP key to view this text.

The Alliance TCP client applications will create extra diagnostic information when the option for application logging is enabled. You should restart the subsystem when changing the logging option. When application logging is enabled there will be additional information written to the job log and to output spooled files in the job.