Certalized log storage and processing based on syslog-ng

The main problem with standard syslog is that it is too weak for modern operating system. There are multiple attempts to "enhance" classic syslog functionality and one of the most successful is syslog-ng. Unfortunately it did not became standard de-facto as it was not adopted by the major Linus distribution -- Red Hat. The latter uses rsyslog.

On Suse 10 and 11 syslog-ng is preinstalled (used instead of standard syslog daemon; although rsyslog can be installed instead and is supported). For Solaris 10 and Solaris 11 syslog-ng packages used to be available from Sun Freeware.

Syslog-ng makes the most sense and is often used for running a centralized log server (LOGHOST). The main problem with centralized syslog  server is that messages need to be pre-filtered to be useful. Sending all syslog messages to central host usually create a huge log files that are more difficult to analyze that separate syslog files on individual hosts.

For individual hosts syslog-ng looks like an overkill, program that is beyond capacity to learn by most sysadmins,  and my impression is that rsyslog is a slightly better deal. Few sysadmin who are not responsible for central loghost server ever managed to learn syslog-ng  configuration files well enough to find its features useful. In most case standard syslog-ng configuration file supplied with the distribution is used, which is the most stupid way to use any syslog daemon.  This is an interesting paradox that I can't explain, but that is what I observed in large corporate environment.

At the same time centralized storage of syslog files makes detection of events that affect multiple servers a lot easier and here capabilities of syslog-ng are really critical for success. As it is often maintained by dedicated "syslog adminisrator" the level of complexity does not matter as much as with regular syslog daemon on servers.

Generally there need to be a compromise between what is stored in centralized database and what is stored on individual servers. A good starting point is FATAL and some other important classes of syslog messages.

On Solaris after starting syslog-ng does not always wants to log messages whatsoever :-).  Pid file is created and syslog-ng daemon is running but it does not accept messages. 

To troubleshoot you need first to check the correctness of the configuration, for example:

    /usr/local/sbin/syslog-ng -v -s -f /etc/syslog-ng/syslog-ng.conf

That helps to detect errors in configuration files.

 If there is no errors you can try to run syslog with debug option (-d). Use

options { long_hostnames(off); sync(0); };

in your syslog-ng.conf.

NEWS CONTENTS

• 20100616 : Centralized Syslog Server Using syslog-NG with web Interface using php-syslog-ng ( Centralized Syslog Server Using syslog-NG with web Interface using php-syslog-ng, Jun 16, 2010 )
• 20100616 : Cool Solutions Centralized Syslogging with Syslog-NG on SUSE Linux ( Cool Solutions Centralized Syslogging with Syslog-NG on SUSE Linux, Jun 16, 2010 )
• 20100224 : Log message classification with syslog-ng by Robert Fekete ( January 13, 2010 , LWN.net )
• 20061215 : Interview syslog-ng 2.0 developer Balázs Scheidler ( Dec 13, 2006 , Linux.com )
• 20061215 : Interview syslog-ng 2.0 developer Balázs Scheidler ( Interview syslog-ng 2.0 developer Balázs Scheidler, Dec 15, 2006 )
• 20060411 : Build a Secure Logging Server with syslog-ng by Carla Schroder ( Build a Secure Logging Server with syslog-ng, Apr 11, 2006 )
• 20060411 : Build a Secure Logging Server with syslog-ng (Part 2) by Carla Schroder ( Build a Secure Logging Server with syslog-ng (Part 2), Apr 11, 2006 )
• 20060411 : [syslog-ng]Configure syslog-ng to act as a central logging server ( [syslog-ng]Configure syslog-ng to act as a central logging server, )
• 20060411 : [syslog-ng]Configure syslog-ng to act as a central logging server ( [syslog-ng]Configure syslog-ng to act as a central logging server, )
• 20050621 : Windows event logs to syslog ( Jun 21, 2005 )
• 20050621 : Syslog-ng 1.6.7 package for Solaris 10, Sparc platform ( Syslog-ng 1.6.7 package for Solaris 10, Sparc platform, )
• 20050620 : Remote Logging with SSH and Syslog-NG by Hal Pomeranz ( Remote Logging with SSH and Syslog-NG, Jun 20, 2005 )

Old news ;-)

[Jun 16, 2010] Centralized Syslog Server Using syslog-NG with web Interface using php-syslog-ng

Syslog-NG My-SQL Integartion with PHP-syslog-NG Web interface

Make sure the you have installed mysql server and phpmyadmin in your machine. If you want to install run the following command

#apt-get install mysql-server

#apt-get install phpmyadmin

You need to install apache Webserver click here for apache with php support installation steps.

What is PHP-syslog-NG ?

php-syslog-ng is a log monitor designed to let the user quickly and easily manage logs from many hosts.

PHP-Syslog-ng is a front-end for viewing syslog-ng messages logged to MySQL in real-time. It features customized searches based on device, time, priori ty, message, and date.

HOW IT WORKS

Syslog-ng by default collects all the messages from the various sources and pipes them to the MySQL Database. PHP-Syslog-ng which is the front end interface t hen looks and reads from the MySQL Database when searched from the web interface.

Download PHP-syslog-NG

http://www.phpwizardry.com/php-syslog-ng/phpsyslogng-2.8.tar.gz

Installing PHP-syslog-NG with Mysql support for syslog-ng

Download the PHP-syslog-NG in your web server root directory and extract this file phpsyslogng-2.8.tar.gz.Now

#cd phpsyslogng-2.8

Now you are in phpsyslogng-2.8 directory

Configure MySQL

The quickest way to do this is to use the dbsetup.sql file in the scripts directory. Just edit the file and set some passwords for the three users that are created (replace PW_HERE). The script will create a table for logs and a table for user authentication and give the three users some sensible priviliges. If you make other changes like changing the name of the database or the name of the tables then make sure you edit config.php to reflect that. After editing the dbsetup.sql file then just run it like this:

shell> mysql -uroot -p < dbsetup.sql

Configure syslog-ng

Now you need to configure syslog-ng to send the desired log messages to a pipe that can be read to send the entries to MySQL. You will need to add two entries to the syslog-ng configuration file. The configuration file is usually in /etc/syslog-ng/syslog-ng.conf.

You first need to add a new 'destination' entry. Add something like this:

destination d_mysql {
pipe("/var/log/mysql.pipe"
template("INSERT INTO logs
(host, facility, priority, level, tag, datetime, program, msg)
VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC',
'$PROGRAM', '$MSG' );\n") template-escape(yes));
};

That will take your log entries and format them into a SQL query that can be run to add it to the database.

You also need to add an entry that determines what log entries to forward to the FIFO pipe. You will usually want to forward everything to MySQL and there should already be a 'source' entry for that in your syslog-ng.conf file (usually called src or s_all). To tie that source to the destination you just created you will add something like this:

log {
source(s_all);
destination(d_mysql);
};

Setup syslog-ng to MySQL pipe

An example for a script that feeds log entries from the FIFO pipe to MySQL is included in the scripts directory. The script is called syslog2mysql.sh.
#!/bin/bash

if [ ! -e /var/log/mysql.pipe ]
then
mkfifo /var/log/mysql.pipe
fi
while [ -e /var/log/mysql.pipe ]
do
mysql -u syslogfeeder --password=PASS_HERE syslog < /var/log/mysql.pipe >/dev/null
done

If you decide to use this script then you have to replace PASS_HERE with the password for the syslogfeeder user. You will also probably want to have this script started automatically whenever you start the server. So add an entry in the inittab or start it through init.d (or whatever is appropriate on your system). But make sure you call it after MySQL has been started.

Now start the syslog2mysql.sh script:

shell> ./syslog2mysql.sh &

or if you created an init.d script:

shell> /etc/init.d/syslog2mysql start

Note:- This syslog2mysql.sh need to run always in background because using this script all the messges are pipe to mysql database

It's finally time to restart the syslog-ng daemon and start sending your logs to the database:

shell> /etc/init.d/syslog-ng restart

Edit config.php

If you are using the default database setup from the dbsetup.sql file then all you need to do is to enter the passwords for the sysloguser and syslogadmin users, set the right host and port for the database server if it is not on the same server as the web server and set the correct URL. Otherwise read through the config.php file and configure things to suit your needs. All the different options are explained in the file.

[Jun 16, 2010] Cool Solutions Centralized Syslogging with Syslog-NG on SUSE Linux

Now that the loghost is ready to receive log messages from hosts on the network, you have to configure your hosts to send messages to it. If the hosts are based on SLES 9 or OES Linux, you have to follow the steps outlined above to enable Syslog-NG by editing /etc/sysconfig/syslog and then running SuSEconfig. Don't forget to stop syslogd first by running /etc/init.d/syslogd stop.

Then, you have to edit /etc/syslog-ng/syslog-ng.conf.in to tell the host to send messages to the syslog server. Open the /etc/syslog-ng/syslog-ng.conf.in file and go to the bottom of the file. Assuming your loghost is located at the ip address 192.168.1.42, add the following lines:

# send everything to log host
destination loghost {
    tcp("192.168.1.42" port(5140));
};
log {
    source(src);
    destination(loghost);
};

After making the changes, save the file and run SuSEconfig --module syslog-ng, and then restart syslogd by running /etc/init.d/syslogd start. Now your host should be configured to send all log messages to your log host.

You can test to see that it works by using the command-line utility logger, which sends messages to the syslogd service. Logger is also a great way to have your shell scripts or cron scripts send status messages to the system log. To test to see if your server is configured to send log messages to your loghost, run the following command.

logger user.warn this is a test logger message that should go to my loghost

Navigate the directories on your loghost to the /var/log/HOSTS/YEAR-MONTH/ directory. You should see a directory for your host. It will either be a hostname or an IP address, depending upon whether reverse-lookup works in your DNS environment. Inside that directory, you should find a log file starting with user that contains your user.warn message

[Feb 24, 2010] Log message classification with syslog-ng by Robert Fekete

In syslog-ng 3.0 a new message-parsing and classifying feature (dubbed pattern database or patterndb) was introduced. With recent improvements in 3.1 and the increasing demand for processing and analyzing log messages, a look at the syslog-ng capabilities is warranted.

January 13, 2010 | LWN.net

The nine-year-old syslog-ng project is a popular, alternative syslog daemon - licensed under GPLv2 - that has established its name with reliable message transfer and flexible message filtering and sorting capabilities. In that time it has gained many new features including the direct logging to SQL databases, TLS-encrypted message transport, and the ability to parse and modify the content of log messages. The SUSE and openSUSE distributions use syslog-ng as their default syslog daemon.

In syslog-ng 3.0 a new message-parsing and classifying feature (dubbed pattern database or patterndb) was introduced. With recent improvements in 3.1 and the increasing demand for processing and analyzing log messages, a look at the syslog-ng capabilities is warranted.

The main task of a central syslog-ng log server is to collect the messages sent by the clients and route the messages to their appropriate destinations depending on the information received in the header of the syslog message or within the log message itself. Using various filters, it is possible to build even complex, tree-like log routes. For example:

It is equally simple to modify the messages by using rewrite rules instead of filters if needed. Rewrite rules can do simple search-and-replace, but can also set a field of the message to a specific value: this comes handy when client does not properly format its log messages to comply with the syslog RFCs. (This is surprisingly common with routers and switches.) Version 3.1 of makes it possible to rewrite the structured data elements in messages that use the latest syslog message format (RFC5424).

Artificial ignorance

Classifying and identifying log messages has many uses. It can be useful for reporting and compliance, but can be also important from the security and system maintenance point of view. The syslog-ng pattern database is also advantageous if you are using the "artificial ignorance" log processing method, which was described by Marcus J. Ranum (MJR):

Artificial Ignorance - a process whereby you throw away the log entries you know aren't interesting. If there's anything left after you've thrown away the stuff you know isn't interesting, then the leftovers must be interesting.

Artificial ignorance is a method to detect the anomalies in a working system. In log analysis, this means recognizing and ignoring the regular, common log messages that result from the normal operation of the system, and therefore are not too interesting. However, new messages that have not appeared in the logs before can signify important events, and should therefore be investigated.

The syslog-ng pattern database

The syslog-ng application can compare the contents of the received log messages to a set of predefined message patterns. That way, syslog-ng is able to identify the exact log message and assign a class to the message that describes the event that has triggered the log message. By default, syslog-ng uses the unknown, system, security, and violation classes, but this can be customized, and further tags can be also assigned to the identified messages.

The traditional approach to identify log messages is to use regular expressions (as the logcheck project does for example). The syslog-ng pattern database uses radix trees for this task, and that has the following important advantages:

• Classifying messages is fast, much faster than with methods based on regular expressions. The speed of processing a message is practically independent from the total number of patterns. What matters is the length of the message and the number of "similar" messages, as this affects the number of junctions in the radix tree.
• Regular-expression based methods become increasingly slower as the number of patterns increases. Radix trees scale very well, because only a relatively small number of simple comparisons must be performed to parse the messages.
• The syslog-ng message patterns are easy to write, understand, and maintain.

For example, compare the following:

A log message from an OpenSSH server:

    Accepted password for joe from 10.50.0.247 port 42156 ssh2

A regular expression that describes this log message and its variants:

    Accepted \ 
        (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam) \
        for [^[:space:]]+ from [^[:space:]]+ port [0-9]+( (ssh|ssh2))? 

An equivalent pattern for the syslog-ng pattern database:

    Accepted @QSTRING:auth_method: @ for @QSTRING:username: @ from \ 
        @QSTRING:client_addr: @ port @NUMBER:port:@ @QSTRING:protocol_version: @

Obviously, log messages describing the same event can be different: they can contain data that varies from message to message, like usernames, IP addresses, timestamps, and so on. This is what makes parsing log messages with regular expressions so difficult. In syslog-ng, these parts of the messages can be covered with special fields called parsers, which are the constructs between '@' in the example. Such parsers process a specific type of data like a string (@STRING@), a number (@NUMBER@ or @FLOAT@), or IP address (@IPV4@, @IPV6@, or @IPVANY@). Also, parsers can be given a name and referenced in filters or as a macro in the names of log files or database tables.

It is also possible to parse the message until a specific ending character or string using the @ESTRING@ parser, or the text between two custom characters with the @QSTRING@ parser.

A syslog-ng pattern database is an XML file that stores patterns and various metadata about the patterns. The message patterns are sample messages that are used to identify the incoming messages; while metadata can include descriptions, custom tags, a message class - which is just a special type of tag - and name-value pairs (which are yet another type of tags).

The syslog-ng application has built-in macros for using the results of the classification: the .classifier.class macro contains the class assigned to the message (e.g., violation, security, or unknown) and the .classifier.rule_id macro contains the identifier of the message pattern that matched the message. It is also possible to filter on the tags assigned to a message. As with syslog, these routing rules are specified in the syslog-ng.conf file.

Using syslog-ng

In order to use these features, get syslog-ng 3.1 - older versions use an earlier and less complete database format. As most distributions still package version 2.x, you will probably have to download it from the syslog-ng download page.

The syntax of the pattern database file might seem a bit intimidating at first, but most of the elements are optional. Check The syslog-ng 3.1 Administrator Guide [PDF] and the sample database files to start with, and write to the mailing list if you run into problems.

A small utility called pdbtool is available in syslog-ng 3.1 to help the testing and management of pattern databases. It allows you to quickly check if a particular log message is recognized by the database, and also to merge the XML files into a single XML for syslog-ng. See pdbtool --help for details.

Closing remarks

The syslog-ng pattern database provides a powerful framework for classifying messages, but it is powerless without the message patterns that make it work. IT systems consist of several components running many applications, which means a lot of message patterns to create. This clearly calls for community effort to create a critical mass of patterns where all this becomes usable.

To start with, BalaBit - the developer of syslog-ng - has made a number of experimental pattern databases available. Currently, these files contain over 8000 patterns for over 200 applications and devices, including Apache, Postfix, Snort, and various common firewall appliances. The syslog-ng pattern databases are freely available for use under the terms of the Creative Commons Attribution-Noncommercial-Share Alike 3.0 (CC by-NC-SA) license.

A community site for sharing pattern databases is reportedly also under construction, but until this becomes a reality, pattern database related discussions and inquiries should go to the general syslog-ng mailing list.

[Dec 15, 2006] Interview syslog-ng 2.0 developer Balázs Scheidler

Dec 13, 2006 | Linux.com

syslog-ng is an alternative system logging tool, a replacement for the standard Unix syslogd system-event logging application. Featuring reliable logging to remote servers via the TCP network protocol, availability on many platforms and architectures, and high-level message filtering capabilities, syslog-ng is part of several Linux distributions. We discussed the highlights of last month's version 2.0 release with the developer, Balázs Scheidler.

NewsForge: How and why did you start the project?

Balázs Scheidler: Back in 1998 the main Hungarian telecommunication company was looking for someone on a local Linux mailing list to port nsyslog to Linux. nsyslog -- developed by Darren Reed -- was at that time incomplete, somewhat buggy, and available only for BSD. While at university, I had been working for an ISP and got often annoyed with syslogd: it creates too many files, it is difficult to find and move the important information, and so on. Developing a better syslog application was a fitting task for me.

NF: Why is it called syslog-ng?

BS: syslog-ng 1.0 was largely based on nsyslog, but nsyslog did not have a real license. I wanted to release the port under GPL, but Darren permitted this only if I renamed the application.

NF: What kind of support is available for the users?

BS: There is a community FAQ and an active mailing list. If you are stuck with the compiling or the configuration, the mailing list is the best place to find help. My company, BalaBit IT Security, offers commercial support for those who need quick support.

NF: Documentation?

BS: The reference guide is mostly up-to-date, but I hope to improve it someday. I am sure there are several howtos floating around on the Internet.

NF: Who uses syslog-ng?

BS: Everyone who takes logging a bit more seriously. I know about people who use it on single workstations, and about companies that manage the centralized logging of several thousand devices with syslog-ng. We have support contracts even with Fortune 500 companies.

NF: What's new in version 2.0?

BS: 1.6 did not have any big problems, only smaller nuances. 2.0 was rewritten from scratch to create a better base for future development and to address small issues. For example, the data structures were optimized, greatly reducing the CPU usage. I have received feedback from a large log center that the new version uses 50% less CPU under the same load.

Every log message may include a timezone. syslog-ng can convert between different timestamps if needed.

It can read and forward logfiles. If an application logs into a file, syslog-ng can read this file and transfer the messages to a remote logcenter.

2.0 supports the IPv6 network protocol, and can also send and receive messages to multicast IP addresses.

It is also possible to include hostnames in the logs without having to use a domain name server. Using a DNS would seriously limit the processing speed in high-traffic environments and requires a network connection. Now you can create a file similar to /etc/hosts that syslog-ng uses to resolve the frequently used IP addresses to hostnames. That makes the logs much easier to read.

syslog-ng 2.0 uses active flow control to prevent message losses. This means that if the output side of syslog-ng is accepting messages slowly, then syslog-ng will wait a bit more between reading messages from the input side. That way the receiver is not flooded with messages it could not process on time, and no messages are lost.

NF: Is syslog-ng available only for Linux, or are other platforms also supported?

BS: It can be compiled for any type of Unix -- it runs on BSD, Solaris, HP-UX, AIX, and probably some others as well. Most bigger Linux distributions have syslog-ng packages: Debian, SUSE, Gentoo.... I think Gentoo installs it by default, replacing syslogd entirely.

NF: What other projects do you work on?

BS: syslog-ng is a hobby for me; that is why it took almost five years to finish version 2.0. My main project is Zorp, an application-level proxy firewall developed by my company. Recently I have been working on an appliance that can transparently proxy and audit the Secure Shell (SSH) protocol.

During development I stumble into many bugs and difficulties, so I have submitted patches to many places, such as glib and the tproxy kernel module.

NF: Are these projects also open source?

BS: No, these are commercial products, but the Zorp firewall does have a GPL version.

NF: Any plans for future syslog-ng features?

BS: I plan to support the syslog protocol that is being developed by IETF.

I would like to add disk-based buffering, so you could configure syslog-ng to log into a file if the network connection goes down, and transmit the messages from the file when the network becomes available again.

It would be also good to transfer the messages securely via TLS, and to have application-layer acknowledgments on the protocol level.

[Dec 15, 2006] Interview syslog-ng 2.0 developer Balázs Scheidler by: Robert Fekete

December 13, 2006 (Linux.com ) syslog-ng is an alternative system logging tool, a replacement for the standard Unix syslogd system-event logging application. Featuring reliable logging to remote servers via the TCP network protocol, availability on many platforms and architectures, and high-level message filtering capabilities, syslog-ng is part of several Linux distributions. We discussed the highlights of last month's version 2.0 release with the developer, Balázs Scheidler.

NewsForge: How and why did you start the project?

Balázs Scheidler: Back in 1998 the main Hungarian telecommunication company was looking for someone on a local Linux mailing list to port nsyslog to Linux. nsyslog -- developed by Darren Reed -- was at that time incomplete, somewhat buggy, and available only for BSD. While at university, I had been working for an ISP and got often annoyed with syslogd: it creates too many files, it is difficult to find and move the important information, and so on. Developing a better syslog application was a fitting task for me.

NF: Why is it called syslog-ng?

BS: syslog-ng 1.0 was largely based on nsyslog, but nsyslog did not have a real license. I wanted to release the port under GPL, but Darren permitted this only if I renamed the application.

NF: What kind of support is available for the users?

BS: There is a community FAQ and an active mailing list. If you are stuck with the compiling or the configuration, the mailing list is the best place to find help. My company, BalaBit IT Security, offers commercial support for those who need quick support.

NF: Documentation?

BS: The reference guide is mostly up-to-date, but I hope to improve it someday. I am sure there are several howtos floating around on the Internet.

NF: Who uses syslog-ng?

BS: Everyone who takes logging a bit more seriously. I know about people who use it on single workstations, and about companies that manage the centralized logging of several thousand devices with syslog-ng. We have support contracts even with Fortune 500 companies.

NF: What's new in version 2.0?

BS: 1.6 did not have any big problems, only smaller nuances. 2.0 was rewritten from scratch to create a better base for future development and to address small issues. For example, the data structures were optimized, greatly reducing the CPU usage. I have received feedback from a large log center that the new version uses 50% less CPU under the same load.

Every log message may include a timezone. syslog-ng can convert between different timestamps if needed.

It can read and forward logfiles. If an application logs into a file, syslog-ng can read this file and transfer the messages to a remote logcenter.

2.0 supports the IPv6 network protocol, and can also send and receive messages to multicast IP addresses.

It is also possible to include hostnames in the logs without having to use a domain name server. Using a DNS would seriously limit the processing speed in high-traffic environments and requires a network connection. Now you can create a file similar to /etc/hosts that syslog-ng uses to resolve the frequently used IP addresses to hostnames. That makes the logs much easier to read.

syslog-ng 2.0 uses active flow control to prevent message losses. This means that if the output side of syslog-ng is accepting messages slowly, then syslog-ng will wait a bit more between reading messages from the input side. That way the receiver is not flooded with messages it could not process on time, and no messages are lost.

NF: Is syslog-ng available only for Linux, or are other platforms also supported?

BS: It can be compiled for any type of Unix -- it runs on BSD, Solaris, HP-UX, AIX, and probably some others as well. Most bigger Linux distributions have syslog-ng packages: Debian, SUSE, Gentoo.... I think Gentoo installs it by default, replacing syslogd entirely.

NF: What other projects do you work on?

BS: syslog-ng is a hobby for me; that is why it took almost five years to finish version 2.0. My main project is Zorp, an application-level proxy firewall developed by my company. Recently I have been working on an appliance that can transparently proxy and audit the Secure Shell (SSH) protocol.

During development I stumble into many bugs and difficulties, so I have submitted patches to many places, such as glib and the tproxy kernel module.

NF: Are these projects also open source?

BS: No, these are commercial products, but the Zorp firewall does have a GPL version.

NF: Any plans for future syslog-ng features?

BS: I plan to support the syslog protocol that is being developed by IETF.

I would like to add disk-based buffering, so you could configure syslog-ng to log into a file if the network connection goes down, and transmit the messages from the file when the network becomes available again.

It would be also good to transfer the messages securely via TLS, and to have application-layer acknowledgments on the protocol level.

[Apr 11, 2006] Build a Secure Logging Server with syslog-ng Part 1 of 2 by Carla Schroder

Configuration File Structure
With all of this flexibility comes a bit of a learning curve. Syslog-ng's configuration file is /etc/syslog-ng/syslog-ng.conf, or on some systems /etc/syslog-ng.conf. You'll need man 5 syslog-ng.conf to understand all the different options and parameters. syslog-ng.conf has five possible sections:

options{}
Global options. These can be overridden in any of the next four sections

source{}
Message sources, such as files, local sockets, or remote hosts

destination{}
Message destinations, such as files, local sockets, or remote hosts

filter{}
Filters are powerful and flexible; you can filter on any aspect of a log message, such as standard syslogd facility names (man 5 syslog.conf), log level, hostname, and arbitrary contents like words or number strings

log{}
Log statements connect the source, destination, and filter statements, and tell syslog-ng what to do with them

Here are some typical global options:

options {

sync (0);
log_fifo_size (2048);
create_dirs (yes);
group (logs);
dir_group (logs);
perm (0640);
dir_perm (0750);
};

Options statements must use options as defined in /etc/syslog-ng.conf. This what the examples mean:

sync
How many lines of messages to keep in the output queue before writing to disk. (Logging output is called "messages.") Zero is the preferred option, since you want to be sure to capture everything, and not risk losing data in the event of a power failure

log_fifo_size
The maximum number of lines of messages in the output queue. The default is 100 lines of messages. You can do some calculations to figure out a suitable value, as this syslog-ng mail list post shows. To quote the relevant bit:

Each message source receives maximum 1024 bytes of data. A single log message is about 20 to 30 bytes. So on a single run each log source may emit 1024/20 = 50 messages. Your log fifo must be able to hold this number of messages for each source. So for two sources, you'll need at least 100 slots in your fifo.

It is easy to get confused by these two options. Incoming messages from remote hosts arrive in bursts, so what you need to do is make sure your log_fifo_size is large enough to handle these bursts. Otherwise you risk losing messages. Ultimately you will be limited by i/o and network speeds

create_dirs
Enable or disable automatic directory creation for destination files. In this article this value is "yes", so that remote host directories can be created on demand

group
dir_group
Set the group owner of logfiles and directories, so you don't have to read and analyze logs as the superuser

perm
dir_perm
Default permissions on new files and directories

Source, Destination, and Filter Statements
Source, destination, and filter names are arbitrary, like these examples show:

source s_local { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); };
destination d_auth { file("/var/log/auth.log"); };
filter f_auth { facility(auth, authpriv); };

So "source s_local" could be "source frederick_remington_depew" if you so desired, and "destination d_auth" could be "destination moon." The convention is to prefix source names with "s", destination names with "d", and filter names with "f", but this is not required. All other elements of source, destination, and filter statements must use parameters as defined in man 5 syslog-ng.conf.

The "source s_local" example collects all locally-generated log messages into a single source statement. The "destination d_auth" statement directs authorization messages to /var/log/auth.log, which are defined by the "filter f_auth" statement. auth and authpriv are standard syslog facility names.

Log statements pull it all together:

log {source(s_local); filter(f_auth); destination(d_auth); };

So these four lines filter out all the auth and authpriv messages from all local messages, and write them to /var/log/auth.log.

Enabling Remote Logging
While it's possible to send log messages from remote clients with good old syslogd, it's really not adequate because it only transmits UDP packets. So you need syslog-ng installed on all client hosts as well. Adding these lines to syslog-ng.conf on the server accepts remote messages from clients and dumps them into a single file per host:

source s_remote { tcp(); };
destination d_clients { file("/var/log/HOSTS/$HOST/"); };
log { source(s_remote); destination(d_clients); };

This is a very simple, but functional example for your client hosts that collects all local messages and sends them to the remote server:

#sample syslog-ng.conf for a remote client
source s_local { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); };
destination d_loghost {tcp("192.168.1.10" port(514));};
log { source(s_local); destination(loghost); };

Sample syslog-ng.conf
A complete sample configuration file is a bit long to include here, so you should take a look at the one that came with your syslog-ng installation. Debian users get a customized file that replicates a syslogd setup. Let's put together our examples here in a single file for the server, set up a remote client, and run some tests to see how it works:

#sample syslog-ng.conf for a central logging server
options {

sync (0);
log_fifo_size (2048);
create_dirs (yes);
group (logs);
dir_group (logs);
perm (0640);
dir_perm (0750);
};

source s_local { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); };
destination d_auth { file("/var/log/auth.log"); };
filter f_auth { facility(auth, authpriv); };

source s_remote { tcp(); };
destination d_clients { file("/var/log/HOSTS/$HOST"); };

log { source(s_remote); destination(d_clients); };
log { source(s_local); filter(f_auth); destination(d_auth); };

Whenever you make changes to syslog-ng.conf you must restart it:

# /etc/init.d/syslog-ng restart

Testing Everything
Now you can runs some simple tests on both the server and the client. Since the only local server messages that are going to be logged are authorization messages, you can test these by opening a new login session on the server, or running su or sudo. Then check /var/log/auth.log. Test the client by doing anything, then see if a new directory was created for the remote client in /var/log/HOSTS.

Another way is to use the useful and excellent logger command:

# logger "this is a test"
# logger -p auth.debug "this is a test"

This will create a line like this in your logfiles:

Apr 1 16:08:42 localhost.localdomain logger: this is a test

Now that we have a grasp of syslog-ng basics, come back next week to learn how to fine-tune and organize syslog-ng just the way you like, for both local and remote logging, and how to securely encrypt all syslog-ng network transmissions.

[Apr 11, 2006] Build a Secure Logging Server with syslog-ng (Part 2) by Carla Schroder

Securing Your Log Server
For your log server to be trustworthy, it should have some good access controls in place. These example iptables rules, run on the logging server itself, illustrate how to allow in only log messages and SSH traffic from the LAN:

iptables -A INPUT -s 192.168.1.0/24 -p tcp --dport 514 --sport 1024:65535 -j ACCEPT
iptables -A INPUT -s 192.168.1.0/24 -p tcp --dport 22 --sport 1024:65535 -m state --state NEW -j ACCEPT

Presumably your default policies are DENY, so you need rules only to allow authorized bits.

You definitely want /var on its own partition, or even a separate drive. If something goes haywire and your log files are flooded with data, you don't want the entire system to go down.

And of course you know to pay attention to hardening the entire system and application-level security.

Organizing Client Messages
Your starting point is the default syslog-ng.conf created by the installer. Typically, though it depends on the distribution, this replicates a standard syslogd configuration. This saves you a bit of labor setting up all the standard facility logs, and gives you a nice model to study.

Syslog.ng lets you slice and dice your log files finely. Don't make yourself crazy and create all manner of separate files for every last little individual criterion. I suggest starting with broader configurations, then directing certain types of messages to their own individual files as you need them. Start with directing unfiltered client messages to the server and let the server sort them out. Then fine-tune the client configurations as you need to. Just keep in mind it's easier to test and debug a single server configuration file than bales of of client files.

The example server configuration from Part 1 can be refined in a number of useful ways. This line creates a new client logfile every day. The client's directory is named for the remote host, then each logfile uses the date for its name:

destination d_clients { file("/var/log/HOSTS/$HOST/$DATE"); };

Other options are MONTH, WEEKDAY, HOUR, FACILITY, PRIORITY, and several others that you can find in the Syslog-ng Reference Manual. Dumping everything into a single file is rather messy, so you may wish to duplicate a stock configuration by creating files by facility, like this:

destination d_clients { file("/var/log/HOSTS/$HOST/$DATE_$FACILITY"); };

This example gives you another way to sort by date, which is helpful if you need to archive log files for a long time:

destination d_clients { file("/var/log/HOSTS/$HOST/$YEAR/$MONTH/$DATE_$FACILITY"); };

Be sure to have the create_dirs (yes); option set as shown in Part 1.

This introduces a new issue: the default logrotate configuration probably doesn't have a clue about your new log configurations. If you want your logs to be rotated instead of archived, you'll have to tweak logrotate.

Logging Both Remotely and Locally
There's no downside to logging remote clients both locally and remotely. Adding these lines to a stock syslog-ng.conf adds remote logging to the existing standard local logging configuration:

source s_local { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); };
destination d_loghost {tcp("192.168.1.10" port(514));};
log { source(s_local); destination(d_loghost); };

Of course you must use your own loghost IP and port number. 514 is the standard syslog-ng port, but you may use any unused port that you desire. Check /etc/services to find unused ports, then be sure to edit /etc/services to include the port you are using.

Encryption With Stunnel
For additional protection you can use stunnel with syslog-ng. Syslog-ng has no native encryption, so stunnel fills this role quite ably. Stunnel must be installed on the server and on all clients. You'll also need OpenSSL, since Stunnel requires it. You'll need to create and distribute SSL certificates, which is the usual many-step dance: create your certificate authority, create .pem files for server and clients, sign the certificates, edit the derned things to remove the private keys, then distribute them.

If you do not already have your own SSL certificate authority, you must now create one. (The following instructions work on most Linux distributions, except Fedora/Red Hat etc., which as usual do things their own weird way. I'm not sure what to do with these, so I'm afraid you're on your own.) Find openssl.conf, and find and edit these lines:

[ CA_default ]
dir   =   ./masterCA    # Where everything is kept
countryName_default             = US
stateOrProvinceName_default     = Oregon
0.organizationName_default      = Alrac Logging Co.
stateOrProvinceName		= Oregon
stateOrProvinceName_default	= OR
localityName			= Portland
organizationalUnitName_default	= CutEmAllDown
commonName			= Alrac AceAdmin
emailAddress			= [email protected]

Now find the CA.sh script, and edit this line to tell CA.sh where to put your new certificates, giving it any name you like:

CATOP=./masterCA

Now change to the directory you want to store your CA in, typically /etc/ssl/, and run the CA.sh script:

# CA.sh -newca

This creates the /etc/ssl/masterCA/ directory, and populates it with all manner of files and directories, including your new server CA, private/cakey.pem. You'll be asked to create a passphrase, which you must write down and keep in a safe place.

Now create the certificate requests for the server, and individual ones for each client:

# openssl req -new -nodes -out logserver_req.pem -keyout logserver.pem
# openssl req -new -nodes -out client1_req.pem -keyout client1.pem
# openssl req -new -nodes -out client2_req.pem -keyout client2.pem

Now each one must be signed by your certificate authority:

# openssl ca -out logserver_cert.pem -infiles logserver_req.pem
# openssl ca -out client1_cert.pem -infiles client1_req.pem
# openssl ca -out client2_cert.pem -infiles client2_req.pem

There are a couple more steps before your shiny new certificates are usable. First, place logserver_cert.pem in /etc/stunnel on the server, and put every client key into each client's /etc/stunnel directory.

Each certificate contains a private key and the certificate itself. The server and client certificate portions need to be exchanged, so that the server has a copy of every client certificate, and every client has a copy of the server certificate. So first copy the certificate portion from every client certificate, and place them all into a single file for the server and store it in /etc/stunnel. The certificate is the bit between

-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----

You can call this file anything you want; I use client_cert.pem. Then you need to delete the private key from a copy of logserver_cert.pem, and place copies on every client in /etc/stunnel.

Configure Stunnel
Finally we get to configure Stunnel. You'll need to edit /etc/stunnel/stunnel.conf. If your installer didn't create one, look for an example file to use. Add these options, using your own server IP, port numbers, and certificate names:

#stunnel.cnf for the logserver
cert = /etc/stunnel/logserver_cert.pem
CAfile = /etc/stunnel/client_cert.pem
verify = 3
[5140]
accept = 192.168.1.10:5140
connect = 127.0.0.1:514

#stunnel.cnf for all clients
client = yes
cert = /etc/stunnel/client_cert.pem
CAfile = /etc/stunnel/logserver_cert.pem
verify = 3
[5140]
accept = 127.0.0.1:514
connect = 192.168.1.10:5140

Syslog-ng.conf
Finally, your syslog-ng.conf file will need a few additions on both clients and server. (These lines reference the examples in both part 1 and part 2.) On the clients, add these lines:

destination d_stunnel {tcp("127.0.0.1" port(514));};
log {source(s_local);destination(d_stunnel);};

The server needs these lines:

source s_stunnel {tcp(ip("127.0.0.1");port(514));};
log {source(s_stunnel); destination(d_clients);};

Start up stunnel by running the stunnel command, and fire up syslog-ng if it isn't running already. Run some tests with the logger command (see Part 1), and there you are, running your own nicely secured remote logging server.

Resources

• the Syslog-ng Reference Manual is an essential reference
  
• Stunnel.org
  
• OpenSSL.org

[syslog-ng]Configure syslog-ng to act as a central logging server

This is the base config file I'm working with..Note: i'm using unix_stream. Should I be using unix_dgram since im on a 2.6x kernel? I'm reading through the section on syslog-ng in the book
"Building Secure Server with Linux", and it mentions something about using "unix-dgram" .. note the "-" instead of "_". I'm assuming its a typo and should be "_".------

source s_sys { file ("/proc/kmsg" log_prefix("kernel: "));
unix_stream("/dev/log"); internal(); };

destination d_cons { file("/dev/console"); };
destination d_mesg { file("/var/log/messages"); };
destination d_auth { file("/var/log/secure"); };
destination d_mail { file("/var/log/maillog"); };
destination d_spol { file("/var/log/spooler"); };
destination d_boot { file("/var/log/boot.log"); };
destination d_cron { file("/var/log/cron"); };
destination d_mlal { usertty("*"); };

filter f_filter1 { facility(kern); };
filter f_filter2 { level(info) and
not (facility(mail)
or facility(authpriv) or facility(cron)); };
filter f_filter3 { facility(authpriv); };
filter f_filter4 { facility(mail); };
filter f_filter5 { level(emerg); };
filter f_filter6 { facility(uucp) or
(facility(news) and level(crit)); };
filter f_filter7 { facility(local7); };
filter f_filter8 { facility(cron); };

#log { source(s_sys); filter(f_filter1); destination(d_cons); };
log { source(s_sys); filter(f_filter2); destination(d_mesg); };
log { source(s_sys); filter(f_filter3); destination(d_auth); };
log { source(s_sys); filter(f_filter4); destination(d_mail); };
log { source(s_sys); filter(f_filter5); destination(d_mlal); };
log { source(s_sys); filter(f_filter6); destination(d_spol); };
log { source(s_sys); filter(f_filter7); destination(d_boot); };
log { source(s_sys); filter(f_filter8); destination(d_cron); };

[syslog-ng]Configure syslog-ng to act as a central logging server

Where can I find documentation that will give me information on how to accept logs from remote hosts and have each of the hosts log to their own file?

destination hosts {
file("/var/syslog/HOSTS/$HOST/$YEAR/$MONTH/$HOST-$FACILITY-$YEAR$MONTH$DAY"

owner(bob) group(llamafood) perm(0755) dir_perm(0755) create_dirs(yes));

};

[Jun 21, 2005] Windows event logs to syslog

• Windows Eventlog to Syslog Utility

  See also discussion at The syslog-ng 2005-April Archive by Thread

• Snare Agent from http://sourceforge.net/projects/snare/
• http://ntsyslog.sourceforge.net
  

Syslog-ng 1.6.7 package for Solaris 10, Sparc platform

[Jun 20, 2005] Remote Logging with SSH and Syslog-NG by Hal Pomeranz

One of the points I make repeatedly in my training classes is the value of centralized logging. Keeping an off-line copy of your site's logs on some central, secure log server not only gives you greater visibility from a systems management perspective but also can prove invaluable after a security incident when the local copies of the log files on the target system(s) have been compromised by the attacker.

Recommended Links

• Syslog-ng - Wikipedia, the free encyclopedia
• Syslog-ng homepage
  • syslog-ng reference manual
• Syslog-ng faq
• Sample chapter from "Building Secure Servers With Linux" (O'Reilly and Associates) covering syslog-ng
• Building a central loghost with syslog-ng
• Cool Solutions Centralized Syslogging with Syslog-NG on SUSE Linux
• Syslog-ng mailing list signup
• Centralized syslog-ng to mysql database
• syslog(-ng) to postgreSQL database howto
• Centralized Syslog Server Using syslog-NG with web Interface using php-syslog-ng
• Central Loghost Mini-HOWTO
• Monitoring - Automated Log Monitoring with LogSentry and a Central ...
• Creating a central syslog server
• Central syslog server | Syslog-ng
• Implementation of Central Logging Server using syslog-ng
• Cool Solutions- Centralized Syslogging with Syslog-NG on SUSE Linux
• Install Centralized Syslog Server with Web Interface - PlanetIT
• BigAdmin Feature Article- Creating a Centralized Secure Log Server ...
• Centralizing Syslog on Solaris - The Blog of Ben Rockwood
• How many of you do centralized syslog-ing- - Web Hosting Talk

Configuration examples

• Sample Solaris 8 syslog-ng conf file (for syslog-ng 1.5.x or higher)
• Sample syslog-ng conf file
• Sample debian syslog-ng conf file to replace the stock syslogd
• An expanded sample syslog-ng conf file
• Nate Campi's sample syslog-ng configuration file

Etc

• Attack Signatures, Tina Bird's syslog Attack Signatures course
• Centralized Syslog Loghost Project
• Log Consolidation with syslog sans.org
• logreport : transform raw data in network/computer system log files into valuable information for you
• modular syslog
• mtsyslogd - Multi Threaded Modular Syslogd Replacement
• Robust event logging with Syslog integrate the Syslog package into your Java programs
• syslog mail archive
• Syslog-ng : syslog-ng adds the possibility to filter based on message contents using regular expressions
  Linux, Solaris and FreeBSD
• Syslog-Sign Protocol IETF draft

Recommended Papers

• Sys Admin Remote Logging with SSH and Syslog-NG by Hal Pomeranz
• CERT India's doc on implementing a central syslog architecture using syslog-ng
• *** Centralized Syslog Loghost using syslog-ng
• Log Analysis Resources (LogAnalysis mailing list)
• Building Secure Servers with Linux, 'Chapter 10: System Log Management and Monitoring'

Mick's syslog-ng chroot howto

Using syslog-ng and Stunnel for a Centralized, Secure Log Server

Article Explains Technique in a Step-by-step Approach A featured article on Sun's BigAdmin Web site by Amy Rich describes how to create a centralized secure log server by using syslog-ng and Stunnel. She covers everything from installing Stunnel and syslog-ng to creating certificate files for syslog-ng over Stunnel, as well as configuring both these tools.

Rich begins by addressing the issue of uncollected information that is compiled with syslog daemon due to time and staff constraints. Centralized logging assists in this matter, however, she acknowledges that syslogd hasn't seen much development since its original implementation and does not address all issues.

"...most centralized log servers using the stock syslog daemon wind up with monolithic log files that are only processed after the syslogd processes close them," Rich writes. "As a result, most sites that centralize logging also wind up replacing the stock syslog daemon with something more secure and more flexible such as Metalog, msyslog, or something similar."

Rich then explains the benefits of syslog-ng, and how when run on each UNIXR host, it encrypts the logging channel with IPSec or the utility Stunnel ensuring its security. She describes the interworkings of Stunnel and its process with SSL sessions and the syslog server. She writes, "In the case of syslog-ng, Stunnel works by accepting log connections on a local port, wrapping them in SSL sessions, and then redirecting them to a secure port on the remote log host. The stunnel process on the remote log host then decrypts the SSL session and hands the information back to the syslog server on the standard port. Once on the log server, organization and parsing of the log files then takes place using the flexibility of syslog-ng."

The article continues by taking the reader through a step-by-step process of installing and configuring syslog-ng and Stunnel on systems running the SolarisTM 8 Operating System (Solaris OS) (SPARCR Platform Edition), with code examples.

To read Rich's complete article, visit:

http://www.sun.com/bigadmin/features/articles/syslog_ng.html

Linux Journal On-Line

Stealthful Logging

Now it's time to combine the previous two techniques in a third one: stealth logging. Normally, a central log server runs syslog or syslog-ng. And indeed, it's possible to capture log packets via a non-IP-addressed interface with Snort and pass them to syslog or syslog-ng. However, it's a lot simpler to let Snort write the packets to a log file itself. The method I'm about to describe uses Snort, tail and awk instead of a traditional logger (on the central logserver, that is; on hosts that send logs you'll still need to configure syslog or syslog-ng as described below).

Suppose you have a LAN segment with several servers whose log messages you'd like sent to a single log server. Suppose further that you're far less concerned with the confidentiality of these log messages than with their integrity. You don't care if anybody eavesdrops the messages, but you don't want the messages tampered with once they've been received by the central log server. Your log server, therefore, is connected to the LAN via a non-IP-addressed interface and will sniff log packets sent to a bogus IP address on the LAN.

Configuring Your Hosts to Send Logs to the Stealth Logger

On each server from which you wish to send logs, you'll need to do two things. The first step is to configure each sender's system logger to send its messages to the bogus IP. By ``bogus'', I only mean that no host actually has that IP address; it must be valid for the LAN in question. Suppose our example LAN's network address is 192.168.1.0/24 and our bogus logger-host address is 192.168.1.111. Servers on the LAN that use standard syslog will each need a line like this in /etc/syslog.conf:

*.info @192.168.1.111

On servers that use syslog-ng, you'll need several lines in /etc/syslog-ng/syslog-ng.conf, like these:

destination d_loghost { udp(ip(192.168.1.111)
port(514)); };
filter f_info { level(info); };
log { filter(f_info); destination(d_loghost); };

In either case you may wish to specify different criteria from simply saying ``all messages whose severity is 'info' or higher'', as shown in the above two examples.

Besides the appropriate lines in its logger's configuration file, each log sender will also need a static ARP entry for the bogus IP address. If your LAN consists of a hub or a series of ``cascaded'' hubs, this ARP address also can be bogus. If your LAN is switched, you'll instead need to specify the real MAC address (hardware address) of the log server. The command to add a static ARP entry on our example log-sending servers is:

arp -s 192.168.1.111 00:04:C2:56:54:58

where 192.168.1.111 is our bogus log-host IP, and 00:04:C2:56:54:58 is either a bogus MAC address or the real MAC address of our stealth logger's non-IP-addressed interface.

Now each server on our protected LAN is configured to send its logs to 192.168.1.111, and in the case of a switched LAN, they'll be sent to the stealth logger's switch port. Now all we need to do is configure the stealth logger itself

[Jun 22nd, 1999] Syslog-ng Syslog-ng By Balazs Scheidler --Historically important paper from LG #43

Reference

syslog-ng [ -dFsvVy ] [ -f <config-filename> ] [ -p <pid-filename> ] [ -C <chroot-dir> ] [ -u <user> ] [ -g <group> ]

DESCRIPTION

syslog-ng reads and logs messages to the system console, log files, other machines and/or users as specified by its configuration file.

The configuration file is read at startup and is reread after receipt of a hangup (HUP) signal. When reloading the configuration file, all destination files are closed and reopened as appropriate. For more information about the configuration file, see syslog-ng.conf(5).

Typically messages are sent to syslog-ng via a Unix domain socket (/dev/log) or via UDP, to port 514 or to whatever syslog/udp is defined to be in /etc/services. To receive messages from the kernel, /dev/klog is opened for reading.

Messages sent to syslog-ng should be an entire line, prefixed by a priority code in between '<' and '>'. Definititions of the recognised values can be found in the include file <sys/syslog.h>.

syslog-ng can be configured to pass messages on to other syslog-ng's and when doing so, it sends the message on with the priority as a prefix.

It is not necessary to terminate a message with a line feed or carriage return.

OPTIONS

-C <directory>, --chroot=<directory>

Chroot to directory.

-d, --debug

Set "debug mode". Prints out various messages to aid in debugging and stops it from becoming a daemon.

-y, --yydebug

Enable debugging messages in the yacc parser, this option is only present if syslog-ng was compiled in debugging mode (--enable-debug configure option).

-F, --foreground

Don't fork into background.

-f <filename>, --cfgfile=<filename>

Instead of reading the normal /etc/syslog-ng/syslog-ng.conf file for configuration information, use the filename given instead.

-g <group>, --group=<group>

Switch to group.

-p <filename>, --pidfile=<filename>

Write the current PID information to the specified file. Defaults to /var/run/syslog-ng.pid.

-s, --syntax-only

Only read and parse the configuration file. See also the option -f.

-u <user>, --group=<user>

Switch to user.

-v, --verbose

Enable verbose mode. Prints out fewer messages, compared to -d.

-V, --version

Print the version number.

DIAGNOSTICS

SEE ALSO

FILES

/etc/syslog-ng/syslog-ng.conf

Configuration file for syslog-ng. See syslog-ng.conf(5) for more information.

/var/run/syslog-ng.pid

The file containing the process id of syslog-ng.

BUGS

If you find any, please send email to the syslog-ng mailing list at [email protected]

Random Findings

• San Diego SuperComputer Center syslog
• minirsyslogd - Mikael Olsson's minimal log server, designed to handle high volumes of syslog traffic from remote devices. Runs under any UNIX, and includes the facility and level tags in the messages it receives. New version released October 2003.
• idsa - an experimental reference monitor, logger and IDS. The log implementation includes verification of system information (such as PIDs), rather than assuming that an application will log the correct information; built in quotas and rotation mechanisms; structured log message formats; and granular filtering capability. The software is licensed under the (L)GPL and includes a syslogd implementation. Marc Welz maintains the package.
• Modular syslog: a syslog replacement that includes data integrity checks, easy database integration, and output redirection using regular expressions.
  • Do you Trust your System Logs?: a layman's description of the integrity checks used by the Modular syslog package, as well as tips on integration.
• nsyslog: nsyslog supports TCP connections for log transfer, and with SSL allows for encrypted delivery of syslog messages across the network. Contact Darren Reed for more information.
• Flexible Logging: OSSP l2 - [From the abstract] OSSP l2 is a C library providing a very flexible and sophisticated Unix logging facility. It is based on the model of arbitrary number of channels, stacked together in a top-down data flow tree structure with filtering channels in internal nodes and output channels on the leaf nodes.
• sysklogd The sysklogd package implements two system log daemons. The syslogd daemon is the general system logging daemon which is responsible for handling requests for syslog services. This version of syslogd is similar to the standard Berkeley product but with a number of compatible extensions. The klogd daemon runs either standalone or as a client of syslogd. Klogd 'listens' to kernel log messages, prioritizes them, and routes them to either output files or to syslogd.
• Secure syslog tools
• socklog: Gerrit Pape's small, secure replacement for syslog.
• algr: a reliable, robust and high performance event logging mechanism.
• Snare: An audit event collection, analysis, reporting and archive service, and security 'agents' for multiple operating systems and applications.
• multilog: Dan Bernstein's high performance, high reliability local replacement for syslog
  • Jeff Saxe's intro to multilog

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: April 23, 2019