Remote Syslog

The default Solaris syslogd daemon is a primitive (old style) daemon that is controlled by /etc/syslog.conf. It can send/receive messages from the remote hosts but the level of control is very primitive (they will be mixed with the logs from the LOGHOST server). syslog-ng is probably a step forward and can be used with Solaris.  In all cases usage of NTP for clock syncronization is a must. This is the most important prerequsite to implementing syslog aggregation on the "master" syslog server in the organization. See RHEL NTP configuration and Troubleshooting NTP on Red Hat Linux for the additional information.

The default syslog daemon uses UDP port 514 for forwarding messages to a centralized log host. There is no acknowledgement and messages are lost if the LOGHOST server is down, path to the master loghost server is not available or if its syslogd  daemon is down. Some nasty surprises are possible is a firewall filters out this port on the route tot he LOGHOST. 

In Solaris syslog is implemented using STEAMS log driver. The specific wording in syslogd(1M):

syslogd reads from the STREAMS log driver, /dev/log, and from any transport provider specified in /etc/netconfig, /etc/net/transport/hosts, and /etc/net/transport/services.

Solaris differs from other Unixes in the methods of sending remote logs: here is the table from syslog-ng reference manual

Platform	Method
Linux	A SOCK_STREAM unix socket named /dev/log, some of the distributions layer switched over to using SOCK_DGRAM, though applications still work with either method.
BSD flavors	A SOCK_DGRAM unix socket named /var/run/log
Solaris (2.5 or below)	An SVR4 style STREAMS device named /dev/log
Solaris (2.6 or above)	In addition to the STREAMS device used in versions below 2.6, uses a new multithreaded IPC method called door. By default the door used by syslogd is /etc/.syslog_door

CPU load on a plain vanilla LOGHOST server is minimal and even an old entry level Sun server (V240) can integrated a couple of hundreds of hosts without much sweat. But that does not make sense as after a hundred hosts it is better to split them into groups representing major classes (webservers, database servers, etc) and use either syslog-ng or virtual machine (zones in Solaris 10) for each.

Also due to the mixture of logs from the remote hosts with LOGHOST server logs, generally you will be better off using Solaris for consolidating logs from Solaris servers, AIX for consolidating syslog from AIX servers, etc.

You can use remote syslog with or without encryption as well. As for encryption IPSEC is one option, Stunnel is an another one. But too much zeal is harmful here ;-)

The simplest remote syslog configuration is an unencrypted one. To activate such remote sysloging on Solaris you need:

• To edit /etc/hosts  (./inet/hosts) and add one or several lines that defines IP address for loghost (Name is arbitrary, and if several remote hosts are defined you can use any name you wish)
• Edit syslog configuration file (/etc/syslog.conf) for each server you need to collect logs from. To send everything to remote syslog server, add this line to your syslogd.conf file:

  *.*        @loghost

  After editing file you need to restart syslogd:

  pname -HUP syslogd

  or

  /etc/rc2.d/S74syslog stop ; /etc/rc2.d/S74syslog start

  You can use a full DNS name like logbase.mycompany.com In this case you do not need to touch hosts file, but remote logging became dependent on the DNS availability.

Solaris uses macro processor M4 to preprocess syslog.conf and you can use all the power of M$ to make the configuration more flexible. Example in the standard syslogd configuration is a pretty weak one: it is based on checking if LOGHOST env. variable is defined (it is only if IP address of the server is defined as loghost in /etc/hosts) The ifdef (‘LOGHOST’, truefield, falsefield) command permits to select two variants of log forwarding depending on this. This permits any server to be used by loghost. Nice, but pretty useless in practice trick as LOGHOST server generally needs to be a specialized and more secure server.

Also you can replace syslogd with syslog-ng. The latter lets filter and route logs. The ability of syslog-ng to filter messages of various types provides the capability to distribute specific message types to people responsible for a given application/function. That makes the environment closer to VM/CMS system ;-). It is possible to invoke a script on the receipt of specific mail messages. If you use Perl the logic can be pretty sophisticated.

Creating a loghost for your servers can be invaluable both for troubleshooting and in the event of a security breach. If you are part of a cluster, make sure your machine logs to another loghost as well as itself. If someone breaks in and erases your log files, and you have only been logging locally, you will have no chance of figuring out where they came from or what they did. Intruders commonly use scripts to erase their presence from the log files. By sending your logs to another more secure machine, you have a better chance of at least tracking the intruder's activities.

Security precautions to take

• Make sure that NFS daemons are configured correctly and are running on all the servers. Otherwise you will have trouble tracing problems and breakins.
• Make sure that "log partition" is large enough to hold logs from all servers for at least 90 days. System logs are traditionally wriiten to the /var partition, althouth you can define /var/adm or var/log as a sepate huge partition on the loghost servers.  This is important to prevent accidental overflows, if some server expriced what is callled syslog messages storm and generate huge amount of messages before the problem is resolved. Generally ten times of normal average volume of logs for 90 days is a good eximate that allo you to handle such situations.
• The default syslog.conf does not do a very good job of logging. Try changing the entry for /var/adm/messages to:

  *.info /var/adm/messages

  If you start getting too much of a certain message (say from sendmail), you can always bump that particular facility down by doing

  *.info;mail.none /var/adm/messages

• To enable more logging of logins, create the "loginlog" on Solaris:

  touch var/adm/loginlog 
  chmod 600 /var/adm/loginlog 
  chgrp sys /var/adm/loginlog

• Enable ssh and tcpwrappers that log important events to syslogd.

Additional security precautions for LOGHOST

• In order to create a loghost, pick one machine and secure it as much as possible. Basically, don’t run anything on this machine besides syslogd. Turn off inetd, sendmail, everything, but make sure you have basic networking up. Possibly don’t even run ssh on this machine. That way, the only way to view the log files on the loghost would be to physically log into the console.
• Make sure the time is always correct on the loghost.
• In order to allow the loghost to receive syslog messages from other machines, you may need to enable it to receive remote logging. (Find out first by reading the syslogd man pages). Do this by adding the –r command line upon syslogd startup. Edit /etc/rc2.d/S74syslog, and find the line:

  /usr/sbin/syslogd

  and change it to:

  /usr/sbin/syslogd -r

• Again, make sure the loghost is as secure as possible: the only thing it should be running is syslogd.

Protocol concerns

• No congestion control with UDP, allowing denial of service attack (Contra argyment: "Simple protocol")
• UDP packets are easily spoofed. (Contra: "No more easily than TCP, aside from sequence number.")
• No authentication - no way for a loghost to reject messages received.
• No confidentiality of message data. (Contra: "Use IPSEC." if you are really concered about security)
• Vulnerable to man-in-middle attack -- packets may be altered undetectably in transit. (Contra: "Use IPSEC.")
• Vulnerable to spoofing/chaffing -- bogus messages can be sent to the logging host. (Contra: "Use IPSEC.")
• Gaps in the message sequence due to lost packets can't be detected by receiver

Implementation woes

• Vulnerable to successful intruders: need to have inviolability of the log itself. Contraargument: you first need to make logs availbale to your personnel that most often does not read them at all. also Loghost server can be easily secured more tightly to aviod common breakins.
• Timestamp is provided at receiver, not originator; forwarding often adds observable delay potencially causing misinterpretation.
• Limited information on originating facility and severity level (priority) of message
• Lack of flexibility in use of facility codes among systems and applications -- making it difficult to write filter scripts to detect security event.

Protocol gripes

1. Firewalls & proxies:  need UDP port 514 proxy
2. Unreliable:  No guarantee that a syslog packet will be received, and no facilities for retransmission (Contra: "Simple protocol, usually not needed on a typical LAN")
3. Better Timestamping:  It should be a requirement that the timestamp specified in the protocol (both by the client, in the packet; and in the server, when the message is logged) are recorded in the logfile.
4. Problems with log message formatting & structure:  "The structure of messages is really a mess."
   • No standard way of formatting of message text and separating various arguments of the log message
   • Priority/facility are encoded in a not particularly human-readable way, but then sent in text form. "This has the worst features of binary protocols (not human-readable) and text protocols (inefficient use of bandwidth - and just to make this one worse, the first byte of every packet is constant!)"
5. Standardisation:  syslog is not standardised, which makes it more difficult than it should be to produce interoperable implementations. (Contra: "Syslog is currently interoperable and actually more or less standardized between all flavours of Unix though there are some individual variations.")

Implementation woes

• Repeat notification: '"...repeated N times" doesn't show up until a different line comes along which is a pain if you're watching the log.
• Problems with multiple syslog.conf lines that name the same file.
• Problem with spraying LAN with useless packets when logserver became unavailable or IP address changed.
• No ability to define daemon's fsync() behavior.
• No standardized logging directory (/var/log, /var/adm, ...).
• No forwarding for sulog.

Syslog is still the most widespread and usable tool for event logging, esp. because it is a buil-in, low-cost protocol that can be easily added to network and embedded devices. Strategies for cautious use include:

• Use physically separate out-of-band networks for all management data including syslog.
• Syslog configuration can use non-IP interfaces between log hosts, e.g. serial links, to keep the destination loghost out of the ARP tables of a compromised access host.
• Firewall filtering, route authentication, VLAN assignments, etc. to limit udp/514 syslog, can isolate syslog regions within a network as well as inside from outside.
• Syslog servers can be provided with CPU and disk capacity far exceeding normal needs.
• Syslog daemon code can be altered for improved filtering and handling of application-specific messages. (This has been widespread in the UNIX and Open Source community, with versions of syslog() and syslogd sometimes distributed with other applications -- e.g. INN.)

Basic security enhancements

• Out-of-band management network can be used for logging.
• Non-IP links can limit visibility of syslog forwarding path.
• IP access list for syslogd can permit receiver to drop all packets not claiming to be from a valid source address.
• Check file permissions for all logfiles and syslog.conf.

NEWS CONTENTS

• 20080702 : Linux Configure Netconsole To Log Messages Over UDP Network by Vivek Gite ( Linux Configure Netconsole To Log Messages Over UDP Network, Jul 2, 2008 )
• 20080702 : http://www.monitorware.com/Common/en/Ar ... server.php ( http://www.monitorware.com/Common/en/Ar ... server.php, )
• 20080702 : SecurityFocus SUN Re Better Syslog server ( SecurityFocus SUN Re Better Syslog server, )
• 20080702 : Syslog.org View topic - Server specs for enterprise syslog servers ( Syslog.org View topic - Server specs for enterprise syslog servers, )
• 20080702 : BigAdmin Feature Article Creating a Centralized Secure Log Server with syslog-ng and Stunnel ( BigAdmin Feature Article Creating a Centralized Secure Log Server with syslog-ng and Stunnel, )
• 20080702 : Central Loghost Mini-HOWTO ( Central Loghost Mini-HOWTO, )
• 20080702 : Aggregate Logs from Remote Sites ( Aggregate Logs from Remote Sites, )

Old News ;-)

[Jul 2, 2008] Linux Configure Netconsole To Log Messages Over UDP Network by Vivek Gite

July 2, 2008

Linux can be configured to log dmesg output to another system via network using syslog. It is done using kernel level networking stuff ia UDP port 514. There is module called netconsole which logs kernel printk messages over udp allowing debugging of problem where disk logging fails and serial consoles are impractical. Most modern distro has this netconsole as a built-in module. netconsole initializes immediately after NIC cards. There are two steps to configure netconsole:

• Syslogd server - Let us assume 192.168.1.100 IP having FQDN - syslogd.nixcraft.in. Please note that the remote host can run either 'netcat -u -l -p <port>' or syslogd.
• All other systems running netconsole module in kernel

Step # 1: Configure Centralized syslogd

Login to syslogd.nixcraft.in server. Open syslogd configuration file. Different UNIX / Linux variant have different configuration files

Red Hat / CentOS / Fedora Linux Configuration

If you are using Red Hat / CentOS / Fedora Linux open /etc/sysconfig/syslog file and set SYSLOGD_OPTIONS option for udp logging.
# vi /etc/sysconfig/syslog
Configure syslogd option as follows:
SYSLOGD_OPTIONS="-m 0 -r -x"
Save and close the file. Restart syslogd, enter:
# service syslog restart

Debian / Ubuntu Linux Configuration

If you are using Debian / Ubuntu Linux open file /etc/default/syslogd set SYSLOGD option for udp logging.
# vi /etc/default/syslogd
Configure syslogd option as follows:
SYSLOGD_OPTIONS="-r"
# /etc/init.d/sysklogd restart

FreeBSD configuration

If you are using FreeBSD open /etc/rc.conf and set syslogd_flags option option for udp logging. Please note that FreeBSD by default accepts network connections. Please refer to syslogd man page for more information.

Firewall configuration

You may need to open UDP port 514 to allow network login. Sample iptables rules to open UDP port 514:
MYNET="192.168.1.0/24"
SLSERVER="192.168.1.100"
iptables -A INPUT -p udp -s $MYNET --sport 1024:65535 -d $SLSERVER --dport 514 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p udp -s $SLSERVER --sport 514 -d $MYNET --dport 1024:65535 -m state --state ESTABLISHED -j ACCEPT

Step # 2: Configure Linux Netconsole

You need to configure netconsole service. Once this service started, you are allowed a remote syslog daemon to record console output from local system. The local port number that the netconsole module will use 6666 (default). You need to set the IP address of the remote syslog server to send messages.

Open /etc/sysconfig/netconsole file under CentOS / RHEL / Fedora Linux, enter:
# vi /etc/sysconfig/netconsole
Set SYSLOGADDR to 192.168.1.100 (IP address of remote syslog server)
SYSLOGADDR=192.168.0.1
Save and close the file. Restart netconsole service, enter:
# /etc/init.d/netconsole restart

A note about Debian / Ubuntu Linux

Red Hat has netconsole init script. However, under Debian / Ubuntu Linux, you need to manually configure netconsole. Type the following command to start netconsole by loading kernel netconsole module, enter:
# modprobe netconsole [email protected]/eth0,[email protected]/00:19:D1:2A:BA:A8
Where,

• 6666 - Local port
• 192.168.1.5 - Local system IP
• eth0 - Local system interface
• 514 - Remote syslogd udp port
• 192.168.1.100 - Remote syslogd IP
• 00:19:D1:2A:BA:A8 - Remote syslogd Mac

You can add above modprobe line to /etc/rc.local to load module automatically. Another recommend option is create /etc/modprobe.d/netconsole file and append following text:
# echo 'options netconsole [email protected]/eth0,[email protected]/00:19:D1:2A:BA:A8 '> /etc/modprobe.d/netconsole

How do I verify netconsole is logging messages over UDP network?

Login to remote syslog udp server (i.e. 192.168.1.100 our sample syslogd system), enter:
# tail -f /var/log/messages
/var/log/messages is default log file under many distributions to log messages. Refer to /etc/syslog.conf for exact location of your file.

How do I use nc / netcat instead of messing with syslogd?

This is called one minute configuration. You can easily get output on 192.168.1.100 without using syslogd. All you have to do is run netcat (nc) command, on 192.168.1.100:
$ nc -l -p 30000 -u
Login to any other box, enter command:
# modprobe netconsole [email protected]/eth0,[email protected]/00:19:D1:2A:BA:A8
Output should start to appear on 192.168.1.100 from 192.168.1.5 without configuring syslogd or anything else.

Further readings:

• nc / netcat command
• modprobe command
• netconsole documentation
• man pages nc, modeprobe

http://www.monitorware.com/Common/en/Ar ... server.php

A paper on optimizing syslog server performance. While it is written for our products, almost all of the same principles apply to generic syslog (except for the multi-threading discussion, because many syslogd's are single-threaded).

SecurityFocus SUN Re Better Syslog server

Date: Mon, 17 Mar 2003 14:08:08 -0500 (EST)
From: Jameel Akari <[email protected]>
To: Matt Harris <[email protected]>

On Mon, 17 Mar 2003, Matt Harris wrote: 

> I've been looking a bit on google/sourceforge/etc to try and find a more configurable and extensible syslog server, to no avail. Does anyone

There is syslog-ng, but IMO it's still lacking.

> # commands for syslog facility "XXX"
> Facility XXX {
> # Send message to stdin of a script which can send alerts, etc
> Severity emerg ACTION /usr/local/libexec/parse-emerg.sh
> # 192.168.47.9 - sample dhcp server
> Host "192.168.47.9" {
> # log stuff from the dhcp server to a seperate file
> Severity warn FILE /var/adm/dhcp.log
> }
> # send other stuff to a default file
> Severity DEFAULT FILE /var/adm/xxx.log
> }

Looks like a good idea to me.

> functionality I'd like - for example, piping out to a smart script,
> sending to different files based on host that sent the message, etc

Logging to a [My|Postgre]SQL backend instead of or in addition to file would be very cool. Stick a PHP frontend on it and make pretty charts and graphs.

Doing things like "tail -f /var/adm/messages | someparserscript.pl" kinda works, but is... non-optimal?

Syslog.org View topic - Server specs for enterprise syslog servers

There really isn't a good rule of thumb, because there are so many variables in the mix. You have 60 hosts going to 3 servers. I'm not sure if that means that 20 hosts will be logging to each, or if all 60 will be sending logs to all 3. The number of logs that each host will be sending per minute is an important item to know. Syslogd is a very very simple application that pulls the data off the wire and shoves it in a file. There are some implementations of syslogd that do write caching, and that will speed up performance.

Otherwise, it really is going to come down to disk speed & IO primarily. If you're able to get 20Mbps out of a traditional disk, that's going to theoretically give you the ability to process 20000 1024 bits syslog packets per second. Now, there are limitations that the OS, file system, etc will impose that will make that less, but it's going to be a pretty big number.

As a rule, drive speed and IO are paramount, then probably memory size and speed, then clock speed of the processor, in order of priority. Unless you are doing something out of the ordinary on the 60 hosts that are sending syslogs, I think you'll be fine with most any server class hardware that you would throw at it, with a lot of room to spare.

Post subject: RE: Server specs for enterprise syslog servers

Feb 19, 2005

Yeah that's the hard part...there won't be a constant flow of traffic, it all depends on what users changing things on the server at a specific time. It's going to be about 60 hosts per syslog server, so IO speed is the main thing and processor is the least then. I'm planning on piping the information into a mysql database as well.

BigAdmin Feature Article Creating a Centralized Secure Log Server with syslog-ng and Stunnel

UNIX system administrators are quite familiar with the syslog daemon, but information that it collects often remains unprocessed unless someone reports a problem. At any site with more than a handful of machines, no one devotes the time to log in and check multiple log files each day or even each month. Automated scripts that might correlate data across these machines are difficult to write because they must access each machine individually. To lessen the burden of both automatic and manual data processing, many sites implement a central log server that collects data for all machines (preferably running NTP to make time/date correlation easier) on a network, including UNIX servers, Windows and Mac desktops, and even networking equipment like routers and switches. Centralized logging is fairly trivial with most stock UNIX syslog daemons, but syslogd is little changed from early incarnations and has some shortcomings.

The standard UNIX syslog daemon transports messages in the clear over UDP, meaning that anyone can sniff potentially sensitive data. The facility.level model is also fairly restrictive, and the default /etc/syslog.conf files supplied with most operating systems neglect to log many messages administrators might be interested in. Parsing out the logs in a way that makes sense for human readability or automated data mining is non-trivial. The UNIX syslog daemon doesn't easily allow one to split log files out by host or match log messages by regular expression, for example. Therefore, most centralized log servers using the stock syslog daemon wind up with monolithic log files that are only processed after the syslogd processes close them.

As a result, most sites that centralize logging also wind up replacing the stock syslog daemon with something more secure and more flexible such as Metalog, msyslog, or something similar. One very popular syslog replacement is an open source program called syslog-ng. An organization can run syslog-ng on each UNIX host or just on the syslog server itself. If syslog-ng is run only on the log host, clients send data over UDP port 514 as usual, but better log organization and manipulation can be accomplished on the server.

The benefit of running syslog-ng on each UNIX host is the ability to encrypt the logging channel with IPSec or the utility Stunnel so that data is not readable by the casual sniffer. When combined with Stunnel as a transport mechanism, an organization has a secure way of centralizing log messages from all desired UNIX hosts for further processing. In the case of syslog-ng, Stunnel works by accepting log connections on a local port, wrapping them in SSL sessions, and then redirecting them to a secure port on the remote log host. The stunnel process on the remote log host then decrypts the SSL session and hands the information back to the syslog server on the standard port. Once on the log server, organization and parsing of the log files then takes place using the flexibility of syslog-ng.

Below I cover the installation and configuration of syslog-ng and Stunnel on machines running the Solaris 8 Operating System (SPARC Platform Edition), but the procedure also generally applies to older and newer versions of the Solaris OS on both SPARC and x86 platforms. Each of the reference machines discussed below comes installed with OpenSSL, tcp wrappers, the Solaris 8 /dev/urandom patch, and the GNU development environment (gcc, and so on) and several other freeware packages. The machine serving as the log server has also been thoroughly hardened since it will store sensitive and security-related information from all machines on the network. These reference machines all reside on the subnet 192.168.1, and the IP address of the log server is 192.168.1.10.

Central Loghost Mini-HOWTO

Aggregate Logs from Remote Sites

Monitoring the logs of a remote site or just a collocated server can often be overlooked when faced with the task of monitoring activity on your local network. You could use the traditional syslog facilities to send logging information from the remote network or systems, but since the syslog daemon uses UDP for sending to remote systems, this is not the ideal solution. UDP provides no reliability in its communications, and so you risk losing logging information. In addition, the traditional syslog daemon has no means to encrypt the traffic that it sends, so your logs might being viewable by anyone with access to the intermediary networks between you and your remote hosts or networks.

To get around these issues, you'll have to look beyond the syslog daemon that comes with your operating system and find a replacement. One such replacement syslog daemon is syslog-ng (http://www.balabit.com/products/syslog_ng/). syslog-ng is not only a fully functional replacement for the traditional syslog daemon, but also adds flexible message filtering capabilities, as well as support for logging to remote systems over TCP (in addition to support for the traditional UDP protocol). With the addition of TCP support, you can also employ stunnel or ssh to securely send the logs across untrusted networks.

To build syslog-ng, you will need the libol library package (http://www.balabit.com/downloads/syslog-ng/libol/) in addition to the syslog-ng package. After downloading these packages, unpack them and then build libol:

$ tar xfz libol-0.3.9.tar.gz
$ cd libol-0.3.9
$ ./configure && make

When you build syslog-ng you can have it statically link to libol, so there is no need to fully install the library.

And now to build syslog-ng:

$ tar xfz syslog-ng-1.5.26.tar.gz
$ cd syslog-ng-1.5.26
$ ./configure --with-libol=../libol-0.3.9
$ make

If you want to compile in TCP wrappers support, you can add the --enable-tcp-wrapper flag to the configure script. After syslog-ng is finished compiling, become root and run make install. This will install the syslog-ng binary and manpages. To configure the daemon, create the /usr/local/etc/syslog-ng directory and then create a syslog-ng.conf to put in it. To start off with, you can use one of the sample configuration files in the doc directory of the syslog-ng distribution.

There are five types of configuration file entries for syslog-ng, each of which begins with a specific keyword. The options entry allows you to tweak the behavior of the daemon, such as how often the daemon will sync the logs to the disk, whether the daemon will create directories automatically, and hostname expansion behavior. source entries tell syslog-ng where to collect log entries from. A source can include Unix sockets, TCP or UDP sockets, files, or pipes. destination entries allow you to specify possible places for syslog-ng to send logs to. You can specify files, pipes, Unix sockets, TCP or UDP sockets, TTYs, or programs. Sources and destinations are then combined with filters (using the filter keyword), which let you select syslog facilities and log levels. Finally, these are all used together in a log entry to define precisely where the information is logged. Thus you can arbitrarily combine any source, select what syslog facilities and levels you want from it, and then route it to any destination. This is what makes syslog-ng an incredibly powerful and flexible tool.

To set up syslog-ng on the remote end so that it can replace the syslogd on the system and send traffic to a remote syslog-ng, you'll first need to translate your syslog.conf to equivalent source, destination, and log entries.

Here's the syslog.conf for a FreeBSD system:

*.err;kern.debug;auth.notice;mail.crit          /dev/console
*.notice;kern.debug;lpr.info;mail.crit;news.err /var/log/messages
security.*                                      /var/log/security
auth.info;authpriv.info                         /var/log/auth.log
mail.info                                       /var/log/maillog
lpr.info                                        /var/log/lpd-errs
cron.*                                          /var/log/cron
*.emerg                                         *

First you'll need to configure a source. Under FreeBSD, /dev/log is a link to /var/run/log. The following source entry tells syslog-ng to read entries from this file:

source src { unix-dgram("/var/run/log"); internal( ); };

If you were using Linux, you would specify unix-stream and /dev/log like this:

source src { unix-stream("/dev/log"); internal( ) };

The internal() entry is for messages generated by syslog-ng itself. Notice that you can include multiple sources in a source entry. Next, include destination entries for each of the actual log files:

destination console { file("/dev/console"); };
destination messages { file("/var/log/messages"); };
destination security { file("/var/log/security"); };
destination authlog { file("/var/log/auth.log"); };
destination maillog { file("/var/log/maillog"); };
destination lpd-errs { file("/var/log/lpd-errs"); };
destination cron { file("/var/log/cron"); };
destination slip { file("/var/log/slip.log"); };
destination ppp { file("/var/log/ppp.log"); };
destination allusers { usertty("*"); };

In addition to these destinations, you'll also want to specify one for remote logging to another syslog-ng process. This can be done with a line similar to this:

destination loghost { tcp("192.168.0.2" port(5140)); };

The port number can be any available TCP port.

Defining the filters is straightforward. You can simply create one for each syslog facility and log level, or you can create them according to those used in your syslog.conf. If you do the latter, you will only have to specify one filter in each log statement, but it will still take some work to create your filters.

Here are example filters for the syslog facilities:

filter f_auth { facility(auth); };
filter f_authpriv { facility(authpriv); };
filter f_console { facility(console); };
filter f_cron { facility(cron); };
filter f_daemon { facility(daemon); };
filter f_ftp { facility(ftp); };
filter f_kern { facility(kern); };
filter f_lpr { facility(lpr); };
filter f_mail { facility(mail); };
filter f_news { facility(news); };
filter f_security { facility(security); };
filter f_user { facility(user); };
filter f_uucp { facility(uucp); };

and examples for the log levels:

filter f_emerg { level(emerg); };
filter f_alert { level(alert..emerg); };
filter f_crit { level(crit..emerg); };
filter f_err { level(err..emerg); };
filter f_warning { level(warning..emerg); };
filter f_notice { level(notice..emerg); };
filter f_info { level(info..emerg); };
filter f_debug { level(debug..emerg); };

Now you can combine the source with the proper filter and destination within the log entries:

# *.err;kern.debug;auth.notice;mail.crit                /dev/console
log { source(src); filter(f_err); destination(console); };
log { source(src); filter(f_kern); filter(f_debug); destination(console); };
log { source(src); filter(f_auth); filter(f_notice); destination(console); };
log { source(src); filter(f_mail); filter(f_crit); destination(console); };

# *.notice;kern.debug;lpr.info;mail.crit;news.err       /var/log/messages
log { source(src); filter(f_notice); destination(messages); };
log { source(src); filter(f_kern); filter(f_debug); destination(messages); };
log { source(src); filter(f_lpr); filter(f_info); destination(messages); };
log { source(src); filter(f_mail); filter(f_crit); destination(messages); };
log { source(src); filter(f_news); filter(f_err); destination(messages); };

# security.*                                            /var/log/security
log { source(src); filter(f_security); destination(security); };

# auth.info;authpriv.info                         /var/log/auth.log
log { source(src); filter(f_auth); filter(f_info); destination(authlog); };
log { source(src); filter(f_authpriv); filter(f_info); destination(authlog); };

# mail.info                                             /var/log/maillog
log { source(src); filter(f_mail); filter(f_info); destination(maillog); };

# lpr.info                                              /var/log/lpd-errs
log { source(src); filter(f_lpr); filter(f_info); destination(lpd-errs); };

# cron.*                                                /var/log/cron
log { source(src); filter(f_cron); destination(cron); };

# *.emerg                                               *
log { source(src); filter(f_emerg); destination(allusers); };

You can set up the machine that will be receiving the logs in much the same way as if you were replacing the currently used syslogd.

To configure syslog-ng to receive messages from a remote host, you must specify a source entry:

source r_src { tcp(ip("192.168.0.2") port(5140)); };

Alternatively, you can dump all the logs from the remote machines into the same destinations that you use for your local log entries. This is not really recommended, because it can be a nightmare to manage, but can be done by including multiple source drivers in the source entry that you use for your local logs:

source src { 
    unix-dgram("/var/run/log"); 
    tcp(ip("192.168.0.2") port(5140));
    internal( );
};

Now logs gathered from remote hosts will appear in any of the destinations that were combined with this source.

If you would like all logs from remote hosts to go into a separate file named for each host in /var/log, you could use a destination like this:

destination r_all { file("/var/log/$HOST"); };

syslog-ng will expand the $HOST macro to the hostname of the system sending it logs and create a file named after it in /var/log. An appropriate log entry to use with this would be:

log { source(r_src); destination(r_all); };

However, an even better method is to recreate all of the remote syslog-ng log files on your central log server. For instance, a destination for a remote machine's messages file would look like this:

destination fbsd_messages { file("/var/log/$HOST/messages"); };

Notice here that the $HOST macro is used in place of a directory name. If you are using a destination entry like this, be sure to create the directory beforehand, or use the create_dirs() option:

options { create_dirs(yes); };

syslog-ng's macros are a very powerful feature. For instance, if you wanted to separate logs by hostname and day, you could use a destination like this:

destination fbsd_messages {
    file("/var/log/$HOST/$YEAR.$MONTH.$DAY/messages"); 
};

You can combine the remote source with the appropriate destinations for the logs coming in from the network just as you did when configuring syslog-ng for local logging-j e with the proper destination and filters.

Another neat thing you can do with syslog-ng is collect logs from a number of remote hosts and then send all of those to yet another syslog-ng daemon. You can do this by combining a remote source and a remote destination with a log entry:

log { source(r_src); destination(loghost); };

Since syslog-ng is now using TCP ports, you can use any encrypting tunnel you like to secure the traffic between your syslog-ng daemons. You can use SSH port forwarding or stunnel to create an encrypted channel between each of your servers. By limiting connections on the listening port to include only localhost (using firewall rules, as in or ), you can eliminate the possibility of bogus log entries or denial-of-service attacks.

Server logs are among the most critical information that a system administrator needs to do her job. Using new tools and strong encryption, you can keep your valuable log data safe from prying eyes.

Recommended Links

Central Loghost Mini-HOWTO

Annotated Papers

ITS Unix Security Overview Solaris Logging

A. How does it work?

Syslogd is the daemon which does the actual logging. The configuration file for syslogd is /etc/syslog.conf. Basically, syslogd puts time-stamped messages into log files as designated by syslog.conf. The log files can then be viewed to construct a history of the events occurring on your system. Syslogd is started along with the other startup scripts in /etc/rc2.d.

Everybody should run syslogd.

B. Why use logs?

Log files can often provide early warning of an impending attack. Logs can also notify you of problems before your good backups are cycled out. Use logs to help keep track of what is happening on your system at all times. Syslogd records and dates things such as login attempts and failures, system errors, possible security problems, etc. It also alerts you to problems which you may otherwise not notice on a daily basis, such as a partition becoming full. In sum, logging will save you a lot of time and pain if it is done correctly.

C. /etc/syslog.conf

In syslog.conf, there are two fields, the selector and the action. The selector field tells what to log and the action field tells where to log it (i.e., to a certain file, or to a certain host on the network).

The selector has the form: facility.level

The action has the form of a path, e.g., /var/adm/messages or /var/log/syslog, or console, or a full host name preceded by an @ sign: @loghostname.caltech.edu

The facility and level have many possible combinations. But for security's sake, it is far easier to just log everything.

Hint: Send logging to one (or a few) specified central hosts to make things easier to manage in large domains.

D. Creating a loghost

Creating a loghost for your cluster of machines can be invaluable in the event of a security breach. If you are part of a cluster, make sure your machine logs to another loghost as well as itself. If someone breaks in and erases your log files, and you have only been logging locally, you will have no chance of figuring out where they came from or what they did. Intruders commonly use scripts to erase their presence from the log files. By sending your logs to another more secure machine, you have a better chance of at least tracking the intruder's activities.

Security precautions to take:

• Make sure the time is correct on your system. Otherwise you will have trouble tracing problems and breakins.
• System logs are generally kept in the /var partition, mainly /var/adm. Make sure that /var is large enough to hold much more than the basic log file. This is to prevent accidental overflows, which could potentially erase important logging info.
• The default syslog.conf does not do a very good job of logging. Try changing the entry for /var/adm/messages to:

*.info /var/adm/messages

If you start getting too much of a certain message (say from sendmail), you can always bump that particular facility down by doing

*.info;mail.none /var/adm/messages

• To enable more logging of logins, create the "loginlog":

touch var/adm/loginlog

chmod 600 /var/adm/loginlog

chgrp sys /var/adm/loginlog

• Many other security programs such as ssh and tcpwrappers will log to syslogd. It is important that syslogd is always running on your system.

Additional security precautions to take:

• In order to create a loghost, pick one machine and secure it as much as possible. Basically, don't run anything on this machine besides syslogd. Turn off inetd, sendmail, everything, but make sure you have basic networking up. Possibly don't even run ssh on this machine. That way, the only way to view the log files on the loghost would be to physically log into the console.
• Make sure the time is always correct on the loghost.
• In order to allow the loghost to receive syslog messages from other machines, you may need to enable it to receive remote logging. (Find out first by reading the syslogd man pages). Do this by adding the –r command line upon syslogd startup. Edit /etc/rc2.d/S74syslog, and find the line:

/usr/sbin/syslogd

and change it to:

/usr/sbin/syslogd -r

• Next, tell all of your clients to log to this machine in addition to their local files. This can by done by adding a line which designates the action as the loghost, with an "@" sign in front of it to the bottom of /etc/syslog.conf:

*.info @logboy.caltech.edu

Once you restart syslogd, the client will send all of its log messages to logboy.caltech.edu, which will also then log the messages to its local files.

Be sure that the client is able to resolve logboy.caltech.edu at all times. A suggestion is to add a special entry in /etc/hosts so that logging is not interrupted if there is a resolving problem.

• Again, make sure the loghost is as secure as possible: the only thing it should be running is syslogd.

BigAdmin Feature Article Creating a Centralized Secure Log Server with syslog-ng and Stunnel

UNIX system administrators are quite familiar with the syslog daemon, but information that it collects often remains unprocessed unless someone reports a problem. At any site with more than a handful of machines, no one devotes the time to log in and check multiple log files each day or even each month. Automated scripts that might correlate data across these machines are difficult to write because they must access each machine individually. To lessen the burden of both automatic and manual data processing, many sites implement a central log server that collects data for all machines (preferably running NTP to make time/date correlation easier) on a network, including UNIX servers, Windows and Mac desktops, and even networking equipment like routers and switches. Centralized logging is fairly trivial with most stock UNIX syslog daemons, but syslogd is little changed from early incarnations and has some shortcomings.

The standard UNIX syslog daemon transports messages in the clear over UDP, meaning that anyone can sniff potentially sensitive data. The facility.level model is also fairly restrictive, and the default /etc/syslog.conf files supplied with most operating systems neglect to log many messages administrators might be interested in. Parsing out the logs in a way that makes sense for human readability or automated data mining is non-trivial. The UNIX syslog daemon doesn't easily allow one to split log files out by host or match log messages by regular expression, for example. Therefore, most centralized log servers using the stock syslog daemon wind up with monolithic log files that are only processed after the syslogd processes close them.

As a result, most sites that centralize logging also wind up replacing the stock syslog daemon with something more secure and more flexible such as Metalog, msyslog, or something similar. One very popular syslog replacement is an open source program called syslog-ng. An organization can run syslog-ng on each UNIX host or just on the syslog server itself. If syslog-ng is run only on the log host, clients send data over UDP port 514 as usual, but better log organization and manipulation can be accomplished on the server.

The benefit of running syslog-ng on each UNIX host is the ability to encrypt the logging channel with IPSec or the utility Stunnel so that data is not readable by the casual sniffer. When combined with Stunnel as a transport mechanism, an organization has a secure way of centralizing log messages from all desired UNIX hosts for further processing. In the case of syslog-ng, Stunnel works by accepting log connections on a local port, wrapping them in SSL sessions, and then redirecting them to a secure port on the remote log host. The stunnel process on the remote log host then decrypts the SSL session and hands the information back to the syslog server on the standard port. Once on the log server, organization and parsing of the log files then takes place using the flexibility of syslog-ng.

Below I cover the installation and configuration of syslog-ng and Stunnel on machines running the Solaris 8 Operating System (SPARC Platform Edition), but the procedure also generally applies to older and newer versions of the Solaris OS on both SPARC and x86 platforms. Each of the reference machines discussed below comes installed with OpenSSL, tcp wrappers, the Solaris 8 /dev/urandom patch, and the GNU development environment (gcc, and so on) and several other freeware packages. The machine serving as the log server has also been thoroughly hardened since it will store sensitive and security-related information from all machines on the network. These reference machines all reside on the subnet 192.168.1, and the IP address of the log server is 192.168.1.10.

Security issues in network event logging (draft-syslog-bcp)

[Chapter 10] 10.5 The UNIX System Log (syslog) Facility

Enterprise Linux Magazine

LJ 11 Linux System Administration How to Log Friends and Influence People by Mark Komarinski

syslog programming

Using syslog as a Generic Event Transport Mechanism Kevin Steves

Securing Your Host -- syslog.conf

Syslog is a flexible logging facility that allows you to:

• Log to a file
• Log to another host
• Mail a user
• Send a notification to a terminal/console

Syslog is event driven. Each event is comprised of two parts, a facility and a level. The notation is "facility.level". Normal operating facilities are:
user, kern, mail, daemon, auth, lpr, news, uucp, cron, mark

Levels are:
emerg, alert, crit, err, warning, notice, info, debug, none

A syslog event might look like "mail.debug" or "auth.notice". Any and all combinations are valid. As a historical note, syslog at one time only logged via a loglevel, defined as local0-7. These days not many systems use this type of logging facility. If you are one of the few, please see your systems man page.

Now ever entry in /etc/syslog.conf follows the same format:
event log_location

Note: inbetween the event and the log location is a tab, not a space

In many cases, the log location will be a filename. "syslogd", the syslog daemon, will append to the file all appropriate messages. The file must exist prior to logging, so if you are setting up a new filename make sure you "touch" the file before putting it in your configuration file.

Sample entries for a syslog file are:

• mail.debug /var/adm/maillog
  This would log all messages of type mail at the level debug to /var/adm/maillog. What you end up finding there is a log of all received and sent mail when you use this setting.
• auth.notice /var/adm/authlog
  All authentication attempts will be logged to /var/adm/authlog
• *.emerg /var/adm/syslog
  All events of type "emergency" will be logged to /var/adm/syslog
• auth.notice /dev/console
  Put out a message to the console every time "/usr/bin/su" is executed.

One nice capability of the syslog daemon is it's ability to log to a loghost. A loghost is a place you want messages sent to. You can either duplicate important messages (such as the filesystem is full) that you write out to your normal file, or have all messages just go there.

Loghost is defined by default as the local host in /etc/hosts. You can change it to whatever machine you want to log, and then simple put in /etc/syslog.conf
event loghost

The second way to define a loghost is simply to specify it in the syslog.conf file by changing the log location to be @loghost.foo.bar.
An example is:
# event @myloghost.foo.bar
mail.debug @myloghost.foo.bar

Remember, myloghost.foo.bar must be accepting syslog notices in order to log messages generated from your machine.

Last, it is recommended that you create the file /var/adm/loginlog. If this file exists, all failed login attempts will be logged there in cleartext. This is a very useful, if undocumented, feature.

SurLink Global Networking, Inc. - FreeBSD SECURITY HOW-TO

syslogd

Many, if not most, serious applications use a facility known as syslog to write and distribute messages. The actual logging of these messages can occur on the local system, a remote system, be sent to specific files, be sent to the system operator, or sent to other users. In most of the following applications I describe in this paper you will see excerpts from the system log created by these applications.

syslog allows you to consolidate your consoles just as PROP does under VM. At NSW TAB we have two AIX-based firewalls which produce copious amounts of log messages tracking the access and usage of our Internet connection. Initially, each of these systems created their own logs. With syslog these messages could be consolidated on, at first, one of the firewalls; and, following the port of syslogd on to our VM/ESA system.

The ability of syslog to route messages of various types then allows us to distribute specific message types to people responsible for a given application/function. The message traffic can also be directed to the VM system operator for integration with PROP. Although we haven't implemented it, it would be possible for an action routine to be invoked by PROP on the receipt of specific messages which would initiate a process to control the remote system.

It is also possible to provide the reverse of this process. That is, VM messages could be sent to a remote *NIX system for consolidation and action. This would involve providing PROP (or the NetView *NCCF service) with routines which used the syslog service.

The following is the /etc/syslog.conf file from our VM/ESA system:

Figure 1. Local syslog Configuration File

# Critical messages go to VM System Operator
*.crit				*SYSOPR
# Dump debug messages to console
*.debug				CON
# News notices go to VM Users MAINT and NEALE
news.notice			MAINT,NEALE
# Syslog debug messages can be logged to file
syslog.debug			/home/neale/SYSLOGD/syslog.log
# Non-syslog debug messages can be logged to file
*.debug				/home/neale/SYSLOGD/debug.log
# News and Mail debug messages can be logged to own file
news.debug,mail.debug	/home/neale/SYSLOGD/newsmail.log

Figure 2. Remote syslog Configuration File

*.debug         /var/adm/messages
*.debug         @vma.tabnsw.com.au
*.crit          @vma.tabnsw.com.au

Man Pages

docs.sun.com man pages section 1M System Administration Commands

/usr/sbin/syslogd [-d] [-f configfile] [-m markinterval] [-p path] [-t| -T]


syslogd reads and forwards system messages to the appropriate log files or users, depending upon the priority of a message and the system facility from which it originates. The configuration file /etc/syslog.conf (see syslog.conf(4)) controls where messages are forwarded. syslogd logs a mark (timestamp) message every markinterval minutes (default 20) at priority LOG_INFO to the facility whose name is given as mark in the syslog.conf file.

A system message consists of a single line of text, which may be prefixed with a priority code number enclosed in angle-brackets (< >); priorities are defined in <sys/syslog.h>.

syslogd reads from the STREAMS log driver, /dev/log, and from any transport provider specified in /etc/netconfig, /etc/net/transport/hosts, and /etc/net/transport/services.

syslogd reads the configuration file when it starts up, and again whenever it receives a HUP signal (see signal(3HEAD), at which time it also closes all files it has open, re-reads its configuration file, and then opens only the log files that are listed in that file. syslogd exits when it receives a TERM signal.

As it starts up, syslogd creates the file /etc/syslog.pid, if possible, containing its process identifier (PID).

If message ID generation is enabled (see log(7D)), each message will be preceded by an identifier in the following format: [ID msgid facility.priority]. msgid is the message's numeric identifier described in msgid(1M). facility and priority are described in syslog.conf(4). [ID 123456 kern.notice] is an example of an identifier when message ID generation is enabled.

If the message originated in a loadable kernel module or driver, the kernel module's name (for example, ufs) will be displayed instead of unix. See EXAMPLES for sample output from syslogd with and without message ID generation enabled.

In an effort to reduce visual clutter, message IDs are not displayed when writing to the console; message IDs are only written to the log file. See EXAMPLES.

The /etc/default/syslogd file contains the following default parameter settings. See FILES.

LOG_FROM_REMOTE

Specifies whether remote messages are logged. LOG_FROM_REMOTE=NO is equivalent to the -t command-line option. The default value for LOG_FROM_REMOTE is YES.

OPTIONS

The following options are supported:

-d

Turn on debugging. This option should only be used interactively in a root shell once the system is in multi-user mode. It should not be used in the system start-up scripts, as this will cause the system to hang at the point where syslogd is started.

-f configfile

Specify an alternate configuration file.

-m markinterval

Specify an interval, in minutes, between mark messages.

-p path

Specify an alternative log device name. The default is /dev/log.

-T

Enable the syslogd UDP port to turn on logging of remote messages. This is the default behavior. See FILES.

-t

Disable the syslogd UDP port to turn off logging of remote messages. See FILES.

Linux.com - Man Reference - syslog.conf

-usr-share-man-cat5-syslog.conf(0)

CHPTR014.HTM -- nice buttons ;-)

System Administration

Troubleshooting

[Sunhelp] Syslog.conf problems

> *.emerg;*.crit;*.alert;*.err                    @sysa
> 
> Where sysa is a valid hostname in the host table. did a kill -HUP on the 
> sylog pid.

what's in /etc/nsswitch.conf for "hosts"?  You can use "getent hosts sysa"
to see where the system really things "sysa" is.  

Also, the spaces between *.err and @sysa have to be tabs.

NEW QUESTION & SUMMARY syslog.conf

The new question is for the below message how can I determine what is the
name of the facility and level ?
*.debug logs this message, daemon.debug does not. What I would like to do
is to filter below messages to a seperate file.

>logmsg: pri 24, flags 2, from beta, msg Jun 25 14:38:41
>in.telnetd[23112]: refused connect from xxx.yyy.tr
>Logging to UNUSED

SUMMARY:

My original question was:

>
>I have following line in my /etc/syslog.conf
>*	/var/adm/test
>daemon.* /var/adm/test
>user.* /var/adm/test
>
>And when I do syslogd -d I track the message:
>
>logmsg: pri 24, flags 2, from beta, msg Jun 25 14:38:41
>in.telnetd[23112]: refused connect from xxx.yyy.tr
>Logging to UNUSED
>
>to UNUSED, I suppose this means it is discarded.
>Ofcourse /var/adm/test is not even created.
>What can I do to make it Logging to /var/adm/test ?

Summary of answers is as below:

You must create the file beforehand (touch //var/adm/test)
and make sure you use TABS, not spaces.


The wildcard entry is *.debug, not *.

Thanks to all who replied.

-cagri

mail Summary - configuring syslog.conf
To all of the people who responded to my questions, many thanks..
(There were just too many responses to thank everyone individually).

Overall, the suggestions were similar.

Don't use spaces, use tabs when configuring syslog.conf.
After making changes, kill -HUP pid for syslog.conf.

The message below is from Kai O'Yang who was one of may who forwarded
their syslog.conf files to share.

I am now receiving auth.notice messages from a remote system to my
loghost(on both the console and authlog file.

The only real problem I have that I haven't been able to resolve with
this is that the name of the remote host is not showing up. Instead,
I am receiving "???" in its place, and garbage on the device name:

Oct 23 14:44:32 ??? su:'su root' succeeded for mconroy on /dev/pts/3^m

I am sure it is configured correctly in dns. So I am at a lost.

Any thought???

Thanks again for everyone's help.

Mark Conroy


First add a loghost alias in /etc/hosts or nis table for the syslog
host. Here's my syslog.conf for all client machines.

#ident "@(#)syslog.conf 1.3 93/12/09 SMI" /* SunOS 5.0
*/ #
# Copyright (c) 1991-1993, by Sun Microsystems, Inc. #
# syslog configuration file.
#
# This file is processed by m4 so be careful to quote (`') names #
that match m4 reserved words. Also, within ifdef's, arguments #
containing commas must be quoted.
#
# Note: Have to exclude user from most lines so that user.alert #
and user.emerg are not included, because old sendmails # will
generate them for debugging information. If you
# have no 4.2BSD based systems doing network logging, you #
can remove all the special cases for "user" logging.
#
*.err;kern.notice;auth.notice;user.none /dev/console
*.err;kern.debug;daemon.notice;mail.crit;user.none @loghost

*.alert;kern.err;daemon.err;user.none operator,@loghost
*.alert;user.none root,@loghost

*.emerg;user.none @loghost auth.info
@loghost
mail.info @loghost daemon.info
@loghost

For the loghost:
#ident "@(#)syslog.conf 1.3 93/12/09 SMI" /* SunOS 5.0
*/ #
# Copyright (c) 1991-1993, by Sun Microsystems, Inc. #
# syslog configuration file.
#
# This file is processed by m4 so be careful to quote (`') names #
that match m4 reserved words. Also, within ifdef's, arguments #
containing commas must be quoted.
#
# Note: Have to exclude user from most lines so that user.alert #
and user.emerg are not included, because old sendmails # will
generate them for debugging information. If you
# have no 4.2BSD based systems doing network logging, you #
can remove all the special cases for "user" logging.
#
*.err;kern.notice;auth.notice;user.none /dev/console
*.err;kern.debug;daemon.notice;mail.crit;user.none
/var/adm/messages

*.alert;kern.err;daemon.err;user.none operator
*.alert;user.none root

*.emerg;user.none * auth.info
/var/log/authlog mail.info
/var/log/mlog
#
# Adding log to daemon information
#
daemon.info /var/log/syslog

Configuration Examples

syslog.conf.html

syslog.conf-server

syslog.conf-client

aixfaq

1.115: Where are the AIX log files kept?
From: [email protected] (Dirk Coldewey)

Such as the equivalent of the SunOS file '/var/log/syslog' and
'/var/adm/messages'.

If you want regular unix syslog files, you can configure syslogd
in /etc/syslog.conf. Here's the one that I use:

#
# See the supplied /etc/syslog.conf file for copious comments.
#
*.err;kern.debug;auth.notice;user.none /dev/console
*.err;kern.debug;daemon,auth.notice;mail.crit;user.none /var/adm/messages
lpr.debug /var/adm/lpd-errs

*.alert;kern.err;daemon.err;user.none operator
*.alert;user.none root

*.emerg;user.none *

# for loghost machines, to have authentication messages (su, login, etc.)
# logged to a file, un-comment out the following line and adjust the
# file name as appropriate.
#
# if a non-loghost machine chooses to have such messages
# sent to the loghost machine, un-comment out the following line.
#
auth.notice /var/log/authlog

mail.debug /var/log/syslog

# following line for compatibility with old sendmails. they will send
# messages with no facility code, which will be turned into "user" messages
# by the local syslog daemon. only the "loghost" machine needs the following
# line, to cause these old sendmail log messages to be logged in the
# mail syslog file.
#
user.alert /var/log/syslog
#
# non-loghost machines will use the following lines to cause "user"
# log messages to be logged locally.
#
user.err /dev/console
user.err /var/adm/messages
user.alert `root, operator'
user.emerg *

FOCUS on Linux Securing Linux

-- begin sample syslog.conf --

*.=info;*.=notice;*.warn;mail.none              /usr/adm/messages
*.=debug                                        /usr/adm/debug
*.warn;*.crit;*.emerg                           /usr/adm/syslog
*.warn;*.crit;*.emerg                           /usr/adm/messages

auth.*                                          /usr/adm/log.auth
cron.*                                          /usr/adm/log.cron
daemon.*                                        /usr/adm/log.daemon
kern.*                                          /usr/adm/log.kern
mail.*                                          /usr/adm/log.mail
mark.*                                          /usr/adm/log.mark
syslog.*                                        /usr/adm/log.syslog
user.*                                          /usr/adm/log.user

-- end sample syslog.conf --

Central Loghost Mini-HOWTO

This page is simply a collection of open source tools you can use to glue together your own centralized (syslog) loghost. Included are example configuration settings so that you can configure your loghost in a manner similar to mine.

There is very little that you need to read and understand in order to use these tools. Also, these tools are widely used and therefore easy to get help with on internet mailing lists.

I established a centralized location for syslog collection in order to facilitate:

1. Log reporting
   • real time alerting
   • periodic (several times per day) summary reporting
2. Log storage
   • long term archival for possible later analysis

Tools used:

• UNIX hosts (Linux and Solaris)
• Modified logcheck script(s).
• Syslog-NG
• Swatch though I'm slowly moving to SEC, this page will be updated once I've completely switched
• Stunnel

Recommended Links

Softpanorama Recommended

Top articles

Sites

Centralized Syslog Server Using syslog-NG with web Interface using php-syslog-ng

Central Loghost Mini-HOWTO

Monitoring - Automated Log Monitoring with LogSentry and a Central ...

Creating a central syslog server

Central syslog server | Syslog-ng

Implementation of Central Logging Server using syslog-ng

Cool Solutions- Centralized Syslogging with Syslog-NG on SUSE Linux

Install Centralized Syslog Server with Web Interface - PlanetIT

BigAdmin Feature Article- Creating a Centralized Secure Log Server ...

Centralizing Syslog on Solaris - The Blog of Ben Rockwood

How many of you do centralized syslog-ing- - Web Hosting Talk

[ubuntu] Need Assistance Setting Up A Central Syslog Server ...

BigAdmin Feature Article Creating a Centralized Secure Log Server with syslog-ng and Stunnel

• Installing Stunnel
• Creating Certificate Files for syslog-ng Over Stunnel
• Configuring Stunnel for Use With syslog-ng
• Installing syslog-ng
• Configuring syslog-ng
• Resources

UNIX system administrators are quite familiar with the syslog daemon, but information that it collects often remains unprocessed unless someone reports a problem. At any site with more than a handful of machines, no one devotes the time to log in and check multiple log files each day or even each month. Automated scripts that might correlate data across these machines are difficult to write because they must access each machine individually. To lessen the burden of both automatic and manual data processing, many sites implement a central log server that collects data for all machines (preferably running NTP to make time/date correlation easier) on a network, including UNIX servers, Windows and Mac desktops, and even networking equipment like routers and switches. Centralized logging is fairly trivial with most stock UNIX syslog daemons, but syslogd is little changed from early incarnations and has some shortcomings.

"The syslog daemon is a very versatile tool that should never be overlooked under any circumstances. The facility itself provides a wealth of information regarding the local system that it monitors.

"However, what happens when the system it's monitoring gets compromised?

"When a system becomes compromised, and the intruder obtains elevated root privileges, he now has the ability, as well as the will, to trash any and all evidence leading up to the intrusion, on top of erasing anything else thereafter, including other key system files.

"That's where remote system logging comes in, and it's real super-easy to set up..."

There may be more elaborate third party proprietary solutions for log consolidation, but the syslog capability within UNIX is simple and ubiquitous on UNIX platforms. This paper will specifically deal with the Sun Solaris environment when not noted otherwise.

Central Loghost Mini-HOWTO

Device Specific syslog Configurations -- central log facility

[1] "File Formats - syslog.conf(4)." 22 January 1997. URL:
http://www.bama.ua.edu/cgi-bin/man-cgi?syslog.conf+4 (23 December 2000).

[2] Internet Engineering Task Force. "Security Issues in Network Event Logging (syslog)." 23 October 2000. URL: http://www.ietf.org/html.charters/syslog-charter.html (23 December 2000).

[3] Carnegie Mellon University. "Manage logging and other data collection mechanisms." 18 October 2000. URL: http://www.cert.org/security-improvement/practices/p092.html (23 December 2000).

[4] Orebaugh, Angela. "Securing Solaris." October 2, 2000. URL: http://www.sans.org/infosecFAQ/sec_solaris.htm (23 December 2000).

[5] Bezroukov, Nikolai. "Softpanorama University Pages: Solaris Hardening and Security." URL: http://www.softpanorama.org/Security/sos.shtml (23 December 2000).

[6] "Maintenance Commands - syslogd(1M)." 27 February 1997. URL: http://www.bama.ua.edu/cgi-bin/man-cgi?syslogd+1M (23 December 2000).

[7] "Tucows Linux Man Pages syslogd.8." 12 October 1998. URL: http://howto.tucows.com/man/man8/syslogd.8.html (23 December 2000).

[8] Carnegie Mellon University. "Configure firewall logging and alert mechanisms." CERT Security Improvement Modules. 2 August 1999. URL:
http://www.cert.org/security-improvement/practices/p059.html (23 December 2000).

My question was:

~~~~> I would like to have some deep understanding about syslog.
~~~~> We intend to use to capture important system messages
~~~~> vide our network consisting of around 250 to 300 sun
~~~~> workstations running 2.3 to 2.5.1 OSs. We want to
~~~~> configure loghost and capture. But the real problem is we
~~~~> are not sure which are notices, which are alerts, which
~~~~> are real critical messages, etc., If some one could throw
~~~~> some light it is highly greatful.

Thanx to: Alexander N. Spitzer([email protected])

He has asked me to go thru:

~~~~man syslog.conf
~~~~man syslogd
~~~~man syslog

Thanx to: "Karl E. Vogel" <[email protected]>

He wrote:

~~ I had problems with the default syslog setup. Below is the syslog.conf
~~ file we use on our production system. Critical messages go to
~~ /var/adm/messages, other stuff goes to /var/log/syslog.

He has included a sample syslog.conf which is very useful ,
But I need still further info capturable from machines.

Thanx to: Michael Hill <[email protected]>

Michael has given me the following info which was very useful.

I wrote an article in the December 1996 issue of SysAdmin magazine,called "Understanding syslog.conf". If you can find that issue, I think my article would help. It included a program I wrote for displaying exactly what syslog will do with every message, and the files are available via FTP from the magazine's Web site at http://www.samag.com

Random Findings

passlogd - passive syslog capture daemon

passlogd is a purpose-built sniffer for capturing syslog messages in transit. this allows for backup logging to be performed on a machine with no open ports. useful if your log server is compromised or you don't want to sift through a terabyte of sniffer logs. passlogd is currently under development, so check back here for bugfixes, new features, etc. if you find bugs, please let me know, and by all means, send me a patch.

NOTE: the latest version (0.1e) contains important security fixes to the parser.

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March, 12, 2019