Softpanorama

Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
May the source be with you, but remember the KISS principle ;-)
Bigger doesn't imply better. Bigger often is a sign of obesity, of lost control, of overcomplexity, of cancerous cells

Rsyslog Configuration

News  Remote Syslog Recommended Links Syslog Messages Classification Loghost server and remote syslog Pipes in syslog
Configuration examples Syslog configuration debugging Syslog Multitail logger utility Syslog analyzers Logwatch
Log rotation logrotate Log Rotation in Solaris Using Logadm How to rotate wtmpx in solaris Syslog-ng Rsyslog Configuration
Solaris System Logs AIX syslog Syslog for Windows Troubleshooting SFU 3.5 syslog Syslog Internals
syslog spoofing Logs Security & Integrity  Horror Stories Tips Humor  Etc
Top Visited
Switchboard
Latest
Past week
Past month

Note of using rsyslog's new or old config style (from Rainer's Blog ):

I got a very interesting question on the rsyslog support forums, and I thought I share it, together with the answer, here at a more prominent spot:

After over a decade of using stock bsd syslog, I finally have a need to do some more complicated processing of logs (splitting off Postgres query logs from general Postgres logs), and after looking at other options (basically syslog-ng), I think rsyslog looks like a better fit. I'm mainly in it so I can use regex matching, but thinks like the log queueing and being able to easily move to db storage in the future look good.
Since I'm new, I'd considered that I might get a jump on things by sticking with the newest config syntax. But after doing some googling for examples and looking at the examples in the rsyslog wiki, it seems like maybe the newest syntax might be a bit too new for a beginner - I learn best by example.

Are there any serious downsides to NOT going with the most current syntax?

The answer is that the old syntax is still fully supported by the versions and will probably remain for quite some while (except for some very few exceptions, which we couldn't carry over for good reasons - this is documented in the compatibility docs on the web site). Some parts of it are considered so important that they most probably never will go away. Actually, if you want to do simple things, the old syntax has its advantages. The more complex your processing gets, the more you benefit from the new syntax. But you can mix and match new and old style in almost all cases.

So my suggestion would be to get started using the old syntax and as soon as you begin to do more complex things, you can switch over to the new style. That's actually the way it is designed ;) A good indicator of when it would be benefitial to move to new style is when you begin to use a lot of directives beginning with $, especially if they modify an action. Also, if you move to action queues, I would strongly suggest to use new style. It is far more intuitive an less error-prone.

To provide a bit more background information, there is an important non-technical reason why the classical syntax is remain for a long time: basic syslog.conf format is extremely well known, covered in a lot of text books, taught in numerous courses and used in a myriad of Internet tutorials. So if we would abandon it, we would thrash a lot of people's knowledge and help resources. In short: we would make it much harder for folks that it would actually need to be. This has never been rsyslog philosophy. Providing the ability to changed gradually and with growing needs is a core goal.

 


NEWS CONTENTS

Old News ;-)

[Oct 21, 2017] Install a Centralized Log Server with Rsyslog in Debian 9

Oct 21, 2017 | www.howtoforge.com

The Rsyslog daemon is automatically installed in most Linux distributions. However, if Rsyslog is not installed on your system you can issue one of the below commands in order to install the service> you will require root privileges to run the commands.

In Debian based distros:

sudo apt-get install rsyslog

In RHEL based distros like CentOS:

sudo yum install rsyslog

In order to verify if Rsyslog daemon is started on a system execute the below commands, depending on your distribution version.

On newer Linux distros with systemd:

systemctl status rsyslog.service

On older Linux versions with init:

service rsyslog status

/etc/init.d/rsyslog status

In order to start the rsyslog daemon issue the following command.

On older Linux versions with init:

service rsyslog start

/etc/init.d/rsyslog start

On latest Linux distros:

systemctl start rsyslog.service

To setup an rsyslog program to run in server mode, edit the main configuration file in /etc/rsyslog.conf. In this file make the following changes as shown in the below sample.

sudo vi /etc/rsyslog.conf

Locate and uncomment by removing the hashtag (#) the following lines in order to allow UDP log message reception on 514 port. By default, the UDP port is used by syslog to send-receive messages.

$ModLoad imudp 
$UDPServerRun 514

Because the UDP protocol is not reliable to exchange data over a network, you can setup Rsyslog to output log messages to a remote server via TCP protocol. To enable TCP reception protocol, open /etc/rsyslog.conf file and uncomment the following lines as shown below. This will allow the rsyslog daemon to bind and listen on a TCP socket on port 514.

$ModLoad imtcp 
$InputTCPServerRun 514

Both protocols can be enabled in rsyslog to run at the same time.

If you want to specify to which senders you permit access to rsyslog daemon, add the following line after the enabled protocol lines:

$AllowedSender TCP, 127.0.0.1, 10.110.50.0/24, *.yourdomain.com

You will also need to create a new template that will be parsed by rsyslog daemon before receiving the incoming logs. The template should instruct the local Rsyslog server where to store the incoming log messages. Define the template right after the $AllowedSender line as shown in the below sample.

$template Incoming-logs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log" 
*.*  ?Incoming-logs
& ~

To log only the messages generated by kern facility use the below syntax.

kern.*   ?Incoming-logs

The received logs are parsed by the above template and will be stored in the local file system in /var/log/ directory, in files named after the client hostname client facility that produced the messages: %HOSTNAME% and %PROGRAMNAME% variables.

The below & ~ redirect rule configures the Rsyslog daemon to save the incoming log messages only to the above files specified by the variables names. Otherwise, the received logs will be further processed and also stored in the content of local logs, such as /var/log/syslog file.

To add a rule to discard all related log messages to mail, you can use the following statement.

mail.* ~

Other variables that can be used to output file names are: %syslogseverity%, %syslogfacility%, %timegenerated%, %HOSTNAME%, %syslogtag%, %msg%, %FROMHOST-IP%, %PRI%, %MSGID%, %APP-NAME%, %TIMESTAMP%, %$year%, %$month%, %$day%

Starting with Rsyslog version 7, a new configuration format can be used to declare a template in an Rsyslog server.

A version 7 template sample can look like shown in the below lines.

template(name="MyTemplate" type="string"
         string="/var/log/%FROMHOST-IP%/%PROGRAMNAME:::secpath-replace%.log"
        )

Another mode you can write the above template can also be as shown below:

template(name="MyTemplate" type="list") {
    constant(value="/var/log/")
    property(name="fromhost-ip")
    constant(value="/")
    property(name="programname" SecurePath="replace")
    constant(value=".log")
    }

In order to apply any changes made to rsyslog configuration file, you must restart the daemon to load the new configuration.

sudo service rsyslog restart

sudo systemctl restart rsyslog

To check which rsyslog sockets in listening state are opened on a Debian Linux system, you can execute the netstat command with root privileges. Pass the results via a filter utility, such as grep

sudo netstat –tulpn | grep rsyslog

Be aware that you must also open Rsyslog ports in firewall in order to allow incoming connections to be established.

In RHEL based distros with Firewalld activated issue the below commands:

firewall-cmd --permanent --add-port=514/tcp

firewall-cmd --permanent --add-port=514/tcp

firewall-cmd –reload

In Debian based distros with UFW firewall active issue the below commands:

ufw allow 514/tcp

ufw allow 514/udp

Iptables firewall rules:

iptables -A INPUT -p tcp -m tcp --dport 514 -j ACCEPT

iptables -A INPUT -p udp --dport 514 -j ACCEPT

Configure Rsyslog as a Client

To enable rsyslog daemon to run in client mode and output local log messages to a remote Rsyslog server, edit /etc/rsyslog.conf file and add one of the following lines:

*. * @IP_REMOTE_RSYSLOG_SERVER:514

*. * @FQDN_RSYSLOG_SERVER:514

This line enables the Rsyslog service to output all internal logs to a distant Rsyslog server on UDP port 514.

To send the logs over TCP protocol use the following template:

*. *  @@IP_reomte_syslog_server:514

To output only cron related logs with all priorities to a rsyslog server, use the below template:

cron.* @ IP_reomte_syslog_server:514

In cases when the Rsyslog server is not reachable via network, append the below lines to /etc/rsyslog.conf file on the client side in order temporarily store the logs in a disk buffered file, until the server comes online.

$ActionQueueFileName queue
$ActionQueueMaxDiskSpace 1g
$ActionQueueSaveOnShutdown on
$ActionQueueType LinkedList
$ActionResumeRetryCount -1

To apply the above rules, Rsyslog daemon needs to be restarted in order to act as a client.

how to use rsyslog's ruleset and call statements

Rsyslog 7.2+ introduced a couple of cool config enhancements, among them a new way to specify rulesets and to call into a ruleset (a much better replacement for omruleset). Unfortunatley, the doc is currently extremely sparse. That will change soon, but in the mean time I thought I provide at least some clues here via the blog.

In essence, the ruleset statement permits to specify a ruleset. It's written as

ruleset(name="rulesetname") { statements here }

Rulesets can be bound to inputs, as usual, and any ruleset can call into another ruleset via the call statement:

call rulesetname

Note that in call, the rulesetname is just plainly specified. We created it that way as we thought this looks most intuitively to people used to any kind of scripting or programming language.

A somewhat larger sample (bascially taken from the rsyslog mailing list, thanks to Brian Knox) is:


module(load="imptcp" keepalive="on")
# use imptcp just as example for bind ruleset below
ruleset(name="rs1") {

*.* /var/log/test1.log
}
ruleset(name="rs2") {
*.* /var/log/test2.log
call rs1
}
input(type="imptcp" port="13514" ruleset="rs2")

All statements NOT specified inside a ruleset become part of the default ruleset. Legacy-style $Ruleset statements are still supported, but cannot be used inside a ruleset() statement (that would create a big mess and is totally unnecessary).

How can I check the config rsyslog

We have often seen the case, that someone has rsyslog running and makes changes to the configuration. And usually, after making the changes, rsyslog gets restarted, but the changed config is invalid. rsyslog has a function to check the configuration for validity. This can be done very easily by invoking this command:
rsyslogd -N1

(Note that rsyslogd may not be in your search path – then it usually is found in /sbin/rsyslogd)

This tells rsyslog to do a config check. It does NOT run in regular mode, but just check configuration file correctness. This option is meant to verify a config file. To do so, run rsyslogd interactively in foreground, specifying -f <config-file> and -N level. The level argument modifies behaviour. Currently, 0 is the same as not specifying the -N option at all (so this makes limited sense) and 1 actually activates the code.

This configuration check will only check the configuration for integrity like syntax. Additionaly, the modules will be loaded to make sure that they work properly. On the downside, since the engine will not be loaded, errors with permissions or alike cannot be checked. These will occur only when running rsyslog normally.

The verdict for this option is, that it is quite useful for a first check if the changes were correct, without running the configuration in live mode. This might help to prevent that rsyslog gets restarted with a basically wrong configuration and thus rendering rsyslog useless, because it might not work or not work properly.

Very simple config -- starting point for modifications - rsyslog wiki

I struggled a bit to figure out where to start with rsyslogd. I wanted to find a complete conf file that I could edit, but everything I found was either really complex or did not include the original syslog information. So here it is... I bolded everything that I made changes in from either the standards in the docs or suggestions.

# -- Loading modules

$ModLoad immark
$ModLoad imudp
$ModLoad imtcp
$ModLoad imuxsock
$ModLoad imklog

# since I am using a uniprocessor pc, I put this in.

$OptimizeForUniprocessor on

# I also wanted to be able to receive syslog traffic

$UDPServerAddress 0.0.0.0
$UDPServerRun 514

# and reduce any duplicates

$RepeatedMsgReduction on
$RepeatedMsgContainsOrigionalMsg on

# this is for Windows events from SNARE

$EscapeControlCharactersOnReceive off

# A basic template mostly from the docs, but I wanted to know what system forwarded the messages so I added some text. Also I added the ":::space" to handle the windows events (based on the other suggestions in this wiki)

$template SyslFormat,"%timegenerated% [WJCG]-%HOSTNAME% %syslogtag%%msg:::space$

# these are right from the default syslog.conf file, adding the ;SyslFormat template at the end

kern.debug /var/adm/syslog.dated/kern.log;SyslFormat
user.debug /var/adm/syslog.dated/user.log;SyslFormat
daemon.debug /var/adm/syslog.dated/daemon.log;SyslFormat
auth.crit;syslog.debug /var/adm/syslog.dated/syslog.log;SyslFormat
mail,lpr.debug /var/adm/syslog/misc.log;SyslFormat
kern.debug /var/adm/messages;SyslFormat
kern.debug /dev/console;SyslFormat
*.emerg *

#this will forward all the logs to another server using TCP port 2010.

*.* @@1.2.3.4:2010;SyslFormat

BSD-Style blocks will go away in rsyslog v7

September 11, 2012 |

Rsyslog supports BSD-style blocks since ages. They were a pretty handy tool to group actions together that should act only on remote hosts or log messages from specific programs. However, the v7 config system with its full nesting capabilities provides a much better - and easy to use - way to specify this. If both systems are mixed, the problem is that BSD-style blocks can be used to violate the nesting structure (examples below). Also, support for them adds a lot to rule engine code complexity. And finally, they are also very seldom used, few users even know they exist.

As a result, I have decided to drop support for BSD-style blocks in rsyslog v7 and above. A poll on the mailing list a week ago did not make anybody speak up against that change. So I assume none is hurt. This is especially the case as the conversion of BSD-style blocks to nested config is a very easy one.

Let's look at this example:

!prog1

*.* /var/log/prog1.log

*.* /var/log/prog1again.log

!prog2

*.* /var/log/prog2.log

*.* /var/log/prog2again.log

This code can very simply be replaced by:

if $programname == 'prog1' then {

/var/log/prog1.log

/var/log/prog1again.log

}

if $programname == 'prog2' then {

/var/log/prog2.log

/var/log/prog2again.log

}


And if you prefer the more powerful action statments (probably not so much benefit for this use case...), you can write:

if $programname == 'prog1' then {

action(type="omfile" file="/var/log/prog1.log")

action(type="omfile" file="/var/log/prog1again.log")

}

if $programname == 'prog2' then {

action(type="omfile" file="/var/log/prog2.log")

action(type="omfile" file="/var/log/prog2again.log")

}

I expect that usually these easy cases happen. HOWEVER, if I had kept support for BSD-style blocks, one could configure things like this:

if $msg contains 'test' then {

action(type="omfile" file="/var/log/somefile")

!prog2

mail.* :mmjsonparse:

&action(type="omfile" file="/var/log/somefile2")

!prog3

&~

!prog4

if $msg contains 'test2' then

/var/log/logfile

else

/var/log/logfile2

}


Can you make out the actual nesting structure of this config? When, for example, programname needs to be "prog3" and what happens then? IMHO this combination can cause considerable user confusion and frustration. As such, I made a sharp cut and removed it.


My apologies for those that need to do the manual conversion. I am sure the time is well-invested in the long term.

Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Sites

Top articles

Sites

Filtering by program name - rsyslog wiki

Jump to: navigation, search

Filtering by program name allows you for instance to segregate log messages by their originators (or maybe to silence a particularly verbose program)

Filtering by program name using the BSD selector syntax

This method doesn't fully work with current versions of rsyslogd. (Included here so you can avoid the trap!) While the BSD selector !programname works, its 'disabling' counterpart !* doesn't! (Noted here as 'NOT YET IMPLEMENTED': [1]). If you use !*, the filtering is not reset and nothing ever matches again until a new program filter is enabled.

If you happen to use this BSD syntax !programname / !* syntax in a rsyslog.d/*.conf file, the result is that you just disabled a the interesting part of the main rsyslog.conf without touching it.

Until it's fixed, you will have to use the 'expression based' syntax instead of the BSD syntax.

Filtering by program name using the expression-based syntax

This is how I filtered popa3d output so that everything higher than info ends up in /var/log/popa3d.log and doesn't appear anywhere else:

# First, I redirect everything that matches the program name 'popa3d' and of priority higher than info (6) into the log file I want

if $programname == 'popa3d' and $syslogseverity <= '6' then /var/log/popa3d.log

# Then I use the same redirect but with ~ as the action, causing the log line not to go into other filters

if $programname == 'popa3d' and $syslogseverity <= '6' then ~

NB: I did put those two lines in the file /etc/rsyslog.d/popa3d.conf, but they can be put in the main rsyslog.conf if you do not use the rsyslog.d method

# If the second part appears after the first filter, you can simply write:

& ~



Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2018 by Dr. Nikolai Bezroukov. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) in the author free time and without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: December, 26, 2017