Rsyslog Configuration

Note of using rsyslog's new or old config style (from Rainer's Blog ):

I got a very interesting question on the rsyslog support forums, and I thought I share it, together with the answer, here at a more prominent spot:

After over a decade of using stock bsd syslog, I finally have a need to do some more complicated processing of logs (splitting off Postgres query logs from general Postgres logs), and after looking at other options (basically syslog-ng), I think rsyslog looks like a better fit. I'm mainly in it so I can use regex matching, but thinks like the log queueing and being able to easily move to db storage in the future look good.
Since I'm new, I'd considered that I might get a jump on things by sticking with the newest config syntax. But after doing some googling for examples and looking at the examples in the rsyslog wiki, it seems like maybe the newest syntax might be a bit too new for a beginner - I learn best by example.

Are there any serious downsides to NOT going with the most current syntax?

The answer is that the old syntax is still fully supported by the versions and will probably remain for quite some while (except for some very few exceptions, which we couldn't carry over for good reasons - this is documented in the compatibility docs on the web site). Some parts of it are considered so important that they most probably never will go away. Actually, if you want to do simple things, the old syntax has its advantages. The more complex your processing gets, the more you benefit from the new syntax. But you can mix and match new and old style in almost all cases.

So my suggestion would be to get started using the old syntax and as soon as you begin to do more complex things, you can switch over to the new style. That's actually the way it is designed ;) A good indicator of when it would be benefitial to move to new style is when you begin to use a lot of directives beginning with $, especially if they modify an action. Also, if you move to action queues, I would strongly suggest to use new style. It is far more intuitive an less error-prone.

To provide a bit more background information, there is an important non-technical reason why the classical syntax is remain for a long time: basic syslog.conf format is extremely well known, covered in a lot of text books, taught in numerous courses and used in a myriad of Internet tutorials. So if we would abandon it, we would thrash a lot of people's knowledge and help resources. In short: we would make it much harder for folks that it would actually need to be. This has never been rsyslog philosophy. Providing the ability to changed gradually and with growing needs is a core goal.

NEWS CONTENTS

• how to use rsyslogs ruleset and call statements
• How can I check the config rsyslog
• Very simple config -- starting point for modifications - rsyslog wiki
• BSD-Style blocks will go away in rsyslog v7

Old News ;-)

how to use rsyslog's ruleset and call statements

Rsyslog 7.2+ introduced a couple of cool config enhancements, among them a new way to specify rulesets and to call into a ruleset (a much better replacement for omruleset). Unfortunatley, the doc is currently extremely sparse. That will change soon, but in the mean time I thought I provide at least some clues here via the blog.

In essence, the ruleset statement permits to specify a ruleset. It's written as

ruleset(name="rulesetname") { statements here }

Rulesets can be bound to inputs, as usual, and any ruleset can call into another ruleset via the call statement:

call rulesetname

Note that in call, the rulesetname is just plainly specified. We created it that way as we thought this looks most intuitively to people used to any kind of scripting or programming language.

A somewhat larger sample (bascially taken from the rsyslog mailing list, thanks to Brian Knox) is:

module(load="imptcp" keepalive="on")
# use imptcp just as example for bind ruleset below
ruleset(name="rs1") {
*.* /var/log/test1.log
}
ruleset(name="rs2") {
*.* /var/log/test2.log
call rs1
}
input(type="imptcp" port="13514" ruleset="rs2")

All statements NOT specified inside a ruleset become part of the default ruleset. Legacy-style $Ruleset statements are still supported, but cannot be used inside a ruleset() statement (that would create a big mess and is totally unnecessary).

How can I check the config rsyslog

We have often seen the case, that someone has rsyslog running and makes changes to the configuration. And usually, after making the changes, rsyslog gets restarted, but the changed config is invalid. rsyslog has a function to check the configuration for validity. This can be done very easily by invoking this command:

rsyslogd -N1

(Note that rsyslogd may not be in your search path – then it usually is found in /sbin/rsyslogd)

This tells rsyslog to do a config check. It does NOT run in regular mode, but just check configuration file correctness. This option is meant to verify a config file. To do so, run rsyslogd interactively in foreground, specifying -f <config-file> and -N level. The level argument modifies behaviour. Currently, 0 is the same as not specifying the -N option at all (so this makes limited sense) and 1 actually activates the code.

This configuration check will only check the configuration for integrity like syntax. Additionaly, the modules will be loaded to make sure that they work properly. On the downside, since the engine will not be loaded, errors with permissions or alike cannot be checked. These will occur only when running rsyslog normally.

The verdict for this option is, that it is quite useful for a first check if the changes were correct, without running the configuration in live mode. This might help to prevent that rsyslog gets restarted with a basically wrong configuration and thus rendering rsyslog useless, because it might not work or not work properly.

Very simple config -- starting point for modifications - rsyslog wiki

I struggled a bit to figure out where to start with rsyslogd. I wanted to find a complete conf file that I could edit, but everything I found was either really complex or did not include the original syslog information. So here it is... I bolded everything that I made changes in from either the standards in the docs or suggestions.

# -- Loading modules

$ModLoad immark
$ModLoad imudp
$ModLoad imtcp
$ModLoad imuxsock
$ModLoad imklog

# since I am using a uniprocessor pc, I put this in.

$OptimizeForUniprocessor on

# I also wanted to be able to receive syslog traffic

$UDPServerAddress 0.0.0.0
$UDPServerRun 514

# and reduce any duplicates

$RepeatedMsgReduction on
$RepeatedMsgContainsOrigionalMsg on

# this is for Windows events from SNARE

$EscapeControlCharactersOnReceive off

# A basic template mostly from the docs, but I wanted to know what system forwarded the messages so I added some text. Also I added the ":::space" to handle the windows events (based on the other suggestions in this wiki)

$template SyslFormat,"%timegenerated% [WJCG]-%HOSTNAME% %syslogtag%%msg:::space$

# these are right from the default syslog.conf file, adding the ;SyslFormat template at the end

kern.debug /var/adm/syslog.dated/kern.log;SyslFormat
user.debug /var/adm/syslog.dated/user.log;SyslFormat
daemon.debug /var/adm/syslog.dated/daemon.log;SyslFormat
auth.crit;syslog.debug /var/adm/syslog.dated/syslog.log;SyslFormat
mail,lpr.debug /var/adm/syslog/misc.log;SyslFormat
kern.debug /var/adm/messages;SyslFormat
kern.debug /dev/console;SyslFormat
*.emerg *

#this will forward all the logs to another server using TCP port 2010.

*.* @@1.2.3.4:2010;SyslFormat

BSD-Style blocks will go away in rsyslog v7

September 11, 2012 |

Rsyslog supports BSD-style blocks since ages. They were a pretty handy tool to group actions together that should act only on remote hosts or log messages from specific programs. However, the v7 config system with its full nesting capabilities provides a much better - and easy to use - way to specify this. If both systems are mixed, the problem is that BSD-style blocks can be used to violate the nesting structure (examples below). Also, support for them adds a lot to rule engine code complexity. And finally, they are also very seldom used, few users even know they exist.

As a result, I have decided to drop support for BSD-style blocks in rsyslog v7 and above. A poll on the mailing list a week ago did not make anybody speak up against that change. So I assume none is hurt. This is especially the case as the conversion of BSD-style blocks to nested config is a very easy one.

Let's look at this example:

!prog1

*.* /var/log/prog1.log

*.* /var/log/prog1again.log

!prog2

*.* /var/log/prog2.log

*.* /var/log/prog2again.log

This code can very simply be replaced by:

if $programname == 'prog1' then {

/var/log/prog1.log

/var/log/prog1again.log

}

if $programname == 'prog2' then {

/var/log/prog2.log

/var/log/prog2again.log

}

And if you prefer the more powerful action statments (probably not so much benefit for this use case...), you can write:

if $programname == 'prog1' then {

action(type="omfile" file="/var/log/prog1.log")

action(type="omfile" file="/var/log/prog1again.log")

}

if $programname == 'prog2' then {

action(type="omfile" file="/var/log/prog2.log")

action(type="omfile" file="/var/log/prog2again.log")

}

I expect that usually these easy cases happen. HOWEVER, if I had kept support for BSD-style blocks, one could configure things like this:

if $msg contains 'test' then {

action(type="omfile" file="/var/log/somefile")

!prog2

mail.* :mmjsonparse:

&action(type="omfile" file="/var/log/somefile2")

!prog3

&~

!prog4

if $msg contains 'test2' then

/var/log/logfile

else

/var/log/logfile2

}

Can you make out the actual nesting structure of this config? When, for example, programname needs to be "prog3" and what happens then? IMHO this combination can cause considerable user confusion and frustration. As such, I made a sharp cut and removed it.

My apologies for those that need to do the manual conversion. I am sure the time is well-invested in the long term.

Recommended Links

Softpanorama Recommended

Filtering by program name - rsyslog wiki

Jump to: navigation, search

Filtering by program name allows you for instance to segregate log messages by their originators (or maybe to silence a particularly verbose program)

Filtering by program name using the BSD selector syntax

This method doesn't fully work with current versions of rsyslogd. (Included here so you can avoid the trap!) While the BSD selector !programname works, its 'disabling' counterpart !* doesn't! (Noted here as 'NOT YET IMPLEMENTED': [1]). If you use !*, the filtering is not reset and nothing ever matches again until a new program filter is enabled.

If you happen to use this BSD syntax !programname / !* syntax in a rsyslog.d/*.conf file, the result is that you just disabled a the interesting part of the main rsyslog.conf without touching it.

Until it's fixed, you will have to use the 'expression based' syntax instead of the BSD syntax.

Filtering by program name using the expression-based syntax

This is how I filtered popa3d output so that everything higher than info ends up in /var/log/popa3d.log and doesn't appear anywhere else:

# First, I redirect everything that matches the program name 'popa3d' and of priority higher than info (6) into the log file I want

if $programname == 'popa3d' and $syslogseverity <= '6' then /var/log/popa3d.log

# Then I use the same redirect but with ~ as the action, causing the log line not to go into other filters

if $programname == 'popa3d' and $syslogseverity <= '6' then ~

NB: I did put those two lines in the file /etc/rsyslog.d/popa3d.conf, but they can be put in the main rsyslog.conf if you do not use the rsyslog.d method

# If the second part appears after the first filter, you can simply write:

& ~

Recommended Links

Softpanorama Recommended

Basic Configuration

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: January, 27, 2015