Log rotation

In modern Unix/Linux distributions  log rotation utility is now supplied with basic OS configuration:

• logrotate -- standard utility for rotation of logs in linux
• logadm -- System Log Rotation Utility (Solaris 9 and later)

They are highly configurable and can be extended to arbitrary log files, not only log files in /var

Basic log rotation is not that difficult to implement in Perl.  There is a dozen or so ready-made scripts, but the problem is that each of them has its own set of preconditions and approaches to rotation so you might be better off writing your own.

There are two approaches, two Unix schools of log rotation: "version based" and "date based.":

• Version based school rotates the log in such a way that the last one has suffix 1, the previous suffix 2, etc. Each time log rotate N files are preserved, numbered from 1 to N.
• Date based school uses daily, weekly of month rotation pattern and each rotated log is marked by timestamp. Log for a certain period are preserved (let's say 6 months; older logs are deleted. This is the approach that I prefer.)

Most Linux distributions have log rotation preconfigured. For example, RHEL/CENTOS and Suse use a separate logrotate utility.

In Unixes such as AIX you need to provide your own tools. It make sense to use logrotation program written in Perl as it is more understandable and modifiable then alternatives for most sysadmins. There is the basic log rotation program in Perl for apache (please note apache has the ability to rotate logs via piping them into special logrotation program without restart):

#!/usr/bin/perl
#
# daily rotate apache logs
# Nikolai Bezroukov, Dec 12, 2008
#
$apache_base="/usr/local/apache2";
$apache_logs="$apache_base/logs";
$apache_bin="$apache_base/bin";
$limit=($ARGG[0])? $ARGG[0]:15; # number of day for logs to preserve

`find $apache_logs -mtime +$limit -exec /usr/bin/rm {} \;`;

#
# Step 1. Obtain the current date and create that timestamp 

$timestamp=`date "+%Y%m%d"`;

#
# Step 2: rename the files
#
chdir("$apache_logs") || die "cannot change to $apache_logs directory";
for $file ("access_log","error_log") {
     if (-e $file) {
        `mv $file $file\.$timestamp`; 
     } 
} # for

#
# Step 3: restart apache
#
`$apache_bin/apachectl restart`;

NEWS CONTENTS

• 191003 : rotatelog a program for log file management ( rotatelog a program for log file management, )
• 191003 : Rotating Linux Log Files - Part 2 logrotate MDLog-sysadmin ( Rotating Linux Log Files - Part 2 logrotate MDLog-sysadmin, )
• 191003 : Perl newsyslog ( Perl newsyslog, )
• 191003 : Perl-Logrotate ( Perl-Logrotate, )
• 191003 : Lumberjack ( Lumberjack, )

Old News ;-)

rotatelog a program for log file management

Well-documented Perl log rotation script

rotatelog - rotates files when they reach a certain trigger size.

rotatelog [-cdht]

This is a program to rotate log files. It can rotate any file listed in its configuration file. The file does not necessarily need to be log file. This program imitates the logrotate and newsyslog programs found on Linux and FreeBSD respectively. I originally wrote this to make life easier in Solaris.

---

OPTIONS

-c

Specify the location of the configuration file. Multiple configuration files can be separated by commas with no spaces.

-d

Turn on debug output. This is verbose output that shows most of what rotatelog does.

-h

Print the help information.

-t

Turn on the test flag. This causes rotatelog to run as if all files need rotation, but disallows the actual rotation. This is most useful in combination with the -d flag.

---

ERRORS

This program will exit if it cannot open its configuration file(s) and if any of the system calls fail. This program will also exit if the there are any syntax errors in the actions section. Any other configuration file errors are ignored and a warning is issued. All error messages are printed to STDERR. This program is meant to be run as a cron job so error messages such as these will be mailed to the owner of the cron job. Many times rotatelog is run in the root crontab.

---

FILES

The rotatelog program depends upon its configuration file for information concerning what files to rotate and when to rotate them. The rotatelog.conf file is located in the /usr/local/etc directory by default. This location is set in the configuration section of the source code along with the paths to certain system utility functions. It is possible to specify a new configuration file via the -c command line option, and it is also possible to specify multiple configuration files in this manner as well using the form -c file1,file2,file3. Note that configuration files that appear later in a multiple configuration specification can override settings in previous configuration files. When installing this program it will be necessary to check this section of the source code. Along with specifying the the files to rotate there is also the ability to specify ownership and mode of the files after rotation, the number of rotated files to keep, the compression used or not used on the rotated files, and an auxiliary action in the form of a shell command to take place when certain files are rotated. The rotatelog program is normally run as root. It is important that the config file is only writable by root due to the ability to execute shell commands!

There are three sections in the configuration file. They can appear in any order. Each section is identified by a keyword. Configuration information follows the keyword and extends until the next keyword. Keywords can come in any order and repeat, but it would be best to follow a simple format such as:

 FILES:

 # File  Trigger  Owner:Group  Mode  Compress  Archive_Limit
 # =========================================================

 /var/log/wtmp      1M   root:wheel  664  none  4
 /var/log/wtmpx     4M   root:wheel  644  none  4
 /var/log/messages  74K  root:wheel  664  Z     5
 /var/log/maillog   60K  root:wheel  664  gz    7

 ACTIONS:

 # Shell Command :file1,file2,file3 ... fileN
 # ==========================================

 rotate1                             : /var/log/wtmp,/var/log/wtmpx
 kill -HUP `cat /var/run/syslog.pid` : /var/log/messages,/var/log/maillog

 NOTIFY:

 # Person to email rotation notification
 # =====================================

 root@localhost

Comment lines must contain a pound sign at the start of the line. These comment lines are ignored along with blank lines and lines containing only whitespace. In general there can be no leading whitespace on configuration lines while there can be any amount of whitespace between elements.

---

Files

The files section specifies which files are to be rotated, when they are to be rotated, what their ownership and modes are to be post rotation, how many rotated copies are to be retained, and the compression used on the files once rotated. The beginning of the files section is noted by the occurrence of the FILES: keyword.

This information must adhere to the following syntax:

 File  Trigger  Owner:Group  Mode  Compress  Archive_Limit

For example:

 /var/log/wtmp      1M   root:wheel  664  none  4
 /var/log/wtmpx     4M   root:wheel  644  none  4
 /var/log/messages  74K  root:wheel  664  Z     5
 /var/log/maillog   60K  root:wheel  664  gz    7

The fields above can be separated by any amount of whitespace. The first field is the full path to the file that requires rotation. The second field is the trigger. The trigger specifies when rotation is to occur. The trigger is the maximum size of a file in bytes, kilobytes, or megabytes. Bytes can be specified with B or b, kilobytes with K or k, or megabytes can be specified with M or m. If a file listed is found to be larger than the trigger it is rotated. The owner:group field specifies who is to own the file once it is rotated. The mode is the file permission mode in octal notation of the rotated file as specified in the chmod(1) manual page. The mode does not include the optional first digit of the four digit chmod(1) octal mode. The mode is only the simple three digit format shown above. The compression field specifies the type of compression to use on the rotated file. A file can be compressed with gzip (gz), compress (Z), or not at all (none). The archive limit field specifies how many copies of a rotated file to keep. This number is the highest count in the rotation scheme. The rotation count starts at 0 so this is actually one higher then the total number of files kept. If the archive limit were set to 4 for the file /var/log/wtmp the maximum number of archived wtmp files would be 5. The files would appear in the /var/log/ directory as:

 wtmp    <-- the current wtmp file.
 wtmp.0  <-- the first archive of wtmp.
 .
 .
 .
 wtmp.4  <-- the last archive of wtmp.

If the last file of an archive exists, wtmp.4 in the above example, it is removed during the rotation and the previous archive file wtmp.3 is renamed accordingly. All other files are rotated in the same manner until the current file is reached and rotated.

---

Actions

The actions section specifies if any actions are to occur for any of the rotated files. The actions section is denoted by the ACTIONS: keyword. Sometimes a process must be notified that it must close its file descriptors and open new ones due to something like a file rotation. Many processes will handle this action when the receive a SIGHUP signal. Most of the time a kill -HUP on the process will accomplish this. One example is syslogd. We can see that syslogd has certain files open all of the time by using lsof to inquire about the status of one of its log files:

 [rowland@darkstar rowland]$ sudo lsof /var/log/messages
 COMMAND PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
 syslogd  99 root    8w  VREG 4,131076     3119  232 /var/log/messages

It is not enough to just move the current file into the archive and touch a new one for writing. The syslogd process must be told that a new file exists and that it is to close the current file descriptor and open a new one on the same file name (not to keep writing to the file pointed to by the current file descriptor). This is what the actions section does. The actions section has the following syntax:

 Shell Command : file1, file2, file3, ... fileN
 - or -
 Shell Command : rotate1, rotate2, ... rotateN

For example:

 rotate1                             : /var/log/wtwp, /var/log/wtmpx
 - or -
 kill -HUP `cat /var/run/syslog.pid` : rotate1
 - or -
 kill -HUP `cat /var/run/syslog.pid` : /var/log/messages,/var/log/maillog

You are free to add whitespace around the : and , characters. This extra whitespace will be ignored. When the /var/log/messages file is to be rotated it is first moved to a new name, but not compressed. The syslogd program will still be writing to this file even though the name has changed because its file descriptor points to this location on the filesystem and the file descriptor is still open at this point. A new file is touched and its permissions and ownership are set to the values specified in the files section. Then the SIGHUP signal is the sent to syslogd, and it closes the rotated file and opens the new file that replaces it. After this is done it is safe to compress the old file if compression was specified in the files section.

There is also a special action called rotate which allows you to bind multiple files together for rotation. All of the files must be present in the files section so that rotatelog knows how to rotate them. When one file is rotated then the other files bound with it are rotated immediately. The format of rotate is:

 rotateN : file1, file2, ... fileN

The rotate command must be followed by an integer. This allows for more than one binding but only for different groups of files. You may not leave the integer off. The wtmp file is a good candidate for this type of rotation. When the wtmp file is rotated it is a good idea to also rotate the wtmpx file. A binding of:

 rotate1 : /var/adm/wtmp, /var/adm/wtmpx

will rotate both files whenever one of them needs rotation. When binding files for rotation in this manner, there is only one way to associate another action (a normal shell command) to the rotation of the bound files. Using the rotate action indicates that all files are to be rotated at the same time. If an action is also to occur, we must be certain that all of the files have begun rotation before executing that action. An action on the group of bound files in the rotate group is specified by using that rotate action as the bound file for the normal shell command action. The syslogd example is perfect for this. In the syslogd case one would probably want to send syslogd a SIGHUP signal once all syslogd files have been rotated. This is how that is accomplished:

 rotate2                             : /var/log/messages,/var/log/maillog
 kill -HUP `cat /var/run/syslog.pid` : rotate2

This ensures the following steps occur during file rotation:

 1.  Begin rotation on a file.
          - Figure out the archive count of the current file.
          - Rotate old logs up one count, removing the last if necessary.
          - Rotate the current file up one count.
          - Touch a new version of the file.

 2.  Perform any actions on the file.
          - Check rotateN bound files.
                 [if file is in a rotateN group]
                 - Begin any file rotation on the other files in the
                   rotateN bound group.  All other files are put
                   through step #1 at this point.
                 - Perform any normal shell command action associated
                   with this group of rotateN bound files.
                 - Finish rotation of all other files bound in the
                   rotateN group.  This is step #3.
                 - Return.  This causes the original file that triggered
                   this recursion to finish its rotation.
          - Check normal shell command actions.
                 [in this case the file was not in a rotateN group]
                 - Perform the shell command action.
                 - Return.  This moves the file into step #3.

 3.  Finish rotating a file.
          - Compress the previous log file if the file has been specified
            for compression in the files section.
          - Add the file to the list of files which have been rotated.

This process ensures that any files bound together are in the beginning of file rotation before any shell commands are executed as a result of this rotation. Once the shell command action occurs, if there is one, all of the files finish their rotation. Due to this flexibility in file rotation actions, the following rules apply to the actions section:

 1.  A file may appear in only one rotateN group.

 2.  A file may appear in only one normal shell command group.

 3.  A file may not appear in both a rotateN group and a normal shell command action.
 4.  A normal shell command may be bound explicitly to a file that is
     not in a rotateN group.

 5.  A normal shell command cannot be bound to a file in a rotateN
     group explicitly.  To bind a shell command to files in a rotateN
     group one must implicitly bind the command to the rotateN action.

If there are no actions to perform when any of the files are rotated this section may be omitted from the configuration file. If you rotate log files written to by syslogd then you will most certainly require one of the examples above with the appropriate path to your syslogd.pid file. If you leave out this step and rotate syslogd files, syslogd will most certainly NOT be your friend. If you choose to bind syslogd files together in a rotateN group, all of the files will be rotated when one file is rotated. If you want to define the action for each file, only when that file is rotated, do not use the rotateN form of the action. It all depends on what you wish to do.

---

Notify

The notify section specifies who is to receive notification in the event that any files are rotated. The notify section is started with the NOTIFY: keyword. The notification is sent out via email to the full email address specified in this section. This is the simplest section. The email address is simply listed as in the following example:

 # Person to receive email rotation notification
 # =============================================

 root@localhost

If there is to be no notification this section may be omitted or a value of ``none'' may be used. There can only be one email address in this section.

---

RESTRICTIONS

There can be no leading whitespace on config file lines. Someday I may fix that. The files must be specified with their full paths in all sections. This is true in the actions section where you can list an action for multiple files. If there are any syntax errors in the configuration file those lines will be ignored and a warning will be printed to STDERR. If this program is run as a cron job this will result in an email message to the owner of the cron job. In most cases this will be root.

---

AUTHOR

This piece of code was written by Shaun Rowland ([email protected]) mostly during the early hours of the morning. In my world that is considered ``day'' while afternoon is considered ``night''. Go figure.

Copyright 1999, 2000, 2001 Interhack Corporation. All rights reserved.

---

HISTORY

$Log: rotatelog.html,v $ Revision 1.2 2003/07/23 15:31:57 cmcurtin committing new content Revision 1.7 2001/05/27 18:21:21 rowland Added hostname information.

Revision 1.6 2001/05/23 15:10:48 rowland Removed history stuff that killed pod2latex.

Revision 1.5 2001/05/23 15:04:47 rowland Fixed a bug where extra whitespace at the end of an ACTIONS line would cause the file eq check file at the end of do action() to fail. Doh! Now we are good to go.

Revision 1.4 2001/05/19 21:23:32 rowland Updated the following in logrotate 1.3:

        * Added the -c and -h command line options.
        * Added the B and b trigger size specification.
        * Changed the release Makefile to create installation directories if
          they do not already exist.

Revision 1.3 2001/04/22 23:01:36 rowland Updated perldoc and config file to better reflect reality in rotating syslogd files. You wouldn not normally put these files in a rotateN group and then define an action for that group because this would cause all of the files to be rotated when just one of them is rotated. You could do this if you wanted, but it makes a perfect example. Better examples will follow later, but the examples here should better reflect reality.

Revision 1.2 2001/04/22 22:45:25 rowland Added more actions section error checking and cleaned up the default settings.

Revision 1.1.1.1 2001/04/22 21:58:26 rowland The rotatelog program was designed to rotate files and perform actions on those files once rotated if desired. This program began its life as logrotate. I changed the name so it would not be confused with the GNU logrotate program. This version includes improved code for handling rotateN bound files and actions on those bound files (in other words it now works properly).

Rotating Linux Log Files - Part 2 logrotate MDLog-sysadmin

• Rotating Linux Log Files - Part 1: syslog
• Rotating Linux Log Files
• Rotate Apache logs using Awstats
• Linux Distributions Trends
• Keep your Debian system up-to-date

Perl newsyslog

newsyslog is a faithful Perl rewrite of the MIT newsyslog utility, with a number of features taken from the FreeBSD and NetBSD variants of newsyslog.

It archives log files based on size, date or interval, and can optionally compress archives with gzip or bzip2. Complete documentation is available via "perldoc newsyslog.pl".

Perl-Logrotate

Perl-Logrotate is a safe log rotation script. It is fully configurable and comes with compression. You can define the number of files to keep rotated and when to rotate. If logfiles cannot be rotated, it will put the log contents back into the file to avoid data loss. It also works with all logging software, as it only truncates the file and does not unlink it. There is no need to signal any processes after this has been run.

Lumberjack (Console/Administration)

Log rotation software, perl replacement for newsyslog
Jun 08th 1999, 08:48 stable: none - devel: none - license: Public Domain

[July 16, 1999]Lumberjack Log rotation software, perl replacement for newsyslog
Jun 08th 1999, 08:48 stable: none - devel: none license: Public Domain. Lumberjack is a perl replacement for the BSD newsyslog. Logs are rotated and renamed to logfile.YYMMDD, and the daemon/signal are configurable. Contrary to the URL for the file, this program should work on all systems, not just NetBSD. http://kludge.psc.edu/~ksulliva/netbsd/lumberjack.man.html

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019