Managing AIX logs

AIX use both the BSD syslogd daemon and AIX proprietary error logging facilities, with a number of function-specific log files to record error events on each node. Most AIX components report errors using the errdemon daemon, an idiosyncratic AIX solution which is totally different from the BSD syslog facility. For example AIX kernel does not use syslog for error logging. 

In AIX the syslogd daemon  is not activated by default. You need to perform several steps in order to be able to use it the way it is used on Solaris and Linux:

• Increase the size of  AIX Error Log facility  logs (as native AIX logging system it contains additional diagnostic information in error messages, which is useful for troubleshooting). 
• Create syslog.conf  manually and specify log rotation parameters (unique, but very useful feature of AIX syslogd daemon implementation)
• The syslogd daemon does not create files unless they already exist. Touch required to create log files. After then you need to restart the daemon or ask it to reread config file with kill -HUP
• Ensure forwarding of the messages from the AIX Error Log facility.
• Optionally integrate log processing into Tivoli (via TEC logfile Adapter) or some other ESM system.  You can also use Apache+PHP based open source log processing systems on AIX (see  Compiling Apache on AIX and Compiling PHP on AIX )

The AIX Error Log facility is configured by default to write files just one megabyte in size. When the log fills up, it wraps around and overwrites existing entries. Therefore, you should set the size to at least 40MB. You can do this once for all nodes with the dsh command after the nodes are installed.

dsh -a /usr/lib/errdemon -s 40960000

AIX syslogd implementation of SYSLOG has built-in log rotation capabilities that are specified with optional fields after the log destination:

facility-level destination rotate size s  files n time t  compress archive path

For example:

*.emerg;*.alert;*.crit;*.warning /var/adm/messages  rotate files 4 time 7d compress

This is probably the most clean solution of the problem in comparison with Linux or Solaris solutions. As you can see in addition to rotation you can also specify compression  When log reaches the specified size and/or age, the current log file will be renamed to something like name.0 and optionally compressed. All previous generations of the same log will have their extensions incremented and the oldest file(s) may be deleted.

Among parameters for the rotate keyword the most important are: 

• size s Size threshold: rotate the log when it is larger than this. s is followed by k or m for KB and MB, respectively.
• time t Time threshold: rotate the log when it is older than this. t is followed by h, d, w, m, or y for hours, days, weeks, months, or years, respectively.
• files n Keep at most n files.
• compress Compress old files.
• archive path Move older files to the specified location.

Strangely enough AIX does not provide any default syslog.conf  at all. But for syslogd activation it needs to exist. If you need standard entries you can borrow the file from Solaris. For example:

mail.debug                                              /var/adm/maillog
mail.none                                               /var/adm/maillog
auth.notice                                             /var/adm/authlog
lpr.debug                                               /var/adm/lpd-errs
kern.debug                                              /var/adm/messages
*.emerg;*.alert;*.crit;*.warning;*.err;*.notice;*.info  /var/adm/messages

Files referenced in syslog.conf need to exist, so you need to "touch" them.  After creating the files required,  you have to reload syslogd again before it will begin logging to the file. Like in Solaris,  you can use  `kill -HUP <syslog_pid>` to reload syslog.conf (there is no pkill command in AIX), but remember that syslogd doesn't complain about a missing file and doesn't create the missing file either. To avoid searching to PID you can use refresh -s syslogd (or stopsrc -s syslogd && startsrc -s syslogd)

Remember, that the syslog.conf file requires tabs as the field delimiter.  If you highlight a line with the mouse, then copy & paste a syslog.conf entry, the tabs can be  converted to spaces and you get a difficult to trace error.  Even more nasty for debugging are cases are when there is an  extra space before or after one of the fields.

The next step is to arrange forwarding error messages to syslog.  To do that you need to created a file called, say,  /root/etc/aix_errd2syslogd.add. This file should contain the following statements:

errnotify:
en_pid = 0
en_name = "syslog"
en_persistenceflg = 1
en_crcid = 0
en_method = "errpt -l $1 | tail -1 | logger -t errpt -p daemon.notice"

After that you need to issued the command

odmadd /root/etc/aix_errd2syslogd.add

The above AIX ODM stanza causes the error daemon to forward the most recent line of the error report to the syslog.

mail.debug   /var/log/mail  rotate size 500m files 4		# maintain 4 files, 500M each
user.debug   /var/log/user  rotate files 12 time 7d		# 12 files, daily rotate
kern.debug   /var/log/kern  rotate files 12 time 7d compress	# 12 files, monthly rotate, compress
...

When any modifications are made to /etc/syslog.conf, make sure to restart syslogd to reread the configuration.

Note: Administrators who get used to other systems need to remember that error messages are also written to the AIX Error Log which usually contains more information about probable cause and possible recovery or diagnostic actions (remember that in AIX, the primary logging facility is "AIX error log" and only code that was ported from 'other sources' contains the calls to syslog and logger.).

Using SMIT for log configuration

This is information from AIX documentation:

To access the Syslog SMIT menu, enter:

smit spsyslog

The format of a syslog log file is:

MMM DD HH:MM:SS node_name resource[pid]: msg

Where:

• MMM DD HH:MM:SS Timestamp (month day hour:minute:second)
• node_name The name of the node that the error occurred on
• resource The name of the failing system
• pid The optionally logged process ID of the failing resource.
• msg A free form error message.

Errors logged by SP components will contain in the message section the following additional information pertaining to the logging resource:

• LPP Licensed program name
• Fn filename
• SID SID_level_of_the_file
• L# Line number or function

The fastpath invocation for the Generate a Syslog Report menu is:

smit spsyslrpt

From the command line

To report to the local node all records logged by ftp for all syslog log files on nodes in the current system partition, enter:

psyslrpt -a -r ftp

To report all records that logged to files selected for the daemon and user facilities starting on March 3 and to report the records to the local node from nodes host1 and host2, enter:

psyslrpt -w host1,host2 -f user,daemon -s 03030000

For complete details on the psyslrpt command, refer to the PSSP: Command and Technical Reference.

If a syslog log file is not configured for rotation it will continue to grow until manually trimmed. The fastpath invocation for the Trim Syslog Log Files menu is:

smit spsyslclr

To trim all entries older than 30 days from all syslog log files specified in the syslog configuration file /etc/syslog.conf, on all nodes in the local system partition, enter:

psyslclr -a -y 30

To trim all records older than 30 days from any log file, for instance the /var/adm/SPlogs/SPdaemon.log file on the local node, use the -l option such as:

psyslclr -y 30 -l /var/adm/SPlogs/SPdaemon.log

To trim all records from all log files specified in an alternate syslog configuration file, for instance one named /u/local/syslog.conf, use the -g option such as:

psyslclr -g /u/local/syslog.conf -y 0

Note: The syslogd daemon is stopped during the trimming process, then restarted with either the default configuration file or an alternate file if specified with the -g option. For complete details see the psyslclr command in the book PSSP: Command and Technical Reference. The psyslclr command can be added as a crontab entry for performing scheduled syslog trimming. On the control workstation, the psyslclr command is used to trim daemon facility messages older than six days. This is done in the /usr/lpp/ssp/bin/cleanup.logs.ws script that is run from the crontabs file on the control workstation. (See Maintaining the crontabs file for more information.)

NEWS CONTENTS

• 20111028 : MyPurgeLogs ( freshmeat.net )
• 20111028 : Sys Admin The AIX Error Logging Facility by Sandor W. Sklar ( www.samag.com )
• 20111028 : Logging from syslog and errlog to the same file ( Logging from syslog and errlog to the same file, )
• 20050828 : convert errlog - comp.unix.aix Google Groups ( convert errlog - comp.unix.aix Google Groups, Aug 28, 2005 )
• 20020423 : how to redirect errpt to syslog [Archive] - dBforums ( how to redirect errpt to syslog [Archive] - dBforums, Apr 23, 2002 )
• 210210 : Rainer's Blog rsyslog on AIX who wants to help porting ( Rainer's Blog rsyslog on AIX who wants to help porting, )

Old News ;-)

[Oct 28, 2011] MyPurgeLogs

freshmeat.net

MyPurgeLogs is a script that can delete files older than a number of days and can rotate and compress other files. It simply reads a configuration file to know what it must do. Wildcards for directories and files are allowed. The compression's method is determined to optimize the free space on the filesystem. This script has only been tested on AIX.

Sys Admin The AIX Error Logging Facility by Sandor W. Sklar

www.samag.com

The AIX error logging facility components are part of the bos.rte and the bos.sysmgt.serv_aid packages, both of which are automatically placed on the system as part of the base operating system installation. Some of these components are shown in Table 1.

Unlike the syslog daemon, which performs no logging at all in its default configuration as shipped, the error logging facility requires no configuration before it can provide useful information about the system. The errdemon is started during system initialization and continuously monitors the special file /dev/error for new entries sent by either the kernel or by applications. The label of each new entry is checked against the contents of the Error Record Template Repository, and if a match is found, additional information about the system environment or hardware status is added, before the entry is posted to the error log.

The actual file in which error entries are stored is configurable; the default is /var/adm/ras/errlog. That file is in a binary format and so should never be truncated or zeroed out manually. The errlog file is a circular log, storing as many entries as can fit within its defined size. A memory buffer is set by the errdemon process, and newly arrived entries are put into the buffer before they are written to the log to minimize the possibility of a lost entry. The name and size of the error log file and the size of the memory buffer may be viewed with the errdemon command:

[aixhost:root:/] # /usr/lib/errdemon -l
Error Log Attributes
--------------------------------------------
Log File                /var/adm/ras/errlog
Log Size                1048576 bytes
Memory Buffer Size      8192 bytes

The parameters displayed may be changed by running the errdemon command with other flags, documented in the errdemon man page. The default sizes and values have always been sufficient on our systems, so I've never had reason to change them.

Due to use of a circular log file, it is not necessary (or even possible) to rotate the error log. Without intervention, errors will remain in the log indefinitely, or until the log fills up with new entries. As shipped, however, the crontab for the root user contains two entries that are executed daily, removing hardware errors that are older than 90 days, and all other errors that are older than 30 days.

0 11  *  *  * /usr/bin/errclear -d S,O 30
0 12  *  *  * /usr/bin/errclear -d H 90

These entries are commented out on my systems, as I prefer that older errors are removed "naturally", when they are replaced by newer entries.

Viewing Errors

Although a record of system errors is a good thing (as most sys admins would agree), logs are useless without a way to read them. Because the error log is stored in binary format, it can't be viewed as logs from syslog and other applications are. Fortunately, AIX provides the errpt command for reading the log.

The errpt command supports a number of optional flags and arguments, each designed to narrow the output to the desired amount. The man page for the errpt command provides detailed usage; Table 2 provides a short summary of the most useful arguments. (Note that all date/time specifications used with the errpt command are in the format of mmddHHMMyy, meaning "month", "day", "hour", "minute", "year"; seconds are not recorded in the error log, and are not specified with any command.)

Each entry in the AIX error log can be classified in a number of ways; the actual values are determined by the entry in the Error Record Template Repository that corresponds with the entry label as passed to the errdemon from the operating system or an application process. This classification system provides a more fine-grained method of prioritizing the severity of entries than does the syslog method of using a facility and priority code. Output from the errpt command may be confined to the types of entries desired by using a combination of the flags in Table 2. Some examples are shown in Table 3.

Dissecting an Error Log Entry

Entries in the error log are formatted in a standard layout, defined by their corresponding template. While different types of errors will provide different information, all error log entries follow a basic format. The one-line summary report (generated by the errpt command without using the "-a" flag) contains the fields shown in Table 4:

Here are several examples of error log entry summaries:

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
D1A1AE6F   0223070601 I H rmt3           TAPE SIM/MIM RECORD
5DFED6F1   0220054301 I O SYSPFS         UNABLE TO ALLOCATE SPACE
                                         IN FILE SYSTEM
1581762B   0219162801 T H hdisk98        DISK OPERATION ERROR

And here is the full entry of the second error summary above:

LABEL:            JFS_FS_FRAGMENTED
IDENTIFIER:       5DFED6F1
Date/Time:        Tue Feb 20 05:43:35
Sequence Number:  146643
Machine Id:       00018294A400
Node Id:          rescue
Class:            O
Type:             INFO
Resource Name:    SYSPFS
Description
UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
Probable Causes
FILE SYSTEM FREE SPACE FRAGMENTED
      Recommended Actions
      CONSOLIDATE FREE SPACE USING DEFRAGFS

      UTILITY
Detail Data
MAJOR/MINOR DEVICE NUMBER
000A 0006
FILE SYSTEM DEVICE AND MOUNT POINT
/dev/hd9var, /var

Monitoring with errreporter

Most, if not all systems administrators have had to deal with an "overload" of information. Multiple log files and process outputs must be monitored constantly for signs of trouble or required intervention. This problem is compounded when the administrator is responsible for a number of systems. Various solutions exist, including those built into the logging application (i.e., the use of a loghost for syslog messages), and free third-party solutions to monitor log files and send alerts when something interesting appears. One such tool that we rely on is "swatch", developed and maintained by Todd Atkins. Swatch excels at monitoring log files for lines that match specific regular expressions, and taking action for each matched entry, such as sending an email or running a command.

For all of the power of swatch, though, I was unable to set up the configuration to perform a specific task: monitoring entries in the AIX error log, ignoring certain specified identifiers, and emailing the full version of the entry to a specified address, with an informative subject line. So, I wrote my own simple program that performs the task I desired. errreporter (Listing 1) is a Perl script runs the errpt command in concurrent mode, checks new entries against a list of identifiers to be ignored, crafts a subject line based upon several fields in the entry, and emails the entire entry to a specified address.

errreporter can be run from the command line, though I have chosen to have it run automatically at system startup, with the following entry in /etc/inittab (all on a single line, but broken here, for convenience):

errrptr:2:respawn:/usr/sec/bin/errreporter \
     -f /usr/sec/etc/errreporter.conf/dev/console 2&1

Of course, if you choose to use this script, be sure to set the proper locations in your inittab entry. The system must have Perl installed; Perl is included with AIX as of version 4.3.3, and is available in source and compiled forms from numerous Web sites. It relies only on modules that are included with the base Perl distribution (see Listing 2 for errreporter.conf file).

Although this script perfectly suits my current needs, there are many areas in which it could be expanded upon or improved. For instance, it may be useful to have entries mailed to different addresses, based upon the entry's identifier. Another useful feature would be to incorporate "loghost"-like functionality, so that a program running on a single server can receive error log entries sent by other systems, communicating via sockets à la the syslog "@loghost" method.

... ... ...

The first source to go to for information on the usage of the commands and programs that are part of the Error Logging Facility is the man pages for the errpt, errdemon, errclear, errinstall, errupdate, errlogger, and errdemon commands, and for the errlog subroutine.

The complete listing of the entries in the Error Template Repository can be generated with the "errpt -t" command, for the one-line summary, or the "errpt -at" command, for the full text of each error template.

A more in-depth overview of the Error Logging Facility can be found in Chapter 10 of the AIX 4.3 Problem Solving Guide and Reference, available online at:
http://www.rs6000.ibm.com/doc_link/en_US/a_doc_lib/aixprob/prbslvgd/errlogfac.htm

[DOC] Logging from syslog and errlog to the same file

1. Clear out the error log

#errclear 0

2. Create the odm stanza for syslogging of error logs

# vi /tmp/syslog.add

errnotify:

en_name = "syslog1"

en_persistenceflg = 1

en_method = "logger –pnotice Msg from Error Log: `errpt –l $1| grep –v 'ERROR_ID TIMESTAMP'`"

3. Add the entry into the odm so that it is recognized

# odmadd /tmp/syslog.add

4. Create file to were the errpt and syslog errors will be monitored

# touch /tmp/all_errors

5. To forward messages to a local temporary file, edit the /etc/syslog.conf

(Caution: when editing /etc/syslog.conf, make sure that you have spaces between the level and the file or commancd)

Add a line at the very bottom

*.notice /tmp/all_errors

To send this to syslogd on another host(example aws3) use :

*.notice @aws3

§ Add '*.notice /tmp/notice' at the bottom of /etc/syslog.conf

§ touch /tmp/notice

§ refresh –s syslogd

6. Refresh syslog and error log then test

# errclear 0

# refresh –s syslogd

7. Now test:

# errlogger "Hello World"

# errpt –a

If you setup to a local file then

# cat /tmp/all_errors

If you setup to go the central host aws3, then on aws3

# cat /tmp/notice

[Aug 28, 2005] convert errlog - comp.unix.aix Google Groups

Hello,

Is there a way to convert the errlog into some type of readable format
without have to use errpt. The reason I want to do this is because some of
the Veritas error messages are reported with a large amount of sense key
data and when converted to ASCII test equals an actual error message that
gets logs in /var/adm/messages on a Solaris system and thus not have to
convert the hex sense data to ASCII data.

Thanks in advance for your help.

John
Aug 28 2005, 12:32 am

yes. there is a way by adding an odm entry. For every new event that comes, you can do an action for that event, such as running logger command and hence using syslog facility via which you can direct your message to an ascii file defined via syslog.conf.

[Apr 23, 2002] how to redirect errpt to syslog [Archive] - dBforums

Frhu
04-23-02, 20:11

Hi, I'd like to know if there is an easy way to redirect the errpt log file
to syslog ??

Or do I have to write a script to do that ??

thanks

---

Thierry Chenus

04-23-02, 23:09

"frhu" <[email protected]> wrote in message
news:<[email protected]>...
> Hi, I'd like to know if there is an easy way to redirect the errpt log file to
> syslog ??
>
> Or do I have to write a script to do that ??
>
> thanks

If you want to redirect errpt entries to a flat file, you can use "errpt -c
>/var/adm/messages" for example.

---

Ton Voon

04-24-02, 14:21

In article <[email protected]>, "frhu"
<[email protected]> wrote:

> Hi, I'd like to know if there is an easy way to redirect the errpt log file to
> syslog ??
>
> Or do I have to write a script to do that ??
>
> thanks
>

I run:

odmdelete -o errnotify -q en_name=TOPX_HWALERT cat <<-""EOF | odmadd errnotify:
en_type=PERM en_class=H en_persistenceflg=1 en_name="TOPX_HWALERT"
en_method="/usr/local/bin/ralert UX106 \"Hardware error: $6 $7 $8 $9\"" EOF

This puts an entry into the errnotify ODM database so that if a permanent hardware
error is entered into errpt, it will run the ralert program immediately. The ralert
program just runs logger to add the entry into syslog. You can then use whatever
usually techniques to monitor syslog.

The beauty of this is that it is event driven: as soon as AIX flags an error in
errpt, you'll get an occurance of ralert. This is the way HACMP is configured to get
AIX hardware errors.

Ton

Rainer's Blog rsyslog on AIX who wants to help porting

Thanks to the recent IBM contribution of a partial rsyslog 5.8.6 port to AIX, we have come across that platform again. Unfortunately, some license issues prevent me from merging the IBM contribution to the current rsyslog release (7.4+). I tried to work with IBM resolving these issues, but it just occurred to me that actually doing the port ourselves is probably easier than wrangling with license issues. I could set some time aside to make changes (as a by-activity, aka "may be sluggish at times"), but I neither have time, machines nor know-how. David Lang has already said has has some machines to try compile on, but we still have know-how issues.

So question now: would there some more folks be interested in rsyslog on AIX? If so, could you join a porting effort and provide us with advise? The most pressing thing currently is setting up an environment where we can compile at all. Possibly, once this is done, we may run into issues that require some more in-depth programming know-how, but that's to be seen (and must must not necessarily happen).

Anyone interested in helping with this please either comment to this blog posting or make yourself heard on the rsyslog mailing list or send me a private mail.

PS: all code derived from this work needs to be placed under ASL 2.0 - just so that you know.

Recommended Links

Softpanorama Recommended

Top articles

Sites

The AIX Error Logging Facility by Sandor W. Sklar

pSeries and AIX Information Center

Redirecting syslog Messages to Error Log

Some applications use syslog for logging errors and other events. To list error log messages and syslog messages in a single report, redirect the syslog messages to the error log. You can do this by specifying errlog as the destination in the syslog configuration file (/etc/syslog.conf). See the syslogd daemon for more information.

Directing Error Log Messages to syslog

You can log error log events in the syslog file by using the logger command with the concurrent error notification capabilities of error log. For example, to log system messages (syslog), add an errnotify object with the following contents:

errnotify:
        en_name = "syslog1"
        en_persistenceflg = 1
        en_method = "logger Msg from Error Log: `errpt -l $1 | grep -v 'ERROR_ID TIMESTAMP'`"

For example, create a file called /tmp/syslog.add with these contents. Then run the odmadd /tmp/syslog.add command (you must be logged in as root user).

For more information about concurrent error notification, see Error Notification.

[PDF] IBM Certification Study Guide - AIX 5L Problem Determination Tools ...

comp.unix.aix Frequently Asked Questions (Part 1 of 5) AIX FAQ

Planning Volume 2, Control Workstation and Software Environment

1.115 Where are the AIX log files kept

AIX logs messages as specified in /etc/syslog.conf. Here's an
example

#
*.err;kern.debug;auth.notice;user.none /dev/console
*.err;kern.debug;daemon,auth.notice;mail.crit;user.none /var/adm/messages
lpr.debug /var/adm/lpd-errs

*.alert;kern.err;daemon.err;user.none operator
*.alert;user.none root
*.emerg;user.none *

# for loghost machines, to have authentication messages (su, login, etc.)
# logged to a file, un-comment out the following line and adjust the
# file name as appropriate.
#
# if a non-loghost machine chooses to have such messages
# sent to the loghost machine, un-comment out the following line.
#
auth.notice /var/log/authlog
mail.debug /var/log/syslog

# following line for compatibility with old sendmails. they will send
# messages with no facility code, which will be turned into "user" messages
# by the local syslog daemon. only the "loghost" machine needs the following
# line, to cause these old sendmail log messages to be logged in the
# mail syslog file.
#
user.alert /var/log/syslog
#
# non-loghost machines will use the following lines to cause "user"
# log messages to be logged locally.
#
user.err /dev/console
user.err /var/adm/messages
user.alert `root, operator'
user.emerg *

Howto configure AIX syslogd (-etc-syslog.conf)

The following steps will guide you trough configuring syslog on an AIX system.
This info is valid for AIX 4.3.3, AIX 5.1, AIX 5.2 and AIX 5.3.

By default, an AIX system will not do syslog'ging. For a strange reason, the default install on an AIX install will not place entry's in /etc/syslog.conf, leaving a totl userless syslogd.

Even more, comparing the syslogd on AIX to another Unix, (ex: Linux), you will notice that <facility>.* doesn't work. So make sure that your priority level is never *.

Comming from a Linux world, you most probably want the following config:

mail.debug      /var/log/mail
user.debug      /var/log/user
kern.debug      /var/log/kern
syslog.debug    /var/log/syslog
daemon.debug    /var/log/daemon
auth.debug      /var/log/secure
local2.debug    /var/log/sudo

As AIX standard has no logrotate daemon running, you want to adapt syslogd todo the rotating on his own, an example would be:

mail.debug   /var/log/mail  rotate size 100k files 4		# 4 files, 100kB each
user.debug   /var/log/user  rotate files 12 time 1m		# 12 files, monthly rotate
kern.debug   /var/log/kern  rotate files 12 time 1m compress	# 12 files, monthly rotate, compress
...

When adaptions are made to /etc/syslog.conf, make sure to restart syslogd.

[root@sys /] refresh -s syslogd
0513-095 The request for subsystem refresh was completed successfully.
[root@sys /]

Recommended Papers

Sys Admin MagazineThe AIX Error Logging Facility Sandor W. Sklar

The primary goal of every UNIX systems administrator is to ensure that the systems that they are responsible for are functioning smoothly and with the best performance possible, 100% of the time. File systems running out of space, applications dumping core, and Ethernet adapter failures are just a sample of the types of things that can trip up a system, impacting that goal. Therefore, it is critical that the people responsible for a system are aware of anything that might have an impact on attaining that 100% system availability.

One of the things that makes AIX my favorite flavor of UNIX is that, besides all the standard tools, daemons, and configuration files that are present in all flavors of UNIX, IBM has provided a number of enhancements that make the monitoring, reliability, and administration of RS/6000 systems second to none. This article will focus on one of those tools: the error logging facility. I'll show you how the AIX error logging facility works, then I'll present a program I wrote that checks the log for error messages, filters out any error messages you wish to ignore, and sends an email to the systems administrator.

The Error Logging Subsystem

On most UNIX systems, information and errors from system events and processes are managed by the syslog daemon (syslogd); depending on settings in the configuration file /etc/syslog.conf, messages are passed from the operating system, daemons, and applications to the console, to log files, or to nowhere at all. AIX includes the syslog daemon, and it is used in the same way that other UNIX-based operating systems use it. In addition to syslog, though, AIX also contains another facility for the management of hardware, operating system, and application messages and errors. This facility, while simple in its operation, provides unique and valuable insight into the health and happiness of an RS/6000 system.

The AIX error logging facility components are part of the bos.rte and the bos.sysmgt.serv_aid packages, both of which are automatically placed on the system as part of the base operating system installation. Some of these components are shown in Table 1.

Unlike the syslog daemon, which performs no logging at all in its default configuration as shipped, the error logging facility requires no configuration before it can provide useful information about the system. The errdemon is started during system initialization and continuously monitors the special file /dev/error for new entries sent by either the kernel or by applications. The label of each new entry is checked against the contents of the Error Record Template Repository, and if a match is found, additional information about the system environment or hardware status is added, before the entry is posted to the error log.

The actual file in which error entries are stored is configurable; the default is /var/adm/ras/errlog. That file is in a binary format and so should never be truncated or zeroed out manually. The errlog file is a circular log, storing as many entries as can fit within its defined size. A memory buffer is set by the errdemon process, and newly arrived entries are put into the buffer before they are written to the log to minimize the possibility of a lost entry. The name and size of the error log file and the size of the memory buffer may be viewed with the errdemon command:

[aixhost:root:/] # /usr/lib/errdemon -l
Error Log Attributes
--------------------------------------------
Log File                /var/adm/ras/errlog
Log Size                1048576 bytes
Memory Buffer Size      8192 bytes

The parameters displayed may be changed by running the errdemon command with other flags, documented in the errdemon man page. The default sizes and values have always been sufficient on our systems, so I've never had reason to change them.

Due to use of a circular log file, it is not necessary (or even possible) to rotate the error log. Without intervention, errors will remain in the log indefinitely, or until the log fills up with new entries. As shipped, however, the crontab for the root user contains two entries that are executed daily, removing hardware errors that are older than 90 days, and all other errors that are older than 30 days.

0 11  *  *  * /usr/bin/errclear -d S,O 30
0 12  *  *  * /usr/bin/errclear -d H 90

These entries are commented out on my systems, as I prefer that older errors are removed "naturally", when they are replaced by newer entries.

Viewing Errors

Although a record of system errors is a good thing (as most sys admins would agree), logs are useless without a way to read them. Because the error log is stored in binary format, it can't be viewed as logs from syslog and other applications are. Fortunately, AIX provides the errpt command for reading the log.

The errpt command supports a number of optional flags and arguments, each designed to narrow the output to the desired amount. The man page for the errpt command provides detailed usage; Table 2 provides a short summary of the most useful arguments. (Note that all date/time specifications used with the errpt command are in the format of mmddHHMMyy, meaning "month", "day", "hour", "minute", "year"; seconds are not recorded in the error log, and are not specified with any command.)

Each entry in the AIX error log can be classified in a number of ways; the actual values are determined by the entry in the Error Record Template Repository that corresponds with the entry label as passed to the errdemon from the operating system or an application process. This classification system provides a more fine-grained method of prioritizing the severity of entries than does the syslog method of using a facility and priority code. Output from the errpt command may be confined to the types of entries desired by using a combination of the flags in Table 2. Some examples are shown in Table 3.

Dissecting an Error Log Entry

Entries in the error log are formatted in a standard layout, defined by their corresponding template. While different types of errors will provide different information, all error log entries follow a basic format. The one-line summary report (generated by the errpt command without using the "-a" flag) contains the fields shown in Table 4:

Here are several examples of error log entry summaries:

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
D1A1AE6F   0223070601 I H rmt3           TAPE SIM/MIM RECORD
5DFED6F1   0220054301 I O SYSPFS         UNABLE TO ALLOCATE SPACE
                                         IN FILE SYSTEM
1581762B   0219162801 T H hdisk98        DISK OPERATION ERROR

And here is the full entry of the second error summary above:

LABEL:            JFS_FS_FRAGMENTED
IDENTIFIER:       5DFED6F1
Date/Time:        Tue Feb 20 05:43:35
Sequence Number:  146643
Machine Id:       00018294A400
Node Id:          rescue
Class:            O
Type:             INFO
Resource Name:    SYSPFS
Description
UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
Probable Causes
FILE SYSTEM FREE SPACE FRAGMENTED
      Recommended Actions
      CONSOLIDATE FREE SPACE USING DEFRAGFS

      UTILITY
Detail Data
MAJOR/MINOR DEVICE NUMBER
000A 0006
FILE SYSTEM DEVICE AND MOUNT POINT
/dev/hd9var, /var

Monitoring with errreporter

Most, if not all systems administrators have had to deal with an "overload" of information. Multiple log files and process outputs must be monitored constantly for signs of trouble or required intervention. This problem is compounded when the administrator is responsible for a number of systems. Various solutions exist, including those built into the logging application (i.e., the use of a loghost for syslog messages), and free third-party solutions to monitor log files and send alerts when something interesting appears. One such tool that we rely on is "swatch", developed and maintained by Todd Atkins. Swatch excels at monitoring log files for lines that match specific regular expressions, and taking action for each matched entry, such as sending an email or running a command.

For all of the power of swatch, though, I was unable to set up the configuration to perform a specific task: monitoring entries in the AIX error log, ignoring certain specified identifiers, and emailing the full version of the entry to a specified address, with an informative subject line. So, I wrote my own simple program that performs the task I desired. errreporter (Listing 1) is a Perl script runs the errpt command in concurrent mode, checks new entries against a list of identifiers to be ignored, crafts a subject line based upon several fields in the entry, and emails the entire entry to a specified address.

errreporter can be run from the command line, though I have chosen to have it run automatically at system startup, with the following entry in /etc/inittab (all on a single line, but broken here, for convenience):

errrptr:2:respawn:/usr/sec/bin/errreporter \
     -f /usr/sec/etc/errreporter.conf/dev/console 2&1

Of course, if you choose to use this script, be sure to set the proper locations in your inittab entry. The system must have Perl installed; Perl is included with AIX as of version 4.3.3, and is available in source and compiled forms from numerous Web sites. It relies only on modules that are included with the base Perl distribution (see Listing 2 for errreporter.conf file).

Although this script perfectly suits my current needs, there are many areas in which it could be expanded upon or improved. For instance, it may be useful to have entries mailed to different addresses, based upon the entry's identifier. Another useful feature would be to incorporate "loghost"-like functionality, so that a program running on a single server can receive error log entries sent by other systems, communicating via sockets à la the syslog "@loghost" method.

Summary

The AIX Error Logging Facility can provide insight into the workings of your system that are not available on other UNIX platforms. I find it to be just one of the many advantages of AIX in a production environment, and I hope that I have helped to explain this simple yet powerful tool.

In this article, I have touched on some of the more commonly used aspects of the Error Logging Facility in AIX. There are numerous other features and capabilities of this subsystem, including the use of the "diag" command for error log analysis and problem determination, the addition of custom error templates, the redirection of error log entries to and from the syslog daemon, and the use of error notification routines in user-developed code to provide notice and error logging to this subsystem. For more information on those topics, and more detail on the items discussed above, please see the documents listed in the References section below.

References

The first source to go to for information on the usage of the commands and programs that are part of the Error Logging Facility is the man pages for the errpt, errdemon, errclear, errinstall, errupdate, errlogger, and errdemon commands, and for the errlog subroutine.

The complete listing of the entries in the Error Template Repository can be generated with the "errpt -t" command, for the one-line summary, or the "errpt -at" command, for the full text of each error template.

A more in-depth overview of the Error Logging Facility can be found in Chapter 10 of the AIX 4.3 Problem Solving Guide and Reference, available online at:
http://www.rs6000.ibm.com/doc_link/en_US/a_doc_lib/aixprob/prbslvgd/errlogfac.htm

Sandor Sklar is a UNIX systems administrator at Standford University. He can be reached at: [email protected].

Reference

syslogd Daemon

Logs system messages.

syslogd [ -d ] [ -s ] [ -f ConfigurationFile ] [ -m MarkInterval ] [ -r] [ -R ] [ -n ] [ -p LogName ]

The syslogd daemon reads a datagram socket and sends each message line to a destination described by the /etc/syslog.conf configuration file. The syslogd daemon reads the configuration file when it is activated and when it receives a hangup signal.

The syslogd daemon creates the /etc/syslog.pid file, which contains a single line with the command process ID used to end or reconfigure the syslogd daemon.

A terminate signal sent to the syslogd daemon ends the daemon. The syslogd daemon logs the end-signal information and terminates immediately.

Each message is one line. A message can contain a priority code, marked by a digit enclosed in < > (angle braces) at the beginning of the line. Messages longer than 900 bytes may be truncated.

The /usr/include/sys/syslog.h include file defines the facility and priority codes used by the configuration file. Locally written applications use the definitions contained in the syslog.h file to log messages via the syslogd daemon.

Flags

Configuration File

The configuration file informs the syslogd daemon where to send a system message, depending on the message's priority level and the facility that generated it.

If you do not use the -f flag, the syslogd daemon reads the default configuration file, the /etc/syslog.conf file.

The syslogd daemon ignores blank lines and lines beginning with a # (pound sign).

Format

Lines in the configuration file for the syslogd daemon contain a selector field, an action field, and an optional rotation field, separated by one or more tabs.

The selector field names a facility and a priority level. Separate facility names with a , (comma). Separate the facility and priority-level portions of the selector field with a . (period). Separate multiple entries in the same selector field with a ; (semicolon). To select all facilities, use an * (asterisk).

The action field identifies a destination (file, host, or user) to receive the messages. If routed to a remote host, the remote system will handle the message as indicated in its own configuration file. To display messages on a user's terminal, the destination field must contain the name of a valid, logged-in system user.

The rotation field identifies how rotation is used. If the action field is a file, then rotation can be based on size or time, or both. One can also compress and/or archive the rotated files.

Facilities

Use the following system facility names in the selector field:

Priority Levels

Use the following message priority levels in the selector field. Messages of the specified priority level and all levels above it are sent as directed.

Destinations

Use the following message destinations in the action field.

Rotation

Use the following rotation keywords in the rotation field.

Examples

1. To log all mail facility messages at the debug level or above to the file /tmp/mailsyslog, type:

   mail.debug /tmp/mailsyslog     

2. To send all system messages except those from the mail facility to a host named rigil, type:

   *.debug;mail.none @rigil

3. To send messages at the emerg priority level from all facilities, and messages at the crit priority level and above from the mail and daemon facilities, to users nick and jam, typer:

   *.emerg;mail,daemon.crit nick, jam

4. To send all mail facility messages to all users' terminal screens, type:

   mail.debug *

5. To log all facility messages at the debug level or above to the file /tmp/syslog.out, and have the file rotated when it gets larger then 500 kilobytes or if a week passes, limit the number of rotated files to 10, use compression and also use /syslogfiles as the archive directory, type:

   *.debug /tmp/syslog.out rotate size 500k time 1w files 10 compress archive /syslogfiles    

Files

Related Information

The syslog subroutine.

syslog.conf

Purpose

Controls output of the syslogd daemon.

Each line must consist of two parts:

1. A selector to determine the message priorities to which the line applies.
2. An action. Each line can contain an optional part.
3. Rotation.

The fields must be separated by one or more tabs or spaces.

Format

msg_src_list destination [rotate [size size k|m] [files files] [time time h|d|w|m|y] [compress] [archive archive]]

where msg_src_list is a semicolon separated list of facility.priority where:

facility

all (except mark)
mark - time marks kern,user,mail,daemon, auth,... For more information, see syslogd AIX 5L Version 5.3 Commands Reference, Volume 5.

priority

is one of (from high to low):
emerg/panic,alert,crit,err(or),warn(ing),notice,info,debug (meaning all messages of this priority or higher)

destination

is:
/filename - log to this file
username[,username2...] - write to user(s)
@hostname - send to syslogd on this machine
* - send to all logged in users

[rotate [size size k|m] [files files] [time time h|d|w|m|y] [compress] [archive archive]] is:

If destination is a regular file and the word rotate is specified, then the destination is limited by either size or time, or both. The size value causes the destination to be limited to size, with files files kept in the rotation. The backup filenames are created by appending a period and a number to destination, starting with .0. The time value causes the destination to be rotated after time. If both time and size are specified, then logfiles will be rotated once the logfile size exceeds size or the after time, whichever is earlier.

If the compress option is specified then the logfile names will be generated with a .Z extension. The files keyword will be applicable to the logfiles which are currently under rotation. For example, if we specify the compress option, then only file with .Z extension will be under rotation and the number of such files will be limited byfiles files. Any logfiles with an extension other than .Z will not be under the rotation scheme and thus will not be under the restriction of files files. Similarly if the compress option is removed then the files which have been generated with .Z extension will no longer be the part of rotation scheme and will not be limited by the files files.

The minimum size that can be specified is 10k. The minimum number of files that can be specified is 2. The default size is 1MB and the default for files is unlimited. Therefore, if only rotate is specified, the log will be rotated with size = 1m. The compress option means that rotated log files that are not in use will be compressed. The archive option will save rotated log files that are not in use to archive.

The default is not to rotate log files.

Example

"mail messages, at debug or higher, go to Log file. File must exist."

"all facilities, at debug and higher, go to console"

"all facilities, at crit or higher, go to all users"

 mail.debug              /usr/spool/mqueue/syslog

 *.debug              /dev/console

 *.crit                     *

 *.debug              /tmp/syslog.out     rotate size 100k files 4

 *.crit               /tmp/syslog.out     rotate time 1d


Files

Related Information

The syslogd daemon.

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 29, 2020