Softpanorama

Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
May the source be with you, but remember the KISS principle ;-)
Bigger doesn't imply better. Bigger often is a sign of obesity, of lost control, of overcomplexity, of cancerous cells

Managing AIX logs

News

AIX for Solaris Administrators

Recommended Links Redbooks Recommended Papers Reference  
Logs Analysis Syslog Syslog MultiTail Syslog Configuration Examples Log analyzers Logger Logrotate
AIX security AIX Hardening TEC logfile Adapter Compiling Apache on AIX Compiling PHP on AIX Humor Etc

AIX use both the BSD syslogd daemon and AIX proprietary error logging facilities, with a number of function-specific log files to record error events on each node. Most AIX components report errors using the errdemon daemon, an idiosyncratic AIX solution which is totally different from the BSD syslog facility. For example AIX kernel does not use syslog for error logging. 

In AIX the syslogd daemon  is not activated by default. You need to perform several steps in order to be able to use it the way it is used on Solaris and Linux:

The AIX Error Log facility is configured by default to write files just one megabyte in size. When the log fills up, it wraps around and overwrites existing entries. Therefore, you should set the size to at least 40MB. You can do this once for all nodes with the dsh command after the nodes are installed.

dsh -a /usr/lib/errdemon -s 40960000

AIX syslogd implementation of SYSLOG has built-in log rotation capabilities that are specified with optional fields after the log destination:

facility-level destination rotate size s  files n time t  compress archive path

For example:

*.emerg;*.alert;*.crit;*.warning /var/adm/messages  rotate files 4 time 7d compress

This is probably the most clean solution of the problem in comparison with Linux or Solaris solutions. As you can see in addition to rotation you can also specify compression  When log reaches the specified size and/or age, the current log file will be renamed to something like name.0 and optionally compressed. All previous generations of the same log will have their extensions incremented and the oldest file(s) may be deleted.

Among parameters for the rotate keyword the most important are: 

Strangely enough AIX does not provide any default syslog.conf  at all. But for syslogd activation it needs to exist. If you need standard entries you can borrow the file from Solaris. For example:

mail.debug                                              /var/adm/maillog
mail.none                                               /var/adm/maillog
auth.notice                                             /var/adm/authlog
lpr.debug                                               /var/adm/lpd-errs
kern.debug                                              /var/adm/messages
*.emerg;*.alert;*.crit;*.warning;*.err;*.notice;*.info  /var/adm/messages

Files referenced in syslog.conf need to exist, so you need to "touch" them.  After creating the files required,  you have to reload syslogd again before it will begin logging to the file. Like in Solaris,  you can use  `kill -HUP <syslog_pid>` to reload syslog.conf (there is no pkill command in AIX), but remember that syslogd doesn't complain about a missing file and doesn't create the missing file either. To avoid searching to PID you can use refresh -s syslogd (or stopsrc -s syslogd && startsrc -s syslogd)

Remember, that the syslog.conf file requires tabs as the field delimiter If you highlight a line with the mouse, then copy & paste a syslog.conf entry, the tabs can be  converted to spaces and you get a difficult to trace error.  Even more nasty for debugging are cases are when there is an  extra space before or after one of the fields.

Remember, that the syslog.conf file requires tabs as the field delimiter.  That's stupid but that's how it is. You need to be careful not to break this rule

The next step is to arrange forwarding error messages to syslog.  To do that you need to created a file called, say,  /root/etc/aix_errd2syslogd.add. This file should contain the following statements:

errnotify:
en_pid = 0
en_name = "syslog"
en_persistenceflg = 1
en_crcid = 0
en_method = "errpt -l $1 | tail -1 | logger -t errpt -p daemon.notice"

After that you need to issued the command

odmadd /root/etc/aix_errd2syslogd.add

The above AIX ODM stanza causes the error daemon to forward the most recent line of the error report to the syslog.

Please remember to configure log rotation: AIX does not rotate syslog logs by default. For example :
mail.debug   /var/log/mail  rotate size 500m files 4		# maintain 4 files, 500M each
user.debug   /var/log/user  rotate files 12 time 7d		# 12 files, daily rotate
kern.debug   /var/log/kern  rotate files 12 time 7d compress	# 12 files, monthly rotate, compress
...

When any modifications are made to /etc/syslog.conf, make sure to restart syslogd to reread the configuration.

Note: Administrators who get used to other systems need to remember that error messages are also written to the AIX Error Log which usually contains more information about probable cause and possible recovery or diagnostic actions (remember that in AIX, the primary logging facility is "AIX error log" and only code that was ported from 'other sources' contains the calls to syslog and logger.).

Using SMIT for log configuration

This is information from AIX documentation:

To access the Syslog SMIT menu, enter:

smit spsyslog

The format of a syslog log file is:

MMM DD HH:MM:SS node_name resource[pid]: msg

Where:

Errors logged by SP components will contain in the message section the following additional information pertaining to the logging resource:

The fastpath invocation for the Generate a Syslog Report menu is:

smit spsyslrpt

From the command line

To report to the local node all records logged by ftp for all syslog log files on nodes in the current system partition, enter:

psyslrpt -a -r ftp

To report all records that logged to files selected for the daemon and user facilities starting on March 3 and to report the records to the local node from nodes host1 and host2, enter:

psyslrpt -w host1,host2 -f user,daemon -s 03030000

For complete details on the psyslrpt command, refer to the PSSP: Command and Technical Reference.

If a syslog log file is not configured for rotation it will continue to grow until manually trimmed. The fastpath invocation for the Trim Syslog Log Files menu is:

smit spsyslclr

To trim all entries older than 30 days from all syslog log files specified in the syslog configuration file /etc/syslog.conf, on all nodes in the local system partition, enter:

psyslclr -a -y 30

To trim all records older than 30 days from any log file, for instance the /var/adm/SPlogs/SPdaemon.log file on the local node, use the -l option such as:

psyslclr -y 30 -l /var/adm/SPlogs/SPdaemon.log

To trim all records from all log files specified in an alternate syslog configuration file, for instance one named /u/local/syslog.conf, use the -g option such as:

psyslclr -g /u/local/syslog.conf -y 0
Note: The syslogd daemon is stopped during the trimming process, then restarted with either the default configuration file or an alternate file if specified with the -g option. For complete details see the psyslclr command in the book PSSP: Command and Technical Reference. The psyslclr command can be added as a crontab entry for performing scheduled syslog trimming. On the control workstation, the psyslclr command is used to trim daemon facility messages older than six days. This is done in the /usr/lpp/ssp/bin/cleanup.logs.ws script that is run from the crontabs file on the control workstation. (See Maintaining the crontabs file for more information.)

Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

[Oct 28, 2011] MyPurgeLogs

freshmeat.net

MyPurgeLogs is a script that can delete files older than a number of days and can rotate and compress other files. It simply reads a configuration file to know what it must do. Wildcards for directories and files are allowed. The compression's method is determined to optimize the free space on the filesystem. This script has only been tested on AIX.

Sys Admin The AIX Error Logging Facility by Sandor W. Sklar

www.samag.com

The AIX error logging facility components are part of the bos.rte and the bos.sysmgt.serv_aid packages, both of which are automatically placed on the system as part of the base operating system installation. Some of these components are shown in Table 1.

Unlike the syslog daemon, which performs no logging at all in its default configuration as shipped, the error logging facility requires no configuration before it can provide useful information about the system. The errdemon is started during system initialization and continuously monitors the special file /dev/error for new entries sent by either the kernel or by applications. The label of each new entry is checked against the contents of the Error Record Template Repository, and if a match is found, additional information about the system environment or hardware status is added, before the entry is posted to the error log.

The actual file in which error entries are stored is configurable; the default is /var/adm/ras/errlog. That file is in a binary format and so should never be truncated or zeroed out manually. The errlog file is a circular log, storing as many entries as can fit within its defined size. A memory buffer is set by the errdemon process, and newly arrived entries are put into the buffer before they are written to the log to minimize the possibility of a lost entry. The name and size of the error log file and the size of the memory buffer may be viewed with the errdemon command:

[aixhost:root:/] # /usr/lib/errdemon -l
Error Log Attributes
--------------------------------------------
Log File                /var/adm/ras/errlog
Log Size                1048576 bytes
Memory Buffer Size      8192 bytes

The parameters displayed may be changed by running the errdemon command with other flags, documented in the errdemon man page. The default sizes and values have always been sufficient on our systems, so I've never had reason to change them.

Due to use of a circular log file, it is not necessary (or even possible) to rotate the error log. Without intervention, errors will remain in the log indefinitely, or until the log fills up with new entries. As shipped, however, the crontab for the root user contains two entries that are executed daily, removing hardware errors that are older than 90 days, and all other errors that are older than 30 days.

0 11  *  *  * /usr/bin/errclear -d S,O 30
0 12  *  *  * /usr/bin/errclear -d H 90

These entries are commented out on my systems, as I prefer that older errors are removed "naturally", when they are replaced by newer entries.

Viewing Errors

Although a record of system errors is a good thing (as most sys admins would agree), logs are useless without a way to read them. Because the error log is stored in binary format, it can't be viewed as logs from syslog and other applications are. Fortunately, AIX provides the errpt command for reading the log.

The errpt command supports a number of optional flags and arguments, each designed to narrow the output to the desired amount. The man page for the errpt command provides detailed usage; Table 2 provides a short summary of the most useful arguments. (Note that all date/time specifications used with the errpt command are in the format of mmddHHMMyy, meaning "month", "day", "hour", "minute", "year"; seconds are not recorded in the error log, and are not specified with any command.)

Each entry in the AIX error log can be classified in a number of ways; the actual values are determined by the entry in the Error Record Template Repository that corresponds with the entry label as passed to the errdemon from the operating system or an application process. This classification system provides a more fine-grained method of prioritizing the severity of entries than does the syslog method of using a facility and priority code. Output from the errpt command may be confined to the types of entries desired by using a combination of the flags in Table 2. Some examples are shown in Table 3.

Dissecting an Error Log Entry

Entries in the error log are formatted in a standard layout, defined by their corresponding template. While different types of errors will provide different information, all error log entries follow a basic format. The one-line summary report (generated by the errpt command without using the "-a" flag) contains the fields shown in Table 4:

Here are several examples of error log entry summaries:

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
D1A1AE6F   0223070601 I H rmt3           TAPE SIM/MIM RECORD
5DFED6F1   0220054301 I O SYSPFS         UNABLE TO ALLOCATE SPACE
                                         IN FILE SYSTEM
1581762B   0219162801 T H hdisk98        DISK OPERATION ERROR

And here is the full entry of the second error summary above:

LABEL:            JFS_FS_FRAGMENTED
IDENTIFIER:       5DFED6F1
Date/Time:        Tue Feb 20 05:43:35
Sequence Number:  146643
Machine Id:       00018294A400
Node Id:          rescue
Class:            O
Type:             INFO
Resource Name:    SYSPFS
Description
UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
Probable Causes
FILE SYSTEM FREE SPACE FRAGMENTED
      Recommended Actions
      CONSOLIDATE FREE SPACE USING DEFRAGFS

      UTILITY
Detail Data
MAJOR/MINOR DEVICE NUMBER
000A 0006
FILE SYSTEM DEVICE AND MOUNT POINT
/dev/hd9var, /var

Monitoring with errreporter

Most, if not all systems administrators have had to deal with an "overload" of information. Multiple log files and process outputs must be monitored constantly for signs of trouble or required intervention. This problem is compounded when the administrator is responsible for a number of systems. Various solutions exist, including those built into the logging application (i.e., the use of a loghost for syslog messages), and free third-party solutions to monitor log files and send alerts when something interesting appears. One such tool that we rely on is "swatch", developed and maintained by Todd Atkins. Swatch excels at monitoring log files for lines that match specific regular expressions, and taking action for each matched entry, such as sending an email or running a command.

For all of the power of swatch, though, I was unable to set up the configuration to perform a specific task: monitoring entries in the AIX error log, ignoring certain specified identifiers, and emailing the full version of the entry to a specified address, with an informative subject line. So, I wrote my own simple program that performs the task I desired. errreporter (Listing 1) is a Perl script runs the errpt command in concurrent mode, checks new entries against a list of identifiers to be ignored, crafts a subject line based upon several fields in the entry, and emails the entire entry to a specified address.

errreporter can be run from the command line, though I have chosen to have it run automatically at system startup, with the following entry in /etc/inittab (all on a single line, but broken here, for convenience):

errrptr:2:respawn:/usr/sec/bin/errreporter \
     -f /usr/sec/etc/errreporter.conf/dev/console 2&1

Of course, if you choose to use this script, be sure to set the proper locations in your inittab entry. The system must have Perl installed; Perl is included with AIX as of version 4.3.3, and is available in source and compiled forms from numerous Web sites. It relies only on modules that are included with the base Perl distribution (see Listing 2 for errreporter.conf file).

Although this script perfectly suits my current needs, there are many areas in which it could be expanded upon or improved. For instance, it may be useful to have entries mailed to different addresses, based upon the entry's identifier. Another useful feature would be to incorporate "loghost"-like functionality, so that a program running on a single server can receive error log entries sent by other systems, communicating via sockets ŕ la the syslog "@loghost" method.

... ... ...

The first source to go to for information on the usage of the commands and programs that are part of the Error Logging Facility is the man pages for the errpt, errdemon, errclear, errinstall, errupdate, errlogger, and errdemon commands, and for the errlog subroutine.

The complete listing of the entries in the Error Template Repository can be generated with the "errpt -t" command, for the one-line summary, or the "errpt -at" command, for the full text of each error template.

A more in-depth overview of the Error Logging Facility can be found in Chapter 10 of the AIX 4.3 Problem Solving Guide and Reference, available online at:
http://www.rs6000.ibm.com/doc_link/en_US/a_doc_lib/aixprob/prbslvgd/errlogfac.htm

[DOC] Logging from syslog and errlog to the same file

1. Clear out the error log

#errclear 0

2. Create the odm stanza for syslogging of error logs

# vi /tmp/syslog.add

errnotify:

en_name = "syslog1"

en_persistenceflg = 1

en_method = "logger –pnotice Msg from Error Log: `errpt –l $1| grep –v 'ERROR_ID TIMESTAMP'`"

3. Add the entry into the odm so that it is recognized

# odmadd /tmp/syslog.add

4. Create file to were the errpt and syslog errors will be monitored

# touch /tmp/all_errors

5. To forward messages to a local temporary file, edit the /etc/syslog.conf

(Caution: when editing /etc/syslog.conf, make sure that you have spaces between the level and the file or commancd)

Add a line at the very bottom

*.notice /tmp/all_errors

To send this to syslogd on another host(example aws3) use :

*.notice @aws3

§ Add '*.notice /tmp/notice' at the bottom of /etc/syslog.conf

§ touch /tmp/notice

§ refresh –s syslogd

6. Refresh syslog and error log then test

# errclear 0

# refresh –s syslogd

7. Now test:

# errlogger "Hello World"

# errpt –a

If you setup to a local file then

# cat /tmp/all_errors

If you setup to go the central host aws3, then on aws3

# cat /tmp/notice

[Aug 28, 2005] convert errlog - comp.unix.aix Google Groups

Hello,

Is there a way to convert the errlog into some type of readable format
without have to use errpt. The reason I want to do this is because some of
the Veritas error messages are reported with a large amount of sense key
data and when converted to ASCII test equals an actual error message that
gets logs in /var/adm/messages on a Solaris system and thus not have to
convert the hex sense data to ASCII data.

Thanks in advance for your help.

John
Aug 28 2005, 12:32 am

yes. there is a way by adding an odm entry. For every new event that comes, you can do an action for that event, such as running logger command and hence using syslog facility via which you can direct your message to an ascii file defined via syslog.conf.

[Apr 23, 2002] how to redirect errpt to syslog [Archive] - dBforums

Frhu
04-23-02, 20:11

Hi, I'd like to know if there is an easy way to redirect the errpt log file
to syslog ??

Or do I have to write a script to do that ??

thanks


Thierry Chenus

04-23-02, 23:09

"frhu" <frederic.huber@ctrceal.caisse-epargne.fr> wrote in message
news:<aa3t4g$ava$1@s1.read.news.oleane.net>...
> Hi, I'd like to know if there is an easy way to redirect the errpt log file to
> syslog ??
>
> Or do I have to write a script to do that ??
>
> thanks

If you want to redirect errpt entries to a flat file, you can use "errpt -c
>/var/adm/messages" for example.


Ton Voon

04-24-02, 14:21

In article <aa3t4g$ava$1@s1.read.news.oleane.net>, "frhu"
<frederic.huber@ctrceal.caisse-epargne.fr> wrote:

> Hi, I'd like to know if there is an easy way to redirect the errpt log file to
> syslog ??
>
> Or do I have to write a script to do that ??
>
> thanks
>

I run:

odmdelete -o errnotify -q en_name=TOPX_HWALERT cat <<-""EOF | odmadd errnotify:
en_type=PERM en_class=H en_persistenceflg=1 en_name="TOPX_HWALERT"
en_method="/usr/local/bin/ralert UX106 \"Hardware error: $6 $7 $8 $9\"" EOF

This puts an entry into the errnotify ODM database so that if a permanent hardware
error is entered into errpt, it will run the ralert program immediately. The ralert
program just runs logger to add the entry into syslog. You can then use whatever
usually techniques to monitor syslog.

The beauty of this is that it is event driven: as soon as AIX flags an error in
errpt, you'll get an occurance of ralert. This is the way HACMP is configured to get
AIX hardware errors.

Ton

Rainer's Blog rsyslog on AIX who wants to help porting

Thanks to the recent IBM contribution of a partial rsyslog 5.8.6 port to AIX, we have come across that platform again. Unfortunately, some license issues prevent me from merging the IBM contribution to the current rsyslog release (7.4+). I tried to work with IBM resolving these issues, but it just occurred to me that actually doing the port ourselves is probably easier than wrangling with license issues. I could set some time aside to make changes (as a by-activity, aka "may be sluggish at times"), but I neither have time, machines nor know-how. David Lang has already said has has some machines to try compile on, but we still have know-how issues.

So question now: would there some more folks be interested in rsyslog on AIX? If so, could you join a porting effort and provide us with advise? The most pressing thing currently is setting up an environment where we can compile at all. Possibly, once this is done, we may run into issues that require some more in-depth programming know-how, but that's to be seen (and must must not necessarily happen).

Anyone interested in helping with this please either comment to this blog posting or make yourself heard on the rsyslog mailing list or send me a private mail.

PS: all code derived from this work needs to be placed under ASL 2.0 - just so that you know.

Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Sites

The AIX Error Logging Facility by Sandor W. Sklar

pSeries and AIX Information Center

Redirecting syslog Messages to Error Log

Some applications use syslog for logging errors and other events. To list error log messages and syslog messages in a single report, redirect the syslog messages to the error log. You can do this by specifying errlog as the destination in the syslog configuration file (/etc/syslog.conf). See the syslogd daemon for more information.

Directing Error Log Messages to syslog

You can log error log events in the syslog file by using the logger command with the concurrent error notification capabilities of error log. For example, to log system messages (syslog), add an errnotify object with the following contents:

errnotify:
        en_name = "syslog1"
        en_persistenceflg = 1
        en_method = "logger Msg from Error Log: `errpt -l $1 | grep -v 'ERROR_ID TIMESTAMP'`"

For example, create a file called /tmp/syslog.add with these contents. Then run the odmadd /tmp/syslog.add command (you must be logged in as root user).

For more information about concurrent error notification, see Error Notification.

[PDF] IBM Certification Study Guide - AIX 5L Problem Determination Tools ...

comp.unix.aix Frequently Asked Questions (Part 1 of 5) AIX FAQ

Planning Volume 2, Control Workstation and Software Environment

  • Chapter 8. Planning to record and diagnose system problems
  • Configuring the AIX error log
  • Configuring the BSD syslog
  • The control workstation
  • SP nodes
  • PSSP system logs
  • Finding and using error messages
  • 1.115 Where are the AIX log files kept

    AIX logs messages as specified in /etc/syslog.conf. Here's an
    example

    #
    *.err;kern.debug;auth.notice;user.none /dev/console
    *.err;kern.debug;daemon,auth.notice;mail.crit;user.none /var/adm/messages
    lpr.debug /var/adm/lpd-errs

    *.alert;kern.err;daemon.err;user.none operator
    *.alert;user.none root
    *.emerg;user.none *

    # for loghost machines, to have authentication messages (su, login, etc.)
    # logged to a file, un-comment out the following line and adjust the
    # file name as appropriate.
    #
    # if a non-loghost machine chooses to have such messages
    # sent to the loghost machine, un-comment out the following line.
    #
    auth.notice /var/log/authlog
    mail.debug /var/log/syslog

    # following line for compatibility with old sendmails. they will send
    # messages with no facility code, which will be turned into "user" messages
    # by the local syslog daemon. only the "loghost" machine needs the following
    # line, to cause these old sendmail log messages to be logged in the
    # mail syslog file.
    #
    user.alert /var/log/syslog
    #
    # non-loghost machines will use the following lines to cause "user"
    # log messages to be logged locally.
    #
    user.err /dev/console
    user.err /var/adm/messages
    user.alert `root, operator'
    user.emerg *

    Howto configure AIX syslogd (-etc-syslog.conf)

    The following steps will guide you trough configuring syslog on an AIX system.
    This info is valid for AIX 4.3.3, AIX 5.1, AIX 5.2 and AIX 5.3.

    By default, an AIX system will not do syslog'ging. For a strange reason, the default install on an AIX install will not place entry's in /etc/syslog.conf, leaving a totl userless syslogd.

    Even more, comparing the syslogd on AIX to another Unix, (ex: Linux), you will notice that <facility>.* doesn't work. So make sure that your priority level is never *.

    Comming from a Linux world, you most probably want the following config:

    mail.debug      /var/log/mail
    user.debug      /var/log/user
    kern.debug      /var/log/kern
    syslog.debug    /var/log/syslog
    daemon.debug    /var/log/daemon
    auth.debug      /var/log/secure
    local2.debug    /var/log/sudo
    

    As AIX standard has no logrotate daemon running, you want to adapt syslogd todo the rotating on his own, an example would be:

    mail.debug   /var/log/mail  rotate size 100k files 4		# 4 files, 100kB each
    user.debug   /var/log/user  rotate files 12 time 1m		# 12 files, monthly rotate
    kern.debug   /var/log/kern  rotate files 12 time 1m compress	# 12 files, monthly rotate, compress
    ...
    

    When adaptions are made to /etc/syslog.conf, make sure to restart syslogd.

    [root@sys /] refresh -s syslogd
    0513-095 The request for subsystem refresh was completed successfully.
    [root@sys /]
    

    Recommended Papers

    Sys Admin MagazineThe AIX Error Logging Facility Sandor W. Sklar

    The primary goal of every UNIX systems administrator is to ensure that the systems that they are responsible for are functioning smoothly and with the best performance possible, 100% of the time. File systems running out of space, applications dumping core, and Ethernet adapter failures are just a sample of the types of things that can trip up a system, impacting that goal. Therefore, it is critical that the people responsible for a system are aware of anything that might have an impact on attaining that 100% system availability.

    One of the things that makes AIX my favorite flavor of UNIX is that, besides all the standard tools, daemons, and configuration files that are present in all flavors of UNIX, IBM has provided a number of enhancements that make the monitoring, reliability, and administration of RS/6000 systems second to none. This article will focus on one of those tools: the error logging facility. I'll show you how the AIX error logging facility works, then I'll present a program I wrote that checks the log for error messages, filters out any error messages you wish to ignore, and sends an email to the systems administrator.

    The Error Logging Subsystem

    On most UNIX systems, information and errors from system events and processes are managed by the syslog daemon (syslogd); depending on settings in the configuration file /etc/syslog.conf, messages are passed from the operating system, daemons, and applications to the console, to log files, or to nowhere at all. AIX includes the syslog daemon, and it is used in the same way that other UNIX-based operating systems use it. In addition to syslog, though, AIX also contains another facility for the management of hardware, operating system, and application messages and errors. This facility, while simple in its operation, provides unique and valuable insight into the health and happiness of an RS/6000 system.

    The AIX error logging facility components are part of the bos.rte and the bos.sysmgt.serv_aid packages, both of which are automatically placed on the system as part of the base operating system installation. Some of these components are shown in Table 1.

    Unlike the syslog daemon, which performs no logging at all in its default configuration as shipped, the error logging facility requires no configuration before it can provide useful information about the system. The errdemon is started during system initialization and continuously monitors the special file /dev/error for new entries sent by either the kernel or by applications. The label of each new entry is checked against the contents of the Error Record Template Repository, and if a match is found, additional information about the system environment or hardware status is added, before the entry is posted to the error log.

    The actual file in which error entries are stored is configurable; the default is /var/adm/ras/errlog. That file is in a binary format and so should never be truncated or zeroed out manually. The errlog file is a circular log, storing as many entries as can fit within its defined size. A memory buffer is set by the errdemon process, and newly arrived entries are put into the buffer before they are written to the log to minimize the possibility of a lost entry. The name and size of the error log file and the size of the memory buffer may be viewed with the errdemon command:

    [aixhost:root:/] # /usr/lib/errdemon -l
    Error Log Attributes
    --------------------------------------------
    Log File                /var/adm/ras/errlog
    Log Size                1048576 bytes
    Memory Buffer Size      8192 bytes

    The parameters displayed may be changed by running the errdemon command with other flags, documented in the errdemon man page. The default sizes and values have always been sufficient on our systems, so I've never had reason to change them.

    Due to use of a circular log file, it is not necessary (or even possible) to rotate the error log. Without intervention, errors will remain in the log indefinitely, or until the log fills up with new entries. As shipped, however, the crontab for the root user contains two entries that are executed daily, removing hardware errors that are older than 90 days, and all other errors that are older than 30 days.

    0 11  *  *  * /usr/bin/errclear -d S,O 30
    0 12  *  *  * /usr/bin/errclear -d H 90

    These entries are commented out on my systems, as I prefer that older errors are removed "naturally", when they are replaced by newer entries.

    Viewing Errors

    Although a record of system errors is a good thing (as most sys admins would agree), logs are useless without a way to read them. Because the error log is stored in binary format, it can't be viewed as logs from syslog and other applications are. Fortunately, AIX provides the errpt command for reading the log.

    The errpt command supports a number of optional flags and arguments, each designed to narrow the output to the desired amount. The man page for the errpt command provides detailed usage; Table 2 provides a short summary of the most useful arguments. (Note that all date/time specifications used with the errpt command are in the format of mmddHHMMyy, meaning "month", "day", "hour", "minute", "year"; seconds are not recorded in the error log, and are not specified with any command.)

    Each entry in the AIX error log can be classified in a number of ways; the actual values are determined by the entry in the Error Record Template Repository that corresponds with the entry label as passed to the errdemon from the operating system or an application process. This classification system provides a more fine-grained method of prioritizing the severity of entries than does the syslog method of using a facility and priority code. Output from the errpt command may be confined to the types of entries desired by using a combination of the flags in Table 2. Some examples are shown in Table 3.

    Dissecting an Error Log Entry

    Entries in the error log are formatted in a standard layout, defined by their corresponding template. While different types of errors will provide different information, all error log entries follow a basic format. The one-line summary report (generated by the errpt command without using the "-a" flag) contains the fields shown in Table 4:

    Here are several examples of error log entry summaries:

    IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
    D1A1AE6F   0223070601 I H rmt3           TAPE SIM/MIM RECORD
    5DFED6F1   0220054301 I O SYSPFS         UNABLE TO ALLOCATE SPACE
                                             IN FILE SYSTEM
    1581762B   0219162801 T H hdisk98        DISK OPERATION ERROR

    And here is the full entry of the second error summary above:

    LABEL:            JFS_FS_FRAGMENTED
    IDENTIFIER:       5DFED6F1
    Date/Time:        Tue Feb 20 05:43:35
    Sequence Number:  146643
    Machine Id:       00018294A400
    Node Id:          rescue
    Class:            O
    Type:             INFO
    Resource Name:    SYSPFS
    Description
    UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
    Probable Causes
    FILE SYSTEM FREE SPACE FRAGMENTED
          Recommended Actions
          CONSOLIDATE FREE SPACE USING DEFRAGFS
    
          UTILITY
    Detail Data
    MAJOR/MINOR DEVICE NUMBER
    000A 0006
    FILE SYSTEM DEVICE AND MOUNT POINT
    /dev/hd9var, /var

    Monitoring with errreporter

    Most, if not all systems administrators have had to deal with an "overload" of information. Multiple log files and process outputs must be monitored constantly for signs of trouble or required intervention. This problem is compounded when the administrator is responsible for a number of systems. Various solutions exist, including those built into the logging application (i.e., the use of a loghost for syslog messages), and free third-party solutions to monitor log files and send alerts when something interesting appears. One such tool that we rely on is "swatch", developed and maintained by Todd Atkins. Swatch excels at monitoring log files for lines that match specific regular expressions, and taking action for each matched entry, such as sending an email or running a command.

    For all of the power of swatch, though, I was unable to set up the configuration to perform a specific task: monitoring entries in the AIX error log, ignoring certain specified identifiers, and emailing the full version of the entry to a specified address, with an informative subject line. So, I wrote my own simple program that performs the task I desired. errreporter (Listing 1) is a Perl script runs the errpt command in concurrent mode, checks new entries against a list of identifiers to be ignored, crafts a subject line based upon several fields in the entry, and emails the entire entry to a specified address.

    errreporter can be run from the command line, though I have chosen to have it run automatically at system startup, with the following entry in /etc/inittab (all on a single line, but broken here, for convenience):

    errrptr:2:respawn:/usr/sec/bin/errreporter \
         -f /usr/sec/etc/errreporter.conf/dev/console 2&1

    Of course, if you choose to use this script, be sure to set the proper locations in your inittab entry. The system must have Perl installed; Perl is included with AIX as of version 4.3.3, and is available in source and compiled forms from numerous Web sites. It relies only on modules that are included with the base Perl distribution (see Listing 2 for errreporter.conf file).

    Although this script perfectly suits my current needs, there are many areas in which it could be expanded upon or improved. For instance, it may be useful to have entries mailed to different addresses, based upon the entry's identifier. Another useful feature would be to incorporate "loghost"-like functionality, so that a program running on a single server can receive error log entries sent by other systems, communicating via sockets ŕ la the syslog "@loghost" method.

    Summary

    The AIX Error Logging Facility can provide insight into the workings of your system that are not available on other UNIX platforms. I find it to be just one of the many advantages of AIX in a production environment, and I hope that I have helped to explain this simple yet powerful tool.

    In this article, I have touched on some of the more commonly used aspects of the Error Logging Facility in AIX. There are numerous other features and capabilities of this subsystem, including the use of the "diag" command for error log analysis and problem determination, the addition of custom error templates, the redirection of error log entries to and from the syslog daemon, and the use of error notification routines in user-developed code to provide notice and error logging to this subsystem. For more information on those topics, and more detail on the items discussed above, please see the documents listed in the References section below.

    References

    The first source to go to for information on the usage of the commands and programs that are part of the Error Logging Facility is the man pages for the errpt, errdemon, errclear, errinstall, errupdate, errlogger, and errdemon commands, and for the errlog subroutine.

    The complete listing of the entries in the Error Template Repository can be generated with the "errpt -t" command, for the one-line summary, or the "errpt -at" command, for the full text of each error template.

    A more in-depth overview of the Error Logging Facility can be found in Chapter 10 of the AIX 4.3 Problem Solving Guide and Reference, available online at:
    http://www.rs6000.ibm.com/doc_link/en_US/a_doc_lib/aixprob/prbslvgd/errlogfac.htm

    Sandor Sklar is a UNIX systems administrator at Standford University. He can be reached at: ssklar@stanford.edu.

    Reference

    syslogd Daemon

    Logs system messages.

    syslogd [ -d ] [ -s ] [ -f ConfigurationFile ] [ -m MarkInterval ] [ -r] [ -R ] [ -n ] [ -p LogName ]

    The syslogd daemon reads a datagram socket and sends each message line to a destination described by the /etc/syslog.conf configuration file. The syslogd daemon reads the configuration file when it is activated and when it receives a hangup signal.

    The syslogd daemon creates the /etc/syslog.pid file, which contains a single line with the command process ID used to end or reconfigure the syslogd daemon.

    A terminate signal sent to the syslogd daemon ends the daemon. The syslogd daemon logs the end-signal information and terminates immediately.

    Each message is one line. A message can contain a priority code, marked by a digit enclosed in < > (angle braces) at the beginning of the line. Messages longer than 900 bytes may be truncated.

    The /usr/include/sys/syslog.h include file defines the facility and priority codes used by the configuration file. Locally written applications use the definitions contained in the syslog.h file to log messages via the syslogd daemon.

    Flags

    -d Turns on debugging.
    -f ConfigurationFile Specifies an alternate configuration file.
    -m MarkInterval Specifies the number of minutes between the mark command messages. If you do not use this flag, the mark command sends a message with LOG_INFO priority sent every 20 minutes. This facility is not enabled by a selector field containing an * (asterisk), which selects all other facilities.
    -s Specifies to forward a "shortened" message to another system (if it is configured to do so) for all the forwarding syslog messages generated on the local system.
    -r Suppresses logging of messages received from remote hosts.
    -R When specified, disables the facility to receive messages from the network using the internet domain socket.
    -n Supresses the "Message forwarded from <log_host_name>: " string added to the beginning of the syslog message that is forwarded to a remote log host.
    -p Specifies an alternate path name for the UNIX datagram socket.

    Configuration File

    The configuration file informs the syslogd daemon where to send a system message, depending on the message's priority level and the facility that generated it.

    If you do not use the -f flag, the syslogd daemon reads the default configuration file, the /etc/syslog.conf file.

    The syslogd daemon ignores blank lines and lines beginning with a # (pound sign).

    Format

    Lines in the configuration file for the syslogd daemon contain a selector field, an action field, and an optional rotation field, separated by one or more tabs.

    The selector field names a facility and a priority level. Separate facility names with a , (comma). Separate the facility and priority-level portions of the selector field with a . (period). Separate multiple entries in the same selector field with a ; (semicolon). To select all facilities, use an * (asterisk).

    The action field identifies a destination (file, host, or user) to receive the messages. If routed to a remote host, the remote system will handle the message as indicated in its own configuration file. To display messages on a user's terminal, the destination field must contain the name of a valid, logged-in system user.

    The rotation field identifies how rotation is used. If the action field is a file, then rotation can be based on size or time, or both. One can also compress and/or archive the rotated files.

    Facilities

    Use the following system facility names in the selector field:

    Facility Description
    kern Kernel
    user User level
    mail Mail subsystem
    daemon System daemons
    auth Security or authorization
    syslog syslogd daemon
    lpr Line-printer subsystem
    news News subsystem
    uucp uucp subsystem
    * All facilities

    Priority Levels

    Use the following message priority levels in the selector field. Messages of the specified priority level and all levels above it are sent as directed.

    Priority Level Description
    emerg Specifies emergency messages (LOG_EMERG). These messages are not distributed to all users. LOG_EMERG priority messages can be logged into a separate file for reviewing.
    alert Specifies important messages (LOG_ALERT), such as a serious hardware error. These messages are distributed to all users.
    crit Specifies critical messages not classified as errors (LOG_CRIT), such as improper login attempts. LOG_CRIT and higher-priority messages are sent to the system console.
    err Specifies messages that represent error conditions (LOG_ERR), such as an unsuccessful disk write.
    warning Specifies messages for abnormal, but recoverable, conditions (LOG_WARNING).
    notice Specifies important informational messages (LOG_NOTICE). Messages without a priority designation are mapped into this priority message.
    info Specifies informational messages (LOG_INFO). These messages can be discarded, but are useful in analyzing the system.
    debug Specifies debugging messages (LOG_DEBUG). These messages may be discarded.
    none Excludes the selected facility. This priority level is useful only if preceded by an entry with an * (asterisk) in the same selector field.

    Destinations

    Use the following message destinations in the action field.

    Destination Description
    File Name Full path name of a file opened in append mode
    @Host Host name, preceded by @ (at sign)
    User[, User][...] User names
    * All users

    Rotation

    Use the following rotation keywords in the rotation field.

    Keyword Description
    rotate This keyword must be specified after the action field.
    size This keyword specifies that rotation is based on size. It is followed by a number and either a k (kilobytes) or m(megabytes).
    time This keyword specifies that rotation is based on time. It is followed by a number and either a h(hour) or d(day) or w(week) or m(month) or y(year).
    files This keyword specifies the total number of rotated files. It is followed by a number. If not specified, then there are unlimited number of rotated files.
    compress This keyword specifies that the saved rotated files will be compressed.
    archive This keyword specifies that the saved rotated files will be copied to a directory. It is followed by the directory name.

    Examples

    1. To log all mail facility messages at the debug level or above to the file /tmp/mailsyslog, type:

      mail.debug /tmp/mailsyslog     
    2. To send all system messages except those from the mail facility to a host named rigil, type:

      *.debug;mail.none @rigil
    3. To send messages at the emerg priority level from all facilities, and messages at the crit priority level and above from the mail and daemon facilities, to users nick and jam, typer:

      *.emerg;mail,daemon.crit nick, jam
    4. To send all mail facility messages to all users' terminal screens, type:

      mail.debug *
    5. To log all facility messages at the debug level or above to the file /tmp/syslog.out, and have the file rotated when it gets larger then 500 kilobytes or if a week passes, limit the number of rotated files to 10, use compression and also use /syslogfiles as the archive directory, type:

      *.debug /tmp/syslog.out rotate size 500k time 1w files 10 compress archive /syslogfiles    

    Files

    /etc/syslog.conf Controls the output of syslogd.
    /etc/syslog.pid Contains the process ID.

    Related Information

    The syslog subroutine.

    syslog.conf

    Purpose

    Controls output of the syslogd daemon.

    Each line must consist of two parts:

    1. A selector to determine the message priorities to which the line applies.
    2. An action. Each line can contain an optional part.
    3. Rotation.

    The fields must be separated by one or more tabs or spaces.

    Format

    msg_src_list destination [rotate [size size k|m] [files files] [time time h|d|w|m|y] [compress] [archive archive]]

    where msg_src_list is a semicolon separated list of facility.priority where:

    facility
    all (except mark)
    mark - time marks kern,user,mail,daemon, auth,... For more information, see syslogd AIX 5L Version 5.3 Commands Reference, Volume 5.
    priority
    is one of (from high to low):
    emerg/panic,alert,crit,err(or),warn(ing),notice,info,debug (meaning all messages of this priority or higher)
    destination
    is:
    /filename - log to this file
    username[,username2...] - write to user(s)
    @hostname - send to syslogd on this machine
    * - send to all logged in users

    [rotate [size size k|m] [files files] [time time h|d|w|m|y] [compress] [archive archive]] is:

    If destination is a regular file and the word rotate is specified, then the destination is limited by either size or time, or both. The size value causes the destination to be limited to size, with files files kept in the rotation. The backup filenames are created by appending a period and a number to destination, starting with .0. The time value causes the destination to be rotated after time. If both time and size are specified, then logfiles will be rotated once the logfile size exceeds size or the after time, whichever is earlier.

    If the compress option is specified then the logfile names will be generated with a .Z extension. The files keyword will be applicable to the logfiles which are currently under rotation. For example, if we specify the compress option, then only file with .Z extension will be under rotation and the number of such files will be limited byfiles files. Any logfiles with an extension other than .Z will not be under the rotation scheme and thus will not be under the restriction of files files. Similarly if the compress option is removed then the files which have been generated with .Z extension will no longer be the part of rotation scheme and will not be limited by the files files.

    The minimum size that can be specified is 10k. The minimum number of files that can be specified is 2. The default size is 1MB and the default for files is unlimited. Therefore, if only rotate is specified, the log will be rotated with size = 1m. The compress option means that rotated log files that are not in use will be compressed. The archive option will save rotated log files that are not in use to archive.

    The default is not to rotate log files.

    Example

    "mail messages, at debug or higher, go to Log file. File must exist."
    
    "all facilities, at debug and higher, go to console"
    
    "all facilities, at crit or higher, go to all users"
    
     mail.debug              /usr/spool/mqueue/syslog
    
     *.debug              /dev/console
    
     *.crit                     *
    
     *.debug              /tmp/syslog.out     rotate size 100k files 4
    
     *.crit               /tmp/syslog.out     rotate time 1d
    
    
    Files
    /etc/syslog.conf Controls the output of syslogd.

    Related Information

    The syslogd daemon.



    Etc

    Society

    Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

    Quotes

    War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

    Bulletin:

    Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

    History:

    Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

    Classic books:

    The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

    Most popular humor pages:

    Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

    The Last but not Least


    Copyright © 1996-2018 by Dr. Nikolai Bezroukov. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) in the author free time and without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

     

    FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

    This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

    You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

    Disclaimer:

    The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

    The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

    Last modified: September 12, 2017