strace

Adapted from Wikipedia

strace is a debugging utility for Linux and some other Unix-like systems to monitor the system calls used by a program and all the signals it receives, similar to "truss" utility in other Unix systems. This is made possible by a kernel feature known as ptrace.

A similar utility is provided by Cygwin.

The most common usage is to start a program using strace, which prints a list of system calls made by the program. This is useful if the program continually crashes, or does not behave as expected; for example using strace may reveal that the program is attempting to access a file which does not exist or cannot be read.

An alternative application is to use the -p flag to attach to a running process. This is useful if a process has stopped responding, and might reveal, for example, that the process is blocking whilst attempting to make a network connection.

As strace only details system calls it cannot be used to detect as many problems as a code debugger such as GNU Debugger (gdb). It is, however, easier to use than a code debugger, and is an extremely useful tool for system administrators.

Example strace output

The following is an example of typical output of the strace command :

open(".", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(3, F_GETFD)                     = 0x1 (flags FD_CLOEXEC)
getdents64(3, /* 18 entries */, 4096)   = 496
getdents64(3, /* 0 entries */, 4096)    = 0
close(3)                                = 0
fstat64(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f2c000
write(1, "autofs\nbackups\ncache\nflexlm\ngames"..., 86autofsA

The above fragment is only a small part of the output of strace when run on the 'ls' command. It shows that the current working directory is opened, inspected and its contents retrieved. The resulting list of file names is written to standard output.

NEWS CONTENTS

• 20171017 : 10 Strace Commands for Troubleshooting and Debugging Linux Processes by Aaron Kili ( Oct 17, 2017 , www.tecmint.com )
• 20111006 : Linux Troubleshooting Wiki ( Linux Troubleshooting Wiki, Oct 06, 2011 )
• 20111006 : What files are trying to be opened ( )
• 20111006 : What is this thing doing to the network? ( )
• 20111006 : Rudimentary profiling ( )
• 20111006 : Following forks and attaching to running processes ( )
• 20111006 : 5-simple-ways-to-troubleshoot-using-strace by Vidar Hokstad ( Jun 11, 2008 , Vidar Hokstad V2.0 )
• 20111006 : Strace analisys of Linux System Calls by Mark L. Mitchell,Jeffrey Oldham ( Oct 12, 2001 , InformIT )

Old News ;-)

[Oct 17, 2017] 10 Strace Commands for Troubleshooting and Debugging Linux Processes by Aaron Kili

Oct 17, 2017 | www.tecmint.com

strace is a powerful command line tool for debugging and trouble shooting programs in Unix-like operating systems such as Linux. It captures and records all system calls made by a process and the signals received by the process.

Read Also : How to Audit Linux Process Using 'autrace' on CentOS/RHEL

It displays the name of each system call together with its arguments enclosed in a parenthesis and its return value to standard error; you can optionally redirect it to a file as well.

In this article, we will explain 10 strace command examples for troubleshooting and debugging programs and processes in a Linux system.

How to Install Strace Process Monitoring Tool in Linux

If strace is not pre-installed on your Linux system, run the appropriate command below for your distribution, to install it.

$ sudo apt install strace  #Debian/Ubuntu 
# yum install strace            #RHEL/CentOS
# dnf install strace            #Fedora 22+

In case a program crashes or behaves in a way not expected, you can go through its systems calls to get a clue of what exactly happened during its execution. As we will see later on, system calls can be categorized under different events: those relating to process management, those that take a file as an argument, those that involve networking, memory mapping, signals, IPC and also file descriptor related system calls.

You can either run a program/command with strace or pass a PID to it using the -p option as in the following examples.

1. Trace Linux Command System Calls

You can simply run a command with strace like this, here we are tracing of all system calls made by the df command .

$ strace df -h
execve("/bin/df", ["df", "-h"], [/* 50 vars */]) = 0
brk(NULL)                               = 0x136e000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f82f78fd000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=147662, ...}) = 0
mmap(NULL, 147662, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f82f78d8000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\t\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1868984, ...}) = 0
mmap(NULL, 3971488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f82f7310000
...

From the output above, you can see various types of system calls made by df command , for example.

open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3

Where:

• open – is the type of system call
• ("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) – system call argument
• – system call return value

Below is an sample output showing the write system calls, that displays df command output on the screen.

mmap(NULL, 26258, PROT_READ, MAP_SHARED, 3, 0) = 0x7f82f78f5000
close(3)                                = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
write(1, "Filesystem      Size  Used Avail"..., 49Filesystem      Size  Used Avail Use% Mounted on
) = 49
write(1, "udev            3.9G     0  3.9G"..., 43udev            3.9G     0  3.9G   0% /dev
) = 43
write(1, "tmpfs           788M  9.6M  779M"..., 43tmpfs           788M  9.6M  779M   2% /run
) = 43
write(1, "/dev/sda10      324G  252G   56G"..., 40/dev/sda10      324G  252G   56G  82% /
) = 40
write(1, "tmpfs           3.9G  104M  3.8G"..., 47tmpfs           3.9G  104M  3.8G   3% /dev/shm
) = 47
write(1, "tmpfs           5.0M  4.0K  5.0M"..., 48tmpfs           5.0M  4.0K  5.0M   1% /run/lock
) = 48
write(1, "tmpfs           3.9G     0  3.9G"..., 53tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
) = 53
write(1, "cgmfs           100K     0  100K"..., 56cgmfs           100K     0  100K   0% /run/cgmanager/fs
) = 56
write(1, "tmpfs           788M   36K  788M"..., 53tmpfs           788M   36K  788M   1% /run/user/1000
) = 53
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

2. Trace Linux Process PID

If a process is already running, you can trace it by simply passing its PID as follows; this will fill your screen with continues output that shows system calls being made by the process, to end it, press [Ctrl + C] .

$ sudo strace -p 3569
strace: Process 3569 attached
restart_syscall(<... resuming interrupted poll ...>) = 1
recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{"U\2\24\300!\247\330\0\3\24\4\0\20\0\0\0\0\0\0\24\24\24\24\24\0\0\3\37%\2\0\0", 4096}], msg_controllen=0, msg_flags=0}, 0) = 32
recvmsg(4, 0x7ffee4dbf870, 0)           = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(4, 0x7ffee4dbf850, 0)           = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=10, events=POLLIN}, {fd=30, events=POLLIN}, {fd=31, events=POLLIN}], 6, -1) = 1 ([{fd=31, revents=POLLIN}])
read(31, "\372", 1)                     = 1
recvmsg(4, 0x7ffee4dbf850, 0)           = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=10, events=POLLIN}, {fd=30, events=POLLIN}, {fd=31, events=POLLIN}], 6, 0) = 1 ([{fd=31, revents=POLLIN}])
read(31, "\372", 1)                     = 1
recvmsg(4, 0x7ffee4dbf850, 0)           = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=10, events=POLLIN}, {fd=30, events=POLLIN}, {fd=31, events=POLLIN}], 6, 0) = 0 (Timeout)
mprotect(0x207faa20000, 8192, PROT_READ|PROT_WRITE) = 0
mprotect(0x207faa20000, 8192, PROT_READ|PROT_EXEC) = 0
mprotect(0x207faa21000, 4096, PROT_READ|PROT_WRITE) = 0
mprotect(0x207faa21000, 4096, PROT_READ|PROT_EXEC) = 0
...

3. Get Summary of Linux Process

Using the -c flag, you can generate a report of total time, calls, and errors for each system call, as follows.

$ sudo strace -c -p 3569
strace: Process 3569 attached
^Cstrace: Process 3569 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
99.73    0.016000           8      1971           poll
0.16    0.000025           0       509        75 futex
0.06    0.000010           0      1985      1966 recvmsg
0.06    0.000009           0      2336           mprotect
0.00    0.000000           0       478           read
0.00    0.000000           0        13           write
0.00    0.000000           0        29           mmap
0.00    0.000000           0         9           munmap
0.00    0.000000           0        18           writev
0.00    0.000000           0       351           madvise
0.00    0.000000           0         1           restart_syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.016044                  7700      2041 total

4. Print Instruction Pointer During System Call

The -i option displays the instruction pointer at the time of each system call made by the program.

$ sudo strace -i df -h
[00007f0d7534c777] execve("/bin/df", ["df", "-h"], [/* 17 vars */]) = 0
[00007faf9cafa4b9] brk(NULL)            = 0x12f0000
[00007faf9cafb387] access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
[00007faf9cafb47a] mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faf9cd03000
[00007faf9cafb387] access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
[00007faf9cafb327] open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
[00007faf9cafb2b4] fstat(3, {st_mode=S_IFREG|0644, st_size=147662, ...}) = 0
[00007faf9cafb47a] mmap(NULL, 147662, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7faf9ccde000
[00007faf9cafb427] close(3)             = 0
[00007faf9cafb387] access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
[00007faf9cafb327] open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
[00007faf9cafb347] read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\t\2\0\0\0\0\0"..., 832) = 832
[00007faf9cafb2b4] fstat(3, {st_mode=S_IFREG|0755, st_size=1868984, ...}) = 0
[00007faf9cafb47a] mmap(NULL, 3971488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7faf9c716000
[00007faf9cafb517] mprotect(0x7faf9c8d6000, 2097152, PROT_NONE) = 0
...

5. Show Time of Day For Each Trace Output Line

You can also print the time of day for each line in the trace output, by passing the -t flag.

$ sudo strace -t df -h
15:19:25 execve("/bin/df", ["df", "-h"], [/* 17 vars */]) = 0
15:19:25 brk(NULL)                      = 0x234c000
15:19:25 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
15:19:25 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8c7f1d9000
15:19:25 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
15:19:25 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
15:19:25 fstat(3, {st_mode=S_IFREG|0644, st_size=147662, ...}) = 0
15:19:25 mmap(NULL, 147662, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f8c7f1b4000
15:19:25 close(3)                       = 0
15:19:25 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
15:19:25 open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
15:19:25 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\t\2\0\0\0\0\0"..., 832) = 832
15:19:25 fstat(3, {st_mode=S_IFREG|0755, st_size=1868984, ...}) = 0
15:19:25 mmap(NULL, 3971488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f8c7ebec000
15:19:25 mprotect(0x7f8c7edac000, 2097152, PROT_NONE) = 0
...

6. Print Command Time Spent in System Calls

To shows the time difference between the starting and the end of each system call made by a program, use the -T option.

$ sudo strace -T df -h
execve("/bin/df", ["df", "-h"], [/* 17 vars */]) = 0 <0.000287>
brk(NULL)                               = 0xeca000 <0.000035>
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory) <0.000028>
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9aff2b1000 <0.000020>
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory) <0.000019>
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 <0.000022>
fstat(3, {st_mode=S_IFREG|0644, st_size=147662, ...}) = 0 <0.000015>
mmap(NULL, 147662, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f9aff28c000 <0.000019>
close(3)                                = 0 <0.000014>
...

7. Trace Only Specific System Calls

In the command below, trace=write is known as a qualifying expression, where trace is a qualifier (others include signal, abbrev, verbose, raw, read, or write). Here, write is the value of the qualifier.

The following command actually shows the system calls to print df output on standard output.

$ sudo strace -e trace=write df -h
write(1, "Filesystem      Size  Used Avail"..., 49Filesystem      Size  Used Avail Use% Mounted on
) = 49
write(1, "udev            3.9G     0  3.9G"..., 43udev            3.9G     0  3.9G   0% /dev
) = 43
write(1, "tmpfs           788M  9.6M  779M"..., 43tmpfs           788M  9.6M  779M   2% /run
) = 43
write(1, "/dev/sda10      324G  252G   56G"..., 40/dev/sda10      324G  252G   56G  82% /
) = 40
write(1, "tmpfs           3.9G  104M  3.8G"..., 47tmpfs           3.9G  104M  3.8G   3% /dev/shm
) = 47
write(1, "tmpfs           5.0M  4.0K  5.0M"..., 48tmpfs           5.0M  4.0K  5.0M   1% /run/lock
) = 48
write(1, "tmpfs           3.9G     0  3.9G"..., 53tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
) = 53
write(1, "cgmfs           100K     0  100K"..., 56cgmfs           100K     0  100K   0% /run/cgmanager/fs
) = 56
write(1, "tmpfs           788M   28K  788M"..., 53tmpfs           788M   28K  788M   1% /run/user/1000
) = 53
+++ exited with 0 +++

Here are some additional commands about trace qualifier.

$ sudo strace -e trace=open,close df -h
$ sudo strace -e trace=open,close,read,write df -h
$ sudo strace -e trace=all df -h        

8. Trace System Calls Based on a Certain Condition

Let's look at how to trace system calls relating to a given class of events. This command can be used to trace all system calls involving process management.

$ sudo strace -q -e trace=process df -h    
execve("/bin/df", ["df", "-h"], [/* 17 vars */]) = 0
arch_prctl(ARCH_SET_FS, 0x7fe2222ff700) = 0
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           788M  9.6M  779M   2% /run
/dev/sda10      324G  252G   56G  82% /
tmpfs           3.9G  104M  3.8G   3% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
cgmfs           100K     0  100K   0% /run/cgmanager/fs
tmpfs           788M   28K  788M   1% /run/user/1000
exit_group(0)                           = ?
+++ exited with 0 +++

Next, to trace all system calls that take a filename as an argument, run this command.

$ sudo strace -q  -e trace=file df -h
execve("/bin/df", ["df", "-h"], [/* 17 vars */]) = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
...

To trace all system calls involving memory mapping, type.

$ sudo strace -q -e trace=memory df -h     
brk(NULL)                               = 0x77a000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe8f4658000
mmap(NULL, 147662, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fe8f4633000
mmap(NULL, 3971488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe8f406b000
mprotect(0x7fe8f422b000, 2097152, PROT_NONE) = 0
mmap(0x7fe8f442b000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c0000) = 0x7fe8f442b000
mmap(0x7fe8f4431000, 14752, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe8f4431000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe8f4632000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe8f4631000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe8f4630000
mprotect(0x7fe8f442b000, 16384, PROT_READ) = 0
mprotect(0x616000, 4096, PROT_READ)     = 0
mprotect(0x7fe8f465a000, 4096, PROT_READ) = 0
munmap(0x7fe8f4633000, 147662)          = 0
mmap(NULL, 2981280, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fe8f3d93000
brk(NULL)                               = 0x77a000
brk(0x79b000)                           = 0x79b000
mmap(NULL, 619, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fe8f4657000
mmap(NULL, 26258, PROT_READ, MAP_SHARED, 3, 0) = 0x7fe8f4650000
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           788M  9.6M  779M   2% /run
/dev/sda10      324G  252G   56G  82% /
tmpfs           3.9G  104M  3.8G   3% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
cgmfs           100K     0  100K   0% /run/cgmanager/fs
tmpfs           788M   28K  788M   1% /run/user/1000
+++ exited with 0 +++

You can trace all network and signals related system calls.

$ sudo strace -e trace=network df -h
$ sudo strace -e trace=signal df -h

9. Redirect Trace Output to File

To write the trace messages sent to standard error to a file, use the -o option. This means that only the command output is printed on the screen as shown below.

$ sudo strace -o df_debug.txt df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           788M  9.6M  779M   2% /run
/dev/sda10      324G  252G   56G  82% /
tmpfs           3.9G  104M  3.8G   3% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
cgmfs           100K     0  100K   0% /run/cgmanager/fs
tmpfs           788M   28K  788M   1% /run/user/1000

To look through the file, use cat command .

$ cat df_debug.txt

10. Show Some Debugging Output of Strace

[Oct 06, 2011] Linux Troubleshooting Wiki

strace

Strace is one of the most powerful tools available for troubleshooting. It allows you to see what an application is doing, to some degree.

`strace` displays all the system calls that an application is making, what arguments it passes to them, and what the return code is. A system call is generally something that requires the kernel to do something. This generally means I/O of all sorts, process management, shared memory and IPC useage, memory allocation, and network useage.

examples

The simplest example of using strace is as follows:

      strace ls -al

This starts the strace process, which then starts `ls -al` and shows every system call. For `ls -al` this is mostly I/O related calls. You can see it calling stat() on files, opening config files, opening the libs it is linked against, allocating memory, and calling write() to output the contents to the screen.

What files are trying to be opened

A common troubleshooting technique is to see what files an app is reading. You might want to make sure it's reading the proper config file, or looking at the correct cache, etc. `strace` by default shows all file I/O operations.

But to make it a bit easier, you can filter strace output. To see just file open()'s

      strace -eopen ls -al

This is a wonderful way to discover any configuration files that might be queried, as well as determining the order of the PATH settings.

What is this thing doing to the network?

To see all network related system calls (name resolution, opening sockets, writing/reading to sockets, etc)

   strace -e trace=network curl --head http://www.redhat.com

Rudimentary profiling

One thing that strace can be used for that is useful for debugging performance problems is some simple profiling.

    strace -c  ls -la

Invoking strace with '-c' will cause a cumulative report of system call usage to be printed. This includes approximate amount of time spent in each call, and how many times a system call is made.

This can sometimes help pinpoint performance issues, especially if an app is doing something like repeatedly opening/closing the same files.

    strace -tt ls -al

the -tt option causes strace to print out the time each call finished, in microseconds.

    strace -r ls -al

the -r option causes strace to print out the time since the last system call. This can be used to spot where a process is spending large amounts of time in user space or especially slow syscalls.

Following forks and attaching to running processes

Often it is difficult or impossible to run a command under strace (an apache httpd for instance). In this case, it's possible to attach to an already running process.

    strace -p 12345

where 12345 is the PID of the process. This is very handy for trying to determine why a process has stalled. Many times a process might be blocking while waiting for I/O. with strace -p, this is easy to detect.

Lot's of processes start other processes. It is often desireable to see a strace of all the processes.

   strace -f /etc/init.d/httpd start

will strace not just the bash process that runs the script, but any helper utilities executed by the script, and httpd itself.

Since strace output is often a handy way to help a developer solve a problem, it's useful to be able to write it to a file. The easiest way to do this is with the -o option.

   strace -o /tmp/strace.out program

Being somewhat familar with the common syscalls for Linux is helpful in understanding strace output. But most of the common ones are simple enough to be able to figure out on context.

A line in strace output is essentially the system call name, the arguments to the call in parentheses (sometimes truncated...), and then the return status. A return status for error is typically -1, but varies sometimes. For more information about the return status of a typical system call invoke `man 2 syscallname`. Usually the return status will be documented in the "RETURN STATUS" section.

Another thing to note about strace is it often shows "errno" status. If you're not familar with UNIX system programming, errno is a global variable that gets set to specific values when some commands execute. This variable gets set to different values based on the error mode of the command. More info on this can be found in `man errno`. But typically, strace will show the brief description for any errno values it gets, e.g.

    open("/foo/bar", O_RDONLY) = -1 ENOENT (No such file or directory)

   strace -s X

the -s option tells strace to show the first X digits of strings. The default is 32 characters, which sometimes is not enough. This will increase the info available to the user.

More info

Overview of linux system calls (http://www.quepublishing.com/articles/article.asp?p=23618&rl=1)

PDF version of Advanced Linux Programming (http://www.advancedlinuxprogramming.com/alp-folder)

5-simple-ways-to-troubleshoot-using-strace by Vidar Hokstad

Jun 11, 2008 | Vidar Hokstad V2.0

Strace is quite simply a tool that traces the execution of system calls. In its simplest form it can trace the execution of a binary from start to end, and output a line of text with the name of the system call, the arguments and the return value for every system call over the lifetime of the process.

But it can do a lot more:

• It can filter based on the specific system call or groups of system calls
• It can profile the use of system calls by tallying up the number of times a specific system call is used, and the time taken, and the number of successes and errors.
• It traces signals sent to the process.
• It can attach to any running process by pid.

If you've used other Unix systems, this is similar to "truss". Another (much more comprehensive) is Sun's Dtrace.

This is just scratching the surface, and in no particular order of importance:

1) Find out which config files a program reads on startup

Ever tried figuring out why some program doesn't read the config file you thought it should? Had to wrestle with custom compiled or distro-specific binaries that read their config from what you consider the "wrong" location?

The naive approach:

$ strace php 2>&1 | grep php.ini
open("/usr/local/bin/php.ini", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/php.ini", O_RDONLY) = 4
lstat64("/usr/local/lib/php.ini", {st_mode=S_IFLNK|0777, st_size=27, ...}) = 0
readlink("/usr/local/lib/php.ini", "/usr/local/Zend/etc/php.ini", 4096) = 27
lstat64("/usr/local/Zend/etc/php.ini", {st_mode=S_IFREG|0664, st_size=40971, ...}) = 0

So this version of PHP reads php.ini from /usr/local/lib/php.ini (but it tries /usr/local/bin first).

The more sophisticated approach if I only care about a specific syscall:

$ strace -e open php 2>&1 | grep php.ini
open("/usr/local/bin/php.ini", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/php.ini", O_RDONLY) = 4

The same approach work for a lot of other things. Have multiple versions of a library installed at different paths and wonder exactly which actually gets loaded? etc.

2) Why does this program not open my file?

Ever run into a program that silently refuse to read a file it doesn't have read access to, but you only figured out after swearing for ages because you thought it didn't actually find the file? Well, you already know what to do:

$ strace -e open,access 2>&1 | grep your-filename

Look for an open() or access() syscall that fails

3) What is that process doing RIGHT NOW?

Ever had a process suddenly hog lots of CPU? Or had a process seem to be hanging?

Then you find the pid, and do this:

root@dev:~# strace -p 15427
Process 15427 attached - interrupt to quit
futex(0x402f4900, FUTEX_WAIT, 2, NULL 
Process 15427 detached

Ah. So in this case it's hanging in a call to futex(). Incidentally in this case it doesn't tell us all that much - hanging on a futex can be caused by a lot of things (a futex is a locking mechanism in the Linux kernel). The above is from a normally working but idle Apache child process that's just waiting to be handed a request.

But "strace -p" is highly useful because it removes a lot of guesswork, and often removes the need for restarting an app with more extensive logging (or even recompile it).

4) What is taking time?

You can always recompile an app with profiling turned on, and for accurate information, especially about what parts of your own code that is taking time that is what you should do. But often it is tremendously useful to be able to just quickly attach strace to a process to see what it's currently spending time on, especially to diagnose problems. Is that 90% CPU use because it's actually doing real work, or is something spinning out of control.

Here's what you do:

root@dev:~# strace -c -p 11084
Process 11084 attached - interrupt to quit
Process 11084 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 94.59    0.001014          48        21           select
  2.89    0.000031           1        21           getppid
  2.52    0.000027           1        21           time
------ ----------- ----------- --------- --------- ----------------
100.00    0.001072                    63           total
root@dev:~# 

After you've started strace with -c -p you just wait for as long as you care to, and then exit with ctrl-c. Strace will spit out profiling data as above.

In this case, it's an idle Postgres "postmaster" process that's spending most of it's time quietly waiting in select(). In this case it's calling getppid() and time() in between each select() call, which is a fairly standard event loop.

You can also run this "start to finish", here with "ls":

root@dev:~# strace -c >/dev/null ls
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 23.62    0.000205         103         2           getdents64
 18.78    0.000163          15        11         1 open
 15.09    0.000131          19         7           read
 12.79    0.000111           7        16           old_mmap
  7.03    0.000061           6        11           close
  4.84    0.000042          11         4           munmap
  4.84    0.000042          11         4           mmap2
  4.03    0.000035           6         6         6 access
  3.80    0.000033           3        11           fstat64
  1.38    0.000012           3         4           brk
  0.92    0.000008           3         3         3 ioctl
  0.69    0.000006           6         1           uname
  0.58    0.000005           5         1           set_thread_area
  0.35    0.000003           3         1           write
  0.35    0.000003           3         1           rt_sigaction
  0.35    0.000003           3         1           fcntl64
  0.23    0.000002           2         1           getrlimit
  0.23    0.000002           2         1           set_tid_address
  0.12    0.000001           1         1           rt_sigprocmask
------ ----------- ----------- --------- --------- ----------------
100.00    0.000868                    87        10 total

Pretty much what you'd expect, it spents most of it's time in two calls to read the directory entries (only two since it was run on a small directory).

5) Why the **** can't I connect to that server?

Debugging why some process isn't connecting to a remote server can be exceedingly frustrating. DNS can fail, connect can hang, the server might send something unexpected back etc. You can use tcpdump to analyze a lot of that, and that too is a very nice tool, but a lot of the time strace will give you less chatter, simply because it will only ever return data related to the syscalls generated by "your" process. If you're trying to figure out what one of hundreds of running processes connecting to the same database server does for example (where picking out the right connection with tcpdump is a nightmare), strace makes life a lot easier.

This is an example of a trace of "nc" connecting to www.news.com on port 80 without any problems:

$ strace -e poll,select,connect,recvfrom,sendto nc www.news.com 80
sendto(3, "\24\0\0\0\26\0\1\3\255\373NH\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("62.30.112.39")}, 28) = 0
poll([{fd=3, events=POLLOUT, revents=POLLOUT}], 1, 0) = 1
sendto(3, "\213\321\1\0\0\1\0\0\0\0\0\0\3www\4news\3com\0\0\34\0\1", 30, MSG_NOSIGNAL, NULL, 0) = 30
poll([{fd=3, events=POLLIN, revents=POLLIN}], 1, 5000) = 1
recvfrom(3, "\213\321\201\200\0\1\0\1\0\1\0\0\3www\4news\3com\0\0\34\0\1\300\f"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("62.30.112.39")}, [16]) = 153
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("62.30.112.39")}, 28) = 0
poll([{fd=3, events=POLLOUT, revents=POLLOUT}], 1, 0) = 1
sendto(3, "k\374\1\0\0\1\0\0\0\0\0\0\3www\4news\3com\0\0\1\0\1", 30, MSG_NOSIGNAL, NULL, 0) = 30
poll([{fd=3, events=POLLIN, revents=POLLIN}], 1, 5000) = 1
recvfrom(3, "k\374\201\200\0\1\0\2\0\0\0\0\3www\4news\3com\0\0\1\0\1\300\f"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("62.30.112.39")}, [16]) = 106
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("62.30.112.39")}, 28) = 0
poll([{fd=3, events=POLLOUT, revents=POLLOUT}], 1, 0) = 1
sendto(3, "\\\2\1\0\0\1\0\0\0\0\0\0\3www\4news\3com\0\0\1\0\1", 30, MSG_NOSIGNAL, NULL, 0) = 30
poll([{fd=3, events=POLLIN, revents=POLLIN}], 1, 5000) = 1
recvfrom(3, "\\\2\201\200\0\1\0\2\0\0\0\0\3www\4news\3com\0\0\1\0\1\300\f"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("62.30.112.39")}, [16]) = 106
connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("216.239.122.102")}, 16) = -1 EINPROGRESS (Operation now in progress)
select(4, NULL, [3], NULL, NULL)        = 1 (out [3])

So what happens here?

Notice the connection attempts to /var/run/nscd/socket? They mean nc first tries to connect to NSCD - the Name Service Cache Daemon - which is usually used in setups that rely on NIS, YP, LDAP or similar directory protocols for name lookups. In this case the connects fails.

It then moves on to DNS (DNS is port 53, hence the "sin_port=htons(53)" in the following connect. You can see it then does a "sendto()" call, sending a DNS packet that contains www.news.com. It then reads back a packet. For whatever reason it tries three times, the last with a slightly different request. My best guess why in this case is that www.news.com is a CNAME (an "alias"), and the multiple requests may just be an artifact of how nc deals with that.

Then in the end, it finally issues a connect() to the IP it found. Notice it returns EINPROGRESS. That means the connect was non-blocking - nc wants to go on processing. It then calls select(), which succeeds when the connection was successful.

Try adding "read" and "write" to the list of syscalls given to strace and enter a string when connected, and you'll get something like this:

read(0, "test\n", 1024)                 = 5
write(3, "test\n", 5)                   = 5
poll([{fd=3, events=POLLIN, revents=POLLIN}, {fd=0, events=POLLIN}], 2, -1) = 1
read(3, "

This shows it reading "test" + linefeed from standard in, and writing it back out to the network connection, then calling poll() to wait for a reply, reading the reply from the network connection and writing it to standard out. Everything seems to be working right. Comments

Brian

strace -c `ps h -u 505 | awk '$1 > 3 {print "-p "$1}' | perl -pe 's/\n/ /g'`

Shows what my oracle (uid 505) is doing. I'd not used -c before I read your post.. THanks!

Prakash

Great Article. I will be of immense help in troubleshooting problems.

[root@www01 ~]# strace -c -p 2079 Process 2079 attached - interrupt to quit Process 2079 detached

I couldn't get profiling data for a Java process. Why is Java different from other system processes?

Thanks, Prakash.

Vidar Hokstad

Prakash,

Most likely the process wasn't executing any system calls while you were tracing it.

If the JVM you're using is multi-threaded it's possible any syscalls was executed by another thread. If not it's possible it simply was executing only in userspace at the time, and strace can't trace that.

You can try "ltrace" that Josh mentioned. "ltrace" is sort of a cousin of strace that traces dynamic library calls instead of syscalls, but that only helps for non-Java code that's dynamically linked into the JVM.

If you have access to a machine with Solaris/OpenSolaris you can also use "dtrace" which supports tracing Java code in much more detail.

2008-06-23 19:21 UTC

Strace analisys of Linux System Calls By Mark L. Mitchell,Jeffrey Oldham

Oct 12, 2001 | InformIT

Sample Chapter is provided courtesy of New Riders.

---

Linux currently provides about 200 different system calls. A listing of system calls for your version of the Linux kernel is in /usr/include/asm/unistd.h. Some of these are for internal use by the system, and others are used only in implementing specialized library functions. In this sample chapter, authors Jeffrey Oldham and Mark Mitchell present a selection of system calls that are likely to be the most useful to application and system programmers.

---

So far, we've presented a variety of functions that your program can invoke to perform system-related functions, such as parsing command-line options, manipulating processes, and mapping memory. If you look under the hood, you'll find that these functions fall into two categories, based on how they are implemented.

• A library function is an ordinary function that resides in a library external to your program. Most of the library functions we've presented so far are in the standard C library, libc. For example, getopt_long and mkstemp are functions provided in the C library.
• A call to a library function is just like any other function call. The arguments are placed in processor registers or onto the stack, and execution is transferred to the start of the function's code, which typically resides in a loaded shared library.
• A system call is implemented in the Linux kernel. When a program makes a system call, the arguments are packaged up and handed to the kernel, which takes over execution of the program until the call completes. A system call isn't an ordinary function call, and a special procedure is required to transfer control to the kernel. However, the GNU C library (the implementation of the standard C library provided with GNU/Linux systems) wraps Linux system calls with functions so that you can call them easily. Low-level I/O functions such as open and read are examples of system calls on Linux.
• The set of Linux system calls forms the most basic interface between programs and the Linux kernel. Each call presents a basic operation or capability.
• Some system calls are very powerful and can exert great influence on the system. For instance, some system calls enable you to shut down the Linux system or to allocate system resources and prevent other users from accessing them. These calls have the restriction that only processes running with superuser privilege (programs run by the root account) can invoke them. These calls fail if invoked by a nonsuperuser process.

Note that a library function may invoke one or more other library functions or system calls as part of its implementation.

Linux currently provides about 200 different system calls. A listing of system calls for your version of the Linux kernel is in /usr/include/asm/unistd.h. Some of these are for internal use by the system, and others are used only in implementing specialized library functions. In this chapter, we'll present a selection of system calls that are likely to be the most useful to application and system programmers.

Most of these system calls are declared in <unistd.h>.

8.1 Using strace

Before we start discussing system calls, it will be useful to present a command with which you can learn about and debug system calls. The strace command traces the execution of another program, listing any system calls the program makes and any signals it receives.

To watch the system calls and signals in a program, simply invoke strace, followed by the program and its command-line arguments. For example, to watch the system calls that are invoked by the hostname ¹ command, use this command:

% strace hostname

This produces a couple screens of output. Each line corresponds to a single system call. For each call, the system call's name is listed, followed by its arguments (or abbreviated arguments, if they are very long) and its return value. Where possible, strace conveniently displays symbolic names instead of numerical values for arguments and return values, and it displays the fields of structures passed by a pointer into the system call. Note that strace does not show ordinary function calls.

In the output from strace hostname, the first line shows the execve system call that invokes the hostname program: ²

execve("/bin/hostname", ["hostname"], [/* 49 vars */]) = 0

The first argument is the name of the program to run; the second is its argument list, consisting of only a single element; and the third is its environment list, which strace omits for brevity. The next 30 or so lines are part of the mechanism that loads the standard C library from a shared library file.

Toward the end are system calls that actually help do the program's work. The uname system call is used to obtain the system's hostname from the kernel,

uname({sys="Linux", node="myhostname", ...}) = 0

Observe that strace helpfully labels the fields (sys and node) of the structure argument. This structure is filled in by the system call-Linux sets the sys field to the operating system name and the node field to the system's hostname. The uname call is discussed further in Section 8.15, "uname."

Finally, the write system call produces output. Recall that file descriptor 1 corresponds to standard output. The third argument is the number of characters to write, and the return value is the number of characters that were actually written.

write(1, "myhostname\n", 11)      = 11

This may appear garbled when you run strace because the output from the hostname program itself is mixed in with the output from strace.

If the program you're tracing produces lots of output, it is sometimes more convenient to redirect the output from strace into a file. Use the option -o filename to do this.

Understanding all the output from strace requires detailed familiarity with the design of the Linux kernel and execution environment. Much of this is of limited interest to application programmers. However, some understanding is useful for debugging tricky problems or understanding how other programs work.

8.2 access: Testing File Permissions

The access system call determines whether the calling process has access permission to a file. It can check any combination of read, write, and execute permission, and it can also check for a file's existence.

The access call takes two arguments. The first is the path to the file to check. The second is a bitwise or of R_OK, W_OK, and X_OK, corresponding to read, write, and execute permission. The return value is 0 if the process has all the specified permissions. If the file exists but the calling process does not have the specified permissions, access returns –1 and sets errno to EACCES (or EROFS, if write permission was requested for a file on a read-only file system).

If the second argument is F_OK, access simply checks for the file's existence. If the file exists, the return value is 0; if not, the return value is –1 and errno is set to ENOENT. Note that errno may instead be set to EACCES if a directory in the file path is inaccessible.

The program shown in Listing 8.1 uses access to check for a file's existence and to determine read and write permissions. Specify the name of the file to check on the command line.

Listing 8.1 (check-access.c) Check File Access Permissions

#include <errno.h>
#include <stdio.h>
#include <unistd.h>

int main (int argc, char* argv[])
{
 char* path = argv[1];
 int rval;

 /* Check file existence. */
 rval = access (path, F_OK);
 if (rval == 0) 
  printf ("%s exists\n", path);
 else {
  if (errno == ENOENT) 
   printf ("%s does not exist\n", path);
  else if (errno == EACCES) 
   printf ("%s is not accessible\n", path);
  return 0;
 }

 /* Check read access. */
 rval = access (path, R_OK);
 if (rval == 0)
  printf ("%s is readable\n", path);
 else
  printf ("%s is not readable (access denied)\n", path);

 /* Check write access. */
 rval = access (path, W_OK);
 if (rval == 0)
  printf ("%s is writable\n", path);
 else if (errno == EACCES)
  printf ("%s is not writable (access denied)\n", path);
 else if (errno == EROFS)
  printf ("%s is not writable (read-only filesystem)\n", path);
 return 0;
}

For example, to check access permissions for a file named README on a CD-ROM, invoke it like this:

% ./check-access /mnt/cdrom/README
/mnt/cdrom/README exists
/mnt/cdrom/README is readable
/mnt/cdrom/README is not writable (read-only filesystem)

8.3 fcntl: Locks and Other File Operations

The fcntl system call is the access point for several advanced operations on file descriptors. The first argument to fcntl is an open file descriptor, and the second is a value that indicates which operation is to be performed. For some operations, fcntl takes an additional argument. We'll describe here one of the most useful fcntl operations, file locking. See the fcntl man page for information about the others.

The fcntl system call allows a program to place a read lock or a write lock on a file, somewhat analogous to the mutex locks discussed in Chapter 5, "Interprocess Communication." A read lock is placed on a readable file descriptor, and a write lock is placed on a writable file descriptor. More than one process may hold a read lock on the same file at the same time, but only one process may hold a write lock, and the same file may not be both locked for read and locked for write. Note that placing a lock does not actually prevent other processes from opening the file, reading from it, or writing to it, unless they acquire locks with fcntl as well.

To place a lock on a file, first create and zero out a struct flock variable. Set the l_type field of the structure to F_RDLCK for a read lock or F_WRLCK for a write lock. Then call fcntl, passing a file descriptor to the file, the F_SETLCKW operation code, and a pointer to the struct flock variable. If another process holds a lock that prevents a new lock from being acquired, fcntl blocks until that lock is released.

The program in Listing 8.2 opens a file for writing whose name is provided on the command line, and then places a write lock on it. The program waits for the user to hit Enter and then unlocks and closes the file.

Listing 8.2 (lock-file.c) Create a Write Lock with fcntl

#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

int main (int argc, char* argv[])
{
 char* file = argv[1];
 int fd;
 struct flock lock;

 printf ("opening %s\n", file);
 /* Open a file descriptor to the file. */
 fd = open (file, O_WRONLY);
 printf ("locking\n");
 /* Initialize the flock structure. */
 memset (&lock, 0, sizeof(lock));
 lock.l_type = F_WRLCK;
 /* Place a write lock on the file. */
 fcntl (fd, F_SETLKW, &lock);

 printf ("locked; hit Enter to unlock... ");
 /* Wait for the user to hit Enter. */
 getchar ();

 printf ("unlocking\n");
 /* Release the lock. */
 lock.l_type = F_UNLCK;
 fcntl (fd, F_SETLKW, &lock);

 close (fd);
 return 0;
}

Compile and run the program on a test file-say, /tmp/test-file-like this:

% cc -o lock-file lock-file.c
% touch /tmp/test-file
% ./lock-file /tmp/test-file
opening /tmp/test-file
locking
locked; hit Enter to unlock...

Now, in another window, try running it again on the same file.

% ./lock-file /tmp/test-file
opening /tmp/test-file
locking

Note that the second instance is blocked while attempting to lock the file. Go back to the first window and press Enter:

unlocking

The program running in the second window immediately acquires the lock.

If you prefer fcntl not to block if the call cannot get the lock you requested, use F_SETLK instead of F_SETLKW. If the lock cannot be acquired, fcntl returns –1 immediately.

Linux provides another implementation of file locking with the flock call. The fcntl version has a major advantage: It works with files on NFS³ file systems (as long as the NFS server is reasonably recent and correctly configured). So, if you have access to two machines that both mount the same file system via NFS, you can repeat the previous example using two different machines. Run lock-file on one machine, specifying a file on an NFS file system, and then run it again on another machine, specifying the same file. NFS wakes up the second program when the lock is released by the first program.

8.4 fsync and fdatasync: Flushing Disk Buffers

On most operating systems, when you write to a file, the data is not immediately written to disk. Instead, the operating system caches the written data in a memory buffer, to reduce the number of required disk writes and improve program responsiveness. When the buffer fills or some other condition occurs (for instance, enough time elapses), the system writes the cached data to disk all at one time.

Linux provides caching of this type as well. Normally, this is a great boon to performance. However, this behavior can make programs that depend on the integrity of disk-based records unreliable. If the system goes down suddenly-for instance, due to a kernel crash or power outage-any data written by a program that is in the memory cache but has not yet been written to disk is lost.

For example, suppose that you are writing a transaction-processing program that keeps a journal file. The journal file contains records of all transactions that have been processed so that if a system failure occurs, the state of the transaction data can be reconstructed. It is obviously important to preserve the integrity of the journal file-whenever a transaction is processed, its journal entry should be sent to the disk drive immediately.

To help you implement this, Linux provides the fsync system call. It takes one argument, a writable file descriptor, and flushes to disk any data written to this file. The fsync call doesn't return until the data has physically been written.

The function in Listing 8.3 illustrates the use of fsync. It writes a single-line entry to a journal file.

Listing 8.3 (write_journal_entry.c) Write and Sync a Journal Entry

#include <fcntl.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

const char* journal_filename = "journal.log";

void write_journal_entry (char* entry)
{
 int fd = open (journal_filename, O_WRONLY | O_CREAT | O_APPEND, 0660);
 write (fd, entry, strlen (entry));
 write (fd, "\n", 1);
 fsync (fd);
 close (fd);
}

Another system call, fdatasync does the same thing. However, although fsync guarantees that the file's modification time will be updated, fdatasync does not; it guarantees only that the file's data will be written. This means that in principal, fdatasync can execute faster than fsync because it needs to force only one disk write instead of two. However, in current versions of Linux, these two system calls actually do the same thing, both updating the file's modification time.

The fsync system call enables you to force a buffer write explicitly. You can also open a file for synchronous I/O, which causes all writes to be committed to disk immediately. To do this, specify the O_SYNC flag when opening the file with the open call.

8.5 getrlimit and setrlimit: Resource Limits

The getrlimit and setrlimit system calls allow a process to read and set limits on the system resources that it can consume. You may be familiar with the ulimit shell command, which enables you to restrict the resource usage of programs you run; ⁴ these system calls allow a program to do this programmatically.

For each resource there are two limits, the hard limit and the soft limit. The soft limit may never exceed the hard limit, and only processes with superuser privilege may change the hard limit. Typically, an application program will reduce the soft limit to place a throttle on the resources it uses.

Both getrlimit and setrlimit take as arguments a code specifying the resource limit type and a pointer to a structrlimit variable. The getrlimit call fills the fields of this structure, while the setrlimit call changes the limit based on its contents. The rlimit structure has two fields: rlim_cur is the soft limit, and rlim_max is the hard limit.

Some of the most useful resource limits that may be changed are listed here, with their codes:

• RLIMIT_CPU-The maximum CPU time, in seconds, used by a program. This is the amount of time that the program is actually executing on the CPU, which is not necessarily the same as wall-clock time. If the program exceeds this time limit, it is terminated with a SIGXCPU signal.
• RLIMIT_DATA-The maximum amount of memory that a program can allocate for its data. Additional allocation beyond this limit will fail.
• RLIMIT_NPROC-The maximum number of child processes that can be running for this user. If the process calls fork and too many processes belonging to this user are running on the system, fork fails.
• RLIMIT_NOFILE-The maximum number of file descriptors that the process may have open at one time.

See the setrlimit man page for a full list of system resources.

The program in Listing 8.4 illustrates setting the limit on CPU time consumed by a program. It sets a 1-second CPU time limit and then spins in an infinite loop. Linux kills the process soon afterward, when it exceeds 1 second of CPU time.

Listing 8.4 (limit-cpu.c) CPU Time Limit Demonstration

#include <sys/resource.h>
#include <sys/time.h>
#include <unistd.h>

int main ()
{
 struct rlimit rl;

 /* Obtain the current limits. */
 getrlimit (RLIMIT_CPU, &rl);
 /* Set a CPU limit of 1 second. */
 rl.rlim_cur = 1;
 setrlimit (RLIMIT_CPU, &rl);
 /* Do busy work. */
 while (1);

 return 0;
}

When the program is terminated by SIGXCPU, the shell helpfully prints out a message interpreting the signal:

% ./limit_cpu
CPU time limit exceeded

8.6 getrusage: Process Statistics

The getrusage system call retrieves process statistics from the kernel. It can be used to obtain statistics either for the current process by passing RUSAGE_SELF as the first argument, or for all terminated child processes that were forked by this process and its children by passing RUSAGE_CHILDREN. The second argument to rusage is a pointer to a struct rusage variable, which is filled with the statistics.

A few of the more interesting fields in struct rusage are listed here:

• ru_utime-A struct timeval field containing the amount of user time, in seconds, that the process has used. User time is CPU time spent executing the user program, rather than in kernel system calls.
• ru_stime-A struct timeval field containing the amount of system time, in seconds, that the process has used. System time is the CPU time spent executing system calls on behalf of the process.
• ru_maxrss-The largest amount of physical memory occupied by the process's data at one time over the course of its execution.

The getrusage man page lists all the available fields. See Section 8.7, "gettimeofday: Wall-Clock Time," for information about struct timeval.

The function in Listing 8.5 prints out the current process's user and system time.

Listing 8.5 (print-cpu-times.c) Display Process User and System Times

#include <stdio.h>
#include <sys/resource.h>
#include <sys/time.h>
#include <unistd.h>

void print_cpu_time()
{
 struct rusage usage;
 getrusage (RUSAGE_SELF, &usage);
 printf ("CPU time: %ld.%06ld sec user, %ld.%06ld sec system\n",
     usage.ru_utime.tv_sec, usage.ru_utime.tv_usec,
     usage.ru_stime.tv_sec, usage.ru_stime.tv_usec);
}

8.7 gettimeofday: Wall-Clock Time

The gettimeofday system call gets the system's wall-clock time. It takes a pointer to a struct timeval variable. This structure represents a time, in seconds, split into two fields. The tv_sec field contains the integral number of seconds, and the tv_usec field contains an additional number of microseconds. This struct timeval value represents the number of seconds elapsed since the start of the UNIX epoch, on midnight UTC on January 1, 1970. The gettimeofday call also takes a second argument, which should be NULL. Include <sys/time.h> if you use this system call.

The number of seconds in the UNIX epoch isn't usually a very handy way of representing dates. The localtime and strftime library functions help manipulate the return value of gettimeofday. The localtime function takes a pointer to the number of seconds (the tv_sec field of struct timeval) and returns a pointer to a struct tm object. This structure contains more useful fields, which are filled according to the local time zone:

• tm_hour, tm_min, tm_sec-The time of day, in hours, minutes, and seconds.
• tm_year, tm_mon, tm_day-The year, month, and date.
• tm_wday-The day of the week. Zero represents Sunday.
• tm_yday-The day of the year.
• tm_isdst-A flag indicating whether daylight savings time is in effect.

The strftime function additionally can produce from the struct tm pointer a customized, formatted string displaying the date and time. The format is specified in a manner similar to printf, as a string with embedded codes indicating which time fields to include. For example, this format string

 "%Y-%m-%d %H:%M:%S"

specifies the date and time in this form:

 2001-01-14 13:09:42

Pass strftime a character buffer to receive the string, the length of that buffer, the format string, and a pointer to a struct tm variable. See the strftime man page for a complete list of codes that can be used in the format string. Notice that neither localtime nor strftime handles the fractional part of the current time more precise than 1 second (the tv_usec field of struct timeval). If you want this in your formatted time strings, you'll have to include it yourself.

Include <time.h> if you call localtime or strftime.

The function in Listing 8.6 prints the current date and time of day, down to the millisecond.

Listing 8.6 (print-time.c) Print Date and Time

#include <stdio.h>
#include <sys/time.h>
#include <time.h>
#include <unistd.h>

void print_time ()
{
 struct timeval tv;
 struct tm* ptm;
 char time_string[40];
 long milliseconds;

 /* Obtain the time of day, and convert it to a tm struct. */
 gettimeofday (&tv, NULL);
 ptm = localtime (&tv.tv_sec);
 /* Format the date and time, down to a single second. */
 strftime (time_string, sizeof (time_string), "%Y-%m-%d %H:%M:%S", ptm);
 /* Compute milliseconds from microseconds. */
 milliseconds = tv.tv_usec / 1000;
 /* Print the formatted time, in seconds, followed by a decimal point
   and the milliseconds. */
 printf ("%s.%03ld\n", time_string, milliseconds);
}

8.8 The mlock Family: Locking Physical Memory

The mlock family of system calls allows a program to lock some or all of its address space into physical memory. This prevents Linux from paging this memory to swap space, even if the program hasn't accessed it for a while.

A time-critical program might lock physical memory because the time delay of paging memory out and back may be too long or too unpredictable. High-security applications may also want to prevent sensitive data from being written out to a swap file, where they might be recovered by an intruder after the program terminates.

Locking a region of memory is as simple as calling mlock with a pointer to the start of the region and the region's length. Linux divides memory into pages and can lock only entire pages at a time; each page that contains part of the memory region specified to mlock is locked. The getpagesize function returns the system's page size, which is 4KB on x86 Linux.

For example, to allocate 32MB of address space and lock it into RAM, you would use this code:

 const int alloc_size = 32 * 1024 * 1024;
 char* memory = malloc (alloc_size);
 mlock (memory, alloc_size);

Note that simply allocating a page of memory and locking it with mlock doesn't reserve physical memory for the calling process because the pages may be copy-on-write. ⁵ Therefore, you should write a dummy value to each page as well:

 size_t i;
 size_t page_size = getpagesize ();
 for (i = 0; i < alloc_size; i += page_size)
  memory[i] = 0;

The write to each page forces Linux to assign a unique, unshared memory page to the process for that page.

To unlock a region, call munlock, which takes the same arguments as mlock.

If you want your program's entire address space locked into physical memory, call mlockall. This system call takes a single flag argument: MCL_CURRENT locks all currently allocated memory, but future allocations are not locked; MCL_FUTURE locks all pages that are allocated after the call. Use MCL_CURRENT|MCL_FUTURE to lock into memory both current and subsequent allocations.

Locking large amounts of memory, especially using mlockall, can be dangerous to the entire Linux system. Indiscriminate memory locking is a good method of bringing your system to a grinding halt because other running processes are forced to compete for smaller physical memory resources and swap rapidly into and back out of memory (this is known as thrashing). If you lock too much memory, the system will run out of memory entirely and Linux will start killing off processes.

For this reason, only processes with superuser privilege may lock memory with mlock or mlockall. If a nonsuperuser process calls one of these functions, it w

urn –1, and set errno to EPERM.

The munlockall call unlocks all memory locked by the current process, including memory locked with mlock and mlockall.

A convenient way to monitor the memory usage of your program is to use the top command. In the output from top, the SIZE column displays the virtual address space size of each program (the total size of your program's code, data, and stack, some of which may be paged out to swap space). The RSS column (for resident set size) shows the size of physical memory that each program currently resides in. The sum of all the RSS values for all running programs cannot exceed your computer's physical memory size, and the sum of all address space sizes is limited to 2GB (for 32-bit versions of Linux).

Include <sys/mman.h> if you use any of the mlock system calls.

8.9 mprotect: Setting Memory Permissions

In Section 5.3, "Mapped Memory," we showed how to use the mmap system call to map a file into memory. Recall that the third argument to mmap is a bitwise or of memory protection flags PROT_READ, PROT_WRITE, and PROT_EXEC for read, write, and execute permission, respectively, or PROT_NONE for no memory access. If a program attempts to perform an operation on a memory location that is not allowed by these permissions, it is terminated with a SIGSEGV (segmentation violation) signal.

After memory has been mapped, these permissions can be modified with the mprotect system call. The arguments to mprotect are an address of a memory region, the size of the region, and a set of protection flags. The memory region must consist of entire pages: The address of the region must be aligned to the system's page size, and the length of the region must be a page size multiple. The protection flags for these pages are replaced with the specified value.

Obtaining Page-Aligned Memory

Note that memory regions returned by malloc are typically not page-aligned, even if the size of the memory is a multiple of the page size. If you want to protect memory obtained from malloc, you will have to allocate a larger memory region and find a page-aligned region within it.

Alternately, you can use the mmap system call to bypass malloc and allocate page-aligned memory directly from the Linux kernel. See Section 5.3, "Mapped Memory," for details.

For example, suppose that your program allocates a page of memory by mapping /dev/zero, as described in Section 5.3.5, "Other Uses for mmap." The memory is initially both readable and writable.

 int fd = open ("/dev/zero", O_RDONLY);
 char* memory = mmap (NULL, page_size, PROT_READ | PROT_WRITE, 
            MAP_PRIVATE, fd, 0);
 close (fd);

Later, your program could make the memory read-only by calling mprotect:

mprotect (memory, page_size, PROT_READ);

An advanced technique to monitor memory access is to protect the region of memory using mmap or mprotect and then handle the SIGSEGV signal that Linux sends to the program when it tries to access that memory. The example in Listing 8.7 illustrates this technique.

Listing 8.7 (mprotect.c) Detect Memory Access Using mprotect

#include <fcntl.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

static int alloc_size;
static char* memory;

void segv_handler (int signal_number) 
{
 printf ("memory accessed!\n");
 mprotect (memory, alloc_size, PROT_READ | PROT_WRITE);
} 

int main ()
{
 int fd;
 struct sigaction sa;

 /* Install segv_handler as the handler for SIGSEGV. */
 memset (&sa, 0, sizeof (sa));
 sa.sa_handler = &segv_handler;
 sigaction (SIGSEGV, &sa, NULL);
 
 /* Allocate one page of memory by mapping /dev/zero. Map the memory
   as write-only, initially. */
 alloc_size = getpagesize ();
 fd = open ("/dev/zero", O_RDONLY);
 memory = mmap (NULL, alloc_size, PROT_WRITE, MAP_PRIVATE, fd, 0);
 close (fd);
 /* Write to the page to obtain a private copy. */
 memory[0] = 0;
 /* Make the memory unwritable. */
 mprotect (memory, alloc_size, PROT_NONE);

 /* Write to the allocated memory region. */
 memory[0] = 1;

 /* All done; unmap the memory. */
 printf ("all done\n");
 munmap (memory, alloc_size);
 return 0;
}

The program follows these steps:

1. The program installs a signal handler for SIGSEGV.
2. The program allocates a page of memory by mapping /dev/zero and writing a value to the allocated page to obtain a private copy.
3. The program protects the memory by calling mprotect with the PROT_NONE permission.
4. When the program subsequently writes to memory, Linux sends it SIGSEGV, which is handled by segv_handler. The signal handler unprotects the memory, which allows the memory access to proceed.
5. When the signal handler completes, control returns to main, where the program deallocates the memory using munmap.

8.10 nanosleep: High-Precision Sleeping

The nanosleep system call is a high-precision version of the standard UNIX sleep call. Instead of sleeping an integral number of seconds, nanosleep takes as its argument a pointer to a struct timespec object, which can express time to nanosecond precision. However, because of the details of how the Linux kernel works, the actual precision provided by nanosleep is 10 milliseconds-still better than that afforded by sleep. This additional precision can be useful, for instance, to schedule frequent operations with short time intervals between them.

The struct timespec structure has two fields: tv_sec, the integral number of seconds, and tv_nsec, an additional number of milliseconds. The value of tv_nsec must be less than 109.

The nanosleep call provides another advantage over sleep. As with sleep, the delivery of a signal interrupts the execution of nanosleep, which sets errno to EINTR and returns –1. However, nanosleep takes a second argument, another pointer to a struct timespec object, which, if not null, is filled with the amount of time remaining (that is, the difference between the requested sleep time and the actual sleep time). This makes it easy to resume the sleep operation.

The function in Listing 8.8 provides an alternate implementation of sleep. Unlike the ordinary system call, this function takes a floating-point value for the number of seconds to sleep and restarts the sleep operation if it's interrupted by a signal.

Listing 8.8 (better_sleep.c) High-Precision Sleep Function

#include <errno.h>
#include <time.h>

int better_sleep (double sleep_time)
{
 struct timespec tv;
 /* Construct the timespec from the number of whole seconds... */
 tv.tv_sec = (time_t) sleep_time;
 /* ... and the remainder in nanoseconds. */
 tv.tv_nsec = (long) ((sleep_time - tv.tv_sec) * 1e+9);

 while (1)
 {
  /* Sleep for the time specified in tv. If interrupted by a
    signal, place the remaining time left to sleep back into tv. */
  int rval = nanosleep (&tv, &tv);
  if (rval == 0)
   /* Completed the entire sleep time; all done. */
   return 0;
  else if (errno == EINTR)
   /* Interrupted by a signal. Try again. */
   continue;
  else 
   /* Some other error; bail out. */
   return rval;
 }
 return 0;
}

8.11 readlink: Reading Symbolic Links

The readlink system call retrieves the target of a symbolic link. It takes three arguments: the path to the symbolic link, a buffer to receive the target of the link, and the length of that buffer. Unusually, readlink does not NUL-terminate the target path that it fills into the buffer. It does, however, return the number of characters in the target path, so NUL-terminating the string is simple.

If the first argument to readlink points to a file that isn't a symbolic link, readlink sets errno to EINVAL and returns –1.

The small program in Listing 8.9 prints the target of the symbolic link specified on its command line.

Listing 8.9 (print-symlink.c) Print the Target of a Symbolic Link

#include <errno.h>
#include <stdio.h>
#include <unistd.h>

int main (int argc, char* argv[])
{
 char target_path[256];
 char* link_path = argv[1];

 /* Attempt to read the target of the symbolic link. */
 int len = readlink (link_path, target_path, sizeof (target_path));

 if (len == -1) {
  /* The call failed. */
  if (errno == EINVAL)
   /* It's not a symbolic link; report that. */
   fprintf (stderr, "%s is not a symbolic link\n", link_path);
  else
   /* Some other problem occurred; print the generic message. */
   perror ("readlink");
  return 1;
 }
 else {
  /* NUL-terminate the target path. */
  target_path[len] = '\0';
  /* Print it. */
  printf ("%s\n", target_path);
  return 0;
 }
}

For example, here's how you could make a symbolic link and use print-symlink to read it back:

% ln -s /usr/bin/wc my_link
% ./print-symlink my_link
/usr/bin/wc

8.12 sendfile: Fast Data Transfers

The sendfile system call provides an efficient mechanism for copying data from one file descriptor to another. The file descriptors may be open to disk files, sockets, or other devices.

Typically, to copy from one file descriptor to another, a program allocates a fixed-size buffer, copies some data from one descriptor into the buffer, writes the buffer out to the other descriptor, and repeats until all the data has been copied. This is inefficient in both time and space because it requires additional memory for the buffer and performs an extra copy of the data into that buffer.

Using sendfile, the intermediate buffer can be eliminated. Call sendfile, passing the file descriptor to write to; the descriptor to read from; a pointer to an offset variable; and the number of bytes to transfer. The offset variable contains the offset in the input file from which the read should start (0 indicates the beginning of the file) and is updated to the position in the file after the transfer. The return value is the number of bytes transferred. Include <sys/sendfile.h> in your program if it uses sendfile.

The program in Listing 8.10 is a simple but extremely efficient implementation of a file copy. When invoked with two filenames on the command line, it copies the contents of the first file into a file named by the second. It uses fstat to determine the size, in bytes, of the source file.

Listing 8.10 (copy.c) File Copy Using sendfile

#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <sys/sendfile.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

int main (int argc, char* argv[])
{
 int read_fd;
 int write_fd;
 struct stat stat_buf;
 off_t offset = 0;

 /* Open the input file. */
 read_fd = open (argv[1], O_RDONLY);
 /* Stat the input file to obtain its size. */
 fstat (read_fd, &stat_buf);
 /* Open the output file for writing, with the same permissions as the
   source file. */
 write_fd = open (argv[2], O_WRONLY | O_CREAT, stat_buf.st_mode);
 /* Blast the bytes from one file to the other. */
 sendfile (write_fd, read_fd, &offset, stat_buf.st_size);
 /* Close up. */
 close (read_fd);
 close (write_fd);

 return 0;
}

The sendfile call can be used in many places to make copies more efficient. One good example is in a Web server or other network daemon, that serves the contents of a file over the network to a client program. Typically, a request is received from a socket connected to the client computer. The server program opens a local disk file to retrieve the data to serve and writes the file's contents to the network socket. Using sendfile can speed up this operation considerably. Other steps need to be taken to make the network transfer as efficient as possible, such as setting the socket parameters correctly. However, these are outside the scope of this book.

8.13 setitimer: Setting Interval Timers

The setitimer system call is a generalization of the alarm call. It schedules the delivery of a signal at some point in the future after a fixed amount of time has elapsed.

A program can set three different types of timers with setitimer:

• If the timer code is ITIMER_REAL, the process is sent a SIGALRM signal after the specified wall-clock time has elapsed.
• If the timer code is ITIMER_VIRTUAL, the process is sent a SIGVTALRM signal after the process has executed for the specified time. Time in which the process is not executing (that is, when the kernel or another process is running) is not counted.
• If the timer code is ITIMER_PROF, the process is sent a SIGPROF signal when the specified time has elapsed either during the process's own execution or the execution of a system call on behalf of the process.

The first argument to setitimer is the timer code, specifying which timer to set. The second argument is a pointer to a struct itimerval object specifying the new settings for that timer. The third argument, if not null, is a pointer to another struct itimerval object that receives the old timer settings.

A struct itimerval variable has two fields:

• it_value is a struct timeval field that contains the time until the timer next expires and a signal is sent. If this is 0, the timer is disabled.
• it_interval is another struct timeval field containing the value to which the timer will be reset after it expires. If this is 0, the timer will be disabled after it expires. If this is nonzero, the timer is set to expire repeatedly after this interval.

The struct timeval type is described in Section 8.7, "gettimeofday: Wall-Clock Time."

The program in Listing 8.11 illustrates the use of setitimer to track the execution time of a program. A timer is configured to expire every 250 milliseconds and send a SIGVTALRM signal.

Listing 8.11 (itemer.c) Timer Example

#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <sys/time.h>

void timer_handler (int signum)
{
 static int count = 0;
 printf ("timer expired %d times\n", ++count);
}

int main ()
{
 struct sigaction sa;
 struct itimerval timer;

 /* Install timer_handler as the signal handler for SIGVTALRM. */
 memset (&sa, 0, sizeof (sa));
 sa.sa_handler = &timer_handler;
 sigaction (SIGVTALRM, &sa, NULL);

 /* Configure the timer to expire after 250 msec... */
 timer.it_value.tv_sec = 0;
 timer.it_value.tv_usec = 250000;
 /* ... and every 250 msec after that. */
 timer.it_interval.tv_sec = 0;
 timer.it_interval.tv_usec = 250000;
 /* Start a virtual timer. It counts down whenever this process is
   executing. */
 setitimer (ITIMER_VIRTUAL, &timer, NULL);

 /* Do busy work. */
 while (1);
}

8.14 sysinfo: Obtaining System Statistics

The sysinfo system call fills a structure with system statistics. Its only argument is a pointer to a struct sysinfo. Some of the more interesting fields of struct sysinfo that are filled include these:

• uptime-Time elapsed since the system booted, in seconds
• totalram-Total available physical RAM
• freeram-Free physical RAM
• procs-Number of processes on the system

See the sysinfo man page for a full description of structsysinfo. Include <linux/kernel.h>, <linux/sys.h>, and <sys/sysinfo.h> if you use sysinfo.

The program in Listing 8.12 prints some statistics about the current system.

Listing 8.12 (sysinfo.c) Print System Statistics

#include <linux/kernel.h>
#include <linux/sys.h>
#include <stdio.h>
#include <sys/sysinfo.h>

int main ()
{
 /* Conversion constants. */
 const long minute = 60;
 const long hour = minute * 60;
 const long day = hour * 24;
 const double megabyte = 1024 * 1024;
 /* Obtain system statistics. */
 struct sysinfo si;
 sysinfo (&si);
 /* Summarize interesting values. */
 printf ("system uptime : %ld days, %ld:%02ld:%02ld\n", 
     si.uptime / day, (si.uptime % day) / hour, 
     (si.uptime % hour) / minute, si.uptime % minute);
 printf ("total RAM   : %5.1f MB\n", si.totalram / megabyte);
 printf ("free RAM   : %5.1f MB\n", si.freeram / megabyte);
 printf ("process count : %d\n", si.procs);
 return 0;
}

8.15 uname

The uname system call fills a structure with various system information, including the computer's network name and domain name, and the operating system version it's running. Pass uname a single argument, a pointer to a struct utsname object. Include <sys/utsname.h> if you use uname.

The call to uname fills in these fields:

• sysname-The name of the operating system (such as Linux).
• release, version-The Linux kernel release number and version level.
• machine-Some information about the hardware platform running Linux. For x86 Linux, this is i386 or i686, depending on the processor.
• node-The computer's unqualified hostname.
• __domain-The computer's domain name.

Each of these fields is a character string.

The small program in Listing 8.13 prints the Linux release and version number and the hardware information.

Listing 8.13 (print-uname) Print Linux Version Number and Hardware Information

#include <stdio.h>
#include <sys/utsname.h>

int main ()
{
 struct utsname u;
 uname (&u);
 printf ("%s release %s (version %s) on %s\n", u.sysname, u.release, 
     u.version, u.machine);
 return 0;

Recommended Links

Softpanorama Recommended

Top articles

Sites

• Using strace to debug application errors
• 5 simple ways to troubleshoot using Strace
• Troubleshooting Apache using strace LG #132
• Articles
• Administration - Using Strace to Determine What Files a Program Opens
• Linux - Using Strace to Debug Code
• Using strace for Fun and Profit | Linux Magazine

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March, 12, 2019