Introduction to Process Management

• Introduction
• Managing Jobs
• Running Jobs in the Foreground and Background
• Moving jobs between the foreground and background
• Command-Line Tools for Process Management
• Using ps to get list of processes running
• Changing process priority with nice
• Sending signals to processes with kill, killall, and pkill
• Using top to Manage Processes
   

Introduction

Running Linux server is essentially a collection of processes. Some privileged, some not.  That means that process management is an important skill for any sysadmin.

Firs of all we can distinguish between two process types:

• User processes

  • Shell jobs are commands started from the command line. They are associated with the shell that was current when the process was started. Shell jobs are also referred to as interactive processes. Unless you take special measures they are killed, if you exit your interactive session.
  • Cron jobs. Those are jobs that are started from scheduler. They are not connected to any interactive session. Cron also has the ability to start process on boot.

• Daemons. those are processes that provide services. They normally are started when a computer is booted and often (but certainly not in all cases) they are running with root privileges. they are not connected to any interactive session.

When a process is started, it can use multiple threads. A thread is a task started by a process and that a dedicated CPU can service. The Linux shell does not offer tools to manage individual threads. Thread management should be taken care of from within the command.

To manage a process efficiently, it is paramount that you know what type of process you are dealing with. Shell jobs require another approach than the processes that are automatically started when a computer boots.

Most processes are in one of the following two states:

• A process that is on the CPU (a running process)
• A process that is off the CPU (a not-running process)

Only one process can run at a time on a single CPU. All other processes have to wait or be in some other state. This is why a process that is not running appears in a different state. Four most important states are as following:

1. Runnable state
2. Sleeping state
3. Uninterruptable sleep state
4. Defunct or Zombie state

In this example of a common process life cycle, the process is as follows:

1. Born or forked
2. Ready to run or runnable
3. Running in user space or running in kernel space
4. Blocked, Waiting, Sleeping, in an "Interruptible Sleep", or in an "Uninterruptable sleep"
5. The process is sleeping, but it is present in main memory
6. The process is sleeping, but it is present in secondary memory storage (swap space on disk)
7. Terminated or stopped

Managing Jobs

When a user types a command and press Enter, a job is started. If no particular measures have been taken, the job is started as a foreground process, occupying the terminal it was started from until it has finished its work. As a Linux administrator, you need to know how to start shell jobs in the foreground or background and what can be done to manage shell jobs.

When a process is started from a shell, it becomes a child process of that shell. In process management, the parent-child relationship between processes is   important. because all processes started from a shell are terminated when that shell is stopped. This also offers an easy way to terminate processes no longer needed.

Processes started in the background will not be killed when the parent shell from which they were started is killed. To terminate these processes, you need to use the kill  or pkill commands. The latter allows to kill all processes belong to a particular user, which is a very important function for sysadmins

Note

In earlier versions of the bash shell, background processes were also killed when the shell they were started from was terminated. To prevent that, the process could be started with the nohup  command in front of it. Using nohup  for this purpose is optional in RHEL 7.

Running Jobs in the Foreground and Background

By default, any executed command is started as a foreground job. For many commands, that does not really matter because the command often takes a little while to complete, after which it returns access to the shell from which it was started. Sometimes it might prove useful to start commands in the background. This makes sense for processes that do not require user interaction. A process that does require user interaction will not be able to get that when running in the background, and for that reason will typically stall when moved to the background. You can take two different approaches to run a process in the background.

If you know that a job will take a long time to complete, you can start it with an &  behind it. This immediately starts the job in the background to make room for other tasks to be started from the command line. To move the last job that was started in the background back as a foreground job, use the fg  command. This command immediately, and with no further questions, brings the last job back to the foreground.

NOTE: RHEL 7 changes the semantic of background jobs -- they now survive the disconnect of terminal session in which they were initialed

A job might sometimes have been started that takes (much) longer than predicted. If that happens, you can use Ctrl+Z  to temporarily stop the job. This does not remove the job from memory; it just pauses the job so that it can be managed. Once paused, it can be continued as a background job using the bg  command.

An alternative key sequence that you can use to manage shell jobs is Ctrl+C. This stops the current job and removes it from memory.

A related keystroke combination is Ctrl+D, which sends the End Of File (EOF) character to the current job. The result is that the job stops waiting for further input so that it can complete what it was currently doing. The result of sending Ctrl+D  is sometimes very similar to the result of sending Ctrl+C, but there is a difference.

When Ctrl+C  is used, the job is just canceled, and some files might not be closed properly. When Ctrl+D  is used, the job stops waiting for further input, which often is just what is needed to complete in a proper way.

Moving jobs between the foreground and background

When moving jobs between the foreground and background, it may be useful to have an overview of all current jobs. To get such an overview, use the jobs  command. Job numbers can be used as an argument to the fg  and bg  commands to perform job management tasks.

Exercise ( modified from Sander van Vugt book) 

You read how to manage interactive shell jobs in this section. Notice that all of these jobs are processes as well. As the user who started the job, you can also manage it. You need to be root to manage process of other users, or be able to switch you identituy to the identity of the other user via sudo.

1.   Open a root shell and type the following commands:

   sleep 3600 &
   dd if=/dev/zero of=/dev/null &
   sleep 7200

2.   Because you started the last command with no &  after the command, you have to wait 2 hours before you get control to the shell back. Type Ctrl+Z  to stop it.

3.   Type jobs. You will see the three jobs that you just started. The first two of them have the Running state, and the last job currently is in the Stopped state.

4.   Type bg 3  to continue running job 3 in the background. Notice that because it was started as the last job, you did not really have to add the number 3.

5.   Type fg 2  to move job 1 to the foreground.

6.   Press Ctrl+C  to cancel job number 2 and use jobs  to confirm that it is now gone.

7. Type jobs  again to see remaining jobs.

NOTES:

• You can use Use xargs -P to submit several parallel jobs t one time.
• There is a batch command what allow you to specify multiple background jobs  which will be be executed one after another,
• You can use at command to schedule time when you should be start running

Exercise  2: The behaviour of background jobs in RHEL7 in case terminal session is terminated.

When you use & you disconnect the job from the terminal. So ending terminal session in RHEL 7 does not kill this background job.  In earlier versions of RHEL background processes were all killed when the shell they were started from was terminated. To prevent that, the process could be started with the nohup  command in front of it. Using nohup  for this purpose is no longer needed in RHEL 7.

Let's check this in this exercise.

1. Type dd if=/dev/zero of=/dev/null &.

2. Open the second terminal and type top. You will see dd job among the top jobs running (this job simply waste a lot of CPU time)
3. Now exit from the first terminal. You will see on the second terminal that the job is still running and is present in top listing -- exiting terminal session does not kill background jobs, which are already disconnected from the terminal.

4. From the top screen use k  to kill the dd job.

   Use Ctrl-C to exit top

Command-Line Tools for Process Management

On a Linux server, many processes are usually running. On an average server or desktop computer, there are often more than a hundred active processes. With so many processes being active, things may go wrong. If that happens, it is good to know how noninteractive processes can be stopped, or how the priority of these processes can be adjusted to make more system resources available for other processes.

Tasks on Linux are typically started as processes. One process can start several worker threads. Working with threads makes sense, because if the process is very busy, the threads can be handled by different CPUs or CPU cores available in the machine. As a Linux administrator, you cannot manage individual threads; you can manage processes, though. It is the programmer of the multithreaded application that has to define how threads relate to one another.

Before talking about different ways to manage processes, it is good to know that there are two different types of background processes. To start, there are kernel threads. These are a part of the Linux kernel, and each of them is started with its own process identification number (PID). When managing processes, it is easy to recognize the kernel processes because they have a name that is between square brackets. For example here is the result off  execution of the command  ps aux | head   which list forst dozen of kernel threads

[0]d620@ROOT:~ # ps aux | head
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.3 128160  6716 ?        Ss   Mar01   0:18 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
root         2  0.0  0.0      0     0 ?        S    Mar01   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    Mar01   0:08 [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S<   Mar01   0:00 [kworker/0:0H]
root         7  0.0  0.0      0     0 ?        S    Mar01   0:00 [migration/0]
root         8  0.0  0.0      0     0 ?        S    Mar01   0:00 [rcu_bh]
root         9  0.0  0.0      0     0 ?        S    Mar01   0:14 [rcu_sched]
root        10  0.0  0.0      0     0 ?        S<   Mar01   0:00 [lru-add-drain]
root        11  0.0  0.0      0     0 ?        S    Mar01   0:04 [watchdog/0]

As you see in RHEL7 the process with  PID 1 is systemd. Kernel threads cannot be managed. 

Using ps to get list of processes running

The most common command to get an overview of currently running processes is ps. If used without any arguments, the ps  command shows only those processes that have been started by the current user. You can use many different options to display different process properties. If you are looking for a short summary of the active processes, use ps aux  (as you saw in Listing 9.1). If you are not only looking for the name of the process but also for the exact command that was used to start the process, use ps -ef. Alternative ways to use ps  exist as well, such as the command ps fax, which shows hierarchical relationships between parent and child processes

TIP: Use ps -ef  to See the Exact Command Used to Start Processes

Click here to view code image

[0]d620@ROOT:~ # ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Jan17 ?        00:00:43 /usr/lib/systemd/systemd --switched-root --sy

... ... ...

Note

For some commands, using a hyphen before options is optional. Some commands do not. The ps  command is one of those latter commands. There is a historical reason why this is the case: The commands derive from the old BSD UNIX flavor, where it was common to specify command-line options without a - in front of them.

Use ps fax  to Show Parent-Child Relationships Between Processes

[0]d620@ROOT:~ # ps fax
  PID TTY      STAT   TIME COMMAND
    2 ?        S      0:00 [kthreadd]
    3 ?        S      0:16  \_ [ksoftirqd/0]
    5 ?        S<     0:00  \_ [kworker/0:0H]
    7 ?        S      0:00  \_ [migration/0]
    8 ?        S      0:00  \_ [rcu_bh]
  ... ... ...
 4217 ?        Ss     0:15 /usr/libexec/postfix/master -w
 4219 ?        S      0:02  \_ qmgr -l -t unix -u
 6450 ?        S      0:00  \_ pickup -l -t unix -u
10647 ?        Ss     0:00 /usr/sbin/sshd -D
 6501 ?        Ss     0:00  \_ sshd: root@pts/0
 6504 pts/0    Ss     0:00      \_ -bash
 6536 pts/0    R+     0:00          \_ ps fax
 

An important piece of information to get out of the ps  command is the PID. Many tasks require the PID to operate, and that is why a command like ps aux | grep dd, which will show process details about dd, including its PID, is quite common. An alternative way to get the same result is to use the pgrep  command. Use pgrep dd  to get a list of all PIDs that have a name containing the string dd.

Changing process priority with nice

As we have now multicore servers, nice is not longer that important as it was in the past.

When Linux processes are started, they are started with a specific priority. By default, all regular processes are equal and are started with the same priority, which is the priority number 20. In some cases, it is useful to change the default priority that was assigned to the process when it was started. You can do that using the nice  and renice  commands. Use nice  if you want to start a process with an adjusted priority. Use renice  to change the priority for a currently active process.

Alternatively, you can use the r  command from the top utility to change the priority of a currently running process.

Changing process priority may make sense in two different scenarios. Suppose, for example, that you are about to start a backup job that does not necessarily have to finish fast. Typically, backup jobs are rather resource intensive, so you might want to start it in a way that it is not annoying other users too much, by lowering its priority.

Another example is where you are about to start a very important calculation job. To ensure that it is handled as fast as possible, you might want to give it an increased priority, taking away CPU time from other processes.

On earlier Linux versions, it could be dangerous to increase the priority of one job too much, because other processes (including vital kernel processes) might risk being blocked out completely. On current Linux kernels, the situation is not that urgent anymore:

• Modern Linux kernels differentiate between essential kernel threads that are started as real-time processes and normal user processes. Increasing the priority of a user process will never be able to block out kernel threads or other processes that were started as real-time processes.
• Modern computers have multiple CPU cores. A single threaded process that is running with the highest priority will never be able to get beyond the boundaries of the CPU it is running on.

When using nice  or renice  to adjust process priority, you can select from values ranging from -20 to 19. The default niceness of a process is set to 0 (which results in the priority value of 20). By applying a negative niceness, you increase the priority. Use a positive niceness to decrease the priority. It is a good idea not to use the ultimate values immediately. Instead, use increments of 5 and see how it affects the application.

TIP:

Do not set process priority to -20; it risks blocking other processes from getting served.

Let’s take a look at examples of how to use nice  and renice. The command nice -n 5 dd if=/dev/zero of=/dev/null &  starts an infinite I/O-intensive job, but with an adjusted niceness so that some place remains for other processes as well. To adjust the niceness of a currently running process, you need the PID of that process. The following two commands show how ps aux  is used to find the PID of the dd job from the previous example. Next, you see how the renice  command is used to change the niceness of that command:

1.  Use ps aux | grep dd  to find the PID of the dd command that you just started. The PID is in the second column of the command output.

2.  Use renice -n 10 -p 1234  (assuming that 1234 is the PID you just found).

Note that regular users can only decrease the priority of a running process. You must be root to give processes increased priority.

Sending signals to processes with kill, killall, and pkill

Before starting to think about using the kill  command or sending other signals to processes, it is good to know that Linux processes have a hierarchical relationship. Every process has a parent process, and as long as it lives, the parent process is responsible for the child processes it has created. This is particularly important when processes are terminated, because it is the parent that has to clean up the resources that were used by the children. When using kill on a parent process that still has active children, you will for that reason not just kill the parent process in question but also all of its currently active child processes.

The Linux kernel allows many signals to be sent to processes. Use man 7 signals  for a complete overview of all the available signals. Three of these signals work for all processes:

• SIGTERM (15) is used to ask a process to stop.
• SIGKILL (9) is used to force a process to stop.
• SIGHUP (1) signal is used to hang up a process. The effect is that the process will reread its configuration files, which makes this a useful signal to use after making modifications to a process configuration file.

To send a signal to a process, the kill  command is used. The most common use is the need to stop a process, which you can do by using the kill  command followed by the PID of the process. This sends the SIGTERM signal to the process, which normally causes the process to cease its activity.

Sometimes the kill  command does not work because the process you want to kill is busy. In that case, you can use kill -9  to send the SIGKILL signal to the process. Because the SIGKILL signal cannot be ignored, it forces the process to stop, but you also risk losing data while using this command. In general, it is a bad idea to use kill -9:

•  You risk losing data.

•  Your system may become unstable if other processes depend on the process you have just killed.

Tip

 kill -l  shows a list of available signals that can be used with kill.

There are two commands that are related to kill: killall  and pkill. The pkill  command easier to use because it takes the name rather than the PID of the process as an argument. You can use the killall  command if multiple processes using the same name need to be killed simultaneously.

Using killall  was particularly common when Linux environments were multiprocessing instead of multithreading. In a multiprocessing environment where a server starts several commands, all with the same name, it is not easy to stop these commands one by one based on their individual PID. Using killall  enables you to terminate all these processes simultaneously.

In a multithreaded environment typically one process that is generating several threads. So all these threads are terminated anyway by stopping the process that started them.

You still can use killall, though, to terminate lots of processes with the same name that have been started on your server.

Exercise: managing processes

In this exercise, you learn how to work with ps, nice, kill, and related utilities to manage processes.

1.  Open a root shell. From this shell, type dd if=/dev/zero of=/dev/null &. Repeat this command three times.

2.  Type ps aux | grep dd. This shows all lines of output that have the letters dd in them; you will see more than just the dd processes, but that should not really matter. The processes you just started are listed last.

3.  Use the PID of one of the dd processes to adjust the niceness, using renice -n 5 <PID>. Notice that in top you cannot easily get an overview of processes and their current priority.

4.  Type ps fax | grep -B5 dd. The -B5  option shows the matching lines, including the five lines before that. Because ps fax  shows hierarchical relationships between processes, you should also find the shell and its PID from which all the dd processes were started.

5.  Find the PID of the shell from which the dd processes were started and type kill -9 <PID>, replacing <PID>  with the PID of the shell you just found. As the dd processes were started as background processes, they are not killed when their parent shell was killed. Instead, they have been moved upwards and are now a child of the systemd process.

Using top to Manage Processes

A convenient tool to manage processes is top. For common process management tasks, top is so great because it gives an overview of the most active processes currently running (hence the name top). This enables you to easily find processes that might need attention. From top, you can also perform common process management tasks, such as adjusting the current process priority and killing processes.

Among the information that you can conveniently obtain from the top utility is the process state.

Now that you know how to use the kill  and nice  commands from the command line, using the same functionality from top is even easier. From top, type k. top will then prompt for the PID of the process you want to send a signal to. By default, the most active process is selected. After you enter the PID, top asks which signal you want to send. By default, signal 15 for SIGTERM is used. However, if you want to insist a bit more, you can type 9  for SIGKILL. Now press Enter  to terminate the process.

To renice a running process from top, type r. You are first prompted for the PID of the process you want to renice. After entering the PID, you are prompted for the nice value you want to use. Enter a positive value to increase process priority or a negative value to decrease process priority.

Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Sites

• Understanding Linux Process States, Red Hat 2012
• Linux Process States - Stack Overflow

Top

• top(1) tasks - Linux man page
• How to Use the Top Command to Show Running Processes
• Linux ps command help and examples

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: December 13, 2020