Linux Kernel Tuning via /proc system

Linux copied Solaris /proc system and extended its functionality as the performance tuning center.  Tuning is usually accomplished via system control variables  stored in /proc/sys. Unlike most other areas of /proc, those variables are typically writable, and are used to adjust the running kernel rather than simply monitor currently running processes and system information. 

The sysctl interface allows administrators to modify variables that the kernel uses to determine behavior. There are two ways to work with sysctl: by directly reading and modifying files in /proc/sys and by using the sysctl program supplied with most distributions. Most documentation on sysctl accesses variables via the /proc/sys file system, and does so using cat for viewing and echo for changing variables, as shown in the following example where IP forwarding is enabled:

# cat /proc/sys/net/ipv4/ip_forward
0
# echo "1" > /proc/sys/net/ipv4/ip_forward
# cat /proc/sys/net/ipv4/ip_forward
1

To start getting a taste of what sysctl can modify, run sysctl -a and you will see all the possible parameters. The list can be quite long: in my current box there are 712 possible settings.

sysctl vm will list all variables that start with "vm."

The -n option to output just the variable values, without the names; -N has the opposite effect, and produces the names but not the values.

You can change any variable by using the -w option with the syntax sysctl -w variable =value. For example, sysctl -w net.ipv6.conf.all.forwarding=1 sets the corresponding variable to true (0 equals "no" or "false"; 1 means "yes" or "true") thus allowing IP6 forwarding.

For more information, run man sysctl  and the post Using Sysctl To Change Kernel Tunables On Linux:

...sysctl is a very versatile command and can be used either in its standalone form, or through the modification of the /etc/sysctl.conf file. First, we'll take a brief look at what the sysctl standalone command can do. It doesn't have too many options, so explaining them quickly upfront will make the rest seem like it makes more sense ;) You can run sysctl with the following flags (maybe more, depending on your distro):

-a to display all the tunable key values currently available
-A to display all the tunable key values currently available, as well as table values
-e to ignore errors (specifically pertaining to unrecognized characters)
-n to "not" print the key names when printing out values
-N to "only" print the key names and forgo printing their values
-p (sometimes -P) to import and apply settings from a specified file. This option will use /etc/sysctl.conf as the default if no file name argument is provided on the command line
-q for your standard quiet mode
-w to change kernel tunable (sysctl) settings - This will make the change in real time, as well as update the /etc/sysctl.conf file

and two more "special" arguments:

variablename <-- Use this on its own to read a key from sysctl matching your variablename
variablename=value <-- Use this to set a variablename (key) to a specific value. Note that this needs to be used with the -w flag (which changes sysctl settings)

Some basic examples of sysctl's use would include:

host # sysctl -a <-- This will produce a huge list of output. The basic format would be: NAME TYPE CHANGEABLE - with each column's name accurately depicting what it represents. For instance you could get an entry with a NAME of "kernel.hostid" and a TYPE of "u_int" (note that this is the datatype) and a notation on whether or not it's CHANGEABLE - in this instance "yes" The changeable field can also return "no" and "raise only" It seems logical to assume that it could return "lower only" as well, but I've yet to see it.

SPECIAL ERRATA NOTICE: If sysctl -a spews a lot of kernel warnings, check out Advisory RHBA-2008:0020-4 on RedHat's website for a patch to fix that issue.

host # sysctl -p /etc/mytestsysctl.conf <-- this will read in and enact all the kernel changes specified in your special /etc/mytestsysctl.conf file. It's a good idea to use a different filename when testing out new sysctl.conf settings, especially if you're making broad changes, since, if you completely screw the pooch and your machine reboots, it will come back up looking for the default /etc/sysctl.conf which will still be good to go)

host # sysctl -p <-- Use the -p option without any arguments if you've made adjustments to your sysctl.conf file and want to reload it, or just to be sure that it is actually being read (if you have serious doubts, you can run "strace -f /sbin/sysctl -p" to get more granular information. If you find that you do need to use strace to run down a problem with sysctl, hopefully our previous post on using strace to debug application issues will help get you off on the right foot and an expedited solution)

host # sysctl -w kernel.hostname="Error.Dumping.core" <-- this will set the hostname of your machine to something that might possibly be amusing. Please ensure that your superiors (or the folks you work for) have a sense of humor before pulling a stunt like this and walking away ;)

It's interesting, also, to note that, while sysctl will work just fine with an /etc/sysctl.conf file that includes nothing but comments (or is completely non-existent), your /proc filesystem "must" be of the type "procfs" in order for it to function correctly. This is picking a nit, really, since you'd have to go out of your way to build your RedHat Linux box to use (for instance) ext3 for the /proc filesystem, but a bit of information that's good to know (maybe... at some point in the future ;) /proc/sys is the base directory for sysctl. In fact, if you wanted to emulate "sysctl -a", you could just do an ls in that directory.

Tomorrow, or sometime later this week, we'll take a look at some of the kernel tunables you'll probably want to change, or may have to modify, most often with sysctl and, with as even a hand as possible, debate the pro's and con's of some of the more "impactful" values that you can mess with.

NEWS CONTENTS

• 20080909 : Linux.com Kernel tuning with sysctl by Federico Kereki ( Linux.com Kernel tuning with sysctl, Sep 09, 2008 )
• 20080909 : Getting somewhere? ( )
• 20080909 : Kernel tuning with sysctl ( )

Old News

[Sep 09, 2008] Linux.com Kernel tuning with sysctl by Federico Kereki

The Linux kernel is flexible, and you can even modify the way it works on the fly by dynamically changing some of its parameters, thanks to the sysctl command. Sysctl provides an interface that allows you to examine and change several hundred kernel parameters in Linux or BSD. Changes take effect immediately, and there's even a way to make them persist after a reboot. By using sysctl judiciously, you can optimize your box without having to recompile your kernel, and get the results immediately.

To start getting a taste of what sysctl can modify, run sysctl -a and you will see all the possible parameters. The list can be quite long: in my current box there are 712 possible settings.

If you want to get the value of just a single variable, use something like sysctl vm.swappiness, or just sysctl vm to list all variables that start with "vm." Add the -n option to output just the variable values, without the names; -N has the opposite effect, and produces the names but not the values.

You can change any variable by using the -w option with the syntax sysctl -w variable=value. For example, sysctl -w net.ipv6.conf.all.forwarding=1 sets the corresponding variable to true (0 equals "no" or "false"; 1 means "yes" or "true") thus allowing IP6 forwarding. You may not even need the -w option -- it seems to be deprecated. Do some experimenting on your own to confirm that.

For more information, run man sysctl to display the standard documentation.

sysctl and the /proc directory

The /proc/sys virtual directory also provides an interface to the sysctl parameters, allowing you to examine and change them. For example, the /proc/sys/vm/swappiness file is equivalent to the vm.swappiness parameter in sysctl.conf; just forget the initial "/proc/sys/" part, substitute dots for the slashes, and you get the corresponding sysctl parameter. (By the way, the substitution is not actually required; slashes are also accepted, though it seems everybody goes for the notation with the dots instead.) Thus, echo 10 >/proc/sys/vm/swappiness is exactly the same as sysctl -w vm.swappiness=10. But as a rule of thumb, if a /proc/sys file is read-only, you cannot set it with sysctl either.

sysctl values are loaded at boot time from the /etc/sysctl.conf file. This file can have blank lines, comments (lines starting either with a "#" character or a semicolon), and lines in the "variable=value" format. For example, my own sysctl.conf file is listed below. If you want to apply it at any time, you can do so with the command sysctl -p.

# Disable response to broadcasts. net.ipv4.icmp_echo_ignore_broadcasts = 1 # enable route verification on all interfaces net.ipv4.conf.all.rp_filter = 1 # enable ipV6 forwarding net.ipv6.conf.all.forwarding = 1 # increase the number of possible inotify(7) watches fs.inotify.max_user_watches = 65536

Getting somewhere?

With so many tunable parameters, how do you decide what to do? Alas, this is a sore point with sysctl: most of the relevant documentation is hidden in the many source files of the Linux kernel, and isn't easily available, and it doesn't help that the explanations given are sometime arcane and difficult to understand. You may find something in the /usr/src/linux/Documentation/sysctl directory, but most (if not all) files there refer to kernel 2.2, and seemingly haven't been updated in the last several years.

Looking around for books on the subject probably won't help much. I found hack #71 in O'Reilly's Linux Server Hacks, Volume 2, from 2005, but that was about it. Several other books include references to sysctl, but as to specific parameters or hints, you are on your own.

As an experiment, I tried looking for information on the swappiness parameter, which can optimize virtual memory management. The /usr/src/Linux/Documentation/sysctl/vm.txt file didn't even refer to it, probably because this parameter appeared around version 2.6 of the kernel. Doing a general search in the complete /usr/src/linux directory turned up five files that mention "swappiness": three "include" (.h) files in include/linux, plus kernel/sysctl.c and mm/vmscan.c. The latter file included the information:

/* * From 0 .. 100. Higher means more swappy. */ int vm_swappiness = 60;

That was it! You can see the default value (60) and a minimal reference to the field meaning. How helpful is that?

My suggestion would be to use sysctl -a to learn the available parameters, then Google around for extra help. You may find, say, an example of changing the shared memory allocation to solve a video program problem, or an explanation on vm.swappiness, or even more suggestions for optimizing IP4 network traffic.

sysctl shows yet another aspect of the great flexibility of Linux systems. While documentation for it is not widely available, learning its features and capabilities on your own can help you get even more performance out of your box. That's system administration at its highest (or lowest?) level.

Read in the original layout at: http://www.linux.com/feature/146599

Kernel tuning with sysctl

Posted by: Anonymous [ip: 96.14.205.198] on September 09, 2008 05:26 PM

There won't be a whole book on sysctl anytime soon--it would be about 14 pages, including title, copyright, TOC, and index. The actual parameters have changed dramatically over time, not only in availability, but also in interpretation. A setting that's valid in a 2.4 kernel might be gone in 2.6, or might have a different set of valid values.

Recommended Links

• Using Sysctl To Change Kernel Tunables On Linux(Jul 29, 2008)
• Tuning the Linux Kernel for Better Network Throughput(Jun 24, 2007)
• Kernel tuning with sysctl Sep 09, 2008
• Use LatencyTOP to find out where process latency is coming from Sep 05, 2008

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019