Awk

AWK  is a simple and elegant pattern scanning and processing language. I would call it the first and last simple scripting language. AWK is also the most portable scripting language in existence. It's the precursor and the main inspiration of Perl. Although originated in Unix it is available and widely used in Windows environment too.

It was created in late 70th of the last century almost simultaneously with Borne shell. The name was composed from the initial letters of three original authors Alfred V. Aho, Brian W. Kernighan, and Peter J. Weinberger. The team was more talented then Stephen Bourne and produced higher quality product. Unfortunately it was never well integrated into the shell. It is commonly used as a command-line filter in pipes to reformat the output of other commands.

AWK takes two inputs: data file and command file. The command file can be absent and necessary commands can be passed as augments. As Ronald P. Loui aptly noted awk is very underappreciated language:

Most people are surprised when I tell them what language we use in our undergraduate AI programming class. That's understandable. We use GAWK. GAWK, Gnu's version of Aho, Weinberger, and Kernighan's old pattern scanning language isn't even viewed as a programming language by most people. Like PERL and TCL, most prefer to view it as a "scripting language." It has no objects; it is not functional; it does no built-in logic programming. Their surprise turns to puzzlement when I confide that (a) while the students are allowed to use any language they want; (b) with a single exception, the best work consistently results from those working in GAWK. (footnote: The exception was a PASCAL programmer who is now an NSF graduate fellow getting a Ph.D. in mathematics at Harvard.) Programmers in C, C++, and LISP haven't even been close (we have not seen work in PROLOG or JAVA).

The main advantage of AWK is that unlike Perl and other "scripting monsters" that it is very slim without feature creep so characteristic of Perl and thus it can be very efficiently used with pipes. Also it has rather simple, clean syntax and like much heavier TCL can be used with C for "dual-language" implementations.

Generally Perl might be better for really complex tasks, but this is not always the case. In reality AWK much better integrates with Unix shell and until probably in 2004 for simple scripts there was no noticeable difference in speed due to the additional time to load and initialize huge Perl interpreter (but Perl 5 still grows and is now looks slim for a typical PC with dual core 3GHz CPU and 2GB of RAM or server, which typically has at least four core CPU and 6GB or more of RAM).

Unfortunately, Larry Wall then decided to throwing in the kitchen sink, and as a side effect sacrificed the simplicity and orthogonally. I would agree that Perl added some nice things, but it probably added too much nice things :-). Perl4 can probably be used as AWK++ but it's not that portable or universally supported. Like I mentioned above, AWK is the most portable scripting language in existence. 

IMHO the original book that describes AWK ( Alfred V. Aho, Brian W. Kernighan, and Peter J. Weinberger The Awk Programming Language, Addison-Wesley, 1988.) can serve as an excellent introduction into scripting. One chapter is available free Chapter 11 The awk Programming Language

AWK has a unique blend of simplicity and power that is especially attractive for novices, who do not have to spend days and weeks learning all those intricacies of Perl before they become productive. In awk you can became productive in several hours. For instance, to print only the second and sixth fields of the date command--the month and year--with a space separating them, use:

  date | awk '{print $2 " " $6}'

The GNU Project produced the most popular version of awk, gawk. gawk has precompiled binaries for MS-DOS and Win32. It has some interesting and useful enhancement. File can be read under control of powerful getline function.  Unlike other implementation GNU AWL contains the dgawk debugger is purposely modeled after GDB.  GNU AWK 4.0 and higher has  "--sandbox" option disables the call of system() and write access to the file system.

NEWS CONTENTS

• 20171031 : Elegant Awk usage by Tom Ryder ( Feb 06, 2012 , sanctum.geek.nz )
• 20171031 : UNIX AWK and SED Programmers Interactive Workbook (UNIX Interactive Workbook) by Peter Patsis ( UNIX AWK and SED Programmer's Interactive Workbook (UNIX Interactive Workbook), )

Old News ;-)

[Oct 31, 2017] Elegant Awk usage by Tom Ryder

It's better to use Perl for this pupose...

Feb 06, 2012 | sanctum.geek.nz

For many system administrators, Awk is used only as a way to print specific columns of data from programs that generate columnar output, such as netstat or ps .

For example, to get a list of all the IP addresses and ports with open TCP connections on a machine, one might run the following:

# netstat -ant | awk '{print $5}'

This works pretty well, but among the data you actually wanted it also includes the fifth word of the opening explanatory note, and the heading of the fifth column:

and
Address
0.0.0.0:*
205.188.17.70:443
172.20.0.236:5222
72.14.203.125:5222

There are varying ways to deal with this.

Matching patterns

One common way is to pipe the output further through a call to grep , perhaps to only include results with at least one number:

# netstat -ant | awk '{print $5}' | grep '[0-9]'

In this case, it's instructive to use the awk call a bit more intelligently by setting a regular expression which the applicable line must match in order for that field to be printed, with the standard / characters as delimiters. This eliminates the need for the call to grep :

# netstat -ant | awk '/[0-9]/ {print $5}'

We can further refine this by ensuring that the regular expression should only match data in the fifth column of the output, using the ~ operator:

# netstat -ant | awk '$5 ~ /[0-9]/ {print $5}'

Skipping lines

Another approach you could take to strip the headers out might be to use sed to skip the first two lines of the output:

# netstat -ant | awk '{print $5}' | sed 1,2d

However, this can also be incorporated into the awk call, using the NR variable and making it part of a conditional checking the line number is greater than two:

# netstat -ant | awk 'NR>2 {print $5}'

Combining and excluding patterns

Another common idiom on systems that don't have the special pgrep command is to filter ps output for a string, but exclude the grep process itself from the output with grep -v grep :

# ps -ef | grep apache | grep -v grep | awk '{print $2}'

If you're using Awk to get columnar data from the output, in this case the second column containing the process ID, both calls to grep can instead be incorporated into the awk call:

# ps -ef | awk '/apache/ && !/awk/ {print $2}'

Again, this can be further refined if necessary to ensure you're only matching the expressions against the command name by specifying the field number for each comparison:

# ps -ef | awk '$8 ~ /apache/ && $8 !~ /awk/ {print $2}'

If you're used to using Awk purely as a column filter, the above might help to increase its utility for you and allow you to write shorter and more efficient command lines. The Awk Primer on Wikibooks is a really good reference for using Awk to its fullest for the sorts of tasks for which it's especially well-suited.

UNIX AWK and SED Programmer's Interactive Workbook (UNIX Interactive Workbook) by Peter Patsis

Paperback: 500 pages
Publisher: Prentice Hall PTR; Workbook edition (December 30, 1998)
Language: English
ISBN-10: 0130826758
ISBN-13: 978-0130826756
Product Dimensions: 9.2 x 7 x 1.8 inches

Customer

The Best AWK and UNIX training manual! June 12, 2000

As a computer consultant, I sometimes need to train 'folks' that haven't had too much time in front of a computer. For those souls that have grown used to Windows, UNIX has the potential to strike fear in a way that I've seldom seen duplicated. This text offers the BEST environment for teaching UNIX and AWK that I have seen.

There are UNIX and AWK reference books galore, but none of them take a systematic approach to TEACHING the subject, (no, not even teach yourself, or the Dummies versions). This book reviews each command and ends with a generous question section. (Not just one or two questions at a high level, and not just one really complicated question - the questions build very nicely to the hardest issues.)

If you are looking for a reference book, this will do, though there are others that I would prefer, but if you are learning UNIX and AWK on your own, or looking to teach it to someone else, this book gets my highest recommendation. I have given it now to everyone I have trained and all have had great success with it. I would feel entirely comfortable handing this book to a trainee, and with no further instruction allowing them to code when they've finished. " An absolute gift"

Recommended Links

Softpanorama Recommended

Top articles

Sites

**** The Awk Programming Language ~ Usually ships in 24 hours

Alfred V. Aho, et al / Paperback / Published 1988
Amazon price: $34.95

As [email protected] put it at customer Reviews at amazon.com "While the Nutshell book is more of a reference guide, Aho's shows how to really use the language with practical examples. From basic examples to flatfile reports and using Awk to try out language issues in compiler design. For those who know Awk there are some great ideas in here. For those who are just starting out it's an excellent way to ease yourself into writing Awk scripts. A awk script writers must have."

Here is web page for the book The AWK Programming Language. Some useful info:

• You can also look it up at the Addison-Wesley home page, which should eventually point you to this.
• Errata for the AWK book.
• Source for the one true awk, updated June 3, 1999. Shell archive (310K); compressed tar file (78K); zip file (90K). There is also a Windows 95 and NT executable (140K).
• Examples from The AWK Programming Language by Aho, Kernighan, and Weinberger as text (120Kb) or zipped (30Kb).
• AWK is also available from GNU (GAWK), courtesy of Arnold Robbins; MAWK by Michael Brennan; from MKS as part of the MKS Toolkit; and from Thompson Automation Software (TAWK). More AWK information can be found on the comp.lang.awk newsgroup.
  

***+ Sed & Awk (Nutshell Handbook) ~ Usually ships in 24 hours

Dale Dougherty, et al / Paperback / Published 1997
Amazon price: $23.96 ~ You Save: $5.99 (20%)

**+ Unix Awk and Sed Programmer's Interactive Workbook ~ Usually ships in 24 hours

Peter Patsis / Paperback / Published 1998
Amazon price: $27.99 ~ You Save: $7.00 (20%)

***+ Effective Awk Programming : A User's Guide ~ Usually ships in 24 hours

Arnold D. Robbins / Paperback / Published 1997
Amazon price: $27.00. Also published by the FSF as "The GNU AWK User's Guide"; Texinfo source is included with the gawk distribution, so you can also print this yourself.

The GNU Awk User's Guide ~ Usually ships in 24 hours

Arnold Robbins / Paperback / Published 1996
Amazon price: $25.00

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: February, 19, 2020