Using extended regular expressions with grep

Dr. Nikolai Bezroukov

Special Characters

Here, we outline the special characters for grep. Note that in egrep (which uses extended regular expressions), which actually are no more functional than standard regular expressions if you use GNU grep ) , the list of special characters increases ( | in grep is the same as \| egrep and vice versa, there are also other differences. Check the man page for details ) The following characters are considered special and need to be "escaped":

?  \  .  [  ]  ^  $

Note that a $ sign loses its meaning if characters follow it (I think) and the carat ^ loses its meaning if other characters precede it.
Square brackets behave a little differently. The rules for square brackets go as follows:

• A closing square bracket loses its special meaning if placed first in a list. for example []12] matches ] , 1, or 2.
• A dash - loses it's usual meaning inside lists if it is placed last.
• A carat ^ loses it's special meaning if it is not placed first
• Most special characters lose their meaning inside square brackets

Quotes

Single quotes are the safest to use, because they protect your regular expression from the shell. For example, grep ! file  will often produce an error (since the shell thinks that "!" is referring to the shell command history) while grep '!' file  will not.

When should you use single quotes ?

The answer is this: if you want to use shell variables, you need double quotes; otherwise always use single quotes.

For example,

egrep "$HOME" file 

searches file for the name of your home directory, while

egrep '$HOME' file 

searches for the string $HOME

Comparison

In complex cases it's always easier to use Perl than to explore intricacies of grep syntax. In any case I strongly recommend you to use option -P if it is available. that's the best way to preserve your sanity.

Grep Regular Expressions Warts

In basic regular expressions the metacharacters `?', `+', `{', `|', `(', and `)' should be backslashed `\?', `\+', `\{', `\|', `\(', and `\)'. As GNU grep manual stated:

Traditional egrep  did not support the `{' metacharacter, and some egrep  implementations support `\{' instead, so portable scripts should avoid `{' in `egrep' patterns and should use `[{]' to match a literal `{'.

GNU egrep  attempts to support traditional usage by assuming that `{' is not special if it would be the start of an invalid interval specification. For example, the shell command

egrep '{1'

searches for the two-character string '{1' instead of reporting a syntax error in the regular expression. POSIX.2 allows this behavior as an extension...

To search for a line containing text hello.gif, the correct command is

grep  hello\\.gif file 

or

grep -P 'hello\.gif' file

More on Regular Expressions

To match a selection of characters, use []. This is often used to make search less case sensitive:

[Hh]ello

Ranges of characters are also permitted.

• [0-3]  is the same as [0123]
• [a-k]  is the same as [abcdefghijk]
• [A-C]  is the same as [ABC]
• [A-Ca-k]  is the same as [ABCabcdefghijk]

There are also some alternate forms :

[[:alpha:]] is the same as [a-zA-Z]
[[:upper:]] is the same as [A-Z]
[[:lower:]] is the same as [a-z]
[[:digit:]] is the same as [0-9]
[[:alnum:]] is the same as [0-9a-zA-Z]
[[:space:]] matches any white space including tabs

Backreferences

Suppose you want to search for a string which contains a certain substring in more than one place. An example is the heading tag in HTML. Suppose I wanted to search for <H1>some string</H1>  . This is easy enough to do. But suppose I wanted to do the same but allow H2 H3 H4 H5 H6  in place of H1. The expression <H[1-6]>.*</H[1-6]>  is not good enough since it matches <H1>Hello world</H3>  but we want the opening tag to match the closing one. To do this, we use a backreference

The expression \n where n is a number, matches the contents of the n'th set of parentheses in the expression

\<H\([1-6]\).*</H\1>

ep to print the name of the file apprend /dev/null to the list of files:

find . -type f -print | xargs fgrep -l 'hahaha'  '{}'  /dev/null 

grep -P -i 'href="[^"]* [^"]*"' *.html

Webliography

• GNU grep 2.5g manual
• Grep is fundamental
• GREP for Linguists by: Stuart Robinson.
• Grep tutorial by Donovan Rebbechi (Rutgers University, NJ) See also adaptation at grep_tutorial
• SimICS Tutorial grep
• Steve Ramsay's Guide to Regular Expressions
• Regular Expression Tutorial
• [PDF] Tutorial: Using regular expressions
• How Regexes Work
• ONLamp.com Finding Things in Unix

Appendix

grep, print lines matching a pattern 2.1 GNU Extensions

GNU Extensions

`-A num'

`--after-context=num'

Print num lines of trailing context after matching lines.

`-B num'

`--before-context=num'

Print num lines of leading context before matching lines.

`-C num'

`--context=num'

Print num lines of output context.

`--colour[=WHEN]'

`--color[=WHEN]'

The matching string is surrounded by the marker specify in GREP_COLOR. WHEN may be `never', `always', or `auto'.

`-num'

Same as `--context=num' lines of leading and trailing context. However, grep will never print any given line more than once.

`-V'

`--version'

Print the version number of grep  to the standard output stream. This version number should be included in all bug reports.

`--help'

Print a usage message briefly summarizing these command-line options and the bug-reporting address, then exit.

`--binary-files=type'

If the first few bytes of a file indicate that the file contains binary data, assume that the file is of type type. By default, type is `binary', and grep  normally outputs either a one-line message saying that a binary file matches, or no message if there is no match. If type is `without-match', grep  assumes that a binary file does not match; this is equivalent to the `-I' option. If type is `text', grep  processes a binary file as if it were text; this is equivalent to the `-a' option. Warning: `--binary-files=text' might output binary garbage, which can have nasty side effects if the output is a terminal and if the terminal driver interprets some of it as commands.

`-b'

`--byte-offset'

Print the byte offset within the input file before each line of output. When grep  runs on MS-DOS or MS-Windows, the printed byte offsets depend on whether the `-u' (`--unix-byte-offsets') option is used; see below.

`-D action'

`--devices=action'

If an input file is a device, FIFO or socket, use action to process it. By default, action is `read', which means that devices are read just as if they were ordinary files. If action is `skip', devices, FIFOs and sockets are silently skipped.

`-d action'

`--directories=action'

If an input file is a directory, use action to process it. By default, action is `read', which means that directories are read just as if they were ordinary files (some operating systems and filesystems disallow this, and will cause grep  to print error messages for every directory or silently skip them). If action is `skip', directories are silently skipped. If action is `recurse', grep  reads all files under each directory, recursively; this is equivalent to the `-r' option.

`-H'

`--with-filename'

Print the filename for each match.

`-h'

`--no-filename'

Suppress the prefixing of filenames on output when multiple files are searched.

`--line-buffered'

Set the line buffering policy, this can be a performance penality.

`--label=LABEL'

Displays input actually coming from standard input as input coming from file LABEL. This is especially useful for tools like zgrep, e.g. gzip -cd foo.gz |grep --label=foo something

`-L'

`--files-without-match'

Suppress normal output; instead print the name of each input file from which no output would normally have been printed. The scanning of every file will stop on the first match.

`-a'

`--text'

Process a binary file as if it were text; this is equivalent to the `--binary-files=text' option.

`-I'

Process a binary file as if it did not contain matching data; this is equivalent to the `--binary-files=without-match' option.

`-w'

`--word-regexp'

Select only those lines containing matches that form whole words. The test is that the matching substring must either be at the beginning of the line, or preceded by a non-word constituent character. Similarly, it must be either at the end of the line or followed by a non-word constituent character. Word-constituent characters are letters, digits, and the underscore.

`-r'

`-R'

`--recursive'

For each directory mentioned in the command line, read and process all files in that directory, recursively. This is the same as the `--directories=recurse' option.

`--include=file_pattern'

When processing directories recursively, only files matching file_pattern will be search.

`--exclude=file_pattern'

When processing directories recursively, skip files matching file_pattern.

`-m num'

`--max-count=num'

Stop reading a file after num matching lines. If the input is standard input from a regular file, and num matching lines are output, grep  ensures that the standard input is positioned to just after the last matching line before exiting, regardless of the presence of trailing context lines. This enables a calling process to resume a search. For example, the following shell script makes use of it:

while grep -m 1 PATTERN
do
  echo xxxx
done < FILE

But the following probably will not work because a pipe is not a regular file:

# This probably will not work.
cat FILE |
while grep -m 1 PATTERN
do
  echo xxxx
done

When grep  stops after NUM matching lines, it outputs any trailing context lines. Since context does not include matching lines, grep  will stop when it encounters another matching line. When the `-c' or `--count' option is also used, grep  does not output a count greater than num. When the `-v' or `--invert-match' option is also used, grep  stops after outputting num non-matching lines.

`-y'

Obsolete synonym for `-i'.

`-U'

`--binary'

Treat the file(s) as binary. By default, under MS-DOS and MS-Windows, grep  guesses the file type by looking at the contents of the first 32kB read from the file. If grep  decides the file is a text file, it strips the CR  characters from the original file contents (to make regular expressions with ^  and $  work correctly). Specifying `-U' overrules this guesswork, causing all files to be read and passed to the matching mechanism verbatim; if the file is a text file with CR/LF  pairs at the end of each line, this will cause some regular expressions to fail. This option has no effect on platforms other than MS-DOS and MS-Windows.

`-u'

`--unix-byte-offsets'

Report Unix-style byte offsets. This switch causes grep  to report byte offsets as if the file were Unix style text file, i.e., the byte offsets ignore the CR  characters which were stripped. This will produce results identical to running grep  on a Unix machine. This option has no effect unless `-b' option is also used; it has no effect on platforms other than MS-DOS and MS-Windows.

`--mmap'

If possible, use the mmap  system call to read input, instead of the default read  system call. In some situations, `--mmap' yields better performance. However, `--mmap' can cause undefined behavior (including core dumps) if an input file shrinks while grep  is operating, or if an I/O error occurs.

`-Z'

`--null'

Output a zero byte (the ASCII NUL  character) instead of the character that normally follows a file name. For example, `grep -lZ' outputs a zero byte after each file name instead of the usual newline. This option makes the output unambiguous, even in the presence of file names containing unusual characters like newlines. This option can be used with commands like `find -print0', `perl -0', `sort -z', and `xargs -0' to process arbitrary file names, even those that contain newline characters.

`-z'

`--null-data'

Treat the input as a set of lines, each terminated by a zero byte (the ASCII NUL  character) instead of a newline. Like the `-Z' or `--null' option, this option can be used with commands like `sort -z' to process arbitrary file names.

Several additional options control which variant of the grep  matching engine is used. See section 4. grep  programs.

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Created: May 16, 1997; Last modified: March 12, 2019