Perl Regular Expressions Best Practices and Tips

Regular expression is a sharp tool. You can cut yourself badly if you are not careful ;-)

Softpanorama Recommedations

It makes a lot of sense first to debug a complex regular expression is a special test script, feeding it with sample strings and observing the output.

Be careful when using the * quantifier because it can match an empty string, which might not be your intention. The regex /b*/ will match any string - even one without any b characters.

Meta-character  .  and modifiers ?, *, + that appear inside the square brackets that define a character class are used in their literal sense. They lose their meta-meaning.

One important feature of capture variables is that only the successful match affect them. If a match is unsuccessful, then  previous values are preserved whatever they may be. That leads to difficult to find errors if you are not careful. You should never use capture variables without checking if the march is successful or not.  What is worse you forget about this "feature" from time to time and make the same mistake again and again. And then spend a day debugging your now semi-forgotten script when you accidentally discover that it misbehaves in certain cases. I think that this is a design blunder of Perl. It should set all capture variables undefined in can of unsuccessful match. Please check your scripts for usage of capture variable and manually check in each case that if statement for matching is used.

The result of a substitution operator in the scalar context is the number of substitutions made

• Modifier i (ignore case) is very useful for both matching and substitution

• Modifier g (global) is very useful for substitution as it replaced all occurrences of regex in the string

If the first method you try doesn't work, try breaking the value into smaller components and match each boundary.

If all else fails, you can always ask for help on the comp.lang.perl.misc newsgroup.

The $. special variable keeps track of the record number. Every time the diamond operators read a line, this variable is incremented.

If you need to save the value of the matched strings stored in the regex memory, make sure to assign them to other variables. regex memory is local to the enclosing block and lasts only until another match is done.

Note: If variable interpolation is used in the replacement string none of the meta-characters can be used in the replacement string

Your regex matching code should protect from unassigned built-in variable regex matching errors.

Two general observation:

• Regular expressions are powerful language that is completely distinct from "rest of Perl". It requires separate study and complex regex require separate testing.
• There are a lot of fine points that you need to understand to use regular expressions correctly. Try to limit yourself to simple regex. Leave complex regex to the authors of books about Perl. That provides a nice material for their errata ;-).

To be successful you need to use to adopt some elements of style that makes mistakes less probable:

• Use non-gready(lazy) matching when you do not need greedy matching. That prevent a lot of errors from happening in case you have several instances of the same pattern in the strong and do not want to find the last one but the first match. 
• If possible try to anchor you regex to either start of the string or to the end of the string or both. That makes matching faster and simplify debugging.
• Always debug complex regex in a separate test program with multiple samples, preferably extracted from your data. Never assume that complex regex is correct on first iteration
• Use extended syntax to commend and pretty print complex regex.
• Try to split complex regex into two or more to make it more simple. Don't try to create a "killer" one-liner everybody will admire and nobody understand.
• If you use variables $1, $1 and so on remember that in case string does not match they preserve their old values. Always check if regex is successful if you use $1. $2 and so on.
• Form ($a, $b, $c)=($string=~/(\d+)\.(\d+)\.(\d+)/ is preferable to usage of separate assignment of each group via $1, $2 and $3.

Below are some reference tables from Medinets book: 

 

Meta Brackets	Description
()	The parentheses let you affect the order of pattern evaluation and act as a form of pattern memory.
(?...)	If a question mark immediately follows the left parentheses, it indicates that an extended mode component is being specified.
{n, m}	The curly braces let specify how many times the "thing" immediately to the left should be matched. {n} means that it should be matched exactly n times. {n,} means it must be matched at least n times. {n, m} means that it must be matched at least n times and not more than m times.
[]	The square brackets let you create a character class. For instance, m/[abc]/ will evaluate to true if any of "a", "b", or "c" is contained in $_. The square brackets are a more readable alternative to the alternation meta-character.

 

Meta Sequences	Description
\	This meta-character "escapes" the following character. This means that any special meaning normally attached to that character is ignored. For instance, if you need to include a dollar sign in a pattern, you must use \$ to avoid Perl's variable interpolation. Use \\ to specify the backslash character in your pattern.
\0nnn	Any Octal byte.
\A	This meta-sequence represents the beginning of the string. Its meaning is not affected by the /m option.
\b	This meta-sequence represents the backspace character inside a character class; otherwise, it represents a word boundary. A word boundary is the spot between word (\w) and non-word(\W) characters. Perl thinks that the \W meta-sequence matches the imaginary characters off the ends of the string.
\B	Match a non-word boundary.
\cn	Any control character.
\d	Match a single digit character.
\D	Match a single non-digit character.
\e	Escape.
\E	Terminate the \L or \U sequence.
\f	Form Feed.
\G	Match only where the previous m//g left off.
\l	Change the next character to lowercase.
\L	Change the following characters to lowercase until a \E sequence is encountered.
\n	Newline.
\Q	Quote Regular Expression meta-characters literally until the \E sequence is encountered.
\r	Carriage Return.
\s	Match a single whitespace character.
\S	Match a single non-whitespace character.
\t	Tab.
\u	Change the next character to uppercase.
\U	Change the following characters to uppercase until a \E sequence is encountered.
\v	Vertical Tab.
\w	Match a single word character. Word characters are the alphanumeric and underscore characters.
\W	Match a single non-word character.
\xnn	Any Hexadecimal byte.
\Z	This meta-sequence represents the end of the string. Its meaning is not affected by the /m option.

 

Regular Expressions form almost a 'language within a language' in Perl. As you can see above, they can be fairly involved, and (lets face it) if you are not familiar with them now, you are not going to learn them without practice. Therefore, we suggest the following path for learning regular expressions.

More about  Modifiers

Modifiers

The material below is adapted from O'Reilly Network Five Habits for Successful Regular Expressions

Use extended syntax for complex patterns

Consider the following regular expression to match a U.S. phone number:

 \(?\d{3}\)?\s?\d{3}[-.]\d{4} 

This regex matches phone numbers like "(973)555-4000". Ask yourself if the regex would match "973-555-4000" or "555-4000". The answer is no in both cases. Writing this pattern on one line conceals both flaws and design decisions. The area code actually can't be omitted and form 973-555-4000 will not be accepted.

Spreading the pattern out over several lines makes the flaws more visible and the necessary modifications easier. In Perl using extended syntax (option x) we can re-write this expression to accept the second form as following:

 /  
    \(?     # optional parentheses
      \d{3} # area code required
    \)?     # optional parentheses
    [-\s.]? # separator is either a dash, a space, or a period.
      \d{3} # 3-digit prefix
    [-.]    # another separator
      \d{4} # 4-digit line number
/x 

The rewritten regex now has an optional separator after the area code so that it matches phone 973-555-4000 as well as (973)555-4000. The area code is still required.

However, a new programmer who wants to make the area code optional can quickly see that it is not optional now and might change the code to use separate regex for each of three major cases instead of merging them like we did. So much for our optimization ;-). Readable code helps immensely but in no way it is a substitute for good design.

Write Tests for each and every complex regex

There are three levels of testing, each adding a higher level of reliability to your code. First, you need to think hard about what you want to match and whether you can deal with false matches. Second, you need to test the regex on example data. Third, you need to formalize the tests into a test suite.

Deciding what to match is a trade-off between making false matches and missing valid matches. If your regex is too strict, it will miss valid matches. If it is too loose, it will generate false matches. Once the regex is released into live code, you probably will not notice either way. Consider the phone regex example above; it would match the text "800-555=0355". False matches are hard to catch, so it's important to plan ahead and test.

Sticking with the phone number example, if you are validating a phone number on a web form, you may settle for ten digits in any format. However, if you are trying to extract phone numbers from a large amount of text, you might want to be more strict to avoid a unacceptable numbers due to  false matches.

When thinking about what you want to match, write down example cases. Then write some code that tests your regular expression against the example cases. Any complicated regular expression is best written in a small test program, as the examples below demonstrate:

In Perl:

#!/usr/bin/perl

my @tests = ( "314-555-4000",
              "800-555-4400",
	      "(314)555-4000",
              "314.555.4000",
              "555-4000",
              "aasdklfjklas",
              "1234-123-12345"          
            );
foreach my $test (@tests) {
    if ( $test =~ m/
       \(?     # optional parentheses
       \d{3} # area code required
       \)?     # optional parentheses
       [-\s.]? # separator is either a dash, a space, or a period.
       \d{3} # 3-digit prefix
       [-\s.]  # another separator
       \d{4} # 4-digit line number/x ) {
        print "Matched on $test\n";
    } else {
        print "Failed match on $test\n";
    } # if
} # foreach

Running the test script exposes yet another problem in the phone number regex: it matched "1234-123-12345". That demonstrates the key principle of text selection: include both tests that you expect to fail (they may succeed) as well as those you expect to match (they may not)

Ideally, you preseve this test as part of the test suite for your entire program. Even if you do not have a test suite already, your regular expression tests are a good foundation for a suite, and now is the perfect opportunity to start on one. Even if now is not the right time (really, it is!), you should make a habit to run your regex tests after every modification. A little extra time here could save you many headaches.

Group the Alternation Operator

The alternation operator (|) has a low precedence. This means that it often alternates over more than the programmer intended. For example, a regex to extract email addresses out of a mail file might look like:

^CC:|To:(.*)

The above attempt is incorrect, but the bugs often go unnoticed. The intent of the above regex is to find lines starting with "CC:" or "To:" and then capture any email addresses on the rest of the line.

Unfortunately, the regex doesn't actually capture anything from lines starting with "CC:" and may capture random text if "To:" appears in the middle of a line. In plain English, the regular expression matches lines beginning with "CC:" and captures nothing, or matches any line containing the text "To:" and then captures the rest of the line. Usually, it will capture plenty of addresses and nobody will notice the failings.

If that were the real intent, you should add parentheses to say it explicitly, like this:

(^CC:)|(To:(.*))

However, the real intent of the regex i to match lines starting with "CC:" or "To:" and then capture the rest of the line. The following regex does that:

^(CC:|To:)(.*)

This is a common and hard-to-catch bug. If you develop the habit of wrapping your alternations in parentheses (or non-capturing parentheses -- (?:)) you can avoid this error.

Use Lazy Quantifiers

Most people avoid using the lazy quantifiers *?, +?, and ??, even though they are easy to understand and make many regular expressions easier to write.

Lazy quantifiers match as little text as possible while still aiding the success of the overall match. If you write foo(.*?)bar, the quantifier will stop matching the first time it sees "bar", not the last time. This may be important if you are trying to capture ###; in the text foo###bar+++bar. A regular quantifier would have captured ###bar+++.

Let's say you want to capture all of the phone numbers from an HTML file. You could use the phone number regular expression example we discussed earlier in this article. However, if you know that the file contains all of the phone numbers in the first column of a table, you can write a much simpler regex using lazy quantifiers:

<tr><td>(.+?)<td>

Many beginning regular expression programmers avoid lazy quantifiers with negated character classes. They write the above code as:

<tr><td>([^>]+)</td>

That works in this case, but leads to trouble if the text you are trying to capture contains common characters from your delimiter (in this case, </td>). If you use lazy quantifiers, you will spend less time kludging character classes and produce clearer regular expressions.

Lazy quantifiers are most valuable when you know the structure surrounding the text you want to capture.

Use Alternative Delimiters

Perl allow you to use any non-alphanumeric or whitespace character as a delimiter. If you switch to a new delimiter, you can avoid having to escape the forward slashes when you are trying to match URLs or HTML tags such as "http://" or "<br />".

For example:

/http:\/\/(\S)*/

could be rewritten as:

#http://(\S)*#

Common delimiters are #, !, |. If you use square brackets, angle brackets, or curly braces, the opening and closing brackets must match. Here are some common uses of delimiters:

Tips

• Usually you should use index function instead of regular expression for simple string matching.
• Sometimes it makes sense to use regular expressions instead of substr. One such task is extraction of component of date, for example:

  $cur_date='20060325'; 
  (year, $month, $day)=$cur_date=~/(\d{4})(\d\d)(\d\d)/;

• Use markers of line start (^) and/or line end($) to anchor your regex to the start and end of the line (Hat tip to RegExp reminder)

  I was just reminded about this small thing, which is so easy to forget – regular expressions that have markers of line start (^) and/or line end($) are so much faster than those regexps that don’t have these markers. The thing is that with line start/end marker regexp engine needs to make only one match/substution, whereas when there is no such markers, it has to repeat the match/substitution operation at every character of the string.

  In practice, it’s unbelievable how much difference this can make. Especially when using complex regular expressions over large data sets.

  P.S.: I understand that it is not always possible to use these markers, but I think that they can be used much more often than they are. Everywhere.

Prev | Up | Contents | Down | Next

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019