NAMEperlretut - Perl regular expressions tutorial
DESCRIPTIONThis page provides a basic tutorial on understanding, creating and
using regular expressions in Perl. It serves as a complement to the
reference page on regular expressions the perlre manpage. Regular expressions
are an integral part of the Perl is widely renowned for excellence in text processing, and regular expressions are one of the big factors behind this fame. Perl regular expressions display an efficiency and flexibility unknown in most other computer languages. Mastering even the basics of regular expressions will allow you to manipulate text with surprising ease. What is a regular expression? A regular expression is simply a string
that describes a pattern. Patterns are in common use these days;
examples are the patterns typed into a search engine to find web pages
and the patterns used to list files in a directory, e.g., Regular expressions have the undeserved reputation of being abstract
and difficult to understand. Regular expressions are constructed using
simple concepts like conditionals and loops and are no more difficult
to understand than the corresponding This tutorial flattens the learning curve by discussing regular expression concepts, along with their notation, one at a time and with many examples. The first part of the tutorial will progress from the simplest word searches to the basic regular expression concepts. If you master the first part, you will have all the tools needed to solve about 98% of your needs. The second part of the tutorial is for those comfortable with the basics and hungry for more power tools. It discusses the more advanced regular expression operators and introduces the latest cutting edge innovations in 5.6.0. A note: to save time, 'regular expression' is often abbreviated as regexp or regex. Regexp is a more natural abbreviation than regex, but is harder to pronounce. The Perl pod documentation is evenly split on regexp vs regex; in Perl, there is more than one way to abbreviate it. We'll use regexp in this tutorial.
Part 1: The basics
Simple word matchingThe simplest regexp is simply a word, or more generally, a string of characters. A regexp consisting of a word matches any string that contains that word: "Hello World" =~ /World/; # matches What is this perl statement all about? if ("Hello World" =~ /World/) { print "It matches\n"; } else { print "It doesn't match\n"; } There are useful variations on this theme. The sense of the match can
be reversed by using if ("Hello World" !~ /World/) { print "It doesn't match\n"; } else { print "It matches\n"; } The literal string in the regexp can be replaced by a variable: $greeting = "World"; if ("Hello World" =~ /$greeting/) { print "It matches\n"; } else { print "It doesn't match\n"; } If you're matching against the special default variable $_ = "Hello World"; if (/World/) { print "It matches\n"; } else { print "It doesn't match\n"; } And finally, the "Hello World" =~ m!World!; # matches, delimited by '!' "Hello World" =~ m{World}; # matches, note the matching '{}' "/usr/bin/perl" =~ m"/perl"; # matches after '/usr/bin', # '/' becomes an ordinary char
Let's consider how different regexps would match "Hello World" =~ /world/; # doesn't match "Hello World" =~ /o W/; # matches "Hello World" =~ /oW/; # doesn't match "Hello World" =~ /World /; # doesn't match The first regexp If a regexp matches in more than one place in the string, perl will always match at the earliest possible point in the string: "Hello World" =~ /o/; # matches 'o' in 'Hello' "That hat is red" =~ /hat/; # matches 'hat' in 'That' With respect to character matching, there are a few more points you need to know about. First of all, not all characters can be used 'as is' in a match. Some characters, called metacharacters, are reserved for use in regexp notation. The metacharacters are {}[]()^$.|*+?\ The significance of each of these will be explained in the rest of the tutorial, but for now, it is important only to know that a metacharacter can be matched by putting a backslash before it: "2+2=4" =~ /2+2/; # doesn't match, + is a metacharacter "2+2=4" =~ /2\+2/; # matches, \+ is treated like an ordinary + "The interval is [0,1)." =~ /[0,1)./ # is a syntax error! "The interval is [0,1)." =~ /\[0,1\)\./ # matches "/usr/bin/perl" =~ /\/usr\/local\/bin\/perl/; # matches In the last regexp, the forward slash The backslash character 'C:\WIN32' =~ /C:\\WIN/; # matches In addition to the metacharacters, there are some ASCII characters
which don't have printable character equivalents and are instead
represented by escape sequences. Common examples are "1000\t2000" =~ m(0\t2) # matches "1000\n2000" =~ /0\n20/ # matches "1000\t2000" =~ /\000\t2/ # doesn't match, "0" ne "\000" "cat" =~ /\143\x61\x74/ # matches, but a weird way to spell cat If you've been around Perl a while, all this talk of escape sequences may seem familiar. Similar escape sequences are used in double-quoted strings and in fact the regexps in Perl are mostly treated as double-quoted strings. This means that variables can be used in regexps as well. Just like double-quoted strings, the values of the variables in the regexp will be substituted in before the regexp is evaluated for matching purposes. So we have: $foo = 'house'; 'housecat' =~ /$foo/; # matches 'cathouse' =~ /cat$foo/; # matches 'housecat' =~ /${foo}cat/; # matches So far, so good. With the knowledge above you can already perform searches with just about any literal string regexp you can dream up. Here is a very simple emulation of the Unix grep program: % cat > simple_grep #!/usr/bin/perl $regexp = shift; while (<>) { print if /$regexp/; } ^D % chmod +x simple_grep % simple_grep abba /usr/dict/words Babbage cabbage cabbages sabbath Sabbathize Sabbathizes sabbatical scabbard scabbards This program is easy to understand. With all of the regexps above, if the regexp matched anywhere in the
string, it was considered a match. Sometimes, however, we'd like to
specify where in the string the regexp should try to match. To do
this, we would use the anchor metacharacters "housekeeper" =~ /keeper/; # matches "housekeeper" =~ /^keeper/; # doesn't match "housekeeper" =~ /keeper$/; # matches "housekeeper\n" =~ /keeper$/; # matches The second regexp doesn't match because When both "keeper" =~ /^keep$/; # doesn't match "keeper" =~ /^keeper$/; # matches "" =~ /^$/; # ^$ matches an empty string The first regexp doesn't match because the string has more to it than
"dogbert" =~ /bert/; # matches, but not what you want "dilbert" =~ /^bert/; # doesn't match, but .. "bertram" =~ /^bert/; # matches, so still not good enough "bertram" =~ /^bert$/; # doesn't match, good "dilbert" =~ /^bert$/; # doesn't match, good "bert" =~ /^bert$/; # matches, perfect Of course, in the case of a literal string, one could just as easily
use the string equivalence
Using character classesAlthough one can already do quite a lot with the literal string regexps above, we've only scratched the surface of regular expression technology. In this and subsequent sections we will introduce regexp concepts (and associated metacharacter notations) that will allow a regexp to not just represent a single character sequence, but a whole class of them. One such concept is that of a character class. A character class
allows a set of possible characters, rather than just a single
character, to match at a particular point in a regexp. Character
classes are denoted by brackets /cat/; # matches 'cat' /[bcr]at/; # matches 'bat, 'cat', or 'rat' /item[0123456789]/; # matches 'item0' or ... or 'item9' "abc" =~ /[cab]/; # matches 'a' In the last statement, even though /[yY][eE][sS]/; # match 'yes' in a case-insensitive way # 'yes', 'Yes', 'YES', etc. This regexp displays a common task: perform a a case-insensitive
match. Perl provides away of avoiding all those brackets by simply
appending an We saw in the section above that there were ordinary characters, which
represented themselves, and special characters, which needed a
backslash /[\]c]def/; # matches ']def' or 'cdef' $x = 'bcr'; /[$x]at/; # matches 'bat', 'cat', or 'rat' /[\$x]at/; # matches '$at' or 'xat' /[\\$x]at/; # matches '\at', 'bat, 'cat', or 'rat' The last two are a little tricky. in The special character /item[0-9]/; # matches 'item0' or ... or 'item9' /[0-9bx-z]aa/; # matches '0aa', ..., '9aa', # 'baa', 'xaa', 'yaa', or 'zaa' /[0-9a-fA-F]/; # matches a hexadecimal digit /[0-9a-zA-Z_]/; # matches a "word" character, # like those in a perl variable name If The special character /[^a]at/; # doesn't match 'aat' or 'at', but matches # all other 'bat', 'cat, '0at', '%at', etc. /[^0-9]/; # matches a non-numeric character /[a^]at/; # matches 'aat' or '^at'; here '^' is ordinary Now, even
The /\d\d:\d\d:\d\d/; # matches a hh:mm:ss time format /[\d\s]/; # matches any digit or whitespace character /\w\W\w/; # matches a word char, followed by a # non-word char, followed by a word char /..rt/; # matches any two chars, followed by 'rt' /end\./; # matches 'end.' /end[.]/; # same thing, matches 'end.' Because a period is a metacharacter, it needs to be escaped to match
as an ordinary period. Because, for example, An anchor useful in basic regexps is the word anchor
$x = "Housecat catenates house and cat"; $x =~ /cat/; # matches cat in 'housecat' $x =~ /\bcat/; # matches cat in 'catenates' $x =~ /cat\b/; # matches cat in 'housecat' $x =~ /\bcat\b/; # matches 'cat' at end of string Note in the last example, the end of the string is considered a word boundary. You might wonder why "" =~ /^$/; # matches "\n" =~ /^$/; # matches, "\n" is ignored "" =~ /./; # doesn't match; it needs a char "" =~ /^.$/; # doesn't match; it needs a char "\n" =~ /^.$/; # doesn't match; it needs a char other than "\n" "a" =~ /^.$/; # matches "a\n" =~ /^.$/; # matches, ignores the "\n" This behavior is convenient, because we usually want to ignore
newlines when we count and match characters in a line. Sometimes,
however, we want to keep track of newlines. We might even want
Here are examples of $x = "There once was a girl\nWho programmed in Perl\n"; $x =~ /^Who/; # doesn't match, "Who" not at start of string $x =~ /^Who/s; # doesn't match, "Who" not at start of string $x =~ /^Who/m; # matches, "Who" at start of second line $x =~ /^Who/sm; # matches, "Who" at start of second line $x =~ /girl.Who/; # doesn't match, "." doesn't match "\n" $x =~ /girl.Who/s; # matches, "." matches "\n" $x =~ /girl.Who/m; # doesn't match, "." doesn't match "\n" $x =~ /girl.Who/sm; # matches, "." matches "\n" Most of the time, the default behavior is what is want, but $x =~ /^Who/m; # matches, "Who" at start of second line $x =~ /\AWho/m; # doesn't match, "Who" is not at start of string $x =~ /girl$/m; # matches, "girl" at end of first line $x =~ /girl\Z/m; # doesn't match, "girl" is not at end of string $x =~ /Perl\Z/m; # matches, "Perl" is at newline before end $x =~ /Perl\z/m; # doesn't match, "Perl" is not at end of string We now know how to create choices among classes of characters in a regexp. What about choices among words or character strings? Such choices are described in the next section.
Matching this or thatSometimes we would like to our regexp to be able to match different
possible words or character strings. This is accomplished by using
the alternation metacharacter "cats and dogs" =~ /cat|dog|bird/; # matches "cat" "cats and dogs" =~ /dog|cat|bird/; # matches "cat" Even though "cats" =~ /c|ca|cat|cats/; # matches "c" "cats" =~ /cats|cat|ca|c/; # matches "cats" Here, all the alternatives match at the first string position, so the first alternative is the one that matches. If some of the alternatives are truncations of the others, put the longest ones first to give them a chance to match. "cab" =~ /a|b|c/ # matches "c" # /a|b|c/ == /[abc]/ The last example points out that character classes are like alternations of characters. At a given character position, the first alternative that allows the regexp match to succeed wil be the one that matches.
Grouping things and hierarchical matchingAlternation allows a regexp to choose among alternatives, but by
itself it unsatisfying. The reason is that each alternative is a whole
regexp, but sometime we want alternatives for just part of a
regexp. For instance, suppose we want to search for housecats or
housekeepers. The regexp The grouping metacharacters /(a|b)b/; # matches 'ab' or 'bb' /(ac|b)b/; # matches 'acb' or 'bb' /(^a|b)c/; # matches 'ac' at start of string or 'bc' anywhere /(a|[bc])d/; # matches 'ad', 'bd', or 'cd' /house(cat|)/; # matches either 'housecat' or 'house' /house(cat(s|)|)/; # matches either 'housecats' or 'housecat' or # 'house'. Note groups can be nested. /(19|20|)\d\d/; # match years 19xx, 20xx, or the Y2K problem, xx "20" =~ /(19|20|)\d\d/; # matches the null alternative '()\d\d', # because '20\d\d' can't match Alternations behave the same way in groups as out of them: at a given
string position, the leftmost alternative that allows the regexp to
match is taken. So in the last example at tth first string position,
The process of trying one alternative, seeing if it matches, and moving on to the next alternative if it doesn't, is called backtracking. The term 'backtracking' comes from the idea that matching a regexp is like a walk in the woods. Successfully matching a regexp is like arriving at a destination. There are many possible trailheads, one for each string position, and each one is tried in order, left to right. From each trailhead there may be many paths, some of which get you there, and some which are dead ends. When you walk along a trail and hit a dead end, you have to backtrack along the trail to an earlier point to try another trail. If you hit your destination, you stop immediately and forget about trying all the other trails. You are persistent, and only if you have tried all the trails from all the trailheads and not arrived at your destination, do you declare failure. To be concrete, here is a step-by-step analysis of what perl does when it tries to match the regexp "abcde" =~ /(abd|abc)(df|d|de)/;
There are a couple of things to note about this analysis. First, the
third alternative in the second group 'de' also allows a match, but we
stopped before we got to it - at a given character position, leftmost
wins. Second, we were able to get a match at the first character
position of the string 'a'. If there were no matches at the first
position, perl would move to the second character position 'b' and
attempt the match all over again. Only when all possible paths at all
possible character positions have been exhausted does perl give give
up and declare Even with all this work, regexp matching happens remarkably fast. To speed things up, during compilation stage, perl compiles the regexp into a compact sequence of opcodes that can often fit inside a processor cache. When the code is executed, these opcodes can then run at full throttle and search very quickly.
Extracting matchesThe grouping metacharacters # extract hours, minutes, seconds $time =~ /(\d\d):(\d\d):(\d\d)/; # match hh:mm:ss format $hours = $1; $minutes = $2; $seconds = $3; Now, we know that in scalar context,
# extract hours, minutes, seconds ($hours, $minutes, $second) = ($time =~ /(\d\d):(\d\d):(\d\d)/); If the groupings in a regexp are nested, /(ab(cd|ef)((gi)|j))/; 1 2 34 so that if the regexp matched, e.g., Closely associated with the matching variables /(\w\w\w)\s\1/; The grouping assigns a value to \1, so that the same 3 letter sequence is used for both parts. Here are some words with repeated parts: % simple_grep '^(\w\w\w\w|\w\w\w|\w\w|\w)\1$' /usr/dict/words beriberi booboo coco mama murmur papa The regexp has a single grouping which considers 4-letter
combinations, then 3-letter combinations, etc. and uses In addition to what was matched, Perl 5.6.0 also provides the
positions of what was matched with the $x = "Mmm...donut, thought Homer"; $x =~ /^(Mmm|Yech)\.\.\.(donut|peas)/; # matches foreach $expr (1..$#-) { print "Match $expr: '${$expr}' at position ($-[$expr],$+[$expr])\n"; } prints Match 1: 'Mmm' at position (0,3) Match 2: 'donut' at position (6,11) Even if there are no groupings in a regexp, it is still possible to
find out what exactly matched in a string. If you use them, perl
will set $x = "the cat caught the mouse"; $x =~ /cat/; # $` = 'the ', $& = 'cat', $' = ' caught the mouse' $x =~ /the/; # $` = '', $& = 'the', $' = ' cat caught the mouse' In the second match, $` is the same as substr( $x, 0, $-[0] ) $& is the same as substr( $x, $-[0], $+[0]-$-[0] ) $' is the same as substr( $x, $+[0] )
Matching repetitionsThe examples in the previous section display an annoying weakness. We
were only matching 3-letter words, or syllables of 4 letters or
less. We'd like to be able to match words or syllables of any length,
without writing out tedious alternatives like
This is exactly the problem the quantifier metacharacters
Here are some examples: /[a-z]+\s+\d*/; # match a lowercase word, at least some space, and # any number of digits /(\w+)\s+\1/; # match doubled words of arbitrary length /y(es)?/i; # matches 'y', 'Y', or a case-insensitive 'yes' $year =~ /\d{2,4}/; # make sure year is at least 2 but not more # than 4 digits $year =~ /\d{4}|\d{2}/; # better match; throw out 3 digit dates $year =~ /\d{2}(\d{2})?/; # same thing written differently. However, # this produces $1 and the other does not. % simple_grep '^(\w+)\1$' /usr/dict/words # isn't this easier? beriberi booboo coco mama murmur papa For all of these quantifiers, perl will try to match as much of the
string as possible, while still allowing the regexp to succeed. Thus
with $x = "the cat in the hat"; $x =~ /^(.*)(cat)(.*)$/; # matches, # $1 = 'the ' # $2 = 'cat' # $3 = ' in the hat' Which is what we might expect, the match finds the only $x =~ /^(.*)(at)(.*)$/; # matches, # $1 = 'the cat in the h' # $2 = 'at' # $3 = '' (0 matches) One might initially guess that perl would find the When a regexp can match a string in several different ways, we can use the principles above to predict which way the regexp will match:
As we have seen above, Principle 0 overrides the others - the regexp will be matched as early as possible, with the other principles determining how the regexp matches at that earliest character position. Here is an example of these principles in action: $x = "The programming republic of Perl"; $x =~ /^(.+)(e|r)(.*)$/; # matches, # $1 = 'The programming republic of Pe' # $2 = 'r' # $3 = 'l' This regexp matches at the earliest string position, $x =~ /(m{1,2})(.*)$/; # matches, # $1 = 'mm' # $2 = 'ing republic of Perl' Here, The earliest possible match is at the first $x =~ /.*(m{1,2})(.*)$/; # matches, # $1 = 'm' # $2 = 'ing republic of Perl' Here, the regexp matches at the start of the string. The first
quantifier $x =~ /(.?)(m{1,2})(.*)$/; # matches, # $1 = 'a' # $2 = 'mm' # $3 = 'ing republic of Perl' Here, "aXXXb" =~ /(X*)/; # matches with $1 = '' because it can match zero copies of Sometimes greed is not good. At times, we would like quantifiers to
match a minimal piece of string, rather than a maximal piece. For
this purpose, Larry Wall created the minimal match or
non-greedy quantifiers
Let's look at the example above, but with minimal quantifiers: $x = "The programming republic of Perl"; $x =~ /^(.+?)(e|r)(.*)$/; # matches, # $1 = 'Th' # $2 = 'e' # $3 = ' programming republic of Perl' The minimal string that will allow both the start of the string $x =~ /(m{1,2}?)(.*?)$/; # matches, # $1 = 'm' # $2 = 'ming republic of Perl' The first string position that this regexp can match is at the first
$x =~ /(.*?)(m{1,2}?)(.*)$/; # matches, # $1 = 'The progra' # $2 = 'm' # $3 = 'ming republic of Perl' In this regexp, you might expect the first minimal quantifier $x =~ /(.??)(m{1,2})(.*)$/; # matches, # $1 = 'a' # $2 = 'mm' # $3 = 'ing republic of Perl' Just as in the previous regexp, the first quantifier We can modify principle 3 above to take into account non-greedy quantifiers:
Just like alternation, quantifiers are also susceptible to backtracking. Here is a step-by-step analysis of the example $x = "the cat in the hat"; $x =~ /^(.*)(at)(.*)$/; # matches, # $1 = 'the cat in the h' # $2 = 'at' # $3 = '' (0 matches)
Most of the time, all this moving forward and backtracking happens quickly and searching is fast. There are some pathological regexps, however, whose execution time exponentially grows with the size of the string. A typical structure that blows up in your face is of the form /(a|b+)*/; The problem is the nested indeterminate quantifiers. There are many
different ways of partitioning a string of length n between the
Building a regexpAt this point, we have all the basic regexp concepts covered, so let's give a more involved example of a regular expression. We will build a regexp that matches numbers. The first task in building a regexp is to decide what we want to match and what we want to exclude. In our case, we want to match both integers and floating point numbers and we want to reject any string that isn't a number. The next task is to break the problem down into smaller problems that are easily converted into a regexp. The simplest case is integers. These consist of a sequence of digits,
with an optional sign in front. The digits we can represent with
/[+-]?\d+/; # matches integers A floating point number potentially has a sign, an integral part, a
decimal point, a fractional part, and an exponent. One or more of these
parts is optional, so we need to check out the different
possibilities. Floating point numbers which are in proper form include
123., 0.345, .34, -1e6, and 25.4E-72. As with integers, the sign out
front is completely optional and can be matched by /[+-]?\d+\./; # 1., 321., etc. /[+-]?\.\d+/; # .1, .234, etc. /[+-]?\d+\.\d+/; # 1.0, 30.56, etc. These can be combined into a single regexp with a three-way alternation: /[+-]?(\d+\.\d+|\d+\.|\.\d+)/; # floating point, no exponent In this alternation, it is important to put Now consider floating point numbers with exponents. The key observation here is that both integers and numbers with decimal points are allowed in front of an exponent. Then exponents, like the overall sign, are independent of whether we are matching numbers with or without decimal points, and can be 'decoupled' from the mantissa. The overall form of the regexp now becomes clear: /^(optional sign)(integer | f.p. mantissa)(optional exponent)$/; The exponent is an /[eE][+-]?\d+/; # exponent Putting all the parts together, we get a regexp that matches numbers: /^[+-]?(\d+\.\d+|\d+\.|\.\d+|\d+)([eE][+-]?\d+)?$/; # Ta da! Long regexps like this may impress your friends, but can be hard to
decipher. In complex situations like this, the /^ [+-]? # first, match an optional sign ( # then match integers or f.p. mantissas: \d+\.\d+ # mantissa of the form a.b |\d+\. # mantissa of the form a. |\.\d+ # mantissa of the form .b |\d+ # integer of the form a ) ([eE][+-]?\d+)? # finally, optionally match an exponent $/x; If whitespace is mostly irrelevant, how does one include space
characters in an extended regexp? The answer is to backslash it
/^ [+-]?\ * # first, match an optional sign *and space* ( # then match integers or f.p. mantissas: \d+\.\d+ # mantissa of the form a.b |\d+\. # mantissa of the form a. |\.\d+ # mantissa of the form .b |\d+ # integer of the form a ) ([eE][+-]?\d+)? # finally, optionally match an exponent $/x; In this form, it is easier to see a way to simplify the
alternation. Alternatives 1, 2, and 4 all start with /^ [+-]?\ * # first, match an optional sign ( # then match integers or f.p. mantissas: \d+ # start out with a ... ( \.\d* # mantissa of the form a.b or a. )? # ? takes care of integers of the form a |\.\d+ # mantissa of the form .b ) ([eE][+-]?\d+)? # finally, optionally match an exponent $/x; or written in the compact form, /^[+-]?\ *(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?$/; This is our final regexp. To recap, we built a regexp by
These are also the typical steps involved in writing a computer program. This makes perfect sense, because regular expressions are essentially programs written a little computer language that specifies patterns.
Using regular expressions in PerlThe last topic of Part 1 briefly covers how regexps are used in Perl programs. Where do they fit into Perl syntax? We have already introduced the matching operator in its default
There are a few more things you might want to know about matching operators. First, we pointed out earlier that variables in regexps are substituted before the regexp is evaluated: $pattern = 'Seuss'; while (<>) { print if /$pattern/; } This will print any lines containing the word #!/usr/bin/perl # Improved simple_grep $regexp = shift; while (<>) { print if /$regexp/o; # a good deal faster } If you change $pattern = 'Seuss'; while (<>) { print if m'$pattern'; # matches '$pattern', not 'Seuss' }
"dog" =~ /d/; # 'd' matches "dogbert =~ //; # this matches the 'd' regexp used before The final two modifiers The use of $x = "cat dog house"; # 3 words $x =~ /^\s*(\w+)\s+(\w+)\s+(\w+)\s*$/; # matches, # $1 = 'cat' # $2 = 'dog' # $3 = 'house' But what if we had an indeterminate number of words? This is the sort
of task while ($x =~ /(\w+)/g) { print "Word is $1, ends at position ", pos $x, "\n"; } prints Word is cat, ends at position 3 Word is dog, ends at position 7 Word is house, ends at position 13 A failed match or changing the target string resets the position. If
you don't want the position reset after failure to match, add the
In list context, @words = ($x =~ /(\w+)/g); # matches, # $word[0] = 'cat' # $word[1] = 'dog' # $word[2] = 'house' Closely associated with the $metric = 1; # use metric units ... $x = <FILE>; # read in measurement $x =~ /^([+-]?\d+)\s*/g; # get magnitude $weight = $1; if ($metric) { # error checking print "Units error!" unless $x =~ /\Gkg\./g; } else { print "Units error!" unless $x =~ /\Glbs\./g; } $x =~ /\G\s+(widget|sprocket)/g; # continue processing The combination of
# expanded, this is "ATC GTT GAA TGC AAA TGA CAT GAC" $dna = "ATCGTTGAATGCAAATGACATGAC"; $dna =~ /TGA/; doesn't work; it may match an while ($dna =~ /(\w\w\w)*?TGA/g) { # note the minimal *? print "Got a TGA stop codon at position ", pos $dna, "\n"; } which prints Got a TGA stop codon at position 18 Got a TGA stop codon at position 23 Position 18 is good, but position 23 is bogus. What happened? The answer is that our regexp works well until we get past the last
real match. Then the regexp will fail to match a synchronized while ($dna =~ /\G(\w\w\w)*?TGA/g) { print "Got a TGA stop codon at position ", pos $dna, "\n"; } This prints Got a TGA stop codon at position 18 which is the correct answer. This example illustrates that it is important not only to match what is desired, but to reject what is not desired. search and replace Regular expressions also play a big role in search and replace
operations in Perl. Search and replace is accomplished with the
$x = "Time to feed the cat!"; $x =~ s/cat/hacker/; # $x contains "Time to feed the hacker!" if ($x =~ s/^(Time.*hacker)!$/$1 now!/) { $more_insistent = 1; } $y = "'quoted words'"; $y =~ s/^'(.*)'$/$1/; # strip single quotes, # $y contains "quoted words" In the last example, the whole string was matched, but only the part
inside the single quotes was grouped. With the $x = "I batted 4 for 4"; $x =~ s/4/four/; # doesn't do it all: # $x contains "I batted four for 4" $x = "I batted 4 for 4"; $x =~ s/4/four/g; # does it all: # $x contains "I batted four for four" If you prefer 'regex' over 'regexp' in this tutorial, you could use the following program to replace it: % cat > simple_replace #!/usr/bin/perl $regexp = shift; $replacement = shift; while (<>) { s/$regexp/$replacement/go; print; } ^D % simple_replace regexp regex perlretut.pod In A modifier available specifically to search and replace is the
$x = "Bill the cat"; $x =~ s/(.)/$chars{$1}++;$1/eg; # final $1 replaces char with itself print "frequency of '$_' is $chars{$_}\n" foreach (sort {$chars{$b} <=> $chars{$a}} keys %chars); This prints frequency of ' ' is 2 frequency of 't' is 2 frequency of 'l' is 2 frequency of 'B' is 1 frequency of 'c' is 1 frequency of 'e' is 1 frequency of 'h' is 1 frequency of 'i' is 1 frequency of 'a' is 1 As with the match The split operator The $x = "Calvin and Hobbes"; @words = split /\s+/, $x; # $word[0] = 'Calvin' # $word[1] = 'and' # $word[2] = 'Hobbes' If the empty regexp $x = "/usr/bin/perl"; @dirs = split m!/!, $x; # $dirs[0] = '' # $dirs[1] = 'usr' # $dirs[2] = 'bin' # $dirs[3] = 'perl' @parts = split m!(/)!, $x; # $parts[0] = '' # $parts[1] = '/' # $parts[2] = 'usr' # $parts[3] = '/' # $parts[4] = 'bin' # $parts[5] = '/' # $parts[6] = 'perl' Since the first character of $x matched the regexp, If you have read this far, congratulations! You now have all the basic tools needed to use regular expressions to solve a wide range of text processing problems. If this is your first time through the tutorial, why not stop here and play around with regexps a while... Part 2 concerns the more esoteric aspects of regular expressions and those concepts certainly aren't needed right at the start.
Part 2: Power toolsOK, you know the basics of regexps and you want to know more. If matching regular expressions is analogous to a walk in the woods, then the tools discussed in Part 1 are analogous to topo maps and a compass, basic tools we use all the time. Most of the tools in part 2 are are analogous to flare guns and satellite phones. They aren't used too often on a hike, but when we are stuck, they can be invaluable. What follows are the more advanced, less used, or sometimes esoteric capabilities of perl regexps. In Part 2, we will assume you are comfortable with the basics and concentrate on the new features.
More on characters, strings, and character classesThere are a number of escape sequences and character classes that we haven't covered yet. There are several escape sequences that convert characters or strings
between upper and lower case. $x = "perl"; $string =~ /\u$x/; # matches 'Perl' in $string $x = "M(rs?|s)\\."; # note the double backslash $string =~ /\l$x/; # matches 'mr.', 'mrs.', and 'ms.',
$x = "This word is in lower case:\L SHOUT\E"; $x =~ /shout/; # matches $x = "I STILL KEYPUNCH CARDS FOR MY 360" $x =~ /\Ukeypunch/; # matches punch card string If there is no Control characters can be escaped with $x = "\QThat !^*&%~& cat!"; $x =~ /\Q!^*&%~&\E/; # check for rough language It does not protect With the advent of 5.6.0, perl regexps can handle more than just the
standard ASCII character set. Perl now supports Unicode, a standard
for encoding the character sets from many of the world's written
languages. Unicode does this by allowing characters to be more than
one byte wide. Perl uses the UTF-8 encoding, in which ASCII characters
are still encoded as one byte, but characters greater than What does this mean for regexps? Well, regexp users don't need to know
much about perl's internal representation of strings. But they do need
to know 1) how to represent Unicode characters in a regexp and 2) when
a matching operation will treat the string to be searched as a
sequence of bytes (the old way) or as a sequence of Unicode characters
(the new way). The answer to 1) is that Unicode characters greater
than use utf8; # We will be doing Unicode processing /\x{263a}/; # match a Unicode smiley face :) Unicode characters in the range of 128-255 use two hexadecimal digits
with braces: Figuring out the hexadecimal sequence of a Unicode character you want
or deciphering someone else's hexadecimal Unicode regexp is about as
much fun as programming in machine code. So another way to specify
Unicode characters is to use the named character escape
sequence use utf8; # We will be doing Unicode processing use charnames ":full"; # use named chars with Unicode full names $x = "abc\N{MERCURY}def"; $x =~ /\N{MERCURY}/; # matches One can also use short names or restrict names to a certain alphabet: use utf8; # We will be doing Unicode processing use charnames ':full'; print "\N{GREEK SMALL LETTER SIGMA} is called sigma.\n"; use charnames ":short"; print "\N{greek:Sigma} is an upper-case sigma.\n"; use charnames qw(greek); print "\N{sigma} is Greek sigma\n"; A list of full names is found in the file Names.txt in the lib/perl5/5.6.0/unicode directory. The answer to requirement 2), as of 5.6.0, is that if a regexp
contains Unicode characters, the string is searched as a sequence of
Unicode characters. Otherwise, the string is searched as a sequence of
bytes. If the string is being searched as a sequence of Unicode
characters, but matching a single byte is required, we can use the use utf8; # We will be doing Unicode processing use charnames ":full"; # use named chars with Unicode full names $x = "a"; $x =~ /\C/; # matches 'a', eats one byte $x = ""; $x =~ /\C/; # doesn't match, no bytes to match $x = "\N{MERCURY}"; # two-byte Unicode character $x =~ /\C/; # matches, but dangerous! The last regexp matches, but is dangerous because the string
character position is no longer synchronized to the string byte
position. This generates the warning 'Malformed UTF-8
character'. Let us now discuss the rest of the character classes. Just as with
Unicode characters, there are named Unicode character classes
represented by the use utf8; # We will be doing Unicode processing use charnames ":full"; # use named chars with Unicode full names $x = "BOB"; $x =~ /^\p{IsUpper}/; # matches, uppercase char class $x =~ /^\P{IsUpper}/; # doesn't match, char class sans uppercase $x =~ /^\p{IsLower}/; # doesn't match, lowercase char class $x =~ /^\P{IsLower}/; # matches, char class sans lowercase Here is the association between some Perl named classes and the traditional Unicode classes: Perl class name Unicode class name or regular expression IsAlpha /^[LM]/ IsAlnum /^[LMN]/ IsASCII $code <= 127 IsCntrl /^C/ IsBlank $code =~ /^(0020|0009)$/ || /^Z[^lp]/ IsDigit Nd IsGraph /^([LMNPS]|Co)/ IsLower Ll IsPrint /^([LMNPS]|Co|Zs)/ IsPunct /^P/ IsSpace /^Z/ || ($code =~ /^(0009|000A|000B|000C|000D)$/ IsSpacePerl /^Z/ || ($code =~ /^(0009|000A|000C|000D)$/ IsUpper /^L[ut]/ IsWord /^[LMN]/ || $code eq "005F" IsXDigit $code =~ /^00(3[0-9]|[46][1-6])$/ You can also use the official Unicode class names with the
As if all those classes weren't enough, Perl also defines POSIX style
character classes. These have the form /\s+[abc[:digit:]xyz]\s*/; # match a,b,c,x,y,z, or a digit /^=item\s[:digit:]/; # match '=item', # followed by a space and a digit use utf8; use charnames ":full"; /\s+[abc\p{IsDigit}xyz]\s+/; # match a,b,c,x,y,z, or a digit /^=item\s\p{IsDigit}/; # match '=item', # followed by a space and a digit Whew! That is all the rest of the characters and character classes.
Compiling and saving regular expressionsIn Part 1 we discussed the $reg = qr/foo+bar?/; # reg contains a compiled regexp Then $x = "fooooba"; $x =~ $reg; # matches, just like /foo+bar?/ $x =~ /$reg/; # same thing, alternate form
$x =~ /(abc)?$reg/; # still matches As with the matching operator, the regexp quote can use different
delimiters, e.g., Pre-compiled regexps are useful for creating dynamic matches that
don't need to be recompiled each time they are encountered. Using
pre-compiled regexps, % cat > multi_grep #!/usr/bin/perl # multi_grep - match any of <number> regexps # usage: multi_grep <number> regexp1 regexp2 ... file1 file2 ... $number = shift; $regexp[$_] = shift foreach (0..$number-1); @compiled = map qr/$_/, @regexp; while ($line = <>) { foreach $pattern (@compiled) { if ($line =~ /$pattern/) { print $line; last; # we matched, so move onto the next line } } } ^D % multi_grep 2 last for multi_grep $regexp[$_] = shift foreach (0..$number-1); foreach $pattern (@compiled) { last; Storing pre-compiled regexps in an array
Embedding comments and modifiers in a regular expressionStarting with this section, we will be discussing Perl's set of
extended patterns. These are extensions to the traditional regular
expression syntax that provide powerful new tools for pattern
matching. We have already seen extensions in the form of the minimal
matching constructs The first extension is an embedded comment /(?# Match an integer:)[+-]?\d+/; This style of commenting has been largely superseded by the raw,
freeform commenting that is allowed with the The modifiers /(?i)yes/; # match 'yes' case insensitively /yes/i; # same thing /(?x)( # freeform version of an integer regexp [+-]? # match an optional sign \d+ # match a sequence of digits ) /x; Embedded modifiers can have two important advantages over the usual modifiers. Embedded modifiers allow a custom set of modifiers to each regexp pattern. This is great for matching an array of regexps that must have different modifiers: $pattern[0] = '(?i)doctor'; $pattern[1] = 'Johnson'; ... while (<>) { foreach $patt (@pattern) { print if /$patt/; } } The second advantage is that embedded modifiers only affect the regexp inside the group the embedded modifier is contained in. So grouping can be used to localize the modifier's effects: /Answer: ((?i)yes)/; # matches 'Answer: yes', 'Answer: YES', etc. Embedded modifiers can also turn off any modifiers already present
by using, e.g.,
Non-capturing groupingsWe noted in Part 1 that groupings # match a number, $1-$4 are set, but we only want $1 /([+-]?\ *(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?)/; # match a number faster , only $1 is set /([+-]?\ *(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?)/; # match a number, get $1 = whole number, $2 = exponent /([+-]?\ *(?:\d+(?:\.\d*)?|\.\d+)(?:[eE]([+-]?\d+))?)/; Non-capturing groupings are also useful for removing nuisance elements gathered from a split operation: $x = '12a34b5'; @num = split /(a|b)/, $x; # @num = ('12','a','34','b','5') @num = split /(?:a|b)/, $x; # @num = ('12','34','5') Non-capturing groupings may also have embedded modifiers:
Looking ahead and looking behindThis section concerns the lookahead and lookbehind assertions. First, a little background. In Perl regular expressions, most regexp elements 'eat up' a certain
amount of string when they match. For instance, the regexp element
Checking the environment entails either looking ahead on the trail,
looking behind, or both. The lookahead and lookbehind assertions are generalizations of the
anchor concept. Lookahead and lookbehind are zero-width assertions
that let us specify which characters we want to test for. The
lookahead assertion is denoted by $x = "I catch the housecat 'Tom-cat' with catnip"; $x =~ /cat(?=\s+)/; # matches 'cat' in 'housecat' @catwords = ($x =~ /(?<=\s)cat\w+/g); # matches, # $catwords[0] = 'catch' # $catwords[1] = 'catnip' $x =~ /\bcat\b/; # matches 'cat' in 'Tom-cat' $x =~ /(?<=\s)cat(?=\s)/; # doesn't match; no isolated 'cat' in # middle of $x Note that the parentheses in $x = "foobar"; $x =~ /foo(?!bar)/; # doesn't match, 'bar' follows 'foo' $x =~ /foo(?!baz)/; # matches, 'baz' doesn't follow 'foo' $x =~ /(?<!\s)foo/; # matches, there is no \s before 'foo'
Using independent subexpressions to prevent backtrackingThe last few extended patterns in this tutorial are experimental as of 5.6.0. Play with them, use them in some code, but don't rely on them just yet for production code. Independent subexpressions are regular expressions, in the
context of a larger regular expression, that function independently of
the larger regular expression. That is, they consume as much or as
little of the string as they wish without regard for the ability of
the larger regexp to match. Independent subexpressions are represented
by $x = "ab"; $x =~ /a*ab/; # matches This obviously matches, but in the process of matching, the
subexpression Contrast that with an independent subexpression: $x =~ /(?>a*)ab/; # doesn't match! The independent subexpression $x = "ab"; $x =~ /a*/g; # matches, eats an 'a' $x =~ /\Gab/g; # doesn't match, no 'a' available Here The ability of an independent subexpression to prevent backtracking can be quite useful. Suppose we want to match a non-empty string enclosed in parentheses up to two levels deep. Then the following regexp matches: $x = "abc(de(fg)h"; # unbalanced parentheses $x =~ /\( ( [^()]+ | \([^()]*\) )+ \)/x; The regexp matches an open parenthesis, one or more copies of an
alternation, and a close parenthesis. The alternation is two-way, with
the first alternative $x =~ /\( ( (?>[^()]+) | \([^()]*\) )+ \)/x; Here,
Conditional expressionsA conditional expression is a form of if-then-else statement
that allows one to choose which patterns are to be matched, based on
some condition. There are two types of conditional expression:
The The integer form of the % simple_grep '^(\w+)(\w+)?(?(2)\2\1|\1)$' /usr/dict/words beriberi coco couscous deed ... toot toto tutu The lookbehind /[ATGC]+(?(?<=AA)G|C)$/; matches a DNA sequence such that it either ends in
A bit of magic: executing Perl code in a regular expressionNormally, regexps are a part of Perl expressions.
Code evaluation expressions turn that around by allowing
arbitrary Perl code to be a part of of a regexp. A code evaluation
expression is denoted Code expressions are zero-width assertions, and the value they return
depends on their environment. There are two possibilities: either the
code expression is used as a conditional in a conditional expression
$x = "abcdef"; $x =~ /abc(?{print "Hi Mom!";})def/; # matches, # prints 'Hi Mom!' $x =~ /aaa(?{print "Hi Mom!";})def/; # doesn't match, # no 'Hi Mom!' Pay careful attention to the next example: $x =~ /abc(?{print "Hi Mom!";})ddd/; # doesn't match, # no 'Hi Mom!' # but why not? At first glance, you'd think that it shouldn't print, because obviously
the $x =~ /abc(?{print "Hi Mom!";})[d]dd/; # doesn't match, # but _does_ print Hmm. What happened here? If you've been following along, you know that the above pattern should be effectively the same as the last one -- enclosing the d in a character class isn't going to change what it matches. So why does the first not print while the second one does? The answer lies in the optimizations the REx engine makes. In the first
case, all the engine sees are plain old characters (aside from the
To take a closer look at how the engine does optimizations, see the section Pragmas and debugging below. More fun with $x =~ /(?{print "Hi Mom!";})/; # matches, # prints 'Hi Mom!' $x =~ /(?{$c = 1;})(?{print "$c";})/; # matches, # prints '1' $x =~ /(?{$c = 1;})(?{print "$^R";})/; # matches, # prints '1' The bit of magic mentioned in the section title occurs when the regexp
backtracks in the process of searching for a match. If the regexp
backtracks over a code expression and if the variables used within are
localized using $x = "aaaa"; $count = 0; # initialize 'a' count $c = "bob"; # test if $c gets clobbered $x =~ /(?{local $c = 0;}) # initialize count ( a # match 'a' (?{local $c = $c + 1;}) # increment count )* # do this any number of times, aa # but match 'aa' at the end (?{$count = $c;}) # copy local $c var into $count /x; print "'a' count is $count, \$c variable is '$c'\n"; This prints 'a' count is 2, $c variable is 'bob' If we replace the 'a' count is 4, $c variable is 'bob' Note that only localized variable changes are undone. Other side effects of code expression execution are permanent. Thus $x = "aaaa"; $x =~ /(a(?{print "Yow\n";}))*aa/; produces Yow Yow Yow Yow The result This example uses a code expression in a conditional to match the article 'the' in either English or German: $lang = 'DE'; # use German ... $text = "das"; print "matched\n" if $text =~ /(?(?{ $lang eq 'EN'; # is the language English? }) the | # if so, then match 'the' (die|das|der) # else, match 'die|das|der' ) /xi; Note that the syntax here is If you try to use code expressions with interpolating variables, perl may surprise you: $bar = 5; $pat = '(?{ 1 })'; /foo(?{ $bar })bar/; # compiles ok, $bar not interpolated /foo(?{ 1 })$bar/; # compile error! /foo${pat}bar/; # compile error! $pat = qr/(?{ $foo = 1 })/; # precompile code regexp /foo${pat}bar/; # compiles ok If a regexp has (1) code expressions and interpolating variables,or (2) a variable that interpolates a code expression, perl treats the regexp as an error. If the code expression is precompiled into a variable, however, interpolating is ok. The question is, why is this an error? The reason is that variable interpolation and code expressions together pose a security risk. The combination is dangerous because many programmers who write search engines often take user input and plug it directly into a regexp: $regexp = <>; # read user-supplied regexp $chomp $regexp; # get rid of possible newline $text =~ /$regexp/; # search $text for the $regexp If the use re 'eval'; # throw caution out the door $bar = 5; $pat = '(?{ 1 })'; /foo(?{ 1 })$bar/; # compiles ok /foo${pat}bar/; # compiles ok Another form of code expression is the pattern code expression . The pattern code expression is like a regular code expression, except that the result of the code evaluation is treated as a regular expression and matched immediately. A simple example is $length = 5; $char = 'a'; $x = 'aaaaabb'; $x =~ /(??{$char x $length})/x; # matches, there are 5 of 'a' This final example contains both ordinary and pattern code
expressions. It detects if a binary string $s0 = 0; $s1 = 1; # initial conditions $x = "1101010010001000001"; print "It is a Fibonacci sequence\n" if $x =~ /^1 # match an initial '1' ( (??{'0' x $s0}) # match $s0 of '0' 1 # and then a '1' (?{ $largest = $s0; # largest seq so far $s2 = $s1 + $s0; # compute next term $s0 = $s1; # in Fibonacci sequence $s1 = $s2; }) )+ # repeat as needed $ # that is all there is /x; print "Largest sequence matched was $largest\n"; This prints It is a Fibonacci sequence Largest sequence matched was 5 Ha! Try that with your garden variety regexp package... Note that the variables The regexp without the /^1((??{'0'x$s0})1(?{$largest=$s0;$s2=$s1+$s0$s0=$s1;$s1=$s2;}))+$/; and is a great start on an Obfuscated Perl entry :-) When working with code and conditional expressions, the extended form of regexps is almost necessary in creating and debugging regexps.
Pragmas and debuggingSpeaking of debugging, there are several pragmas available to control
and debug regexps in Perl. We have already encountered one pragma in
the previous section, use re 'taint'; $tainted = <>; @parts = ($tainted =~ /(\w+)\s+(\w+)/; # @parts is now tainted The use re 'debug'; /^(.*)$/s; # output debugging info use re 'debugcolor'; /^(.*)$/s; # output debugging info in living color The global % perl -e 'use re "debug"; "abc" =~ /a*b+c/;' Compiling REx `a*b+c' size 9 first at 1 1: STAR(4) 2: EXACT <a>(0) 4: PLUS(7) 5: EXACT <b>(0) 7: EXACT <c>(9) 9: END(0) floating `bc' at 0..2147483647 (checking floating) minlen 2 Guessing start of match, REx `a*b+c' against `abc'... Found floating substr `bc' at offset 1... Guessed: match at offset 0 Matching REx `a*b+c' against `abc' Setting an EVAL scope, savestack=3 0 <> <abc> | 1: STAR EXACT <a> can match 1 times out of 32767... Setting an EVAL scope, savestack=3 1 <a> <bc> | 4: PLUS EXACT <b> can match 1 times out of 32767... Setting an EVAL scope, savestack=3 2 <ab> <c> | 7: EXACT <c> 3 <abc> <> | 9: END Match successful! Freeing REx: `a*b+c' If you have gotten this far into the tutorial, you can probably guess what the different parts of the debugging output tell you. The first part Compiling REx `a*b+c' size 9 first at 1 1: STAR(4) 2: EXACT <a>(0) 4: PLUS(7) 5: EXACT <b>(0) 7: EXACT <c>(9) 9: END(0) describes the compilation stage. floating `bc' at 0..2147483647 (checking floating) minlen 2 Guessing start of match, REx `a*b+c' against `abc'... Found floating substr `bc' at offset 1... Guessed: match at offset 0 Then the match is executed and the remaining lines describe the process: Matching REx `a*b+c' against `abc' Setting an EVAL scope, savestack=3 0 <> <abc> | 1: STAR EXACT <a> can match 1 times out of 32767... Setting an EVAL scope, savestack=3 1 <a> <bc> | 4: PLUS EXACT <b> can match 1 times out of 32767... Setting an EVAL scope, savestack=3 2 <ab> <c> | 7: EXACT <c> 3 <abc> <> | 9: END Match successful! Freeing REx: `a*b+c' Each step is of the form An alternative method of debugging regexps is to embed "that this" =~ m@(?{print "Start at position ", pos, "\n";}) t(?{print "t1\n";}) h(?{print "h1\n";}) i(?{print "i1\n";}) s(?{print "s1\n";}) | t(?{print "t2\n";}) h(?{print "h2\n";}) a(?{print "a2\n";}) t(?{print "t2\n";}) (?{print "Done at position ", pos, "\n";}) @x; prints Start at position 0 t1 h1 t2 h2 a2 t2 Done at position 4
BUGSCode expressions, conditional expressions, and independent expressions are experimental. Don't use them in production code. Yet.
SEE ALSOThis is just a tutorial. For the full story on perl regular expressions, see the the perlre manpage regular expressions reference page. For more information on the matching For an excellent all-around resource on the care and feeding of regular expressions, see the book Mastering Regular Expressions by Jeffrey Friedl (published by O'Reilly, ISBN 1556592-257-3).
AUTHOR AND COPYRIGHTCopyright (c) 2000 Mark Kvale All rights reserved. This document may be distributed under the same terms as Perl itself.
AcknowledgmentsThe inspiration for the stop codon DNA example came from the ZIP code example in chapter 7 of Mastering Regular Expressions. The author would like to thank Jeff Pinyan, Andrew Johnson, Peter Haworth, Ronald J Kimball, and Joe Smith for all their helpful comments.
|