ch26_04.htm

the unix power tools

><A
CLASS="indexterm"
NAME="AUTOID-28807"
></A
>Searching for a word isn't quite as simple as it at first appears.
The string
<CODE
CLASS="literal"
>the</CODE
>
will match the word 
<CODE
CLASS="literal"
>other</CODE
>.
You can put spaces before and after the letters and use this regular
expression:
<IMG
SRC="../chars/squ.gif"
ALT=" "><CODE
CLASS="literal"
>the</CODE
><IMG
SRC="../chars/squ.gif"
ALT=" ">.
However, this does not match words at the beginning or the end of the line.
And it does not match the case where there is a punctuation mark
after the word. </P
><P
CLASS="para"
>There is an easy solution&nbsp;- at least in many versions of <EM
CLASS="emphasis"
>ed</EM
>, <EM
CLASS="emphasis"
>ex</EM
>, and
<EM
CLASS="emphasis"
>vi</EM
>.
The characters 
<CODE
CLASS="literal"
>\&lt;</CODE
>
and
<CODE
CLASS="literal"
>\&gt;</CODE
>
are similar to the
<CODE
CLASS="literal"
>^</CODE
>
and
<CODE
CLASS="literal"
>$</CODE
>
anchors,
as they don't occupy a position of a character.
They do 
<EM
CLASS="emphasis"
>anchor</EM
>
the expression between to match only if it is on a word boundary.
The pattern to search for the words
<CODE
CLASS="literal"
>the</CODE
> and <CODE
CLASS="literal"
>The</CODE
>
would be:
<CODE
CLASS="literal"
>\&lt;[tT]he\&gt;</CODE
>.</P
><P
CLASS="para"
>Let's define a &quot;word boundary.&quot;
The character before the
<CODE
CLASS="literal"
>t</CODE
> or <CODE
CLASS="literal"
>T</CODE
>
must be either a newline character or anything except a letter,
digit, or underscore (&nbsp;<CODE
CLASS="literal"
>_</CODE
>&nbsp;).
The character after the
<CODE
CLASS="literal"
>e</CODE
>
must

also be a character other than a digit, letter, or underscore,
or it could be the end-of-line character.</P
></DIV
><DIV
CLASS="sect2"
><H3
CLASS="sect2"
><A
CLASS="title"
NAME="UPT-ART-427-SECT-1.9"
>26.4.9 Remembering Patterns with \&nbsp;(, \&nbsp;), and \1 </A
></H3
><P
CLASS="para"
><A
CLASS="indexterm"
NAME="AUTOID-28833"
></A
><A
CLASS="indexterm"
NAME="AUTOID-28836"
></A
><A
CLASS="indexterm"
NAME="AUTOID-28839"
></A
>Another pattern that requires a special mechanism is searching for
repeated words.
The expression
<CODE
CLASS="literal"
>[a-z][a-z]</CODE
>
will match any two lowercase letters.
If you wanted to search for lines that had two adjoining identical
letters, the above pattern wouldn't help.
You need a way to remember what you found and see if 
the same pattern occurs again.
In some programs, 
you can mark part of a pattern using
<CODE
CLASS="literal"
>\(</CODE
>
and
<CODE
CLASS="literal"
>\)</CODE
>.
You can recall the remembered pattern with
<CODE
CLASS="literal"
>\</CODE
> 
followed by a single digit.
Therefore, to search for two identical letters, use:
<CODE
CLASS="literal"
>\([a-z]\)\1</CODE
>.
You can have nine different remembered patterns. 
Each occurrence of 
<CODE
CLASS="literal"
>\(</CODE
>
starts a new pattern.
The regular expression to match a five-letter palindrome 
(e.g., &quot;radar&quot;) is:
<CODE
CLASS="literal"
>\([a-z]\)\([a-z]\)[a-z]\2\1</CODE
>.
[Some versions of some programs can't handle <CODE
CLASS="literal"
>\(&nbsp;\)</CODE
> in the same
regular expression as <CODE
CLASS="literal"
>\</CODE
><CODE
CLASS="replaceable"
><I
>1</I
></CODE
>, etc.
In all versions of <EM
CLASS="emphasis"
>sed</EM
>, you're safe if you use
<SPAN
CLASS="link"
> <CODE
CLASS="literal"
>/( /)</CODE
> on the pattern side of an <EM
CLASS="emphasis"
>s</EM
> command-and<CODE
CLASS="literal"
>/</CODE
><CODE
CLASS="replaceable"
><I
>1</I
></CODE
>, etc., on the replacement side . (<A
CLASS="linkend"
HREF="ch34_10.htm"
TITLE="Referencing Portions of a Search String "
>34.10</A
>)</SPAN
>
<EM
CLASS="emphasis"
>-JP</EM
> ]</P
></DIV
><DIV
CLASS="sect2"
><H3
CLASS="sect2"
><A
CLASS="title"
NAME="UPT-ART-427-SECT-1.10"
>26.4.10 Potential Problems </A
></H3
><P
CLASS="para"
>That completes a discussion of simple regular expressions.
Before I discuss the extensions that extended expressions offer, I
want to mention two potential problem areas.</P
><P
CLASS="para"
>The 
<CODE
CLASS="literal"
>/&lt;</CODE
>
and
<CODE
CLASS="literal"
>/&gt;</CODE
>
characters were introduced in the
<EM
CLASS="emphasis"
>vi</EM
>
editor. The other programs didn't have this ability at that time.
Also, the
<CODE
CLASS="literal"
>/{</CODE
><CODE
CLASS="replaceable"
><I
>min</I
></CODE
><CODE
CLASS="literal"
>,</CODE
><CODE
CLASS="replaceable"
><I
>max</I
></CODE
><CODE
CLASS="literal"
>/}</CODE
>
modifier is new, and earlier utilities didn't have this ability.
This makes it difficult for the novice user of regular expressions,
because it seems as if each utility has a different convention.
Sun has retrofitted the newest regular expression library to all of
their programs, so they all have the same ability.
If you try to use these newer features on other vendors' machines, you
might find they don't work the same way.</P
><P
CLASS="para"
>The other potential point of confusion is the 
<SPAN
CLASS="link"
>extent of the pattern matches (<A
CLASS="linkend"
HREF="ch26_06.htm"
TITLE="Just What Does a Regular Expression Match? "
>26.6</A
>)</SPAN
>.
Regular expressions match the longest possible pattern.
That is, the regular expression
<CODE
CLASS="literal"
>A.*B</CODE
>
matches 
<CODE
CLASS="literal"
>AAB</CODE
>
as well as 
<CODE
CLASS="literal"
>AAAABBBBABCCCCBBBAAAB</CODE
>.
This doesn't cause many problems using
<EM
CLASS="emphasis"
>grep</EM
>,
because an oversight in a regular expression will just match more
lines than desired.
If you use 
<EM
CLASS="emphasis"
>sed</EM
>,
and your patterns get carried away, you may end up deleting or
changing more than you want to.</P
></DIV
><DIV
CLASS="sect2"
><H3
CLASS="sect2"
><A
CLASS="title"
NAME="UPT-ART-427-SECT-1.11"
>26.4.11 Extended Regular Expressions </A
></H3
><P
CLASS="para"
><A
CLASS="indexterm"
NAME="AUTOID-28881"
></A
><A
CLASS="indexterm"
NAME="AUTOID-28883"
></A
><A
CLASS="indexterm"
NAME="AUTOID-28886"
></A
><A
CLASS="indexterm"
NAME="AUTOID-28888"
></A
><A
CLASS="indexterm"
NAME="AUTOID-28891"
></A
>Two programs use extended regular expressions:
<EM
CLASS="emphasis"
>egrep</EM
>
and 
<EM
CLASS="emphasis"
>awk</EM
>.
[<EM
CLASS="emphasis"
>perl</EM
> uses expressions that are even more extended. <EM
CLASS="emphasis"
>-JP</EM
> ]
With these extensions, those special characters preceded by a backslash
no longer have special meaning:
<CODE
CLASS="literal"
>/{</CODE
>,
<CODE
CLASS="literal"
>/}</CODE
>,
<CODE
CLASS="literal"
>/&lt;</CODE
>,
<CODE
CLASS="literal"
>/&gt;</CODE
>,
<CODE
CLASS="literal"
>/(</CODE
>,
<CODE
CLASS="literal"
>/)</CODE
>,
as well as 
<CODE
CLASS="literal"
>/</CODE
><CODE
CLASS="replaceable"
><I
>digit</I
></CODE
>.
There is a very good reason for this, which I will
delay explaining to build up suspense.</P
><P
CLASS="para"
><A
CLASS="indexterm"
NAME="AUTOID-28906"
></A
>The 
question mark (<CODE
CLASS="literal"
>?</CODE
>) 
matches zero or one instances of the character set before it, and the<A
CLASS="indexterm"
NAME="AUTOID-28910"
></A
>
plus sign (<CODE
CLASS="literal"
>+</CODE
>)
matches one or more copies of the character set.
You can't use <CODE
CLASS="literal"
>/{</CODE
> and <CODE
CLASS="literal"
>/}</CODE
> in extended regular expressions,
but if you could, you might consider
<CODE
CLASS="literal"
>?</CODE
>
to be the same as
<CODE
CLASS="literal"
>/{0,1/}</CODE
>
and 
<CODE
CLASS="literal"
>+</CODE
>
to be the same as
<CODE
CLASS="literal"
>/{1,/}</CODE
>.</P
><P
CLASS="para"
>By now, you are wondering why the extended regular expressions 
are even worth using. Except for two abbreviations, there seem to be no
advantages and a lot of disadvantages.
Therefore, examples would be useful.</P
><P
CLASS="para"
>The three important characters in the expanded regular expressions are
<CODE
CLASS="literal"
>(</CODE
>,
<CODE
CLASS="literal"
>|</CODE
>,
and 
<CODE
CLASS="literal"
>)</CODE
>.<A
CLASS="indexterm"
NAME="AUTOID-28925"
></A
><A
CLASS="indexterm"
NAME="AUTOID-28928"
></A
>
Parentheses are used to group expressions; the vertical bar acts an
an OR operator.
Together, they let you match a
<EM
CLASS="emphasis"
>choice</EM
>
of patterns.
As an example, you can 
use <EM
CLASS="emphasis"
>egrep</EM
>
to print all 
<CODE
CLASS="literal"
>From:</CODE
> 
and 
<CODE
CLASS="literal"
>Subject:</CODE
>
lines from your incoming mail:</P
><P
CLASS="para"
><BLOCKQUOTE
CLASS="screen"
><PRE
CLASS="screen"
>% <CODE
CLASS="userinput"
><B
>egrep '^(From|Subject): ' /usr/spool/mail/$USER</B
></CODE
></PRE
></BLOCKQUOTE
></P
><P
CLASS="para"
>All lines starting with 
<CODE
CLASS="literal"
>From:</CODE
>
or
<CODE
CLASS="literal"
>Subject:</CODE
>
will be printed. There is no easy way to do this with simple
regular expressions. You could try something like
<CODE
CLASS="literal"
>^[FS][ru][ob][mj]e*c*t*:</CODE
>
and hope you don't have any lines that start with
<CODE
CLASS="literal"
>Sromeet:</CODE
>.
Extended expressions don't have
the 
<CODE
CLASS="literal"
>/&lt;</CODE
>
and
<CODE
CLASS="literal"
>/&gt;</CODE
>
characters.
You can compensate by using the alternation mechanism.
Matching the word
&quot;the&quot;
in the beginning, middle, or end of a sentence or at the end of a line can be
done with the extended regular expression:
<CODE
CLASS="literal"
>(^| )the([^a-z]|$)</CODE
>.
There are two choices before the word: a space or the beginning of a
line.
Following the word, there must be something besides a lowercase letter or
else the end of the line.
One extra bonus with extended regular expressions is the ability to
use the
<CODE
CLASS="literal"
>*</CODE
>,
<CODE
CLASS="literal"
>+</CODE
>,
and 
<CODE
CLASS="literal"
>?</CODE
>
modifiers after a 
<CODE
CLASS="literal"
>(...)</CODE
>
grouping.
Here are two ways to match
&quot;a simple problem&quot;,
&quot;an easy problem&quot;,
as well as
&quot;a problem&quot;;
the second expression is more exact:</P
><P
CLASS="para"
><BLOCKQUOTE
CLASS="screen"
><PRE
CLASS="screen"
>% <CODE
CLASS="userinput"
><B
>egrep &quot;a[n]? (simple|easy)? ?problem&quot; data</B
></CODE
>
% <CODE
CLASS="userinput"
><B
>egrep &quot;a[n]? ((simple|easy) )?problem&quot; data</B
></CODE
></PRE
></BLOCKQUOTE
></P
><P
CLASS="para"
>I promised to explain why the backslash characters don't work in
extended regular expressions.
Well, perhaps the
<CODE
CLASS="literal"
>/{.../}</CODE
>
and
<A
CLASS="indexterm"
NAME="AUTOID-28956"
></A
><A
CLASS="indexterm"
NAME="AUTOID-28959"
></A
>
<A
CLASS="indexterm"
NAME="AUTOID-28962"
></A
><A
CLASS="indexterm"
NAME="AUTOID-28965"
></A
>

<CODE
CLASS="literal"
>/&lt;.../&gt;</CODE
>
could be added to the extended expressions, but 
it might confuse people if those characters are added and the
<CODE
CLASS="literal"
>/(.../)</CODE
>
are not. And there is no way to add that functionality to the extended
expressions without changing the current usage. Do you see why?
It's quite simple. If
<CODE
CLASS="literal"
>(</CODE
>
has a special meaning, then 
<CODE
CLASS="literal"
>/(</CODE
>
must be the ordinary character.
This is the opposite of the simple regular expressions,
where
<CODE
CLASS="literal"
>(</CODE
>
is ordinary and
<CODE
CLASS="literal"
>/(</CODE
>
is special.
The usage of the parentheses is incompatible, and any change could
break old programs.</P
><P
CLASS="para"
>If the extended expression used 
<CODE
CLASS="literal"
>(...|...)</CODE
>
as regular characters, and
<CODE
CLASS="literal"
>/(.../|.../)</CODE
>
for specifying alternate patterns, then it is possible to have one set
of regular expressions that has full functionality.
This is exactly
what
<SPAN
CLASS="link"
>GNU Emacs (<A
CLASS="linkend"
HREF="ch32_01.htm#UPT-ART-5540"
TITLE="Emacs: The Other Editor "
>32.1</A
>)</SPAN
>
does, by the way-it combines
all of the features of regular and
extended expressions with one syntax.<A
CLASS="indexterm"
NAME="AUTOID-28978"
></A
></P
></DIV
><DIV
CLASS="sect1info"
><P
CLASS="SECT1INFO"
>- <SPAN
CLASS="authorinitials"
>BB</SPAN
></P
></DIV
></DIV
><DIV
CLASS="htmlnav"
><P
></P
><HR
ALIGN="LEFT"
WIDTH="515"
TITLE="footer"><TABLE
WIDTH="515"
BORDER="0"
CELLSPACING="0"
CELLPADDING="0"
><TR
><TD
ALIGN="LEFT"
VALIGN="TOP"
WIDTH="172"
><A
CLASS="SECT1"
HREF="ch26_03.htm"
TITLE="26.3 Understanding Expressions "
><IMG
SRC="gifs/txtpreva.gif"
SRC="gifs/txtpreva.gif"
ALT="Previous: 26.3 Understanding Expressions "
BORDER="0"></A
></TD
><TD
ALIGN="CENTER"
VALIGN="TOP"
WIDTH="171"
><A
CLASS="book"
HREF="index.htm"
TITLE="UNIX Power Tools"
><IMG
SRC="gifs/txthome.gif"
SRC="gifs/txthome.gif"
ALT="UNIX Power Tools"
BORDER="0"></A
></TD
><TD
ALIGN="RIGHT"
VALIGN="TOP"
WIDTH="172"
><A
CLASS="SECT1"
HREF="ch26_05.htm"
TITLE="26.5 Getting Regular Expressions Right "
><IMG
SRC="gifs/txtnexta.gif"
SRC="gifs/txtnexta.gif"
ALT="Next: 26.5 Getting Regular Expressions Right "
BORDER="0"></A
></TD
></TR
><TR
><TD
ALIGN="LEFT"
VALIGN="TOP"
WIDTH="172"
>26.3 Understanding Expressions </TD
><TD
ALIGN="CENTER"
VALIGN="TOP"
WIDTH="171"
><A
CLASS="index"
HREF="index/idx_0.htm"
TITLE="Book Index"
><IMG
SRC="gifs/index.gif"
SRC="gifs/index.gif"
ALT="Book Index"
BORDER="0"></A
></TD
><TD
ALIGN="RIGHT"
VALIGN="TOP"
WIDTH="172"
>26.5 Getting Regular Expressions Right </TD
></TR
></TABLE
><HR
ALIGN="LEFT"
WIDTH="515"
TITLE="footer"><IMG
SRC="gifs/smnavbar.gif"
SRC="gifs/smnavbar.gif"
USEMAP="#map"
BORDER="0"
ALT="The UNIX CD Bookshelf Navigation"><MAP
NAME="map"
><AREA
SHAPE="RECT"
COORDS="0,0,73,21"
HREF="../index.htm"
ALT="The UNIX CD Bookshelf"><AREA
SHAPE="RECT"
COORDS="74,0,163,21"
HREF="index.htm"
ALT="UNIX Power Tools"><AREA
SHAPE="RECT"
COORDS="164,0,257,21"
HREF="../unixnut/index.htm"
ALT="UNIX in a Nutshell"><AREA
SHAPE="RECT"
COORDS="258,0,321,21"
HREF="../vi/index.htm"
ALT="Learning the vi Editor"><AREA
SHAPE="RECT"
COORDS="322,0,378,21"
HREF="../sedawk/index.htm"
ALT="sed &amp; awk"><AREA
SHAPE="RECT"
COORDS="379,0,438,21"
HREF="../ksh/index.htm"
ALT="Learning the Korn Shell"><AREA
SHAPE="RECT"
COORDS="439,0,514,21"
HREF="../lrnunix/index.htm"
ALT="Learning the UNIX Operating System"></MAP
></DIV
></BODY
></HTML
>