lex.html

IEEE 1003.1-2003, Single Unix Specification v3

<tr valign="top">
<td align="left">
<p class="tent">%<b>e</b> <i>n</i></p>
</td>
<td align="left">
<p class="tent">Number of parse tree nodes</p>
</td>
<td align="left">
<p class="tent">1000</p>
</td>
</tr>

<tr valign="top">
<td align="left">
<p class="tent">%<b>k</b> <i>n</i></p>
</td>
<td align="left">
<p class="tent">Number of packed character classes</p>
</td>
<td align="left">
<p class="tent">1000</p>
</td>
</tr>

<tr valign="top">
<td align="left">
<p class="tent">%<b>o</b> <i>n</i></p>
</td>
<td align="left">
<p class="tent">Size of the output array</p>
</td>
<td align="left">
<p class="tent">3000</p>
</td>
</tr>
</table>
</center>

<p>In the table, <i>n</i> represents a positive decimal integer, preceded by one or more &lt;blank&gt;s. The exact meaning of these
table size numbers is implementation-defined. The implementation shall document how these numbers affect the <i>lex</i> utility and
how they are related to any output that may be generated by the implementation should limitations be encountered during the
execution of <i>lex</i>. It shall be possible to determine from this output which of the table size values needs to be modified to
permit <i>lex</i> to successfully generate tables for the input language. The values in the column Minimum Value represent the
lowest values conforming implementations shall provide.</p>

<h5><a name="tag_04_73_13_02"></a>Rules in lex</h5>

<p>The rules in <i>lex</i> source files are a table in which the left column contains regular expressions and the right column
contains actions (C program fragments) to be executed when the expressions are recognized.</p>

<pre>
<i>ERE action
ERE action</i><tt>...
</tt>
</pre>

<p>The extended regular expression (ERE) portion of a row shall be separated from <i>action</i> by one or more &lt;blank&gt;s. A
regular expression containing &lt;blank&gt;s shall be recognized under one of the following conditions:</p>

<ul>
<li>
<p>The entire expression appears within double-quotes.</p>
</li>

<li>
<p>The &lt;blank&gt;s appear within double-quotes or square brackets.</p>
</li>

<li>
<p>Each &lt;blank&gt; is preceded by a backslash character.</p>
</li>
</ul>

<h5><a name="tag_04_73_13_03"></a>User Subroutines in lex</h5>

<p>Anything in the user subroutines section shall be copied to <b>lex.yy.c</b> following <i>yylex</i>().</p>

<h5><a name="tag_04_73_13_04"></a>Regular Expressions in lex</h5>

<p>The <i>lex</i> utility shall support the set of extended regular expressions (see the Base Definitions volume of
IEEE&nbsp;Std&nbsp;1003.1-2001, <a href="../basedefs/xbd_chap09.html#tag_09_04">Section 9.4, Extended Regular Expressions</a>),
with the following additions and exceptions to the syntax:</p>

<dl compact>
<dt><tt>"..."</tt></dt>

<dd>Any string enclosed in double-quotes shall represent the characters within the double-quotes as themselves, except that
backslash escapes (which appear in the following table) shall be recognized. Any backslash-escape sequence shall be terminated by
the closing quote. For example, <tt>"\01"</tt> <tt>"1"</tt> represents a single string: the octal value 1 followed by the character
<tt>'1'</tt> .</dd>

<dt>&lt;<i>state</i>&gt;<i>r</i>,&nbsp;&lt;<i>state1,state2,</i>...&gt;<i>r</i></dt>

<dd><br>
The regular expression <i>r</i> shall be matched only when the program is in one of the start conditions indicated by <i>state</i>,
<i>state1</i>, and so on; see <a href="#tag_04_73_13_05">Actions in lex</a> . (As an exception to the typographical conventions of
the rest of this volume of IEEE&nbsp;Std&nbsp;1003.1-2001, in this case &lt;<i>state</i>&gt; does not represent a metavariable, but
the literal angle-bracket characters surrounding a symbol.) The start condition shall be recognized as such only at the beginning
of a regular expression.</dd>

<dt><i>r</i>/<i>x</i></dt>

<dd>The regular expression <i>r</i> shall be matched only if it is followed by an occurrence of regular expression <i>x</i> (
<i>x</i> is the instance of trailing context, further defined below). The token returned in <i>yytext</i> shall only match
<i>r</i>. If the trailing portion of <i>r</i> matches the beginning of <i>x</i>, the result is unspecified. The <i>r</i> expression
cannot include further trailing context or the <tt>'$'</tt> (match-end-of-line) operator; <i>x</i> cannot include the <tt>'^'</tt>
(match-beginning-of-line) operator, nor trailing context, nor the <tt>'$'</tt> operator. That is, only one occurrence of trailing
context is allowed in a <i>lex</i> regular expression, and the <tt>'^'</tt> operator only can be used at the beginning of such an
expression.</dd>

<dt>{<i>name</i>}</dt>

<dd>When <i>name</i> is one of the substitution symbols from the <i>Definitions</i> section, the string, including the enclosing
braces, shall be replaced by the <i>substitute</i> value. The <i>substitute</i> value shall be treated in the extended regular
expression as if it were enclosed in parentheses. No substitution shall occur if { <i>name</i>} occurs within a bracket expression
or within double-quotes.</dd>
</dl>

<p>Within an ERE, a backslash character shall be considered to begin an escape sequence as specified in the table in the Base
Definitions volume of IEEE&nbsp;Std&nbsp;1003.1-2001, <a href="../basedefs/xbd_chap05.html">Chapter 5, File Format Notation</a> (
<tt>'\\'</tt> , <tt>'\a'</tt> , <tt>'\b'</tt> , <tt>'\f'</tt> , <tt>'\n'</tt> , <tt>'\r'</tt> , <tt>'\t'</tt> , <tt>'\v'</tt> ). In
addition, the escape sequences in the following table shall be recognized.</p>

<p>A literal &lt;newline&gt; cannot occur within an ERE; the escape sequence <tt>'\n'</tt> can be used to represent a
&lt;newline&gt;. A &lt;newline&gt; shall not be matched by a period operator.<br>
</p>

<center><b>Table: Escape Sequences in <i>lex</i></b></center>

<center>
<table border="1" cellpadding="3" align="center">
<tr valign="top">
<th align="center">
<p class="tent"><b>Escape</b></p>
</th>
<th align="center">
<p class="tent"><b>&nbsp;</b></p>
</th>
<th align="center">
<p class="tent"><b>&nbsp;</b></p>
</th>
</tr>

<tr valign="top">
<th align="center">
<p class="tent"><b>Sequence</b></p>
</th>
<th align="center">
<p class="tent"><b>Description</b></p>
</th>
<th align="center">
<p class="tent"><b>Meaning</b></p>
</th>
</tr>

<tr valign="top">
<td align="left">
<p class="tent">\<i>digits</i></p>
</td>
<td align="left">
<p class="tent">A backslash character followed by the longest sequence of one, two, or three octal-digit characters (01234567). If
all of the digits are 0 (that is, representation of the NUL character), the behavior is undefined.</p>
</td>
<td align="left">
<p class="tent">The character whose encoding is represented by the one, two, or three-digit octal integer. If the size of a byte on
the system is greater than nine bits, the valid escape sequence used to represent a byte is implementation-defined. Multi-byte
characters require multiple, concatenated escape sequences of this type, including the leading <tt>'\'</tt> for each byte.</p>
</td>
</tr>

<tr valign="top">
<td align="left">
<p class="tent">\x<i>digits</i></p>
</td>
<td align="left">
<p class="tent">A backslash character followed by the longest sequence of hexadecimal-digit characters (01234567abcdefABCDEF). If
all of the digits are 0 (that is, representation of the NUL character), the behavior is undefined.</p>
</td>
<td align="left">
<p class="tent">The character whose encoding is represented by the hexadecimal integer.</p>
</td>
</tr>

<tr valign="top">
<td align="left">
<p class="tent">\c</p>
</td>
<td align="left">
<p class="tent">A backslash character followed by any character not described in this table or in the table in the Base Definitions
volume of IEEE&nbsp;Std&nbsp;1003.1-2001, <a href="../basedefs/xbd_chap05.html">Chapter 5, File Format Notation</a> ( <tt>'\\'</tt>
, <tt>'\a'</tt> , <tt>'\b'</tt> , <tt>'\f'</tt> , <tt>'\n'</tt> , <tt>'\r'</tt> , <tt>'\t'</tt> , <tt>'\v'</tt> ).</p>
</td>
<td align="left">
<p class="tent">The character <tt>'c'</tt> , unchanged.</p>
</td>
</tr>
</table>
</center>

<basefont size="2"> 

<dl>
<dt><b>Note:</b></dt>

<dd>If a <tt>'\x'</tt> sequence needs to be immediately followed by a hexadecimal digit character, a sequence such as
<tt>"\x1"</tt> <tt>"1"</tt> can be used, which represents a character containing the value 1, followed by the character
<tt>'1'</tt> .</dd>
</dl>

<basefont size="3"> 

<p>The order of precedence given to extended regular expressions for <i>lex</i> differs from that specified in the Base Definitions
volume of IEEE&nbsp;Std&nbsp;1003.1-2001, <a href="../basedefs/xbd_chap09.html#tag_09_04">Section 9.4, Extended Regular
Expressions</a>. The order of precedence for <i>lex</i> shall be as shown in the following table, from high to low. <basefont size=
"2"></p>

<dl>
<dt><b>Note:</b></dt>

<dd>The escaped characters entry is not meant to imply that these are operators, but they are included in the table to show their
relationships to the true operators. The start condition, trailing context, and anchoring notations have been omitted from the
table because of the placement restrictions described in this section; they can only appear at the beginning or ending of an
ERE.</dd>
</dl>

<basefont size="3"><br>
<center><b>Table: ERE Precedence in <i>lex</i></b></center>

<center>
<table border="1" cellpadding="3" align="center">
<tr valign="top">
<th align="center">
<p class="tent"><b>Extended Regular Expression</b></p>
</th>
<th align="center">
<p class="tent"><b>Precedence</b></p>
</th>
</tr>

<tr valign="top">
<td align="left">
<p class="tent">collation-related bracket symbols</p>
</td>
<td align="left">
<p class="tent">[= =] [: :] [. .]</p>
</td>
</tr>

<tr valign="top">
<td align="left">
<p class="tent">escaped characters</p>
</td>
<td align="left">
<p class="tent">\&lt;<i>special character</i>&gt;</p>
</td>
</tr>

<tr valign="top">
<td align="left">
<p class="tent">bracket expression</p>
</td>
<td align="left">
<p class="tent">[ ]</p>
</td>
</tr>

<tr valign="top">
<td align="left">
<p class="tent">quoting</p>
</td>
<td align="left">
<p class="tent">"..."</p>
</td>
</tr>

<tr valign="top">
<td align="left">
<p class="tent">grouping</p>
</td>
<td align="left">
<p class="tent">( )</p>
</td>
</tr>

<tr valign="top">
<td align="left">
<p class="tent">definition</p>
</td>
<td align="left">
<p class="tent">{<i>name</i>}</p>
</td>
</tr>

<tr valign="top">
<td align="left">
<p class="tent">single-character RE duplication</p>
</td>
<td align="left">
<p class="tent">* + ?</p>
</td>
</tr>

<tr valign="top">
<td align="left">
<p class="tent">concatenation</p>
</td>
<td align="left">
<p class="tent">&nbsp;</p>
</td>
</tr>

<tr valign="top">
<td align="left">
<p class="tent">interval expression</p>
</td>
<td align="left">
<p class="tent">{m,n}</p>
</td>
</tr>

<tr valign="top">
<td align="left">
<p class="tent">alternation</p>
</td>
<td align="left">
<p class="tent">|</p>
</td>
</tr>
</table>
</center>

<p>The ERE anchoring operators <tt>'^'</tt> and <tt>'$'</tt> do not appear in the table. With <i>lex</i> regular expressions, these
operators are restricted in their use: the <tt>'^'</tt> operator can only be used at the beginning of an entire regular expression,
and the <tt>'$'</tt> operator only at the end. The operators apply to the entire regular expression. Thus, for example, the pattern
<tt>"(^abc)|(def$)"</tt> is undefined; it can instead be written as two separate rules, one with the regular expression
<tt>"^abc"</tt> and one with <tt>"def$"</tt> , which share a common action via the special <tt>'|'</tt> action (see below). If the
pattern were written <tt>"^abc|def$"</tt> , it would match either <tt>"abc"</tt> or <tt>"def"</tt> on a line by itself.</p>

<p>Unlike the general ERE rules, embedded anchoring is not allowed by most historical <i>lex</i> implementations. An example of
embedded anchoring would be for patterns such as <tt>"(^|&nbsp;)foo(&nbsp;|$)"</tt> to match <tt>"foo"</tt> when it exists as a
complete word. This functionality can be obtained using existing <i>lex</i> features:</p>

<pre>
<tt>^foo/[ \n]      |
" foo"/[ \n]    /* Found foo as a separate word. */
</tt>
</pre>

<p>Note also that <tt>'$'</tt> is a form of trailing context (it is equivalent to <tt>"/\n"</tt> ) and as such cannot be used with
regular expressions containing another instance of the operator (see the preceding discussion of trailing context).</p>

<p>The additional regular expressions trailing-context operator <tt>'/'</tt> can be used as an ordinary character if presented
within double-quotes, <tt>"/"</tt> ; preceded by a backslash, <tt>"\/"</tt> ; or within a bracket expression, <tt>"[/]"</tt> . The