flex.texi

Flex词法/语法分析器源码

allows actions to begin with @samp{%@{} and will consider the
action to be all the text up to the next @samp{%@}} (regardless of
ordinary braces inside the action).

An action consisting solely of a vertical bar ('|') means
"same as the action for the next rule." See below for an
illustration.

Actions can include arbitrary C code, including @code{return}
statements to return a value to whatever routine called
@samp{yylex()}.  Each time @samp{yylex()} is called it continues
processing tokens from where it last left off until it either
reaches the end of the file or executes a return.

Actions are free to modify @code{yytext} except for lengthening
it (adding characters to its end--these will overwrite
later characters in the input stream).  This however does
not apply when using @samp{%array} (see above); in that case,
@code{yytext} may be freely modified in any way.

Actions are free to modify @code{yyleng} except they should not
do so if the action also includes use of @samp{yymore()} (see
below).

There are a number of special directives which can be
included within an action:

@itemize -
@item
@samp{ECHO} copies yytext to the scanner's output.

@item
@code{BEGIN} followed by the name of a start condition
places the scanner in the corresponding start
condition (see below).

@item
@code{REJECT} directs the scanner to proceed on to the
"second best" rule which matched the input (or a
prefix of the input).  The rule is chosen as
described above in "How the Input is Matched", and
@code{yytext} and @code{yyleng} set up appropriately.  It may
either be one which matched as much text as the
originally chosen rule but came later in the @code{flex}
input file, or one which matched less text.  For
example, the following will both count the words in
the input and call the routine special() whenever
"frob" is seen:

@example
        int word_count = 0;
%%

frob        special(); REJECT;
[^ \t\n]+   ++word_count;
@end example

Without the @code{REJECT}, any "frob"'s in the input would
not be counted as words, since the scanner normally
executes only one action per token.  Multiple
@code{REJECT's} are allowed, each one finding the next
best choice to the currently active rule.  For
example, when the following scanner scans the token
"abcd", it will write "abcdabcaba" to the output:

@example
%%
a        |
ab       |
abc      |
abcd     ECHO; REJECT;
.|\n     /* eat up any unmatched character */
@end example

(The first three rules share the fourth's action
since they use the special '|' action.)  @code{REJECT} is
a particularly expensive feature in terms of
scanner performance; if it is used in @emph{any} of the
scanner's actions it will slow down @emph{all} of the
scanner's matching.  Furthermore, @code{REJECT} cannot be used
with the @samp{-Cf} or @samp{-CF} options (see below).

Note also that unlike the other special actions,
@code{REJECT} is a @emph{branch}; code immediately following it
in the action will @emph{not} be executed.

@item
@samp{yymore()} tells the scanner that the next time it
matches a rule, the corresponding token should be
@emph{appended} onto the current value of @code{yytext} rather
than replacing it.  For example, given the input
"mega-kludge" the following will write
"mega-mega-kludge" to the output:

@example
%%
mega-    ECHO; yymore();
kludge   ECHO;
@end example

First "mega-" is matched and echoed to the output.
Then "kludge" is matched, but the previous "mega-"
is still hanging around at the beginning of @code{yytext}
so the @samp{ECHO} for the "kludge" rule will actually
write "mega-kludge".
@end itemize

Two notes regarding use of @samp{yymore()}.  First, @samp{yymore()}
depends on the value of @code{yyleng} correctly reflecting the
size of the current token, so you must not modify @code{yyleng}
if you are using @samp{yymore()}.  Second, the presence of
@samp{yymore()} in the scanner's action entails a minor
performance penalty in the scanner's matching speed.

@itemize -
@item
@samp{yyless(n)} returns all but the first @var{n} characters of
the current token back to the input stream, where
they will be rescanned when the scanner looks for
the next match.  @code{yytext} and @code{yyleng} are adjusted
appropriately (e.g., @code{yyleng} will now be equal to @var{n}
).  For example, on the input "foobar" the
following will write out "foobarbar":

@example
%%
foobar    ECHO; yyless(3);
[a-z]+    ECHO;
@end example

An argument of 0 to @code{yyless} will cause the entire
current input string to be scanned again.  Unless
you've changed how the scanner will subsequently
process its input (using @code{BEGIN}, for example), this
will result in an endless loop.

Note that @code{yyless} is a macro and can only be used in the
flex input file, not from other source files.

@item
@samp{unput(c)} puts the character @code{c} back onto the input
stream.  It will be the next character scanned.
The following action will take the current token
and cause it to be rescanned enclosed in
parentheses.

@example
@{
int i;
/* Copy yytext because unput() trashes yytext */
char *yycopy = strdup( yytext );
unput( ')' );
for ( i = yyleng - 1; i >= 0; --i )
    unput( yycopy[i] );
unput( '(' );
free( yycopy );
@}
@end example

Note that since each @samp{unput()} puts the given
character back at the @emph{beginning} of the input stream,
pushing back strings must be done back-to-front.
An important potential problem when using @samp{unput()} is that
if you are using @samp{%pointer} (the default), a call to @samp{unput()}
@emph{destroys} the contents of @code{yytext}, starting with its
rightmost character and devouring one character to the left
with each call.  If you need the value of yytext preserved
after a call to @samp{unput()} (as in the above example), you
must either first copy it elsewhere, or build your scanner
using @samp{%array} instead (see How The Input Is Matched).

Finally, note that you cannot put back @code{EOF} to attempt to
mark the input stream with an end-of-file.

@item
@samp{input()} reads the next character from the input
stream.  For example, the following is one way to
eat up C comments:

@example
%%
"/*"        @{
            register int c;

            for ( ; ; )
                @{
                while ( (c = input()) != '*' &&
                        c != EOF )
                    ;    /* eat up text of comment */

                if ( c == '*' )
                    @{
                    while ( (c = input()) == '*' )
                        ;
                    if ( c == '/' )
                        break;    /* found the end */
                    @}

                if ( c == EOF )
                    @{
                    error( "EOF in comment" );
                    break;
                    @}
                @}
            @}
@end example

(Note that if the scanner is compiled using @samp{C++},
then @samp{input()} is instead referred to as @samp{yyinput()},
in order to avoid a name clash with the @samp{C++} stream
by the name of @code{input}.)

@item YY_FLUSH_BUFFER
flushes the scanner's internal buffer so that the next time the scanner
attempts to match a token, it will first refill the buffer using
@code{YY_INPUT} (see The Generated Scanner, below).  This action is
a special case of the more general @samp{yy_flush_buffer()} function,
described below in the section Multiple Input Buffers.

@item
@samp{yyterminate()} can be used in lieu of a return
statement in an action.  It terminates the scanner
and returns a 0 to the scanner's caller, indicating
"all done".  By default, @samp{yyterminate()} is also
called when an end-of-file is encountered.  It is a
macro and may be redefined.
@end itemize

@node Generated scanner, Start conditions, Actions, Top
@section The generated scanner

The output of @code{flex} is the file @file{lex.yy.c}, which contains
the scanning routine @samp{yylex()}, a number of tables used by
it for matching tokens, and a number of auxiliary routines
and macros.  By default, @samp{yylex()} is declared as follows:

@example
int yylex()
    @{
    @dots{} various definitions and the actions in here @dots{}
    @}
@end example

(If your environment supports function prototypes, then it
will be "int yylex( void  )".)   This  definition  may  be
changed by defining the "YY_DECL" macro.  For example, you
could use:

@example
#define YY_DECL float lexscan( a, b ) float a, b;
@end example

to give the scanning routine the name @code{lexscan}, returning a
float, and taking two floats as arguments.  Note that if
you give arguments to the scanning routine using a
K&R-style/non-prototyped function declaration, you must
terminate the definition with a semi-colon (@samp{;}).

Whenever @samp{yylex()} is called, it scans tokens from the
global input file @code{yyin} (which defaults to stdin).  It
continues until it either reaches an end-of-file (at which
point it returns the value 0) or one of its actions
executes a @code{return} statement.

If the scanner reaches an end-of-file, subsequent calls are undefined
unless either @code{yyin} is pointed at a new input file (in which case
scanning continues from that file), or @samp{yyrestart()} is called.
@samp{yyrestart()} takes one argument, a @samp{FILE *} pointer (which
can be nil, if you've set up @code{YY_INPUT} to scan from a source
other than @code{yyin}), and initializes @code{yyin} for scanning from
that file.  Essentially there is no difference between just assigning
@code{yyin} to a new input file or using @samp{yyrestart()} to do so;
the latter is available for compatibility with previous versions of
@code{flex}, and because it can be used to switch input files in the
middle of scanning.  It can also be used to throw away the current
input buffer, by calling it with an argument of @code{yyin}; but
better is to use @code{YY_FLUSH_BUFFER} (see above).  Note that
@samp{yyrestart()} does @emph{not} reset the start condition to
@code{INITIAL} (see Start Conditions, below).


If @samp{yylex()} stops scanning due to executing a @code{return}
statement in one of the actions, the scanner may then be called
again and it will resume scanning where it left off.

By default (and for purposes of efficiency), the scanner
uses block-reads rather than simple @samp{getc()} calls to read
characters from @code{yyin}.  The nature of how it gets its input
can be controlled by defining the @code{YY_INPUT} macro.
YY_INPUT's calling sequence is
"YY_INPUT(buf,result,max_size)".  Its action is to place
up to @var{max_size} characters in the character array @var{buf} and
return in the integer variable @var{result} either the number of
characters read or the constant YY_NULL (0 on Unix
systems) to indicate EOF.  The default YY_INPUT reads from
the global file-pointer "yyin".

A sample definition of YY_INPUT (in the definitions
section of the input file):

@example
%@{
#define YY_INPUT(buf,result,max_size) \
    @{ \
    int c = getchar(); \
    result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \
    @}
%@}
@end example

This definition will change the input processing to occur
one character at a time.

When the scanner receives an end-of-file indication from
YY_INPUT, it then checks the @samp{yywrap()} function.  If
@samp{yywrap()} returns false (zero), then it is assumed that the
function has gone ahead and set up @code{yyin} to point to
another input file, and scanning continues.  If it returns
true (non-zero), then the scanner terminates, returning 0
to its caller.  Note that in either case, the start
condition remains unchanged; it does @emph{not} revert to @code{INITIAL}.

If you do not supply your own version of @samp{yywrap()}, then you
must either use @samp{%option noyywrap} (in which case the scanner
behaves as though @samp{yywrap()} returned 1), or you must link with
@samp{-lfl} to obtain the default version of the routine, which always
returns 1.

Three routines are available for scanning from in-memory
buffers rather than files: @samp{yy_scan_string()},
@samp{yy_scan_bytes()}, and @samp{yy_scan_buffer()}.  See the discussion
of them below in the section Multiple Input Buffers.

The scanner writes its @samp{ECHO} output to the @code{yyout} global
(default, stdout), which may be redefined by the user
simply by assigning it to some other @code{FILE} pointer.

@node Start conditions, Multiple buffers, Generated scanner, Top
@section Start conditions

@code{flex} provides a mechanism for conditionally activating
rules.  Any rule whose pattern is prefixed with "<sc>"
will only be active when the scanner is in the start
condition named "sc".  For example,

@example
<STRING>[^"]*        @{ /* eat up the string body ... */
            @dots{}
            @}
@end example

@noindent
will be active only when the scanner is in the "STRING"
start condition, and

@example
<INITIAL,STRING,QUOTE>\.        @{ /* handle an escape ... */
            @dots{}
            @}
@end example

@noindent
will be active only when the current start condition is
either "INITIAL", "STRING", or "QUOTE".

Start conditions are declared in the definitions (first)
section of the input using unindented lines beginning with
either @samp{%s} or @samp{%x} followed by a list of names.  The former
declares @emph{inclusive} start conditions, the latter @emph{exclusive}
start conditions.  A start condition is activated using
the @code{BEGIN} action.  Until the next @code{BEGIN} action is
executed, rules with the given start condition will be active
and rules with other start conditions will be inactive.
If the start condition is @emph{inclusive}, then rules with no
start conditions at all will also be active.  If it is
@emph{exclusive}, then @emph{only} rules qualified with the start
condition will be active.  A set of rules contingent on the
same exclusive start condition describe a scanner which is
independent of any of the other rules in the @code{flex} input.
Because of this, exclusive start conditions make it easy
to specify "mini-scanners" which scan portions of the
input that are syntactically different from the rest
(e.g., comments).

If the distinction between inclusive and exclusive start
conditions is still a little vague, here's a simple
example illustrating the connection between the two.  The set
of rules:

@example
%s example
%%

<example>foo   do_something();

bar            something_else();
@end example

@noindent
is equivalent to

@example
%x example
%%