flex.texinfo

早期freebsd实现

\input texinfo       @c                    -*- Texinfo -*-
@setfilename flex.info

@ifinfo
@format
START-INFO-DIR-ENTRY

* flex: (flex).	
		Fast lexical analyzer generator.

END-INFO-DIR-ENTRY
@end format
@end ifinfo

@synindex ky cp
@c
@c This file documents ``flex'', the fast lexical analyzer generator.
@c
@c This text is based on the file ``flexdoc.1'' in the UCB flex
@c distribution.  The text may be freely distributed under the terms of
@c the UCB license; see file COPYING.
@c
@c $Id: flex.texinfo,v 1.5 1992/05/29 20:40:09 pesch Exp $

@setchapternewpage odd
@settitle Using @code{flex}
@titlepage
@finalout
@c @smallbook
@c @cropmarks
@title Using @code{flex}
@subtitle A Fast Lexical Analyzer Generator
@sp 1
@subtitle 26 May 1990---Version 2.3
@sp 4
@quotation
This product includes software developed by the University of
California, Berkeley and its contributors.
@end quotation
@author Vern Paxson
@page

@tex
\def\$#1${{#1}} % Kluge: collect RCS revision info without $...$
\xdef\manvers{\$Revision: 1.5 $} % For use in headers, footers too
{\parskip=0pt \hfill {\it Using flex} \par \hfill \manvers\par \hfill
\TeX{}info \texinfoversion\par
\hfill Original text: {\tt vern@@ee.lbl.gov}\par 
\hfill Texinfo conversion:{\tt pesch@@cygnus.com}\par}
@end tex

@vskip 0pt plus 1filll
Copyright (c) 1990 The Regents of the University of California.
All rights reserved.

This code is derived from software contributed to Berkeley by
Vern Paxson.

The United States Government has rights in this work pursuant
to contract no. DE-AC03-76SF00098 between the United States
Department of Energy and the University of California.

Redistribution and use in source and binary forms are permitted
provided that: (1) source distributions retain this entire
copyright notice and comment, and (2) distributions including
binaries display the following acknowledgement:  ``This product
includes software developed by the University of California,
Berkeley and its contributors'' in the documentation or other
materials provided with the distribution and in all advertising
materials mentioning features or use of this software.  Neither the
name of the University nor the names of its contributors may be
used to endorse or promote products derived from this software
without specific prior written permission.


THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE.
@end titlepage

@ifinfo
@node Top
@top FLEX---fast lexical analyzer generator

@quotation
This product includes software developed by the University of
California, Berkeley and its contributors.
@end quotation

@menu
* Introduction::        An Overview of @code{flex}, with Examples
* Files::               Input and Output Files
* Invoking::            Command-line Options
* Performance::         Performance Considerations
* Incompatibilities::   Incompatibilities with @code{lex} and @sc{posix}
* Diagnostics::         Diagnostic Messages
* Bugs::                Deficiencies and Bugs
* Acknowledgements::    Contributors to flex
@end menu
@end ifinfo

@node Introduction
@chapter An Overview of @code{flex}, with Examples

@code{flex} is a tool for generating scanners: programs which recognize
lexical patterns in text.  @code{flex} reads the given input files (or
its standard input if no file names are given) for a description of the
scanner to generate.  The description is in the form of pairs of regular
expressions and C code, called @dfn{rules}.  @code{flex} generates as
output a C source file, @file{lex.yy.c}, which defines a routine
@code{yylex}. Compile and link this file with the @samp{-lfl} library to
produce an executable.  When the executable runs, it analyzes its input
for occurrences of the regular expressions.  Whenever it finds one, it
executes the corresponding C code.

Some simple examples follow, to give you the flavor of using
@code{flex}.

@menu
* Text-Substitution::   Trivial Text-Substitution
* Counter::             Count Lines and Characters
* Toy::                 Simplified Pascal-like Language
@end menu

@node Text-Substitution
@section Text-Substitution Scanner

The following @code{flex} input specifies a scanner which,
whenever it encounters the string @samp{username}, will replace it
with the user's login name:

@example
%%
username    printf( "%s", getlogin() );
@end example

By default, any text not matched by a @code{flex} scanner is copied to
the output, so the net effect of this scanner is to copy its input file
to its output with each occurrence of @samp{username} expanded.  In this
input, there is just one rule.  @samp{username} is the pattern and the
@code{printf} is the action.  The @samp{%%} marks the beginning of the
rules.

@node Counter
@section A Scanner to Count Lines and Characters

Here's another simple example:

@example
    int num_lines = 0, num_chars = 0;

%%
\n    ++num_lines; ++num_chars;
.     ++num_chars;

%%
main()
    @{
    yylex();
    printf( "# of lines = %d, # of chars = %d\n",
            num_lines, num_chars );
    @}
@end example

This scanner counts the number of characters and the  number
of  lines in its input (it produces no output other than the
final report on the counts).  The first  line  declares  two
globals,  @code{num_lines}  and @code{num_chars}, which are accessible
both inside @code{yylex} and in the @code{main} routine declared after
the  second  @samp{%%}.  There are two rules, one which matches a
newline (@samp{\n}) and increments both the line  count  and  the
character  count,  and one which matches any character other
than a newline (indicated by the @samp{.} regular expression).

@node Toy
@section Simplified Pascal-like Language Scanner

A somewhat more complicated example:

@smallexample
/* scanner for a toy Pascal-like language */

%@{
/* need this for the call to atof() below */
#include <math.h>
%@}

DIGIT    [0-9]
ID       [a-z][a-z0-9]*

%%

@{DIGIT@}+    @{
            printf( "An integer: %s (%d)\n", yytext,
                    atoi( yytext ) );
            @}

@{DIGIT@}+"."@{DIGIT@}*        @{
            printf( "A float: %s (%g)\n", yytext,
                    atof( yytext ) );
            @}

if|then|begin|end|procedure|function        @{
            printf( "A keyword: %s\n", yytext );
            @}

@{ID@}        printf( "An identifier: %s\n", yytext );

"+"|"-"|"*"|"/"   printf( "An operator: %s\n", yytext );

"@{"[^@}\n]*"@}"     /* eat up one-line comments */

[ \t\n]+          /* eat up whitespace */

.           printf( "Unrecognized character: %s\n", yytext );

%%

main( argc, argv )
int argc;
char **argv;
    @{
    ++argv, --argc;  /* skip over program name */
    if ( argc > 0 )
            yyin = fopen( argv[0], "r" );
    else
            yyin = stdin;

    yylex();
    @}
@end smallexample

This is the beginnings of a simple scanner for a language like Pascal.
It identifies different types of tokens and reports on what it has seen.

The details of this example are explained in the following chapters.

@node Files
@chapter Input and Output Files

@code{flex}'s actions are specified by definitions (which may include
embedded C code) in one or more input files.  The primary
output file is @file{lex.yy.c}.  You can also use some of the
command-line options to get diagnostic output
(@pxref{Invoking,,Command-line options}).
This chapter gives the details of how to structure your input to
define the scanner you need.

@menu
* Input Format::        Format of the Input File
* Scanner::             The Generated Scanner
* Start::               Start Conditions
* Multiple Input::      Multiple Input Buffers
* EOF::                 End-of-File Rules
* Misc::                Miscellaneous Macros
* Parsers::             Interfacing with Parser Generators
* Translation::         Translation Table
@end menu

@node Input Format
@section Format of the Input File

The @code{flex} input file consists of three sections, separated by a
line with just @samp{%%} in it:

@example
@var{definitions}
%%
@var{rules}
%%
@var{user code}
@end example

The @var{definitions} section contains declarations of simple name
definitions to simplify the scanner specification, and declarations of
start conditions, which are explained in a later section.

@noindent
Name definitions have the form:

@example
@var{name} @var{definition}
@end example

The @var{name} is a word beginning with a letter or an underscore
(@samp{_}) followed by zero or more letters, digits, @samp{_}, or
@samp{-} (dash).  The definition is taken to begin at the first
non-whitespace character following the name, and continuing to the end
of the line.  The definition can subsequently be referred to using
@samp{@{@var{name}@}}, which will expand to @samp{(@var{definition})}.
For example,

@example
DIGIT    [0-9]
ID       [a-z][a-z0-9]*
@end example

defines @samp{DIGIT} to be a regular expression which matches a single
digit, and @samp{ID} to be a regular expression which matches a letter
followed by zero or more letters or digits.  A subsequent reference to

@example
@{DIGIT@}+"."@{DIGIT@}*
@end example

@noindent
is identical to

@example
([0-9])+"."([0-9])*
@end example

@noindent
and matches one or more digits followed by a @samp{.} followed by zero
or more digits.

The rules section of the @code{flex} input contains a series of rules of
the form:

@example
@var{pattern}   @var{action}
@end example

@noindent
where the @var{pattern} must be unindented and the @var{action} must
begin on the same line.

See below for a further description of patterns and actions.

Finally, the user code section is simply copied to @file{lex.yy.c}
verbatim.  It is used for companion routines which call or are called by
the scanner.  The presence of this section is optional; if it is
missing, the second @samp{%%} in the input file may be skipped, too.

In the definitions and rules sections, any indented text or text
enclosed in @samp{%@{} and @samp{%@}} is copied verbatim to the output
(with the @samp{%@{@}} removed).  The @samp{%@{@}} must appear
unindented on lines by themselves.

In the rules section, any indented or @samp{%@{@}} text appearing before
the first rule may be used to declare variables which are local to the
scanning routine and (after the declarations) code which is to be
executed whenever the scanning routine is entered.  Other indented or
@samp{%@{@}} text in the rule section is still copied to the output, but
its meaning is not well defined and it may well cause compile-time
errors (this feature is present for @sc{posix} compliance; see below for
other such features).

In the definitions section, an unindented comment (i.e., a line
beginning with @samp{/*}) is also copied verbatim to the output up to
the next @samp{*/}.  Also, any line in the definitions section beginning
with @samp{#} is ignored, though this style of comment is deprecated and
may go away in the future.

@menu
* Patterns::    Patterns in the input
* Matching::    How the input is matched
* Actions::     Actions
@end menu

@node Patterns
@subsection Patterns in the Input

The patterns in the input are written using an extended set of regular
expressions.  These are:

@table @code
@item @var{x}
match the character @samp{@var{x}}

@item .
any character except newline

@item [xyz]
a ``character class''; in this case, the pattern matches either an
@samp{x}, a @samp{y}, or a @samp{z}

@item [abj-oZ]
a ``character class'' with a range in it; matches an @samp{a}, a
@samp{b}, any letter from @samp{j} through @samp{o}, or a @samp{Z}

@item [^A-Z]
a ``negated character class'', i.e., any character but those in the
class.  In this case, any character @emph{except} an uppercase letter.

@item [^A-Z\n]
any character @emph{except} an uppercase letter or a newline