flex.1

早期freebsd实现

or the '$' operator).  The start condition, '^', and "<<EOF>>" patterns
can only occur at the beginning of a pattern, and, as well as with '/' and '$',
cannot be grouped inside parentheses.  The following are all illegal:
.nf

    foo/bar$
    foo|(bar$)
    foo|^bar
    <sc1>foo<sc2>bar

.fi
.SH SUMMARY OF SPECIAL ACTIONS
In addition to arbitrary C code, the following can appear in actions:
.IP -
.B ECHO
copies yytext to the scanner's output.
.IP -
.B BEGIN
followed by the name of a start condition places the scanner in the
corresponding start condition.
.IP -
.B REJECT
directs the scanner to proceed on to the "second best" rule which matched the
input (or a prefix of the input).
.B yytext
and
.B yyleng
are set up appropriately.  Note that
.B REJECT
is a particularly expensive feature in terms scanner performance;
if it is used in
.I any
of the scanner's actions it will slow down
.I all
of the scanner's matching.  Furthermore,
.B REJECT
cannot be used with the
.B \-f
or
.B \-F
options.
.IP
Note also that unlike the other special actions,
.B REJECT
is a
.I branch;
code immediately following it in the action will
.I not
be executed.
.IP -
.B yymore()
tells the scanner that the next time it matches a rule, the corresponding
token should be
.I appended
onto the current value of
.B yytext
rather than replacing it.
.IP -
.B yyless(n)
returns all but the first
.I n
characters of the current token back to the input stream, where they
will be rescanned when the scanner looks for the next match.
.B yytext
and
.B yyleng
are adjusted appropriately (e.g.,
.B yyleng
will now be equal to
.I n
).
.IP -
.B unput(c)
puts the character
.I c
back onto the input stream.  It will be the next character scanned.
.IP -
.B input()
reads the next character from the input stream (this routine is called
.B yyinput()
if the scanner is compiled using
.B C++).
.IP -
.B yyterminate()
can be used in lieu of a return statement in an action.  It terminates
the scanner and returns a 0 to the scanner's caller, indicating "all done".
.IP
By default,
.B yyterminate()
is also called when an end-of-file is encountered.  It is a macro and
may be redefined.
.IP -
.B YY_NEW_FILE
is an action available only in <<EOF>> rules.  It means "Okay, I've
set up a new input file, continue scanning".  It is no longer required;
you can just assign
.I yyin
to point to a new file in the <<EOF>> action.
.IP -
.B yy_create_buffer( file, size )
takes a
.I FILE
pointer and an integer
.I size.
It returns a YY_BUFFER_STATE
handle to a new input buffer large enough to accomodate
.I size
characters and associated with the given file.  When in doubt, use
.B YY_BUF_SIZE
for the size.
.IP -
.B yy_switch_to_buffer( new_buffer )
switches the scanner's processing to scan for tokens from
the given buffer, which must be a YY_BUFFER_STATE.
.IP -
.B yy_delete_buffer( buffer )
deletes the given buffer.
.SH VALUES AVAILABLE TO THE USER
.IP -
.B char *yytext
holds the text of the current token.  It may be modified but not lengthened
(you cannot append characters to the end).  Modifying the last character
may affect the activity of rules anchored using '^' during the next scan;
see
.B lexdoc(1)
for details.
.IP
If the special directive
.B %array
appears in the first section of the scanner description, then
.B yytext
is instead declared
.B char yytext[YYLMAX],
where
.B YYLMAX
is a macro definition that you can redefine in the first section
if you don't like the default value (generally 8KB).  Using
.B %array
results in somewhat slower scanners, but the value of
.B yytext
becomes immune to calls to
.I input()
and
.I unput(),
which potentially destroy its value when
.B yytext
is a character pointer.  The opposite of
.B %array
is
.B %pointer,
which is the default.
.IP -
.B int yyleng
holds the length of the current token.
.IP -
.B FILE *yyin
is the file which by default
.I lex
reads from.  It may be redefined but doing so only makes sense before
scanning begins or after an EOF has been encountered.  Changing it in
the midst of scanning will have unexpected results since
.I lex
buffers its input; use
.B yyrestart()
instead.
Once scanning terminates because an end-of-file
has been seen,
.B
you can assign
.I yyin
at the new input file and then call the scanner again to continue scanning.
.IP -
.B void yyrestart( FILE *new_file )
may be called to point
.I yyin
at the new input file.  The switch-over to the new file is immediate
(any previously buffered-up input is lost).  Note that calling
.B yyrestart()
with
.I yyin
as an argument thus throws away the current input buffer and continues
scanning the same input file.
.IP -
.B FILE *yyout
is the file to which
.B ECHO
actions are done.  It can be reassigned by the user.
.IP -
.B YY_CURRENT_BUFFER
returns a
.B YY_BUFFER_STATE
handle to the current buffer.
.IP -
.B YY_START
returns an integer value corresponding to the current start
condition.  You can subsequently use this value with
.B BEGIN
to return to that start condition.
.SH MACROS AND FUNCTIONS YOU CAN REDEFINE
.IP -
.B YY_DECL
controls how the scanning routine is declared.
By default, it is "int yylex()", or, if prototypes are being
used, "int yylex(void)".  This definition may be changed by redefining
the "YY_DECL" macro.  Note that
if you give arguments to the scanning routine using a
K&R-style/non-prototyped function declaration, you must terminate
the definition with a semi-colon (;).
.IP -
The nature of how the scanner
gets its input can be controlled by redefining the
.B YY_INPUT
macro.
YY_INPUT's calling sequence is "YY_INPUT(buf,result,max_size)".  Its
action is to place up to
.I max_size
characters in the character array
.I buf
and return in the integer variable
.I result
either the
number of characters read or the constant YY_NULL (0 on Unix systems)
to indicate EOF.  The default YY_INPUT reads from the
global file-pointer "yyin".
A sample redefinition of YY_INPUT (in the definitions
section of the input file):
.nf

    %{
    #undef YY_INPUT
    #define YY_INPUT(buf,result,max_size) \\
        { \\
        int c = getchar(); \\
        result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \\
        }
    %}

.fi
.IP -
When the scanner receives an end-of-file indication from YY_INPUT,
it then checks the function
.B yywrap()
function.  If
.B yywrap()
returns false (zero), then it is assumed that the
function has gone ahead and set up
.I yyin
to point to another input file, and scanning continues.  If it returns
true (non-zero), then the scanner terminates, returning 0 to its
caller.
.IP
The default
.B yywrap()
always returns 1.
.IP -
YY_USER_ACTION
can be redefined to provide an action
which is always executed prior to the matched rule's action.
.IP -
The macro
.B YY_USER_INIT
may be redefined to provide an action which is always executed before
the first scan.
.IP -
In the generated scanner, the actions are all gathered in one large
switch statement and separated using
.B YY_BREAK,
which may be redefined.  By default, it is simply a "break", to separate
each rule's action from the following rule's.
.SH FILES
.TP
.B \-ll
library with which scanners may be linked to obtain default versions
of
.I yywrap()
and
.I main().
.TP
.I lex.yy.c
generated scanner (called
.I lexyy.c
on some systems).
.TP
.I lex.backup
backing-up information for
.B \-b
flag (called
.I lex.bck
on some systems).
.SH "SEE ALSO"
.PP
lexdoc(1), yacc(1), sed(1), awk(1).
.PP
M. E. Lesk and E. Schmidt,
.I LEX \- Lexical Analyzer Generator
.SH DIAGNOSTICS
.PP
.I reject_used_but_not_detected undefined
or
.PP
.I yymore_used_but_not_detected undefined -
These errors can occur at compile time.  They indicate that the
scanner uses
.B REJECT
or
.B yymore()
but that
.I lex
failed to notice the fact, meaning that
.I lex
scanned the first two sections looking for occurrences of these actions
and failed to find any, but somehow you snuck some in (via a #include
file, for example).  Make an explicit reference to the action in your
.I lex
input file.  (Note that previously
.I lex
supported a
.B %used/%unused
mechanism for dealing with this problem; this feature is still supported
but now deprecated, and will go away soon unless the author hears from
people who can argue compellingly that they need it.)
.PP
.I lex scanner jammed -
a scanner compiled with
.B \-s
has encountered an input string which wasn't matched by
any of its rules.
.PP
.I warning, rule cannot be matched
indicates that the given rule
cannot be matched because it follows other rules that will
always match the same text as it.  See
.I lexdoc(1)
for an example.
.PP
.I warning,
.B \-s
.I
option given but default rule can be matched
means that it is possible (perhaps only in a particular start condition)
that the default rule (match any single character) is the only one
that will match a particular input.  Since
.PP
.I scanner input buffer overflowed -
a scanner rule matched more text than the available dynamic memory.
.PP
.I token too large, exceeds YYLMAX -
your scanner uses
.B %array
and one of its rules matched a string longer than the
.B YYLMAX
constant (8K bytes by default).  You can increase the value by
#define'ing
.B YYLMAX
in the definitions section of your
.I lex
input.
.PP
.I scanner requires \-8 flag to
.I use the character 'x' -
Your scanner specification includes recognizing the 8-bit character
.I 'x'
and you did not specify the \-8 flag, and your scanner defaulted to 7-bit
because you used the
.B \-Cf
or
.B \-CF
table compression options.
.PP
.I lex scanner push-back overflow -
you used
.B unput()
to push back so much text that the scanner's buffer could not hold
both the pushed-back text and the current token in
.B yytext.
Ideally the scanner should dynamically resize the buffer in this case, but at
present it does not.
.PP
.I
input buffer overflow, can't enlarge buffer because scanner uses REJECT -
the scanner was working on matching an extremely large token and needed
to expand the input buffer.  This doesn't work with scanners that use
.B
REJECT.
.PP
.I
fatal lex scanner internal error--end of buffer missed -
This can occur in an scanner which is reentered after a long-jump
has jumped out (or over) the scanner's activation frame.  Before
reentering the scanner, use:
.nf

    yyrestart( yyin );

.fi
.SH AUTHOR
Vern Paxson, with the help of many ideas and much inspiration from
Van Jacobson.  Original version by Jef Poskanzer.
.PP
See lexdoc(1) for additional credits and the address to send comments to.
.SH DEFICIENCIES / BUGS
.PP
Some trailing context
patterns cannot be properly matched and generate
warning messages ("dangerous trailing context").  These are
patterns where the ending of the
first part of the rule matches the beginning of the second
part, such as "zx*/xy*", where the 'x*' matches the 'x' at
the beginning of the trailing context.  (Note that the POSIX draft
states that the text matched by such patterns is undefined.)
.PP
For some trailing context rules, parts which are actually fixed-length are
not recognized as such, leading to the abovementioned performance loss.
In particular, parts using '|' or {n} (such as "foo{3}") are always
considered variable-length.
.PP
Combining trailing context with the special '|' action can result in
.I fixed
trailing context being turned into the more expensive
.I variable
trailing context.  For example, in the following:
.nf

    %%
    abc      |
    xyz/def

.fi
.PP
Use of
.B unput()
or
.B input()
invalidates yytext and yyleng, unless the
.B %array
directive
or the
.B \-l
option has been used.
.PP
Use of unput() to push back more text than was matched can
result in the pushed-back text matching a beginning-of-line ('^')
rule even though it didn't come at the beginning of the line
(though this is rare!).
.PP
Pattern-matching of NUL's is substantially slower than matching other
characters.
.PP
Dynamic resizing of the input buffer is slow, as it entails rescanning
all the text matched so far by the current (generally huge) token.
.PP
.I lex
does not generate correct #line directives for code internal
to the scanner; thus, bugs in its skeleton file
yield bogus line numbers.
.PP
Due to both buffering of input and read-ahead, you cannot intermix
calls to <stdio.h> routines, such as, for example,
.B getchar(),
with
.I lex
rules and expect it to work.  Call
.B input()
instead.
.PP
The total table entries listed by the
.B \-v
flag excludes the number of table entries needed to determine
what rule has been matched.  The number of entries is equal
to the number of DFA states if the scanner does not use
.B REJECT,
and somewhat greater than the number of states if it does.
.PP
.B REJECT
cannot be used with the
.B \-f
or
.B \-F
options.
.PP
The
.I lex
internal algorithms need documentation.