flexdoc.1

早期freebsd实现

.B lex.foo.c.
Here are all of the names affected:
.nf

    yyFlexLexer
    yy_create_buffer
    yy_delete_buffer
    yy_flex_debug
    yy_init_buffer
    yy_load_buffer_state
    yy_switch_to_buffer
    yyin
    yyleng
    yylex
    yyout
    yyrestart
    yytext
    yywrap

.fi
Within your scanner itself, you can still refer to the global variables
and functions using either version of their name; but eternally, they
have the modified name.
.IP
This option lets you easily link together multiple
.I lex
programs into the same executable.  Note, though, that using this
option also renames
.B yywrap(),
so you now
.I must
provide your own (appropriately-named) version of the routine for your
scanner, as linking with
.B \-ll
no longer provides one for you by default.
.TP
.B \-Sskeleton_file
overrides the default skeleton file from which
.I lex
constructs its scanners.  You'll never need this option unless you are doing
.I lex
maintenance or development.
.SH PERFORMANCE CONSIDERATIONS
The main design goal of
.I lex
is that it generate high-performance scanners.  It has been optimized
for dealing well with large sets of rules.  Aside from the effects on
scanner speed of the table compression
.B \-C
options outlined above,
there are a number of options/actions which degrade performance.  These
are, from most expensive to least:
.nf

    REJECT

    pattern sets that require backing up
    arbitrary trailing context

    yymore()
    '^' beginning-of-line operator

.fi
with the first three all being quite expensive and the last two
being quite cheap.  Note also that
.B unput()
is implemented as a routine call that potentially does quite a bit of
work, while
.B yyless()
is a quite-cheap macro; so if just putting back some excess text you
scanned, use
.B yyless().
.PP
.B REJECT
should be avoided at all costs when performance is important.
It is a particularly expensive option.
.PP
Getting rid of backing up is messy and often may be an enormous
amount of work for a complicated scanner.  In principal, one begins
by using the
.B \-b 
flag to generate a
.I lex.backup
file.  For example, on the input
.nf

    %%
    foo        return TOK_KEYWORD;
    foobar     return TOK_KEYWORD;

.fi
the file looks like:
.nf

    State #6 is non-accepting -
     associated rule line numbers:
           2       3
     out-transitions: [ o ]
     jam-transitions: EOF [ \\001-n  p-\\177 ]

    State #8 is non-accepting -
     associated rule line numbers:
           3
     out-transitions: [ a ]
     jam-transitions: EOF [ \\001-`  b-\\177 ]

    State #9 is non-accepting -
     associated rule line numbers:
           3
     out-transitions: [ r ]
     jam-transitions: EOF [ \\001-q  s-\\177 ]

    Compressed tables always back up.

.fi
The first few lines tell us that there's a scanner state in
which it can make a transition on an 'o' but not on any other
character, and that in that state the currently scanned text does not match
any rule.  The state occurs when trying to match the rules found
at lines 2 and 3 in the input file.
If the scanner is in that state and then reads
something other than an 'o', it will have to back up to find
a rule which is matched.  With
a bit of headscratching one can see that this must be the
state it's in when it has seen "fo".  When this has happened,
if anything other than another 'o' is seen, the scanner will
have to back up to simply match the 'f' (by the default rule).
.PP
The comment regarding State #8 indicates there's a problem
when "foob" has been scanned.  Indeed, on any character other
than an 'a', the scanner will have to back up to accept "foo".
Similarly, the comment for State #9 concerns when "fooba" has
been scanned and an 'r' does not follow.
.PP
The final comment reminds us that there's no point going to
all the trouble of removing backing up from the rules unless
we're using
.B \-Cf
or
.B \-CF,
since there's no performance gain doing so with compressed scanners.
.PP
The way to remove the backing up is to add "error" rules:
.nf

    %%
    foo         return TOK_KEYWORD;
    foobar      return TOK_KEYWORD;

    fooba       |
    foob        |
    fo          {
                /* false alarm, not really a keyword */
                return TOK_ID;
                }

.fi
.PP
Eliminating backing up among a list of keywords can also be
done using a "catch-all" rule:
.nf

    %%
    foo         return TOK_KEYWORD;
    foobar      return TOK_KEYWORD;

    [a-z]+      return TOK_ID;

.fi
This is usually the best solution when appropriate.
.PP
Backing up messages tend to cascade.
With a complicated set of rules it's not uncommon to get hundreds
of messages.  If one can decipher them, though, it often
only takes a dozen or so rules to eliminate the backing up (though
it's easy to make a mistake and have an error rule accidentally match
a valid token.  A possible future
.I lex
feature will be to automatically add rules to eliminate backing up).
.PP
.I Variable
trailing context (where both the leading and trailing parts do not have
a fixed length) entails almost the same performance loss as
.B REJECT
(i.e., substantial).  So when possible a rule like:
.nf

    %%
    mouse|rat/(cat|dog)   run();

.fi
is better written:
.nf

    %%
    mouse/cat|dog         run();
    rat/cat|dog           run();

.fi
or as
.nf

    %%
    mouse|rat/cat         run();
    mouse|rat/dog         run();

.fi
Note that here the special '|' action does
.I not
provide any savings, and can even make things worse (see
.PP
A final note regarding performance: as mentioned above in the section
How the Input is Matched, dynamically resizing
.B yytext
to accomodate huge tokens is a slow process because it presently requires that
the (huge) token be rescanned from the beginning.  Thus if performance is
vital, you should attempt to match "large" quantities of text but not
"huge" quantities, where the cutoff between the two is at about 8K
characters/token.
.PP
Another area where the user can increase a scanner's performance
(and one that's easier to implement) arises from the fact that
the longer the tokens matched, the faster the scanner will run.
This is because with long tokens the processing of most input
characters takes place in the (short) inner scanning loop, and
does not often have to go through the additional work of setting up
the scanning environment (e.g.,
.B yytext)
for the action.  Recall the scanner for C comments:
.nf

    %x comment
    %%
            int line_num = 1;

    "/*"         BEGIN(comment);

    <comment>[^*\\n]*
    <comment>"*"+[^*/\\n]*
    <comment>\\n             ++line_num;
    <comment>"*"+"/"        BEGIN(INITIAL);

.fi
This could be sped up by writing it as:
.nf

    %x comment
    %%
            int line_num = 1;

    "/*"         BEGIN(comment);

    <comment>[^*\\n]*
    <comment>[^*\\n]*\\n      ++line_num;
    <comment>"*"+[^*/\\n]*
    <comment>"*"+[^*/\\n]*\\n ++line_num;
    <comment>"*"+"/"        BEGIN(INITIAL);

.fi
Now instead of each newline requiring the processing of another
action, recognizing the newlines is "distributed" over the other rules
to keep the matched text as long as possible.  Note that
.I adding
rules does
.I not
slow down the scanner!  The speed of the scanner is independent
of the number of rules or (modulo the considerations given at the
beginning of this section) how complicated the rules are with
regard to operators such as '*' and '|'.
.PP
A final example in speeding up a scanner: suppose you want to scan
through a file containing identifiers and keywords, one per line
and with no other extraneous characters, and recognize all the
keywords.  A natural first approach is:
.nf

    %%
    asm      |
    auto     |
    break    |
    ... etc ...
    volatile |
    while    /* it's a keyword */

    .|\\n     /* it's not a keyword */

.fi
To eliminate the back-tracking, introduce a catch-all rule:
.nf

    %%
    asm      |
    auto     |
    break    |
    ... etc ...
    volatile |
    while    /* it's a keyword */

    [a-z]+   |
    .|\\n     /* it's not a keyword */

.fi
Now, if it's guaranteed that there's exactly one word per line,
then we can reduce the total number of matches by a half by
merging in the recognition of newlines with that of the other
tokens:
.nf

    %%
    asm\\n    |
    auto\\n   |
    break\\n  |
    ... etc ...
    volatile\\n |
    while\\n  /* it's a keyword */

    [a-z]+\\n |
    .|\\n     /* it's not a keyword */

.fi
One has to be careful here, as we have now reintroduced backing up
into the scanner.  In particular, while
.I we
know that there will never be any characters in the input stream
other than letters or newlines,
.I lex
can't figure this out, and it will plan for possibly needing to back up
when it has scanned a token like "auto" and then the next character
is something other than a newline or a letter.  Previously it would
then just match the "auto" rule and be done, but now it has no "auto"
rule, only a "auto\\n" rule.  To eliminate the possibility of backing up,
we could either duplicate all rules but without final newlines, or,
since we never expect to encounter such an input and therefore don't
how it's classified, we can introduce one more catch-all rule, this
one which doesn't include a newline:
.nf

    %%
    asm\\n    |
    auto\\n   |
    break\\n  |
    ... etc ...
    volatile\\n |
    while\\n  /* it's a keyword */

    [a-z]+\\n |
    [a-z]+   |
    .|\\n     /* it's not a keyword */

.fi
Compiled with
.B \-Cf,
this is about as fast as one can get a
.I lex 
scanner to go for this particular problem.
.PP
A final note:
.I lex
is slow when matching NUL's, particularly when a token contains
multiple NUL's.
It's best to write rules which match
.I short
amounts of text if it's anticipated that the text will often include NUL's.
.SH INCOMPATIBILITIES WITH AT&T LEX AND POSIX
BSD
.I lex
is a rewrite of the AT&T Unix
.I lex
tool (the two implementations do not share any code, though),
with some extensions and incompatibilities, both of which
are of concern to those who wish to write scanners acceptable
to either implementation.  The POSIX
.I lex
specification is closer to BSD
.I lex's
behavior than that of the original AT&T
.I lex
implementation, but there also remain some incompatibilities between BSD
.I lex
and POSIX.  The intent is that ultimately BSD
.I lex
will be fully POSIX-conformant.  In this section we discuss all of
the known areas of incompatibility.
.PP
BSD
.I lex's
.B \-l
option turns on maximum compatibility with the original AT&T
.I lex
implementation, at the cost of a major loss in the generated scanner's
performance.  We note below which incompatibilities can be overcome
using the
.B \-l
option.
.PP
BSD
.I lex
is fully compatible with AT&T
.I lex
with the following exceptions:
.IP -
The undocumented AT&T
.I lex
scanner internal variable
.B yylineno
is not supported unless
.B \-l
is used.
.IP
yylineno is not part of the POSIX specification.
.IP -
The
.B input()
routine is not redefinable, though it may be called to read characters
following whatever has been matched by a rule.  If
.B input()
encounters an end-of-file the normal
.B yywrap()
processing is done.  A ``real'' end-of-file is returned by
.B input()
as
.I EOF.
.IP
Input is instead controlled by defining the
.B YY_INPUT
macro.
.IP
The BSD
.I lex
restriction that
.B input()
cannot be redefined is in accordance with the POSIX specification,
which simply does not specify any way of controlling the
scanner's input other than by making an initial assignment to
.I yyin.
.IP -
BSD
.I lex
scanners are not as reentrant as AT&T
.I lex
scanners.  In particular, if you have an interactive scanner and
an interrupt handler which long-jumps out of the scanner, and
the scanner is subsequently called again, you may get the following
message:
.nf

    fatal lex scanner internal error--end of buffer missed

.fi
To reenter the scanner, first use
.nf

    yyrestart( yyin );

.fi
Note that this call will throw away any buffered input; usually this
isn't a problem with an interactive scanner.
.IP -
.B output()
is not supported.
Output from the
.B ECHO
macro is done to the file-pointer
.I yyout
(default
.I stdout).
.IP
.B output()
is not part of the POSIX specification.
.IP -
AT&T
.I lex
does not support exclusive start conditions (%x), though they
are in the POSIX specification.
.IP -
When definitions are expanded, BSD
.I lex
encloses them in parentheses.
With AT&T lex, the following:
.nf

    NAME    [A-Z][A-Z0-9]*
    %%
    foo{NAME}?      printf( "Found it\\n" );
    %%

.fi
will not match the string "foo" because when the macro
is expanded the rule is equivalent to "foo[A-Z][A-Z0-9]*?"
and the precedence is such that the '?' is associated with
"[A-Z0-9]*".  With BSD
.I lex,
the rule will be expanded to
"foo([A-Z][A-Z0-9]*)?" and so the string "foo" will match.
.IP
N