re2c.1

来自「a little DFA compiler.」· 1 代码 · 共 598 行 · 第 1/2 页

./" 
./" $Id: re2c.1.in 663 2007-04-01 11:22:15Z helly $
./"
.TH RE2C 1 "22 April 2005" "Version 0.12.3"
.ds re \fBre2c\fP
.ds le \fBlex\fP
.ds rx regular expression
.ds lx \fIl\fP-expression
.SH NAME
re2c \- convert regular expressions to C/C++

.SH SYNOPSIS
\*(re [\fB-bdefghisuvVw1\fP] [\fB-o output\fP] file\fP

.SH DESCRIPTION
\*(re is a preprocessor that generates C-based recognizers from regular
expressions.
The input to \*(re consists of C/C++ source interleaved with
comments of the form \fC/*!re2c\fP ... \fC*/\fP which contain
scanner specifications.
In the output these comments are replaced with code that, when
executed, will find the next input token and then execute
some user-supplied token-specific code.

For example, given the following code

.in +3
.nf
char *scan(char *p)
{
/*!re2c
        re2c:define:YYCTYPE  = "unsigned char";
        re2c:define:YYCURSOR = p;
        re2c:yyfill:enable   = 0;
        re2c:yych:conversion = 1;
        re2c:indent:top      = 1;
        [0-9]+          {return p;}
        [\000-\377]     {return (char*)0;}
*/
}
.fi
.in -3

\*(re -is will generate

.in +3
.nf
/* Generated by re2c on Sat Apr 16 11:40:58 1994 */
char *scan(char *p)
{
    {
        unsigned char yych;

        yych = (unsigned char)*p;
        if(yych <= '/') goto yy4;
        if(yych >= ':') goto yy4;
        ++p;
        yych = (unsigned char)*p;
        goto yy7;
yy3:
        {return p;}
yy4:
        ++p;
        yych = (unsigned char)*p;
        {return char*)0;}
yy6:
        ++p;
        yych = (unsigned char)*p;
yy7:
        if(yych <= '/') goto yy3;
        if(yych <= '9') goto yy6;
        goto yy3;
    }

}
.fi
.in -3

You can place one \fC/*!max:re2c */\fP comment that will output a "#define 
\fCYYMAXFILL\fP <n>" line that holds the maximum number of characters 
required to parse the input. That is the maximum value \fCYYFILL\fP(n)
will receive. If -1 is in effect then YYMAXFILL can only be triggered once
after the last \fC/*!re2c */\fP.

You can also use \fC/*!ignore:re2c */\fP blocks that allows to document the
scanner code and will not be part of the output.

.SH OPTIONS
\*(re provides the following options:
.TP
\fB-?\fP
\fB-h\fP
Invoke a short help.
.TP
\fB-b\fP
Implies \fB-s\fP.  Use bit vectors as well in the attempt to coax better
code out of the compiler.  Most useful for specifications with more than a
few keywords (e.g. for most programming languages).
.TP
\fB-d\fP
Creates a parser that dumps information about the current position and in 
which state the parser is while parsing the input. This is useful to debug 
parser issues and states. If you use this switch you need to define a macro
\fIYYDEBUG\fP that is called like a function with two parameters:
\fIvoid YYDEBUG(int state, char current)\fP. The first parameter receives the 
state or -1 and the second parameter receives the input at the current cursor.
.TP
\fB-e\fP
Cross-compile from an ASCII platform to an EBCDIC one. 
.TP
\fB-f\fP
Generate a scanner with support for storable state.
For details see below at \fBSCANNER WITH STORABLE STATES\fP.
.TP
\fB-g\fP
Generate a scanner that utilizes GCC's computed goto feature. That is \*(re
generates jump tables whenever a decision is of a certain complexity (e.g. a 
lot of if conditions are otherwise necessary). This is only useable with GCC 
and produces output that cannot be compiled with any other compiler. Note that
this implies -b and that the complexity threshold can be configured using the
inplace configuration "cgoto:threshold".
.TP
\fB-i\fP
Do not output #line information. This is usefull when you want use a CMS tool
with the \*(re output which you might want if you do not require your users to 
have \*(re themselves when building from your source.
\fB-o output\fP
Specify the output file.
.TP
\fB-s\fP
Generate nested \fCif\fPs for some \fCswitch\fPes.  Many compilers need this
assist to generate better code.
.TP
\fB-u\fP
Generate a parser that supports Unicode chars (UTF-32). This means the 
generated code can deal with any valid Unicode character up to 0x10FFFF. When
UTF-8 or UTF-16 needs to be supported you need to convert the incoming stream
to UTF-32 upon input yourself.
.TP
\fB-v\fP
Show version information.
.TP
\fB-V\fP
Show the version as a number XXYYZZ.
.TP
\fB-w\fP
Create a parser that supports wide chars (UCS-2). This implies \fB-s\fP and 
cannot be used together with \fB-e\fP switch.
.TP
\fB-1\fP
Force single pass generation, this cannot be combined with -f and disables 
YYMAXFILL generation prior to last \*(re block.
.TP
\fb--no-generation-date\fP
Suppress date output in the generated output so that it only shows the re2c
version.
.SH "INTERFACE CODE"
Unlike other scanner generators, \*(re does not generate complete scanners:
the user must supply some interface code.
In particular, the user must define the following macros or use the 
corresponding inplace configurations:
.TP
\fCYYCTYPE\fP
Type used to hold an input symbol.
Usually \fCchar\fP or \fCunsigned char\fP.
.TP
\fCYYCURSOR\fP
\*(lx of type \fC*YYCTYPE\fP that points to the current input symbol.
The generated code advances \fCYYCURSOR\fP as symbols are matched.
On entry, \fCYYCURSOR\fP is assumed to point to the first character of the
current token.  On exit, \fCYYCURSOR\fP will point to the first character of
the following token.
.TP
\fCYYLIMIT\fP
Expression of type \fC*YYCTYPE\fP that marks the end of the buffer
(\fCYYLIMIT[-1]\fP is the last character in the buffer).
The generated code repeatedly compares \fCYYCURSOR\fP to \fCYYLIMIT\fP
to determine when the buffer needs (re)filling.
.TP
\fCYYMARKER\fP
\*(lx of type \fC*YYCTYPE\fP.
The generated code saves backtracking information in \fCYYMARKER\fP. Some easy
scanners might not use this.
.TP
\fCYYCTXMARKER\fP
\*(lx of type \fC*YYCTYPE\fP.
The generated code saves trailing context backtracking information in \fCYYCTXMARKER\fP.
The user only needs to define this macro if a scanner specification uses trailing
context in one or more of its regular expressions.
.TP
\fCYYFILL\fP(\fIn\fP\fC\fP)
The generated code "calls" \fCYYFILL\fP(n) when the buffer needs
(re)filling:  at least \fIn\fP additional characters should
be provided.  \fCYYFILL\fP(n) should adjust \fCYYCURSOR\fP, \fCYYLIMIT\fP,
\fCYYMARKER\fP and \fCYYCTXMARKER\fP as needed.  Note that for typical 
programming languages \fIn\fP will be the length of the longest keyword plus one.
The user can place a comment of the form \fC/*!max:re2c */\fP once to insert 
a \fCYYMAXFILL\fP(n) definition that is set to the maximum length value. If -1 
switch is used then \fCYYMAXFILL\fP can be triggered only once after the 
last \fC/*!re2c */\fP
block.
.TP
\fCYYGETSTATE\fP()
The user only needs to define this macro if the \fB-f\fP flag was specified.
In that case, the generated code "calls" \fCYYGETSTATE\fP() at the very beginning
of the scanner in order to obtain the saved state. \fCYYGETSTATE\fP() must return a signed
integer. The value must be either -1, indicating that the scanner is entered for the
first time, or a value previously saved by \fCYYSETSTATE\fP(s).  In the second case, the
scanner will resume operations right after where the last \fCYYFILL\fP(n) was called.
.TP
\fCYYSETSTATE(\fP\fIs\fP\fC)\fP
The user only needs to define this macro if the \fB-f\fP flag was specified.
In that case, the generated code "calls" \fCYYSETSTATE\fP just before calling
\fCYYFILL\fP(n).  The parameter to \fCYYSETSTATE\fP is a signed integer that uniquely
identifies the specific instance of \fCYYFILL\fP(n) that is about to be called.
Should the user wish to save the state of the scanner and have \fCYYFILL\fP(n) return
to the caller, all he has to do is store that unique identifer in a variable.
Later, when the scannered is called again, it will call \fCYYGETSTATE()\fP and
resume execution right where it left off. The generated code will contain 
both \fCYYSETSTATE\fP(s) and \fCYYGETSTATE\fP even if \fCYYFILL\fP(n) is being
disabled.
.TP
\fCYYDEBUG(\fP\fIstate\fP,\fIcurrent\fC)\fP
This is only needed if the \fB-d\fP flag was specified. It allows to easily debug
the generated parser by calling a user defined function for every state. The function
should have the following signature: \fIvoid YYDEBUG(int state, char current)\fP. 
The first parameter receives the state or -1 and the second parameter receives the 
input at the current cursor.
.TP
\fCYYMAXFILL
This will be automatically defined by \fC/*!max:re2c */\fP blocks as explained above.

.SH "SCANNER WITH STORABLE STATES"
When the \fB-f\fP flag is specified, \*(re generates a scanner that
can store its current state, return to the caller, and later resume
operations exactly where it left off.

The default operation of \*(re is a "pull" model, where the scanner asks
for extra input whenever it needs it. However, this mode of operation
assumes that the scanner is the "owner" the parsing loop, and that may
not always be convenient.

Typically, if there is a preprocessor ahead of the scanner in the stream,
or for that matter any other procedural source of data, the scanner cannot
"ask" for more data unless both scanner and source live in a separate threads.

The \fB-f\fP flag is useful for just this situation : it lets users design
scanners that work in a "push" model, i.e. where data is fed to the scanner
chunk by chunk. When the scanner runs out of data to consume, it just stores
its state, and return to the caller. When more input data is fed to the scanner,
it resumes operations exactly where it left off.

When using the -f option \*(re does not accept stdin because it has to do the 
full generation process twice which means it has to read the input twice. That
means \*(re would fail in case it cannot open the input twice or reading the
input for the first time influences the second read attempt.

Changes needed compared to the "pull" model.

1. User has to supply macros YYSETSTATE() and YYGETSTATE(state)

2. The \fB-f\fP option inhibits declaration of \fIyych\fP and
\fIyyaccept\fP. So the user has to declare these. Also the user has
to save and restore these. In the example \fIexamples/push.re\fP these
are declared as fields of the (C++) class of which the scanner is a
method, so they do not need to be saved/restored explicitly. For C
they could e.g. be made macros that select fields from a structure
passed in as parameter. Alternatively, they could be declared as local
variables, saved with YYFILL(n) when it decides to return and restored
at entry to the function. Also, it could be more efficient to save the
state from YYFILL(n) because YYSETSTATE(state) is called
unconditionally. YYFILL(n) however does not get \fIstate\fP as
parameter, so we would have to store state in a local variable by
YYSETSTATE(state).

3. Modify YYFILL(n) to return (from the function calling it) if more
input is needed.

4. Modify caller to recognise "more input is needed" and respond
appropriately.

5. The generated code will contain a switch block that is used to restores 
the last state by jumping behind the corrspoding YYFILL(n) call. This code is
automatically generated in the epilog of the first "\fC/*!re2c */\fP" block. 
It is possible to trigger generation of the YYGETSTATE() block earlier by 
placing a "\fC/*!getstate:re2c */\fP" comment. This is especially useful when
the scanner code should be wrapped inside a loop.

Please see examples/push.re for push-model scanner. The generated code can be
tweaked using inplace configurations "\fBstate:abort\fP" and "\fBstate:nextlabel\fP".

.SH "SCANNER SPECIFICATIONS"
Each scanner specification consists of a set of \fIrules\fP, \fInamed
definitions\fP and \fIconfigurations\fP.
.LP
\fIRules\fP consist of a regular expression along with a block of C/C++ code that
is to be executed when the associated \fIregular expression\fP is matched.
.P
.RS