elex.texi

用于词法分析的词法分析器

this grammar you need a scanner that matches the following symbols: `+',
`-', `/', `*', `mod', `(', `)', plus @var{variable} and @var{number}
(which are defined above as a regular expressions).  A @var{variable} is
any string of letters or underscores, and a @var{number} can be anything
from an integer to a floating point value with an exponent.

So, the first thing you need for the calculator is a scanner that can
recognize these symbols --- this is where @var{Elex} comes in.

@c ----------------------------------------------------------------------
@node       The Scanner Script, Testing the Script, The Calculator, An Example
@section    The Scanner Script
@cindex scanner script, example
@cindex example, scanner script
@cindex CalcScanner.elx

After much head-scratching and cursing over the flippant prose in the
manual, you finally produce an @var{Elex} scanner script that you think
will work.  The complete script is listed below (it's called
@file{demo/CalcScanner.elx} in the @var{Elex} distribution).

The comments (lines starting with `#') within the script give a brief
description of the meaning of each part.  @xref{Scanner Scripts} for the
details.

@example
# This is the start of the scanner definition.
scanner CalcScanner

@group
  # This section tells the code generator to declare these names to be
  # unique integer constants that will be used to represent each
  # symbol.

symbols

  SymNumber SymVariable SymMod SymPlus SymMinus SymStar SymSlash
  SymLBracket SymRBracket
@end group

@group
  # This section is used to define symbolic expressions that can be
  # used in later regular expressions.

defines
  number = "0-9+"
@end group

begin

  # This is the main section, composed of a number of productions that
  # each define a symbol.  Each production is of the form
  #
  #  on [@var{productionName}] "@var{regexp}"
  #  <
  #    @var{code to be executed on a match of regexp}
  #  >
  #
  # The productionName part is used to give a meaningful name to the
  # production.  If no name is given, the production's position in the
  # script is used as the name (the first production is numbered 1).

@group
  # Matches numbers like 1, -1.2, +1.1e10, -10e-2
  on Number "<number>(\.<number>)?([eE][\+\-]?<number>)?"
  <
    // SymNumber, like all the constants used in return statements
    // below, appears in the `symbols' section and is declared by the
    // code generator.

    return SymNumber;
  >
@end group

@group
  # Matches variable names beginning with a letter, followed by zero
  # or more letters or underscores.
  on Variable "[a-zA-Z][a-zA-Z_]*"
  <
    return SymVariable;
  >
@end group

@group
  # This matches `mod' as a keyword.  Note that while the Variable
  # production above also matches 'mod', this production has
  # precedence because it comes later in the script.
  on Mod "mod"
  <
    return SymMod;
  >
@end group

  on Plus "\+"
  <
    return SymPlus;
  >

  on Minus "\-"
  <
    return SymMinus;
  >

  on Star "\*"
  <
    return SymStar;
  >

  on Slash "/"
  <
    return SymSlash;
  >

  on LBracket "\("
  <
    return SymLBracket;
  >

  on RBracket "\)"
  <
    return SymRBracket;
  >

@group
  # This production causes the scanner to ignore whitespace.
  on WhiteSpace "[ \t\n]+"
  <
    return SymNULL;
  >
@end group

@group
  # The error production gets matched when the scanner reads something
  # that cannot be matched by any other production.  The default
  # behavior if this production does not exist is to raise `invalid
  # symbol' error.  This default behavior is overridden if this
  # production returns SymNULL, but will still happen if the
  # production returns SymERROR.
  on error
  <
    // This production has no effect, since returning SymERROR means
    // error is handed as usual.

    return SymERROR;
  >
@end group

end
@end example

@c ----------------------------------------------------------------------
@node       Testing the Script, Generating the Scanner, The Scanner Script, An Example
@section    Testing the Script
@cindex testing example
@cindex debugging example
@cindex example, debugging
@cindex example, debugging output
@cindex CalcScanner.elx, debugging

Now you've written a script it's a well-known rule that you and the
computer will have different ideas what the script actually does.
Typically you don't find this out until you compile the code and test it
in program, but @var{Elex} has a way of testing the regular expressions
before you generate any code.  You don't have to have written any code
at all to use this feature.

To test the @file{CalcScanner.elx} script, run the @var{Elex} front
end in debug mode like this:

@example
> elex -d CalcScanner.elx
@end example

If the script compiles OK, the @var{Elex} script debugger runs each
line of the standard input through the scanner, matching symbols and
printing them in this format:

@example
@var{row}:@var{column} @var{production_name} "@var{matched_text}"
@end example

@var{row} and @var{column} indicate where the symbol was found,
@var{production_name} is the name you gave the production and
@var{matched_text} is the actual text matched for the production.
Error symbols have a production name of `<error>'.

Below is the transcript of a session with the @var{Elex} debugger and
the @file{CalcScanner.elx} script (bold lines are those entered by the
user):

@cartouche
@example
@group
> @b{elex -d CalcScanner.elx} 
@b{1.2 3.14e-2 8+a_variable/(mod)}

0:0: Number: "1.2"
0:3: WhiteSpace: " "
0:4: Number: "3.14e-2"
0:11: WhiteSpace: " "
0:12: Number: "8"
0:13: Plus: "+"
0:14: Variable: "a_variable"
0:24: Slash: "/"
0:25: LBracket: "("
0:26: Mod: "mod"

@b{123.2e 1. !#$%^&}
0:29: RBracket: ")"
0:30: WhiteSpace: "
"
1:0: Number: "123.2"
1:5: Variable: "e"
1:6: WhiteSpace: " "
1:7: Number: "1"
1:8: <Error>: "."
1:9: WhiteSpace: " "
1:10: <Error>: "!#$%^&"
1:17: WhiteSpace: "
"
@end group
@end example
@end cartouche

You might notice that the `)' ending the first line of user input does
not get matched until after the second line has been entered.  This
due to the @var{Elex} scanner engine's backtracking design
(@pxref{Backtracking Scanners}).

@c ----------------------------------------------------------------------
@node       Generating the Scanner, Using the Scanner, Testing the Script, An Example
@section    Generating the Scanner
@cindex generating C++ code
@cindex CalcScanner.h, listing
@cindex ElexScanner.h
@cindex ElexScanner
@cindex CalcScanner, class
@cindex C++ code, generating with Elex
@cindex example, generating a C++ scanner
@cindex example, C++ scanner implementation

Now you're happy with the scanner script, it's time to use @var{Elex} to
generate some code:

@example
> elex CalcScanner.elx
@end example

If no errors occur you'll get two output files, @file{ElexScanner.h} and
@file{ElexScanner.cpp}.  These files define a C++ class called
@code{CalcScanner} which implements the scanner defined by the script.
@file{ElexScanner.h} will look something like this (minus the comments):

@example
#include <cppscan/ElexScanner.h>

class CalcScanner : public ElexScanner
@{
public:

  // These are from the script `symbols' section.
  enum @{SymLBracket, SymMinus, SymMod, SymNumber, SymPlus,
        SymRBracket, SymSlash, SymStar, SymVariable@};

  // These are used internally.
  enum @{ProdNumber, ProdVariable, ProdMod, ProdPlus, ProdMinus,
        ProdStar, ProdSlash, ProdLBracket, ProdRBracket,
        ProdWhiteSpace, Proderror@};

  // Create a CalcScanner reading from `i'.
  CalcScanner (XInputStream &i);

  // Called with one of the ProdXXX constants to invoke the code for
  // the associated production.
  virtual int invokeProduction (int prod);

@group
  // Production handler member functions generated from the code
  // fragments in the script.
  int prodNumber ();
  int prodVariable ();
  int prodMod ();
  int prodPlus ();
  int prodMinus ();
  int prodStar ();
  int prodSlash ();
  int prodLBracket ();
  int prodRBracket ();
  int prodWhiteSpace ();
  int proderror ();
@end group
@};
@end example

@c ----------------------------------------------------------------------
@node       Using the Scanner,  , Generating the Scanner, An Example
@section    Using the Scanner
@cindex compiling a scanner
@cindex using a scanner
@cindex scanner, compiling
@cindex C++ scanner, compiling
@cindex CalcScanner, compiling
@cindex example, using a scanner
@cindex example, compiling a scanner

To use the scanner in an application, you simply create an instance of
the scanner class, passing it an @code{XInputStream} object that
encapsulates an @code{istream}.  You can then use the @code{getNext ()}
function to read each symbol, the @code{getSymbol ()} and @code{getText
()} functions to read the symbol and matched text and the @code{eof ()}
function to test for the end of input.  For example:

@example
@group
XInputStream str (cin);    // read from standard input
CalcScanner scanner (str); // create the scanner

do
@{
  scanner.getNext ();      // read next symbol

  cout << "Symbol: : " << scanner.getSymbol () << endl;
  cout << "Text: : "   << scanner.getText () << endl;
@} while (!scanner.eof ())
@end group
@end example

To compile the C++ calculator application you need to include
@file{CalcScanner.h} and ensure that the appropriate @var{Elex} header
files and libraries are accessible to the compiler.  The @var{Elex}
C++ header files can be found in the @file{cppscan} directory and the
@var{Elex} library is called @file{lib/elex.a}, so the commands to
compile and link the @file{calc.cpp} application are:

@example
@group
> g++ -c calc.cpp CalcParser.cpp CalcScanner.cpp -I$ELEX_HOME/cppscan
> g++ -o calc calc.o CalcParser.o CalcScanner.o $ELEX_HOME/lib/elex.a