changes_from_1.33

本工具提供一个词法分析器和语法分析器的集成开发环境

            (A B)? => <<test(LT(1))>>?  someRule

      are computed during an early phase of antlr processing.  As
      a result, prior to MR13, complex expressions such as:

            ~B
            L..U
            ~L..U
            TokClassName
            ~TokClassName

      were not computed properly.  This resulted in incorrect
      context being computed for such expressions.

      In MR13 these context guards are verified for proper semantics
      in the initial phase and then re-evaluated after complex token
      expressions have been computed in order to produce the correct
      behavior.

      Reported by Arpad Beszedes ([e-mail]).

#167. (Changed in MR13) ~L..U

      Prior to MR13, the complement of a token range was
      not properly computed.

#166. (Changed in MR13) token expression L..U

      The token U was represented as an unsigned char, restricting
      the use of L..U to cases where U was assigned a token number
      less than 256.  This is corrected in MR13.

#165. (Changed in MR13) option -newAST

      To create ASTs from an ANTLRTokenPtr antlr usually calls
      "new AST(ANTLRTokenPtr)".  This option generates a call
      to "newAST(ANTLRTokenPtr)" instead.  This allows a user
      to define a parser member function to create an AST object.

      Similar changes for ASTBase::tmake and ASTBase::link were not
      thought necessary since they do not create AST object, only
      use existing ones.

#164. (Changed in MR13) Unused variable _astp

      For many compilations, we have lived with warnings about
      the unused variable _astp.  It turns out that this varible
      can *never* be used because the code which references it was
      commented out.

      This investigation was sparked by a note from Erwin Achermann
      ([e-mail]).

#163. (Changed in MR13) Incorrect makefiles for testcpp examples

      All the examples in pccts/testcpp/* had incorrect definitions
      in the makefiles for the symbol "CCC".  Instead of CCC=CC they
      had CC=$(CCC).

      There was an additional problem in testcpp/1/test.g due to the
      change in ANTLRToken::getText() to a const member function
      (Item #137).

      Reported by Maurice Mass ([e-mail]).

#162. (Changed in MR13) Combining #token with #tokdefs

      When it became possible to change the print-name of a
      #token (Item #148) it became useful to give a #token
      statement whose only purpose was to giving a print name
      to the #token.  Prior to this change this could not be
      combined with the #tokdefs feature.

#161. (Changed in MR13) Switch -gxt inhibits generation of tokens.h

#160. (Changed in MR13) Omissions in list of names for remap.h

      When a user selects the -gp option antlr creates a list
      of macros in remap.h to rename some of the standard
      antlr routines from zzXXX to userprefixXXX.

      There were number of omissions from the remap.h name
      list related to the new trace facility.  This was reported,
      along with a fix, by Bernie Solomon ([e-mail]).

#159. (Changed in MR13) Violations of classic C rules

      There were a number of violations of classic C style in
      the distribution kit.  This was reported, along with fixes,
      by Bernie Solomon ([e-mail]).

#158. (Changed in MR13) #header causes problem for pre-processors

      A user who runs the C pre-processor on antlr source suggested
      that another syntax be allowed.  With MR13 such directives
      such as #header, #pragma, etc. may be written as "\#header",
      "\#pragma", etc.  For escaping pre-processor directives inside
      a #header use something like the following:

            \#header
            <<
                \#include <stdio.h>
            >>

#157. (Fixed in MR13) empty error sets for rules with infinite recursion

      When the first set for a rule cannot be computed due to infinite
      left recursion and it is the only alternative for a block then
      the error set for the block would be empty.  This would result
      in a fatal error.

      Reported by Darin Creason ([e-mail])

#156. (Changed in MR13) DLGLexerBase::getToken() now public

#155. (Changed in MR13) Context behind predicates can suppress

      With -mrhoist enabled the context behind a guarded predicate can
      be used to suppress other predicates.  Consider the following grammar:

        r0 : (r1)+;

        r1  : rp
            | rq
            ;
        rp  : <<p LATEXT(1)>>? B ;
        rq : (A)? => <<q LATEXT(1)>>? (A|B);

      In earlier versions both predicates "p" and "q" would be hoisted into
      rule r0. With MR12c predicate p is suppressed because the context which
      follows predicate q includes "B" which can "cover" predicate "p".  In
      other words, in trying to decide in r0 whether to call r1, it doesn't
      really matter whether p is false or true because, either way, there is
      a valid choice within r1.

#154. (Changed in MR13) Making hoist suppression explicit using <<nohoist>>

      A common error, even among experienced pccts users, is to code
      an init-action to inhibit hoisting rather than a leading action.
      An init-action does not inhibit hoisting.

      This was coded:

        rule1 : <<;>> rule2

      This is what was meant:

        rule1 : <<;>> <<;>> rule2

      With MR13, the user can code:

        rule1 : <<;>> <<nohoist>> rule2

      The following will give an error message:

        rule1 : <<nohoist>> rule2

      If the <<nohoist>> appears as an init-action rather than a leading
      action an error message is issued.  The meaning of an init-action
      containing "nohoist" is unclear: does it apply to just one
      alternative or to all alternatives ?

#153. (Changed in MR12b) Bug in computation of -mrhoist suppression set

      Consider the following grammar with k=1 and "-mrhoist on":

            r1  : (A)? => ((p>>? x      /* l1 */
                | r2                    /* l2 */
                ;
            r2  :  A                    /* l4 */
                | (B)? => <<q>>? y      /* l5 */
                ;

      In earlier versions the mrhoist routine would see that both l1 and
      l2 contained predicates and would assume that this prevented either
      from acting to suppress the other predicate.  In the example above
      it didn't realize the A at line l4 is capable of suppressing the
      predicate at l1 even though alt l2 contains (indirectly) a predicate.

      This is fixed  in MR12b.

      Reported by Reinier van den Born ([e-mail])

#153. (Changed in MR12a) Bug in computation of -mrhoist suppression set

      An oversight similar to that described in Item #152 appeared in
      the computation of the set that "covered" a predicate.  If a
      predicate expression included a term such as p=AND(q,r) the context
      of p was taken to be context(q) & context(r), when it should have
      been context(q) | context(r).  This is fixed in MR12a.

#152. (Changed in MR12) Bug in generation of predicate expressions

      The primary purpose for MR12 is to make quite clear that MR11 is
      obsolete and to fix the bug related to predicate expressions.

      In MR10 code was added to optimize the code generated for
      predicate expression tests.  Unfortunately, there was a
      significant oversight in the code which resulted in a bug in
      the generation of code for predicate expression tests which
      contained predicates combined using AND:

            r0 : (r1)* "@" ;
            r1 : (AAA)? => <<p LATEXT(1)>>? r2 ;
            r2 : (BBB)? => <<q LATEXT(1)>>? Q
               | (BBB)? => <<r LATEXT(1)>>? Q
               ;

      In MR11 (and MR10 when using "-mrhoist on") the code generated
      for r0 to predict r1 would be equivalent to:

        if ( LA(1)==Q &&
                (LA(1)==AAA && LA(1)==BBB) &&
                    ( p && ( q || r )) ) {

      This is incorrect because it expresses the idea that LA(1)
      *must* be AAA in order to attempt r1, and *must* be BBB to
      attempt r2.  The result was that r1 became unreachable since
      both condition can not be simultaneously true.

      The general philosophy of code generation for predicates
      can be summarized as follows:

            a. If the context is true don't enter an alt
               for which the corresponding predicate is false.

               If the context is false then it is okay to enter
               the alt without evaluating the predicate at all.

            b. A predicate created by ORing of predicates has
               context which is the OR of their individual contexts.

            c. A predicate created by ANDing of predicates has
               (surprise) context which is the OR of their individual
               contexts.

            d. Apply these rules recursively.

            e. Remember rule (a)

      The correct code should express the idea that *if* LA(1) is
      AAA then p must be true to attempt r1, but if LA(1) is *not*
      AAA then it is okay to attempt r1, provided that *if* LA(1) is
      BBB then one of q or r must be true.

        if ( LA(1)==Q &&
                ( !(LA(1)==AAA || LA(1)==BBB) ||
                    ( ! LA(1) == AAA || p) &&
                    ( ! LA(1) == BBB || q || r ) ) ) {

      I believe this is fixed in MR12.

      Reported by Reinier van den Born ([e-mail])

#151a. (Changed in MR12) ANTLRParser::getLexer()

      As a result of several requests, I have added public methods to
      get a pointer to the lexer belonging to a parser.

            ANTLRTokenStream *ANTLRParser::getLexer() const

                Returns a pointer to the lexer being used by the
                parser.  ANTLRTokenStream is the base class of
                DLGLexer

            ANTLRTokenStream *ANTLRTokenBuffer::getLexer() const

                Returns a pointer to the lexer being used by the
                ANTLRTokenBuffer.  ANTLRTokenStream is the base
                class of DLGLexer

      You must manually cast the ANTLRTokenStream to your program's
      lexer class. Because the name of the lexer's class is not fixed.
      Thus it is impossible to incorporate it into the DLGLexerBase
      class.

#151b.(Changed in MR12) ParserBlackBox member getLexer()

      The template class ParserBlackBox now has a member getLexer()
      which returns a pointer to the lexer.

#150. (Changed in MR12) syntaxErrCount and lexErrCount now public

      See Item #127 for more information.

#149. (Changed in MR12) antlr option -info o (letter o for orphan)

      If there is more than one rule which is not referenced by any
      other rule then all such rules are listed.  This is useful for
      alerting one to rules which are not used, but which can still
      contribute to ambiguity.  For example:

            start : a Z ;
            unused: a A ;
            a     : (A)+ ;

      will cause an ambiguity report for rule "a" which will be
      difficult to understand if the user forgets about rule "unused"
      simply because it is not used in the grammar.

#148. (Changed in MR11) #token names appearing in zztokens,token_tbl

      In a #token statement like the following:

            #token Plus "\+"

      the string "Plus" appears in the zztokens array (C mode) and
      token_tbl (C++ mode).  This string is used in most error
      messages.  In MR11 one has the option of using some other string,
      (e.g.  "+") in those tables.

      In MR11 one can write:

            #token Plus ("+")             "\+"
            #token RP   ("(")             "\("
            #token COM  ("comment begin") "/\*"

      A #token statement is allowed to appear in more than one #lexclass
      with different regular expressions.  However, the token name appears
      only once in the zztokens/token_tbl array.  This means that only
      one substitute can be specified for a given #token name.  The second
      attempt to define a substitute name (different from the first) will
      result in an error message.

#147. (Changed in MR11) Bug in follow set computation

      There is a bug in 1.33 vanilla and all maintenance releases
      prior to MR11 in the computation of the follow set.  The bug is
      different than that described in Item #82 and probably more
      common.  It was discovered in the ansi.g grammar while testing
      the "ambiguity aid" (Item #119). The search for a bug started
      when the ambiguity aid was unable to discover the actual source
      of an ambiguity reported by antlr.

      The problem appears when an optimization of the follow set
      computation is used inappropriately.  The result is that the
      follow set used is the "worst case".  In other words, the error
      can lead to false reports of ambiguity.  The good news is that
      if you have a grammar in which you have addressed all reported