news

Flex词法/语法分析器源码

Changes between release 2.5.2 (25Apr95) and release 2.5.1:

	- The --prefix configuration option now works.

	- A bug that completely broke the "-Cf" table compression
	  option has been fixed.

	- A major headache involving "const" declarators and Solaris
	  systems has been fixed.

	- An octal escape sequence in a flex regular expression must
	  now contain only the digits 0-7.

	- You can now use "--" on the flex command line to mark the
	  end of flex options.

	- You can now specify the filename '-' as a synonym for stdin.

	- By default, the scanners generated by flex no longer
	  statically initialize yyin and yyout to stdin and stdout.
	  This change is necessary because in some ANSI environments,
	  stdin and stdout are not compile-time constant.  You can
	  force the initialization using "%option stdinit" in the first
	  section of your flex input.

	- "%option nounput" now correctly omits the unput() routine
	  from the output.

	- "make clean" now removes config.log, config.cache, and the
	  flex binary.  The fact that it removes the flex binary means
	  you should take care if making changes to scan.l, to make
	  sure you don't wind up in a bootstrap problem.

	- In general, the Makefile has been reworked somewhat (thanks
	  to Francois Pinard) for added flexibility - more changes will
	  follow in subsequent releases.

	- The .texi and .info files in MISC/texinfo/ have been updated,
	  thanks also to Francois Pinard.

	- The FlexLexer::yylex(istream* new_in, ostream* new_out) method
	  now does not have a default for the first argument, to disambiguate
	  it from FlexLexer::yylex().

	- A bug in destructing a FlexLexer object before doing any scanning
	  with it has been fixed.

	- A problem with including FlexLexer.h multiple times has been fixed.

	- The alloca() chud necessary to accommodate bison has grown
	  even uglier, but hopefully more correct.

	- A portability tweak has been added to accommodate compilers that
	  use char* generic pointers.

	- EBCDIC contact information in the file MISC/EBCDIC has been updated.

	- An OS/2 Makefile and config.h for flex 2.5 is now available in
	  MISC/OS2/, contributed by Kai Uwe Rommel.

	- The descrip.mms file for building flex under VMS has been updated,
	  thanks to Pat Rankin.

	- The notes on building flex for the Amiga have been updated for
	  flex 2.5, contributed by Andreas Scherer.


Changes between release 2.5.1 (28Mar95) and release 2.4.7:

	- A new concept of "start condition" scope has been introduced.
	  A start condition scope is begun with:

		<SCs>{

	  where SCs is a list of one or more start conditions.  Inside
	  the start condition scope, every rule automatically has the
	  prefix <SCs> applied to it, until a '}' which matches the
	  initial '{'.  So, for example:

		<ESC>{
			"\\n"	return '\n';
			"\\r"	return '\r';
			"\\f"	return '\f';
			"\\0"	return '\0';
		}

	  is equivalent to:

		<ESC>"\\n"	return '\n';
		<ESC>"\\r"	return '\r';
		<ESC>"\\f"	return '\f';
		<ESC>"\\0"	return '\0';

	  As indicated in this example, rules inside start condition scopes
	  (and any rule, actually, other than the first) can be indented,
	  to better show the extent of the scope.

	  Start condition scopes may be nested.

	- The new %option directive can be used in the first section of
	  a flex scanner to control scanner-generation options.  Most
	  options are given simply as names, optionally preceded by the
	  word "no" (with no intervening whitespace) to negate their
	  meaning.  Some are equivalent to flex flags, so putting them
	  in your scanner source is equivalent to always specifying
	  the flag (%option's take precedence over flags):

		7bit	-7 option
		8bit	-8 option
		align	-Ca option
		backup	-b option
		batch	-B option
		c++	-+ option
		caseful	opposite of -i option (caseful is the default);
		case-sensitive	same as above
		caseless	-i option;
		case-insensitive	same as above
		debug	-d option
		default	opposite of -s option
		ecs	-Ce option
		fast	-F option
		full	-f option
		interactive	-I option
		lex-compat	-l option
		meta-ecs	-Cm option
		perf-report	-p option
		read	-Cr option
		stdout	-t option
		verbose	-v option
		warn	opposite of -w option (so use "%option nowarn" for -w)

		array	equivalent to "%array"
		pointer	equivalent to "%pointer" (default)

	  Some provide new features:

		always-interactive	generate a scanner which always
			considers its input "interactive" (no call to isatty()
			will be made when the scanner runs)
		main	supply a main program for the scanner, which
			simply calls yylex().  Implies %option noyywrap.
		never-interactive	generate a scanner which never
			considers its input "interactive" (no call to isatty()
			will be made when the scanner runs)
		stack	if set, enable start condition stacks (see below)
		stdinit	if unset ("%option nostdinit"), initialize yyin
			and yyout statically to nil FILE* pointers, instead
			of stdin and stdout
		yylineno	if set, keep track of the current line
			number in global yylineno (this option is expensive
			in terms of performance).  The line number is available
			to C++ scanning objects via the new member function
			lineno().
		yywrap	if unset ("%option noyywrap"), scanner does not
			call yywrap() upon EOF but simply assumes there
			are no more files to scan

	  Flex scans your rule actions to determine whether you use the
	  REJECT or yymore features (this is not new).  Two %options can be
	  used to override its decision, either by setting them to indicate
	  the feature is indeed used, or unsetting them to indicate it
	  actually is not used:

		reject
		yymore

	  Three %option's take string-delimited values, offset with '=':

		outfile="<name>"	equivalent to -o<name>
		prefix="<name>"		equivalent to -P<name>
		yyclass="<name>"	set the name of the C++ scanning class
					(see below)

	  A number of %option's are available for lint purists who
	  want to suppress the appearance of unneeded routines in
	  the generated scanner.  Each of the following, if unset,
	  results in the corresponding routine not appearing in the
	  generated scanner:

		input, unput
		yy_push_state, yy_pop_state, yy_top_state
		yy_scan_buffer, yy_scan_bytes, yy_scan_string

	  You can specify multiple options with a single %option directive,
	  and multiple directives in the first section of your flex input file.

	- The new function:

		YY_BUFFER_STATE yy_scan_string( const char *str )

	  returns a YY_BUFFER_STATE (which also becomes the current input
	  buffer) for scanning the given string, which occurs starting
	  with the next call to yylex().  The string must be NUL-terminated.
	  A related function:

		YY_BUFFER_STATE yy_scan_bytes( const char *bytes, int len )

	  creates a buffer for scanning "len" bytes (including possibly NUL's)
	  starting at location "bytes".

	  Note that both of these functions create and scan a *copy* of
	  the string/bytes.  (This may be desirable, since yylex() modifies
	  the contents of the buffer it is scanning.)  You can avoid the
	  copy by using:

		YY_BUFFER_STATE yy_scan_buffer( char *base, yy_size_t size )

	  which scans in place the buffer starting at "base", consisting
	  of "size" bytes, the last two bytes of which *must* be
	  YY_END_OF_BUFFER_CHAR (these bytes are not scanned; thus, scanning
	  consists of base[0] through base[size-2], inclusive).  If you
	  fail to set up "base" in this manner, yy_scan_buffer returns a
	  nil pointer instead of creating a new input buffer.

	  The type yy_size_t is an integral type to which you can cast
	  an integer expression reflecting the size of the buffer.

	- Three new routines are available for manipulating stacks of
	  start conditions:

		void yy_push_state( int new_state )

	  pushes the current start condition onto the top of the stack
	  and BEGIN's "new_state" (recall that start condition names are
	  also integers).

		void yy_pop_state()

	  pops the top of the stack and BEGIN's to it, and

		int yy_top_state()

	  returns the top of the stack without altering the stack's
	  contents.

	  The start condition stack grows dynamically and so has no built-in
	  size limitation.  If memory is exhausted, program execution
	  is aborted.

	  To use start condition stacks, your scanner must include
	  a "%option stack" directive.

	- flex now supports POSIX character class expressions.  These
	  are expressions enclosed inside "[:" and ":]" delimiters (which
	  themselves must appear between the '[' and ']' of a character
	  class; other elements may occur inside the character class, too).
	  The expressions flex recognizes are:

		[:alnum:] [:alpha:] [:blank:] [:cntrl:] [:digit:] [:graph:]	
		[:lower:] [:print:] [:punct:] [:space:] [:upper:] [:xdigit:]

	  These expressions all designate a set of characters equivalent to
	  the corresponding isXXX function (for example, [:alnum:] designates
	  those characters for which isalnum() returns true - i.e., any
	  alphabetic or numeric).  Some systems don't provide isblank(),
	  so flex defines [:blank:] as a blank or a tab.

	  For example, the following character classes are all equivalent:

		[[:alnum:]]
		[[:alpha:][:digit:]
		[[:alpha:]0-9]
		[a-zA-Z0-9]

	  If your scanner is case-insensitive (-i flag), then [:upper:]
	  and [:lower:] are equivalent to [:alpha:].

	- The promised rewrite of the C++ FlexLexer class has not yet
	  been done.  Support for FlexLexer is limited at the moment to
	  fixing show-stopper bugs, so, for example, the new functions
	  yy_scan_string() & friends are not available to FlexLexer
	  objects.

	- The new macro

		yy_set_interactive(is_interactive)

	  can be used to control whether the current buffer is considered
	  "interactive".  An interactive buffer is processed more slowly,
	  but must be used when the scanner's input source is indeed
	  interactive to avoid problems due to waiting to fill buffers
	  (see the discussion of the -I flag in flex.1).  A non-zero value
	  in the macro invocation marks the buffer as interactive, a zero
	  value as non-interactive.  Note that use of this macro overrides
	  "%option always-interactive" or "%option never-interactive".

	  yy_set_interactive() must be invoked prior to beginning to
	  scan the buffer.

	- The new macro

		yy_set_bol(at_bol)

	  can be used to control whether the current buffer's scanning
	  context for the next token match is done as though at the
	  beginning of a line (non-zero macro argument; makes '^' anchored
	  rules active) or not at the beginning of a line (zero argument,
	  '^' rules inactive).

	- Related to this change, the mechanism for determining when a scan is
	  starting at the beginning of a line has changed.  It used to be
	  that '^' was active iff the character prior to that at which the
	  scan started was a newline.  The mechanism now is that '^' is
	  active iff the last token ended in a newline (or the last call to
	  input() returned a newline).  For most users, the difference in
	  mechanisms is negligible.  Where it will make a difference,
	  however, is if unput() or yyless() is used to alter the input
	  stream.  When in doubt, use yy_set_bol().

	- The new beginning-of-line mechanism involved changing some fairly
	  twisted code, so it may have introduced bugs - beware ...

	- The macro YY_AT_BOL() returns true if the next token scanned from
	  the current buffer will have '^' rules active, false otherwise.

	- The new function

		void yy_flush_buffer( struct yy_buffer_state* b )

	  flushes the contents of the current buffer (i.e., next time
	  the scanner attempts to match a token using b as the current
	  buffer, it will begin by invoking YY_INPUT to fill the buffer).
	  This routine is also available to C++ scanners (unlike some
	  of the other new routines).

	  The related macro

		YY_FLUSH_BUFFER

	  flushes the contents of the current buffer.

	- A new "-ooutput" option writes the generated scanner to "output".
	  If used with -t, the scanner is still written to stdout, but
	  its internal #line directives (see previous item) use "output".

	- Flex now generates #line directives relating the code it
	  produces to the output file; this means that error messages
	  in the flex-generated code should be correctly pinpointed.

	- When generating #line directives, filenames with embedded '\'s
	  have those characters escaped (i.e., turned into '\\').  This
	  feature helps with reporting filenames for some MS-DOS and OS/2
	  systems.

	- The FlexLexer class includes two new public member functions:

		virtual void switch_streams( istream* new_in = 0,
						ostream* new_out = 0 )

	  reassigns yyin to new_in (if non-nil) and yyout to new_out
	  (ditto), deleting the previous input buffer if yyin is
	  reassigned.  It is used by:

		int yylex( istream* new_in = 0, ostream* new_out = 0 )

	  which first calls switch_streams() and then returns the value
	  of calling yylex().

	- C++ scanners now have yy_flex_debug as a member variable of
	  FlexLexer rather than a global, and member functions for testing
	  and setting it.

	- When generating a C++ scanning class, you can now use

		%option yyclass="foo"

	  to inform flex that you have derived "foo" as a subclass of
	  yyFlexLexer, so flex will place your actions in the member
	  function foo::yylex() instead of yyFlexLexer::yylex().  It also
	  generates a yyFlexLexer::yylex() member function that generates a
	  run-time error if called (by invoking yyFlexLexer::LexerError()).
	  This feature is necessary if your subclass "foo" introduces some
	  additional member functions or variables that you need to access
	  from yylex().

	- Current texinfo files in MISC/texinfo, contributed by Francois
	  Pinard.

	- You can now change the name "flex" to something else (e.g., "lex")
	  by redefining $(FLEX) in the Makefile.

	- Two bugs (one serious) that could cause "bigcheck" to fail have
	  been fixed.

	- A number of portability/configuration changes have been made
	  for easier portability.

	- You can use "YYSTATE" in your scanner as an alias for YY_START
	  (for AT&T lex compatibility).

	- input() now maintains yylineno.

	- input() no longer trashes yytext.

	- interactive scanners now read characters in YY_INPUT up to a
	  newline, a large performance gain.

	- C++ scanner objects now work with the -P option.  You include
	  <FlexLexer.h> once per scanner - see comments in <FlexLexer.h>
	  (or flex.1) for details.

	- C++ FlexLexer objects now use the "cerr" stream to report -d output
	  instead of stdio.