tcltest.n

来自「tcl是工具命令语言」· N 代码 · 共 1,066 行 · 第 1/3 页

.TP
\fBnormalizePath\fR \fIpathVar\fR
Resolves symlinks in a path, thus creating a path without internal
redirection.  It is assumed that \fIpathVar\fR is absolute.
\fIpathVar\fR is modified in place.  The Tcl command [\fBfile normalize\fR]
is a sufficient replacement.
.TP
\fBbytestring\fR \fIstring\fR
Construct a string that consists of the requested sequence of bytes,
as opposed to a string of properly formed UTF-8 characters using the
value supplied in \fIstring\fR.  This allows the tester to create
denormalized or improperly formed strings to pass to C procedures that
are supposed to accept strings with embedded NULL types and confirm
that a string result has a certain pattern of bytes.  This is
exactly equivalent to the Tcl command [\fBencoding convertfrom identity\fR].
.SH TESTS
.PP
The [\fBtest\fR] command is the heart of the \fBtcltest\fR package.
Its essential function is to evaluate a Tcl script and compare
the result with an expected result.  The options of [\fBtest\fR]
define the test script, the environment in which to evaluate it,
the expected result, and how the compare the actual result to
the expected result.  Some configuration options of \fBtcltest\fR
also influence how [\fBtest\fR] operates.
.PP
The valid options for [\fBtest\fR] are summarized:
.CS
.ta 0.8i
test \fIname\fR \fIdescription\fR
	?-constraints \fIkeywordList|expression\fR?
	?-setup \fIsetupScript\fR?
	?-body \fItestScript\fR?
	?-cleanup \fIcleanupScript\fR?
	?-result \fIexpectedAnswer\fR?
	?-output \fIexpectedOutput\fR?
	?-errorOutput \fIexpectedError\fR?
	?-returnCodes \fIcodeList\fR?
	?-match \fImode\fR?
.CE
The \fIname\fR may be any string.  It is conventional to choose
a \fIname\fR according to the pattern:
.CS
\fItarget\fR-\fImajorNum\fR.\fIminorNum\fR
.CE
For white-box (regression) tests, the target should be the name of the
C function or Tcl procedure being tested.  For black-box tests, the
target should be the name of the feature being tested.  Some conventions
call for the names of black-box tests to have the suffix \fB_bb\fR.
Related tests should share a major number.  As a test suite evolves,
it is best to have the same test name continue to correspond to the
same test, so that it remains meaningful to say things like ``Test
foo-1.3 passed in all releases up to 3.4, but began failing in
release 3.5.''
.PP
During evaluation of [\fBtest\fR], the \fIname\fR will be compared
to the lists of string matching patterns returned by
[\fBconfigure -match\fR], and [\fBconfigure -skip\fR].  The test
will be run only if \fIname\fR matches any of the patterns from
[\fBconfigure -match\fR] and matches none of the patterns
from [\fBconfigure -skip\fR].
.PP
The \fIdescription\fR should be a short textual description of the
test.  The \fIdescription\fR is included in output produced by the
test, typically test failure messages.  Good \fIdescription\fR values
should briefly explain the purpose of the test to users of a test suite.
The name of a Tcl or C function being tested should be included in the
description for regression tests.  If the test case exists to reproduce
a bug, include the bug ID in the description. 
.PP
Valid attributes and associated values are:
.TP
\fB-constraints \fIkeywordList|expression\fR
The optional \fB-constraints\fR attribute can be list of one or more
keywords or an expression.  If the \fB-constraints\fR value is a list of
keywords, each of these keywords should be the name of a constraint
defined by a call to [\fBtestConstraint\fR].  If any of the listed
constraints is false or does not exist, the test is skipped.  If the
\fB-constraints\fR value is an expression, that expression
is evaluated. If the expression evaluates to true, then the test is run.
Note that the expression form of \fB-constraints\fR may interfere with the
operation of [\fBconfigure -constraints\fR] and
[\fBconfigure -limitconstraints\fR], and is not recommended.
Appropriate constraints should be added to any tests that should
not always be run.  That is, conditional evaluation of a test
should be accomplished by the \fB-constraints\fR option, not by
conditional evaluation of [\fBtest\fR].  In that way, the same
number of tests are always reported by the test suite, though
the number skipped may change based on the testing environment.
The default value is an empty list.  
See \fBTEST CONSTRAINTS\fR below for a list of built-in constraints 
and information on how to add your own constraints.
.TP
\fB-setup \fIscript\fR
The optional \fB-setup\fR attribute indicates a \fIscript\fR that will be run
before the script indicated by the \fB-body\fR attribute.  If evaluation
of \fIscript\fR raises an error, the test will fail.  The default value
is an empty script.
.TP
\fB-body \fIscript\fR
The \fB-body\fR attribute indicates the \fIscript\fR to run to carry out the 
test.  It must return a result that can be checked for correctness.
If evaluation of \fIscript\fR raises an error, the test will fail.
The default value is an empty script.
.TP
\fB-cleanup \fIscript\fR
The optional \fB-cleanup\fR attribute indicates a \fIscript\fR that will be
run after the script indicated by the \fB-body\fR attribute.
If evaluation of \fIscript\fR raises an error, the test will fail.
The default value is an empty script.
.TP
\fB-match \fImode\fR
The \fB-match\fR attribute determines how expected answers supplied by
\fB-result\fR, \fB-output\fR, and \fB-errorOutput\fR are compared.  Valid
values for \fImode\fR are \fBregexp\fR, \fBglob\fR, \fBexact\fR, and
any value registered by a prior call to [\fBcustomMatch\fR].  The default
value is \fBexact\fR.
.TP
\fB-result \fIexpectedValue\fR
The \fB-result\fR attribute supplies the \fIexpectedValue\fR against which
the return value from script will be compared. The default value is
an empty string.
.TP
\fB-output \fIexpectedValue\fR
The \fB-output\fR attribute supplies the \fIexpectedValue\fR against which
any output sent to \fBstdout\fR or [\fBoutputChannel\fR] during evaluation
of the script(s) will be compared.  Note that only output printed using
[\fBputs\fR] is used for comparison.  If \fB-output\fR is not specified,
output sent to \fBstdout\fR and [\fBoutputChannel\fR] is not processed for
comparison.
.TP
\fB-errorOutput \fIexpectedValue\fR
The \fB-errorOutput\fR attribute supplies the \fIexpectedValue\fR against
which any output sent to \fBstderr\fR or [\fBerrorChannel\fR] during 
evaluation of the script(s) will be compared. Note that only output
printed using [\fBputs\fR] is used for comparison.  If \fB-errorOutput\fR
is not specified, output sent to \fBstderr\fR and [\fBerrorChannel\fR] is
not processed for comparison.
.TP
\fB-returnCodes \fIexpectedCodeList\fR
The optional \fB-returnCodes\fR attribute supplies \fIexpectedCodeList\fR,
a list of return codes that may be accepted from evaluation of the
\fB-body\fR script.  If evaluation of the \fB-body\fR script returns
a code not in the \fIexpectedCodeList\fR, the test fails.  All
return codes known to [\fBreturn\fR], in both numeric and symbolic
form, including extended return codes, are acceptable elements in
the \fIexpectedCodeList\fR.  Default value is \fB{ok return}\fR.
.PP
To pass, a test must successfully evaluate its \fB-setup\fR, \fB-body\fR,
and \fB-cleanup\fR scripts.  The return code of the \fB-body\fR script and
its result must match expected values, and if specified, output and error
data from the test must match expected \fB-output\fR and \fB-errorOutput\fR
values.  If any of these conditions are not met, then the test fails.
Note that all scripts are evaluated in the context of the caller
of [\fBtest\fR].
.PP
As long as [\fBtest\fR] is called with valid syntax and legal
values for all attributes, it will not raise an error.  Test
failures are instead reported as output written to [\fBoutputChannel\fR].
In default operation, a successful test produces no output.  The output
messages produced by [\fBtest\fR] are controlled by the
[\fBconfigure -verbose\fR] option as described in \fBCONFIGURABLE OPTIONS\fR
below.  Any output produced by the test scripts themselves should be
produced using [\fBputs\fR] to [\fBoutputChannel\fR] or
[\fBerrorChannel\fR], so that users of the test suite may
easily capture output with the [\fBconfigure -outfile\fR] and
[\fBconfigure -errfile\fR] options, and so that the \fB-output\fR
and \fB-errorOutput\fR attributes work properly.
.SH "TEST CONSTRAINTS"
.PP
Constraints are used to determine whether or not a test should be skipped.
Each constraint has a name, which may be any string, and a boolean
value.  Each [\fBtest\fR] has a \fB-constraints\fR value which is a
list of constraint names.  There are two modes of constraint control.
Most frequently, the default mode is used, indicated by a setting
of [\fBconfigure -limitconstraints\fR] to false.  The test will run
only if all constraints in the list are true-valued.  Thus,
the \fB-constraints\fR option of [\fBtest\fR] is a convenient, symbolic
way to define any conditions required for the test to be possible or
meaningful.  For example, a [\fBtest\fR] with \fB-constraints unix\fR
will only be run if the constraint \fBunix\fR is true, which indicates
the test suite is being run on a Unix platform.
.PP
Each [\fBtest\fR] should include whatever \fB-constraints\fR are
required to constrain it to run only where appropriate.  Several
constraints are pre-defined in the \fBtcltest\fR package, listed
below.  The registration of user-defined constraints is performed
by the [\fBtestConstraint\fR] command.  User-defined constraints
may appear within a test file, or within the script specified
by the [\fBconfigure -load\fR] or [\fBconfigure -loadfile\fR]
options.
.PP
The following is a list of constraints pre-defined by the
\fBtcltest\fR package itself:
.TP
\fIsingleTestInterp\fR
test can only be run if all test files are sourced into a single interpreter
.TP
\fIunix\fR
test can only be run on any Unix platform
.TP
\fIwin\fR
test can only be run on any Windows platform
.TP
\fInt\fR
test can only be run on any Windows NT platform
.TP
\fI95\fR
test can only be run on any Windows 95 platform
.TP
\fI98\fR
test can only be run on any Windows 98 platform
.TP
\fImac\fR
test can only be run on any Mac platform
.TP
\fIunixOrWin\fR
test can only be run on a Unix or Windows platform
.TP
\fImacOrWin\fR
test can only be run on a Mac or Windows platform
.TP
\fImacOrUnix\fR
test can only be run on a Mac or Unix platform
.TP
\fItempNotWin\fR
test can not be run on Windows.  This flag is used to temporarily
disable a test. 
.TP
\fItempNotMac\fR
test can not be run on a Mac.  This flag is used
to temporarily disable a test.
.TP
\fIunixCrash\fR
test crashes if it's run on Unix.  This flag is used to temporarily
disable a test. 
.TP
\fIwinCrash\fR
test crashes if it's run on Windows.  This flag is used to temporarily
disable a test. 
.TP
\fImacCrash\fR
test crashes if it's run on a Mac.  This flag is used to temporarily
disable a test. 
.TP
\fIemptyTest\fR
test is empty, and so not worth running, but it remains as a
place-holder for a test to be written in the future.  This constraint
has value false to cause tests to be skipped unless the user specifies
otherwise.
.TP
\fIknownBug\fR
test is known to fail and the bug is not yet fixed.  This constraint
has value false to cause tests to be skipped unless the user specifies
otherwise.
.TP
\fInonPortable\fR
test can only be run in some known development environment.
Some tests are inherently non-portable because they depend on things
like word length, file system configuration, window manager, etc.
This constraint has value false to cause tests to be skipped unless
the user specifies otherwise.  
.TP
\fIuserInteraction\fR
test requires interaction from the user.  This constraint has
value false to causes tests to be skipped unless the user specifies
otherwise.  
.TP
\fIinteractive\fR
test can only be run in if the interpreter is in interactive mode 
(when the global tcl_interactive variable is set to 1).
.TP
\fInonBlockFiles\fR
test can only be run if platform supports setting files into
nonblocking mode 
.TP
\fIasyncPipeClose\fR
test can only be run if platform supports async flush and async close
on a pipe 
.TP
\fIunixExecs\fR
test can only be run if this machine has Unix-style commands
\fBcat\fR, \fBecho\fR, \fBsh\fR, \fBwc\fR, \fBrm\fR, \fBsleep\fR,
\fBfgrep\fR, \fBps\fR, \fBchmod\fR, and \fBmkdir\fR available
.TP
\fIhasIsoLocale\fR
test can only be run if can switch to an ISO locale
.TP
\fIroot\fR
test can only run if Unix user is root
.TP
\fInotRoot\fR
test can only run if Unix user is not root
.TP
\fIeformat\fR
test can only run if app has a working version of sprintf with respect
to the "e" format of floating-point numbers.
.TP
\fIstdio\fR
test can only be run if [\fBinterpreter\fR] can be [\fBopen\fR]ed
as a pipe.
.PP
The alternative mode of constraint control is enabled by setting
[\fBconfigure -limitconstraints\fR] to true.  With that configuration
setting, all existing constraints other than those in the constraint
list returned by [\fBconfigure -constraints\fR] are set to false.
When the value of [\fBconfigure -constraints\fR]
is set, all those constraints are set to true.  The effect is that
when both options [\fBconfigure -constraints\fR] and
[\fBconfigure -limitconstraints\fR] are in use, only those tests including
only constraints from the [\fBconfigure -constraints\fR] list
are run; all others are skipped.  For example, one might set
up a configuration with
.CS
configure -constraints knownBug \e
          -limitconstraints true \e
          -verbose pass
.CE
to run exactly those tests that exercise known bugs, and discover
whether any of them pass, indicating the bug had been fixed.  
.SH "RUNNING ALL TESTS"
.PP
The single command [\fBrunAllTests\fR] is evaluated to run an entire
test suite, spanning many files and directories.  The configuration
options of \fBtcltest\fR control the precise operations.  The
[\fBrunAllTests\fR] command begins by printing a summary of its
configuration to [\fBoutputChannel\fR].
.PP
Test files to be evaluated are sought in the directory
[\fBconfigure -testdir\fR].  The list of files in that directory
that match any of the patterns in [\fBconfigure -file\fR] and
match none of the patterns in [\fBconfigure -notfile\fR] is generated
and sorted.  Then each file will be evaluated in turn.  If
[\fBconfigure -singleproc\fR] is true, then each file will
be [\fBsource\fR]d in the caller's context.  If if is false,
then a copy of [\fBinterpreter\fR] will be [\fBexec\fR]d to
evaluate each file.  The multi-process operation is useful
when testing can cause errors so severe that a process 
terminates.  Although such an error may terminate a child
process evaluating one file, the master process can continue
with the rest of the test suite.  In multi-process operation,
the configuration of \fBtcltest\fR in the master process is
passed to the child processes as command line arguments,
with the exception of [\fBconfigure -outfile\fR].  The
[\fBrunAllTests\fR] command in the
master process collects all output from the child processes
and collates their results into one master report.  Any
reports of individual test failures, or messages requested
by a [\fBconfigure -verbose\fR] setting are passed directly
on to [\fBoutputChannel\fR] by the master process.
.PP
After evaluating all selected test files, a summary of the
results is printed to [\fBoutputChannel\fR].  The summary
includes the total number of [\fBtest\fR]s evaluated, broken
down into those skipped, those passed, and those failed.
The summary also notes the number of files evaluated, and the names
of any files with failing tests or errors.  A list of