readme

来自「mallet是自然语言处理、机器学习领域的一个开源项目。」· 代码 · 共 373 行 · 第 1/2 页

                      Writing Python Regression Tests
                      -------------------------------
                               Skip Montanaro
                              ([e-mail])


Introduction

If you add a new module to Python or modify the functionality of an existing
module, you should write one or more test cases to exercise that new
functionality.  There are different ways to do this within the regression
testing facility provided with Python; any particular test should use only
one of these options.  Each option requires writing a test module using the
conventions of the the selected option:

    - PyUnit based tests
    - doctest based tests
    - "traditional" Python test modules

Regardless of the mechanics of the testing approach you choose,
you will be writing unit tests (isolated tests of functions and objects
defined by the module) using white box techniques.  Unlike black box
testing, where you only have the external interfaces to guide your test case
writing, in white box testing you can see the code being tested and tailor
your test cases to exercise it more completely.  In particular, you will be
able to refer to the C and Python code in the CVS repository when writing
your regression test cases.


PyUnit based tests

The PyUnit framework is based on the ideas of unit testing as espoused
by Kent Beck and the Extreme Programming (XP) movement.  The specific
interface provided by the framework is tightly based on the JUnit
Java implementation of Beck's original SmallTalk test framework.  Please
see the documentation of the unittest module for detailed information on
the interface and general guidelines on writing PyUnit based tests.

The test_support helper module provides a two functions for use by
PyUnit based tests in the Python regression testing framework:
run_unittest() takes a unittest.TestCase derived class as a parameter
and runs the tests defined in that class, and run_suite() takes a
populated TestSuite instance and runs the tests..  All test methods in
the Python regression framework have names that start with "test_" and
use lower-case names with words separated with underscores.

All PyUnit-based tests in the Python test suite use boilerplate that
looks like this:

    import unittest
    import test_support

    class MyTestCase(unittest.TestCase):
        # define test methods here...

    def test_main():
        test_support.run_unittest(MyTestCase)

    if __name__ == "__main__":
        test_main()

This has the advantage that it allows the unittest module to be used
as a script to run individual tests as well as working well with the
regrtest framework.


doctest based tests

Tests written to use doctest are actually part of the docstrings for
the module being tested.  Each test is written as a display of an
interactive session, including the Python prompts, statements that would
be typed by the user, and the output of those statements (including
tracebacks, although only the exception msg needs to be retained then).
The module in the test package is simply a wrapper that causes doctest
to run over the tests in the module.  The test for the difflib module
provides a convenient example:

    import difflib, test_support
    test_support.run_doctest(difflib)

If the test is successful, nothing is written to stdout (so you should not
create a corresponding output/test_difflib file), but running regrtest
with -v will give a detailed report, the same as if passing -v to doctest.

A second argument can be passed to run_doctest to tell doctest to search
sys.argv for -v instead of using test_support's idea of verbosity.  This
is useful for writing doctest-based tests that aren't simply running a
doctest'ed Lib module, but contain the doctests themselves.  Then at
times you may want to run such a test directly as a doctest, independent
of the regrtest framework.  The tail end of test_descrtut.py is a good
example:

    def test_main(verbose=None):
        import test_support, test.test_descrtut
        test_support.run_doctest(test.test_descrtut, verbose)

    if __name__ == "__main__":
        test_main(1)

If run via regrtest, test_main() is called (by regrtest) without specifying
verbose, and then test_support's idea of verbosity is used.  But when
run directly, test_main(1) is called, and then doctest's idea of verbosity
is used.

See the documentation for the doctest module for information on
writing tests using the doctest framework.


"traditional" Python test modules

The mechanics of how the "traditional" test system operates are fairly
straightforward.  When a test case is run, the output is compared with the
expected output that is stored in .../Lib/test/output.  If the test runs to
completion and the actual and expected outputs match, the test succeeds, if
not, it fails.  If an ImportError or test_support.TestSkipped error is
raised, the test is not run.


Executing Test Cases

If you are writing test cases for module spam, you need to create a file
in .../Lib/test named test_spam.py.  In addition, if the tests are expected
to write to stdout during a successful run, you also need to create an
expected output file in .../Lib/test/output named test_spam ("..."
represents the top-level directory in the Python source tree, the directory
containing the configure script).  If needed, generate the initial version
of the test output file by executing:

    ./python Lib/test/regrtest.py -g test_spam.py

from the top-level directory.

Any time you modify test_spam.py you need to generate a new expected
output file.  Don't forget to desk check the generated output to make sure
it's really what you expected to find!  All in all it's usually better
not to have an expected-out file (note that doctest- and unittest-based
tests do not).

To run a single test after modifying a module, simply run regrtest.py
without the -g flag:

    ./python Lib/test/regrtest.py test_spam.py

While debugging a regression test, you can of course execute it
independently of the regression testing framework and see what it prints:

    ./python Lib/test/test_spam.py

To run the entire test suite:

[UNIX, + other platforms where "make" works] Make the "test" target at the
top level:

    make test

{WINDOWS] Run rt.bat from your PCBuild directory.  Read the comments at
the top of rt.bat for the use of special -d, -O and -q options processed
by rt.bat.

[OTHER] You can simply execute the two runs of regrtest (optimized and
non-optimized) directly:

    ./python Lib/test/regrtest.py
    ./python -O Lib/test/regrtest.py

But note that this way picks up whatever .pyc and .pyo files happen to be
around.  The makefile and rt.bat ways run the tests twice, the first time
removing all .pyc and .pyo files from the subtree rooted at Lib/.

Test cases generate output based upon values computed by the test code.
When executed, regrtest.py compares the actual output generated by executing
the test case with the expected output and reports success or failure.  It
stands to reason that if the actual and expected outputs are to match, they
must not contain any machine dependencies.  This means your test cases
should not print out absolute machine addresses (e.g. the return value of
the id() builtin function) or floating point numbers with large numbers of
significant digits (unless you understand what you are doing!).


Test Case Writing Tips

Writing good test cases is a skilled task and is too complex to discuss in
detail in this short document.  Many books have been written on the subject.
I'll show my age by suggesting that Glenford Myers' "The Art of Software
Testing", published in 1979, is still the best introduction to the subject
available.  It is short (177 pages), easy to read, and discusses the major
elements of software testing, though its publication predates the