/Lib/test/README

  1+++++++++++++++++++++++++++++++
  2Writing Python Regression Tests
  3+++++++++++++++++++++++++++++++
  4
  5:Author: Skip Montanaro
  6:Contact: skip@pobox.com
  7
  8Introduction
  9============
 10
 11If you add a new module to Python or modify the functionality of an existing
 12module, you should write one or more test cases to exercise that new
 13functionality.  There are different ways to do this within the regression
 14testing facility provided with Python; any particular test should use only
 15one of these options.  Each option requires writing a test module using the
 16conventions of the selected option:
 17
 18    - unittest_ based tests
 19    - doctest_ based tests
 20    - "traditional" Python test modules
 21
 22Regardless of the mechanics of the testing approach you choose,
 23you will be writing unit tests (isolated tests of functions and objects
 24defined by the module) using white box techniques.  Unlike black box
 25testing, where you only have the external interfaces to guide your test case
 26writing, in white box testing you can see the code being tested and tailor
 27your test cases to exercise it more completely.  In particular, you will be
 28able to refer to the C and Python code in the CVS repository when writing
 29your regression test cases.
 30
 31.. _unittest: http://www.python.org/doc/current/lib/module-unittest.html
 32.. _doctest: http://www.python.org/doc/current/lib/module-doctest.html
 33
 34unittest-based tests
 35------------------
 36The unittest_ framework is based on the ideas of unit testing as espoused
 37by Kent Beck and the `Extreme Programming`_ (XP) movement.  The specific
 38interface provided by the framework is tightly based on the JUnit_
 39Java implementation of Beck's original SmallTalk test framework.  Please
 40see the documentation of the unittest_ module for detailed information on
 41the interface and general guidelines on writing unittest-based tests.
 42
 43The test_support helper module provides a function for use by
 44unittest-based tests in the Python regression testing framework,
 45``run_unittest()``. This is the primary way of running tests in the
 46standard library. You can pass it any number of the following:
 47
 48- classes derived from or instances of ``unittest.TestCase`` or
 49  ``unittest.TestSuite``. These will be handed off to unittest for
 50  converting into a proper TestSuite instance.
 51
 52- a string; this must be a key in sys.modules. The module associated with
 53  that string will be scanned by ``unittest.TestLoader.loadTestsFromModule``.
 54  This is usually seen as ``test_support.run_unittest(__name__)`` in a test
 55  module's ``test_main()`` function. This has the advantage of picking up
 56  new tests automatically, without you having to add each new test case
 57  manually.
 58   
 59All test methods in the Python regression framework have names that
 60start with "``test_``" and use lower-case names with words separated with
 61underscores.
 62
 63Test methods should *not* have docstrings!  The unittest module prints
 64the docstring if there is one, but otherwise prints the function name
 65and the full class name.  When there's a problem with a test, the
 66latter information makes it easier to find the source for the test
 67than the docstring.
 68
 69All unittest-based tests in the Python test suite use boilerplate that
 70looks like this (with minor variations)::
 71
 72    import unittest
 73    from test import test_support
 74
 75    class MyTestCase1(unittest.TestCase):
 76
 77        # Define setUp and tearDown only if needed
 78
 79        def setUp(self):
 80            unittest.TestCase.setUp(self)
 81            ... additional initialization...
 82
 83        def tearDown(self):
 84            ... additional finalization...
 85            unittest.TestCase.tearDown(self)
 86
 87        def test_feature_one(self):
 88            # Testing feature one
 89            ...unit test for feature one...
 90
 91        def test_feature_two(self):
 92            # Testing feature two
 93            ...unit test for feature two...
 94
 95        ...etc...
 96
 97    class MyTestCase2(unittest.TestCase):
 98        ...same structure as MyTestCase1...
 99
100    ...etc...
101
102    def test_main():
103        test_support.run_unittest(__name__)
104
105    if __name__ == "__main__":
106        test_main()
107
108This has the advantage that it allows the unittest module to be used
109as a script to run individual tests as well as working well with the
110regrtest framework.
111
112.. _Extreme Programming: http://www.extremeprogramming.org/
113.. _JUnit: http://www.junit.org/
114
115doctest based tests
116-------------------
117Tests written to use doctest_ are actually part of the docstrings for
118the module being tested.  Each test is written as a display of an
119interactive session, including the Python prompts, statements that would
120be typed by the user, and the output of those statements (including
121tracebacks, although only the exception msg needs to be retained then).
122The module in the test package is simply a wrapper that causes doctest
123to run over the tests in the module.  The test for the difflib module
124provides a convenient example::
125
126    import difflib
127    from test import test_support
128    test_support.run_doctest(difflib)
129
130If the test is successful, nothing is written to stdout (so you should not
131create a corresponding output/test_difflib file), but running regrtest
132with -v will give a detailed report, the same as if passing -v to doctest.
133
134A second argument can be passed to run_doctest to tell doctest to search
135``sys.argv`` for -v instead of using test_support's idea of verbosity.  This
136is useful for writing doctest-based tests that aren't simply running a
137doctest'ed Lib module, but contain the doctests themselves.  Then at
138times you may want to run such a test directly as a doctest, independent
139of the regrtest framework.  The tail end of test_descrtut.py is a good
140example::
141
142    def test_main(verbose=None):
143        from test import test_support, test_descrtut
144        test_support.run_doctest(test_descrtut, verbose)
145
146    if __name__ == "__main__":
147        test_main(1)
148
149If run via regrtest, ``test_main()`` is called (by regrtest) without
150specifying verbose, and then test_support's idea of verbosity is used.  But
151when run directly, ``test_main(1)`` is called, and then doctest's idea of
152verbosity is used.
153
154See the documentation for the doctest module for information on
155writing tests using the doctest framework.
156
157"traditional" Python test modules
158---------------------------------
159The mechanics of how the "traditional" test system operates are fairly
160straightforward.  When a test case is run, the output is compared with the
161expected output that is stored in .../Lib/test/output.  If the test runs to
162completion and the actual and expected outputs match, the test succeeds, if
163not, it fails.  If an ``ImportError`` or ``test_support.TestSkipped`` error
164is raised, the test is not run.
165
166Executing Test Cases
167====================
168If you are writing test cases for module spam, you need to create a file
169in .../Lib/test named test_spam.py.  In addition, if the tests are expected
170to write to stdout during a successful run, you also need to create an
171expected output file in .../Lib/test/output named test_spam ("..."
172represents the top-level directory in the Python source tree, the directory
173containing the configure script).  If needed, generate the initial version
174of the test output file by executing::
175
176    ./python Lib/test/regrtest.py -g test_spam.py
177
178from the top-level directory.
179
180Any time you modify test_spam.py you need to generate a new expected
181output file.  Don't forget to desk check the generated output to make sure
182it's really what you expected to find!  All in all it's usually better
183not to have an expected-out file (note that doctest- and unittest-based
184tests do not).
185
186To run a single test after modifying a module, simply run regrtest.py
187without the -g flag::
188
189    ./python Lib/test/regrtest.py test_spam.py
190
191While debugging a regression test, you can of course execute it
192independently of the regression testing framework and see what it prints::
193
194    ./python Lib/test/test_spam.py
195
196To run the entire test suite:
197
198- [UNIX, + other platforms where "make" works] Make the "test" target at the
199  top level::
200
201    make test
202
203- [WINDOWS] Run rt.bat from your PCBuild directory.  Read the comments at
204  the top of rt.bat for the use of special -d, -O and -q options processed
205  by rt.bat.
206
207- [OTHER] You can simply execute the two runs of regrtest (optimized and
208  non-optimized) directly::
209
210    ./python Lib/test/regrtest.py
211    ./python -O Lib/test/regrtest.py
212
213But note that this way picks up whatever .pyc and .pyo files happen to be
214around.  The makefile and rt.bat ways run the tests twice, the first time
215removing all .pyc and .pyo files from the subtree rooted at Lib/.
216
217Test cases generate output based upon values computed by the test code.
218When executed, regrtest.py compares the actual output generated by executing
219the test case with the expected output and reports success or failure.  It
220stands to reason that if the actual and expected outputs are to match, they
221must not contain any machine dependencies.  This means your test cases
222should not print out absolute machine addresses (e.g. the return value of
223the id() builtin function) or floating point numbers with large numbers of
224significant digits (unless you understand what you are doing!).
225
226
227Test Case Writing Tips
228======================
229Writing good test cases is a skilled task and is too complex to discuss in
230detail in this short document.  Many books have been written on the subject.
231I'll show my age by suggesting that Glenford Myers' `"The Art of Software
232Testing"`_, published in 1979, is still the best introduction to the subject
233available.  It is short (177 pages), easy to read, and discusses the major
234elements of software testing, though its publication predates the
235object-oriented software revolution, so doesn't cover that subject at all.
236Unfortunately, it is very expensive (about $100 new).  If you can borrow it
237or find it used (around $20), I strongly urge you to pick up a copy.
238
239The most important goal when writing test cases is to break things.  A test
240case that doesn't uncover a bug is much less valuable than one that does.
241In designing test cases you should pay attention to the following:
242
243    * Your test cases should exercise all the functions and objects defined
244      in the module, not just the ones meant to be called by users of your
245      module.  This may require you to write test code that uses the module
246      in ways you don't expect (explicitly calling internal functions, for
247      example - see test_atexit.py).
248
249    * You should consider any boundary values that may tickle exceptional
250      conditions (e.g. if you were writing regression tests for division,
251      you might well want to generate tests with numerators and denominators
252      at the limits of floating point and integer numbers on the machine
253      performing the tests as well as a denominator of zero).
254
255    * You should exercise as many paths through the code as possible.  This
256      may not always be possible, but is a goal to strive for.  In
257      particular, when considering if statements (or their equivalent), you
258      want to create test cases that exercise both the true and false
259      branches.  For loops, you should create test cases that exercise the
260      loop zero, one and multiple times.
261
262    * You should test with obviously invalid input.  If you know that a
263      function requires an integer input, try calling it with other types of
264      objects to see how it responds.
265
266    * You should test with obviously out-of-range input.  If the domain of a
267      function is only defined for positive integers, try calling it with a
268      negative integer.
269
270    * If you are going to fix a bug that wasn't uncovered by an existing
271      test, try to write a test case that exposes the bug (preferably before
272      fixing it).
273
274    * If you need to create a temporary file, you can use the filename in
275      ``test_support.TESTFN`` to do so.  It is important to remove the file
276      when done; other tests should be able to use the name without cleaning
277      up after your test.
278
279.. _"The Art of Software Testing": 
280        http://www.amazon.com/exec/obidos/ISBN=0471043281
281
282Regression Test Writing Rules
283=============================
284Each test case is different.  There is no "standard" form for a Python
285regression test case, though there are some general rules (note that
286these mostly apply only to the "classic" tests; unittest_- and doctest_-
287based tests should follow the conventions natural to those frameworks)::
288
289    * If your test case detects a failure, raise ``TestFailed`` (found in
290      ``test.test_support``).
291
292    * Import everything you'll need as early as possible.
293
294    * If you'll be importing objects from a module that is at least
295      partially platform-dependent, only import those objects you need for
296      the current test case to avoid spurious ``ImportError`` exceptions
297      that prevent the test from running to completion.
298
299    * Print all your test case results using the ``print`` statement.  For
300      non-fatal errors, print an error message (or omit a successful
301      completion print) to indicate the failure, but proceed instead of
302      raising ``TestFailed``.
303
304    * Use ``assert`` sparingly, if at all.  It's usually better to just print
305      what you got, and rely on regrtest's got-vs-expected comparison to
306      catch deviations from what you expect.  ``assert`` statements aren't
307      executed at all when regrtest is run in -O mode; and, because they
308      cause the test to stop immediately, can lead to a long & tedious
309      test-fix, test-fix, test-fix, ... cycle when things are badly broken
310      (and note that "badly broken" often includes running the test suite
311      for the first time on new platforms or under new implementations of
312      the language).
313
314Miscellaneous
315=============
316There is a test_support module in the test package you can import for
317your test case.  Import this module using either::
318
319    import test.test_support
320
321or::
322
323    from test import test_support
324
325test_support provides the following useful objects:
326
327    * ``TestFailed`` - raise this exception when your regression test detects
328      a failure.
329
330    * ``TestSkipped`` - raise this if the test could not be run because the
331      platform doesn't offer all the required facilities (like large
332      file support), even if all the required modules are available.
333
334    * ``ResourceDenied`` - this is raised when a test requires a resource that
335      is not available.  Primarily used by 'requires'.
336
337    * ``verbose`` - you can use this variable to control print output.  Many
338      modules use it.  Search for "verbose" in the test_*.py files to see
339      lots of examples.
340
341    * ``forget(module_name)`` - attempts to cause Python to "forget" that it
342      loaded a module and erase any PYC files.
343
344    * ``is_resource_enabled(resource)`` - Returns a boolean based on whether
345      the resource is enabled or not.
346
347    * ``requires(resource [, msg])`` - if the required resource is not
348      available the ResourceDenied exception is raised.
349    
350    * ``verify(condition, reason='test failed')``.  Use this instead of::
351
352          assert condition[, reason]
353
354      ``verify()`` has two advantages over ``assert``:  it works even in -O
355      mode, and it raises ``TestFailed`` on failure instead of
356      ``AssertionError``.
357
358    * ``have_unicode`` - true if Unicode is available, false otherwise.
359
360    * ``is_jython`` - true if the interpreter is Jython, false otherwise.
361
362    * ``TESTFN`` - a string that should always be used as the filename when
363      you need to create a temp file.  Also use ``try``/``finally`` to
364      ensure that your temp files are deleted before your test completes.
365      Note that you cannot unlink an open file on all operating systems, so
366      also be sure to close temp files before trying to unlink them.
367
368    * ``sortdict(dict)`` - acts like ``repr(dict.items())``, but sorts the
369      items first.  This is important when printing a dict value, because
370      the order of items produced by ``dict.items()`` is not defined by the
371      language.
372
373    * ``findfile(file)`` - you can call this function to locate a file
374      somewhere along sys.path or in the Lib/test tree - see
375      test_linuxaudiodev.py for an example of its use.
376
377    * ``fcmp(x,y)`` - you can call this function to compare two floating
378      point numbers when you expect them to only be approximately equal
379      withing a fuzz factor (``test_support.FUZZ``, which defaults to 1e-6).
380
381    * ``check_syntax_error(testcase, statement)`` - make sure that the
382      statement is *not* correct Python syntax.
383
384
385Some Non-Obvious regrtest Features
386==================================
387    * Automagic test detection:  When you create a new test file
388      test_spam.py, you do not need to modify regrtest (or anything else)
389      to advertise its existence.  regrtest searches for and runs all
390      modules in the test directory with names of the form test_xxx.py.
391
392    * Miranda output:  If, when running test_spam.py, regrtest does not
393      find an expected-output file test/output/test_spam, regrtest
394      pretends that it did find one, containing the single line
395
396      test_spam
397
398      This allows new tests that don't expect to print anything to stdout
399      to not bother creating expected-output files.
400
401    * Two-stage testing:  To run test_spam.py, regrtest imports test_spam
402      as a module.  Most tests run to completion as a side-effect of
403      getting imported.  After importing test_spam, regrtest also executes
404      ``test_spam.test_main()``, if test_spam has a ``test_main`` attribute.
405      This is rarely required with the "traditional" Python tests, and
406      you shouldn't create a module global with name test_main unless
407      you're specifically exploiting this gimmick.  This usage does
408      prove useful with unittest-based tests as well, however; defining
409      a ``test_main()`` which is run by regrtest and a script-stub in the
410      test module ("``if __name__ == '__main__': test_main()``") allows
411      the test to be used like any other Python test and also work
412      with the unittest.py-as-a-script approach, allowing a developer
413      to run specific tests from the command line.