/Doc/library/test.rst

  1
  2:mod:`test` --- Regression tests package for Python
  3===================================================
  4
  5.. module:: test
  6   :synopsis: Regression tests package containing the testing suite for Python.
  7.. sectionauthor:: Brett Cannon <brett@python.org>
  8
  9
 10The :mod:`test` package contains all regression tests for Python as well as the
 11modules :mod:`test.test_support` and :mod:`test.regrtest`.
 12:mod:`test.test_support` is used to enhance your tests while
 13:mod:`test.regrtest` drives the testing suite.
 14
 15Each module in the :mod:`test` package whose name starts with ``test_`` is a
 16testing suite for a specific module or feature. All new tests should be written
 17using the :mod:`unittest` or :mod:`doctest` module.  Some older tests are
 18written using a "traditional" testing style that compares output printed to
 19``sys.stdout``; this style of test is considered deprecated.
 20
 21
 22.. seealso::
 23
 24   Module :mod:`unittest`
 25      Writing PyUnit regression tests.
 26
 27   Module :mod:`doctest`
 28      Tests embedded in documentation strings.
 29
 30
 31.. _writing-tests:
 32
 33Writing Unit Tests for the :mod:`test` package
 34----------------------------------------------
 35
 36It is preferred that tests that use the :mod:`unittest` module follow a few
 37guidelines. One is to name the test module by starting it with ``test_`` and end
 38it with the name of the module being tested. The test methods in the test module
 39should start with ``test_`` and end with a description of what the method is
 40testing. This is needed so that the methods are recognized by the test driver as
 41test methods. Also, no documentation string for the method should be included. A
 42comment (such as ``# Tests function returns only True or False``) should be used
 43to provide documentation for test methods. This is done because documentation
 44strings get printed out if they exist and thus what test is being run is not
 45stated.
 46
 47A basic boilerplate is often used::
 48
 49   import unittest
 50   from test import test_support
 51
 52   class MyTestCase1(unittest.TestCase):
 53
 54       # Only use setUp() and tearDown() if necessary
 55
 56       def setUp(self):
 57           ... code to execute in preparation for tests ...
 58
 59       def tearDown(self):
 60           ... code to execute to clean up after tests ...
 61
 62       def test_feature_one(self):
 63           # Test feature one.
 64           ... testing code ...
 65
 66       def test_feature_two(self):
 67           # Test feature two.
 68           ... testing code ...
 69
 70       ... more test methods ...
 71
 72   class MyTestCase2(unittest.TestCase):
 73       ... same structure as MyTestCase1 ...
 74
 75   ... more test classes ...
 76
 77   def test_main():
 78       test_support.run_unittest(MyTestCase1,
 79                                 MyTestCase2,
 80                                 ... list other tests ...
 81                                )
 82
 83   if __name__ == '__main__':
 84       test_main()
 85
 86This boilerplate code allows the testing suite to be run by :mod:`test.regrtest`
 87as well as on its own as a script.
 88
 89The goal for regression testing is to try to break code. This leads to a few
 90guidelines to be followed:
 91
 92* The testing suite should exercise all classes, functions, and constants. This
 93  includes not just the external API that is to be presented to the outside world
 94  but also "private" code.
 95
 96* Whitebox testing (examining the code being tested when the tests are being
 97  written) is preferred. Blackbox testing (testing only the published user
 98  interface) is not complete enough to make sure all boundary and edge cases are
 99  tested.
100
101* Make sure all possible values are tested including invalid ones. This makes
102  sure that not only all valid values are acceptable but also that improper values
103  are handled correctly.
104
105* Exhaust as many code paths as possible. Test where branching occurs and thus
106  tailor input to make sure as many different paths through the code are taken.
107
108* Add an explicit test for any bugs discovered for the tested code. This will
109  make sure that the error does not crop up again if the code is changed in the
110  future.
111
112* Make sure to clean up after your tests (such as close and remove all temporary
113  files).
114
115* If a test is dependent on a specific condition of the operating system then
116  verify the condition already exists before attempting the test.
117
118* Import as few modules as possible and do it as soon as possible. This
119  minimizes external dependencies of tests and also minimizes possible anomalous
120  behavior from side-effects of importing a module.
121
122* Try to maximize code reuse. On occasion, tests will vary by something as small
123  as what type of input is used. Minimize code duplication by subclassing a basic
124  test class with a class that specifies the input::
125
126     class TestFuncAcceptsSequences(unittest.TestCase):
127
128         func = mySuperWhammyFunction
129
130         def test_func(self):
131             self.func(self.arg)
132
133     class AcceptLists(TestFuncAcceptsSequences):
134         arg = [1,2,3]
135
136     class AcceptStrings(TestFuncAcceptsSequences):
137         arg = 'abc'
138
139     class AcceptTuples(TestFuncAcceptsSequences):
140         arg = (1,2,3)
141
142
143.. seealso::
144
145   Test Driven Development
146      A book by Kent Beck on writing tests before code.
147
148
149.. _regrtest:
150
151Running tests using :mod:`test.regrtest`
152----------------------------------------
153
154:mod:`test.regrtest` can be used as a script to drive Python's regression test
155suite. Running the script by itself automatically starts running all regression
156tests in the :mod:`test` package. It does this by finding all modules in the
157package whose name starts with ``test_``, importing them, and executing the
158function :func:`test_main` if present. The names of tests to execute may also be
159passed to the script. Specifying a single regression test (:program:`python
160regrtest.py` :option:`test_spam.py`) will minimize output and only print whether
161the test passed or failed and thus minimize output.
162
163Running :mod:`test.regrtest` directly allows what resources are available for
164tests to use to be set. You do this by using the :option:`-u` command-line
165option. Run :program:`python regrtest.py` :option:`-uall` to turn on all
166resources; specifying :option:`all` as an option for :option:`-u` enables all
167possible resources. If all but one resource is desired (a more common case), a
168comma-separated list of resources that are not desired may be listed after
169:option:`all`. The command :program:`python regrtest.py`
170:option:`-uall,-audio,-largefile` will run :mod:`test.regrtest` with all
171resources except the :option:`audio` and :option:`largefile` resources. For a
172list of all resources and more command-line options, run :program:`python
173regrtest.py` :option:`-h`.
174
175Some other ways to execute the regression tests depend on what platform the
176tests are being executed on. On Unix, you can run :program:`make` :option:`test`
177at the top-level directory where Python was built. On Windows, executing
178:program:`rt.bat` from your :file:`PCBuild` directory will run all regression
179tests.
180
181
182:mod:`test.test_support` --- Utility functions for tests
183========================================================
184
185.. module:: test.test_support
186   :synopsis: Support for Python regression tests.
187
188.. note::
189
190   The :mod:`test.test_support` module has been renamed to :mod:`test.support`
191   in Python 3.0.  The :term:`2to3` tool will automatically adapt imports when
192   converting your sources to 3.0.
193
194
195
196
197The :mod:`test.test_support` module provides support for Python's regression
198tests.
199
200This module defines the following exceptions:
201
202
203.. exception:: TestFailed
204
205   Exception to be raised when a test fails. This is deprecated in favor of
206   :mod:`unittest`\ -based tests and :class:`unittest.TestCase`'s assertion
207   methods.
208
209
210.. exception:: TestSkipped
211
212   Subclass of :exc:`TestFailed`. Raised when a test is skipped. This occurs when a
213   needed resource (such as a network connection) is not available at the time of
214   testing.
215
216
217.. exception:: ResourceDenied
218
219   Subclass of :exc:`TestSkipped`. Raised when a resource (such as a network
220   connection) is not available. Raised by the :func:`requires` function.
221
222The :mod:`test.test_support` module defines the following constants:
223
224
225.. data:: verbose
226
227   :const:`True` when verbose output is enabled. Should be checked when more
228   detailed information is desired about a running test. *verbose* is set by
229   :mod:`test.regrtest`.
230
231
232.. data:: have_unicode
233
234   :const:`True` when Unicode support is available.
235
236
237.. data:: is_jython
238
239   :const:`True` if the running interpreter is Jython.
240
241
242.. data:: TESTFN
243
244   Set to the path that a temporary file may be created at. Any temporary that is
245   created should be closed and unlinked (removed).
246
247The :mod:`test.test_support` module defines the following functions:
248
249
250.. function:: forget(module_name)
251
252   Removes the module named *module_name* from ``sys.modules`` and deletes any
253   byte-compiled files of the module.
254
255
256.. function:: is_resource_enabled(resource)
257
258   Returns :const:`True` if *resource* is enabled and available. The list of
259   available resources is only set when :mod:`test.regrtest` is executing the
260   tests.
261
262
263.. function:: requires(resource[, msg])
264
265   Raises :exc:`ResourceDenied` if *resource* is not available. *msg* is the
266   argument to :exc:`ResourceDenied` if it is raised. Always returns true if called
267   by a function whose ``__name__`` is ``'__main__'``. Used when tests are executed
268   by :mod:`test.regrtest`.
269
270
271.. function:: findfile(filename)
272
273   Return the path to the file named *filename*. If no match is found *filename* is
274   returned. This does not equal a failure since it could be the path to the file.
275
276
277.. function:: run_unittest(*classes)
278
279   Execute :class:`unittest.TestCase` subclasses passed to the function. The
280   function scans the classes for methods starting with the prefix ``test_`` and
281   executes the tests individually.
282
283   It is also legal to pass strings as parameters; these should be keys in
284   ``sys.modules``. Each associated module will be scanned by
285   ``unittest.TestLoader.loadTestsFromModule()``. This is usually seen in the
286   following :func:`test_main` function::
287
288      def test_main():
289          test_support.run_unittest(__name__)
290
291   This will run all tests defined in the named module.
292
293
294.. function:: check_warnings()
295
296   A convenience wrapper for ``warnings.catch_warnings()`` that makes
297   it easier to test that a warning was correctly raised with a single
298   assertion. It is approximately equivalent to calling
299   ``warnings.catch_warnings(record=True)``.
300
301   The main difference is that on entry to the context manager, a
302   :class:`WarningRecorder` instance is returned instead of a simple list.
303   The underlying warnings list is available via the recorder object's
304   :attr:`warnings` attribute, while the attributes of the last raised
305   warning are also accessible directly on the object. If no warning has
306   been raised, then the latter attributes will all be :const:`None`.
307
308   A :meth:`reset` method is also provided on the recorder object. This
309   method simply clears the warning list.
310
311   The context manager is used like this::
312
313      with check_warnings() as w:
314          warnings.simplefilter("always")
315          warnings.warn("foo")
316          assert str(w.message) == "foo"
317          warnings.warn("bar")
318          assert str(w.message) == "bar"
319          assert str(w.warnings[0].message) == "foo"
320          assert str(w.warnings[1].message) == "bar"
321          w.reset()
322          assert len(w.warnings) == 0
323
324   .. versionadded:: 2.6
325
326
327.. function:: captured_stdout()
328
329   This is a context manager than runs the :keyword:`with` statement body using
330   a :class:`StringIO.StringIO` object as sys.stdout.  That object can be
331   retrieved using the ``as`` clause of the :keyword:`with` statement.
332
333   Example use::
334
335      with captured_stdout() as s:
336          print "hello"
337      assert s.getvalue() == "hello"
338
339   .. versionadded:: 2.6
340
341
342The :mod:`test.test_support` module defines the following classes:
343
344.. class:: TransientResource(exc[, **kwargs])
345
346   Instances are a context manager that raises :exc:`ResourceDenied` if the
347   specified exception type is raised.  Any keyword arguments are treated as
348   attribute/value pairs to be compared against any exception raised within the
349   :keyword:`with` statement.  Only if all pairs match properly against
350   attributes on the exception is :exc:`ResourceDenied` raised.
351
352   .. versionadded:: 2.6
353.. class:: EnvironmentVarGuard()
354
355   Class used to temporarily set or unset environment variables.  Instances can be
356   used as a context manager.
357
358   .. versionadded:: 2.6
359
360
361.. method:: EnvironmentVarGuard.set(envvar, value)
362
363   Temporarily set the environment variable ``envvar`` to the value of ``value``.
364
365
366.. method:: EnvironmentVarGuard.unset(envvar)
367
368   Temporarily unset the environment variable ``envvar``.
369
370.. class:: WarningsRecorder()
371
372   Class used to record warnings for unit tests. See documentation of
373   :func:`check_warnings` above for more details.
374
375   .. versionadded:: 2.6
376