/Doc/library/unittest.rst

  1
  2:mod:`unittest` --- Unit testing framework
  3==========================================
  4
  5.. module:: unittest
  6   :synopsis: Unit testing framework for Python.
  7.. moduleauthor:: Steve Purcell <stephen_purcell@yahoo.com>
  8.. sectionauthor:: Steve Purcell <stephen_purcell@yahoo.com>
  9.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>
 10.. sectionauthor:: Raymond Hettinger <python@rcn.com>
 11
 12
 13.. versionadded:: 2.1
 14
 15The Python unit testing framework, sometimes referred to as "PyUnit," is a
 16Python language version of JUnit, by Kent Beck and Erich Gamma. JUnit is, in
 17turn, a Java version of Kent's Smalltalk testing framework.  Each is the de
 18facto standard unit testing framework for its respective language.
 19
 20:mod:`unittest` supports test automation, sharing of setup and shutdown code for
 21tests, aggregation of tests into collections, and independence of the tests from
 22the reporting framework.  The :mod:`unittest` module provides classes that make
 23it easy to support these qualities for a set of tests.
 24
 25To achieve this, :mod:`unittest` supports some important concepts:
 26
 27test fixture
 28   A :dfn:`test fixture` represents the preparation needed to perform one or more
 29   tests, and any associate cleanup actions.  This may involve, for example,
 30   creating temporary or proxy databases, directories, or starting a server
 31   process.
 32
 33test case
 34   A :dfn:`test case` is the smallest unit of testing.  It checks for a specific
 35   response to a particular set of inputs.  :mod:`unittest` provides a base class,
 36   :class:`TestCase`, which may be used to create new test cases.
 37
 38test suite
 39   A :dfn:`test suite` is a collection of test cases, test suites, or both.  It is
 40   used to aggregate tests that should be executed together.
 41
 42test runner
 43   A :dfn:`test runner` is a component which orchestrates the execution of tests
 44   and provides the outcome to the user.  The runner may use a graphical interface,
 45   a textual interface, or return a special value to indicate the results of
 46   executing the tests.
 47
 48The test case and test fixture concepts are supported through the
 49:class:`TestCase` and :class:`FunctionTestCase` classes; the former should be
 50used when creating new tests, and the latter can be used when integrating
 51existing test code with a :mod:`unittest`\ -driven framework. When building test
 52fixtures using :class:`TestCase`, the :meth:`setUp` and :meth:`tearDown` methods
 53can be overridden to provide initialization and cleanup for the fixture.  With
 54:class:`FunctionTestCase`, existing functions can be passed to the constructor
 55for these purposes.  When the test is run, the fixture initialization is run
 56first; if it succeeds, the cleanup method is run after the test has been
 57executed, regardless of the outcome of the test.  Each instance of the
 58:class:`TestCase` will only be used to run a single test method, so a new
 59fixture is created for each test.
 60
 61Test suites are implemented by the :class:`TestSuite` class.  This class allows
 62individual tests and test suites to be aggregated; when the suite is executed,
 63all tests added directly to the suite and in "child" test suites are run.
 64
 65A test runner is an object that provides a single method, :meth:`run`, which
 66accepts a :class:`TestCase` or :class:`TestSuite` object as a parameter, and
 67returns a result object.  The class :class:`TestResult` is provided for use as
 68the result object. :mod:`unittest` provides the :class:`TextTestRunner` as an
 69example test runner which reports test results on the standard error stream by
 70default.  Alternate runners can be implemented for other environments (such as
 71graphical environments) without any need to derive from a specific class.
 72
 73
 74.. seealso::
 75
 76   Module :mod:`doctest`
 77      Another test-support module with a very different flavor.
 78
 79   `Simple Smalltalk Testing: With Patterns <http://www.XProgramming.com/testfram.htm>`_
 80      Kent Beck's original paper on testing frameworks using the pattern shared by
 81      :mod:`unittest`.
 82
 83   `Nose <http://code.google.com/p/python-nose/>`_ and `py.test <http://pytest.org>`_
 84      Third-party unittest frameworks with a lighter-weight syntax
 85      for writing tests.  For example, ``assert func(10) == 42``.
 86
 87   `python-mock <http://python-mock.sourceforge.net/>`_ and `minimock <http://blog.ianbicking.org/minimock.html>`_
 88      Tools for creating mock test objects (objects simulating external resources).
 89
 90.. _unittest-minimal-example:
 91
 92Basic example
 93-------------
 94
 95The :mod:`unittest` module provides a rich set of tools for constructing and
 96running tests.  This section demonstrates that a small subset of the tools
 97suffice to meet the needs of most users.
 98
 99Here is a short script to test three functions from the :mod:`random` module::
100
101   import random
102   import unittest
103
104   class TestSequenceFunctions(unittest.TestCase):
105
106       def setUp(self):
107           self.seq = range(10)
108
109       def testshuffle(self):
110           # make sure the shuffled sequence does not lose any elements
111           random.shuffle(self.seq)
112           self.seq.sort()
113           self.assertEqual(self.seq, range(10))
114
115       def testchoice(self):
116           element = random.choice(self.seq)
117           self.assert_(element in self.seq)
118
119       def testsample(self):
120           self.assertRaises(ValueError, random.sample, self.seq, 20)
121           for element in random.sample(self.seq, 5):
122               self.assert_(element in self.seq)
123
124   if __name__ == '__main__':
125       unittest.main()
126
127A testcase is created by subclassing :class:`unittest.TestCase`. The three
128individual tests are defined with methods whose names start with the letters
129``test``.  This naming convention informs the test runner about which methods
130represent tests.
131
132The crux of each test is a call to :meth:`assertEqual` to check for an expected
133result; :meth:`assert_` to verify a condition; or :meth:`assertRaises` to verify
134that an expected exception gets raised.  These methods are used instead of the
135:keyword:`assert` statement so the test runner can accumulate all test results
136and produce a report.
137
138When a :meth:`setUp` method is defined, the test runner will run that method
139prior to each test.  Likewise, if a :meth:`tearDown` method is defined, the test
140runner will invoke that method after each test.  In the example, :meth:`setUp`
141was used to create a fresh sequence for each test.
142
143The final block shows a simple way to run the tests. :func:`unittest.main`
144provides a command line interface to the test script.  When run from the command
145line, the above script produces an output that looks like this::
146
147   ...
148   ----------------------------------------------------------------------
149   Ran 3 tests in 0.000s
150
151   OK
152
153Instead of :func:`unittest.main`, there are other ways to run the tests with a
154finer level of control, less terse output, and no requirement to be run from the
155command line.  For example, the last two lines may be replaced with::
156
157   suite = unittest.TestLoader().loadTestsFromTestCase(TestSequenceFunctions)
158   unittest.TextTestRunner(verbosity=2).run(suite)
159
160Running the revised script from the interpreter or another script produces the
161following output::
162
163   testchoice (__main__.TestSequenceFunctions) ... ok
164   testsample (__main__.TestSequenceFunctions) ... ok
165   testshuffle (__main__.TestSequenceFunctions) ... ok
166
167   ----------------------------------------------------------------------
168   Ran 3 tests in 0.110s
169
170   OK
171
172The above examples show the most commonly used :mod:`unittest` features which
173are sufficient to meet many everyday testing needs.  The remainder of the
174documentation explores the full feature set from first principles.
175
176
177.. _organizing-tests:
178
179Organizing test code
180--------------------
181
182The basic building blocks of unit testing are :dfn:`test cases` --- single
183scenarios that must be set up and checked for correctness.  In :mod:`unittest`,
184test cases are represented by instances of :mod:`unittest`'s :class:`TestCase`
185class. To make your own test cases you must write subclasses of
186:class:`TestCase`, or use :class:`FunctionTestCase`.
187
188An instance of a :class:`TestCase`\ -derived class is an object that can
189completely run a single test method, together with optional set-up and tidy-up
190code.
191
192The testing code of a :class:`TestCase` instance should be entirely self
193contained, such that it can be run either in isolation or in arbitrary
194combination with any number of other test cases.
195
196The simplest :class:`TestCase` subclass will simply override the :meth:`runTest`
197method in order to perform specific testing code::
198
199   import unittest
200
201   class DefaultWidgetSizeTestCase(unittest.TestCase):
202       def runTest(self):
203           widget = Widget('The widget')
204           self.assertEqual(widget.size(), (50, 50), 'incorrect default size')
205
206Note that in order to test something, we use the one of the :meth:`assert\*` or
207:meth:`fail\*` methods provided by the :class:`TestCase` base class.  If the
208test fails, an exception will be raised, and :mod:`unittest` will identify the
209test case as a :dfn:`failure`.  Any other exceptions will be treated as
210:dfn:`errors`. This helps you identify where the problem is: :dfn:`failures` are
211caused by incorrect results - a 5 where you expected a 6. :dfn:`Errors` are
212caused by incorrect code - e.g., a :exc:`TypeError` caused by an incorrect
213function call.
214
215The way to run a test case will be described later.  For now, note that to
216construct an instance of such a test case, we call its constructor without
217arguments::
218
219   testCase = DefaultWidgetSizeTestCase()
220
221Now, such test cases can be numerous, and their set-up can be repetitive.  In
222the above case, constructing a :class:`Widget` in each of 100 Widget test case
223subclasses would mean unsightly duplication.
224
225Luckily, we can factor out such set-up code by implementing a method called
226:meth:`setUp`, which the testing framework will automatically call for us when
227we run the test::
228
229   import unittest
230
231   class SimpleWidgetTestCase(unittest.TestCase):
232       def setUp(self):
233           self.widget = Widget('The widget')
234
235   class DefaultWidgetSizeTestCase(SimpleWidgetTestCase):
236       def runTest(self):
237           self.failUnless(self.widget.size() == (50,50),
238                           'incorrect default size')
239
240   class WidgetResizeTestCase(SimpleWidgetTestCase):
241       def runTest(self):
242           self.widget.resize(100,150)
243           self.failUnless(self.widget.size() == (100,150),
244                           'wrong size after resize')
245
246If the :meth:`setUp` method raises an exception while the test is running, the
247framework will consider the test to have suffered an error, and the
248:meth:`runTest` method will not be executed.
249
250Similarly, we can provide a :meth:`tearDown` method that tidies up after the
251:meth:`runTest` method has been run::
252
253   import unittest
254
255   class SimpleWidgetTestCase(unittest.TestCase):
256       def setUp(self):
257           self.widget = Widget('The widget')
258
259       def tearDown(self):
260           self.widget.dispose()
261           self.widget = None
262
263If :meth:`setUp` succeeded, the :meth:`tearDown` method will be run whether
264:meth:`runTest` succeeded or not.
265
266Such a working environment for the testing code is called a :dfn:`fixture`.
267
268Often, many small test cases will use the same fixture.  In this case, we would
269end up subclassing :class:`SimpleWidgetTestCase` into many small one-method
270classes such as :class:`DefaultWidgetSizeTestCase`.  This is time-consuming and
271discouraging, so in the same vein as JUnit, :mod:`unittest` provides a simpler
272mechanism::
273
274   import unittest
275
276   class WidgetTestCase(unittest.TestCase):
277       def setUp(self):
278           self.widget = Widget('The widget')
279
280       def tearDown(self):
281           self.widget.dispose()
282           self.widget = None
283
284       def testDefaultSize(self):
285           self.failUnless(self.widget.size() == (50,50),
286                           'incorrect default size')
287
288       def testResize(self):
289           self.widget.resize(100,150)
290           self.failUnless(self.widget.size() == (100,150),
291                           'wrong size after resize')
292
293Here we have not provided a :meth:`runTest` method, but have instead provided
294two different test methods.  Class instances will now each run one of the
295:meth:`test\*`  methods, with ``self.widget`` created and destroyed separately
296for each instance.  When creating an instance we must specify the test method it
297is to run.  We do this by passing the method name in the constructor::
298
299   defaultSizeTestCase = WidgetTestCase('testDefaultSize')
300   resizeTestCase = WidgetTestCase('testResize')
301
302Test case instances are grouped together according to the features they test.
303:mod:`unittest` provides a mechanism for this: the :dfn:`test suite`,
304represented by :mod:`unittest`'s :class:`TestSuite` class::
305
306   widgetTestSuite = unittest.TestSuite()
307   widgetTestSuite.addTest(WidgetTestCase('testDefaultSize'))
308   widgetTestSuite.addTest(WidgetTestCase('testResize'))
309
310For the ease of running tests, as we will see later, it is a good idea to
311provide in each test module a callable object that returns a pre-built test
312suite::
313
314   def suite():
315       suite = unittest.TestSuite()
316       suite.addTest(WidgetTestCase('testDefaultSize'))
317       suite.addTest(WidgetTestCase('testResize'))
318       return suite
319
320or even::
321
322   def suite():
323       tests = ['testDefaultSize', 'testResize']
324
325       return unittest.TestSuite(map(WidgetTestCase, tests))
326
327Since it is a common pattern to create a :class:`TestCase` subclass with many
328similarly named test functions, :mod:`unittest` provides a :class:`TestLoader`
329class that can be used to automate the process of creating a test suite and
330populating it with individual tests. For example, ::
331
332   suite = unittest.TestLoader().loadTestsFromTestCase(WidgetTestCase)
333
334will create a test suite that will run ``WidgetTestCase.testDefaultSize()`` and
335``WidgetTestCase.testResize``. :class:`TestLoader` uses the ``'test'`` method
336name prefix to identify test methods automatically.
337
338Note that the order in which the various test cases will be run is determined by
339sorting the test function names with the built-in :func:`cmp` function.
340
341Often it is desirable to group suites of test cases together, so as to run tests
342for the whole system at once.  This is easy, since :class:`TestSuite` instances
343can be added to a :class:`TestSuite` just as :class:`TestCase` instances can be
344added to a :class:`TestSuite`::
345
346   suite1 = module1.TheTestSuite()
347   suite2 = module2.TheTestSuite()
348   alltests = unittest.TestSuite([suite1, suite2])
349
350You can place the definitions of test cases and test suites in the same modules
351as the code they are to test (such as :file:`widget.py`), but there are several
352advantages to placing the test code in a separate module, such as
353:file:`test_widget.py`:
354
355* The test module can be run standalone from the command line.
356
357* The test code can more easily be separated from shipped code.
358
359* There is less temptation to change test code to fit the code it tests without
360  a good reason.
361
362* Test code should be modified much less frequently than the code it tests.
363
364* Tested code can be refactored more easily.
365
366* Tests for modules written in C must be in separate modules anyway, so why not
367  be consistent?
368
369* If the testing strategy changes, there is no need to change the source code.
370
371
372.. _legacy-unit-tests:
373
374Re-using old test code
375----------------------
376
377Some users will find that they have existing test code that they would like to
378run from :mod:`unittest`, without converting every old test function to a
379:class:`TestCase` subclass.
380
381For this reason, :mod:`unittest` provides a :class:`FunctionTestCase` class.
382This subclass of :class:`TestCase` can be used to wrap an existing test
383function.  Set-up and tear-down functions can also be provided.
384
385Given the following test function::
386
387   def testSomething():
388       something = makeSomething()
389       assert something.name is not None
390       # ...
391
392one can create an equivalent test case instance as follows::
393
394   testcase = unittest.FunctionTestCase(testSomething)
395
396If there are additional set-up and tear-down methods that should be called as
397part of the test case's operation, they can also be provided like so::
398
399   testcase = unittest.FunctionTestCase(testSomething,
400                                        setUp=makeSomethingDB,
401                                        tearDown=deleteSomethingDB)
402
403To make migrating existing test suites easier, :mod:`unittest` supports tests
404raising :exc:`AssertionError` to indicate test failure. However, it is
405recommended that you use the explicit :meth:`TestCase.fail\*` and
406:meth:`TestCase.assert\*` methods instead, as future versions of :mod:`unittest`
407may treat :exc:`AssertionError` differently.
408
409.. note::
410
411   Even though :class:`FunctionTestCase` can be used to quickly convert an existing
412   test base over to a :mod:`unittest`\ -based system, this approach is not
413   recommended.  Taking the time to set up proper :class:`TestCase` subclasses will
414   make future test refactorings infinitely easier.
415
416
417.. _unittest-contents:
418
419Classes and functions
420---------------------
421
422
423.. class:: TestCase([methodName])
424
425   Instances of the :class:`TestCase` class represent the smallest testable units
426   in the :mod:`unittest` universe.  This class is intended to be used as a base
427   class, with specific tests being implemented by concrete subclasses.  This class
428   implements the interface needed by the test runner to allow it to drive the
429   test, and methods that the test code can use to check for and report various
430   kinds of failure.
431
432   Each instance of :class:`TestCase` will run a single test method: the method
433   named *methodName*.  If you remember, we had an earlier example that went
434   something like this::
435
436      def suite():
437          suite = unittest.TestSuite()
438          suite.addTest(WidgetTestCase('testDefaultSize'))
439          suite.addTest(WidgetTestCase('testResize'))
440          return suite
441
442   Here, we create two instances of :class:`WidgetTestCase`, each of which runs a
443   single test.
444
445   *methodName* defaults to ``'runTest'``.
446
447
448.. class:: FunctionTestCase(testFunc[, setUp[, tearDown[, description]]])
449
450   This class implements the portion of the :class:`TestCase` interface which
451   allows the test runner to drive the test, but does not provide the methods which
452   test code can use to check and report errors. This is used to create test cases
453   using legacy test code, allowing it to be integrated into a :mod:`unittest`\
454   -based test framework.
455
456
457.. class:: TestSuite([tests])
458
459   This class represents an aggregation of individual tests cases and test suites.
460   The class presents the interface needed by the test runner to allow it to be run
461   as any other test case.  Running a :class:`TestSuite` instance is the same as
462   iterating over the suite, running each test individually.
463
464   If *tests* is given, it must be an iterable of individual test cases or other
465   test suites that will be used to build the suite initially. Additional methods
466   are provided to add test cases and suites to the collection later on.
467
468
469.. class:: TestLoader()
470
471   This class is responsible for loading tests according to various criteria and
472   returning them wrapped in a :class:`TestSuite`. It can load all tests within a
473   given module or :class:`TestCase` subclass.
474
475
476.. class:: TestResult()
477
478   This class is used to compile information about which tests have succeeded and
479   which have failed.
480
481
482.. data:: defaultTestLoader
483
484   Instance of the :class:`TestLoader` class intended to be shared.  If no
485   customization of the :class:`TestLoader` is needed, this instance can be used
486   instead of repeatedly creating new instances.
487
488
489.. class:: TextTestRunner([stream[, descriptions[, verbosity]]])
490
491   A basic test runner implementation which prints results on standard error.  It
492   has a few configurable parameters, but is essentially very simple.  Graphical
493   applications which run test suites should provide alternate implementations.
494
495
496.. function:: main([module[, defaultTest[, argv[, testRunner[, testLoader]]]]])
497
498   A command-line program that runs a set of tests; this is primarily for making
499   test modules conveniently executable.  The simplest use for this function is to
500   include the following line at the end of a test script::
501
502      if __name__ == '__main__':
503          unittest.main()
504
505   The *testRunner* argument can either be a test runner class or an already
506   created instance of it.
507
508In some cases, the existing tests may have been written using the :mod:`doctest`
509module.  If so, that module provides a  :class:`DocTestSuite` class that can
510automatically build :class:`unittest.TestSuite` instances from the existing
511:mod:`doctest`\ -based tests.
512
513.. versionadded:: 2.3
514
515
516.. _testcase-objects:
517
518TestCase Objects
519----------------
520
521Each :class:`TestCase` instance represents a single test, but each concrete
522subclass may be used to define multiple tests --- the concrete class represents
523a single test fixture.  The fixture is created and cleaned up for each test
524case.
525
526:class:`TestCase` instances provide three groups of methods: one group used to
527run the test, another used by the test implementation to check conditions and
528report failures, and some inquiry methods allowing information about the test
529itself to be gathered.
530
531Methods in the first group (running the test) are:
532
533
534.. method:: TestCase.setUp()
535
536   Method called to prepare the test fixture.  This is called immediately before
537   calling the test method; any exception raised by this method will be considered
538   an error rather than a test failure. The default implementation does nothing.
539
540
541.. method:: TestCase.tearDown()
542
543   Method called immediately after the test method has been called and the result
544   recorded.  This is called even if the test method raised an exception, so the
545   implementation in subclasses may need to be particularly careful about checking
546   internal state.  Any exception raised by this method will be considered an error
547   rather than a test failure.  This method will only be called if the
548   :meth:`setUp` succeeds, regardless of the outcome of the test method. The
549   default implementation does nothing.
550
551
552.. method:: TestCase.run([result])
553
554   Run the test, collecting the result into the test result object passed as
555   *result*.  If *result* is omitted or :const:`None`, a temporary result object is
556   created (by calling the :meth:`defaultTestCase` method) and used; this result
557   object is not returned to :meth:`run`'s caller.
558
559   The same effect may be had by simply calling the :class:`TestCase` instance.
560
561
562.. method:: TestCase.debug()
563
564   Run the test without collecting the result.  This allows exceptions raised by
565   the test to be propagated to the caller, and can be used to support running
566   tests under a debugger.
567
568The test code can use any of the following methods to check for and report
569failures.
570
571
572.. method:: TestCase.assert_(expr[, msg])
573            TestCase.failUnless(expr[, msg])
574            TestCase.assertTrue(expr[, msg])
575
576   Signal a test failure if *expr* is false; the explanation for the error will be
577   *msg* if given, otherwise it will be :const:`None`.
578
579
580.. method:: TestCase.assertEqual(first, second[, msg])
581            TestCase.failUnlessEqual(first, second[, msg])
582
583   Test that *first* and *second* are equal.  If the values do not compare equal,
584   the test will fail with the explanation given by *msg*, or :const:`None`.  Note
585   that using :meth:`failUnlessEqual` improves upon doing the comparison as the
586   first parameter to :meth:`failUnless`:  the default value for *msg* can be
587   computed to include representations of both *first* and *second*.
588
589
590.. method:: TestCase.assertNotEqual(first, second[, msg])
591            TestCase.failIfEqual(first, second[, msg])
592
593   Test that *first* and *second* are not equal.  If the values do compare equal,
594   the test will fail with the explanation given by *msg*, or :const:`None`.  Note
595   that using :meth:`failIfEqual` improves upon doing the comparison as the first
596   parameter to :meth:`failUnless` is that the default value for *msg* can be
597   computed to include representations of both *first* and *second*.
598
599
600.. method:: TestCase.assertAlmostEqual(first, second[, places[, msg]])
601            TestCase.failUnlessAlmostEqual(first, second[, places[, msg]])
602
603   Test that *first* and *second* are approximately equal by computing the
604   difference, rounding to the given number of decimal *places* (default 7),
605   and comparing to zero.
606   Note that comparing a given number of decimal places is not the same as
607   comparing a given number of significant digits. If the values do not compare
608   equal, the test will fail with the explanation given by *msg*, or :const:`None`.
609
610
611.. method:: TestCase.assertNotAlmostEqual(first, second[, places[, msg]])
612            TestCase.failIfAlmostEqual(first, second[, places[, msg]])
613
614   Test that *first* and *second* are not approximately equal by computing the
615   difference, rounding to the given number of decimal *places* (default 7),
616   and comparing to zero.
617   Note that comparing a given number of decimal places is not the same as
618   comparing a given number of significant digits. If the values do not compare
619   equal, the test will fail with the explanation given by *msg*, or :const:`None`.
620
621
622.. method:: TestCase.assertRaises(exception, callable, ...)
623            TestCase.failUnlessRaises(exception, callable, ...)
624
625   Test that an exception is raised when *callable* is called with any positional
626   or keyword arguments that are also passed to :meth:`assertRaises`.  The test
627   passes if *exception* is raised, is an error if another exception is raised, or
628   fails if no exception is raised.  To catch any of a group of exceptions, a tuple
629   containing the exception classes may be passed as *exception*.
630
631
632.. method:: TestCase.failIf(expr[, msg])
633            TestCase.assertFalse(expr[, msg])
634
635   The inverse of the :meth:`failUnless` method is the :meth:`failIf` method.  This
636   signals a test failure if *expr* is true, with *msg* or :const:`None` for the
637   error message.
638
639
640.. method:: TestCase.fail([msg])
641
642   Signals a test failure unconditionally, with *msg* or :const:`None` for the
643   error message.
644
645
646.. attribute:: TestCase.failureException
647
648   This class attribute gives the exception raised by the :meth:`test` method.  If
649   a test framework needs to use a specialized exception, possibly to carry
650   additional information, it must subclass this exception in order to "play fair"
651   with the framework.  The initial value of this attribute is
652   :exc:`AssertionError`.
653
654Testing frameworks can use the following methods to collect information on the
655test:
656
657
658.. method:: TestCase.countTestCases()
659
660   Return the number of tests represented by this test object.  For
661   :class:`TestCase` instances, this will always be ``1``.
662
663
664.. method:: TestCase.defaultTestResult()
665
666   Return an instance of the test result class that should be used for this test
667   case class (if no other result instance is provided to the :meth:`run` method).
668
669   For :class:`TestCase` instances, this will always be an instance of
670   :class:`TestResult`;  subclasses of :class:`TestCase` should override this as
671   necessary.
672
673
674.. method:: TestCase.id()
675
676   Return a string identifying the specific test case.  This is usually the full
677   name of the test method, including the module and class name.
678
679
680.. method:: TestCase.shortDescription()
681
682   Returns a one-line description of the test, or :const:`None` if no description
683   has been provided.  The default implementation of this method returns the first
684   line of the test method's docstring, if available, or :const:`None`.
685
686
687.. _testsuite-objects:
688
689TestSuite Objects
690-----------------
691
692:class:`TestSuite` objects behave much like :class:`TestCase` objects, except
693they do not actually implement a test.  Instead, they are used to aggregate
694tests into groups of tests that should be run together. Some additional methods
695are available to add tests to :class:`TestSuite` instances:
696
697
698.. method:: TestSuite.addTest(test)
699
700   Add a :class:`TestCase` or :class:`TestSuite` to the suite.
701
702
703.. method:: TestSuite.addTests(tests)
704
705   Add all the tests from an iterable of :class:`TestCase` and :class:`TestSuite`
706   instances to this test suite.
707
708   This is equivalent to iterating over *tests*, calling :meth:`addTest` for each
709   element.
710
711:class:`TestSuite` shares the following methods with :class:`TestCase`:
712
713
714.. method:: TestSuite.run(result)
715
716   Run the tests associated with this suite, collecting the result into the test
717   result object passed as *result*.  Note that unlike :meth:`TestCase.run`,
718   :meth:`TestSuite.run` requires the result object to be passed in.
719
720
721.. method:: TestSuite.debug()
722
723   Run the tests associated with this suite without collecting the result. This
724   allows exceptions raised by the test to be propagated to the caller and can be
725   used to support running tests under a debugger.
726
727
728.. method:: TestSuite.countTestCases()
729
730   Return the number of tests represented by this test object, including all
731   individual tests and sub-suites.
732
733In the typical usage of a :class:`TestSuite` object, the :meth:`run` method is
734invoked by a :class:`TestRunner` rather than by the end-user test harness.
735
736
737.. _testresult-objects:
738
739TestResult Objects
740------------------
741
742A :class:`TestResult` object stores the results of a set of tests.  The
743:class:`TestCase` and :class:`TestSuite` classes ensure that results are
744properly recorded; test authors do not need to worry about recording the outcome
745of tests.
746
747Testing frameworks built on top of :mod:`unittest` may want access to the
748:class:`TestResult` object generated by running a set of tests for reporting
749purposes; a :class:`TestResult` instance is returned by the
750:meth:`TestRunner.run` method for this purpose.
751
752:class:`TestResult` instances have the following attributes that will be of
753interest when inspecting the results of running a set of tests:
754
755
756.. attribute:: TestResult.errors
757
758   A list containing 2-tuples of :class:`TestCase` instances and strings holding
759   formatted tracebacks. Each tuple represents a test which raised an unexpected
760   exception.
761
762   .. versionchanged:: 2.2
763      Contains formatted tracebacks instead of :func:`sys.exc_info` results.
764
765
766.. attribute:: TestResult.failures
767
768   A list containing 2-tuples of :class:`TestCase` instances and strings holding
769   formatted tracebacks. Each tuple represents a test where a failure was
770   explicitly signalled using the :meth:`TestCase.fail\*` or
771   :meth:`TestCase.assert\*` methods.
772
773   .. versionchanged:: 2.2
774      Contains formatted tracebacks instead of :func:`sys.exc_info` results.
775
776
777.. attribute:: TestResult.testsRun
778
779   The total number of tests run so far.
780
781
782.. method:: TestResult.wasSuccessful()
783
784   Returns :const:`True` if all tests run so far have passed, otherwise returns
785   :const:`False`.
786
787
788.. method:: TestResult.stop()
789
790   This method can be called to signal that the set of tests being run should be
791   aborted by setting the :class:`TestResult`'s ``shouldStop`` attribute to
792   :const:`True`.  :class:`TestRunner` objects should respect this flag and return
793   without running any additional tests.
794
795   For example, this feature is used by the :class:`TextTestRunner` class to stop
796   the test framework when the user signals an interrupt from the keyboard.
797   Interactive tools which provide :class:`TestRunner` implementations can use this
798   in a similar manner.
799
800The following methods of the :class:`TestResult` class are used to maintain the
801internal data structures, and may be extended in subclasses to support
802additional reporting requirements.  This is particularly useful in building
803tools which support interactive reporting while tests are being run.
804
805
806.. method:: TestResult.startTest(test)
807
808   Called when the test case *test* is about to be run.
809
810   The default implementation simply increments the instance's ``testsRun``
811   counter.
812
813
814.. method:: TestResult.stopTest(test)
815
816   Called after the test case *test* has been executed, regardless of the outcome.
817
818   The default implementation does nothing.
819
820
821.. method:: TestResult.addError(test, err)
822
823   Called when the test case *test* raises an unexpected exception *err* is a tuple
824   of the form returned by :func:`sys.exc_info`: ``(type, value, traceback)``.
825
826   The default implementation appends a tuple ``(test, formatted_err)`` to the
827   instance's ``errors`` attribute, where *formatted_err* is a formatted
828   traceback derived from *err*.
829
830
831.. method:: TestResult.addFailure(test, err)
832
833   Called when the test case *test* signals a failure. *err* is a tuple of the form
834   returned by :func:`sys.exc_info`:  ``(type, value, traceback)``.
835
836   The default implementation appends a tuple ``(test, formatted_err)`` to the
837   instance's ``failures`` attribute, where *formatted_err* is a formatted
838   traceback derived from *err*.
839
840
841.. method:: TestResult.addSuccess(test)
842
843   Called when the test case *test* succeeds.
844
845   The default implementation does nothing.
846
847
848.. _testloader-objects:
849
850TestLoader Objects
851------------------
852
853The :class:`TestLoader` class is used to create test suites from classes and
854modules.  Normally, there is no need to create an instance of this class; the
855:mod:`unittest` module provides an instance that can be shared as
856``unittest.defaultTestLoader``. Using a subclass or instance, however, allows
857customization of some configurable properties.
858
859:class:`TestLoader` objects have the following methods:
860
861
862.. method:: TestLoader.loadTestsFromTestCase(testCaseClass)
863
864   Return a suite of all tests cases contained in the :class:`TestCase`\ -derived
865   :class:`testCaseClass`.
866
867
868.. method:: TestLoader.loadTestsFromModule(module)
869
870   Return a suite of all tests cases contained in the given module. This method
871   searches *module* for classes derived from :class:`TestCase` and creates an
872   instance of the class for each test method defined for the class.
873
874   .. warning::
875
876      While using a hierarchy of :class:`TestCase`\ -derived classes can be convenient
877      in sharing fixtures and helper functions, defining test methods on base classes
878      that are not intended to be instantiated directly does not play well with this
879      method.  Doing so, however, can be useful when the fixtures are different and
880      defined in subclasses.
881
882
883.. method:: TestLoader.loadTestsFromName(name[, module])
884
885   Return a suite of all tests cases given a string specifier.
886
887   The specifier *name* is a "dotted name" that may resolve either to a module, a
888   test case class, a test method within a test case class, a :class:`TestSuite`
889   instance, or a callable object which returns a :class:`TestCase` or
890   :class:`TestSuite` instance.  These checks are applied in the order listed here;
891   that is, a method on a possible test case class will be picked up as "a test
892   method within a test case class", rather than "a callable object".
893
894   For example, if you have a module :mod:`SampleTests` containing a
895   :class:`TestCase`\ -derived class :class:`SampleTestCase` with three test
896   methods (:meth:`test_one`, :meth:`test_two`, and :meth:`test_three`), the
897   specifier ``'SampleTests.SampleTestCase'`` would cause this method to return a
898   suite which will run all three test methods.  Using the specifier
899   ``'SampleTests.SampleTestCase.test_two'`` would cause it to return a test suite
900   which will run only the :meth:`test_two` test method.  The specifier can refer
901   to modules and packages which have not been imported; they will be imported as a
902   side-effect.
903
904   The method optionally resolves *name* relative to the given *module*.
905
906
907.. method:: TestLoader.loadTestsFromNames(names[, module])
908
909   Similar to :meth:`loadTestsFromName`, but takes a sequence of names rather than
910   a single name.  The return value is a test suite which supports all the tests
911   defined for each name.
912
913
914.. method:: TestLoader.getTestCaseNames(testCaseClass)
915
916   Return a sorted sequence of method names found within *testCaseClass*; this
917   should be a subclass of :class:`TestCase`.
918
919The following attributes of a :class:`TestLoader` can be configured either by
920subclassing or assignment on an instance:
921
922
923.. attribute:: TestLoader.testMethodPrefix
924
925   String giving the prefix of method names which will be interpreted as test
926   methods.  The default value is ``'test'``.
927
928   This affects :meth:`getTestCaseNames` and all the :meth:`loadTestsFrom\*`
929   methods.
930
931
932.. attribute:: TestLoader.sortTestMethodsUsing
933
934   Function to be used to compare method names when sorting them in
935   :meth:`getTestCaseNames` and all the :meth:`loadTestsFrom\*` methods. The
936   default value is the built-in :func:`cmp` function; the attribute can also be
937   set to :const:`None` to disable the sort.
938
939
940.. attribute:: TestLoader.suiteClass
941
942   Callable object that constructs a test suite from a list of tests. No methods on
943   the resulting object are needed.  The default value is the :class:`TestSuite`
944   class.
945
946   This affects all the :meth:`loadTestsFrom\*` methods.
947