/Lib/test/regrtest.py

   1#! /usr/bin/env python
   2
   3"""Regression test.
   4
   5This will find all modules whose name is "test_*" in the test
   6directory, and run them.  Various command line options provide
   7additional facilities.
   8
   9Command line options:
  10
  11-v: verbose    -- run tests in verbose mode with output to stdout
  12-w: verbose2   -- re-run failed tests in verbose mode
  13-q: quiet      -- don't print anything except if a test fails
  14-x: exclude    -- arguments are tests to *exclude*
  15-s: single     -- run only a single test (see below)
  16-S: slow       -- print the slowest 10 tests
  17-r: random     -- randomize test execution order
  18-f: fromfile   -- read names of tests to run from a file (see below)
  19-l: findleaks  -- if GC is available detect tests that leak memory
  20-u: use        -- specify which special resource intensive tests to run
  21-h: help       -- print this text and exit
  22-t: threshold  -- call gc.set_threshold(N)
  23-T: coverage   -- turn on code coverage using the trace module
  24-D: coverdir   -- Directory where coverage files are put
  25-N: nocoverdir -- Put coverage files alongside modules
  26-L: runleaks   -- run the leaks(1) command just before exit
  27-R: huntrleaks -- search for reference leaks (needs debug build, v. slow)
  28-M: memlimit   -- run very large memory-consuming tests
  29--failfast     -- stop after a single test fails, rather than running all tests
  30
  31If non-option arguments are present, they are names for tests to run,
  32unless -x is given, in which case they are names for tests not to run.
  33If no test names are given, all tests are run.
  34
  35-r randomizes test execution order. You can use --randseed=int to provide a
  36int seed value for the randomizer; this is useful for reproducing troublesome
  37test orders.
  38
  39-T turns on code coverage tracing with the trace module.
  40
  41-D specifies the directory where coverage files are put.
  42
  43-N Put coverage files alongside modules.
  44
  45-s means to run only a single test and exit.  This is useful when
  46doing memory analysis on the Python interpreter (which tend to consume
  47too many resources to run the full regression test non-stop).  The
  48file /tmp/pynexttest is read to find the next test to run.  If this
  49file is missing, the first test_*.py file in testdir or on the command
  50line is used.  (actually tempfile.gettempdir() is used instead of
  51/tmp).
  52
  53-f reads the names of tests from the file given as f's argument, one
  54or more test names per line.  Whitespace is ignored.  Blank lines and
  55lines beginning with '#' are ignored.  This is especially useful for
  56whittling down failures involving interactions among tests.
  57
  58-L causes the leaks(1) command to be run just before exit if it exists.
  59leaks(1) is available on Mac OS X and presumably on some other
  60FreeBSD-derived systems.
  61
  62-R runs each test several times and examines sys.gettotalrefcount() to
  63see if the test appears to be leaking references.  The argument should
  64be of the form stab:run:fname where 'stab' is the number of times the
  65test is run to let gettotalrefcount settle down, 'run' is the number
  66of times further it is run and 'fname' is the name of the file the
  67reports are written to.  These parameters all have defaults (5, 4 and
  68"reflog.txt" respectively), so the minimal invocation is '-R ::'.
  69
  70-M runs tests that require an exorbitant amount of memory. These tests
  71typically try to ascertain containers keep working when containing more than
  722 billion objects, which only works on 64-bit systems. There are also some
  73tests that try to exhaust the address space of the process, which only makes
  74sense on 32-bit systems with at least 2Gb of memory. The passed-in memlimit,
  75which is a string in the form of '2.5Gb', determines howmuch memory the
  76tests will limit themselves to (but they may go slightly over.) The number
  77shouldn't be more memory than the machine has (including swap memory). You
  78should also keep in mind that swap memory is generally much, much slower
  79than RAM, and setting memlimit to all available RAM or higher will heavily
  80tax the machine. On the other hand, it is no use running these tests with a
  81limit of less than 2.5Gb, and many require more than 20Gb. Tests that expect
  82to use more than memlimit memory will be skipped. The big-memory tests
  83generally run very, very long.
  84
  85--failfast causes regrtest.py to exit with status 1 as soon as the first test
  86fails. This is useful in combination with --randseed for reproducing
  87hard-to-trigger test failures.
  88
  89-u is used to specify which special resource intensive tests to run,
  90such as those requiring large file support or network connectivity.
  91The argument is a comma-separated list of words indicating the
  92resources to test.  Currently only the following are defined:
  93
  94    all -       Enable all special resources.
  95
  96    audio -     Tests that use the audio device.  (There are known
  97                cases of broken audio drivers that can crash Python or
  98                even the Linux kernel.)
  99
 100    curses -    Tests that use curses and will modify the terminal's
 101                state and output modes.
 102
 103    lib2to3 -   Run the tests for 2to3 (They take a while.)
 104
 105    largefile - It is okay to run some test that may create huge
 106                files.  These tests can take a long time and may
 107                consume >2GB of disk space temporarily.
 108
 109    network -   It is okay to run tests that use external network
 110                resource, e.g. testing SSL support for sockets.
 111
 112    bsddb -     It is okay to run the bsddb testsuite, which takes
 113                a long time to complete.
 114
 115    decimal -   Test the decimal module against a large suite that
 116                verifies compliance with standards.
 117
 118    compiler -  Test the compiler package by compiling all the source
 119                in the standard library and test suite.  This takes
 120                a long time.  Enabling this resource also allows
 121                test_tokenize to verify round-trip lexing on every
 122                file in the test library.
 123
 124    subprocess  Run all tests for the subprocess module.
 125
 126    urlfetch -  It is okay to download files required on testing.
 127
 128To enable all resources except one, use '-uall,-<resource>'.  For
 129example, to run all the tests except for the bsddb tests, give the
 130option '-uall,-bsddb'.
 131"""
 132
 133import cStringIO
 134import getopt
 135import os
 136import random
 137import re
 138import sys
 139import time
 140import traceback
 141import types
 142import warnings
 143try:
 144    import _llvm
 145except ImportError:
 146    _llvm = None
 147    sys.exc_clear()
 148
 149# I see no other way to suppress these warnings;
 150# putting them in test_grammar.py has no effect:
 151warnings.filterwarnings("ignore", "hex/oct constants", FutureWarning,
 152                        ".*test.test_grammar$")
 153if sys.maxint > 0x7fffffff:
 154    # Also suppress them in <string>, because for 64-bit platforms,
 155    # that's where test_grammar.py hides them.
 156    warnings.filterwarnings("ignore", "hex/oct constants", FutureWarning,
 157                            "<string>")
 158
 159# Ignore ImportWarnings that only occur in the source tree,
 160# (because of modules with the same name as source-directories in Modules/)
 161for mod in ("ctypes", "gzip", "zipfile", "tarfile", "encodings.zlib_codec",
 162            "test.test_zipimport", "test.test_zlib", "test.test_zipfile",
 163            "test.test_codecs", "test.string_tests"):
 164    warnings.filterwarnings(module=".*%s$" % (mod,),
 165                            action="ignore", category=ImportWarning)
 166
 167# MacOSX (a.k.a. Darwin) has a default stack size that is too small
 168# for deeply recursive regular expressions.  We see this as crashes in
 169# the Python test suite when running test_re.py and test_sre.py.  The
 170# fix is to set the stack limit to 2048.
 171# This approach may also be useful for other Unixy platforms that
 172# suffer from small default stack limits.
 173if sys.platform == 'darwin':
 174    try:
 175        import resource
 176    except ImportError:
 177        pass
 178    else:
 179        soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
 180        newsoft = min(hard, max(soft, 1024*2048))
 181        resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard))
 182
 183from test import test_support
 184
 185RESOURCE_NAMES = ('audio', 'curses', 'largefile', 'network', 'bsddb',
 186                  'decimal', 'compiler', 'subprocess', 'urlfetch')
 187
 188
 189def usage(code, msg=''):
 190    print __doc__
 191    if msg: print msg
 192    sys.exit(code)
 193
 194
 195def main(tests=None, testdir=None, verbose=0, quiet=False,
 196         exclude=False, single=False, randomize=False, fromfile=None,
 197         findleaks=False, use_resources=None, trace=False, coverdir='coverage',
 198         runleaks=False, huntrleaks=False, verbose2=False, print_slow=False,
 199         random_seed=None, failfast=False):
 200    """Execute a test suite.
 201
 202    This also parses command-line options and modifies its behavior
 203    accordingly.
 204
 205    tests -- a list of strings containing test names (optional)
 206    testdir -- the directory in which to look for tests (optional)
 207
 208    Users other than the Python test suite will certainly want to
 209    specify testdir; if it's omitted, the directory containing the
 210    Python test suite is searched for.
 211
 212    If the tests argument is omitted, the tests listed on the
 213    command-line will be used.  If that's empty, too, then all *.py
 214    files beginning with test_ will be used.
 215
 216    The other default arguments (verbose, quiet, exclude,
 217    single, randomize, findleaks, use_resources, trace, coverdir, print_slow and
 218    random_seed, failfast) allow programmers calling main() directly to set the
 219    values that would normally be set by flags on the command line.
 220    """
 221
 222    test_support.record_original_stdout(sys.stdout)
 223    try:
 224        opts, args = getopt.getopt(sys.argv[1:], 'hvgqxsSrf:lu:t:TD:NLR:wM:',
 225                                   ['help', 'verbose', 'quiet', 'exclude',
 226                                    'single', 'slow', 'random', 'fromfile',
 227                                    'findleaks', 'use=', 'threshold=', 'trace',
 228                                    'coverdir=', 'nocoverdir', 'runleaks',
 229                                    'huntrleaks=', 'verbose2', 'memlimit=',
 230                                    'randseed=', 'failfast',
 231                                    ])
 232    except getopt.error, msg:
 233        usage(2, msg)
 234
 235    # Defaults
 236    if random_seed is None:
 237        random_seed = int(1000000 * random.random())
 238    if use_resources is None:
 239        use_resources = []
 240    for o, a in opts:
 241        if o in ('-h', '--help'):
 242            usage(0)
 243        elif o in ('-v', '--verbose'):
 244            verbose += 1
 245        elif o in ('-w', '--verbose2'):
 246            verbose2 = True
 247        elif o in ('-q', '--quiet'):
 248            quiet = True;
 249            verbose = 0
 250        elif o in ('-x', '--exclude'):
 251            exclude = True
 252        elif o in ('-s', '--single'):
 253            single = True
 254        elif o in ('-S', '--slow'):
 255            print_slow = True
 256        elif o in ('-r', '--randomize'):
 257            randomize = True
 258        elif o == '--randseed':
 259            random_seed = int(a)
 260        elif o in ('-f', '--fromfile'):
 261            fromfile = a
 262        elif o == '--failfast':
 263            failfast = True
 264        elif o in ('-l', '--findleaks'):
 265            findleaks = True
 266        elif o in ('-L', '--runleaks'):
 267            runleaks = True
 268        elif o in ('-t', '--threshold'):
 269            import gc
 270            gc.set_threshold(int(a))
 271        elif o in ('-T', '--coverage'):
 272            trace = True
 273        elif o in ('-D', '--coverdir'):
 274            coverdir = os.path.join(os.getcwd(), a)
 275        elif o in ('-N', '--nocoverdir'):
 276            coverdir = None
 277        elif o in ('-R', '--huntrleaks'):
 278            huntrleaks = a.split(':')
 279            if len(huntrleaks) != 3:
 280                print a, huntrleaks
 281                usage(2, '-R takes three colon-separated arguments')
 282            if len(huntrleaks[0]) == 0:
 283                huntrleaks[0] = 5
 284            else:
 285                huntrleaks[0] = int(huntrleaks[0])
 286            if len(huntrleaks[1]) == 0:
 287                huntrleaks[1] = 4
 288            else:
 289                huntrleaks[1] = int(huntrleaks[1])
 290            if len(huntrleaks[2]) == 0:
 291                huntrleaks[2] = "reflog.txt"
 292        elif o in ('-M', '--memlimit'):
 293            test_support.set_memlimit(a)
 294        elif o in ('-u', '--use'):
 295            u = [x.lower() for x in a.split(',')]
 296            for r in u:
 297                if r == 'all':
 298                    use_resources[:] = RESOURCE_NAMES
 299                    continue
 300                remove = False
 301                if r[0] == '-':
 302                    remove = True
 303                    r = r[1:]
 304                if r not in RESOURCE_NAMES:
 305                    usage(1, 'Invalid -u/--use option: ' + a)
 306                if remove:
 307                    if r in use_resources:
 308                        use_resources.remove(r)
 309                elif r not in use_resources:
 310                    use_resources.append(r)
 311    if single and fromfile:
 312        usage(2, "-s and -f don't go together!")
 313
 314    good = []
 315    bad = []
 316    skipped = []
 317    resource_denieds = []
 318
 319    if findleaks:
 320        try:
 321            import gc
 322        except ImportError:
 323            print 'No GC available, disabling findleaks.'
 324            findleaks = False
 325        else:
 326            # Uncomment the line below to report garbage that is not
 327            # freeable by reference counting alone.  By default only
 328            # garbage that is not collectable by the GC is reported.
 329            #gc.set_debug(gc.DEBUG_SAVEALL)
 330            found_garbage = []
 331
 332    if single:
 333        from tempfile import gettempdir
 334        filename = os.path.join(gettempdir(), 'pynexttest')
 335        try:
 336            fp = open(filename, 'r')
 337            next = fp.read().strip()
 338            tests = [next]
 339            fp.close()
 340        except IOError:
 341            pass
 342
 343    if fromfile:
 344        tests = []
 345        fp = open(fromfile)
 346        for line in fp:
 347            guts = line.split() # assuming no test has whitespace in its name
 348            if guts and not guts[0].startswith('#'):
 349                tests.extend(guts)
 350        fp.close()
 351
 352    # Strip .py extensions.
 353    if args:
 354        args = map(removepy, args)
 355    if tests:
 356        tests = map(removepy, tests)
 357
 358    stdtests = STDTESTS[:]
 359    nottests = NOTTESTS[:]
 360    if exclude:
 361        for arg in args:
 362            if arg in stdtests:
 363                stdtests.remove(arg)
 364        nottests[:0] = args
 365        args = []
 366    tests = tests or args or findtests(testdir, stdtests, nottests)
 367    if single:
 368        tests = tests[:1]
 369    if randomize:
 370        if random_seed:
 371            random.seed(random_seed)
 372        print "Using random seed", random_seed
 373        random.shuffle(tests)
 374    if trace:
 375        import trace
 376        tracer = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix],
 377                             trace=False, count=True)
 378    print "Testing with flags:", sys.flags
 379    test_times = []
 380    test_support.verbose = verbose      # Tell tests to be moderately quiet
 381    test_support.use_resources = use_resources
 382    save_modules = sys.modules.keys()
 383    for test in tests:
 384        if not quiet:
 385            print test
 386            sys.stdout.flush()
 387        if trace:
 388            # If we're tracing code coverage, then we don't exit with status
 389            # if on a false return value from main.
 390            tracer.runctx('runtest(test, verbose, quiet,'
 391                          '        test_times, testdir)',
 392                          globals=globals(), locals=vars())
 393        else:
 394            try:
 395                ok = runtest(test, verbose, quiet, test_times,
 396                             testdir, huntrleaks)
 397            except KeyboardInterrupt:
 398                # print a newline separate from the ^C
 399                print
 400                break
 401            except:
 402                raise
 403            if ok > 0:
 404                good.append(test)
 405            elif ok == 0:
 406                bad.append(test)
 407                if failfast:
 408                    break
 409            else:
 410                skipped.append(test)
 411                if ok == -2:
 412                    resource_denieds.append(test)
 413        if findleaks:
 414            gc.collect()
 415            if gc.garbage:
 416                print "Warning: test created", len(gc.garbage),
 417                print "uncollectable object(s)."
 418                # move the uncollectable objects somewhere so we don't see
 419                # them again
 420                found_garbage.extend(gc.garbage)
 421                del gc.garbage[:]
 422        # Unload the newly imported modules (best effort finalization)
 423        for module in sys.modules.keys():
 424            if module not in save_modules and module.startswith("test."):
 425                test_support.unload(module)
 426
 427    # The lists won't be sorted if running with -r
 428    good.sort()
 429    bad.sort()
 430    skipped.sort()
 431
 432    if good and not quiet:
 433        if not bad and not skipped and len(good) > 1:
 434            print "All",
 435        print count(len(good), "test"), "OK."
 436    if print_slow:
 437        test_times.sort(reverse=True)
 438        print "10 slowest tests:"
 439        for time, test in test_times[:10]:
 440            print "%s: %.1fs" % (test, time)
 441    if bad:
 442        print count(len(bad), "test"), "failed:"
 443        printlist(bad)
 444    if skipped and not quiet:
 445        print count(len(skipped), "test"), "skipped:"
 446        printlist(skipped)
 447
 448        e = _ExpectedSkips()
 449        plat = sys.platform
 450        if e.isvalid():
 451            surprise = set(skipped) - e.getexpected() - set(resource_denieds)
 452            if surprise:
 453                print count(len(surprise), "skip"), \
 454                      "unexpected on", plat + ":"
 455                printlist(surprise)
 456            else:
 457                print "Those skips are all expected on", plat + "."
 458        else:
 459            print "Ask someone to teach regrtest.py about which tests are"
 460            print "expected to get skipped on", plat + "."
 461
 462    if verbose2 and bad:
 463        print "Re-running failed tests in verbose mode"
 464        for test in bad:
 465            print "Re-running test %r in verbose mode" % test
 466            sys.stdout.flush()
 467            try:
 468                test_support.verbose = True
 469                ok = runtest(test, True, quiet, test_times, testdir,
 470                             huntrleaks)
 471            except KeyboardInterrupt:
 472                # print a newline separate from the ^C
 473                print
 474                break
 475            except:
 476                raise
 477
 478    if single:
 479        alltests = findtests(testdir, stdtests, nottests)
 480        for i in range(len(alltests)):
 481            if tests[0] == alltests[i]:
 482                if i == len(alltests) - 1:
 483                    os.unlink(filename)
 484                else:
 485                    fp = open(filename, 'w')
 486                    fp.write(alltests[i+1] + '\n')
 487                    fp.close()
 488                break
 489        else:
 490            os.unlink(filename)
 491
 492    if trace:
 493        r = tracer.results()
 494        r.write_results(show_missing=True, summary=True, coverdir=coverdir)
 495
 496    if runleaks:
 497        os.system("leaks %d" % os.getpid())
 498
 499    sys.exit(len(bad) > 0)
 500
 501
 502STDTESTS = [
 503    'test_grammar',
 504    'test_opcodes',
 505    'test_dict',
 506    'test_builtin',
 507    'test_exceptions',
 508    'test_types',
 509    'test_unittest',
 510    'test_doctest',
 511    'test_doctest2',
 512   ]
 513
 514NOTTESTS = [
 515    'test_support',
 516    'test_future1',
 517    'test_future2',
 518    ]
 519
 520def findtests(testdir=None, stdtests=STDTESTS, nottests=NOTTESTS):
 521    """Return a list of all applicable test modules."""
 522    if not testdir: testdir = findtestdir()
 523    names = os.listdir(testdir)
 524    tests = []
 525    for name in names:
 526        if name[:5] == "test_" and name[-3:] == os.extsep+"py":
 527            modname = name[:-3]
 528            if modname not in stdtests and modname not in nottests:
 529                tests.append(modname)
 530    tests.sort()
 531    return stdtests + tests
 532
 533def runtest(test, verbose, quiet, test_times,
 534            testdir=None, huntrleaks=False):
 535    """Run a single test.
 536
 537    test -- the name of the test
 538    verbose -- if true, print more messages
 539    quiet -- if true, don't print 'skipped' messages (probably redundant)
 540    test_times -- a list of (time, test_name) pairs
 541    testdir -- test directory
 542    huntrleaks -- run multiple times to test for leaks; requires a debug
 543                  build; a triple corresponding to -R's three arguments
 544    Return:
 545        -2  test skipped because resource denied
 546        -1  test skipped for some other reason
 547         0  test failed
 548         1  test passed
 549    """
 550
 551    try:
 552        return runtest_inner(test, verbose, quiet, test_times,
 553                             testdir, huntrleaks)
 554    finally:
 555        cleanup_test_droppings(test, verbose)
 556
 557# See runtest() for info on parameters and possibly return values.
 558def runtest_inner(test, verbose, quiet, test_times,
 559                  testdir=None, huntrleaks=False):
 560    test_support.unload(test)
 561    if not testdir:
 562        testdir = findtestdir()
 563    if verbose:
 564        capture_stdout = None
 565    else:
 566        capture_stdout = cStringIO.StringIO()
 567
 568    refleak = False  # True if the test leaked references.
 569    try:
 570        save_stdout = sys.stdout
 571        try:
 572            if capture_stdout:
 573                sys.stdout = capture_stdout
 574            if test.startswith('test.'):
 575                abstest = test
 576            else:
 577                # Always import it from the test package
 578                abstest = 'test.' + test
 579            start_time = time.time()
 580            the_package = __import__(abstest, globals(), locals(), [])
 581            the_module = getattr(the_package, test)
 582            # Old tests run to completion simply as a side-effect of
 583            # being imported.  For tests based on unittest or doctest,
 584            # explicitly invoke their test_main() function (if it exists).
 585            indirect_test = getattr(the_module, "test_main", None)
 586            if indirect_test is not None:
 587                indirect_test()
 588            if huntrleaks:
 589                refleak = dash_R(the_module, test, indirect_test, huntrleaks)
 590            test_time = time.time() - start_time
 591            test_times.append((test_time, test))
 592        finally:
 593            sys.stdout = save_stdout
 594    except test_support.ResourceDenied, msg:
 595        if not quiet:
 596            print test, "skipped --", msg
 597            sys.stdout.flush()
 598        return -2
 599    except (ImportError, test_support.TestSkipped), msg:
 600        if not quiet:
 601            print test, "skipped --", msg
 602            sys.stdout.flush()
 603        return -1
 604    except KeyboardInterrupt:
 605        raise
 606    except test_support.TestFailed, msg:
 607        print "test", test, "failed --", msg
 608        sys.stdout.flush()
 609        return 0
 610    except:
 611        type, value = sys.exc_info()[:2]
 612        print "test", test, "crashed --", str(type) + ":", value
 613        sys.stdout.flush()
 614        if verbose:
 615            traceback.print_exc(file=sys.stdout)
 616            sys.stdout.flush()
 617        return 0
 618    else:
 619        if refleak:
 620            return 0
 621        # Except in verbose mode, tests should not print anything
 622        if verbose or huntrleaks:
 623            return 1
 624        output = capture_stdout.getvalue()
 625        if not output:
 626            return 1
 627        print "test", test, "produced unexpected output:"
 628        print "*" * 70
 629        print output
 630        print "*" * 70
 631        sys.stdout.flush()
 632        return 0
 633
 634def cleanup_test_droppings(testname, verbose):
 635    import shutil
 636
 637    # Try to clean up junk commonly left behind.  While tests shouldn't leave
 638    # any files or directories behind, when a test fails that can be tedious
 639    # for it to arrange.  The consequences can be especially nasty on Windows,
 640    # since if a test leaves a file open, it cannot be deleted by name (while
 641    # there's nothing we can do about that here either, we can display the
 642    # name of the offending test, which is a real help).
 643    for name in (test_support.TESTFN,
 644                 "db_home",
 645                ):
 646        if not os.path.exists(name):
 647            continue
 648
 649        if os.path.isdir(name):
 650            kind, nuker = "directory", shutil.rmtree
 651        elif os.path.isfile(name):
 652            kind, nuker = "file", os.unlink
 653        else:
 654            raise SystemError("os.path says %r exists but is neither "
 655                              "directory nor file" % name)
 656
 657        if verbose:
 658            print "%r left behind %s %r" % (testname, kind, name)
 659        try:
 660            nuker(name)
 661        except Exception, msg:
 662            print >> sys.stderr, ("%r left behind %s %r and it couldn't be "
 663                "removed: %s" % (testname, kind, name, msg))
 664
 665def dash_R(the_module, test, indirect_test, huntrleaks):
 666    """Run a test multiple times, looking for reference leaks.
 667
 668    Returns:
 669        False if the test didn't leak references; True if we detected refleaks.
 670    """
 671    # This code is hackish and inelegant, but it seems to do the job.
 672    import copy_reg, _abcoll, io
 673
 674    if not hasattr(sys, 'gettotalrefcount'):
 675        raise Exception("Tracking reference leaks requires a debug build "
 676                        "of Python")
 677
 678    # Save current values for dash_R_cleanup() to restore.
 679    fs = warnings.filters[:]
 680    ps = copy_reg.dispatch_table.copy()
 681    pic = sys.path_importer_cache.copy()
 682    abcs = {}
 683    modules = _abcoll, io
 684    for abc in [getattr(mod, a) for mod in modules for a in mod.__all__]:
 685        # XXX isinstance(abc, ABCMeta) leads to infinite recursion
 686        if not hasattr(abc, '_abc_registry'):
 687            continue
 688        for obj in abc.__subclasses__() + [abc]:
 689            abcs[obj] = obj._abc_registry.copy()
 690
 691    if indirect_test:
 692        def run_the_test():
 693            indirect_test()
 694    else:
 695        def run_the_test():
 696            reload(the_module)
 697
 698    deltas = []
 699    nwarmup, ntracked, fname = huntrleaks
 700    repcount = nwarmup + ntracked
 701    print >> sys.stderr, "beginning", repcount, "repetitions"
 702    print >> sys.stderr, ("1234567890"*(repcount//10 + 1))[:repcount]
 703    if _llvm:
 704        _llvm.set_jit_control("never")
 705    for i in range(repcount):
 706        dash_R_cleanup(fs, ps, pic, abcs)
 707        rc_before = sys.gettotalrefcount()
 708        run_the_test()
 709        sys.stderr.write('.')
 710        dash_R_cleanup(fs, ps, pic, abcs)
 711        rc_after = sys.gettotalrefcount()
 712        if i >= nwarmup:
 713            deltas.append(rc_after - rc_before)
 714    print >> sys.stderr
 715    if any(deltas):
 716        msg = '%s leaked %s references, sum=%s' % (test, deltas, sum(deltas))
 717        print >> sys.stderr, msg
 718        refrep = open(fname, "a")
 719        print >> refrep, msg
 720        refrep.close()
 721        return True
 722    return False
 723
 724def dash_R_cleanup(fs, ps, pic, abcs):
 725    import gc, copy_reg
 726    import _strptime, linecache
 727    dircache = test_support.import_module('dircache', deprecated=True)
 728    import urlparse, urllib, urllib2, mimetypes, doctest
 729    import struct, filecmp
 730    from distutils.dir_util import _path_created
 731
 732    # Clear the warnings registry, so they can be displayed again
 733    for mod in sys.modules.values():
 734        if hasattr(mod, '__warningregistry__'):
 735            del mod.__warningregistry__
 736
 737    # Restore some original values.
 738    warnings.filters[:] = fs
 739    copy_reg.dispatch_table.clear()
 740    copy_reg.dispatch_table.update(ps)
 741    sys.path_importer_cache.clear()
 742    sys.path_importer_cache.update(pic)
 743
 744    # clear type cache
 745    sys._clear_type_cache()
 746
 747    # Clear ABC registries, restoring previously saved ABC registries.
 748    for abc, registry in abcs.items():
 749        abc._abc_registry = registry.copy()
 750        abc._abc_cache.clear()
 751        abc._abc_negative_cache.clear()
 752
 753    # Clear assorted module caches.
 754    _path_created.clear()
 755    re.purge()
 756    _strptime._regex_cache.clear()
 757    urlparse.clear_cache()
 758    urllib.urlcleanup()
 759    urllib2.install_opener(None)
 760    dircache.reset()
 761    linecache.clearcache()
 762    mimetypes._default_mime_types()
 763    filecmp._cache.clear()
 764    struct._clearcache()
 765    doctest.master = None
 766
 767    if _llvm:
 768        code_types = (types.CodeType, types.FunctionType, types.MethodType)
 769        for obj in gc.get_objects():
 770            if isinstance(obj, code_types):
 771                _llvm.clear_feedback(obj)
 772
 773    # Collect cyclic trash.
 774    gc.collect()
 775
 776def findtestdir():
 777    if __name__ == '__main__':
 778        file = sys.argv[0]
 779    else:
 780        file = __file__
 781    testdir = os.path.dirname(file) or os.curdir
 782    return testdir
 783
 784def removepy(name):
 785    if name.endswith(os.extsep + "py"):
 786        name = name[:-3]
 787    return name
 788
 789def count(n, word):
 790    if n == 1:
 791        return "%d %s" % (n, word)
 792    else:
 793        return "%d %ss" % (n, word)
 794
 795def printlist(x, width=70, indent=4):
 796    """Print the elements of iterable x to stdout.
 797
 798    Optional arg width (default 70) is the maximum line length.
 799    Optional arg indent (default 4) is the number of blanks with which to
 800    begin each line.
 801    """
 802
 803    from textwrap import fill
 804    blanks = ' ' * indent
 805    print fill(' '.join(map(str, x)), width,
 806               initial_indent=blanks, subsequent_indent=blanks)
 807
 808# Map sys.platform to a string containing the basenames of tests
 809# expected to be skipped on that platform.
 810#
 811# Special cases:
 812#     test_pep277
 813#         The _ExpectedSkips constructor adds this to the set of expected
 814#         skips if not os.path.supports_unicode_filenames.
 815#     test_socket_ssl
 816#         Controlled by test_socket_ssl.skip_expected.  Requires the network
 817#         resource, and a socket module with ssl support.
 818#     test_timeout
 819#         Controlled by test_timeout.skip_expected.  Requires the network
 820#         resource and a socket module.
 821#
 822# Tests that are expected to be skipped everywhere except on one platform
 823# are also handled separately.
 824
 825_expectations = {
 826    'win32':
 827        """
 828        test__locale
 829        test_bsddb185
 830        test_bsddb3
 831        test_commands
 832        test_crypt
 833        test_curses
 834        test_dbm
 835        test_dl
 836        test_fcntl
 837        test_fork1
 838        test_epoll
 839        test_gdbm
 840        test_grp
 841        test_ioctl
 842        test_jit_gdb
 843        test_largefile
 844        test_kqueue
 845        test_mhlib
 846        test_openpty
 847        test_ossaudiodev
 848        test_pipes
 849        test_poll
 850        test_posix
 851        test_pty
 852        test_pwd
 853        test_resource
 854        test_signal
 855        test_threadsignals
 856        test_timing
 857        test_wait3
 858        test_wait4
 859        """,
 860    'linux2':
 861        """
 862        test_bsddb185
 863        test_curses
 864        test_dl
 865        test_largefile
 866        test_kqueue
 867        test_ossaudiodev
 868        """,
 869   'mac':
 870        """
 871        test_atexit
 872        test_bsddb
 873        test_bsddb185
 874        test_bsddb3
 875        test_bz2
 876        test_commands
 877        test_crypt
 878        test_curses
 879        test_dbm
 880        test_dl
 881        test_fcntl
 882        test_fork1
 883        test_epoll
 884        test_grp
 885        test_ioctl
 886        test_jit_gdb
 887        test_largefile
 888        test_locale
 889        test_kqueue
 890        test_mmap
 891        test_openpty
 892        test_ossaudiodev
 893        test_poll
 894        test_popen
 895        test_popen2
 896        test_posix
 897        test_pty
 898        test_pwd
 899        test_resource
 900        test_signal
 901        test_sundry
 902        test_tarfile
 903        test_timing
 904        """,
 905    'unixware7':
 906        """
 907        test_bsddb
 908        test_bsddb185
 909        test_dl
 910        test_epoll
 911        test_jit_gdb
 912        test_largefile
 913        test_kqueue
 914        test_minidom
 915        test_openpty
 916        test_pyexpat
 917        test_sax
 918        test_sundry
 919        """,
 920    'openunix8':
 921        """
 922        test_bsddb
 923        test_bsddb185
 924        test_dl
 925        test_epoll
 926        test_jit_gdb
 927        test_largefile
 928        test_kqueue
 929        test_minidom
 930        test_openpty
 931        test_pyexpat
 932        test_sax
 933        test_sundry
 934        """,
 935    'sco_sv3':
 936        """
 937        test_asynchat
 938        test_bsddb
 939        test_bsddb185
 940        test_dl
 941        test_fork1
 942        test_epoll
 943        test_gettext
 944        test_jit_gdb
 945        test_largefile
 946        test_locale
 947        test_kqueue
 948        test_minidom
 949        test_openpty
 950        test_pyexpat
 951        test_queue
 952        test_sax
 953        test_sundry
 954        test_thread
 955        test_threaded_import
 956        test_threadedtempfile
 957        test_threading
 958        """,
 959    'riscos':
 960        """
 961        test_asynchat
 962        test_atexit
 963        test_bsddb
 964        test_bsddb185
 965        test_bsddb3
 966        test_commands
 967        test_crypt
 968        test_dbm
 969        test_dl
 970        test_fcntl
 971        test_fork1
 972        test_epoll
 973        test_gdbm
 974        test_grp
 975        test_jit_gdb
 976        test_largefile
 977        test_locale
 978        test_kqueue
 979        test_mmap
 980        test_openpty
 981        test_poll
 982        test_popen2
 983        test_pty
 984        test_pwd
 985        test_strop
 986        test_sundry
 987        test_thread
 988        test_threaded_import
 989        test_threadedtempfile
 990        test_threading
 991        test_timing
 992        """,
 993    'darwin':
 994        """
 995        test__locale
 996        test_bsddb
 997        test_bsddb3
 998        test_curses
 999        test_epoll
1000        test_gdbm
1001        test_jit_gdb
1002        test_largefile
1003        test_locale
1004        test_kqueue
1005        test_minidom
1006        test_ossaudiodev
1007        test_poll
1008        """,
1009    'sunos5':
1010        """
1011        test_bsddb
1012        test_bsddb185
1013        test_curses
1014        test_dbm
1015        test_epoll
1016        test_jit_gdb
1017        test_kqueue
1018        test_gdbm
1019        test_gzip
1020        test_openpty
1021        test_zipfile
1022        test_zlib
1023        """,
1024    'hp-ux11':
1025        """
1026        test_bsddb
1027        test_bsddb185
1028        test_curses
1029        test_dl
1030        test_epoll
1031        test_gdbm
1032        test_gzip
1033        test_jit_gdb
1034        test_largefile
1035        test_locale
1036        test_kqueue
1037        test_minidom
1038        test_openpty
1039        test_pyexpat
1040        test_sax
1041        test_zipfile
1042        test_zlib
1043        """,
1044    'atheos':
1045        """
1046        test_bsddb185
1047        test_curses
1048        test_dl
1049        test_gdbm
1050        test_epoll
1051        test_jit_gdb
1052        test_largefile
1053        test_locale
1054        test_kqueue
1055        test_mhlib
1056        test_mmap
1057        test_poll
1058        test_popen2
1059        test_resource
1060        """,
1061    'cygwin':
1062        """
1063        test_bsddb185
1064        test_bsddb3
1065        test_curses
1066        test_dbm
1067        test_epoll
1068        test_ioctl
1069        test_jit_gdb
1070        test_kqueue
1071        test_largefile
1072        test_locale
1073        test_ossaudiodev
1074        test_socketserver
1075        """,
1076    'os2emx':
1077        """
1078        test_audioop
1079        test_bsddb185
1080        test_bsddb3
1081        test_commands
1082        test_curses
1083        test_dl
1084        test_epoll
1085        test_jit_gdb
1086        test_kqueue
1087        test_largefile
1088        test_mhlib
1089        test_mmap
1090        test_openpty
1091        test_ossaudiodev
1092        test_pty
1093        test_resource
1094        test_signal
1095        """,
1096    'freebsd4':
1097        """
1098        test_bsddb
1099        test_bsddb3
1100        test_epoll
1101        test_gdbm
1102        test_jit_gdb
1103        test_locale
1104        test_ossaudiodev
1105        test_pep277
1106        test_pty
1107        test_socket_ssl
1108        test_socketserver
1109        test_tcl
1110        test_timeout
1111        test_urllibnet
1112        test_multiprocessing
1113        """,
1114    'aix5':
1115        """
1116        test_bsddb
1117        test_bsddb185
1118        test_bsddb3
1119        test_bz2
1120        test_dl
1121        test_epoll
1122        test_gdbm
1123        test_gzip
1124        test_jit_gdb
1125        test_kqueue
1126        test_ossaudiodev
1127        test_tcl
1128        test_zipimport
1129        test_zlib
1130        """,
1131    'openbsd3':
1132        """
1133        test_bsddb
1134        test_bsddb3
1135        test_ctypes
1136        test_dl
1137        test_epoll
1138        test_gdbm
1139        test_jit_gdb
1140        test_locale
1141        test_normalization
1142        test_ossaudiodev
1143        test_pep277
1144        test_tcl
1145        test_multiprocessing
1146        """,
1147    'netbsd3':
1148        """
1149        test_bsddb
1150        test_bsddb185
1151        test_bsddb3
1152        test_ctypes
1153        test_curses
1154        test_dl
1155        test_epoll
1156        test_gdbm
1157        test_jit_gdb
1158        test_locale
1159        test_ossaudiodev
1160        test_pep277
1161        test_tcl
1162        test_multiprocessing
1163        """,
1164}
1165_expectations['freebsd5'] = _expectations['freebsd4']
1166_expectations['freebsd6'] = _expectations['freebsd4']
1167_expectations['freebsd7'] = _expectations['freebsd4']
1168_expectations['freebsd8'] = _expectations['freebsd4']
1169
1170# If Python was configured with --without-llvm, these tests will always skip.
1171if _llvm is None:
1172    for plat, skips in _expectations.items():
1173        _expectations[plat] = skips + "\ntest_jit_gdb\ntest_llvm"
1174
1175
1176class _ExpectedSkips:
1177    def __init__(self):
1178        import os.path
1179        from test import test_timeout
1180
1181        self.valid = False
1182        if sys.platform in _expectations:
1183            s = _expectations[sys.platform]
1184            self.expected = set(s.split())
1185
1186            # expected to be skipped on every platform, even Linux
1187            self.expected.add('test_linuxaudiodev')
1188
1189            if not os.path.supports_unicode_filenames:
1190                self.expected.add('test_pep277')
1191
1192            try:
1193                from test import test_socket_ssl
1194            except ImportError:
1195                pass
1196            else:
1197                if test_socket_ssl.skip_expected:
1198                    self.expected.add('test_socket_ssl')
1199
1200            if test_timeout.skip_expected:
1201                self.expected.add('test_timeout')
1202
1203            if sys.maxint == 9223372036854775807L:
1204                self.expected.add('test_imageop')
1205
1206            if not sys.platform in ("mac", "darwin"):
1207                MAC_ONLY = ["test_macos", "test_macostools", "test_aepack",
1208                            "test_plistlib", "test_scriptpackages",
1209                            "test_applesingle"]
1210                for skip in MAC_ONLY:
1211                    self.expected.add(skip)
1212            elif len(u'\0'.encode('unicode-internal')) == 4:
1213                self.expected.add("test_macostools")
1214
1215
1216            if sys.platform != "win32":
1217                # test_sqlite is only reliable on Windows where the library
1218                # is distributed with Python
1219                WIN_ONLY = ["test_unicode_file", "test_winreg",
1220                            "test_winsound", "test_startfile",
1221                            "test_sqlite"]
1222                for skip in WIN_ONLY:
1223                    self.expected.add(skip)
1224
1225            if sys.platform != 'irix':
1226                IRIX_ONLY = ["test_imageop", "test_al", "test_cd", "test_cl",
1227                             "test_gl", "test_imgfile"]
1228                for skip in IRIX_ONLY:
1229                    self.expected.add(skip)
1230
1231            if sys.platform != 'sunos5':
1232                self.expected.add('test_sunaudiodev')
1233                self.expected.add('test_nis')
1234
1235            if not sys.py3kwarning:
1236                self.expected.add('test_py3kwarn')
1237
1238            self.valid = True
1239
1240    def isvalid(self):
1241        "Return true iff _ExpectedSkips knows about the current platform."
1242        return self.valid
1243
1244    def getexpected(self):
1245        """Return set of test names we expect to skip on current platform.
1246
1247        self.isvalid() must be true.
1248        """
1249
1250        assert self.isvalid()
1251        return self.expected
1252
1253if __name__ == '__main__':
1254    # Remove regrtest.py's own directory from the module search path.  This
1255    # prevents relative imports from working, and relative imports will screw
1256    # up the testing framework.  E.g. if both test.test_support and
1257    # test_support are imported, they will not contain the same globals, and
1258    # much of the testing framework relies on the globals in the
1259    # test.test_support module.
1260    mydir = os.path.abspath(os.path.normpath(os.path.dirname(sys.argv[0])))
1261    i = pathlen = len(sys.path)
1262    while i >= 0:
1263        i -= 1
1264        if os.path.abspath(os.path.normpath(sys.path[i])) == mydir:
1265            del sys.path[i]
1266    if len(sys.path) == pathlen:
1267        print 'Could not find %r in sys.path to remove it' % mydir
1268    main()