/Tools/pybench/README
http://unladen-swallow.googlecode.com/ · #! · 368 lines · 299 code · 69 blank · 0 comment · 0 complexity · 6c324c69c502bf966c8ac69e2dbec960 MD5 · raw file
- ________________________________________________________________________
- PYBENCH - A Python Benchmark Suite
- ________________________________________________________________________
- Extendable suite of of low-level benchmarks for measuring
- the performance of the Python implementation
- (interpreter, compiler or VM).
- pybench is a collection of tests that provides a standardized way to
- measure the performance of Python implementations. It takes a very
- close look at different aspects of Python programs and let's you
- decide which factors are more important to you than others, rather
- than wrapping everything up in one number, like the other performance
- tests do (e.g. pystone which is included in the Python Standard
- Library).
- pybench has been used in the past by several Python developers to
- track down performance bottlenecks or to demonstrate the impact of
- optimizations and new features in Python.
- The command line interface for pybench is the file pybench.py. Run
- this script with option '--help' to get a listing of the possible
- options. Without options, pybench will simply execute the benchmark
- and then print out a report to stdout.
- Micro-Manual
- ------------
- Run 'pybench.py -h' to see the help screen. Run 'pybench.py' to run
- the benchmark suite using default settings and 'pybench.py -f <file>'
- to have it store the results in a file too.
- It is usually a good idea to run pybench.py multiple times to see
- whether the environment, timers and benchmark run-times are suitable
- for doing benchmark tests.
- You can use the comparison feature of pybench.py ('pybench.py -c
- <file>') to check how well the system behaves in comparison to a
- reference run.
- If the differences are well below 10% for each test, then you have a
- system that is good for doing benchmark testings. Of you get random
- differences of more than 10% or significant differences between the
- values for minimum and average time, then you likely have some
- background processes running which cause the readings to become
- inconsistent. Examples include: web-browsers, email clients, RSS
- readers, music players, backup programs, etc.
- If you are only interested in a few tests of the whole suite, you can
- use the filtering option, e.g. 'pybench.py -t string' will only
- run/show the tests that have 'string' in their name.
- This is the current output of pybench.py --help:
- """
- ------------------------------------------------------------------------
- PYBENCH - a benchmark test suite for Python interpreters/compilers.
- ------------------------------------------------------------------------
- Synopsis:
- pybench.py [option] files...
- Options and default settings:
- -n arg number of rounds (10)
- -f arg save benchmark to file arg ()
- -c arg compare benchmark with the one in file arg ()
- -s arg show benchmark in file arg, then exit ()
- -w arg set warp factor to arg (10)
- -t arg run only tests with names matching arg ()
- -C arg set the number of calibration runs to arg (20)
- -d hide noise in comparisons (0)
- -v verbose output (not recommended) (0)
- --with-gc enable garbage collection (0)
- --with-syscheck use default sys check interval (0)
- --timer arg use given timer (time.time)
- -h show this help text
- --help show this help text
- --debug enable debugging
- --copyright show copyright
- --examples show examples of usage
- Version:
- 2.0
- The normal operation is to run the suite and display the
- results. Use -f to save them for later reuse or comparisons.
- Available timers:
- time.time
- time.clock
- systimes.processtime
- Examples:
- python2.1 pybench.py -f p21.pybench
- python2.5 pybench.py -f p25.pybench
- python pybench.py -s p25.pybench -c p21.pybench
- """
- License
- -------
- See LICENSE file.
- Sample output
- -------------
- """
- -------------------------------------------------------------------------------
- PYBENCH 2.0
- -------------------------------------------------------------------------------
- * using Python 2.4.2
- * disabled garbage collection
- * system check interval set to maximum: 2147483647
- * using timer: time.time
- Calibrating tests. Please wait...
- Running 10 round(s) of the suite at warp factor 10:
- * Round 1 done in 6.388 seconds.
- * Round 2 done in 6.485 seconds.
- * Round 3 done in 6.786 seconds.
- ...
- * Round 10 done in 6.546 seconds.
- -------------------------------------------------------------------------------
- Benchmark: 2006-06-12 12:09:25
- -------------------------------------------------------------------------------
- Rounds: 10
- Warp: 10
- Timer: time.time
- Machine Details:
- Platform ID: Linux-2.6.8-24.19-default-x86_64-with-SuSE-9.2-x86-64
- Processor: x86_64
- Python:
- Executable: /usr/local/bin/python
- Version: 2.4.2
- Compiler: GCC 3.3.4 (pre 3.3.5 20040809)
- Bits: 64bit
- Build: Oct 1 2005 15:24:35 (#1)
- Unicode: UCS2
- Test minimum average operation overhead
- -------------------------------------------------------------------------------
- BuiltinFunctionCalls: 126ms 145ms 0.28us 0.274ms
- BuiltinMethodLookup: 124ms 130ms 0.12us 0.316ms
- CompareFloats: 109ms 110ms 0.09us 0.361ms
- CompareFloatsIntegers: 100ms 104ms 0.12us 0.271ms
- CompareIntegers: 137ms 138ms 0.08us 0.542ms
- CompareInternedStrings: 124ms 127ms 0.08us 1.367ms
- CompareLongs: 100ms 104ms 0.10us 0.316ms
- CompareStrings: 111ms 115ms 0.12us 0.929ms
- CompareUnicode: 108ms 128ms 0.17us 0.693ms
- ConcatStrings: 142ms 155ms 0.31us 0.562ms
- ConcatUnicode: 119ms 127ms 0.42us 0.384ms
- CreateInstances: 123ms 128ms 1.14us 0.367ms
- CreateNewInstances: 121ms 126ms 1.49us 0.335ms
- CreateStringsWithConcat: 130ms 135ms 0.14us 0.916ms
- CreateUnicodeWithConcat: 130ms 135ms 0.34us 0.361ms
- DictCreation: 108ms 109ms 0.27us 0.361ms
- DictWithFloatKeys: 149ms 153ms 0.17us 0.678ms
- DictWithIntegerKeys: 124ms 126ms 0.11us 0.915ms
- DictWithStringKeys: 114ms 117ms 0.10us 0.905ms
- ForLoops: 110ms 111ms 4.46us 0.063ms
- IfThenElse: 118ms 119ms 0.09us 0.685ms
- ListSlicing: 116ms 120ms 8.59us 0.103ms
- NestedForLoops: 125ms 137ms 0.09us 0.019ms
- NormalClassAttribute: 124ms 136ms 0.11us 0.457ms
- NormalInstanceAttribute: 110ms 117ms 0.10us 0.454ms
- PythonFunctionCalls: 107ms 113ms 0.34us 0.271ms
- PythonMethodCalls: 140ms 149ms 0.66us 0.141ms
- Recursion: 156ms 166ms 3.32us 0.452ms
- SecondImport: 112ms 118ms 1.18us 0.180ms
- SecondPackageImport: 118ms 127ms 1.27us 0.180ms
- SecondSubmoduleImport: 140ms 151ms 1.51us 0.180ms
- SimpleComplexArithmetic: 128ms 139ms 0.16us 0.361ms
- SimpleDictManipulation: 134ms 136ms 0.11us 0.452ms
- SimpleFloatArithmetic: 110ms 113ms 0.09us 0.571ms
- SimpleIntFloatArithmetic: 106ms 111ms 0.08us 0.548ms
- SimpleIntegerArithmetic: 106ms 109ms 0.08us 0.544ms
- SimpleListManipulation: 103ms 113ms 0.10us 0.587ms
- SimpleLongArithmetic: 112ms 118ms 0.18us 0.271ms
- SmallLists: 105ms 116ms 0.17us 0.366ms
- SmallTuples: 108ms 128ms 0.24us 0.406ms
- SpecialClassAttribute: 119ms 136ms 0.11us 0.453ms
- SpecialInstanceAttribute: 143ms 155ms 0.13us 0.454ms
- StringMappings: 115ms 121ms 0.48us 0.405ms
- StringPredicates: 120ms 129ms 0.18us 2.064ms
- StringSlicing: 111ms 127ms 0.23us 0.781ms
- TryExcept: 125ms 126ms 0.06us 0.681ms
- TryRaiseExcept: 133ms 137ms 2.14us 0.361ms
- TupleSlicing: 117ms 120ms 0.46us 0.066ms
- UnicodeMappings: 156ms 160ms 4.44us 0.429ms
- UnicodePredicates: 117ms 121ms 0.22us 2.487ms
- UnicodeProperties: 115ms 153ms 0.38us 2.070ms
- UnicodeSlicing: 126ms 129ms 0.26us 0.689ms
- -------------------------------------------------------------------------------
- Totals: 6283ms 6673ms
- """
- ________________________________________________________________________
- Writing New Tests
- ________________________________________________________________________
- pybench tests are simple modules defining one or more pybench.Test
- subclasses.
- Writing a test essentially boils down to providing two methods:
- .test() which runs .rounds number of .operations test operations each
- and .calibrate() which does the same except that it doesn't actually
- execute the operations.
- Here's an example:
- ------------------
- from pybench import Test
- class IntegerCounting(Test):
- # Version number of the test as float (x.yy); this is important
- # for comparisons of benchmark runs - tests with unequal version
- # number will not get compared.
- version = 1.0
-
- # The number of abstract operations done in each round of the
- # test. An operation is the basic unit of what you want to
- # measure. The benchmark will output the amount of run-time per
- # operation. Note that in order to raise the measured timings
- # significantly above noise level, it is often required to repeat
- # sets of operations more than once per test round. The measured
- # overhead per test round should be less than 1 second.
- operations = 20
- # Number of rounds to execute per test run. This should be
- # adjusted to a figure that results in a test run-time of between
- # 1-2 seconds (at warp 1).
- rounds = 100000
- def test(self):
- """ Run the test.
- The test needs to run self.rounds executing
- self.operations number of operations each.
- """
- # Init the test
- a = 1
- # Run test rounds
- #
- # NOTE: Use xrange() for all test loops unless you want to face
- # a 20MB process !
- #
- for i in xrange(self.rounds):
- # Repeat the operations per round to raise the run-time
- # per operation significantly above the noise level of the
- # for-loop overhead.
- # Execute 20 operations (a += 1):
- a += 1
- a += 1
- a += 1
- a += 1
- a += 1
- a += 1
- a += 1
- a += 1
- a += 1
- a += 1
- a += 1
- a += 1
- a += 1
- a += 1
- a += 1
- a += 1
- a += 1
- a += 1
- a += 1
- a += 1
- def calibrate(self):
- """ Calibrate the test.
- This method should execute everything that is needed to
- setup and run the test - except for the actual operations
- that you intend to measure. pybench uses this method to
- measure the test implementation overhead.
- """
- # Init the test
- a = 1
- # Run test rounds (without actually doing any operation)
- for i in xrange(self.rounds):
- # Skip the actual execution of the operations, since we
- # only want to measure the test's administration overhead.
- pass
- Registering a new test module
- -----------------------------
- To register a test module with pybench, the classes need to be
- imported into the pybench.Setup module. pybench will then scan all the
- symbols defined in that module for subclasses of pybench.Test and
- automatically add them to the benchmark suite.
- Breaking Comparability
- ----------------------
- If a change is made to any individual test that means it is no
- longer strictly comparable with previous runs, the '.version' class
- variable should be updated. Therefafter, comparisons with previous
- versions of the test will list as "n/a" to reflect the change.
- Version History
- ---------------
- 2.0: rewrote parts of pybench which resulted in more repeatable
- timings:
- - made timer a parameter
- - changed the platform default timer to use high-resolution
- timers rather than process timers (which have a much lower
- resolution)
- - added option to select timer
- - added process time timer (using systimes.py)
- - changed to use min() as timing estimator (average
- is still taken as well to provide an idea of the difference)
- - garbage collection is turned off per default
- - sys check interval is set to the highest possible value
- - calibration is now a separate step and done using
- a different strategy that allows measuring the test
- overhead more accurately
- - modified the tests to each give a run-time of between
- 100-200ms using warp 10
- - changed default warp factor to 10 (from 20)
- - compared results with timeit.py and confirmed measurements
- - bumped all test versions to 2.0
- - updated platform.py to the latest version
- - changed the output format a bit to make it look
- nicer
- - refactored the APIs somewhat
- 1.3+: Steve Holden added the NewInstances test and the filtering
- option during the NeedForSpeed sprint; this also triggered a long
- discussion on how to improve benchmark timing and finally
- resulted in the release of 2.0
- 1.3: initial checkin into the Python SVN repository
- Have fun,
- --
- Marc-Andre Lemburg
- mal@lemburg.com