PageRenderTime 14ms CodeModel.GetById 141ms app.highlight 168ms RepoModel.GetById 33ms app.codeStats 1ms

/Doc/whatsnew/2.3.rst

http://unladen-swallow.googlecode.com/
ReStructuredText | 2078 lines | 1595 code | 483 blank | 0 comment | 0 complexity | 8db9b7b78675d404b6b3341c06d9e9af MD5 | raw file
   1****************************
   2  What's New in Python 2.3
   3****************************
   4
   5:Author: A.M. Kuchling
   6
   7.. |release| replace:: 1.01
   8
   9.. $Id: whatsnew23.tex 54631 2007-03-31 11:58:36Z georg.brandl $
  10
  11This article explains the new features in Python 2.3.  Python 2.3 was released
  12on July 29, 2003.
  13
  14The main themes for Python 2.3 are polishing some of the features added in 2.2,
  15adding various small but useful enhancements to the core language, and expanding
  16the standard library.  The new object model introduced in the previous version
  17has benefited from 18 months of bugfixes and from optimization efforts that have
  18improved the performance of new-style classes.  A few new built-in functions
  19have been added such as :func:`sum` and :func:`enumerate`.  The :keyword:`in`
  20operator can now be used for substring searches (e.g. ``"ab" in "abc"`` returns
  21:const:`True`).
  22
  23Some of the many new library features include Boolean, set, heap, and date/time
  24data types, the ability to import modules from ZIP-format archives, metadata
  25support for the long-awaited Python catalog, an updated version of IDLE, and
  26modules for logging messages, wrapping text, parsing CSV files, processing
  27command-line options, using BerkeleyDB databases...  the list of new and
  28enhanced modules is lengthy.
  29
  30This article doesn't attempt to provide a complete specification of the new
  31features, but instead provides a convenient overview.  For full details, you
  32should refer to the documentation for Python 2.3, such as the Python Library
  33Reference and the Python Reference Manual.  If you want to understand the
  34complete implementation and design rationale, refer to the PEP for a particular
  35new feature.
  36
  37.. ======================================================================
  38
  39
  40PEP 218: A Standard Set Datatype
  41================================
  42
  43The new :mod:`sets` module contains an implementation of a set datatype.  The
  44:class:`Set` class is for mutable sets, sets that can have members added and
  45removed.  The :class:`ImmutableSet` class is for sets that can't be modified,
  46and instances of :class:`ImmutableSet` can therefore be used as dictionary keys.
  47Sets are built on top of dictionaries, so the elements within a set must be
  48hashable.
  49
  50Here's a simple example::
  51
  52   >>> import sets
  53   >>> S = sets.Set([1,2,3])
  54   >>> S
  55   Set([1, 2, 3])
  56   >>> 1 in S
  57   True
  58   >>> 0 in S
  59   False
  60   >>> S.add(5)
  61   >>> S.remove(3)
  62   >>> S
  63   Set([1, 2, 5])
  64   >>>
  65
  66The union and intersection of sets can be computed with the :meth:`union` and
  67:meth:`intersection` methods; an alternative notation uses the bitwise operators
  68``&`` and ``|``. Mutable sets also have in-place versions of these methods,
  69:meth:`union_update` and :meth:`intersection_update`. ::
  70
  71   >>> S1 = sets.Set([1,2,3])
  72   >>> S2 = sets.Set([4,5,6])
  73   >>> S1.union(S2)
  74   Set([1, 2, 3, 4, 5, 6])
  75   >>> S1 | S2                  # Alternative notation
  76   Set([1, 2, 3, 4, 5, 6])
  77   >>> S1.intersection(S2)
  78   Set([])
  79   >>> S1 & S2                  # Alternative notation
  80   Set([])
  81   >>> S1.union_update(S2)
  82   >>> S1
  83   Set([1, 2, 3, 4, 5, 6])
  84   >>>
  85
  86It's also possible to take the symmetric difference of two sets.  This is the
  87set of all elements in the union that aren't in the intersection.  Another way
  88of putting it is that the symmetric difference contains all elements that are in
  89exactly one set.  Again, there's an alternative notation (``^``), and an in-
  90place version with the ungainly name :meth:`symmetric_difference_update`. ::
  91
  92   >>> S1 = sets.Set([1,2,3,4])
  93   >>> S2 = sets.Set([3,4,5,6])
  94   >>> S1.symmetric_difference(S2)
  95   Set([1, 2, 5, 6])
  96   >>> S1 ^ S2
  97   Set([1, 2, 5, 6])
  98   >>>
  99
 100There are also :meth:`issubset` and :meth:`issuperset` methods for checking
 101whether one set is a subset or superset of another::
 102
 103   >>> S1 = sets.Set([1,2,3])
 104   >>> S2 = sets.Set([2,3])
 105   >>> S2.issubset(S1)
 106   True
 107   >>> S1.issubset(S2)
 108   False
 109   >>> S1.issuperset(S2)
 110   True
 111   >>>
 112
 113
 114.. seealso::
 115
 116   :pep:`218` - Adding a Built-In Set Object Type
 117      PEP written by Greg V. Wilson. Implemented by Greg V. Wilson, Alex Martelli, and
 118      GvR.
 119
 120.. ======================================================================
 121
 122
 123.. _section-generators:
 124
 125PEP 255: Simple Generators
 126==========================
 127
 128In Python 2.2, generators were added as an optional feature, to be enabled by a
 129``from __future__ import generators`` directive.  In 2.3 generators no longer
 130need to be specially enabled, and are now always present; this means that
 131:keyword:`yield` is now always a keyword.  The rest of this section is a copy of
 132the description of generators from the "What's New in Python 2.2" document; if
 133you read it back when Python 2.2 came out, you can skip the rest of this
 134section.
 135
 136You're doubtless familiar with how function calls work in Python or C. When you
 137call a function, it gets a private namespace where its local variables are
 138created.  When the function reaches a :keyword:`return` statement, the local
 139variables are destroyed and the resulting value is returned to the caller.  A
 140later call to the same function will get a fresh new set of local variables.
 141But, what if the local variables weren't thrown away on exiting a function?
 142What if you could later resume the function where it left off?  This is what
 143generators provide; they can be thought of as resumable functions.
 144
 145Here's the simplest example of a generator function::
 146
 147   def generate_ints(N):
 148       for i in range(N):
 149           yield i
 150
 151A new keyword, :keyword:`yield`, was introduced for generators.  Any function
 152containing a :keyword:`yield` statement is a generator function; this is
 153detected by Python's bytecode compiler which compiles the function specially as
 154a result.
 155
 156When you call a generator function, it doesn't return a single value; instead it
 157returns a generator object that supports the iterator protocol.  On executing
 158the :keyword:`yield` statement, the generator outputs the value of ``i``,
 159similar to a :keyword:`return` statement.  The big difference between
 160:keyword:`yield` and a :keyword:`return` statement is that on reaching a
 161:keyword:`yield` the generator's state of execution is suspended and local
 162variables are preserved.  On the next call to the generator's ``.next()``
 163method, the function will resume executing immediately after the
 164:keyword:`yield` statement.  (For complicated reasons, the :keyword:`yield`
 165statement isn't allowed inside the :keyword:`try` block of a :keyword:`try`...\
 166:keyword:`finally` statement; read :pep:`255` for a full explanation of the
 167interaction between :keyword:`yield` and exceptions.)
 168
 169Here's a sample usage of the :func:`generate_ints` generator::
 170
 171   >>> gen = generate_ints(3)
 172   >>> gen
 173   <generator object at 0x8117f90>
 174   >>> gen.next()
 175   0
 176   >>> gen.next()
 177   1
 178   >>> gen.next()
 179   2
 180   >>> gen.next()
 181   Traceback (most recent call last):
 182     File "stdin", line 1, in ?
 183     File "stdin", line 2, in generate_ints
 184   StopIteration
 185
 186You could equally write ``for i in generate_ints(5)``, or ``a,b,c =
 187generate_ints(3)``.
 188
 189Inside a generator function, the :keyword:`return` statement can only be used
 190without a value, and signals the end of the procession of values; afterwards the
 191generator cannot return any further values. :keyword:`return` with a value, such
 192as ``return 5``, is a syntax error inside a generator function.  The end of the
 193generator's results can also be indicated by raising :exc:`StopIteration`
 194manually, or by just letting the flow of execution fall off the bottom of the
 195function.
 196
 197You could achieve the effect of generators manually by writing your own class
 198and storing all the local variables of the generator as instance variables.  For
 199example, returning a list of integers could be done by setting ``self.count`` to
 2000, and having the :meth:`next` method increment ``self.count`` and return it.
 201However, for a moderately complicated generator, writing a corresponding class
 202would be much messier. :file:`Lib/test/test_generators.py` contains a number of
 203more interesting examples.  The simplest one implements an in-order traversal of
 204a tree using generators recursively. ::
 205
 206   # A recursive generator that generates Tree leaves in in-order.
 207   def inorder(t):
 208       if t:
 209           for x in inorder(t.left):
 210               yield x
 211           yield t.label
 212           for x in inorder(t.right):
 213               yield x
 214
 215Two other examples in :file:`Lib/test/test_generators.py` produce solutions for
 216the N-Queens problem (placing $N$ queens on an $NxN$ chess board so that no
 217queen threatens another) and the Knight's Tour (a route that takes a knight to
 218every square of an $NxN$ chessboard without visiting any square twice).
 219
 220The idea of generators comes from other programming languages, especially Icon
 221(http://www.cs.arizona.edu/icon/), where the idea of generators is central.  In
 222Icon, every expression and function call behaves like a generator.  One example
 223from "An Overview of the Icon Programming Language" at
 224http://www.cs.arizona.edu/icon/docs/ipd266.htm gives an idea of what this looks
 225like::
 226
 227   sentence := "Store it in the neighboring harbor"
 228   if (i := find("or", sentence)) > 5 then write(i)
 229
 230In Icon the :func:`find` function returns the indexes at which the substring
 231"or" is found: 3, 23, 33.  In the :keyword:`if` statement, ``i`` is first
 232assigned a value of 3, but 3 is less than 5, so the comparison fails, and Icon
 233retries it with the second value of 23.  23 is greater than 5, so the comparison
 234now succeeds, and the code prints the value 23 to the screen.
 235
 236Python doesn't go nearly as far as Icon in adopting generators as a central
 237concept.  Generators are considered part of the core Python language, but
 238learning or using them isn't compulsory; if they don't solve any problems that
 239you have, feel free to ignore them. One novel feature of Python's interface as
 240compared to Icon's is that a generator's state is represented as a concrete
 241object (the iterator) that can be passed around to other functions or stored in
 242a data structure.
 243
 244
 245.. seealso::
 246
 247   :pep:`255` - Simple Generators
 248      Written by Neil Schemenauer, Tim Peters, Magnus Lie Hetland.  Implemented mostly
 249      by Neil Schemenauer and Tim Peters, with other fixes from the Python Labs crew.
 250
 251.. ======================================================================
 252
 253
 254.. _section-encodings:
 255
 256PEP 263: Source Code Encodings
 257==============================
 258
 259Python source files can now be declared as being in different character set
 260encodings.  Encodings are declared by including a specially formatted comment in
 261the first or second line of the source file.  For example, a UTF-8 file can be
 262declared with::
 263
 264   #!/usr/bin/env python
 265   # -*- coding: UTF-8 -*-
 266
 267Without such an encoding declaration, the default encoding used is 7-bit ASCII.
 268Executing or importing modules that contain string literals with 8-bit
 269characters and have no encoding declaration will result in a
 270:exc:`DeprecationWarning` being signalled by Python 2.3; in 2.4 this will be a
 271syntax error.
 272
 273The encoding declaration only affects Unicode string literals, which will be
 274converted to Unicode using the specified encoding.  Note that Python identifiers
 275are still restricted to ASCII characters, so you can't have variable names that
 276use characters outside of the usual alphanumerics.
 277
 278
 279.. seealso::
 280
 281   :pep:`263` - Defining Python Source Code Encodings
 282      Written by Marc-André Lemburg and Martin von Löwis; implemented by Suzuki Hisao
 283      and Martin von Löwis.
 284
 285.. ======================================================================
 286
 287
 288PEP 273: Importing Modules from ZIP Archives
 289============================================
 290
 291The new :mod:`zipimport` module adds support for importing modules from a ZIP-
 292format archive.  You don't need to import the module explicitly; it will be
 293automatically imported if a ZIP archive's filename is added to ``sys.path``.
 294For example::
 295
 296   amk@nyman:~/src/python$ unzip -l /tmp/example.zip
 297   Archive:  /tmp/example.zip
 298     Length     Date   Time    Name
 299    --------    ----   ----    ----
 300        8467  11-26-02 22:30   jwzthreading.py
 301    --------                   -------
 302        8467                   1 file
 303   amk@nyman:~/src/python$ ./python
 304   Python 2.3 (#1, Aug 1 2003, 19:54:32)
 305   >>> import sys
 306   >>> sys.path.insert(0, '/tmp/example.zip')  # Add .zip file to front of path
 307   >>> import jwzthreading
 308   >>> jwzthreading.__file__
 309   '/tmp/example.zip/jwzthreading.py'
 310   >>>
 311
 312An entry in ``sys.path`` can now be the filename of a ZIP archive. The ZIP
 313archive can contain any kind of files, but only files named :file:`\*.py`,
 314:file:`\*.pyc`, or :file:`\*.pyo` can be imported.  If an archive only contains
 315:file:`\*.py` files, Python will not attempt to modify the archive by adding the
 316corresponding :file:`\*.pyc` file, meaning that if a ZIP archive doesn't contain
 317:file:`\*.pyc` files, importing may be rather slow.
 318
 319A path within the archive can also be specified to only import from a
 320subdirectory; for example, the path :file:`/tmp/example.zip/lib/` would only
 321import from the :file:`lib/` subdirectory within the archive.
 322
 323
 324.. seealso::
 325
 326   :pep:`273` - Import Modules from Zip Archives
 327      Written by James C. Ahlstrom,  who also provided an implementation. Python 2.3
 328      follows the specification in :pep:`273`,  but uses an implementation written by
 329      Just van Rossum  that uses the import hooks described in :pep:`302`. See section
 330      :ref:`section-pep302` for a description of the new import hooks.
 331
 332.. ======================================================================
 333
 334
 335PEP 277: Unicode file name support for Windows NT
 336=================================================
 337
 338On Windows NT, 2000, and XP, the system stores file names as Unicode strings.
 339Traditionally, Python has represented file names as byte strings, which is
 340inadequate because it renders some file names inaccessible.
 341
 342Python now allows using arbitrary Unicode strings (within the limitations of the
 343file system) for all functions that expect file names, most notably the
 344:func:`open` built-in function. If a Unicode string is passed to
 345:func:`os.listdir`, Python now returns a list of Unicode strings.  A new
 346function, :func:`os.getcwdu`, returns the current directory as a Unicode string.
 347
 348Byte strings still work as file names, and on Windows Python will transparently
 349convert them to Unicode using the ``mbcs`` encoding.
 350
 351Other systems also allow Unicode strings as file names but convert them to byte
 352strings before passing them to the system, which can cause a :exc:`UnicodeError`
 353to be raised. Applications can test whether arbitrary Unicode strings are
 354supported as file names by checking :attr:`os.path.supports_unicode_filenames`,
 355a Boolean value.
 356
 357Under MacOS, :func:`os.listdir` may now return Unicode filenames.
 358
 359
 360.. seealso::
 361
 362   :pep:`277` - Unicode file name support for Windows NT
 363      Written by Neil Hodgson; implemented by Neil Hodgson, Martin von Löwis, and Mark
 364      Hammond.
 365
 366.. ======================================================================
 367
 368
 369PEP 278: Universal Newline Support
 370==================================
 371
 372The three major operating systems used today are Microsoft Windows, Apple's
 373Macintosh OS, and the various Unix derivatives.  A minor irritation of cross-
 374platform work  is that these three platforms all use different characters to
 375mark the ends of lines in text files.  Unix uses the linefeed (ASCII character
 37610), MacOS uses the carriage return (ASCII character 13), and Windows uses a
 377two-character sequence of a carriage return plus a newline.
 378
 379Python's file objects can now support end of line conventions other than the one
 380followed by the platform on which Python is running. Opening a file with the
 381mode ``'U'`` or ``'rU'`` will open a file for reading in universal newline mode.
 382All three line ending conventions will be translated to a ``'\n'`` in the
 383strings returned by the various file methods such as :meth:`read` and
 384:meth:`readline`.
 385
 386Universal newline support is also used when importing modules and when executing
 387a file with the :func:`execfile` function.  This means that Python modules can
 388be shared between all three operating systems without needing to convert the
 389line-endings.
 390
 391This feature can be disabled when compiling Python by specifying the
 392:option:`--without-universal-newlines` switch when running Python's
 393:program:`configure` script.
 394
 395
 396.. seealso::
 397
 398   :pep:`278` - Universal Newline Support
 399      Written and implemented by Jack Jansen.
 400
 401.. ======================================================================
 402
 403
 404.. _section-enumerate:
 405
 406PEP 279: enumerate()
 407====================
 408
 409A new built-in function, :func:`enumerate`, will make certain loops a bit
 410clearer.  ``enumerate(thing)``, where *thing* is either an iterator or a
 411sequence, returns a iterator that will return ``(0, thing[0])``, ``(1,
 412thing[1])``, ``(2, thing[2])``, and so forth.
 413
 414A common idiom to change every element of a list looks like this::
 415
 416   for i in range(len(L)):
 417       item = L[i]
 418       # ... compute some result based on item ...
 419       L[i] = result
 420
 421This can be rewritten using :func:`enumerate` as::
 422
 423   for i, item in enumerate(L):
 424       # ... compute some result based on item ...
 425       L[i] = result
 426
 427
 428.. seealso::
 429
 430   :pep:`279` - The enumerate() built-in function
 431      Written and implemented by Raymond D. Hettinger.
 432
 433.. ======================================================================
 434
 435
 436PEP 282: The logging Package
 437============================
 438
 439A standard package for writing logs, :mod:`logging`, has been added to Python
 4402.3.  It provides a powerful and flexible mechanism for generating logging
 441output which can then be filtered and processed in various ways.  A
 442configuration file written in a standard format can be used to control the
 443logging behavior of a program.  Python includes handlers that will write log
 444records to standard error or to a file or socket, send them to the system log,
 445or even e-mail them to a particular address; of course, it's also possible to
 446write your own handler classes.
 447
 448The :class:`Logger` class is the primary class. Most application code will deal
 449with one or more :class:`Logger` objects, each one used by a particular
 450subsystem of the application. Each :class:`Logger` is identified by a name, and
 451names are organized into a hierarchy using ``.``  as the component separator.
 452For example, you might have :class:`Logger` instances named ``server``,
 453``server.auth`` and ``server.network``.  The latter two instances are below
 454``server`` in the hierarchy.  This means that if you turn up the verbosity for
 455``server`` or direct ``server`` messages to a different handler, the changes
 456will also apply to records logged to ``server.auth`` and ``server.network``.
 457There's also a root :class:`Logger` that's the parent of all other loggers.
 458
 459For simple uses, the :mod:`logging` package contains some convenience functions
 460that always use the root log::
 461
 462   import logging
 463
 464   logging.debug('Debugging information')
 465   logging.info('Informational message')
 466   logging.warning('Warning:config file %s not found', 'server.conf')
 467   logging.error('Error occurred')
 468   logging.critical('Critical error -- shutting down')
 469
 470This produces the following output::
 471
 472   WARNING:root:Warning:config file server.conf not found
 473   ERROR:root:Error occurred
 474   CRITICAL:root:Critical error -- shutting down
 475
 476In the default configuration, informational and debugging messages are
 477suppressed and the output is sent to standard error.  You can enable the display
 478of informational and debugging messages by calling the :meth:`setLevel` method
 479on the root logger.
 480
 481Notice the :func:`warning` call's use of string formatting operators; all of the
 482functions for logging messages take the arguments ``(msg, arg1, arg2, ...)`` and
 483log the string resulting from ``msg % (arg1, arg2, ...)``.
 484
 485There's also an :func:`exception` function that records the most recent
 486traceback.  Any of the other functions will also record the traceback if you
 487specify a true value for the keyword argument *exc_info*. ::
 488
 489   def f():
 490       try:    1/0
 491       except: logging.exception('Problem recorded')
 492
 493   f()
 494
 495This produces the following output::
 496
 497   ERROR:root:Problem recorded
 498   Traceback (most recent call last):
 499     File "t.py", line 6, in f
 500       1/0
 501   ZeroDivisionError: integer division or modulo by zero
 502
 503Slightly more advanced programs will use a logger other than the root logger.
 504The :func:`getLogger(name)` function is used to get a particular log, creating
 505it if it doesn't exist yet. :func:`getLogger(None)` returns the root logger. ::
 506
 507   log = logging.getLogger('server')
 508    ...
 509   log.info('Listening on port %i', port)
 510    ...
 511   log.critical('Disk full')
 512    ...
 513
 514Log records are usually propagated up the hierarchy, so a message logged to
 515``server.auth`` is also seen by ``server`` and ``root``, but a :class:`Logger`
 516can prevent this by setting its :attr:`propagate` attribute to :const:`False`.
 517
 518There are more classes provided by the :mod:`logging` package that can be
 519customized.  When a :class:`Logger` instance is told to log a message, it
 520creates a :class:`LogRecord` instance that is sent to any number of different
 521:class:`Handler` instances.  Loggers and handlers can also have an attached list
 522of filters, and each filter can cause the :class:`LogRecord` to be ignored or
 523can modify the record before passing it along.  When they're finally output,
 524:class:`LogRecord` instances are converted to text by a :class:`Formatter`
 525class.  All of these classes can be replaced by your own specially-written
 526classes.
 527
 528With all of these features the :mod:`logging` package should provide enough
 529flexibility for even the most complicated applications.  This is only an
 530incomplete overview of its features, so please see the package's reference
 531documentation for all of the details.  Reading :pep:`282` will also be helpful.
 532
 533
 534.. seealso::
 535
 536   :pep:`282` - A Logging System
 537      Written by Vinay Sajip and Trent Mick; implemented by Vinay Sajip.
 538
 539.. ======================================================================
 540
 541
 542.. _section-bool:
 543
 544PEP 285: A Boolean Type
 545=======================
 546
 547A Boolean type was added to Python 2.3.  Two new constants were added to the
 548:mod:`__builtin__` module, :const:`True` and :const:`False`.  (:const:`True` and
 549:const:`False` constants were added to the built-ins in Python 2.2.1, but the
 5502.2.1 versions are simply set to integer values of 1 and 0 and aren't a
 551different type.)
 552
 553The type object for this new type is named :class:`bool`; the constructor for it
 554takes any Python value and converts it to :const:`True` or :const:`False`. ::
 555
 556   >>> bool(1)
 557   True
 558   >>> bool(0)
 559   False
 560   >>> bool([])
 561   False
 562   >>> bool( (1,) )
 563   True
 564
 565Most of the standard library modules and built-in functions have been changed to
 566return Booleans. ::
 567
 568   >>> obj = []
 569   >>> hasattr(obj, 'append')
 570   True
 571   >>> isinstance(obj, list)
 572   True
 573   >>> isinstance(obj, tuple)
 574   False
 575
 576Python's Booleans were added with the primary goal of making code clearer.  For
 577example, if you're reading a function and encounter the statement ``return 1``,
 578you might wonder whether the ``1`` represents a Boolean truth value, an index,
 579or a coefficient that multiplies some other quantity.  If the statement is
 580``return True``, however, the meaning of the return value is quite clear.
 581
 582Python's Booleans were *not* added for the sake of strict type-checking.  A very
 583strict language such as Pascal would also prevent you performing arithmetic with
 584Booleans, and would require that the expression in an :keyword:`if` statement
 585always evaluate to a Boolean result.  Python is not this strict and never will
 586be, as :pep:`285` explicitly says.  This means you can still use any expression
 587in an :keyword:`if` statement, even ones that evaluate to a list or tuple or
 588some random object.  The Boolean type is a subclass of the :class:`int` class so
 589that arithmetic using a Boolean still works. ::
 590
 591   >>> True + 1
 592   2
 593   >>> False + 1
 594   1
 595   >>> False * 75
 596   0
 597   >>> True * 75
 598   75
 599
 600To sum up :const:`True` and :const:`False` in a sentence: they're alternative
 601ways to spell the integer values 1 and 0, with the single difference that
 602:func:`str` and :func:`repr` return the strings ``'True'`` and ``'False'``
 603instead of ``'1'`` and ``'0'``.
 604
 605
 606.. seealso::
 607
 608   :pep:`285` - Adding a bool type
 609      Written and implemented by GvR.
 610
 611.. ======================================================================
 612
 613
 614PEP 293: Codec Error Handling Callbacks
 615=======================================
 616
 617When encoding a Unicode string into a byte string, unencodable characters may be
 618encountered.  So far, Python has allowed specifying the error processing as
 619either "strict" (raising :exc:`UnicodeError`), "ignore" (skipping the
 620character), or "replace" (using a question mark in the output string), with
 621"strict" being the default behavior. It may be desirable to specify alternative
 622processing of such errors, such as inserting an XML character reference or HTML
 623entity reference into the converted string.
 624
 625Python now has a flexible framework to add different processing strategies.  New
 626error handlers can be added with :func:`codecs.register_error`, and codecs then
 627can access the error handler with :func:`codecs.lookup_error`. An equivalent C
 628API has been added for codecs written in C. The error handler gets the necessary
 629state information such as the string being converted, the position in the string
 630where the error was detected, and the target encoding.  The handler can then
 631either raise an exception or return a replacement string.
 632
 633Two additional error handlers have been implemented using this framework:
 634"backslashreplace" uses Python backslash quoting to represent unencodable
 635characters and "xmlcharrefreplace" emits XML character references.
 636
 637
 638.. seealso::
 639
 640   :pep:`293` - Codec Error Handling Callbacks
 641      Written and implemented by Walter Dörwald.
 642
 643.. ======================================================================
 644
 645
 646.. _section-pep301:
 647
 648PEP 301: Package Index and Metadata for Distutils
 649=================================================
 650
 651Support for the long-requested Python catalog makes its first appearance in 2.3.
 652
 653The heart of the catalog is the new Distutils :command:`register` command.
 654Running ``python setup.py register`` will collect the metadata describing a
 655package, such as its name, version, maintainer, description, &c., and send it to
 656a central catalog server.  The resulting catalog is available from
 657http://www.python.org/pypi.
 658
 659To make the catalog a bit more useful, a new optional *classifiers* keyword
 660argument has been added to the Distutils :func:`setup` function.  A list of
 661`Trove <http://catb.org/~esr/trove/>`_-style strings can be supplied to help
 662classify the software.
 663
 664Here's an example :file:`setup.py` with classifiers, written to be compatible
 665with older versions of the Distutils::
 666
 667   from distutils import core
 668   kw = {'name': "Quixote",
 669         'version': "0.5.1",
 670         'description': "A highly Pythonic Web application framework",
 671         # ...
 672         }
 673
 674   if (hasattr(core, 'setup_keywords') and
 675       'classifiers' in core.setup_keywords):
 676       kw['classifiers'] = \
 677           ['Topic :: Internet :: WWW/HTTP :: Dynamic Content',
 678            'Environment :: No Input/Output (Daemon)',
 679            'Intended Audience :: Developers'],
 680
 681   core.setup(**kw)
 682
 683The full list of classifiers can be obtained by running  ``python setup.py
 684register --list-classifiers``.
 685
 686
 687.. seealso::
 688
 689   :pep:`301` - Package Index and Metadata for Distutils
 690      Written and implemented by Richard Jones.
 691
 692.. ======================================================================
 693
 694
 695.. _section-pep302:
 696
 697PEP 302: New Import Hooks
 698=========================
 699
 700While it's been possible to write custom import hooks ever since the
 701:mod:`ihooks` module was introduced in Python 1.3, no one has ever been really
 702happy with it because writing new import hooks is difficult and messy.  There
 703have been various proposed alternatives such as the :mod:`imputil` and :mod:`iu`
 704modules, but none of them has ever gained much acceptance, and none of them were
 705easily usable from C code.
 706
 707:pep:`302` borrows ideas from its predecessors, especially from Gordon
 708McMillan's :mod:`iu` module.  Three new items  are added to the :mod:`sys`
 709module:
 710
 711* ``sys.path_hooks`` is a list of callable objects; most  often they'll be
 712  classes.  Each callable takes a string containing a path and either returns an
 713  importer object that will handle imports from this path or raises an
 714  :exc:`ImportError` exception if it can't handle this path.
 715
 716* ``sys.path_importer_cache`` caches importer objects for each path, so
 717  ``sys.path_hooks`` will only need to be traversed once for each path.
 718
 719* ``sys.meta_path`` is a list of importer objects that will be traversed before
 720  ``sys.path`` is checked.  This list is initially empty, but user code can add
 721  objects to it.  Additional built-in and frozen modules can be imported by an
 722  object added to this list.
 723
 724Importer objects must have a single method, :meth:`find_module(fullname,
 725path=None)`.  *fullname* will be a module or package name, e.g. ``string`` or
 726``distutils.core``.  :meth:`find_module` must return a loader object that has a
 727single method, :meth:`load_module(fullname)`, that creates and returns the
 728corresponding module object.
 729
 730Pseudo-code for Python's new import logic, therefore, looks something like this
 731(simplified a bit; see :pep:`302` for the full details)::
 732
 733   for mp in sys.meta_path:
 734       loader = mp(fullname)
 735       if loader is not None:
 736           <module> = loader.load_module(fullname)
 737
 738   for path in sys.path:
 739       for hook in sys.path_hooks:
 740           try:
 741               importer = hook(path)
 742           except ImportError:
 743               # ImportError, so try the other path hooks
 744               pass
 745           else:
 746               loader = importer.find_module(fullname)
 747               <module> = loader.load_module(fullname)
 748
 749   # Not found!
 750   raise ImportError
 751
 752
 753.. seealso::
 754
 755   :pep:`302` - New Import Hooks
 756      Written by Just van Rossum and Paul Moore. Implemented by Just van Rossum.
 757
 758.. ======================================================================
 759
 760
 761.. _section-pep305:
 762
 763PEP 305: Comma-separated Files
 764==============================
 765
 766Comma-separated files are a format frequently used for exporting data from
 767databases and spreadsheets.  Python 2.3 adds a parser for comma-separated files.
 768
 769Comma-separated format is deceptively simple at first glance::
 770
 771   Costs,150,200,3.95
 772
 773Read a line and call ``line.split(',')``: what could be simpler? But toss in
 774string data that can contain commas, and things get more complicated::
 775
 776   "Costs",150,200,3.95,"Includes taxes, shipping, and sundry items"
 777
 778A big ugly regular expression can parse this, but using the new  :mod:`csv`
 779package is much simpler::
 780
 781   import csv
 782
 783   input = open('datafile', 'rb')
 784   reader = csv.reader(input)
 785   for line in reader:
 786       print line
 787
 788The :func:`reader` function takes a number of different options. The field
 789separator isn't limited to the comma and can be changed to any character, and so
 790can the quoting and line-ending characters.
 791
 792Different dialects of comma-separated files can be defined and registered;
 793currently there are two dialects, both used by Microsoft Excel. A separate
 794:class:`csv.writer` class will generate comma-separated files from a succession
 795of tuples or lists, quoting strings that contain the delimiter.
 796
 797
 798.. seealso::
 799
 800   :pep:`305` - CSV File API
 801      Written and implemented  by Kevin Altis, Dave Cole, Andrew McNamara, Skip
 802      Montanaro, Cliff Wells.
 803
 804.. ======================================================================
 805
 806
 807.. _section-pep307:
 808
 809PEP 307: Pickle Enhancements
 810============================
 811
 812The :mod:`pickle` and :mod:`cPickle` modules received some attention during the
 8132.3 development cycle.  In 2.2, new-style classes could be pickled without
 814difficulty, but they weren't pickled very compactly; :pep:`307` quotes a trivial
 815example where a new-style class results in a pickled string three times longer
 816than that for a classic class.
 817
 818The solution was to invent a new pickle protocol.  The :func:`pickle.dumps`
 819function has supported a text-or-binary flag  for a long time.  In 2.3, this
 820flag is redefined from a Boolean to an integer: 0 is the old text-mode pickle
 821format, 1 is the old binary format, and now 2 is a new 2.3-specific format.  A
 822new constant, :const:`pickle.HIGHEST_PROTOCOL`, can be used to select the
 823fanciest protocol available.
 824
 825Unpickling is no longer considered a safe operation.  2.2's :mod:`pickle`
 826provided hooks for trying to prevent unsafe classes from being unpickled
 827(specifically, a :attr:`__safe_for_unpickling__` attribute), but none of this
 828code was ever audited and therefore it's all been ripped out in 2.3.  You should
 829not unpickle untrusted data in any version of Python.
 830
 831To reduce the pickling overhead for new-style classes, a new interface for
 832customizing pickling was added using three special methods:
 833:meth:`__getstate__`, :meth:`__setstate__`, and :meth:`__getnewargs__`.  Consult
 834:pep:`307` for the full semantics  of these methods.
 835
 836As a way to compress pickles yet further, it's now possible to use integer codes
 837instead of long strings to identify pickled classes. The Python Software
 838Foundation will maintain a list of standardized codes; there's also a range of
 839codes for private use.  Currently no codes have been specified.
 840
 841
 842.. seealso::
 843
 844   :pep:`307` - Extensions to the pickle protocol
 845      Written and implemented  by Guido van Rossum and Tim Peters.
 846
 847.. ======================================================================
 848
 849
 850.. _section-slices:
 851
 852Extended Slices
 853===============
 854
 855Ever since Python 1.4, the slicing syntax has supported an optional third "step"
 856or "stride" argument.  For example, these are all legal Python syntax:
 857``L[1:10:2]``, ``L[:-1:1]``, ``L[::-1]``.  This was added to Python at the
 858request of the developers of Numerical Python, which uses the third argument
 859extensively.  However, Python's built-in list, tuple, and string sequence types
 860have never supported this feature, raising a :exc:`TypeError` if you tried it.
 861Michael Hudson contributed a patch to fix this shortcoming.
 862
 863For example, you can now easily extract the elements of a list that have even
 864indexes::
 865
 866   >>> L = range(10)
 867   >>> L[::2]
 868   [0, 2, 4, 6, 8]
 869
 870Negative values also work to make a copy of the same list in reverse order::
 871
 872   >>> L[::-1]
 873   [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
 874
 875This also works for tuples, arrays, and strings::
 876
 877   >>> s='abcd'
 878   >>> s[::2]
 879   'ac'
 880   >>> s[::-1]
 881   'dcba'
 882
 883If you have a mutable sequence such as a list or an array you can assign to or
 884delete an extended slice, but there are some differences between assignment to
 885extended and regular slices.  Assignment to a regular slice can be used to
 886change the length of the sequence::
 887
 888   >>> a = range(3)
 889   >>> a
 890   [0, 1, 2]
 891   >>> a[1:3] = [4, 5, 6]
 892   >>> a
 893   [0, 4, 5, 6]
 894
 895Extended slices aren't this flexible.  When assigning to an extended slice, the
 896list on the right hand side of the statement must contain the same number of
 897items as the slice it is replacing::
 898
 899   >>> a = range(4)
 900   >>> a
 901   [0, 1, 2, 3]
 902   >>> a[::2]
 903   [0, 2]
 904   >>> a[::2] = [0, -1]
 905   >>> a
 906   [0, 1, -1, 3]
 907   >>> a[::2] = [0,1,2]
 908   Traceback (most recent call last):
 909     File "<stdin>", line 1, in ?
 910   ValueError: attempt to assign sequence of size 3 to extended slice of size 2
 911
 912Deletion is more straightforward::
 913
 914   >>> a = range(4)
 915   >>> a
 916   [0, 1, 2, 3]
 917   >>> a[::2]
 918   [0, 2]
 919   >>> del a[::2]
 920   >>> a
 921   [1, 3]
 922
 923One can also now pass slice objects to the :meth:`__getitem__` methods of the
 924built-in sequences::
 925
 926   >>> range(10).__getitem__(slice(0, 5, 2))
 927   [0, 2, 4]
 928
 929Or use slice objects directly in subscripts::
 930
 931   >>> range(10)[slice(0, 5, 2)]
 932   [0, 2, 4]
 933
 934To simplify implementing sequences that support extended slicing, slice objects
 935now have a method :meth:`indices(length)` which, given the length of a sequence,
 936returns a ``(start, stop, step)`` tuple that can be passed directly to
 937:func:`range`. :meth:`indices` handles omitted and out-of-bounds indices in a
 938manner consistent with regular slices (and this innocuous phrase hides a welter
 939of confusing details!).  The method is intended to be used like this::
 940
 941   class FakeSeq:
 942       ...
 943       def calc_item(self, i):
 944           ...
 945       def __getitem__(self, item):
 946           if isinstance(item, slice):
 947               indices = item.indices(len(self))
 948               return FakeSeq([self.calc_item(i) for i in range(*indices)])
 949           else:
 950               return self.calc_item(i)
 951
 952From this example you can also see that the built-in :class:`slice` object is
 953now the type object for the slice type, and is no longer a function.  This is
 954consistent with Python 2.2, where :class:`int`, :class:`str`, etc., underwent
 955the same change.
 956
 957.. ======================================================================
 958
 959
 960Other Language Changes
 961======================
 962
 963Here are all of the changes that Python 2.3 makes to the core Python language.
 964
 965* The :keyword:`yield` statement is now always a keyword, as described in
 966  section :ref:`section-generators` of this document.
 967
 968* A new built-in function :func:`enumerate` was added, as described in section
 969  :ref:`section-enumerate` of this document.
 970
 971* Two new constants, :const:`True` and :const:`False` were added along with the
 972  built-in :class:`bool` type, as described in section :ref:`section-bool` of this
 973  document.
 974
 975* The :func:`int` type constructor will now return a long integer instead of
 976  raising an :exc:`OverflowError` when a string or floating-point number is too
 977  large to fit into an integer.  This can lead to the paradoxical result that
 978  ``isinstance(int(expression), int)`` is false, but that seems unlikely to cause
 979  problems in practice.
 980
 981* Built-in types now support the extended slicing syntax, as described in
 982  section :ref:`section-slices` of this document.
 983
 984* A new built-in function, :func:`sum(iterable, start=0)`,  adds up the numeric
 985  items in the iterable object and returns their sum.  :func:`sum` only accepts
 986  numbers, meaning that you can't use it to concatenate a bunch of strings.
 987  (Contributed by Alex Martelli.)
 988
 989* ``list.insert(pos, value)`` used to  insert *value* at the front of the list
 990  when *pos* was negative.  The behaviour has now been changed to be consistent
 991  with slice indexing, so when *pos* is -1 the value will be inserted before the
 992  last element, and so forth.
 993
 994* ``list.index(value)``, which searches for *value*  within the list and returns
 995  its index, now takes optional  *start* and *stop* arguments to limit the search
 996  to  only part of the list.
 997
 998* Dictionaries have a new method, :meth:`pop(key[, *default*])`, that returns
 999  the value corresponding to *key* and removes that key/value pair from the
1000  dictionary.  If the requested key isn't present in the dictionary, *default* is
1001  returned if it's specified and :exc:`KeyError` raised if it isn't. ::
1002
1003     >>> d = {1:2}
1004     >>> d
1005     {1: 2}
1006     >>> d.pop(4)
1007     Traceback (most recent call last):
1008       File "stdin", line 1, in ?
1009     KeyError: 4
1010     >>> d.pop(1)
1011     2
1012     >>> d.pop(1)
1013     Traceback (most recent call last):
1014       File "stdin", line 1, in ?
1015     KeyError: 'pop(): dictionary is empty'
1016     >>> d
1017     {}
1018     >>>
1019
1020  There's also a new class method,  :meth:`dict.fromkeys(iterable, value)`, that
1021  creates a dictionary with keys taken from the supplied iterator *iterable* and
1022  all values set to *value*, defaulting to ``None``.
1023
1024  (Patches contributed by Raymond Hettinger.)
1025
1026  Also, the :func:`dict` constructor now accepts keyword arguments to simplify
1027  creating small dictionaries::
1028
1029     >>> dict(red=1, blue=2, green=3, black=4)
1030     {'blue': 2, 'black': 4, 'green': 3, 'red': 1}
1031
1032  (Contributed by Just van Rossum.)
1033
1034* The :keyword:`assert` statement no longer checks the ``__debug__`` flag, so
1035  you can no longer disable assertions by assigning to ``__debug__``. Running
1036  Python with the :option:`-O` switch will still generate code that doesn't
1037  execute any assertions.
1038
1039* Most type objects are now callable, so you can use them to create new objects
1040  such as functions, classes, and modules.  (This means that the :mod:`new` module
1041  can be deprecated in a future Python version, because you can now use the type
1042  objects available in the :mod:`types` module.) For example, you can create a new
1043  module object with the following code:
1044
1045  ::
1046
1047     >>> import types
1048     >>> m = types.ModuleType('abc','docstring')
1049     >>> m
1050     <module 'abc' (built-in)>
1051     >>> m.__doc__
1052     'docstring'
1053
1054* A new warning, :exc:`PendingDeprecationWarning` was added to indicate features
1055  which are in the process of being deprecated.  The warning will *not* be printed
1056  by default.  To check for use of features that will be deprecated in the future,
1057  supply :option:`-Walways::PendingDeprecationWarning::` on the command line or
1058  use :func:`warnings.filterwarnings`.
1059
1060* The process of deprecating string-based exceptions, as in ``raise "Error
1061  occurred"``, has begun.  Raising a string will now trigger
1062  :exc:`PendingDeprecationWarning`.
1063
1064* Using ``None`` as a variable name will now result in a :exc:`SyntaxWarning`
1065  warning.  In a future version of Python, ``None`` may finally become a keyword.
1066
1067* The :meth:`xreadlines` method of file objects, introduced in Python 2.1, is no
1068  longer necessary because files now behave as their own iterator.
1069  :meth:`xreadlines` was originally introduced as a faster way to loop over all
1070  the lines in a file, but now you can simply write ``for line in file_obj``.
1071  File objects also have a new read-only :attr:`encoding` attribute that gives the
1072  encoding used by the file; Unicode strings written to the file will be
1073  automatically  converted to bytes using the given encoding.
1074
1075* The method resolution order used by new-style classes has changed, though
1076  you'll only notice the difference if you have a really complicated inheritance
1077  hierarchy.  Classic classes are unaffected by this change.  Python 2.2
1078  originally used a topological sort of a class's ancestors, but 2.3 now uses the
1079  C3 algorithm as described in the paper `"A Monotonic Superclass Linearization
1080  for Dylan" <http://www.webcom.com/haahr/dylan/linearization-oopsla96.html>`_. To
1081  understand the motivation for this change,  read Michele Simionato's article
1082  `"Python 2.3 Method Resolution Order" <http://www.python.org/2.3/mro.html>`_, or
1083  read the thread on python-dev starting with the message at
1084  http://mail.python.org/pipermail/python-dev/2002-October/029035.html. Samuele
1085  Pedroni first pointed out the problem and also implemented the fix by coding the
1086  C3 algorithm.
1087
1088* Python runs multithreaded programs by switching between threads after
1089  executing N bytecodes.  The default value for N has been increased from 10 to
1090  100 bytecodes, speeding up single-threaded applications by reducing the
1091  switching overhead.  Some multithreaded applications may suffer slower response
1092  time, but that's easily fixed by setting the limit back to a lower number using
1093  :func:`sys.setcheckinterval(N)`. The limit can be retrieved with the new
1094  :func:`sys.getcheckinterval` function.
1095
1096* One minor but far-reaching change is that the names of extension types defined
1097  by the modules included with Python now contain the module and a ``'.'`` in
1098  front of the type name.  For example, in Python 2.2, if you created a socket and
1099  printed its :attr:`__class__`, you'd get this output::
1100
1101     >>> s = socket.socket()
1102     >>> s.__class__
1103     <type 'socket'>
1104
1105  In 2.3, you get this::
1106
1107     >>> s.__class__
1108     <type '_socket.socket'>
1109
1110* One of the noted incompatibilities between old- and new-style classes has been
1111  removed: you can now assign to the :attr:`__name__` and :attr:`__bases__`
1112  attributes of new-style classes.  There are some restrictions on what can be
1113  assigned to :attr:`__bases__` along the lines of those relating to assigning to
1114  an instance's :attr:`__class__` attribute.
1115
1116.. ======================================================================
1117
1118
1119String Changes
1120--------------
1121
1122* The :keyword:`in` operator now works differently for strings. Previously, when
1123  evaluating ``X in Y`` where *X* and *Y* are strings, *X* could only be a single
1124  character. That's now changed; *X* can be a string of any length, and ``X in Y``
1125  will return :const:`True` if *X* is a substring of *Y*.  If *X* is the empty
1126  string, the result is always :const:`True`. ::
1127
1128     >>> 'ab' in 'abcd'
1129     True
1130     >>> 'ad' in 'abcd'
1131     False
1132     >>> '' in 'abcd'
1133     True
1134
1135  Note that this doesn't tell you where the substring starts; if you need that
1136  information, use the :meth:`find` string method.
1137
1138* The :meth:`strip`, :meth:`lstrip`, and :meth:`rstrip` string methods now have
1139  an optional argument for specifying the characters to strip.  The default is
1140  still to remove all whitespace characters::
1141
1142     >>> '   abc '.strip()
1143     'abc'
1144     >>> '><><abc<><><>'.strip('<>')
1145     'abc'
1146     >>> '><><abc<><><>\n'.strip('<>')
1147     'abc<><><>\n'
1148     >>> u'\u4000\u4001abc\u4000'.strip(u'\u4000')
1149     u'\u4001abc'
1150     >>>
1151
1152  (Suggested by Simon Brunning and implemented by Walter Dörwald.)
1153
1154* The :meth:`startswith` and :meth:`endswith` string methods now accept negative
1155  numbers for the *start* and *end* parameters.
1156
1157* Another new string method is :meth:`zfill`, originally a function in the
1158  :mod:`string` module.  :meth:`zfill` pads a numeric string with zeros on the
1159  left until it's the specified width. Note that the ``%`` operator is still more
1160  flexible and powerful than :meth:`zfill`. ::
1161
1162     >>> '45'.zfill(4)
1163     '0045'
1164     >>> '12345'.zfill(4)
1165     '12345'
1166     >>> 'goofy'.zfill(6)
1167     '0goofy'
1168
1169  (Contributed by Walter Dörwald.)
1170
1171* A new type object, :class:`basestring`, has been added. Both 8-bit strings and
1172  Unicode strings inherit from this type, so ``isinstance(obj, basestring)`` will
1173  return :const:`True` for either kind of string.  It's a completely abstract
1174  type, so you can't create :class:`basestring` instances.
1175
1176* Interned strings are no longer immortal and will now be garbage-collected in
1177  the usual way when the only reference to them is from the internal dictionary of
1178  interned strings.  (Implemented by Oren Tirosh.)
1179
1180.. ======================================================================
1181
1182
1183Optimizations
1184-------------
1185
1186* The creation of new-style class instances has been made much faster; they're
1187  now faster than classic classes!
1188
1189* The :meth:`sort` method of list objects has been extensively rewritten by Tim
1190  Peters, and the implementation is significantly faster.
1191
1192* Multiplication of large long integers is now much faster thanks to an
1193  implementation of Karatsuba multiplication, an algorithm that scales better than
1194  the O(n\*n) required for the grade-school multiplication algorithm.  (Original
1195  patch by Christopher A. Craig, and significantly reworked by Tim Peters.)
1196
1197* The ``SET_LINENO`` opcode is now gone.  This may provide a small speed
1198  increase, depending on your compiler's idiosyncrasies. See section
1199  :ref:`section-other` for a longer explanation. (Removed by Michael Hudson.)
1200
1201* :func:`xrange` objects now have their own iterator, making ``for i in
1202  xrange(n)`` slightly faster than ``for i in range(n)``.  (Patch by Raymond
1203  Hettinger.)
1204
1205* A number of small rearrangements have been made in various hotspots to improve
1206  performance, such as inlining a function or removing some code.  (Implemented
1207  mostly by GvR, but lots of people have contributed single changes.)
1208
1209The net result of the 2.3 optimizations is that Python 2.3 runs the  pystone
1210benchmark around 25% faster than Python 2.2.
1211
1212.. ======================================================================
1213
1214
1215New, Improved, and Deprecated Modules
1216=====================================
1217
1218As usual, Python's standard library received a number of enhancements and bug
1219fixes.  Here's a partial list of the most notable changes, sorted alphabetically
1220by module name. Consult the :file:`Misc/NEWS` file in the source tree for a more
1221complete list of changes, or look through the CVS logs for all the details.
1222
1223* The :mod:`array` module now supports arrays of Unicode characters using the
1224  ``'u'`` format character.  Arrays also now support using the ``+=`` assignment
1225  operator to add another array's contents, and the ``*=`` assignment operator to
1226  repeat an array. (Contributed by Jason Orendorff.)
1227
1228* The :mod:`bsddb` module has been replaced by version 4.1.6 of the `PyBSDDB
1229  <http://pybsddb.sourceforge.net>`_ package, providing a more complete interface
1230  to the transactional features of the BerkeleyDB library.
1231
1232  The old version of the module has been renamed to  :mod:`bsddb185` and is no
1233  longer built automatically; you'll  have to edit :file:`Modules/Setup` to enable
1234  it.  Note that the new :mod:`bsddb` package is intended to be compatible with
1235  the  old module, so be sure to file bugs if you discover any incompatibilities.
1236  When upgrading to Python 2.3, if the new interpreter is compiled with a new
1237  version of  the underlying BerkeleyDB library, you will almost certainly have to
1238  convert your database files to the new version.  You can do this fairly easily
1239  with the new scripts :file:`db2pickle.py` and :file:`pickle2db.py` which you
1240  will find in the distribution's :file:`Tools/scripts` directory.  If you've
1241  already been using the PyBSDDB package and importing it as :mod:`bsddb3`, you
1242  will have to change your ``import`` statements to import it as :mod:`bsddb`.
1243
1244* The new :mod:`bz2` module is an interface to the bz2 data compression library.
1245  bz2-compressed data is usually smaller than  corresponding :mod:`zlib`\
1246  -compressed data. (Contributed by Gustavo Niemeyer.)
1247
1248* A set of standard date/time types has been added in the new :mod:`datetime`
1249  module.  See the following section for more details.
1250
1251* The Distutils :class:`Extension` class now supports an extra constructor
1252  argument named *depends* for listing additional source files that an extension
1253  depends on.  This lets Distutils recompile the module if any of the dependency
1254  files are modified.  For example, if :file:`sampmodule.c` includes the header
1255  file :file:`sample.h`, you would create the :class:`Extension` object like
1256  this::
1257
1258     ext = Extension("samp",
1259                     sources=["sampmodule.c"],
1260                     depends=["sample.h"])
1261
1262  Modifying :file:`sample.h` would then cause the module to be recompiled.
1263  (Contributed by Jeremy Hylton.)
1264
1265* Other minor changes to Distutils: it now checks for the :envvar:`CC`,
1266  :envvar:`CFLAGS`, :envvar:`CPP`, :envvar:`LDFLAGS`, and :envvar:`CPPFLAGS`
1267  environment variables, using them to override the settings in Python's
1268  configuration (contributed by Robert Weber).
1269
1270* Previously the :mod:`doctest` module would only search the docstrings of
1271  public methods and functions for test cases, but it now also examines private
1272  ones as well.  The :func:`DocTestSuite(` function creates a
1273  :class:`unittest.TestSuite` object from a set of :mod:`doctest` tests.
1274
1275* The new :func:`gc.get_referents(object)` function returns a list of all the
1276  objects referenced by *object*.
1277
1278* The :mod:`getopt` module gained a new function, :func:`gnu_getopt`, that
1279  supports the same arguments as the existing :func:`getopt` function but uses
1280  GNU-style scanning mode. The existing :func:`getopt` stops processing options as
1281  soon as a non-option argument is encountered, but in GNU-style mode processing
1282  continues, meaning that options and arguments can be mixed.  For example::
1283
1284     >>> getopt.getopt(['-f', 'filename', 'output', '-v'], 'f:v')
1285     ([('-f', 'filename')], ['output', '-v'])
1286     >>> getopt.gnu_getopt(['-f', 'filename', 'output', '-v'], 'f:v')
1287     ([('-f', 'filename'), ('-v', '')], ['output'])
1288
1289  (Contributed by Peter Åstrand.)
1290
1291* The :mod:`grp`, :mod:`pwd`, and :mod:`resource` modules now return enhanced
1292  tuples::
1293
1294     >>> import grp
1295     >>> g = grp.getgrnam('amk')
1296     >>> g.gr_name, g.gr_gid
1297     ('amk', 500)
1298
1299* The :mod:`gzip` module can now handle files exceeding 2 GiB.
1300
1301* The new :mod:`heapq` module contains an implementation of a heap queue
1302  algorithm.  A heap is an array-like data structure that keeps items in a
1303  partially sorted order such that, for every index *k*, ``heap[k] <=
1304  heap[2*k+1]`` and ``heap[k] <= heap[2*k+2]``.  This makes it quick to remove the
1305  smallest item, and inserting a new item while maintaining the heap property is
1306  O(lg n).  (See http://www.nist.gov/dads/HTML/priorityque.html for more
1307  information about the priority queue data structure.)
1308
1309  The :mod:`heapq` module provides :func:`heappush` and :func:`heappop` functions
1310  for adding and removing items while maintaining the heap property on top of some
1311  other mutable Python sequence type.  Here's an example that uses a Python list::
1312
1313     >>> import heapq
1314     >>> heap = []
1315     >>> for item in [3, 7, 5, 11, 1]:
1316     ...    heapq.heappush(heap, item)
1317     ...
1318     >>> heap
1319     [1, 3, 5, 11, 7]
1320     >>> heapq.heappop(heap)
1321     1
1322     >>> heapq.heappop(heap)
1323     3
1324     >>> heap
1325     [5, 7, 11]
1326
1327  (Contributed by Kevin O'Connor.)
1328
1329* The IDLE integrated development environment has been updated using the code
1330  from the IDLEfork project (http://idlefork.sf.net).  The most notable feature is
1331  that the code being developed is now executed in a subprocess, meaning that
1332  there's no longer any need for manual ``reload()`` operations. IDLE's core code
1333  has been incorporated into the standard library as the :mod:`idlelib` package.
1334
1335* The :mod:`imaplib` module now supports IMAP over SSL. (Contributed by Piers
1336  Lauder and Tino Lange.)
1337
1338* The :mod:`itertools` contains a number of useful functions for use with
1339  iterators, inspired by various functions provided by the ML and Haskell
1340  languages.  For example, ``itertools.ifilter(predicate, iterator)`` returns all
1341  elements in the iterator for which the function :func:`predicate` returns
1342  :const:`True`, and ``itertools.repeat(obj, N)`` returns ``obj`` *N* times.
1343  There are a number of other functions in the module; see the package's reference
1344  documentation for details.
1345  (Contributed by Raymond Hettinger.)
1346
1347* Two new functions in the :mod:`math` module, :func:`degrees(rads)` and
1348  :func:`radians(degs)`, convert between radians and degrees.  Other functions in
1349  the :mod:`math` module such as :func:`math.sin` and :func:`math.cos` have always
1350  required input values measured in radians.  Also, an optional *base* argument
1351  was added to :func:`math.log` to make it easier to compute logarithms for bases
1352  other than ``e`` and ``10``.  (Contributed by Raymond Hettinger.)
1353
1354* Several new POSIX functions (:func:`getpgid`, :func:`killpg`, :func:`lchown`,
1355  :func:`loadavg`, :func:`major`, :func:`makedev`, :func:`minor`, and
1356  :func:`mknod`) were added to the :mod:`posix` module that underlies the
1357  :mod:`os` module. (Contributed by Gustavo Niemeyer, Geert Jansen, and Denis S.
1358  Otkidach.)
1359
1360* In the :mod:`os` module, the :func:`\*stat` family of functions can now report
1361  fractions of a second in a timestamp.  Such time stamps are represented as
1362  floats, similar to the value returned by :func:`time.time`.
1363
1364  During testing, it was found that some applications will break if time stamps
1365  are floats.  For compatibility, when using the tuple interface of the
1366  :class:`stat_result` time stamps will be represented as integers. When using
1367  named fields (a feature first introduced in Python 2.2), time stamps are still
1368  represented as integers, unless :func:`os.stat_float_times` is invoked to enable
1369  float return values::
1370
1371     >>> os.stat("/tmp").st_mtime
1372     1034791200
1373     >>> os.stat_float_times(True)
1374     >>> os.stat("/tmp").st_mtime
1375     1034791200.6335014
1376
1377  In Python 2.4, the default will change to always returning floats.
1378
1379  Application developers should enable this feature only if all their libraries
1380  work properly when confronted with floating point time stamps, or if they use
1381  the tuple API. If used, the feature should be activated on an application level
1382  instead of trying to enable it on a per-use basis.
1383
1384* The :mod:`optparse` module contains a new parser for command-line arguments
1385  that can convert option values to a particular Python type  and will
1386  automatically generate a usage message.  See the following section for  more
1387  details.
1388
1389* The old and never-documented :mod:`linuxaudiodev` module has been deprecated,
1390  and a new version named :mod:`ossaudiodev` has been added.  The module was
1391  renamed because the OSS sound drivers can be used on platforms other than Linux,
1392  and the interface has also been tidied and brought up to date in various ways.
1393  (Contributed by Greg Ward and Nicholas FitzRoy-Dale.)
1394
1395* The new :mod:`platform` module contains a number of functions that try to
1396  determine various properties of the platform you're running on.  There are
1397  functions for getting the architecture, CPU type, the Windows OS version, and
1398  even the Linux distribution version. (Contributed by Marc-André Lemburg.)
1399
1400* The parser objects provided by the :mod:`pyexpat` module can now optionally
1401  buffer character data, resulting in fewer calls to your character data handler
1402  and therefore faster performance.  Setting the parser object's
1403  :attr:`buffer_text` attribute to :const:`True` will enable buffering.
1404
1405* The :func:`sample(population, k)` function was added to the :mod:`random`
1406  module.  *population* is a sequence or :class:`xrange` object containing the
1407  elements of a population, and :func:`sample` chooses *k* elements from the
1408  population without replacing chosen elements.  *k* can be any value up to
1409  ``len(population)``. For example::
1410
1411     >>> days = ['Mo', 'Tu', 'We', 'Th', 'Fr', 'St', 'Sn']
1412     >>> random.sample(days, 3)      # Choose 3 elements
1413     ['St', 'Sn', 'Th']
1414     >>> random.sample(days, 7)      # Choose 7 elements
1415     ['Tu', 'Th', 'Mo', 'We', 'St', 'Fr', 'Sn']
1416     >>> random.sample(days, 7)      # Choose 7 again
1417     ['We', 'Mo', 'Sn', 'Fr', 'Tu', 'St', 'Th']
1418     >>> random.sample(days, 8)      # Can't choose eight
1419     Traceback (most recent call last):
1420       File "<stdin>", line 1, in ?
1421       File "random.py", line 414, in sample
1422           raise ValueError, "sample larger than population"
1423     ValueError: sample larger than population
1424     >>> random.sample(xrange(1,10000,2), 10)   # Choose ten odd nos. under 10000
1425     [3407, 3805, 1505, 7023, 2401, 2267, 9733, 3151, 8083, 9195]
1426
1427  The :mod:`random` module now uses a new algorithm, the Mersenne Twister,
1428  implemented in C.  It's faster and more extensively studied than the previous
1429  algorithm.
1430
1431  (All changes contributed by Raymond Hettinger.)
1432
1433* The :mod:`readline` module also gained a number of new functions:
1434  :func:`get_history_item`, :func:`get_current_history_length`, and
1435  :func:`redisplay`.
1436
1437* The :mod:`rexec` and :mod:`Bastion` modules have been declared dead, and
1438  attempts to import them will fail with a :exc:`RuntimeError`.  New-style classes
1439  provide new ways to break out of the restricted execution environment provided
1440  by :mod:`rexec`, and no one has interest in fixing them or time to do so.  If
1441  you have applications using :mod:`rexec`, rewrite them to use something else.
1442
1443  (Sticking with Python 2.2 or 2.1 will not make your applications any safer
1444  because there are known bugs in the :mod:`rexec` module in those versions.  To
1445  repeat: if you're using :mod:`rexec`, stop using it immediately.)
1446
1447* The :mod:`rotor` module has been deprecated because the  algorithm it uses for
1448  encryption is not believed to be secure.  If you need encryption, use one of the
1449  several AES Python modules that are available separately.
1450
1451* The :mod:`shutil` module gained a :func:`move(src, dest)` function that
1452  recursively moves a file or directory to a new location.
1453
1454* Support for more advanced POSIX signal handling was added to the :mod:`signal`
1455  but then removed again as it proved impossible to make it work reliably across
1456  platforms.
1457
1458* The :mod:`socket` module now supports timeouts.  You can call the
1459  :meth:`settimeout(t)` method on a socket object to set a timeout of *t* seconds.
1460  Subsequent socket operations that take longer than *t* seconds to complete will
1461  abort and raise a :exc:`socket.timeout` exception.
1462
1463  The original timeout implementation was by Tim O'Malley.  Michael Gilfix
1464  integrated it into the Python :mod:`socket` module and shepherded it through a
1465  lengthy review.  After the code was checked in, Guido van Rossum rewrote parts
1466  of it.  (This is a good example of a collaborative development process in
1467  action.)
1468
1469* On Windows, the :mod:`socket` module now ships with Secure  Sockets Layer
1470  (SSL) support.
1471
1472* The value of the C :const:`PYTHON_API_VERSION` macro is now exposed at the
1473  Python level as ``sys.api_version``.  The current exception can be cleared by
1474  calling the new :func:`sys.exc_clear` function.
1475
1476* The new :mod:`tarfile` module  allows reading from and writing to
1477  :program:`tar`\ -format archive files. (Contributed by Lars Gustäbel.)
1478
1479* The new :mod:`textwrap` module contains functions for wrapping strings
1480  containing paragraphs of text.  The :func:`wrap(text, width)` function takes a
1481  string and returns a list containing the text split into lines of no more than
1482  the chosen width.  The :func:`fill(text, width)` function returns a single
1483  string, reformatted to fit into lines no longer than the chosen width. (As you
1484  can guess, :func:`fill` is built on top of :func:`wrap`.  For example::
1485
1486     >>> import textwrap
1487     >>> paragraph = "Not a whit, we defy augury: ... more text ..."
1488     >>> textwrap.wrap(paragraph, 60)
1489     ["Not a whit, we defy augury: there's a special providence in",
1490      "the fall of a sparrow. If it be now, 'tis not to come; if it",
1491      ...]
1492     >>> print textwrap.fill(paragraph, 35)
1493     Not a whit, we defy augury: there's
1494     a special providence in the fall of
1495     a sparrow. If it be now, 'tis not
1496     to come; if it be not to come, it
1497     will be now; if it be not now, yet
1498     it will come: the readiness is all.
1499     >>>
1500
1501  The module also contains a :class:`TextWrapper` class that actually implements
1502  the text wrapping strategy.   Both the :class:`TextWrapper` class and the
1503  :func:`wrap` and :func:`fill` functions support a number of additional keyword
1504  arguments for fine-tuning the formatting; consult the module's documentation
1505  for details. (Contributed by Greg Ward.)
1506
1507* The :mod:`thread` and :mod:`threading` modules now have companion modules,
1508  :mod:`dummy_thread` and :mod:`dummy_threading`, that provide a do-nothing
1509  implementation of the :mod:`thread` module's interface for platforms where
1510  threads are not supported.  The intention is to simplify thread-aware modules
1511  (ones that *don't* rely on threads to run) by putting the following code at the
1512  top::
1513
1514     try:
1515         import threading as _threading
1516     except ImportError:
1517         import dummy_threading as _threading
1518
1519  In this example, :mod:`_threading` is used as the module name to make it clear
1520  that the module being used is not necessarily the actual :mod:`threading`
1521  module. Code can call functions and use classes in :mod:`_threading` whether or
1522  not threads are supported, avoiding an :keyword:`if` statement and making the
1523  code slightly clearer.  This module will not magically make multithreaded code
1524  run without threads; code that waits for another thread to return or to do
1525  something will simply hang forever.
1526
1527* The :mod:`time` module's :func:`strptime` function has long been an annoyance
1528  because it uses the platform C library's :func:`strptime` implementation, and
1529  different platforms sometimes have odd bugs.  Brett Cannon contributed a
1530  portable implementation that's written in pure Python and should behave
1531  identically on all platforms.
1532
1533* The new :mod:`timeit` module helps measure how long snippets of Python code
1534  take to execute.  The :file:`timeit.py` file can be run directly from the
1535  command line, or the module's :class:`Timer` class can be imported and used
1536  directly.  Here's a short example that figures out whether it's faster to
1537  convert an 8-bit string to Unicode by appending an empty Unicode string to it or
1538  by using the :func:`unicode` function::
1539
1540     import timeit
1541
1542     timer1 = timeit.Timer('unicode("abc")')
1543     timer2 = timeit.Timer('"abc" + u""')
1544
1545     # Run three trials
1546     print timer1.repeat(repeat=3, number=100000)
1547     print timer2.repeat(repeat=3, number=100000)
1548
1549     # On my laptop this outputs:
1550     # [0.36831796169281006, 0.37441694736480713, 0.35304892063140869]
1551     # [0.17574405670166016, 0.18193507194519043, 0.17565798759460449]
1552
1553* The :mod:`Tix` module has received various bug fixes and updates for the
1554  current version of the Tix package.
1555
1556* The :mod:`Tkinter` module now works with a thread-enabled  version of Tcl.
1557  Tcl's threading model requires that widgets only be accessed from the thread in
1558  which they're created; accesses from another thread can cause Tcl to panic.  For
1559  certain Tcl interfaces, :mod:`Tkinter` will now automatically avoid this  when a
1560  widget is accessed from a different thread by marshalling a command, passing it
1561  to the correct thread, and waiting for the results.  Other interfaces can't be
1562  handled automatically but :mod:`Tkinter` will now raise an exception on such an
1563  access so that you can at least find out about the problem.  See
1564  http://mail.python.org/pipermail/python-dev/2002-December/031107.html for a more
1565  detailed explanation of this change.  (Implemented by Martin von Löwis.)
1566
1567* Calling Tcl methods through :mod:`_tkinter` no longer  returns only strings.
1568  Instead, if Tcl returns other objects those objects are converted to their
1569  Python equivalent, if one exists, or wrapped with a :class:`_tkinter.Tcl_Obj`
1570  object if no Python equivalent exists. This behavior can be controlled through
1571  the :meth:`wantobjects` method of :class:`tkapp` objects.
1572
1573  When using :mod:`_tkinter` through the :mod:`Tkinter` module (as most Tkinter
1574  applications will), this feature is always activated. It should not cause
1575  compatibility problems, since Tkinter would always convert string results to
1576  Python types where possible.
1577
1578  If any incompatibilities are found, the old behavior can be restored by setting
1579  the :attr:`wantobjects` variable in the :mod:`Tkinter` module to false before
1580  creating the first :class:`tkapp` object. ::
1581
1582     import Tkinter
1583     Tkinter.wantobjects = 0
1584
1585  Any breakage caused by this change should be reported as a bug.
1586
1587* The :mod:`UserDict` module has a new :class:`DictMixin` class which defines
1588  all dictionary methods for classes that already have a minimum mapping
1589  interface.  This greatly simplifies writing classes that need to be
1590  substitutable for dictionaries, such as the classes in  the :mod:`shelve`
1591  module.
1592
1593  Adding the mix-in as a superclass provides the full dictionary interface
1594  whenever the class defines :meth:`__getitem__`, :meth:`__setitem__`,
1595  :meth:`__delitem__`, and :meth:`keys`. For example::
1596
1597     >>> import UserDict
1598     >>> class SeqDict(UserDict.DictMixin):
1599     ...     """Dictionary lookalike implemented with lists."""
1600     ...     def __init__(self):
1601     ...         self.keylist = []
1602     ...         self.valuelist = []
1603     ...     def __getitem__(self, key):
1604     ...         try:
1605     ...             i = self.keylist.index(key)
1606     ...         except ValueError:
1607     ...             raise KeyError
1608     ...         return self.valuelist[i]
1609     ...     def __setitem__(self, key, value):
1610     ...         try:
1611     ...             i = self.keylist.index(key)
1612     ...             self.valuelist[i] = value
1613     ...         except ValueError:
1614     ...             self.keylist.append(key)
1615     ...             self.valuelist.append(value)
1616     ...     def __delitem__(self, key):
1617     ...         try:
1618     ...             i = self.keylist.index(key)
1619     ...         except ValueError:
1620     ...             raise KeyError
1621     ...         self.keylist.pop(i)
1622     ...         self.valuelist.pop(i)
1623     ...     def keys(self):
1624     ...         return list(self.keylist)
1625     ...
1626     >>> s = SeqDict()
1627     >>> dir(s)      # See that other dictionary methods are implemented
1628     ['__cmp__', '__contains__', '__delitem__', '__doc__', '__getitem__',
1629      '__init__', '__iter__', '__len__', '__module__', '__repr__',
1630      '__setitem__', 'clear', 'get', 'has_key', 'items', 'iteritems',
1631      'iterkeys', 'itervalues', 'keylist', 'keys', 'pop', 'popitem',
1632      'setdefault', 'update', 'valuelist', 'values']
1633
1634  (Contributed by Raymond Hettinger.)
1635
1636* The DOM implementation in :mod:`xml.dom.minidom` can now generate XML output
1637  in a particular encoding by providing an optional encoding argument to the
1638  :meth:`toxml` and :meth:`toprettyxml` methods of DOM nodes.
1639
1640* The :mod:`xmlrpclib` module now supports an XML-RPC extension for handling nil
1641  data values such as Python's ``None``.  Nil values are always supported on
1642  unmarshalling an XML-RPC response.  To generate requests containing ``None``,
1643  you must supply a true value for the *allow_none* parameter when creating a
1644  :class:`Marshaller` instance.
1645
1646* The new :mod:`DocXMLRPCServer` module allows writing self-documenting XML-RPC
1647  servers. Run it in demo mode (as a program) to see it in action.   Pointing the
1648  Web browser to the RPC server produces pydoc-style documentation; pointing
1649  xmlrpclib to the server allows invoking the actual methods. (Contributed by
1650  Brian Quinlan.)
1651
1652* Support for internationalized domain names (RFCs 3454, 3490, 3491, and 3492)
1653  has been added. The "idna" encoding can be used to convert between a Unicode
1654  domain name and the ASCII-compatible encoding (ACE) of that name. ::
1655
1656     >{}>{}> u"www.Alliancefrançaise.nu".encode("idna")
1657     'www.xn--alliancefranaise-npb.nu'
1658
1659  The :mod:`socket` module has also been extended to transparently convert
1660  Unicode hostnames to the ACE version before passing them to the C library.
1661  Modules that deal with hostnames such as :mod:`httplib` and :mod:`ftplib`)
1662  also support Unicode host names; :mod:`httplib` also sends HTTP ``Host``
1663  headers using the ACE version of the domain name.  :mod:`urllib` supports
1664  Unicode URLs with non-ASCII host names as long as the ``path`` part of the URL
1665  is ASCII only.
1666
1667  To implement this change, the :mod:`stringprep` module, the  ``mkstringprep``
1668  tool and the ``punycode`` encoding have been added.
1669
1670.. ======================================================================
1671
1672
1673Date/Time Type
1674--------------
1675
1676Date and time types suitable for expressing timestamps were added as the
1677:mod:`datetime` module.  The types don't support different calendars or many
1678fancy features, and just stick to the basics of representing time.
1679
1680The three primary types are: :class:`date`, representing a day, month, and year;
1681:class:`time`, consisting of hour, minute, and second; and :class:`datetime`,
1682which contains all the attributes of both :class:`date` and :class:`time`.
1683There's also a :class:`timedelta` class representing differences between two
1684points in time, and time zone logic is implemented by classes inheriting from
1685the abstract :class:`tzinfo` class.
1686
1687You can create instances of :class:`date` and :class:`time` by either supplying
1688keyword arguments to the appropriate constructor, e.g.
1689``datetime.date(year=1972, month=10, day=15)``, or by using one of a number of
1690class methods.  For example, the :meth:`date.today` class method returns the
1691current local date.
1692
1693Once created, instances of the date/time classes are all immutable. There are a
1694number of methods for producing formatted strings from objects::
1695
1696   >>> import datetime
1697   >>> now = datetime.datetime.now()
1698   >>> now.isoformat()
1699   '2002-12-30T21:27:03.994956'
1700   >>> now.ctime()  # Only available on date, datetime
1701   'Mon Dec 30 21:27:03 2002'
1702   >>> now.strftime('%Y %d %b')
1703   '2002 30 Dec'
1704
1705The :meth:`replace` method allows modifying one or more fields  of a
1706:class:`date` or :class:`datetime` instance, returning a new instance::
1707
1708   >>> d = datetime.datetime.now()
1709   >>> d
1710   datetime.datetime(2002, 12, 30, 22, 15, 38, 827738)
1711   >>> d.replace(year=2001, hour = 12)
1712   datetime.datetime(2001, 12, 30, 12, 15, 38, 827738)
1713   >>>
1714
1715Instances can be compared, hashed, and converted to strings (the result is the
1716same as that of :meth:`isoformat`).  :class:`date` and :class:`datetime`
1717instances can be subtracted from each other, and added to :class:`timedelta`
1718instances.  The largest missing feature is that there's no standard library
1719support for parsing strings and getting back a :class:`date` or
1720:class:`datetime`.
1721
1722For more information, refer to the module's reference documentation.
1723(Contributed by Tim Peters.)
1724
1725.. ======================================================================
1726
1727
1728The optparse Module
1729-------------------
1730
1731The :mod:`getopt` module provides simple parsing of command-line arguments.  The
1732new :mod:`optparse` module (originally named Optik) provides more elaborate
1733command-line parsing that follows the Unix conventions, automatically creates
1734the output for :option:`--help`, and can perform different actions for different
1735options.
1736
1737You start by creating an instance of :class:`OptionParser` and telling it what
1738your program's options are. ::
1739
1740   import sys
1741   from optparse import OptionParser
1742
1743   op = OptionParser()
1744   op.add_option('-i', '--input',
1745                 action='store', type='string', dest='input',
1746                 help='set input filename')
1747   op.add_option('-l', '--length',
1748                 action='store', type='int', dest='length',
1749                 help='set maximum length of output')
1750
1751Parsing a command line is then done by calling the :meth:`parse_args` method. ::
1752
1753   options, args = op.parse_args(sys.argv[1:])
1754   print options
1755   print args
1756
1757This returns an object containing all of the option values, and a list of
1758strings containing the remaining arguments.
1759
1760Invoking the script with the various arguments now works as you'd expect it to.
1761Note that the length argument is automatically converted to an integer. ::
1762
1763   $ ./python opt.py -i data arg1
1764   <Values at 0x400cad4c: {'input': 'data', 'length': None}>
1765   ['arg1']
1766   $ ./python opt.py --input=data --length=4
1767   <Values at 0x400cad2c: {'input': 'data', 'length': 4}>
1768   []
1769   $
1770
1771The help message is automatically generated for you::
1772
1773   $ ./python opt.py --help
1774   usage: opt.py [options]
1775
1776   options:
1777     -h, --help            show this help message and exit
1778     -iINPUT, --input=INPUT
1779                           set input filename
1780     -lLENGTH, --length=LENGTH
1781                           set maximum length of output
1782   $
1783
1784See the module's documentation for more details.
1785
1786
1787Optik was written by Greg Ward, with suggestions from the readers of the Getopt
1788SIG.
1789
1790.. ======================================================================
1791
1792
1793.. _section-pymalloc:
1794
1795Pymalloc: A Specialized Object Allocator
1796========================================
1797
1798Pymalloc, a specialized object allocator written by Vladimir Marangozov, was a
1799feature added to Python 2.1.  Pymalloc is intended to be faster than the system
1800:cfunc:`malloc` and to have less memory overhead for allocation patterns typical
1801of Python programs. The allocator uses C's :cfunc:`malloc` function to get large
1802pools of memory and then fulfills smaller memory requests from these pools.
1803
1804In 2.1 and 2.2, pymalloc was an experimental feature and wasn't enabled by
1805default; you had to explicitly enable it when compiling Python by providing the
1806:option:`--with-pymalloc` option to the :program:`configure` script.  In 2.3,
1807pymalloc has had further enhancements and is now enabled by default; you'll have
1808to supply :option:`--without-pymalloc` to disable it.
1809
1810This change is transparent to code written in Python; however, pymalloc may
1811expose bugs in C extensions.  Authors of C extension modules should test their
1812code with pymalloc enabled, because some incorrect code may cause core dumps at
1813runtime.
1814
1815There's one particularly common error that causes problems.  There are a number
1816of memory allocation functions in Python's C API that have previously just been
1817aliases for the C library's :cfunc:`malloc` and :cfunc:`free`, meaning that if
1818you accidentally called mismatched functions the error wouldn't be noticeable.
1819When the object allocator is enabled, these functions aren't aliases of
1820:cfunc:`malloc` and :cfunc:`free` any more, and calling the wrong function to
1821free memory may get you a core dump.  For example, if memory was allocated using
1822:cfunc:`PyObject_Malloc`, it has to be freed using :cfunc:`PyObject_Free`, not
1823:cfunc:`free`.  A few modules included with Python fell afoul of this and had to
1824be fixed; doubtless there are more third-party modules that will have the same
1825problem.
1826
1827As part of this change, the confusing multiple interfaces for allocating memory
1828have been consolidated down into two API families. Memory allocated with one
1829family must not be manipulated with functions from the other family.  There is
1830one family for allocating chunks of memory and another family of functions
1831specifically for allocating Python objects.
1832
1833* To allocate and free an undistinguished chunk of memory use the "raw memory"
1834  family: :cfunc:`PyMem_Malloc`, :cfunc:`PyMem_Realloc`, and :cfunc:`PyMem_Free`.
1835
1836* The "object memory" family is the interface to the pymalloc facility described
1837  above and is biased towards a large number of "small" allocations:
1838  :cfunc:`PyObject_Malloc`, :cfunc:`PyObject_Realloc`, and :cfunc:`PyObject_Free`.
1839
1840* To allocate and free Python objects, use the "object" family
1841  :cfunc:`PyObject_New`, :cfunc:`PyObject_NewVar`, and :cfunc:`PyObject_Del`.
1842
1843Thanks to lots of work by Tim Peters, pymalloc in 2.3 also provides debugging
1844features to catch memory overwrites and doubled frees in both extension modules
1845and in the interpreter itself.  To enable this support, compile a debugging
1846version of the Python interpreter by running :program:`configure` with
1847:option:`--with-pydebug`.
1848
1849To aid extension writers, a header file :file:`Misc/pymemcompat.h` is
1850distributed with the source to Python 2.3 that allows Python extensions to use
1851the 2.3 interfaces to memory allocation while compiling against any version of
1852Python since 1.5.2.  You would copy the file from Python's source distribution
1853and bundle it with the source of your extension.
1854
1855
1856.. seealso::
1857
1858   http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Objects/obmalloc.c
1859      For the full details of the pymalloc implementation, see the comments at the top
1860      of the file :file:`Objects/obmalloc.c` in the Python source code.  The above
1861      link points to the file within the SourceForge CVS browser.
1862
1863.. ======================================================================
1864
1865
1866Build and C API Changes
1867=======================
1868
1869Changes to Python's build process and to the C API include:
1870
1871* The cycle detection implementation used by the garbage collection has proven
1872  to be stable, so it's now been made mandatory.  You can no longer compile Python
1873  without it, and the :option:`--with-cycle-gc` switch to :program:`configure` has
1874  been removed.
1875
1876* Python can now optionally be built as a shared library
1877  (:file:`libpython2.3.so`) by supplying :option:`--enable-shared` when running
1878  Python's :program:`configure` script.  (Contributed by Ondrej Palkovsky.)
1879
1880* The :cmacro:`DL_EXPORT` and :cmacro:`DL_IMPORT` macros are now deprecated.
1881  Initialization functions for Python extension modules should now be declared
1882  using the new macro :cmacro:`PyMODINIT_FUNC`, while the Python core will
1883  generally use the :cmacro:`PyAPI_FUNC` and :cmacro:`PyAPI_DATA` macros.
1884
1885* The interpreter can be compiled without any docstrings for the built-in
1886  functions and modules by supplying :option:`--without-doc-strings` to the
1887  :program:`configure` script. This makes the Python executable about 10% smaller,
1888  but will also mean that you can't get help for Python's built-ins.  (Contributed
1889  by Gustavo Niemeyer.)
1890
1891* The :cfunc:`PyArg_NoArgs` macro is now deprecated, and code that uses it
1892  should be changed.  For Python 2.2 and later, the method definition table can
1893  specify the :const:`METH_NOARGS` flag, signalling that there are no arguments,
1894  and the argument checking can then be removed.  If compatibility with pre-2.2
1895  versions of Python is important, the code could use ``PyArg_ParseTuple(args,
1896  "")`` instead, but this will be slower than using :const:`METH_NOARGS`.
1897
1898* :cfunc:`PyArg_ParseTuple` accepts new format characters for various sizes of
1899  unsigned integers: ``B`` for :ctype:`unsigned char`, ``H`` for :ctype:`unsigned
1900  short int`,  ``I`` for :ctype:`unsigned int`,  and ``K`` for :ctype:`unsigned
1901  long long`.
1902
1903* A new function, :cfunc:`PyObject_DelItemString(mapping, char \*key)` was added
1904  as shorthand for ``PyObject_DelItem(mapping, PyString_New(key))``.
1905
1906* File objects now manage their internal string buffer differently, increasing
1907  it exponentially when needed.  This results in the benchmark tests in
1908  :file:`Lib/test/test_bufio.py` speeding up considerably (from 57 seconds to 1.7
1909  seconds, according to one measurement).
1910
1911* It's now possible to define class and static methods for a C extension type by
1912  setting either the :const:`METH_CLASS` or :const:`METH_STATIC` flags in a
1913  method's :ctype:`PyMethodDef` structure.
1914
1915* Python now includes a copy of the Expat XML parser's source code, removing any
1916  dependence on a system version or local installation of Expat.
1917
1918* If you dynamically allocate type objects in your extension, you should be
1919  aware of a change in the rules relating to the :attr:`__module__` and
1920  :attr:`__name__` attributes.  In summary, you will want to ensure the type's
1921  dictionary contains a ``'__module__'`` key; making the module name the part of
1922  the type name leading up to the final period will no longer have the desired
1923  effect.  For more detail, read the API reference documentation or the  source.
1924
1925.. ======================================================================
1926
1927
1928Port-Specific Changes
1929---------------------
1930
1931Support for a port to IBM's OS/2 using the EMX runtime environment was merged
1932into the main Python source tree.  EMX is a POSIX emulation layer over the OS/2
1933system APIs.  The Python port for EMX tries to support all the POSIX-like
1934capability exposed by the EMX runtime, and mostly succeeds; :func:`fork` and
1935:func:`fcntl` are restricted by the limitations of the underlying emulation
1936layer.  The standard OS/2 port, which uses IBM's Visual Age compiler, also
1937gained support for case-sensitive import semantics as part of the integration of
1938the EMX port into CVS.  (Contributed by Andrew MacIntyre.)
1939
1940On MacOS, most toolbox modules have been weaklinked to improve backward
1941compatibility.  This means that modules will no longer fail to load if a single
1942routine is missing on the current OS version. Instead calling the missing
1943routine will raise an exception. (Contributed by Jack Jansen.)
1944
1945The RPM spec files, found in the :file:`Misc/RPM/` directory in the Python
1946source distribution, were updated for 2.3.  (Contributed by Sean Reifschneider.)
1947
1948Other new platforms now supported by Python include AtheOS
1949(http://www.atheos.cx/), GNU/Hurd, and OpenVMS.
1950
1951.. ======================================================================
1952
1953
1954.. _section-other:
1955
1956Other Changes and Fixes
1957=======================
1958
1959As usual, there were a bunch of other improvements and bugfixes scattered
1960throughout the source tree.  A search through the CVS change logs finds there
1961were 523 patches applied and 514 bugs fixed between Python 2.2 and 2.3.  Both
1962figures are likely to be underestimates.
1963
1964Some of the more notable changes are:
1965
1966* If the :envvar:`PYTHONINSPECT` environment variable is set, the Python
1967  interpreter will enter the interactive prompt after running a Python program, as
1968  if Python had been invoked with the :option:`-i` option. The environment
1969  variable can be set before running the Python interpreter, or it can be set by
1970  the Python program as part of its execution.
1971
1972* The :file:`regrtest.py` script now provides a way to allow "all resources
1973  except *foo*."  A resource name passed to the :option:`-u` option can now be
1974  prefixed with a hyphen (``'-'``) to mean "remove this resource."  For example,
1975  the option '``-uall,-bsddb``' could be used to enable the use of all resources
1976  except ``bsddb``.
1977
1978* The tools used to build the documentation now work under Cygwin as well as
1979  Unix.
1980
1981* The ``SET_LINENO`` opcode has been removed.  Back in the mists of time, this
1982  opcode was needed to produce line numbers in tracebacks and support trace
1983  functions (for, e.g., :mod:`pdb`). Since Python 1.5, the line numbers in
1984  tracebacks have been computed using a different mechanism that works with
1985  "python -O".  For Python 2.3 Michael Hudson implemented a similar scheme to
1986  determine when to call the trace function, removing the need for ``SET_LINENO``
1987  entirely.
1988
1989  It would be difficult to detect any resulting difference from Python code, apart
1990  from a slight speed up when Python is run without :option:`-O`.
1991
1992  C extensions that access the :attr:`f_lineno` field of frame objects should
1993  instead call ``PyCode_Addr2Line(f->f_code, f->f_lasti)``. This will have the
1994  added effect of making the code work as desired under "python -O" in earlier
1995  versions of Python.
1996
1997  A nifty new feature is that trace functions can now assign to the
1998  :attr:`f_lineno` attribute of frame objects, changing the line that will be
1999  executed next.  A ``jump`` command has been added to the :mod:`pdb` debugger
2000  taking advantage of this new feature. (Implemented by Richie Hindle.)
2001
2002.. ======================================================================
2003
2004
2005Porting to Python 2.3
2006=====================
2007
2008This section lists previously described changes that may require changes to your
2009code:
2010
2011* :keyword:`yield` is now always a keyword; if it's used as a variable name in
2012  your code, a different name must be chosen.
2013
2014* For strings *X* and *Y*, ``X in Y`` now works if *X* is more than one
2015  character long.
2016
2017* The :func:`int` type constructor will now return a long integer instead of
2018  raising an :exc:`OverflowError` when a string or floating-point number is too
2019  large to fit into an integer.
2020
2021* If you have Unicode strings that contain 8-bit characters, you must declare
2022  the file's encoding (UTF-8, Latin-1, or whatever) by adding a comment to the top
2023  of the file.  See section :ref:`section-encodings` for more information.
2024
2025* Calling Tcl methods through :mod:`_tkinter` no longer  returns only strings.
2026  Instead, if Tcl returns other objects those objects are converted to their
2027  Python equivalent, if one exists, or wrapped with a :class:`_tkinter.Tcl_Obj`
2028  object if no Python equivalent exists.
2029
2030* Large octal and hex literals such as ``0xffffffff`` now trigger a
2031  :exc:`FutureWarning`. Currently they're stored as 32-bit numbers and result in a
2032  negative value, but in Python 2.4 they'll become positive long integers.
2033
2034  There are a few ways to fix this warning.  If you really need a positive number,
2035  just add an ``L`` to the end of the literal.  If you're trying to get a 32-bit
2036  integer with low bits set and have previously used an expression such as ``~(1
2037  << 31)``, it's probably clearest to start with all bits set and clear the
2038  desired upper bits. For example, to clear just the top bit (bit 31), you could
2039  write ``0xffffffffL &~(1L<<31)``.
2040
2041* You can no longer disable assertions by assigning to ``__debug__``.
2042
2043* The Distutils :func:`setup` function has gained various new keyword arguments
2044  such as *depends*.  Old versions of the Distutils will abort if passed unknown
2045  keywords.  A solution is to check for the presence of the new
2046  :func:`get_distutil_options` function in your :file:`setup.py` and only uses the
2047  new keywords with a version of the Distutils that supports them::
2048
2049     from distutils import core
2050
2051     kw = {'sources': 'foo.c', ...}
2052     if hasattr(core, 'get_distutil_options'):
2053         kw['depends'] = ['foo.h']
2054     ext = Extension(**kw)
2055
2056* Using ``None`` as a variable name will now result in a :exc:`SyntaxWarning`
2057  warning.
2058
2059* Names of extension types defined by the modules included with Python now
2060  contain the module and a ``'.'`` in front of the type name.
2061
2062.. ======================================================================
2063
2064
2065.. _23acks:
2066
2067Acknowledgements
2068================
2069
2070The author would like to thank the following people for offering suggestions,
2071corrections and assistance with various drafts of this article: Jeff Bauer,
2072Simon Brunning, Brett Cannon, Michael Chermside, Andrew Dalke, Scott David
2073Daniels, Fred L. Drake, Jr., David Fraser,  Kelly Gerber, Raymond Hettinger,
2074Michael Hudson, Chris Lambert, Detlef Lannert, Martin von Löwis, Andrew
2075MacIntyre, Lalo Martins, Chad Netzer, Gustavo Niemeyer, Neal Norwitz, Hans
2076Nowak, Chris Reedy, Francesco Ricciardi, Vinay Sajip, Neil Schemenauer, Roman
2077Suzi, Jason Tishler, Just van Rossum.
2078