PageRenderTime 772ms CodeModel.GetById 125ms app.highlight 196ms RepoModel.GetById 124ms app.codeStats 0ms

/Doc/extending/extending.rst

http://unladen-swallow.googlecode.com/
ReStructuredText | 1285 lines | 1003 code | 282 blank | 0 comment | 0 complexity | e856fcfebec3d0757532bec6ebdc9537 MD5 | raw file

Large files files are truncated, but you can click here to view the full file

   1.. highlightlang:: c
   2
   3
   4.. _extending-intro:
   5
   6******************************
   7Extending Python with C or C++
   8******************************
   9
  10It is quite easy to add new built-in modules to Python, if you know how to
  11program in C.  Such :dfn:`extension modules` can do two things that can't be
  12done directly in Python: they can implement new built-in object types, and they
  13can call C library functions and system calls.
  14
  15To support extensions, the Python API (Application Programmers Interface)
  16defines a set of functions, macros and variables that provide access to most
  17aspects of the Python run-time system.  The Python API is incorporated in a C
  18source file by including the header ``"Python.h"``.
  19
  20The compilation of an extension module depends on its intended use as well as on
  21your system setup; details are given in later chapters.
  22
  23
  24.. _extending-simpleexample:
  25
  26A Simple Example
  27================
  28
  29Let's create an extension module called ``spam`` (the favorite food of Monty
  30Python fans...) and let's say we want to create a Python interface to the C
  31library function :cfunc:`system`. [#]_ This function takes a null-terminated
  32character string as argument and returns an integer.  We want this function to
  33be callable from Python as follows::
  34
  35   >>> import spam
  36   >>> status = spam.system("ls -l")
  37
  38Begin by creating a file :file:`spammodule.c`.  (Historically, if a module is
  39called ``spam``, the C file containing its implementation is called
  40:file:`spammodule.c`; if the module name is very long, like ``spammify``, the
  41module name can be just :file:`spammify.c`.)
  42
  43The first line of our file can be::
  44
  45   #include <Python.h>
  46
  47which pulls in the Python API (you can add a comment describing the purpose of
  48the module and a copyright notice if you like).
  49
  50.. note::
  51
  52   Since Python may define some pre-processor definitions which affect the standard
  53   headers on some systems, you *must* include :file:`Python.h` before any standard
  54   headers are included.
  55
  56All user-visible symbols defined by :file:`Python.h` have a prefix of ``Py`` or
  57``PY``, except those defined in standard header files. For convenience, and
  58since they are used extensively by the Python interpreter, ``"Python.h"``
  59includes a few standard header files: ``<stdio.h>``, ``<string.h>``,
  60``<errno.h>``, and ``<stdlib.h>``.  If the latter header file does not exist on
  61your system, it declares the functions :cfunc:`malloc`, :cfunc:`free` and
  62:cfunc:`realloc` directly.
  63
  64The next thing we add to our module file is the C function that will be called
  65when the Python expression ``spam.system(string)`` is evaluated (we'll see
  66shortly how it ends up being called)::
  67
  68   static PyObject *
  69   spam_system(PyObject *self, PyObject *args)
  70   {
  71       const char *command;
  72       int sts;
  73
  74       if (!PyArg_ParseTuple(args, "s", &command))
  75           return NULL;
  76       sts = system(command);
  77       return Py_BuildValue("i", sts);
  78   }
  79
  80There is a straightforward translation from the argument list in Python (for
  81example, the single expression ``"ls -l"``) to the arguments passed to the C
  82function.  The C function always has two arguments, conventionally named *self*
  83and *args*.
  84
  85The *self* argument is only used when the C function implements a built-in
  86method, not a function. In the example, *self* will always be a *NULL* pointer,
  87since we are defining a function, not a method.  (This is done so that the
  88interpreter doesn't have to understand two different types of C functions.)
  89
  90The *args* argument will be a pointer to a Python tuple object containing the
  91arguments.  Each item of the tuple corresponds to an argument in the call's
  92argument list.  The arguments are Python objects --- in order to do anything
  93with them in our C function we have to convert them to C values.  The function
  94:cfunc:`PyArg_ParseTuple` in the Python API checks the argument types and
  95converts them to C values.  It uses a template string to determine the required
  96types of the arguments as well as the types of the C variables into which to
  97store the converted values.  More about this later.
  98
  99:cfunc:`PyArg_ParseTuple` returns true (nonzero) if all arguments have the right
 100type and its components have been stored in the variables whose addresses are
 101passed.  It returns false (zero) if an invalid argument list was passed.  In the
 102latter case it also raises an appropriate exception so the calling function can
 103return *NULL* immediately (as we saw in the example).
 104
 105
 106.. _extending-errors:
 107
 108Intermezzo: Errors and Exceptions
 109=================================
 110
 111An important convention throughout the Python interpreter is the following: when
 112a function fails, it should set an exception condition and return an error value
 113(usually a *NULL* pointer).  Exceptions are stored in a static global variable
 114inside the interpreter; if this variable is *NULL* no exception has occurred.  A
 115second global variable stores the "associated value" of the exception (the
 116second argument to :keyword:`raise`).  A third variable contains the stack
 117traceback in case the error originated in Python code.  These three variables
 118are the C equivalents of the Python variables ``sys.exc_type``,
 119``sys.exc_value`` and ``sys.exc_traceback`` (see the section on module
 120:mod:`sys` in the Python Library Reference).  It is important to know about them
 121to understand how errors are passed around.
 122
 123The Python API defines a number of functions to set various types of exceptions.
 124
 125The most common one is :cfunc:`PyErr_SetString`.  Its arguments are an exception
 126object and a C string.  The exception object is usually a predefined object like
 127:cdata:`PyExc_ZeroDivisionError`.  The C string indicates the cause of the error
 128and is converted to a Python string object and stored as the "associated value"
 129of the exception.
 130
 131Another useful function is :cfunc:`PyErr_SetFromErrno`, which only takes an
 132exception argument and constructs the associated value by inspection of the
 133global variable :cdata:`errno`.  The most general function is
 134:cfunc:`PyErr_SetObject`, which takes two object arguments, the exception and
 135its associated value.  You don't need to :cfunc:`Py_INCREF` the objects passed
 136to any of these functions.
 137
 138You can test non-destructively whether an exception has been set with
 139:cfunc:`PyErr_Occurred`.  This returns the current exception object, or *NULL*
 140if no exception has occurred.  You normally don't need to call
 141:cfunc:`PyErr_Occurred` to see whether an error occurred in a function call,
 142since you should be able to tell from the return value.
 143
 144When a function *f* that calls another function *g* detects that the latter
 145fails, *f* should itself return an error value (usually *NULL* or ``-1``).  It
 146should *not* call one of the :cfunc:`PyErr_\*` functions --- one has already
 147been called by *g*. *f*'s caller is then supposed to also return an error
 148indication to *its* caller, again *without* calling :cfunc:`PyErr_\*`, and so on
 149--- the most detailed cause of the error was already reported by the function
 150that first detected it.  Once the error reaches the Python interpreter's main
 151loop, this aborts the currently executing Python code and tries to find an
 152exception handler specified by the Python programmer.
 153
 154(There are situations where a module can actually give a more detailed error
 155message by calling another :cfunc:`PyErr_\*` function, and in such cases it is
 156fine to do so.  As a general rule, however, this is not necessary, and can cause
 157information about the cause of the error to be lost: most operations can fail
 158for a variety of reasons.)
 159
 160To ignore an exception set by a function call that failed, the exception
 161condition must be cleared explicitly by calling :cfunc:`PyErr_Clear`.  The only
 162time C code should call :cfunc:`PyErr_Clear` is if it doesn't want to pass the
 163error on to the interpreter but wants to handle it completely by itself
 164(possibly by trying something else, or pretending nothing went wrong).
 165
 166Every failing :cfunc:`malloc` call must be turned into an exception --- the
 167direct caller of :cfunc:`malloc` (or :cfunc:`realloc`) must call
 168:cfunc:`PyErr_NoMemory` and return a failure indicator itself.  All the
 169object-creating functions (for example, :cfunc:`PyInt_FromLong`) already do
 170this, so this note is only relevant to those who call :cfunc:`malloc` directly.
 171
 172Also note that, with the important exception of :cfunc:`PyArg_ParseTuple` and
 173friends, functions that return an integer status usually return a positive value
 174or zero for success and ``-1`` for failure, like Unix system calls.
 175
 176Finally, be careful to clean up garbage (by making :cfunc:`Py_XDECREF` or
 177:cfunc:`Py_DECREF` calls for objects you have already created) when you return
 178an error indicator!
 179
 180The choice of which exception to raise is entirely yours.  There are predeclared
 181C objects corresponding to all built-in Python exceptions, such as
 182:cdata:`PyExc_ZeroDivisionError`, which you can use directly. Of course, you
 183should choose exceptions wisely --- don't use :cdata:`PyExc_TypeError` to mean
 184that a file couldn't be opened (that should probably be :cdata:`PyExc_IOError`).
 185If something's wrong with the argument list, the :cfunc:`PyArg_ParseTuple`
 186function usually raises :cdata:`PyExc_TypeError`.  If you have an argument whose
 187value must be in a particular range or must satisfy other conditions,
 188:cdata:`PyExc_ValueError` is appropriate.
 189
 190You can also define a new exception that is unique to your module. For this, you
 191usually declare a static object variable at the beginning of your file::
 192
 193   static PyObject *SpamError;
 194
 195and initialize it in your module's initialization function (:cfunc:`initspam`)
 196with an exception object (leaving out the error checking for now)::
 197
 198   PyMODINIT_FUNC
 199   initspam(void)
 200   {
 201       PyObject *m;
 202
 203       m = Py_InitModule("spam", SpamMethods);
 204       if (m == NULL)
 205           return;
 206
 207       SpamError = PyErr_NewException("spam.error", NULL, NULL);
 208       Py_INCREF(SpamError);
 209       PyModule_AddObject(m, "error", SpamError);
 210   }
 211
 212Note that the Python name for the exception object is :exc:`spam.error`.  The
 213:cfunc:`PyErr_NewException` function may create a class with the base class
 214being :exc:`Exception` (unless another class is passed in instead of *NULL*),
 215described in :ref:`bltin-exceptions`.
 216
 217Note also that the :cdata:`SpamError` variable retains a reference to the newly
 218created exception class; this is intentional!  Since the exception could be
 219removed from the module by external code, an owned reference to the class is
 220needed to ensure that it will not be discarded, causing :cdata:`SpamError` to
 221become a dangling pointer. Should it become a dangling pointer, C code which
 222raises the exception could cause a core dump or other unintended side effects.
 223
 224We discuss the use of PyMODINIT_FUNC as a function return type later in this
 225sample.
 226
 227
 228.. _backtoexample:
 229
 230Back to the Example
 231===================
 232
 233Going back to our example function, you should now be able to understand this
 234statement::
 235
 236   if (!PyArg_ParseTuple(args, "s", &command))
 237       return NULL;
 238
 239It returns *NULL* (the error indicator for functions returning object pointers)
 240if an error is detected in the argument list, relying on the exception set by
 241:cfunc:`PyArg_ParseTuple`.  Otherwise the string value of the argument has been
 242copied to the local variable :cdata:`command`.  This is a pointer assignment and
 243you are not supposed to modify the string to which it points (so in Standard C,
 244the variable :cdata:`command` should properly be declared as ``const char
 245*command``).
 246
 247The next statement is a call to the Unix function :cfunc:`system`, passing it
 248the string we just got from :cfunc:`PyArg_ParseTuple`::
 249
 250   sts = system(command);
 251
 252Our :func:`spam.system` function must return the value of :cdata:`sts` as a
 253Python object.  This is done using the function :cfunc:`Py_BuildValue`, which is
 254something like the inverse of :cfunc:`PyArg_ParseTuple`: it takes a format
 255string and an arbitrary number of C values, and returns a new Python object.
 256More info on :cfunc:`Py_BuildValue` is given later. ::
 257
 258   return Py_BuildValue("i", sts);
 259
 260In this case, it will return an integer object.  (Yes, even integers are objects
 261on the heap in Python!)
 262
 263If you have a C function that returns no useful argument (a function returning
 264:ctype:`void`), the corresponding Python function must return ``None``.   You
 265need this idiom to do so (which is implemented by the :cmacro:`Py_RETURN_NONE`
 266macro)::
 267
 268   Py_INCREF(Py_None);
 269   return Py_None;
 270
 271:cdata:`Py_None` is the C name for the special Python object ``None``.  It is a
 272genuine Python object rather than a *NULL* pointer, which means "error" in most
 273contexts, as we have seen.
 274
 275
 276.. _methodtable:
 277
 278The Module's Method Table and Initialization Function
 279=====================================================
 280
 281I promised to show how :cfunc:`spam_system` is called from Python programs.
 282First, we need to list its name and address in a "method table"::
 283
 284   static PyMethodDef SpamMethods[] = {
 285       ...
 286       {"system",  spam_system, METH_VARARGS,
 287        "Execute a shell command."},
 288       ...
 289       {NULL, NULL, 0, NULL}        /* Sentinel */
 290   };
 291
 292Note the third entry (``METH_VARARGS``).  This is a flag telling the interpreter
 293the calling convention to be used for the C function.  It should normally always
 294be ``METH_VARARGS`` or ``METH_VARARGS | METH_KEYWORDS``; a value of ``0`` means
 295that an obsolete variant of :cfunc:`PyArg_ParseTuple` is used.
 296
 297When using only ``METH_VARARGS``, the function should expect the Python-level
 298parameters to be passed in as a tuple acceptable for parsing via
 299:cfunc:`PyArg_ParseTuple`; more information on this function is provided below.
 300
 301The :const:`METH_KEYWORDS` bit may be set in the third field if keyword
 302arguments should be passed to the function.  In this case, the C function should
 303accept a third ``PyObject *`` parameter which will be a dictionary of keywords.
 304Use :cfunc:`PyArg_ParseTupleAndKeywords` to parse the arguments to such a
 305function.
 306
 307The method table must be passed to the interpreter in the module's
 308initialization function.  The initialization function must be named
 309:cfunc:`initname`, where *name* is the name of the module, and should be the
 310only non-\ ``static`` item defined in the module file::
 311
 312   PyMODINIT_FUNC
 313   initspam(void)
 314   {
 315       (void) Py_InitModule("spam", SpamMethods);
 316   }
 317
 318Note that PyMODINIT_FUNC declares the function as ``void`` return type,
 319declares any special linkage declarations required by the platform, and for  C++
 320declares the function as ``extern "C"``.
 321
 322When the Python program imports module :mod:`spam` for the first time,
 323:cfunc:`initspam` is called. (See below for comments about embedding Python.)
 324It calls :cfunc:`Py_InitModule`, which creates a "module object" (which is
 325inserted in the dictionary ``sys.modules`` under the key ``"spam"``), and
 326inserts built-in function objects into the newly created module based upon the
 327table (an array of :ctype:`PyMethodDef` structures) that was passed as its
 328second argument. :cfunc:`Py_InitModule` returns a pointer to the module object
 329that it creates (which is unused here).  It may abort with a fatal error for
 330certain errors, or return *NULL* if the module could not be initialized
 331satisfactorily.
 332
 333When embedding Python, the :cfunc:`initspam` function is not called
 334automatically unless there's an entry in the :cdata:`_PyImport_Inittab` table.
 335The easiest way to handle this is to statically initialize your
 336statically-linked modules by directly calling :cfunc:`initspam` after the call
 337to :cfunc:`Py_Initialize`::
 338
 339   int
 340   main(int argc, char *argv[])
 341   {
 342       /* Pass argv[0] to the Python interpreter */
 343       Py_SetProgramName(argv[0]);
 344
 345       /* Initialize the Python interpreter.  Required. */
 346       Py_Initialize();
 347
 348       /* Add a static module */
 349       initspam();
 350
 351An example may be found in the file :file:`Demo/embed/demo.c` in the Python
 352source distribution.
 353
 354.. note::
 355
 356   Removing entries from ``sys.modules`` or importing compiled modules into
 357   multiple interpreters within a process (or following a :cfunc:`fork` without an
 358   intervening :cfunc:`exec`) can create problems for some extension modules.
 359   Extension module authors should exercise caution when initializing internal data
 360   structures. Note also that the :func:`reload` function can be used with
 361   extension modules, and will call the module initialization function
 362   (:cfunc:`initspam` in the example), but will not load the module again if it was
 363   loaded from a dynamically loadable object file (:file:`.so` on Unix,
 364   :file:`.dll` on Windows).
 365
 366A more substantial example module is included in the Python source distribution
 367as :file:`Modules/xxmodule.c`.  This file may be used as a  template or simply
 368read as an example.  The :program:`modulator.py` script included in the source
 369distribution or Windows install provides  a simple graphical user interface for
 370declaring the functions and objects which a module should implement, and can
 371generate a template which can be filled in.  The script lives in the
 372:file:`Tools/modulator/` directory; see the :file:`README` file there for more
 373information.
 374
 375
 376.. _compilation:
 377
 378Compilation and Linkage
 379=======================
 380
 381There are two more things to do before you can use your new extension: compiling
 382and linking it with the Python system.  If you use dynamic loading, the details
 383may depend on the style of dynamic loading your system uses; see the chapters
 384about building extension modules (chapter :ref:`building`) and additional
 385information that pertains only to building on Windows (chapter
 386:ref:`building-on-windows`) for more information about this.
 387
 388If you can't use dynamic loading, or if you want to make your module a permanent
 389part of the Python interpreter, you will have to change the configuration setup
 390and rebuild the interpreter.  Luckily, this is very simple on Unix: just place
 391your file (:file:`spammodule.c` for example) in the :file:`Modules/` directory
 392of an unpacked source distribution, add a line to the file
 393:file:`Modules/Setup.local` describing your file::
 394
 395   spam spammodule.o
 396
 397and rebuild the interpreter by running :program:`make` in the toplevel
 398directory.  You can also run :program:`make` in the :file:`Modules/`
 399subdirectory, but then you must first rebuild :file:`Makefile` there by running
 400':program:`make` Makefile'.  (This is necessary each time you change the
 401:file:`Setup` file.)
 402
 403If your module requires additional libraries to link with, these can be listed
 404on the line in the configuration file as well, for instance::
 405
 406   spam spammodule.o -lX11
 407
 408
 409.. _callingpython:
 410
 411Calling Python Functions from C
 412===============================
 413
 414So far we have concentrated on making C functions callable from Python.  The
 415reverse is also useful: calling Python functions from C. This is especially the
 416case for libraries that support so-called "callback" functions.  If a C
 417interface makes use of callbacks, the equivalent Python often needs to provide a
 418callback mechanism to the Python programmer; the implementation will require
 419calling the Python callback functions from a C callback.  Other uses are also
 420imaginable.
 421
 422Fortunately, the Python interpreter is easily called recursively, and there is a
 423standard interface to call a Python function.  (I won't dwell on how to call the
 424Python parser with a particular string as input --- if you're interested, have a
 425look at the implementation of the :option:`-c` command line option in
 426:file:`Modules/main.c` from the Python source code.)
 427
 428Calling a Python function is easy.  First, the Python program must somehow pass
 429you the Python function object.  You should provide a function (or some other
 430interface) to do this.  When this function is called, save a pointer to the
 431Python function object (be careful to :cfunc:`Py_INCREF` it!) in a global
 432variable --- or wherever you see fit. For example, the following function might
 433be part of a module definition::
 434
 435   static PyObject *my_callback = NULL;
 436
 437   static PyObject *
 438   my_set_callback(PyObject *dummy, PyObject *args)
 439   {
 440       PyObject *result = NULL;
 441       PyObject *temp;
 442
 443       if (PyArg_ParseTuple(args, "O:set_callback", &temp)) {
 444           if (!PyCallable_Check(temp)) {
 445               PyErr_SetString(PyExc_TypeError, "parameter must be callable");
 446               return NULL;
 447           }
 448           Py_XINCREF(temp);         /* Add a reference to new callback */
 449           Py_XDECREF(my_callback);  /* Dispose of previous callback */
 450           my_callback = temp;       /* Remember new callback */
 451           /* Boilerplate to return "None" */
 452           Py_INCREF(Py_None);
 453           result = Py_None;
 454       }
 455       return result;
 456   }
 457
 458This function must be registered with the interpreter using the
 459:const:`METH_VARARGS` flag; this is described in section :ref:`methodtable`.  The
 460:cfunc:`PyArg_ParseTuple` function and its arguments are documented in section
 461:ref:`parsetuple`.
 462
 463The macros :cfunc:`Py_XINCREF` and :cfunc:`Py_XDECREF` increment/decrement the
 464reference count of an object and are safe in the presence of *NULL* pointers
 465(but note that *temp* will not be  *NULL* in this context).  More info on them
 466in section :ref:`refcounts`.
 467
 468.. index:: single: PyObject_CallObject()
 469
 470Later, when it is time to call the function, you call the C function
 471:cfunc:`PyObject_CallObject`.  This function has two arguments, both pointers to
 472arbitrary Python objects: the Python function, and the argument list.  The
 473argument list must always be a tuple object, whose length is the number of
 474arguments.  To call the Python function with no arguments, pass in NULL, or
 475an empty tuple; to call it with one argument, pass a singleton tuple.
 476:cfunc:`Py_BuildValue` returns a tuple when its format string consists of zero
 477or more format codes between parentheses.  For example::
 478
 479   int arg;
 480   PyObject *arglist;
 481   PyObject *result;
 482   ...
 483   arg = 123;
 484   ...
 485   /* Time to call the callback */
 486   arglist = Py_BuildValue("(i)", arg);
 487   result = PyObject_CallObject(my_callback, arglist);
 488   Py_DECREF(arglist);
 489
 490:cfunc:`PyObject_CallObject` returns a Python object pointer: this is the return
 491value of the Python function.  :cfunc:`PyObject_CallObject` is
 492"reference-count-neutral" with respect to its arguments.  In the example a new
 493tuple was created to serve as the argument list, which is :cfunc:`Py_DECREF`\
 494-ed immediately after the call.
 495
 496The return value of :cfunc:`PyObject_CallObject` is "new": either it is a brand
 497new object, or it is an existing object whose reference count has been
 498incremented.  So, unless you want to save it in a global variable, you should
 499somehow :cfunc:`Py_DECREF` the result, even (especially!) if you are not
 500interested in its value.
 501
 502Before you do this, however, it is important to check that the return value
 503isn't *NULL*.  If it is, the Python function terminated by raising an exception.
 504If the C code that called :cfunc:`PyObject_CallObject` is called from Python, it
 505should now return an error indication to its Python caller, so the interpreter
 506can print a stack trace, or the calling Python code can handle the exception.
 507If this is not possible or desirable, the exception should be cleared by calling
 508:cfunc:`PyErr_Clear`.  For example::
 509
 510   if (result == NULL)
 511       return NULL; /* Pass error back */
 512   ...use result...
 513   Py_DECREF(result);
 514
 515Depending on the desired interface to the Python callback function, you may also
 516have to provide an argument list to :cfunc:`PyObject_CallObject`.  In some cases
 517the argument list is also provided by the Python program, through the same
 518interface that specified the callback function.  It can then be saved and used
 519in the same manner as the function object.  In other cases, you may have to
 520construct a new tuple to pass as the argument list.  The simplest way to do this
 521is to call :cfunc:`Py_BuildValue`.  For example, if you want to pass an integral
 522event code, you might use the following code::
 523
 524   PyObject *arglist;
 525   ...
 526   arglist = Py_BuildValue("(l)", eventcode);
 527   result = PyObject_CallObject(my_callback, arglist);
 528   Py_DECREF(arglist);
 529   if (result == NULL)
 530       return NULL; /* Pass error back */
 531   /* Here maybe use the result */
 532   Py_DECREF(result);
 533
 534Note the placement of ``Py_DECREF(arglist)`` immediately after the call, before
 535the error check!  Also note that strictly speaking this code is not complete:
 536:cfunc:`Py_BuildValue` may run out of memory, and this should be checked.
 537
 538You may also call a function with keyword arguments by using
 539:cfunc:`PyObject_Call`, which supports arguments and keyword arguments.  As in
 540the above example, we use :cfunc:`Py_BuildValue` to construct the dictionary. ::
 541
 542   PyObject *dict;
 543   ...
 544   dict = Py_BuildValue("{s:i}", "name", val);
 545   result = PyObject_Call(my_callback, NULL, dict);
 546   Py_DECREF(dict);
 547   if (result == NULL)
 548       return NULL; /* Pass error back */
 549   /* Here maybe use the result */
 550   Py_DECREF(result);
 551
 552
 553.. _parsetuple:
 554
 555Extracting Parameters in Extension Functions
 556============================================
 557
 558.. index:: single: PyArg_ParseTuple()
 559
 560The :cfunc:`PyArg_ParseTuple` function is declared as follows::
 561
 562   int PyArg_ParseTuple(PyObject *arg, char *format, ...);
 563
 564The *arg* argument must be a tuple object containing an argument list passed
 565from Python to a C function.  The *format* argument must be a format string,
 566whose syntax is explained in :ref:`arg-parsing` in the Python/C API Reference
 567Manual.  The remaining arguments must be addresses of variables whose type is
 568determined by the format string.
 569
 570Note that while :cfunc:`PyArg_ParseTuple` checks that the Python arguments have
 571the required types, it cannot check the validity of the addresses of C variables
 572passed to the call: if you make mistakes there, your code will probably crash or
 573at least overwrite random bits in memory.  So be careful!
 574
 575Note that any Python object references which are provided to the caller are
 576*borrowed* references; do not decrement their reference count!
 577
 578Some example calls::
 579
 580   int ok;
 581   int i, j;
 582   long k, l;
 583   const char *s;
 584   int size;
 585
 586   ok = PyArg_ParseTuple(args, ""); /* No arguments */
 587       /* Python call: f() */
 588
 589::
 590
 591   ok = PyArg_ParseTuple(args, "s", &s); /* A string */
 592       /* Possible Python call: f('whoops!') */
 593
 594::
 595
 596   ok = PyArg_ParseTuple(args, "lls", &k, &l, &s); /* Two longs and a string */
 597       /* Possible Python call: f(1, 2, 'three') */
 598
 599::
 600
 601   ok = PyArg_ParseTuple(args, "(ii)s#", &i, &j, &s, &size);
 602       /* A pair of ints and a string, whose size is also returned */
 603       /* Possible Python call: f((1, 2), 'three') */
 604
 605::
 606
 607   {
 608       const char *file;
 609       const char *mode = "r";
 610       int bufsize = 0;
 611       ok = PyArg_ParseTuple(args, "s|si", &file, &mode, &bufsize);
 612       /* A string, and optionally another string and an integer */
 613       /* Possible Python calls:
 614          f('spam')
 615          f('spam', 'w')
 616          f('spam', 'wb', 100000) */
 617   }
 618
 619::
 620
 621   {
 622       int left, top, right, bottom, h, v;
 623       ok = PyArg_ParseTuple(args, "((ii)(ii))(ii)",
 624                &left, &top, &right, &bottom, &h, &v);
 625       /* A rectangle and a point */
 626       /* Possible Python call:
 627          f(((0, 0), (400, 300)), (10, 10)) */
 628   }
 629
 630::
 631
 632   {
 633       Py_complex c;
 634       ok = PyArg_ParseTuple(args, "D:myfunction", &c);
 635       /* a complex, also providing a function name for errors */
 636       /* Possible Python call: myfunction(1+2j) */
 637   }
 638
 639
 640.. _parsetupleandkeywords:
 641
 642Keyword Parameters for Extension Functions
 643==========================================
 644
 645.. index:: single: PyArg_ParseTupleAndKeywords()
 646
 647The :cfunc:`PyArg_ParseTupleAndKeywords` function is declared as follows::
 648
 649   int PyArg_ParseTupleAndKeywords(PyObject *arg, PyObject *kwdict,
 650                                   char *format, char *kwlist[], ...);
 651
 652The *arg* and *format* parameters are identical to those of the
 653:cfunc:`PyArg_ParseTuple` function.  The *kwdict* parameter is the dictionary of
 654keywords received as the third parameter from the Python runtime.  The *kwlist*
 655parameter is a *NULL*-terminated list of strings which identify the parameters;
 656the names are matched with the type information from *format* from left to
 657right.  On success, :cfunc:`PyArg_ParseTupleAndKeywords` returns true, otherwise
 658it returns false and raises an appropriate exception.
 659
 660.. note::
 661
 662   Nested tuples cannot be parsed when using keyword arguments!  Keyword parameters
 663   passed in which are not present in the *kwlist* will cause :exc:`TypeError` to
 664   be raised.
 665
 666.. index:: single: Philbrick, Geoff
 667
 668Here is an example module which uses keywords, based on an example by Geoff
 669Philbrick (philbrick@hks.com)::
 670
 671   #include "Python.h"
 672
 673   static PyObject *
 674   keywdarg_parrot(PyObject *self, PyObject *args, PyObject *keywds)
 675   {
 676       int voltage;
 677       char *state = "a stiff";
 678       char *action = "voom";
 679       char *type = "Norwegian Blue";
 680
 681       static char *kwlist[] = {"voltage", "state", "action", "type", NULL};
 682
 683       if (!PyArg_ParseTupleAndKeywords(args, keywds, "i|sss", kwlist,
 684                                        &voltage, &state, &action, &type))
 685           return NULL;
 686
 687       printf("-- This parrot wouldn't %s if you put %i Volts through it.\n",
 688              action, voltage);
 689       printf("-- Lovely plumage, the %s -- It's %s!\n", type, state);
 690
 691       Py_INCREF(Py_None);
 692
 693       return Py_None;
 694   }
 695
 696   static PyMethodDef keywdarg_methods[] = {
 697       /* The cast of the function is necessary since PyCFunction values
 698        * only take two PyObject* parameters, and keywdarg_parrot() takes
 699        * three.
 700        */
 701       {"parrot", (PyCFunction)keywdarg_parrot, METH_VARARGS | METH_KEYWORDS,
 702        "Print a lovely skit to standard output."},
 703       {NULL, NULL, 0, NULL}   /* sentinel */
 704   };
 705
 706::
 707
 708   void
 709   initkeywdarg(void)
 710   {
 711     /* Create the module and add the functions */
 712     Py_InitModule("keywdarg", keywdarg_methods);
 713   }
 714
 715
 716.. _buildvalue:
 717
 718Building Arbitrary Values
 719=========================
 720
 721This function is the counterpart to :cfunc:`PyArg_ParseTuple`.  It is declared
 722as follows::
 723
 724   PyObject *Py_BuildValue(char *format, ...);
 725
 726It recognizes a set of format units similar to the ones recognized by
 727:cfunc:`PyArg_ParseTuple`, but the arguments (which are input to the function,
 728not output) must not be pointers, just values.  It returns a new Python object,
 729suitable for returning from a C function called from Python.
 730
 731One difference with :cfunc:`PyArg_ParseTuple`: while the latter requires its
 732first argument to be a tuple (since Python argument lists are always represented
 733as tuples internally), :cfunc:`Py_BuildValue` does not always build a tuple.  It
 734builds a tuple only if its format string contains two or more format units. If
 735the format string is empty, it returns ``None``; if it contains exactly one
 736format unit, it returns whatever object is described by that format unit.  To
 737force it to return a tuple of size 0 or one, parenthesize the format string.
 738
 739Examples (to the left the call, to the right the resulting Python value)::
 740
 741   Py_BuildValue("")                        None
 742   Py_BuildValue("i", 123)                  123
 743   Py_BuildValue("iii", 123, 456, 789)      (123, 456, 789)
 744   Py_BuildValue("s", "hello")              'hello'
 745   Py_BuildValue("ss", "hello", "world")    ('hello', 'world')
 746   Py_BuildValue("s#", "hello", 4)          'hell'
 747   Py_BuildValue("()")                      ()
 748   Py_BuildValue("(i)", 123)                (123,)
 749   Py_BuildValue("(ii)", 123, 456)          (123, 456)
 750   Py_BuildValue("(i,i)", 123, 456)         (123, 456)
 751   Py_BuildValue("[i,i]", 123, 456)         [123, 456]
 752   Py_BuildValue("{s:i,s:i}",
 753                 "abc", 123, "def", 456)    {'abc': 123, 'def': 456}
 754   Py_BuildValue("((ii)(ii)) (ii)",
 755                 1, 2, 3, 4, 5, 6)          (((1, 2), (3, 4)), (5, 6))
 756
 757
 758.. _refcounts:
 759
 760Reference Counts
 761================
 762
 763In languages like C or C++, the programmer is responsible for dynamic allocation
 764and deallocation of memory on the heap.  In C, this is done using the functions
 765:cfunc:`malloc` and :cfunc:`free`.  In C++, the operators ``new`` and
 766``delete`` are used with essentially the same meaning and we'll restrict
 767the following discussion to the C case.
 768
 769Every block of memory allocated with :cfunc:`malloc` should eventually be
 770returned to the pool of available memory by exactly one call to :cfunc:`free`.
 771It is important to call :cfunc:`free` at the right time.  If a block's address
 772is forgotten but :cfunc:`free` is not called for it, the memory it occupies
 773cannot be reused until the program terminates.  This is called a :dfn:`memory
 774leak`.  On the other hand, if a program calls :cfunc:`free` for a block and then
 775continues to use the block, it creates a conflict with re-use of the block
 776through another :cfunc:`malloc` call.  This is called :dfn:`using freed memory`.
 777It has the same bad consequences as referencing uninitialized data --- core
 778dumps, wrong results, mysterious crashes.
 779
 780Common causes of memory leaks are unusual paths through the code.  For instance,
 781a function may allocate a block of memory, do some calculation, and then free
 782the block again.  Now a change in the requirements for the function may add a
 783test to the calculation that detects an error condition and can return
 784prematurely from the function.  It's easy to forget to free the allocated memory
 785block when taking this premature exit, especially when it is added later to the
 786code.  Such leaks, once introduced, often go undetected for a long time: the
 787error exit is taken only in a small fraction of all calls, and most modern
 788machines have plenty of virtual memory, so the leak only becomes apparent in a
 789long-running process that uses the leaking function frequently.  Therefore, it's
 790important to prevent leaks from happening by having a coding convention or
 791strategy that minimizes this kind of errors.
 792
 793Since Python makes heavy use of :cfunc:`malloc` and :cfunc:`free`, it needs a
 794strategy to avoid memory leaks as well as the use of freed memory.  The chosen
 795method is called :dfn:`reference counting`.  The principle is simple: every
 796object contains a counter, which is incremented when a reference to the object
 797is stored somewhere, and which is decremented when a reference to it is deleted.
 798When the counter reaches zero, the last reference to the object has been deleted
 799and the object is freed.
 800
 801An alternative strategy is called :dfn:`automatic garbage collection`.
 802(Sometimes, reference counting is also referred to as a garbage collection
 803strategy, hence my use of "automatic" to distinguish the two.)  The big
 804advantage of automatic garbage collection is that the user doesn't need to call
 805:cfunc:`free` explicitly.  (Another claimed advantage is an improvement in speed
 806or memory usage --- this is no hard fact however.)  The disadvantage is that for
 807C, there is no truly portable automatic garbage collector, while reference
 808counting can be implemented portably (as long as the functions :cfunc:`malloc`
 809and :cfunc:`free` are available --- which the C Standard guarantees). Maybe some
 810day a sufficiently portable automatic garbage collector will be available for C.
 811Until then, we'll have to live with reference counts.
 812
 813While Python uses the traditional reference counting implementation, it also
 814offers a cycle detector that works to detect reference cycles.  This allows
 815applications to not worry about creating direct or indirect circular references;
 816these are the weakness of garbage collection implemented using only reference
 817counting.  Reference cycles consist of objects which contain (possibly indirect)
 818references to themselves, so that each object in the cycle has a reference count
 819which is non-zero.  Typical reference counting implementations are not able to
 820reclaim the memory belonging to any objects in a reference cycle, or referenced
 821from the objects in the cycle, even though there are no further references to
 822the cycle itself.
 823
 824The cycle detector is able to detect garbage cycles and can reclaim them so long
 825as there are no finalizers implemented in Python (:meth:`__del__` methods).
 826When there are such finalizers, the detector exposes the cycles through the
 827:mod:`gc` module (specifically, the
 828``garbage`` variable in that module).  The :mod:`gc` module also exposes a way
 829to run the detector (the :func:`collect` function), as well as configuration
 830interfaces and the ability to disable the detector at runtime.  The cycle
 831detector is considered an optional component; though it is included by default,
 832it can be disabled at build time using the :option:`--without-cycle-gc` option
 833to the :program:`configure` script on Unix platforms (including Mac OS X) or by
 834removing the definition of ``WITH_CYCLE_GC`` in the :file:`pyconfig.h` header on
 835other platforms.  If the cycle detector is disabled in this way, the :mod:`gc`
 836module will not be available.
 837
 838
 839.. _refcountsinpython:
 840
 841Reference Counting in Python
 842----------------------------
 843
 844There are two macros, ``Py_INCREF(x)`` and ``Py_DECREF(x)``, which handle the
 845incrementing and decrementing of the reference count. :cfunc:`Py_DECREF` also
 846frees the object when the count reaches zero. For flexibility, it doesn't call
 847:cfunc:`free` directly --- rather, it makes a call through a function pointer in
 848the object's :dfn:`type object`.  For this purpose (and others), every object
 849also contains a pointer to its type object.
 850
 851The big question now remains: when to use ``Py_INCREF(x)`` and ``Py_DECREF(x)``?
 852Let's first introduce some terms.  Nobody "owns" an object; however, you can
 853:dfn:`own a reference` to an object.  An object's reference count is now defined
 854as the number of owned references to it.  The owner of a reference is
 855responsible for calling :cfunc:`Py_DECREF` when the reference is no longer
 856needed.  Ownership of a reference can be transferred.  There are three ways to
 857dispose of an owned reference: pass it on, store it, or call :cfunc:`Py_DECREF`.
 858Forgetting to dispose of an owned reference creates a memory leak.
 859
 860It is also possible to :dfn:`borrow` [#]_ a reference to an object.  The
 861borrower of a reference should not call :cfunc:`Py_DECREF`.  The borrower must
 862not hold on to the object longer than the owner from which it was borrowed.
 863Using a borrowed reference after the owner has disposed of it risks using freed
 864memory and should be avoided completely. [#]_
 865
 866The advantage of borrowing over owning a reference is that you don't need to
 867take care of disposing of the reference on all possible paths through the code
 868--- in other words, with a borrowed reference you don't run the risk of leaking
 869when a premature exit is taken.  The disadvantage of borrowing over owning is
 870that there are some subtle situations where in seemingly correct code a borrowed
 871reference can be used after the owner from which it was borrowed has in fact
 872disposed of it.
 873
 874A borrowed reference can be changed into an owned reference by calling
 875:cfunc:`Py_INCREF`.  This does not affect the status of the owner from which the
 876reference was borrowed --- it creates a new owned reference, and gives full
 877owner responsibilities (the new owner must dispose of the reference properly, as
 878well as the previous owner).
 879
 880
 881.. _ownershiprules:
 882
 883Ownership Rules
 884---------------
 885
 886Whenever an object reference is passed into or out of a function, it is part of
 887the function's interface specification whether ownership is transferred with the
 888reference or not.
 889
 890Most functions that return a reference to an object pass on ownership with the
 891reference.  In particular, all functions whose function it is to create a new
 892object, such as :cfunc:`PyInt_FromLong` and :cfunc:`Py_BuildValue`, pass
 893ownership to the receiver.  Even if the object is not actually new, you still
 894receive ownership of a new reference to that object.  For instance,
 895:cfunc:`PyInt_FromLong` maintains a cache of popular values and can return a
 896reference to a cached item.
 897
 898Many functions that extract objects from other objects also transfer ownership
 899with the reference, for instance :cfunc:`PyObject_GetAttrString`.  The picture
 900is less clear, here, however, since a few common routines are exceptions:
 901:cfunc:`PyTuple_GetItem`, :cfunc:`PyList_GetItem`, :cfunc:`PyDict_GetItem`, and
 902:cfunc:`PyDict_GetItemString` all return references that you borrow from the
 903tuple, list or dictionary.
 904
 905The function :cfunc:`PyImport_AddModule` also returns a borrowed reference, even
 906though it may actually create the object it returns: this is possible because an
 907owned reference to the object is stored in ``sys.modules``.
 908
 909When you pass an object reference into another function, in general, the
 910function borrows the reference from you --- if it needs to store it, it will use
 911:cfunc:`Py_INCREF` to become an independent owner.  There are exactly two
 912important exceptions to this rule: :cfunc:`PyTuple_SetItem` and
 913:cfunc:`PyList_SetItem`.  These functions take over ownership of the item passed
 914to them --- even if they fail!  (Note that :cfunc:`PyDict_SetItem` and friends
 915don't take over ownership --- they are "normal.")
 916
 917When a C function is called from Python, it borrows references to its arguments
 918from the caller.  The caller owns a reference to the object, so the borrowed
 919reference's lifetime is guaranteed until the function returns.  Only when such a
 920borrowed reference must be stored or passed on, it must be turned into an owned
 921reference by calling :cfunc:`Py_INCREF`.
 922
 923The object reference returned from a C function that is called from Python must
 924be an owned reference --- ownership is transferred from the function to its
 925caller.
 926
 927
 928.. _thinice:
 929
 930Thin Ice
 931--------
 932
 933There are a few situations where seemingly harmless use of a borrowed reference
 934can lead to problems.  These all have to do with implicit invocations of the
 935interpreter, which can cause the owner of a reference to dispose of it.
 936
 937The first and most important case to know about is using :cfunc:`Py_DECREF` on
 938an unrelated object while borrowing a reference to a list item.  For instance::
 939
 940   void
 941   bug(PyObject *list)
 942   {
 943       PyObject *item = PyList_GetItem(list, 0);
 944
 945       PyList_SetItem(list, 1, PyInt_FromLong(0L));
 946       PyObject_Print(item, stdout, 0); /* BUG! */
 947   }
 948
 949This function first borrows a reference to ``list[0]``, then replaces
 950``list[1]`` with the value ``0``, and finally prints the borrowed reference.
 951Looks harmless, right?  But it's not!
 952
 953Let's follow the control flow into :cfunc:`PyList_SetItem`.  The list owns
 954references to all its items, so when item 1 is replaced, it has to dispose of
 955the original item 1.  Now let's suppose the original item 1 was an instance of a
 956user-defined class, and let's further suppose that the class defined a
 957:meth:`__del__` method.  If this class instance has a reference count of 1,
 958disposing of it will call its :meth:`__del__` method.
 959
 960Since it is written in Python, the :meth:`__del__` method can execute arbitrary
 961Python code.  Could it perhaps do something to invalidate the reference to
 962``item`` in :cfunc:`bug`?  You bet!  Assuming that the list passed into
 963:cfunc:`bug` is accessible to the :meth:`__del__` method, it could execute a
 964statement to the effect of ``del list[0]``, and assuming this was the last
 965reference to that object, it would free the memory associated with it, thereby
 966invalidating ``item``.
 967
 968The solution, once you know the source of the problem, is easy: temporarily
 969increment the reference count.  The correct version of the function reads::
 970
 971   void
 972   no_bug(PyObject *list)
 973   {
 974       PyObject *item = PyList_GetItem(list, 0);
 975
 976       Py_INCREF(item);
 977       PyList_SetItem(list, 1, PyInt_FromLong(0L));
 978       PyObject_Print(item, stdout, 0);
 979       Py_DECREF(item);
 980   }
 981
 982This is a true story.  An older version of Python contained variants of this bug
 983and someone spent a considerable amount of time in a C debugger to figure out
 984why his :meth:`__del__` methods would fail...
 985
 986The second case of problems with a borrowed reference is a variant involving
 987threads.  Normally, multiple threads in the Python interpreter can't get in each
 988other's way, because there is a global lock protecting Python's entire object
 989space.  However, it is possible to temporarily release this lock using the macro
 990:cmacro:`Py_BEGIN_ALLOW_THREADS`, and to re-acquire it using
 991:cmacro:`Py_END_ALLOW_THREADS`.  This is common around blocking I/O calls, to
 992let other threads use the processor while waiting for the I/O to complete.
 993Obviously, the following function has the same problem as the previous one::
 994
 995   void
 996   bug(PyObject *list)
 997   {
 998       PyObject *item = PyList_GetItem(list, 0);
 999       Py_BEGIN_ALLOW_THREADS
1000       ...some blocking I/O call...
1001       Py_END_ALLOW_THREADS
1002       PyObject_Print(item, stdout, 0); /* BUG! */
1003   }
1004
1005
1006.. _nullpointers:
1007
1008NULL Pointers
1009-------------
1010
1011In general, functions that take object references as arguments do not expect you
1012to pass them *NULL* pointers, and will dump core (or cause later core dumps) if
1013you do so.  Functions that return object references generally return *NULL* only
1014to indicate that an exception occurred.  The reason for not testing for *NULL*
1015arguments is that functions often pass the objects they receive on to other
1016function --- if each function were to test for *NULL*, there would be a lot of
1017redundant tests and the code would run more slowly.
1018
1019It is better to test for *NULL* only at the "source:" when a pointer that may be
1020*NULL* is received, for example, from :cfunc:`malloc` or from a function that
1021may raise an exception.
1022
1023The macros :cfunc:`Py_INCREF` and :cfunc:`Py_DECREF` do not check for *NULL*
1024pointers --- however, their variants :cfunc:`Py_XINCREF` and :cfunc:`Py_XDECREF`
1025do.
1026
1027The macros for checking for a particular object type (``Pytype_Check()``) don't
1028check for *NULL* pointers --- again, there is much code that calls several of
1029these in a row to test an object against various different expected types, and
1030this would generate redundant tests.  There are no variants with *NULL*
1031checking.
1032
1033The C function calling mechanism guarantees that the argument list passed to C
1034functions (``args`` in the examples) is never *NULL* --- in fact it guarantees
1035that it is always a tuple. [#]_
1036
1037It is a severe error to ever let a *NULL* pointer "escape" to the Python user.
1038
1039.. Frank Stajano:
1040   A pedagogically buggy example, along the lines of the previous listing, would
1041   be helpful here -- showing in more concrete terms what sort of actions could
1042   cause the problem. I can't very well imagine it from the description.
1043
1044
1045.. _cplusplus:
1046
1047Writing Extensions in C++
1048=========================
1049
1050It is possible to write extension modules in C++.  Some restrictions apply.  If
1051the main program (the Python interpreter) is compiled and linked by the C
1052compiler, global or static objects with constructors cannot be used.  This is
1053not a problem if the main program is linked by the C++ compiler.  Functions that
1054will be called by the Python interpreter (in particular, module initialization
1055functions) have to be declared using ``extern "C"``. It is unnecessary to
1056enclose the Python header files in ``extern "C" {...}`` --- they use this form
1057already if the symbol ``__cplusplus`` is defined (all recent C++ compilers
1058define this symbol).
1059
1060
1061.. _using-cobjects:
1062
1063Providing a C API for an Extension Module
1064=========================================
1065
1066.. sectionauthor:: Konrad Hinsen <hinsen@cnrs-orleans.fr>
1067
1068
1069Many extension modules just provide new functions and types to be used from
1070Python, but sometimes the code in an extension module can be useful for other
1071extension modules. For example, an extension module could implement a type
1072"collection" which works like lists without order. Just like the standard Python
1073list type has a C API which permits extension modules to create and manipulate
1074lists, this new collection type should have a set of C functions for direct
1075manipulation from other extension modules.
1076
1077At first sight this seems easy: just write the functions (without declaring them
1078``static``, of course), provide an appropriate header file, and document
1079the C API. And in fact this would work if all extension modules were always
1080linked statically with the Python interpreter. When modules are used as shared
1081libraries, however, the symbols defined in one module may not be visible to
1082another module. The details of visibility depend on the operating system; some
1083systems use one global namespace for the Python interpreter and all extension
1084modules (Windows, for example), whereas others require an explicit list of
1085imported symbols at module link time (AIX is one example), or offer a choice of
1086different strategies (most Unices). And even if symbols are globally visible,
1087the module whose functions one wishes to call might not have been loaded yet!
1088
1089Portability therefore requires not to make any assumptions about symbol
1090visibility. This means that all symbols in extension modules should be declared
1091``static``, except for the module's initialization function, in order to
1092avoid name clashes with other extension modules (as discussed in section
1093:ref:`methodtable`). And it means that symbols that *should* be accessible from
1094other extension modules must be exported in a different way.
1095
1096Python provides a special mechanism to pass C-level information (pointers) from
1097one extension module to another one: CObjects. A CObject is a Python data type
1098which stores a pointer (:ctype:`void \*`).  CObjects can only be created and
1099accessed via their C API, but they can be passed around like any other Python
1100object. In particular,  they can be assigned to a name in an extension module's
1101namespace. Other extension modules can then import this module, retrieve the
1102value of this name, and then retrieve the pointer from the CObject.
1103
1104There are many ways in which CObjects can be used to export the C API of an
1105extension module. Each name could get its own CObject, or all C API pointers
1106could be stored in an array whose address is published in a CObject. And theā€¦

Large files files are truncated, but you can click here to view the full file