/Doc/tutorial/modules.rst

  1.. _tut-modules:
  2
  3*******
  4Modules
  5*******
  6
  7If you quit from the Python interpreter and enter it again, the definitions you
  8have made (functions and variables) are lost. Therefore, if you want to write a
  9somewhat longer program, you are better off using a text editor to prepare the
 10input for the interpreter and running it with that file as input instead.  This
 11is known as creating a *script*.  As your program gets longer, you may want to
 12split it into several files for easier maintenance.  You may also want to use a
 13handy function that you've written in several programs without copying its
 14definition into each program.
 15
 16To support this, Python has a way to put definitions in a file and use them in a
 17script or in an interactive instance of the interpreter. Such a file is called a
 18*module*; definitions from a module can be *imported* into other modules or into
 19the *main* module (the collection of variables that you have access to in a
 20script executed at the top level and in calculator mode).
 21
 22A module is a file containing Python definitions and statements.  The file name
 23is the module name with the suffix :file:`.py` appended.  Within a module, the
 24module's name (as a string) is available as the value of the global variable
 25``__name__``.  For instance, use your favorite text editor to create a file
 26called :file:`fibo.py` in the current directory with the following contents::
 27
 28   # Fibonacci numbers module
 29
 30   def fib(n):    # write Fibonacci series up to n
 31       a, b = 0, 1
 32       while b < n:
 33           print b,
 34           a, b = b, a+b
 35
 36   def fib2(n): # return Fibonacci series up to n
 37       result = []
 38       a, b = 0, 1
 39       while b < n:
 40           result.append(b)
 41           a, b = b, a+b
 42       return result
 43
 44Now enter the Python interpreter and import this module with the following
 45command::
 46
 47   >>> import fibo
 48
 49This does not enter the names of the functions defined in ``fibo``  directly in
 50the current symbol table; it only enters the module name ``fibo`` there. Using
 51the module name you can access the functions::
 52
 53   >>> fibo.fib(1000)
 54   1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987
 55   >>> fibo.fib2(100)
 56   [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89]
 57   >>> fibo.__name__
 58   'fibo'
 59
 60If you intend to use a function often you can assign it to a local name::
 61
 62   >>> fib = fibo.fib
 63   >>> fib(500)
 64   1 1 2 3 5 8 13 21 34 55 89 144 233 377
 65
 66
 67.. _tut-moremodules:
 68
 69More on Modules
 70===============
 71
 72A module can contain executable statements as well as function definitions.
 73These statements are intended to initialize the module. They are executed only
 74the *first* time the module is imported somewhere. [#]_
 75
 76Each module has its own private symbol table, which is used as the global symbol
 77table by all functions defined in the module. Thus, the author of a module can
 78use global variables in the module without worrying about accidental clashes
 79with a user's global variables. On the other hand, if you know what you are
 80doing you can touch a module's global variables with the same notation used to
 81refer to its functions, ``modname.itemname``.
 82
 83Modules can import other modules.  It is customary but not required to place all
 84:keyword:`import` statements at the beginning of a module (or script, for that
 85matter).  The imported module names are placed in the importing module's global
 86symbol table.
 87
 88There is a variant of the :keyword:`import` statement that imports names from a
 89module directly into the importing module's symbol table.  For example::
 90
 91   >>> from fibo import fib, fib2
 92   >>> fib(500)
 93   1 1 2 3 5 8 13 21 34 55 89 144 233 377
 94
 95This does not introduce the module name from which the imports are taken in the
 96local symbol table (so in the example, ``fibo`` is not defined).
 97
 98There is even a variant to import all names that a module defines::
 99
100   >>> from fibo import *
101   >>> fib(500)
102   1 1 2 3 5 8 13 21 34 55 89 144 233 377
103
104This imports all names except those beginning with an underscore (``_``).
105
106.. note::
107
108   For efficiency reasons, each module is only imported once per interpreter
109   session.  Therefore, if you change your modules, you must restart the
110   interpreter -- or, if it's just one module you want to test interactively,
111   use :func:`reload`, e.g. ``reload(modulename)``.
112
113
114.. _tut-modulesasscripts:
115
116Executing modules as scripts
117----------------------------
118
119When you run a Python module with ::
120
121   python fibo.py <arguments>
122
123the code in the module will be executed, just as if you imported it, but with
124the ``__name__`` set to ``"__main__"``.  That means that by adding this code at
125the end of your module::
126
127   if __name__ == "__main__":
128       import sys
129       fib(int(sys.argv[1]))
130
131you can make the file usable as a script as well as an importable module,
132because the code that parses the command line only runs if the module is
133executed as the "main" file::
134
135   $ python fibo.py 50
136   1 1 2 3 5 8 13 21 34
137
138If the module is imported, the code is not run::
139
140   >>> import fibo
141   >>>
142
143This is often used either to provide a convenient user interface to a module, or
144for testing purposes (running the module as a script executes a test suite).
145
146
147.. _tut-searchpath:
148
149The Module Search Path
150----------------------
151
152.. index:: triple: module; search; path
153
154When a module named :mod:`spam` is imported, the interpreter searches for a file
155named :file:`spam.py` in the current directory, and then in the list of
156directories specified by the environment variable :envvar:`PYTHONPATH`.  This
157has the same syntax as the shell variable :envvar:`PATH`, that is, a list of
158directory names.  When :envvar:`PYTHONPATH` is not set, or when the file is not
159found there, the search continues in an installation-dependent default path; on
160Unix, this is usually :file:`.:/usr/local/lib/python`.
161
162Actually, modules are searched in the list of directories given by the variable
163``sys.path`` which is initialized from the directory containing the input script
164(or the current directory), :envvar:`PYTHONPATH` and the installation- dependent
165default.  This allows Python programs that know what they're doing to modify or
166replace the module search path.  Note that because the directory containing the
167script being run is on the search path, it is important that the script not have
168the same name as a standard module, or Python will attempt to load the script as
169a module when that module is imported. This will generally be an error.  See
170section :ref:`tut-standardmodules` for more information.
171
172
173"Compiled" Python files
174-----------------------
175
176As an important speed-up of the start-up time for short programs that use a lot
177of standard modules, if a file called :file:`spam.pyc` exists in the directory
178where :file:`spam.py` is found, this is assumed to contain an
179already-"byte-compiled" version of the module :mod:`spam`. The modification time
180of the version of :file:`spam.py` used to create :file:`spam.pyc` is recorded in
181:file:`spam.pyc`, and the :file:`.pyc` file is ignored if these don't match.
182
183Normally, you don't need to do anything to create the :file:`spam.pyc` file.
184Whenever :file:`spam.py` is successfully compiled, an attempt is made to write
185the compiled version to :file:`spam.pyc`.  It is not an error if this attempt
186fails; if for any reason the file is not written completely, the resulting
187:file:`spam.pyc` file will be recognized as invalid and thus ignored later.  The
188contents of the :file:`spam.pyc` file are platform independent, so a Python
189module directory can be shared by machines of different architectures.
190
191Some tips for experts:
192
193* When the Python interpreter is invoked with the :option:`-O` flag, optimized
194  code is generated and stored in :file:`.pyo` files.  The optimizer currently
195  doesn't help much; it only removes :keyword:`assert` statements.  When
196  :option:`-O` is used, *all* :term:`bytecode` is optimized; ``.pyc`` files are
197  ignored and ``.py`` files are compiled to optimized bytecode.
198
199* Passing two :option:`-O` flags to the Python interpreter (:option:`-OO`) will
200  cause the bytecode compiler to perform optimizations that could in some rare
201  cases result in malfunctioning programs.  Currently only ``__doc__`` strings are
202  removed from the bytecode, resulting in more compact :file:`.pyo` files.  Since
203  some programs may rely on having these available, you should only use this
204  option if you know what you're doing.
205
206* A program doesn't run any faster when it is read from a :file:`.pyc` or
207  :file:`.pyo` file than when it is read from a :file:`.py` file; the only thing
208  that's faster about :file:`.pyc` or :file:`.pyo` files is the speed with which
209  they are loaded.
210
211* When a script is run by giving its name on the command line, the bytecode for
212  the script is never written to a :file:`.pyc` or :file:`.pyo` file.  Thus, the
213  startup time of a script may be reduced by moving most of its code to a module
214  and having a small bootstrap script that imports that module.  It is also
215  possible to name a :file:`.pyc` or :file:`.pyo` file directly on the command
216  line.
217
218* It is possible to have a file called :file:`spam.pyc` (or :file:`spam.pyo`
219  when :option:`-O` is used) without a file :file:`spam.py` for the same module.
220  This can be used to distribute a library of Python code in a form that is
221  moderately hard to reverse engineer.
222
223  .. index:: module: compileall
224
225* The module :mod:`compileall` can create :file:`.pyc` files (or :file:`.pyo`
226  files when :option:`-O` is used) for all modules in a directory.
227
228
229.. _tut-standardmodules:
230
231Standard Modules
232================
233
234.. index:: module: sys
235
236Python comes with a library of standard modules, described in a separate
237document, the Python Library Reference ("Library Reference" hereafter).  Some
238modules are built into the interpreter; these provide access to operations that
239are not part of the core of the language but are nevertheless built in, either
240for efficiency or to provide access to operating system primitives such as
241system calls.  The set of such modules is a configuration option which also
242depends on the underlying platform For example, the :mod:`winreg` module is only
243provided on Windows systems. One particular module deserves some attention:
244:mod:`sys`, which is built into every Python interpreter.  The variables
245``sys.ps1`` and ``sys.ps2`` define the strings used as primary and secondary
246prompts::
247
248   >>> import sys
249   >>> sys.ps1
250   '>>> '
251   >>> sys.ps2
252   '... '
253   >>> sys.ps1 = 'C> '
254   C> print 'Yuck!'
255   Yuck!
256   C>
257
258
259These two variables are only defined if the interpreter is in interactive mode.
260
261The variable ``sys.path`` is a list of strings that determines the interpreter's
262search path for modules. It is initialized to a default path taken from the
263environment variable :envvar:`PYTHONPATH`, or from a built-in default if
264:envvar:`PYTHONPATH` is not set.  You can modify it using standard list
265operations::
266
267   >>> import sys
268   >>> sys.path.append('/ufs/guido/lib/python')
269
270
271.. _tut-dir:
272
273The :func:`dir` Function
274========================
275
276The built-in function :func:`dir` is used to find out which names a module
277defines.  It returns a sorted list of strings::
278
279   >>> import fibo, sys
280   >>> dir(fibo)
281   ['__name__', 'fib', 'fib2']
282   >>> dir(sys)
283   ['__displayhook__', '__doc__', '__excepthook__', '__name__', '__stderr__',
284    '__stdin__', '__stdout__', '_getframe', 'api_version', 'argv',
285    'builtin_module_names', 'byteorder', 'callstats', 'copyright',
286    'displayhook', 'exc_clear', 'exc_info', 'exc_type', 'excepthook',
287    'exec_prefix', 'executable', 'exit', 'getdefaultencoding', 'getdlopenflags',
288    'getrecursionlimit', 'getrefcount', 'hexversion', 'maxint', 'maxunicode',
289    'meta_path', 'modules', 'path', 'path_hooks', 'path_importer_cache',
290    'platform', 'prefix', 'ps1', 'ps2', 'setcheckinterval', 'setdlopenflags',
291    'setprofile', 'setrecursionlimit', 'settrace', 'stderr', 'stdin', 'stdout',
292    'version', 'version_info', 'warnoptions']
293
294Without arguments, :func:`dir` lists the names you have defined currently::
295
296   >>> a = [1, 2, 3, 4, 5]
297   >>> import fibo
298   >>> fib = fibo.fib
299   >>> dir()
300   ['__builtins__', '__doc__', '__file__', '__name__', 'a', 'fib', 'fibo', 'sys']
301
302Note that it lists all types of names: variables, modules, functions, etc.
303
304.. index:: module: __builtin__
305
306:func:`dir` does not list the names of built-in functions and variables.  If you
307want a list of those, they are defined in the standard module
308:mod:`__builtin__`::
309
310   >>> import __builtin__
311   >>> dir(__builtin__)
312   ['ArithmeticError', 'AssertionError', 'AttributeError', 'DeprecationWarning',
313    'EOFError', 'Ellipsis', 'EnvironmentError', 'Exception', 'False',
314    'FloatingPointError', 'FutureWarning', 'IOError', 'ImportError',
315    'IndentationError', 'IndexError', 'KeyError', 'KeyboardInterrupt',
316    'LookupError', 'MemoryError', 'NameError', 'None', 'NotImplemented',
317    'NotImplementedError', 'OSError', 'OverflowError',
318    'PendingDeprecationWarning', 'ReferenceError', 'RuntimeError',
319    'RuntimeWarning', 'StandardError', 'StopIteration', 'SyntaxError',
320    'SyntaxWarning', 'SystemError', 'SystemExit', 'TabError', 'True',
321    'TypeError', 'UnboundLocalError', 'UnicodeDecodeError',
322    'UnicodeEncodeError', 'UnicodeError', 'UnicodeTranslateError',
323    'UserWarning', 'ValueError', 'Warning', 'WindowsError',
324    'ZeroDivisionError', '_', '__debug__', '__doc__', '__import__',
325    '__name__', 'abs', 'apply', 'basestring', 'bool', 'buffer',
326    'callable', 'chr', 'classmethod', 'cmp', 'coerce', 'compile',
327    'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'divmod',
328    'enumerate', 'eval', 'execfile', 'exit', 'file', 'filter', 'float',
329    'frozenset', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex',
330    'id', 'input', 'int', 'intern', 'isinstance', 'issubclass', 'iter',
331    'len', 'license', 'list', 'locals', 'long', 'map', 'max', 'min',
332    'object', 'oct', 'open', 'ord', 'pow', 'property', 'quit', 'range',
333    'raw_input', 'reduce', 'reload', 'repr', 'reversed', 'round', 'set',
334    'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super',
335    'tuple', 'type', 'unichr', 'unicode', 'vars', 'xrange', 'zip']
336
337
338.. _tut-packages:
339
340Packages
341========
342
343Packages are a way of structuring Python's module namespace by using "dotted
344module names".  For example, the module name :mod:`A.B` designates a submodule
345named ``B`` in a package named ``A``.  Just like the use of modules saves the
346authors of different modules from having to worry about each other's global
347variable names, the use of dotted module names saves the authors of multi-module
348packages like NumPy or the Python Imaging Library from having to worry about
349each other's module names.
350
351Suppose you want to design a collection of modules (a "package") for the uniform
352handling of sound files and sound data.  There are many different sound file
353formats (usually recognized by their extension, for example: :file:`.wav`,
354:file:`.aiff`, :file:`.au`), so you may need to create and maintain a growing
355collection of modules for the conversion between the various file formats.
356There are also many different operations you might want to perform on sound data
357(such as mixing, adding echo, applying an equalizer function, creating an
358artificial stereo effect), so in addition you will be writing a never-ending
359stream of modules to perform these operations.  Here's a possible structure for
360your package (expressed in terms of a hierarchical filesystem)::
361
362   sound/                          Top-level package
363         __init__.py               Initialize the sound package
364         formats/                  Subpackage for file format conversions
365                 __init__.py
366                 wavread.py
367                 wavwrite.py
368                 aiffread.py
369                 aiffwrite.py
370                 auread.py
371                 auwrite.py
372                 ...
373         effects/                  Subpackage for sound effects
374                 __init__.py
375                 echo.py
376                 surround.py
377                 reverse.py
378                 ...
379         filters/                  Subpackage for filters
380                 __init__.py
381                 equalizer.py
382                 vocoder.py
383                 karaoke.py
384                 ...
385
386When importing the package, Python searches through the directories on
387``sys.path`` looking for the package subdirectory.
388
389The :file:`__init__.py` files are required to make Python treat the directories
390as containing packages; this is done to prevent directories with a common name,
391such as ``string``, from unintentionally hiding valid modules that occur later
392on the module search path. In the simplest case, :file:`__init__.py` can just be
393an empty file, but it can also execute initialization code for the package or
394set the ``__all__`` variable, described later.
395
396Users of the package can import individual modules from the package, for
397example::
398
399   import sound.effects.echo
400
401This loads the submodule :mod:`sound.effects.echo`.  It must be referenced with
402its full name. ::
403
404   sound.effects.echo.echofilter(input, output, delay=0.7, atten=4)
405
406An alternative way of importing the submodule is::
407
408   from sound.effects import echo
409
410This also loads the submodule :mod:`echo`, and makes it available without its
411package prefix, so it can be used as follows::
412
413   echo.echofilter(input, output, delay=0.7, atten=4)
414
415Yet another variation is to import the desired function or variable directly::
416
417   from sound.effects.echo import echofilter
418
419Again, this loads the submodule :mod:`echo`, but this makes its function
420:func:`echofilter` directly available::
421
422   echofilter(input, output, delay=0.7, atten=4)
423
424Note that when using ``from package import item``, the item can be either a
425submodule (or subpackage) of the package, or some  other name defined in the
426package, like a function, class or variable.  The ``import`` statement first
427tests whether the item is defined in the package; if not, it assumes it is a
428module and attempts to load it.  If it fails to find it, an :exc:`ImportError`
429exception is raised.
430
431Contrarily, when using syntax like ``import item.subitem.subsubitem``, each item
432except for the last must be a package; the last item can be a module or a
433package but can't be a class or function or variable defined in the previous
434item.
435
436
437.. _tut-pkg-import-star:
438
439Importing \* From a Package
440---------------------------
441
442.. index:: single: __all__
443
444Now what happens when the user writes ``from sound.effects import *``?  Ideally,
445one would hope that this somehow goes out to the filesystem, finds which
446submodules are present in the package, and imports them all.  Unfortunately,
447this operation does not work very well on Windows platforms, where the
448filesystem does not always have accurate information about the case of a
449filename!  On these platforms, there is no guaranteed way to know whether a file
450:file:`ECHO.PY` should be imported as a module :mod:`echo`, :mod:`Echo` or
451:mod:`ECHO`.  (For example, Windows 95 has the annoying practice of showing all
452file names with a capitalized first letter.)  The DOS 8+3 filename restriction
453adds another interesting problem for long module names.
454
455The only solution is for the package author to provide an explicit index of the
456package.  The import statement uses the following convention: if a package's
457:file:`__init__.py` code defines a list named ``__all__``, it is taken to be the
458list of module names that should be imported when ``from package import *`` is
459encountered.  It is up to the package author to keep this list up-to-date when a
460new version of the package is released.  Package authors may also decide not to
461support it, if they don't see a use for importing \* from their package.  For
462example, the file :file:`sounds/effects/__init__.py` could contain the following
463code::
464
465   __all__ = ["echo", "surround", "reverse"]
466
467This would mean that ``from sound.effects import *`` would import the three
468named submodules of the :mod:`sound` package.
469
470If ``__all__`` is not defined, the statement ``from sound.effects import *``
471does *not* import all submodules from the package :mod:`sound.effects` into the
472current namespace; it only ensures that the package :mod:`sound.effects` has
473been imported (possibly running any initialization code in :file:`__init__.py`)
474and then imports whatever names are defined in the package.  This includes any
475names defined (and submodules explicitly loaded) by :file:`__init__.py`.  It
476also includes any submodules of the package that were explicitly loaded by
477previous import statements.  Consider this code::
478
479   import sound.effects.echo
480   import sound.effects.surround
481   from sound.effects import *
482
483In this example, the echo and surround modules are imported in the current
484namespace because they are defined in the :mod:`sound.effects` package when the
485``from...import`` statement is executed.  (This also works when ``__all__`` is
486defined.)
487
488Note that in general the practice of importing ``*`` from a module or package is
489frowned upon, since it often causes poorly readable code. However, it is okay to
490use it to save typing in interactive sessions, and certain modules are designed
491to export only names that follow certain patterns.
492
493Remember, there is nothing wrong with using ``from Package import
494specific_submodule``!  In fact, this is the recommended notation unless the
495importing module needs to use submodules with the same name from different
496packages.
497
498
499Intra-package References
500------------------------
501
502The submodules often need to refer to each other.  For example, the
503:mod:`surround` module might use the :mod:`echo` module.  In fact, such
504references are so common that the :keyword:`import` statement first looks in the
505containing package before looking in the standard module search path. Thus, the
506:mod:`surround` module can simply use ``import echo`` or ``from echo import
507echofilter``.  If the imported module is not found in the current package (the
508package of which the current module is a submodule), the :keyword:`import`
509statement looks for a top-level module with the given name.
510
511When packages are structured into subpackages (as with the :mod:`sound` package
512in the example), you can use absolute imports to refer to submodules of siblings
513packages.  For example, if the module :mod:`sound.filters.vocoder` needs to use
514the :mod:`echo` module in the :mod:`sound.effects` package, it can use ``from
515sound.effects import echo``.
516
517Starting with Python 2.5, in addition to the implicit relative imports described
518above, you can write explicit relative imports with the ``from module import
519name`` form of import statement. These explicit relative imports use leading
520dots to indicate the current and parent packages involved in the relative
521import. From the :mod:`surround` module for example, you might use::
522
523   from . import echo
524   from .. import formats
525   from ..filters import equalizer
526
527Note that both explicit and implicit relative imports are based on the name of
528the current module. Since the name of the main module is always ``"__main__"``,
529modules intended for use as the main module of a Python application should
530always use absolute imports.
531
532
533Packages in Multiple Directories
534--------------------------------
535
536Packages support one more special attribute, :attr:`__path__`.  This is
537initialized to be a list containing the name of the directory holding the
538package's :file:`__init__.py` before the code in that file is executed.  This
539variable can be modified; doing so affects future searches for modules and
540subpackages contained in the package.
541
542While this feature is not often needed, it can be used to extend the set of
543modules found in a package.
544
545
546.. rubric:: Footnotes
547
548.. [#] In fact function definitions are also 'statements' that are 'executed'; the
549   execution enters the function name in the module's global symbol table.
550