/Doc/library/ast.rst

  1.. _ast:
  2
  3Abstract Syntax Trees
  4=====================
  5
  6.. module:: ast
  7   :synopsis: Abstract Syntax Tree classes and manipulation.
  8
  9.. sectionauthor:: Martin v. Lรถwis <martin@v.loewis.de>
 10.. sectionauthor:: Georg Brandl <georg@python.org>
 11
 12.. versionadded:: 2.5
 13   The low-level ``_ast`` module containing only the node classes.
 14
 15.. versionadded:: 2.6
 16   The high-level ``ast`` module containing all helpers.
 17
 18
 19The :mod:`ast` module helps Python applications to process trees of the Python
 20abstract syntax grammar.  The abstract syntax itself might change with each
 21Python release; this module helps to find out programmatically what the current
 22grammar looks like.
 23
 24An abstract syntax tree can be generated by passing :data:`ast.PyCF_ONLY_AST` as
 25a flag to the :func:`compile` builtin function, or using the :func:`parse`
 26helper provided in this module.  The result will be a tree of objects whose
 27classes all inherit from :class:`ast.AST`.  An abstract syntax tree can be
 28compiled into a Python code object using the built-in :func:`compile` function.
 29
 30
 31Node classes
 32------------
 33
 34.. class:: AST
 35
 36   This is the base of all AST node classes.  The actual node classes are
 37   derived from the :file:`Parser/Python.asdl` file, which is reproduced
 38   :ref:`below <abstract-grammar>`.  They are defined in the :mod:`_ast` C
 39   module and re-exported in :mod:`ast`.
 40
 41   There is one class defined for each left-hand side symbol in the abstract
 42   grammar (for example, :class:`ast.stmt` or :class:`ast.expr`).  In addition,
 43   there is one class defined for each constructor on the right-hand side; these
 44   classes inherit from the classes for the left-hand side trees.  For example,
 45   :class:`ast.BinOp` inherits from :class:`ast.expr`.  For production rules
 46   with alternatives (aka "sums"), the left-hand side class is abstract: only
 47   instances of specific constructor nodes are ever created.
 48
 49   .. attribute:: _fields
 50
 51      Each concrete class has an attribute :attr:`_fields` which gives the names
 52      of all child nodes.
 53
 54      Each instance of a concrete class has one attribute for each child node,
 55      of the type as defined in the grammar.  For example, :class:`ast.BinOp`
 56      instances have an attribute :attr:`left` of type :class:`ast.expr`.
 57
 58      If these attributes are marked as optional in the grammar (using a
 59      question mark), the value might be ``None``.  If the attributes can have
 60      zero-or-more values (marked with an asterisk), the values are represented
 61      as Python lists.  All possible attributes must be present and have valid
 62      values when compiling an AST with :func:`compile`.
 63
 64   .. attribute:: lineno
 65                  col_offset
 66
 67      Instances of :class:`ast.expr` and :class:`ast.stmt` subclasses have
 68      :attr:`lineno` and :attr:`col_offset` attributes.  The :attr:`lineno` is
 69      the line number of source text (1-indexed so the first line is line 1) and
 70      the :attr:`col_offset` is the UTF-8 byte offset of the first token that
 71      generated the node.  The UTF-8 offset is recorded because the parser uses
 72      UTF-8 internally.
 73
 74   The constructor of a class :class:`ast.T` parses its arguments as follows:
 75
 76   * If there are positional arguments, there must be as many as there are items
 77     in :attr:`T._fields`; they will be assigned as attributes of these names.
 78   * If there are keyword arguments, they will set the attributes of the same
 79     names to the given values.
 80
 81   For example, to create and populate an :class:`ast.UnaryOp` node, you could
 82   use ::
 83
 84      node = ast.UnaryOp()
 85      node.op = ast.USub()
 86      node.operand = ast.Num()
 87      node.operand.n = 5
 88      node.operand.lineno = 0
 89      node.operand.col_offset = 0
 90      node.lineno = 0
 91      node.col_offset = 0
 92
 93   or the more compact ::
 94
 95      node = ast.UnaryOp(ast.USub(), ast.Num(5, lineno=0, col_offset=0),
 96                         lineno=0, col_offset=0)
 97
 98   .. versionadded:: 2.6
 99      The constructor as explained above was added.  In Python 2.5 nodes had
100      to be created by calling the class constructor without arguments and
101      setting the attributes afterwards.
102
103
104.. _abstract-grammar:
105
106Abstract Grammar
107----------------
108
109The module defines a string constant ``__version__`` which is the decimal
110Subversion revision number of the file shown below.
111
112The abstract grammar is currently defined as follows:
113
114.. literalinclude:: ../../Parser/Python.asdl
115
116
117:mod:`ast` Helpers
118------------------
119
120.. versionadded:: 2.6
121
122Apart from the node classes, :mod:`ast` module defines these utility functions
123and classes for traversing abstract syntax trees:
124
125.. function:: parse(expr, filename='<unknown>', mode='exec')
126
127   Parse an expression into an AST node.  Equivalent to ``compile(expr,
128   filename, mode, ast.PyCF_ONLY_AST)``.
129
130
131.. function:: literal_eval(node_or_string)
132
133   Safely evaluate an expression node or a string containing a Python
134   expression.  The string or node provided may only consist of the following
135   Python literal structures: strings, numbers, tuples, lists, dicts, booleans,
136   and ``None``.
137
138   This can be used for safely evaluating strings containing Python expressions
139   from untrusted sources without the need to parse the values oneself.
140
141
142.. function:: get_docstring(node, clean=True)
143
144   Return the docstring of the given *node* (which must be a
145   :class:`FunctionDef`, :class:`ClassDef` or :class:`Module` node), or ``None``
146   if it has no docstring.  If *clean* is true, clean up the docstring's
147   indentation with :func:`inspect.cleandoc`.
148
149
150.. function:: fix_missing_locations(node)
151
152   When you compile a node tree with :func:`compile`, the compiler expects
153   :attr:`lineno` and :attr:`col_offset` attributes for every node that supports
154   them.  This is rather tedious to fill in for generated nodes, so this helper
155   adds these attributes recursively where not already set, by setting them to
156   the values of the parent node.  It works recursively starting at *node*.
157
158
159.. function:: increment_lineno(node, n=1)
160
161   Increment the line number of each node in the tree starting at *node* by *n*.
162   This is useful to "move code" to a different location in a file.
163
164
165.. function:: copy_location(new_node, old_node)
166
167   Copy source location (:attr:`lineno` and :attr:`col_offset`) from *old_node*
168   to *new_node* if possible, and return *new_node*.
169
170
171.. function:: iter_fields(node)
172
173   Yield a tuple of ``(fieldname, value)`` for each field in ``node._fields``
174   that is present on *node*.
175
176
177.. function:: iter_child_nodes(node)
178
179   Yield all direct child nodes of *node*, that is, all fields that are nodes
180   and all items of fields that are lists of nodes.
181
182
183.. function:: walk(node)
184
185   Recursively yield all child nodes of *node*, in no specified order.  This is
186   useful if you only want to modify nodes in place and don't care about the
187   context.
188
189
190.. class:: NodeVisitor()
191
192   A node visitor base class that walks the abstract syntax tree and calls a
193   visitor function for every node found.  This function may return a value
194   which is forwarded by the :meth:`visit` method.
195
196   This class is meant to be subclassed, with the subclass adding visitor
197   methods.
198
199   .. method:: visit(node)
200
201      Visit a node.  The default implementation calls the method called
202      :samp:`self.visit_{classname}` where *classname* is the name of the node
203      class, or :meth:`generic_visit` if that method doesn't exist.
204
205   .. method:: generic_visit(node)
206
207      This visitor calls :meth:`visit` on all children of the node.
208
209      Note that child nodes of nodes that have a custom visitor method won't be
210      visited unless the visitor calls :meth:`generic_visit` or visits them
211      itself.
212
213   Don't use the :class:`NodeVisitor` if you want to apply changes to nodes
214   during traversal.  For this a special visitor exists
215   (:class:`NodeTransformer`) that allows modifications.
216
217
218.. class:: NodeTransformer()
219
220   A :class:`NodeVisitor` subclass that walks the abstract syntax tree and
221   allows modification of nodes.
222
223   The :class:`NodeTransformer` will walk the AST and use the return value of
224   the visitor methods to replace or remove the old node.  If the return value
225   of the visitor method is ``None``, the node will be removed from its
226   location, otherwise it is replaced with the return value.  The return value
227   may be the original node in which case no replacement takes place.
228
229   Here is an example transformer that rewrites all occurrences of name lookups
230   (``foo``) to ``data['foo']``::
231
232      class RewriteName(NodeTransformer):
233
234          def visit_Name(self, node):
235              return copy_location(Subscript(
236                  value=Name(id='data', ctx=Load()),
237                  slice=Index(value=Str(s=node.id)),
238                  ctx=node.ctx
239              ), node)
240
241   Keep in mind that if the node you're operating on has child nodes you must
242   either transform the child nodes yourself or call the :meth:`generic_visit`
243   method for the node first.
244
245   For nodes that were part of a collection of statements (that applies to all
246   statement nodes), the visitor may also return a list of nodes rather than
247   just a single node.
248
249   Usually you use the transformer like this::
250
251      node = YourTransformer().visit(node)
252
253
254.. function:: dump(node, annotate_fields=True, include_attributes=False)
255
256   Return a formatted dump of the tree in *node*.  This is mainly useful for
257   debugging purposes.  The returned string will show the names and the values
258   for fields.  This makes the code impossible to evaluate, so if evaluation is
259   wanted *annotate_fields* must be set to False.  Attributes such as line
260   numbers and column offsets are not dumped by default.  If this is wanted,
261   *include_attributes* can be set to ``True``.