/Demo/metaclasses/index.html

http://unladen-swallow.googlecode.com/ · HTML · 605 lines · 452 code · 153 blank · 0 comment · 0 complexity · 8c1e9a3d66a373c7241b6b02d8761ff0 MD5 · raw file

  1. <HTML>
  2. <HEAD>
  3. <TITLE>Metaclasses in Python 1.5</TITLE>
  4. </HEAD>
  5. <BODY BGCOLOR="FFFFFF">
  6. <H1>Metaclasses in Python 1.5</H1>
  7. <H2>(A.k.a. The Killer Joke :-)</H2>
  8. <HR>
  9. (<i>Postscript:</i> reading this essay is probably not the best way to
  10. understand the metaclass hook described here. See a <A
  11. HREF="meta-vladimir.txt">message posted by Vladimir Marangozov</A>
  12. which may give a gentler introduction to the matter. You may also
  13. want to search Deja News for messages with "metaclass" in the subject
  14. posted to comp.lang.python in July and August 1998.)
  15. <HR>
  16. <P>In previous Python releases (and still in 1.5), there is something
  17. called the ``Don Beaudry hook'', after its inventor and champion.
  18. This allows C extensions to provide alternate class behavior, thereby
  19. allowing the Python class syntax to be used to define other class-like
  20. entities. Don Beaudry has used this in his infamous <A
  21. HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> package; Jim
  22. Fulton has used it in his <A
  23. HREF="http://www.digicool.com/releases/ExtensionClass/">Extension
  24. Classes</A> package. (It has also been referred to as the ``Don
  25. Beaudry <i>hack</i>,'' but that's a misnomer. There's nothing hackish
  26. about it -- in fact, it is rather elegant and deep, even though
  27. there's something dark to it.)
  28. <P>(On first reading, you may want to skip directly to the examples in
  29. the section "Writing Metaclasses in Python" below, unless you want
  30. your head to explode.)
  31. <P>
  32. <HR>
  33. <P>Documentation of the Don Beaudry hook has purposefully been kept
  34. minimal, since it is a feature of incredible power, and is easily
  35. abused. Basically, it checks whether the <b>type of the base
  36. class</b> is callable, and if so, it is called to create the new
  37. class.
  38. <P>Note the two indirection levels. Take a simple example:
  39. <PRE>
  40. class B:
  41. pass
  42. class C(B):
  43. pass
  44. </PRE>
  45. Take a look at the second class definition, and try to fathom ``the
  46. type of the base class is callable.''
  47. <P>(Types are not classes, by the way. See questions 4.2, 4.19 and in
  48. particular 6.22 in the <A
  49. HREF="http://www.python.org/cgi-bin/faqw.py" >Python FAQ</A>
  50. for more on this topic.)
  51. <P>
  52. <UL>
  53. <LI>The <b>base class</b> is B; this one's easy.<P>
  54. <LI>Since B is a class, its type is ``class''; so the <b>type of the
  55. base class</b> is the type ``class''. This is also known as
  56. types.ClassType, assuming the standard module <code>types</code> has
  57. been imported.<P>
  58. <LI>Now is the type ``class'' <b>callable</b>? No, because types (in
  59. core Python) are never callable. Classes are callable (calling a
  60. class creates a new instance) but types aren't.<P>
  61. </UL>
  62. <P>So our conclusion is that in our example, the type of the base
  63. class (of C) is not callable. So the Don Beaudry hook does not apply,
  64. and the default class creation mechanism is used (which is also used
  65. when there is no base class). In fact, the Don Beaudry hook never
  66. applies when using only core Python, since the type of a core object
  67. is never callable.
  68. <P>So what do Don and Jim do in order to use Don's hook? Write an
  69. extension that defines at least two new Python object types. The
  70. first would be the type for ``class-like'' objects usable as a base
  71. class, to trigger Don's hook. This type must be made callable.
  72. That's why we need a second type. Whether an object is callable
  73. depends on its type. So whether a type object is callable depends on
  74. <i>its</i> type, which is a <i>meta-type</i>. (In core Python there
  75. is only one meta-type, the type ``type'' (types.TypeType), which is
  76. the type of all type objects, even itself.) A new meta-type must
  77. be defined that makes the type of the class-like objects callable.
  78. (Normally, a third type would also be needed, the new ``instance''
  79. type, but this is not an absolute requirement -- the new class type
  80. could return an object of some existing type when invoked to create an
  81. instance.)
  82. <P>Still confused? Here's a simple device due to Don himself to
  83. explain metaclasses. Take a simple class definition; assume B is a
  84. special class that triggers Don's hook:
  85. <PRE>
  86. class C(B):
  87. a = 1
  88. b = 2
  89. </PRE>
  90. This can be though of as equivalent to:
  91. <PRE>
  92. C = type(B)('C', (B,), {'a': 1, 'b': 2})
  93. </PRE>
  94. If that's too dense for you, here's the same thing written out using
  95. temporary variables:
  96. <PRE>
  97. creator = type(B) # The type of the base class
  98. name = 'C' # The name of the new class
  99. bases = (B,) # A tuple containing the base class(es)
  100. namespace = {'a': 1, 'b': 2} # The namespace of the class statement
  101. C = creator(name, bases, namespace)
  102. </PRE>
  103. This is analogous to what happens without the Don Beaudry hook, except
  104. that in that case the creator function is set to the default class
  105. creator.
  106. <P>In either case, the creator is called with three arguments. The
  107. first one, <i>name</i>, is the name of the new class (as given at the
  108. top of the class statement). The <i>bases</i> argument is a tuple of
  109. base classes (a singleton tuple if there's only one base class, like
  110. the example). Finally, <i>namespace</i> is a dictionary containing
  111. the local variables collected during execution of the class statement.
  112. <P>Note that the contents of the namespace dictionary is simply
  113. whatever names were defined in the class statement. A little-known
  114. fact is that when Python executes a class statement, it enters a new
  115. local namespace, and all assignments and function definitions take
  116. place in this namespace. Thus, after executing the following class
  117. statement:
  118. <PRE>
  119. class C:
  120. a = 1
  121. def f(s): pass
  122. </PRE>
  123. the class namespace's contents would be {'a': 1, 'f': &lt;function f
  124. ...&gt;}.
  125. <P>But enough already about writing Python metaclasses in C; read the
  126. documentation of <A
  127. HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> or <A
  128. HREF="http://www.digicool.com/papers/ExtensionClass.html" >Extension
  129. Classes</A> for more information.
  130. <P>
  131. <HR>
  132. <H2>Writing Metaclasses in Python</H2>
  133. <P>In Python 1.5, the requirement to write a C extension in order to
  134. write metaclasses has been dropped (though you can still do
  135. it, of course). In addition to the check ``is the type of the base
  136. class callable,'' there's a check ``does the base class have a
  137. __class__ attribute.'' If so, it is assumed that the __class__
  138. attribute refers to a class.
  139. <P>Let's repeat our simple example from above:
  140. <PRE>
  141. class C(B):
  142. a = 1
  143. b = 2
  144. </PRE>
  145. Assuming B has a __class__ attribute, this translates into:
  146. <PRE>
  147. C = B.__class__('C', (B,), {'a': 1, 'b': 2})
  148. </PRE>
  149. This is exactly the same as before except that instead of type(B),
  150. B.__class__ is invoked. If you have read <A HREF=
  151. "http://www.python.org/cgi-bin/faqw.py?req=show&file=faq06.022.htp"
  152. >FAQ question 6.22</A> you will understand that while there is a big
  153. technical difference between type(B) and B.__class__, they play the
  154. same role at different abstraction levels. And perhaps at some point
  155. in the future they will really be the same thing (at which point you
  156. would be able to derive subclasses from built-in types).
  157. <P>At this point it may be worth mentioning that C.__class__ is the
  158. same object as B.__class__, i.e., C's metaclass is the same as B's
  159. metaclass. In other words, subclassing an existing class creates a
  160. new (meta)inststance of the base class's metaclass.
  161. <P>Going back to the example, the class B.__class__ is instantiated,
  162. passing its constructor the same three arguments that are passed to
  163. the default class constructor or to an extension's metaclass:
  164. <i>name</i>, <i>bases</i>, and <i>namespace</i>.
  165. <P>It is easy to be confused by what exactly happens when using a
  166. metaclass, because we lose the absolute distinction between classes
  167. and instances: a class is an instance of a metaclass (a
  168. ``metainstance''), but technically (i.e. in the eyes of the python
  169. runtime system), the metaclass is just a class, and the metainstance
  170. is just an instance. At the end of the class statement, the metaclass
  171. whose metainstance is used as a base class is instantiated, yielding a
  172. second metainstance (of the same metaclass). This metainstance is
  173. then used as a (normal, non-meta) class; instantiation of the class
  174. means calling the metainstance, and this will return a real instance.
  175. And what class is that an instance of? Conceptually, it is of course
  176. an instance of our metainstance; but in most cases the Python runtime
  177. system will see it as an instance of a a helper class used by the
  178. metaclass to implement its (non-meta) instances...
  179. <P>Hopefully an example will make things clearer. Let's presume we
  180. have a metaclass MetaClass1. It's helper class (for non-meta
  181. instances) is callled HelperClass1. We now (manually) instantiate
  182. MetaClass1 once to get an empty special base class:
  183. <PRE>
  184. BaseClass1 = MetaClass1("BaseClass1", (), {})
  185. </PRE>
  186. We can now use BaseClass1 as a base class in a class statement:
  187. <PRE>
  188. class MySpecialClass(BaseClass1):
  189. i = 1
  190. def f(s): pass
  191. </PRE>
  192. At this point, MySpecialClass is defined; it is a metainstance of
  193. MetaClass1 just like BaseClass1, and in fact the expression
  194. ``BaseClass1.__class__ == MySpecialClass.__class__ == MetaClass1''
  195. yields true.
  196. <P>We are now ready to create instances of MySpecialClass. Let's
  197. assume that no constructor arguments are required:
  198. <PRE>
  199. x = MySpecialClass()
  200. y = MySpecialClass()
  201. print x.__class__, y.__class__
  202. </PRE>
  203. The print statement shows that x and y are instances of HelperClass1.
  204. How did this happen? MySpecialClass is an instance of MetaClass1
  205. (``meta'' is irrelevant here); when an instance is called, its
  206. __call__ method is invoked, and presumably the __call__ method defined
  207. by MetaClass1 returns an instance of HelperClass1.
  208. <P>Now let's see how we could use metaclasses -- what can we do
  209. with metaclasses that we can't easily do without them? Here's one
  210. idea: a metaclass could automatically insert trace calls for all
  211. method calls. Let's first develop a simplified example, without
  212. support for inheritance or other ``advanced'' Python features (we'll
  213. add those later).
  214. <PRE>
  215. import types
  216. class Tracing:
  217. def __init__(self, name, bases, namespace):
  218. """Create a new class."""
  219. self.__name__ = name
  220. self.__bases__ = bases
  221. self.__namespace__ = namespace
  222. def __call__(self):
  223. """Create a new instance."""
  224. return Instance(self)
  225. class Instance:
  226. def __init__(self, klass):
  227. self.__klass__ = klass
  228. def __getattr__(self, name):
  229. try:
  230. value = self.__klass__.__namespace__[name]
  231. except KeyError:
  232. raise AttributeError, name
  233. if type(value) is not types.FunctionType:
  234. return value
  235. return BoundMethod(value, self)
  236. class BoundMethod:
  237. def __init__(self, function, instance):
  238. self.function = function
  239. self.instance = instance
  240. def __call__(self, *args):
  241. print "calling", self.function, "for", self.instance, "with", args
  242. return apply(self.function, (self.instance,) + args)
  243. Trace = Tracing('Trace', (), {})
  244. class MyTracedClass(Trace):
  245. def method1(self, a):
  246. self.a = a
  247. def method2(self):
  248. return self.a
  249. aninstance = MyTracedClass()
  250. aninstance.method1(10)
  251. print "the answer is %d" % aninstance.method2()
  252. </PRE>
  253. Confused already? The intention is to read this from top down. The
  254. Tracing class is the metaclass we're defining. Its structure is
  255. really simple.
  256. <P>
  257. <UL>
  258. <LI>The __init__ method is invoked when a new Tracing instance is
  259. created, e.g. the definition of class MyTracedClass later in the
  260. example. It simply saves the class name, base classes and namespace
  261. as instance variables.<P>
  262. <LI>The __call__ method is invoked when a Tracing instance is called,
  263. e.g. the creation of aninstance later in the example. It returns an
  264. instance of the class Instance, which is defined next.<P>
  265. </UL>
  266. <P>The class Instance is the class used for all instances of classes
  267. built using the Tracing metaclass, e.g. aninstance. It has two
  268. methods:
  269. <P>
  270. <UL>
  271. <LI>The __init__ method is invoked from the Tracing.__call__ method
  272. above to initialize a new instance. It saves the class reference as
  273. an instance variable. It uses a funny name because the user's
  274. instance variables (e.g. self.a later in the example) live in the same
  275. namespace.<P>
  276. <LI>The __getattr__ method is invoked whenever the user code
  277. references an attribute of the instance that is not an instance
  278. variable (nor a class variable; but except for __init__ and
  279. __getattr__ there are no class variables). It will be called, for
  280. example, when aninstance.method1 is referenced in the example, with
  281. self set to aninstance and name set to the string "method1".<P>
  282. </UL>
  283. <P>The __getattr__ method looks the name up in the __namespace__
  284. dictionary. If it isn't found, it raises an AttributeError exception.
  285. (In a more realistic example, it would first have to look through the
  286. base classes as well.) If it is found, there are two possibilities:
  287. it's either a function or it isn't. If it's not a function, it is
  288. assumed to be a class variable, and its value is returned. If it's a
  289. function, we have to ``wrap'' it in instance of yet another helper
  290. class, BoundMethod.
  291. <P>The BoundMethod class is needed to implement a familiar feature:
  292. when a method is defined, it has an initial argument, self, which is
  293. automatically bound to the relevant instance when it is called. For
  294. example, aninstance.method1(10) is equivalent to method1(aninstance,
  295. 10). In the example if this call, first a temporary BoundMethod
  296. instance is created with the following constructor call: temp =
  297. BoundMethod(method1, aninstance); then this instance is called as
  298. temp(10). After the call, the temporary instance is discarded.
  299. <P>
  300. <UL>
  301. <LI>The __init__ method is invoked for the constructor call
  302. BoundMethod(method1, aninstance). It simply saves away its
  303. arguments.<P>
  304. <LI>The __call__ method is invoked when the bound method instance is
  305. called, as in temp(10). It needs to call method1(aninstance, 10).
  306. However, even though self.function is now method1 and self.instance is
  307. aninstance, it can't call self.function(self.instance, args) directly,
  308. because it should work regardless of the number of arguments passed.
  309. (For simplicity, support for keyword arguments has been omitted.)<P>
  310. </UL>
  311. <P>In order to be able to support arbitrary argument lists, the
  312. __call__ method first constructs a new argument tuple. Conveniently,
  313. because of the notation *args in __call__'s own argument list, the
  314. arguments to __call__ (except for self) are placed in the tuple args.
  315. To construct the desired argument list, we concatenate a singleton
  316. tuple containing the instance with the args tuple: (self.instance,) +
  317. args. (Note the trailing comma used to construct the singleton
  318. tuple.) In our example, the resulting argument tuple is (aninstance,
  319. 10).
  320. <P>The intrinsic function apply() takes a function and an argument
  321. tuple and calls the function for it. In our example, we are calling
  322. apply(method1, (aninstance, 10)) which is equivalent to calling
  323. method(aninstance, 10).
  324. <P>From here on, things should come together quite easily. The output
  325. of the example code is something like this:
  326. <PRE>
  327. calling &lt;function method1 at ae8d8&gt; for &lt;Instance instance at 95ab0&gt; with (10,)
  328. calling &lt;function method2 at ae900&gt; for &lt;Instance instance at 95ab0&gt; with ()
  329. the answer is 10
  330. </PRE>
  331. <P>That was about the shortest meaningful example that I could come up
  332. with. A real tracing metaclass (for example, <A
  333. HREF="#Trace">Trace.py</A> discussed below) needs to be more
  334. complicated in two dimensions.
  335. <P>First, it needs to support more advanced Python features such as
  336. class variables, inheritance, __init__ methods, and keyword arguments.
  337. <P>Second, it needs to provide a more flexible way to handle the
  338. actual tracing information; perhaps it should be possible to write
  339. your own tracing function that gets called, perhaps it should be
  340. possible to enable and disable tracing on a per-class or per-instance
  341. basis, and perhaps a filter so that only interesting calls are traced;
  342. it should also be able to trace the return value of the call (or the
  343. exception it raised if an error occurs). Even the Trace.py example
  344. doesn't support all these features yet.
  345. <P>
  346. <HR>
  347. <H1>Real-life Examples</H1>
  348. <P>Have a look at some very preliminary examples that I coded up to
  349. teach myself how to write metaclasses:
  350. <DL>
  351. <DT><A HREF="Enum.py">Enum.py</A>
  352. <DD>This (ab)uses the class syntax as an elegant way to define
  353. enumerated types. The resulting classes are never instantiated --
  354. rather, their class attributes are the enumerated values. For
  355. example:
  356. <PRE>
  357. class Color(Enum):
  358. red = 1
  359. green = 2
  360. blue = 3
  361. print Color.red
  362. </PRE>
  363. will print the string ``Color.red'', while ``Color.red==1'' is true,
  364. and ``Color.red + 1'' raise a TypeError exception.
  365. <P>
  366. <DT><A NAME=Trace></A><A HREF="Trace.py">Trace.py</A>
  367. <DD>The resulting classes work much like standard
  368. classes, but by setting a special class or instance attribute
  369. __trace_output__ to point to a file, all calls to the class's methods
  370. are traced. It was a bit of a struggle to get this right. This
  371. should probably redone using the generic metaclass below.
  372. <P>
  373. <DT><A HREF="Meta.py">Meta.py</A>
  374. <DD>A generic metaclass. This is an attempt at finding out how much
  375. standard class behavior can be mimicked by a metaclass. The
  376. preliminary answer appears to be that everything's fine as long as the
  377. class (or its clients) don't look at the instance's __class__
  378. attribute, nor at the class's __dict__ attribute. The use of
  379. __getattr__ internally makes the classic implementation of __getattr__
  380. hooks tough; we provide a similar hook _getattr_ instead.
  381. (__setattr__ and __delattr__ are not affected.)
  382. (XXX Hm. Could detect presence of __getattr__ and rename it.)
  383. <P>
  384. <DT><A HREF="Eiffel.py">Eiffel.py</A>
  385. <DD>Uses the above generic metaclass to implement Eiffel style
  386. pre-conditions and post-conditions.
  387. <P>
  388. <DT><A HREF="Synch.py">Synch.py</A>
  389. <DD>Uses the above generic metaclass to implement synchronized
  390. methods.
  391. <P>
  392. <DT><A HREF="Simple.py">Simple.py</A>
  393. <DD>The example module used above.
  394. <P>
  395. </DL>
  396. <P>A pattern seems to be emerging: almost all these uses of
  397. metaclasses (except for Enum, which is probably more cute than useful)
  398. mostly work by placing wrappers around method calls. An obvious
  399. problem with that is that it's not easy to combine the features of
  400. different metaclasses, while this would actually be quite useful: for
  401. example, I wouldn't mind getting a trace from the test run of the
  402. Synch module, and it would be interesting to add preconditions to it
  403. as well. This needs more research. Perhaps a metaclass could be
  404. provided that allows stackable wrappers...
  405. <P>
  406. <HR>
  407. <H2>Things You Could Do With Metaclasses</H2>
  408. <P>There are lots of things you could do with metaclasses. Most of
  409. these can also be done with creative use of __getattr__, but
  410. metaclasses make it easier to modify the attribute lookup behavior of
  411. classes. Here's a partial list.
  412. <P>
  413. <UL>
  414. <LI>Enforce different inheritance semantics, e.g. automatically call
  415. base class methods when a derived class overrides<P>
  416. <LI>Implement class methods (e.g. if the first argument is not named
  417. 'self')<P>
  418. <LI>Implement that each instance is initialized with <b>copies</b> of
  419. all class variables<P>
  420. <LI>Implement a different way to store instance variables (e.g. in a
  421. list kept outside the instance but indexed by the instance's id())<P>
  422. <LI>Automatically wrap or trap all or certain methods
  423. <UL>
  424. <LI>for tracing
  425. <LI>for precondition and postcondition checking
  426. <LI>for synchronized methods
  427. <LI>for automatic value caching
  428. </UL>
  429. <P>
  430. <LI>When an attribute is a parameterless function, call it on
  431. reference (to mimic it being an instance variable); same on assignment<P>
  432. <LI>Instrumentation: see how many times various attributes are used<P>
  433. <LI>Different semantics for __setattr__ and __getattr__ (e.g. disable
  434. them when they are being used recursively)<P>
  435. <LI>Abuse class syntax for other things<P>
  436. <LI>Experiment with automatic type checking<P>
  437. <LI>Delegation (or acquisition)<P>
  438. <LI>Dynamic inheritance patterns<P>
  439. <LI>Automatic caching of methods<P>
  440. </UL>
  441. <P>
  442. <HR>
  443. <H4>Credits</H4>
  444. <P>Many thanks to David Ascher and Donald Beaudry for their comments
  445. on earlier draft of this paper. Also thanks to Matt Conway and Tommy
  446. Burnette for putting a seed for the idea of metaclasses in my
  447. mind, nearly three years ago, even though at the time my response was
  448. ``you can do that with __getattr__ hooks...'' :-)
  449. <P>
  450. <HR>
  451. </BODY>
  452. </HTML>