/Demo/metaclasses/index.html
HTML | 605 lines | 452 code | 153 blank | 0 comment | 0 complexity | 8c1e9a3d66a373c7241b6b02d8761ff0 MD5 | raw file
1<HTML> 2 3<HEAD> 4<TITLE>Metaclasses in Python 1.5</TITLE> 5</HEAD> 6 7<BODY BGCOLOR="FFFFFF"> 8 9<H1>Metaclasses in Python 1.5</H1> 10<H2>(A.k.a. The Killer Joke :-)</H2> 11 12<HR> 13 14(<i>Postscript:</i> reading this essay is probably not the best way to 15understand the metaclass hook described here. See a <A 16HREF="meta-vladimir.txt">message posted by Vladimir Marangozov</A> 17which may give a gentler introduction to the matter. You may also 18want to search Deja News for messages with "metaclass" in the subject 19posted to comp.lang.python in July and August 1998.) 20 21<HR> 22 23<P>In previous Python releases (and still in 1.5), there is something 24called the ``Don Beaudry hook'', after its inventor and champion. 25This allows C extensions to provide alternate class behavior, thereby 26allowing the Python class syntax to be used to define other class-like 27entities. Don Beaudry has used this in his infamous <A 28HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> package; Jim 29Fulton has used it in his <A 30HREF="http://www.digicool.com/releases/ExtensionClass/">Extension 31Classes</A> package. (It has also been referred to as the ``Don 32Beaudry <i>hack</i>,'' but that's a misnomer. There's nothing hackish 33about it -- in fact, it is rather elegant and deep, even though 34there's something dark to it.) 35 36<P>(On first reading, you may want to skip directly to the examples in 37the section "Writing Metaclasses in Python" below, unless you want 38your head to explode.) 39 40<P> 41 42<HR> 43 44<P>Documentation of the Don Beaudry hook has purposefully been kept 45minimal, since it is a feature of incredible power, and is easily 46abused. Basically, it checks whether the <b>type of the base 47class</b> is callable, and if so, it is called to create the new 48class. 49 50<P>Note the two indirection levels. Take a simple example: 51 52<PRE> 53class B: 54 pass 55 56class C(B): 57 pass 58</PRE> 59 60Take a look at the second class definition, and try to fathom ``the 61type of the base class is callable.'' 62 63<P>(Types are not classes, by the way. See questions 4.2, 4.19 and in 64particular 6.22 in the <A 65HREF="http://www.python.org/cgi-bin/faqw.py" >Python FAQ</A> 66for more on this topic.) 67 68<P> 69 70<UL> 71 72<LI>The <b>base class</b> is B; this one's easy.<P> 73 74<LI>Since B is a class, its type is ``class''; so the <b>type of the 75base class</b> is the type ``class''. This is also known as 76types.ClassType, assuming the standard module <code>types</code> has 77been imported.<P> 78 79<LI>Now is the type ``class'' <b>callable</b>? No, because types (in 80core Python) are never callable. Classes are callable (calling a 81class creates a new instance) but types aren't.<P> 82 83</UL> 84 85<P>So our conclusion is that in our example, the type of the base 86class (of C) is not callable. So the Don Beaudry hook does not apply, 87and the default class creation mechanism is used (which is also used 88when there is no base class). In fact, the Don Beaudry hook never 89applies when using only core Python, since the type of a core object 90is never callable. 91 92<P>So what do Don and Jim do in order to use Don's hook? Write an 93extension that defines at least two new Python object types. The 94first would be the type for ``class-like'' objects usable as a base 95class, to trigger Don's hook. This type must be made callable. 96That's why we need a second type. Whether an object is callable 97depends on its type. So whether a type object is callable depends on 98<i>its</i> type, which is a <i>meta-type</i>. (In core Python there 99is only one meta-type, the type ``type'' (types.TypeType), which is 100the type of all type objects, even itself.) A new meta-type must 101be defined that makes the type of the class-like objects callable. 102(Normally, a third type would also be needed, the new ``instance'' 103type, but this is not an absolute requirement -- the new class type 104could return an object of some existing type when invoked to create an 105instance.) 106 107<P>Still confused? Here's a simple device due to Don himself to 108explain metaclasses. Take a simple class definition; assume B is a 109special class that triggers Don's hook: 110 111<PRE> 112class C(B): 113 a = 1 114 b = 2 115</PRE> 116 117This can be though of as equivalent to: 118 119<PRE> 120C = type(B)('C', (B,), {'a': 1, 'b': 2}) 121</PRE> 122 123If that's too dense for you, here's the same thing written out using 124temporary variables: 125 126<PRE> 127creator = type(B) # The type of the base class 128name = 'C' # The name of the new class 129bases = (B,) # A tuple containing the base class(es) 130namespace = {'a': 1, 'b': 2} # The namespace of the class statement 131C = creator(name, bases, namespace) 132</PRE> 133 134This is analogous to what happens without the Don Beaudry hook, except 135that in that case the creator function is set to the default class 136creator. 137 138<P>In either case, the creator is called with three arguments. The 139first one, <i>name</i>, is the name of the new class (as given at the 140top of the class statement). The <i>bases</i> argument is a tuple of 141base classes (a singleton tuple if there's only one base class, like 142the example). Finally, <i>namespace</i> is a dictionary containing 143the local variables collected during execution of the class statement. 144 145<P>Note that the contents of the namespace dictionary is simply 146whatever names were defined in the class statement. A little-known 147fact is that when Python executes a class statement, it enters a new 148local namespace, and all assignments and function definitions take 149place in this namespace. Thus, after executing the following class 150statement: 151 152<PRE> 153class C: 154 a = 1 155 def f(s): pass 156</PRE> 157 158the class namespace's contents would be {'a': 1, 'f': <function f 159...>}. 160 161<P>But enough already about writing Python metaclasses in C; read the 162documentation of <A 163HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> or <A 164HREF="http://www.digicool.com/papers/ExtensionClass.html" >Extension 165Classes</A> for more information. 166 167<P> 168 169<HR> 170 171<H2>Writing Metaclasses in Python</H2> 172 173<P>In Python 1.5, the requirement to write a C extension in order to 174write metaclasses has been dropped (though you can still do 175it, of course). In addition to the check ``is the type of the base 176class callable,'' there's a check ``does the base class have a 177__class__ attribute.'' If so, it is assumed that the __class__ 178attribute refers to a class. 179 180<P>Let's repeat our simple example from above: 181 182<PRE> 183class C(B): 184 a = 1 185 b = 2 186</PRE> 187 188Assuming B has a __class__ attribute, this translates into: 189 190<PRE> 191C = B.__class__('C', (B,), {'a': 1, 'b': 2}) 192</PRE> 193 194This is exactly the same as before except that instead of type(B), 195B.__class__ is invoked. If you have read <A HREF= 196"http://www.python.org/cgi-bin/faqw.py?req=show&file=faq06.022.htp" 197>FAQ question 6.22</A> you will understand that while there is a big 198technical difference between type(B) and B.__class__, they play the 199same role at different abstraction levels. And perhaps at some point 200in the future they will really be the same thing (at which point you 201would be able to derive subclasses from built-in types). 202 203<P>At this point it may be worth mentioning that C.__class__ is the 204same object as B.__class__, i.e., C's metaclass is the same as B's 205metaclass. In other words, subclassing an existing class creates a 206new (meta)inststance of the base class's metaclass. 207 208<P>Going back to the example, the class B.__class__ is instantiated, 209passing its constructor the same three arguments that are passed to 210the default class constructor or to an extension's metaclass: 211<i>name</i>, <i>bases</i>, and <i>namespace</i>. 212 213<P>It is easy to be confused by what exactly happens when using a 214metaclass, because we lose the absolute distinction between classes 215and instances: a class is an instance of a metaclass (a 216``metainstance''), but technically (i.e. in the eyes of the python 217runtime system), the metaclass is just a class, and the metainstance 218is just an instance. At the end of the class statement, the metaclass 219whose metainstance is used as a base class is instantiated, yielding a 220second metainstance (of the same metaclass). This metainstance is 221then used as a (normal, non-meta) class; instantiation of the class 222means calling the metainstance, and this will return a real instance. 223And what class is that an instance of? Conceptually, it is of course 224an instance of our metainstance; but in most cases the Python runtime 225system will see it as an instance of a a helper class used by the 226metaclass to implement its (non-meta) instances... 227 228<P>Hopefully an example will make things clearer. Let's presume we 229have a metaclass MetaClass1. It's helper class (for non-meta 230instances) is callled HelperClass1. We now (manually) instantiate 231MetaClass1 once to get an empty special base class: 232 233<PRE> 234BaseClass1 = MetaClass1("BaseClass1", (), {}) 235</PRE> 236 237We can now use BaseClass1 as a base class in a class statement: 238 239<PRE> 240class MySpecialClass(BaseClass1): 241 i = 1 242 def f(s): pass 243</PRE> 244 245At this point, MySpecialClass is defined; it is a metainstance of 246MetaClass1 just like BaseClass1, and in fact the expression 247``BaseClass1.__class__ == MySpecialClass.__class__ == MetaClass1'' 248yields true. 249 250<P>We are now ready to create instances of MySpecialClass. Let's 251assume that no constructor arguments are required: 252 253<PRE> 254x = MySpecialClass() 255y = MySpecialClass() 256print x.__class__, y.__class__ 257</PRE> 258 259The print statement shows that x and y are instances of HelperClass1. 260How did this happen? MySpecialClass is an instance of MetaClass1 261(``meta'' is irrelevant here); when an instance is called, its 262__call__ method is invoked, and presumably the __call__ method defined 263by MetaClass1 returns an instance of HelperClass1. 264 265<P>Now let's see how we could use metaclasses -- what can we do 266with metaclasses that we can't easily do without them? Here's one 267idea: a metaclass could automatically insert trace calls for all 268method calls. Let's first develop a simplified example, without 269support for inheritance or other ``advanced'' Python features (we'll 270add those later). 271 272<PRE> 273import types 274 275class Tracing: 276 def __init__(self, name, bases, namespace): 277 """Create a new class.""" 278 self.__name__ = name 279 self.__bases__ = bases 280 self.__namespace__ = namespace 281 def __call__(self): 282 """Create a new instance.""" 283 return Instance(self) 284 285class Instance: 286 def __init__(self, klass): 287 self.__klass__ = klass 288 def __getattr__(self, name): 289 try: 290 value = self.__klass__.__namespace__[name] 291 except KeyError: 292 raise AttributeError, name 293 if type(value) is not types.FunctionType: 294 return value 295 return BoundMethod(value, self) 296 297class BoundMethod: 298 def __init__(self, function, instance): 299 self.function = function 300 self.instance = instance 301 def __call__(self, *args): 302 print "calling", self.function, "for", self.instance, "with", args 303 return apply(self.function, (self.instance,) + args) 304 305Trace = Tracing('Trace', (), {}) 306 307class MyTracedClass(Trace): 308 def method1(self, a): 309 self.a = a 310 def method2(self): 311 return self.a 312 313aninstance = MyTracedClass() 314 315aninstance.method1(10) 316 317print "the answer is %d" % aninstance.method2() 318</PRE> 319 320Confused already? The intention is to read this from top down. The 321Tracing class is the metaclass we're defining. Its structure is 322really simple. 323 324<P> 325 326<UL> 327 328<LI>The __init__ method is invoked when a new Tracing instance is 329created, e.g. the definition of class MyTracedClass later in the 330example. It simply saves the class name, base classes and namespace 331as instance variables.<P> 332 333<LI>The __call__ method is invoked when a Tracing instance is called, 334e.g. the creation of aninstance later in the example. It returns an 335instance of the class Instance, which is defined next.<P> 336 337</UL> 338 339<P>The class Instance is the class used for all instances of classes 340built using the Tracing metaclass, e.g. aninstance. It has two 341methods: 342 343<P> 344 345<UL> 346 347<LI>The __init__ method is invoked from the Tracing.__call__ method 348above to initialize a new instance. It saves the class reference as 349an instance variable. It uses a funny name because the user's 350instance variables (e.g. self.a later in the example) live in the same 351namespace.<P> 352 353<LI>The __getattr__ method is invoked whenever the user code 354references an attribute of the instance that is not an instance 355variable (nor a class variable; but except for __init__ and 356__getattr__ there are no class variables). It will be called, for 357example, when aninstance.method1 is referenced in the example, with 358self set to aninstance and name set to the string "method1".<P> 359 360</UL> 361 362<P>The __getattr__ method looks the name up in the __namespace__ 363dictionary. If it isn't found, it raises an AttributeError exception. 364(In a more realistic example, it would first have to look through the 365base classes as well.) If it is found, there are two possibilities: 366it's either a function or it isn't. If it's not a function, it is 367assumed to be a class variable, and its value is returned. If it's a 368function, we have to ``wrap'' it in instance of yet another helper 369class, BoundMethod. 370 371<P>The BoundMethod class is needed to implement a familiar feature: 372when a method is defined, it has an initial argument, self, which is 373automatically bound to the relevant instance when it is called. For 374example, aninstance.method1(10) is equivalent to method1(aninstance, 37510). In the example if this call, first a temporary BoundMethod 376instance is created with the following constructor call: temp = 377BoundMethod(method1, aninstance); then this instance is called as 378temp(10). After the call, the temporary instance is discarded. 379 380<P> 381 382<UL> 383 384<LI>The __init__ method is invoked for the constructor call 385BoundMethod(method1, aninstance). It simply saves away its 386arguments.<P> 387 388<LI>The __call__ method is invoked when the bound method instance is 389called, as in temp(10). It needs to call method1(aninstance, 10). 390However, even though self.function is now method1 and self.instance is 391aninstance, it can't call self.function(self.instance, args) directly, 392because it should work regardless of the number of arguments passed. 393(For simplicity, support for keyword arguments has been omitted.)<P> 394 395</UL> 396 397<P>In order to be able to support arbitrary argument lists, the 398__call__ method first constructs a new argument tuple. Conveniently, 399because of the notation *args in __call__'s own argument list, the 400arguments to __call__ (except for self) are placed in the tuple args. 401To construct the desired argument list, we concatenate a singleton 402tuple containing the instance with the args tuple: (self.instance,) + 403args. (Note the trailing comma used to construct the singleton 404tuple.) In our example, the resulting argument tuple is (aninstance, 40510). 406 407<P>The intrinsic function apply() takes a function and an argument 408tuple and calls the function for it. In our example, we are calling 409apply(method1, (aninstance, 10)) which is equivalent to calling 410method(aninstance, 10). 411 412<P>From here on, things should come together quite easily. The output 413of the example code is something like this: 414 415<PRE> 416calling <function method1 at ae8d8> for <Instance instance at 95ab0> with (10,) 417calling <function method2 at ae900> for <Instance instance at 95ab0> with () 418the answer is 10 419</PRE> 420 421<P>That was about the shortest meaningful example that I could come up 422with. A real tracing metaclass (for example, <A 423HREF="#Trace">Trace.py</A> discussed below) needs to be more 424complicated in two dimensions. 425 426<P>First, it needs to support more advanced Python features such as 427class variables, inheritance, __init__ methods, and keyword arguments. 428 429<P>Second, it needs to provide a more flexible way to handle the 430actual tracing information; perhaps it should be possible to write 431your own tracing function that gets called, perhaps it should be 432possible to enable and disable tracing on a per-class or per-instance 433basis, and perhaps a filter so that only interesting calls are traced; 434it should also be able to trace the return value of the call (or the 435exception it raised if an error occurs). Even the Trace.py example 436doesn't support all these features yet. 437 438<P> 439 440<HR> 441 442<H1>Real-life Examples</H1> 443 444<P>Have a look at some very preliminary examples that I coded up to 445teach myself how to write metaclasses: 446 447<DL> 448 449<DT><A HREF="Enum.py">Enum.py</A> 450 451<DD>This (ab)uses the class syntax as an elegant way to define 452enumerated types. The resulting classes are never instantiated -- 453rather, their class attributes are the enumerated values. For 454example: 455 456<PRE> 457class Color(Enum): 458 red = 1 459 green = 2 460 blue = 3 461print Color.red 462</PRE> 463 464will print the string ``Color.red'', while ``Color.red==1'' is true, 465and ``Color.red + 1'' raise a TypeError exception. 466 467<P> 468 469<DT><A NAME=Trace></A><A HREF="Trace.py">Trace.py</A> 470 471<DD>The resulting classes work much like standard 472classes, but by setting a special class or instance attribute 473__trace_output__ to point to a file, all calls to the class's methods 474are traced. It was a bit of a struggle to get this right. This 475should probably redone using the generic metaclass below. 476 477<P> 478 479<DT><A HREF="Meta.py">Meta.py</A> 480 481<DD>A generic metaclass. This is an attempt at finding out how much 482standard class behavior can be mimicked by a metaclass. The 483preliminary answer appears to be that everything's fine as long as the 484class (or its clients) don't look at the instance's __class__ 485attribute, nor at the class's __dict__ attribute. The use of 486__getattr__ internally makes the classic implementation of __getattr__ 487hooks tough; we provide a similar hook _getattr_ instead. 488(__setattr__ and __delattr__ are not affected.) 489(XXX Hm. Could detect presence of __getattr__ and rename it.) 490 491<P> 492 493<DT><A HREF="Eiffel.py">Eiffel.py</A> 494 495<DD>Uses the above generic metaclass to implement Eiffel style 496pre-conditions and post-conditions. 497 498<P> 499 500<DT><A HREF="Synch.py">Synch.py</A> 501 502<DD>Uses the above generic metaclass to implement synchronized 503methods. 504 505<P> 506 507<DT><A HREF="Simple.py">Simple.py</A> 508 509<DD>The example module used above. 510 511<P> 512 513</DL> 514 515<P>A pattern seems to be emerging: almost all these uses of 516metaclasses (except for Enum, which is probably more cute than useful) 517mostly work by placing wrappers around method calls. An obvious 518problem with that is that it's not easy to combine the features of 519different metaclasses, while this would actually be quite useful: for 520example, I wouldn't mind getting a trace from the test run of the 521Synch module, and it would be interesting to add preconditions to it 522as well. This needs more research. Perhaps a metaclass could be 523provided that allows stackable wrappers... 524 525<P> 526 527<HR> 528 529<H2>Things You Could Do With Metaclasses</H2> 530 531<P>There are lots of things you could do with metaclasses. Most of 532these can also be done with creative use of __getattr__, but 533metaclasses make it easier to modify the attribute lookup behavior of 534classes. Here's a partial list. 535 536<P> 537 538<UL> 539 540<LI>Enforce different inheritance semantics, e.g. automatically call 541base class methods when a derived class overrides<P> 542 543<LI>Implement class methods (e.g. if the first argument is not named 544'self')<P> 545 546<LI>Implement that each instance is initialized with <b>copies</b> of 547all class variables<P> 548 549<LI>Implement a different way to store instance variables (e.g. in a 550list kept outside the instance but indexed by the instance's id())<P> 551 552<LI>Automatically wrap or trap all or certain methods 553 554<UL> 555 556<LI>for tracing 557 558<LI>for precondition and postcondition checking 559 560<LI>for synchronized methods 561 562<LI>for automatic value caching 563 564</UL> 565<P> 566 567<LI>When an attribute is a parameterless function, call it on 568reference (to mimic it being an instance variable); same on assignment<P> 569 570<LI>Instrumentation: see how many times various attributes are used<P> 571 572<LI>Different semantics for __setattr__ and __getattr__ (e.g. disable 573them when they are being used recursively)<P> 574 575<LI>Abuse class syntax for other things<P> 576 577<LI>Experiment with automatic type checking<P> 578 579<LI>Delegation (or acquisition)<P> 580 581<LI>Dynamic inheritance patterns<P> 582 583<LI>Automatic caching of methods<P> 584 585</UL> 586 587<P> 588 589<HR> 590 591<H4>Credits</H4> 592 593<P>Many thanks to David Ascher and Donald Beaudry for their comments 594on earlier draft of this paper. Also thanks to Matt Conway and Tommy 595Burnette for putting a seed for the idea of metaclasses in my 596mind, nearly three years ago, even though at the time my response was 597``you can do that with __getattr__ hooks...'' :-) 598 599<P> 600 601<HR> 602 603</BODY> 604 605</HTML>