PageRenderTime 55ms CodeModel.GetById 10ms app.highlight 33ms RepoModel.GetById 1ms app.codeStats 0ms

/Lib/pickletools.py

http://unladen-swallow.googlecode.com/
Python | 2271 lines | 2254 code | 6 blank | 11 comment | 27 complexity | b6b9de2d5986fc864ed0e3e5c39bb97f MD5 | raw file

Large files files are truncated, but you can click here to view the full file

   1'''"Executable documentation" for the pickle module.
   2
   3Extensive comments about the pickle protocols and pickle-machine opcodes
   4can be found here.  Some functions meant for external use:
   5
   6genops(pickle)
   7   Generate all the opcodes in a pickle, as (opcode, arg, position) triples.
   8
   9dis(pickle, out=None, memo=None, indentlevel=4)
  10   Print a symbolic disassembly of a pickle.
  11'''
  12
  13__all__ = ['dis', 'genops', 'optimize']
  14
  15# Other ideas:
  16#
  17# - A pickle verifier:  read a pickle and check it exhaustively for
  18#   well-formedness.  dis() does a lot of this already.
  19#
  20# - A protocol identifier:  examine a pickle and return its protocol number
  21#   (== the highest .proto attr value among all the opcodes in the pickle).
  22#   dis() already prints this info at the end.
  23#
  24# - A pickle optimizer:  for example, tuple-building code is sometimes more
  25#   elaborate than necessary, catering for the possibility that the tuple
  26#   is recursive.  Or lots of times a PUT is generated that's never accessed
  27#   by a later GET.
  28
  29
  30"""
  31"A pickle" is a program for a virtual pickle machine (PM, but more accurately
  32called an unpickling machine).  It's a sequence of opcodes, interpreted by the
  33PM, building an arbitrarily complex Python object.
  34
  35For the most part, the PM is very simple:  there are no looping, testing, or
  36conditional instructions, no arithmetic and no function calls.  Opcodes are
  37executed once each, from first to last, until a STOP opcode is reached.
  38
  39The PM has two data areas, "the stack" and "the memo".
  40
  41Many opcodes push Python objects onto the stack; e.g., INT pushes a Python
  42integer object on the stack, whose value is gotten from a decimal string
  43literal immediately following the INT opcode in the pickle bytestream.  Other
  44opcodes take Python objects off the stack.  The result of unpickling is
  45whatever object is left on the stack when the final STOP opcode is executed.
  46
  47The memo is simply an array of objects, or it can be implemented as a dict
  48mapping little integers to objects.  The memo serves as the PM's "long term
  49memory", and the little integers indexing the memo are akin to variable
  50names.  Some opcodes pop a stack object into the memo at a given index,
  51and others push a memo object at a given index onto the stack again.
  52
  53At heart, that's all the PM has.  Subtleties arise for these reasons:
  54
  55+ Object identity.  Objects can be arbitrarily complex, and subobjects
  56  may be shared (for example, the list [a, a] refers to the same object a
  57  twice).  It can be vital that unpickling recreate an isomorphic object
  58  graph, faithfully reproducing sharing.
  59
  60+ Recursive objects.  For example, after "L = []; L.append(L)", L is a
  61  list, and L[0] is the same list.  This is related to the object identity
  62  point, and some sequences of pickle opcodes are subtle in order to
  63  get the right result in all cases.
  64
  65+ Things pickle doesn't know everything about.  Examples of things pickle
  66  does know everything about are Python's builtin scalar and container
  67  types, like ints and tuples.  They generally have opcodes dedicated to
  68  them.  For things like module references and instances of user-defined
  69  classes, pickle's knowledge is limited.  Historically, many enhancements
  70  have been made to the pickle protocol in order to do a better (faster,
  71  and/or more compact) job on those.
  72
  73+ Backward compatibility and micro-optimization.  As explained below,
  74  pickle opcodes never go away, not even when better ways to do a thing
  75  get invented.  The repertoire of the PM just keeps growing over time.
  76  For example, protocol 0 had two opcodes for building Python integers (INT
  77  and LONG), protocol 1 added three more for more-efficient pickling of short
  78  integers, and protocol 2 added two more for more-efficient pickling of
  79  long integers (before protocol 2, the only ways to pickle a Python long
  80  took time quadratic in the number of digits, for both pickling and
  81  unpickling).  "Opcode bloat" isn't so much a subtlety as a source of
  82  wearying complication.
  83
  84
  85Pickle protocols:
  86
  87For compatibility, the meaning of a pickle opcode never changes.  Instead new
  88pickle opcodes get added, and each version's unpickler can handle all the
  89pickle opcodes in all protocol versions to date.  So old pickles continue to
  90be readable forever.  The pickler can generally be told to restrict itself to
  91the subset of opcodes available under previous protocol versions too, so that
  92users can create pickles under the current version readable by older
  93versions.  However, a pickle does not contain its version number embedded
  94within it.  If an older unpickler tries to read a pickle using a later
  95protocol, the result is most likely an exception due to seeing an unknown (in
  96the older unpickler) opcode.
  97
  98The original pickle used what's now called "protocol 0", and what was called
  99"text mode" before Python 2.3.  The entire pickle bytestream is made up of
 100printable 7-bit ASCII characters, plus the newline character, in protocol 0.
 101That's why it was called text mode.  Protocol 0 is small and elegant, but
 102sometimes painfully inefficient.
 103
 104The second major set of additions is now called "protocol 1", and was called
 105"binary mode" before Python 2.3.  This added many opcodes with arguments
 106consisting of arbitrary bytes, including NUL bytes and unprintable "high bit"
 107bytes.  Binary mode pickles can be substantially smaller than equivalent
 108text mode pickles, and sometimes faster too; e.g., BININT represents a 4-byte
 109int as 4 bytes following the opcode, which is cheaper to unpickle than the
 110(perhaps) 11-character decimal string attached to INT.  Protocol 1 also added
 111a number of opcodes that operate on many stack elements at once (like APPENDS
 112and SETITEMS), and "shortcut" opcodes (like EMPTY_DICT and EMPTY_TUPLE).
 113
 114The third major set of additions came in Python 2.3, and is called "protocol
 1152".  This added:
 116
 117- A better way to pickle instances of new-style classes (NEWOBJ).
 118
 119- A way for a pickle to identify its protocol (PROTO).
 120
 121- Time- and space- efficient pickling of long ints (LONG{1,4}).
 122
 123- Shortcuts for small tuples (TUPLE{1,2,3}}.
 124
 125- Dedicated opcodes for bools (NEWTRUE, NEWFALSE).
 126
 127- The "extension registry", a vector of popular objects that can be pushed
 128  efficiently by index (EXT{1,2,4}).  This is akin to the memo and GET, but
 129  the registry contents are predefined (there's nothing akin to the memo's
 130  PUT).
 131
 132Another independent change with Python 2.3 is the abandonment of any
 133pretense that it might be safe to load pickles received from untrusted
 134parties -- no sufficient security analysis has been done to guarantee
 135this and there isn't a use case that warrants the expense of such an
 136analysis.
 137
 138To this end, all tests for __safe_for_unpickling__ or for
 139copy_reg.safe_constructors are removed from the unpickling code.
 140References to these variables in the descriptions below are to be seen
 141as describing unpickling in Python 2.2 and before.
 142"""
 143
 144# Meta-rule:  Descriptions are stored in instances of descriptor objects,
 145# with plain constructors.  No meta-language is defined from which
 146# descriptors could be constructed.  If you want, e.g., XML, write a little
 147# program to generate XML from the objects.
 148
 149##############################################################################
 150# Some pickle opcodes have an argument, following the opcode in the
 151# bytestream.  An argument is of a specific type, described by an instance
 152# of ArgumentDescriptor.  These are not to be confused with arguments taken
 153# off the stack -- ArgumentDescriptor applies only to arguments embedded in
 154# the opcode stream, immediately following an opcode.
 155
 156# Represents the number of bytes consumed by an argument delimited by the
 157# next newline character.
 158UP_TO_NEWLINE = -1
 159
 160# Represents the number of bytes consumed by a two-argument opcode where
 161# the first argument gives the number of bytes in the second argument.
 162TAKEN_FROM_ARGUMENT1 = -2   # num bytes is 1-byte unsigned int
 163TAKEN_FROM_ARGUMENT4 = -3   # num bytes is 4-byte signed little-endian int
 164
 165class ArgumentDescriptor(object):
 166    __slots__ = (
 167        # name of descriptor record, also a module global name; a string
 168        'name',
 169
 170        # length of argument, in bytes; an int; UP_TO_NEWLINE and
 171        # TAKEN_FROM_ARGUMENT{1,4} are negative values for variable-length
 172        # cases
 173        'n',
 174
 175        # a function taking a file-like object, reading this kind of argument
 176        # from the object at the current position, advancing the current
 177        # position by n bytes, and returning the value of the argument
 178        'reader',
 179
 180        # human-readable docs for this arg descriptor; a string
 181        'doc',
 182    )
 183
 184    def __init__(self, name, n, reader, doc):
 185        assert isinstance(name, str)
 186        self.name = name
 187
 188        assert isinstance(n, int) and (n >= 0 or
 189                                       n in (UP_TO_NEWLINE,
 190                                             TAKEN_FROM_ARGUMENT1,
 191                                             TAKEN_FROM_ARGUMENT4))
 192        self.n = n
 193
 194        self.reader = reader
 195
 196        assert isinstance(doc, str)
 197        self.doc = doc
 198
 199from struct import unpack as _unpack
 200
 201def read_uint1(f):
 202    r"""
 203    >>> import StringIO
 204    >>> read_uint1(StringIO.StringIO('\xff'))
 205    255
 206    """
 207
 208    data = f.read(1)
 209    if data:
 210        return ord(data)
 211    raise ValueError("not enough data in stream to read uint1")
 212
 213uint1 = ArgumentDescriptor(
 214            name='uint1',
 215            n=1,
 216            reader=read_uint1,
 217            doc="One-byte unsigned integer.")
 218
 219
 220def read_uint2(f):
 221    r"""
 222    >>> import StringIO
 223    >>> read_uint2(StringIO.StringIO('\xff\x00'))
 224    255
 225    >>> read_uint2(StringIO.StringIO('\xff\xff'))
 226    65535
 227    """
 228
 229    data = f.read(2)
 230    if len(data) == 2:
 231        return _unpack("<H", data)[0]
 232    raise ValueError("not enough data in stream to read uint2")
 233
 234uint2 = ArgumentDescriptor(
 235            name='uint2',
 236            n=2,
 237            reader=read_uint2,
 238            doc="Two-byte unsigned integer, little-endian.")
 239
 240
 241def read_int4(f):
 242    r"""
 243    >>> import StringIO
 244    >>> read_int4(StringIO.StringIO('\xff\x00\x00\x00'))
 245    255
 246    >>> read_int4(StringIO.StringIO('\x00\x00\x00\x80')) == -(2**31)
 247    True
 248    """
 249
 250    data = f.read(4)
 251    if len(data) == 4:
 252        return _unpack("<i", data)[0]
 253    raise ValueError("not enough data in stream to read int4")
 254
 255int4 = ArgumentDescriptor(
 256           name='int4',
 257           n=4,
 258           reader=read_int4,
 259           doc="Four-byte signed integer, little-endian, 2's complement.")
 260
 261
 262def read_stringnl(f, decode=True, stripquotes=True):
 263    r"""
 264    >>> import StringIO
 265    >>> read_stringnl(StringIO.StringIO("'abcd'\nefg\n"))
 266    'abcd'
 267
 268    >>> read_stringnl(StringIO.StringIO("\n"))
 269    Traceback (most recent call last):
 270    ...
 271    ValueError: no string quotes around ''
 272
 273    >>> read_stringnl(StringIO.StringIO("\n"), stripquotes=False)
 274    ''
 275
 276    >>> read_stringnl(StringIO.StringIO("''\n"))
 277    ''
 278
 279    >>> read_stringnl(StringIO.StringIO('"abcd"'))
 280    Traceback (most recent call last):
 281    ...
 282    ValueError: no newline found when trying to read stringnl
 283
 284    Embedded escapes are undone in the result.
 285    >>> read_stringnl(StringIO.StringIO(r"'a\n\\b\x00c\td'" + "\n'e'"))
 286    'a\n\\b\x00c\td'
 287    """
 288
 289    data = f.readline()
 290    if not data.endswith('\n'):
 291        raise ValueError("no newline found when trying to read stringnl")
 292    data = data[:-1]    # lose the newline
 293
 294    if stripquotes:
 295        for q in "'\"":
 296            if data.startswith(q):
 297                if not data.endswith(q):
 298                    raise ValueError("strinq quote %r not found at both "
 299                                     "ends of %r" % (q, data))
 300                data = data[1:-1]
 301                break
 302        else:
 303            raise ValueError("no string quotes around %r" % data)
 304
 305    # I'm not sure when 'string_escape' was added to the std codecs; it's
 306    # crazy not to use it if it's there.
 307    if decode:
 308        data = data.decode('string_escape')
 309    return data
 310
 311stringnl = ArgumentDescriptor(
 312               name='stringnl',
 313               n=UP_TO_NEWLINE,
 314               reader=read_stringnl,
 315               doc="""A newline-terminated string.
 316
 317                   This is a repr-style string, with embedded escapes, and
 318                   bracketing quotes.
 319                   """)
 320
 321def read_stringnl_noescape(f):
 322    return read_stringnl(f, decode=False, stripquotes=False)
 323
 324stringnl_noescape = ArgumentDescriptor(
 325                        name='stringnl_noescape',
 326                        n=UP_TO_NEWLINE,
 327                        reader=read_stringnl_noescape,
 328                        doc="""A newline-terminated string.
 329
 330                        This is a str-style string, without embedded escapes,
 331                        or bracketing quotes.  It should consist solely of
 332                        printable ASCII characters.
 333                        """)
 334
 335def read_stringnl_noescape_pair(f):
 336    r"""
 337    >>> import StringIO
 338    >>> read_stringnl_noescape_pair(StringIO.StringIO("Queue\nEmpty\njunk"))
 339    'Queue Empty'
 340    """
 341
 342    return "%s %s" % (read_stringnl_noescape(f), read_stringnl_noescape(f))
 343
 344stringnl_noescape_pair = ArgumentDescriptor(
 345                             name='stringnl_noescape_pair',
 346                             n=UP_TO_NEWLINE,
 347                             reader=read_stringnl_noescape_pair,
 348                             doc="""A pair of newline-terminated strings.
 349
 350                             These are str-style strings, without embedded
 351                             escapes, or bracketing quotes.  They should
 352                             consist solely of printable ASCII characters.
 353                             The pair is returned as a single string, with
 354                             a single blank separating the two strings.
 355                             """)
 356
 357def read_string4(f):
 358    r"""
 359    >>> import StringIO
 360    >>> read_string4(StringIO.StringIO("\x00\x00\x00\x00abc"))
 361    ''
 362    >>> read_string4(StringIO.StringIO("\x03\x00\x00\x00abcdef"))
 363    'abc'
 364    >>> read_string4(StringIO.StringIO("\x00\x00\x00\x03abcdef"))
 365    Traceback (most recent call last):
 366    ...
 367    ValueError: expected 50331648 bytes in a string4, but only 6 remain
 368    """
 369
 370    n = read_int4(f)
 371    if n < 0:
 372        raise ValueError("string4 byte count < 0: %d" % n)
 373    data = f.read(n)
 374    if len(data) == n:
 375        return data
 376    raise ValueError("expected %d bytes in a string4, but only %d remain" %
 377                     (n, len(data)))
 378
 379string4 = ArgumentDescriptor(
 380              name="string4",
 381              n=TAKEN_FROM_ARGUMENT4,
 382              reader=read_string4,
 383              doc="""A counted string.
 384
 385              The first argument is a 4-byte little-endian signed int giving
 386              the number of bytes in the string, and the second argument is
 387              that many bytes.
 388              """)
 389
 390
 391def read_string1(f):
 392    r"""
 393    >>> import StringIO
 394    >>> read_string1(StringIO.StringIO("\x00"))
 395    ''
 396    >>> read_string1(StringIO.StringIO("\x03abcdef"))
 397    'abc'
 398    """
 399
 400    n = read_uint1(f)
 401    assert n >= 0
 402    data = f.read(n)
 403    if len(data) == n:
 404        return data
 405    raise ValueError("expected %d bytes in a string1, but only %d remain" %
 406                     (n, len(data)))
 407
 408string1 = ArgumentDescriptor(
 409              name="string1",
 410              n=TAKEN_FROM_ARGUMENT1,
 411              reader=read_string1,
 412              doc="""A counted string.
 413
 414              The first argument is a 1-byte unsigned int giving the number
 415              of bytes in the string, and the second argument is that many
 416              bytes.
 417              """)
 418
 419
 420def read_unicodestringnl(f):
 421    r"""
 422    >>> import StringIO
 423    >>> read_unicodestringnl(StringIO.StringIO("abc\uabcd\njunk"))
 424    u'abc\uabcd'
 425    """
 426
 427    data = f.readline()
 428    if not data.endswith('\n'):
 429        raise ValueError("no newline found when trying to read "
 430                         "unicodestringnl")
 431    data = data[:-1]    # lose the newline
 432    return unicode(data, 'raw-unicode-escape')
 433
 434unicodestringnl = ArgumentDescriptor(
 435                      name='unicodestringnl',
 436                      n=UP_TO_NEWLINE,
 437                      reader=read_unicodestringnl,
 438                      doc="""A newline-terminated Unicode string.
 439
 440                      This is raw-unicode-escape encoded, so consists of
 441                      printable ASCII characters, and may contain embedded
 442                      escape sequences.
 443                      """)
 444
 445def read_unicodestring4(f):
 446    r"""
 447    >>> import StringIO
 448    >>> s = u'abcd\uabcd'
 449    >>> enc = s.encode('utf-8')
 450    >>> enc
 451    'abcd\xea\xaf\x8d'
 452    >>> n = chr(len(enc)) + chr(0) * 3  # little-endian 4-byte length
 453    >>> t = read_unicodestring4(StringIO.StringIO(n + enc + 'junk'))
 454    >>> s == t
 455    True
 456
 457    >>> read_unicodestring4(StringIO.StringIO(n + enc[:-1]))
 458    Traceback (most recent call last):
 459    ...
 460    ValueError: expected 7 bytes in a unicodestring4, but only 6 remain
 461    """
 462
 463    n = read_int4(f)
 464    if n < 0:
 465        raise ValueError("unicodestring4 byte count < 0: %d" % n)
 466    data = f.read(n)
 467    if len(data) == n:
 468        return unicode(data, 'utf-8')
 469    raise ValueError("expected %d bytes in a unicodestring4, but only %d "
 470                     "remain" % (n, len(data)))
 471
 472unicodestring4 = ArgumentDescriptor(
 473                    name="unicodestring4",
 474                    n=TAKEN_FROM_ARGUMENT4,
 475                    reader=read_unicodestring4,
 476                    doc="""A counted Unicode string.
 477
 478                    The first argument is a 4-byte little-endian signed int
 479                    giving the number of bytes in the string, and the second
 480                    argument-- the UTF-8 encoding of the Unicode string --
 481                    contains that many bytes.
 482                    """)
 483
 484
 485def read_decimalnl_short(f):
 486    r"""
 487    >>> import StringIO
 488    >>> read_decimalnl_short(StringIO.StringIO("1234\n56"))
 489    1234
 490
 491    >>> read_decimalnl_short(StringIO.StringIO("1234L\n56"))
 492    Traceback (most recent call last):
 493    ...
 494    ValueError: trailing 'L' not allowed in '1234L'
 495    """
 496
 497    s = read_stringnl(f, decode=False, stripquotes=False)
 498    if s.endswith("L"):
 499        raise ValueError("trailing 'L' not allowed in %r" % s)
 500
 501    # It's not necessarily true that the result fits in a Python short int:
 502    # the pickle may have been written on a 64-bit box.  There's also a hack
 503    # for True and False here.
 504    if s == "00":
 505        return False
 506    elif s == "01":
 507        return True
 508
 509    try:
 510        return int(s)
 511    except OverflowError:
 512        return long(s)
 513
 514def read_decimalnl_long(f):
 515    r"""
 516    >>> import StringIO
 517
 518    >>> read_decimalnl_long(StringIO.StringIO("1234\n56"))
 519    Traceback (most recent call last):
 520    ...
 521    ValueError: trailing 'L' required in '1234'
 522
 523    Someday the trailing 'L' will probably go away from this output.
 524
 525    >>> read_decimalnl_long(StringIO.StringIO("1234L\n56"))
 526    1234L
 527
 528    >>> read_decimalnl_long(StringIO.StringIO("123456789012345678901234L\n6"))
 529    123456789012345678901234L
 530    """
 531
 532    s = read_stringnl(f, decode=False, stripquotes=False)
 533    if not s.endswith("L"):
 534        raise ValueError("trailing 'L' required in %r" % s)
 535    return long(s)
 536
 537
 538decimalnl_short = ArgumentDescriptor(
 539                      name='decimalnl_short',
 540                      n=UP_TO_NEWLINE,
 541                      reader=read_decimalnl_short,
 542                      doc="""A newline-terminated decimal integer literal.
 543
 544                          This never has a trailing 'L', and the integer fit
 545                          in a short Python int on the box where the pickle
 546                          was written -- but there's no guarantee it will fit
 547                          in a short Python int on the box where the pickle
 548                          is read.
 549                          """)
 550
 551decimalnl_long = ArgumentDescriptor(
 552                     name='decimalnl_long',
 553                     n=UP_TO_NEWLINE,
 554                     reader=read_decimalnl_long,
 555                     doc="""A newline-terminated decimal integer literal.
 556
 557                         This has a trailing 'L', and can represent integers
 558                         of any size.
 559                         """)
 560
 561
 562def read_floatnl(f):
 563    r"""
 564    >>> import StringIO
 565    >>> read_floatnl(StringIO.StringIO("-1.25\n6"))
 566    -1.25
 567    """
 568    s = read_stringnl(f, decode=False, stripquotes=False)
 569    return float(s)
 570
 571floatnl = ArgumentDescriptor(
 572              name='floatnl',
 573              n=UP_TO_NEWLINE,
 574              reader=read_floatnl,
 575              doc="""A newline-terminated decimal floating literal.
 576
 577              In general this requires 17 significant digits for roundtrip
 578              identity, and pickling then unpickling infinities, NaNs, and
 579              minus zero doesn't work across boxes, or on some boxes even
 580              on itself (e.g., Windows can't read the strings it produces
 581              for infinities or NaNs).
 582              """)
 583
 584def read_float8(f):
 585    r"""
 586    >>> import StringIO, struct
 587    >>> raw = struct.pack(">d", -1.25)
 588    >>> raw
 589    '\xbf\xf4\x00\x00\x00\x00\x00\x00'
 590    >>> read_float8(StringIO.StringIO(raw + "\n"))
 591    -1.25
 592    """
 593
 594    data = f.read(8)
 595    if len(data) == 8:
 596        return _unpack(">d", data)[0]
 597    raise ValueError("not enough data in stream to read float8")
 598
 599
 600float8 = ArgumentDescriptor(
 601             name='float8',
 602             n=8,
 603             reader=read_float8,
 604             doc="""An 8-byte binary representation of a float, big-endian.
 605
 606             The format is unique to Python, and shared with the struct
 607             module (format string '>d') "in theory" (the struct and cPickle
 608             implementations don't share the code -- they should).  It's
 609             strongly related to the IEEE-754 double format, and, in normal
 610             cases, is in fact identical to the big-endian 754 double format.
 611             On other boxes the dynamic range is limited to that of a 754
 612             double, and "add a half and chop" rounding is used to reduce
 613             the precision to 53 bits.  However, even on a 754 box,
 614             infinities, NaNs, and minus zero may not be handled correctly
 615             (may not survive roundtrip pickling intact).
 616             """)
 617
 618# Protocol 2 formats
 619
 620from pickle import decode_long
 621
 622def read_long1(f):
 623    r"""
 624    >>> import StringIO
 625    >>> read_long1(StringIO.StringIO("\x00"))
 626    0L
 627    >>> read_long1(StringIO.StringIO("\x02\xff\x00"))
 628    255L
 629    >>> read_long1(StringIO.StringIO("\x02\xff\x7f"))
 630    32767L
 631    >>> read_long1(StringIO.StringIO("\x02\x00\xff"))
 632    -256L
 633    >>> read_long1(StringIO.StringIO("\x02\x00\x80"))
 634    -32768L
 635    """
 636
 637    n = read_uint1(f)
 638    data = f.read(n)
 639    if len(data) != n:
 640        raise ValueError("not enough data in stream to read long1")
 641    return decode_long(data)
 642
 643long1 = ArgumentDescriptor(
 644    name="long1",
 645    n=TAKEN_FROM_ARGUMENT1,
 646    reader=read_long1,
 647    doc="""A binary long, little-endian, using 1-byte size.
 648
 649    This first reads one byte as an unsigned size, then reads that
 650    many bytes and interprets them as a little-endian 2's-complement long.
 651    If the size is 0, that's taken as a shortcut for the long 0L.
 652    """)
 653
 654def read_long4(f):
 655    r"""
 656    >>> import StringIO
 657    >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\xff\x00"))
 658    255L
 659    >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\xff\x7f"))
 660    32767L
 661    >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\x00\xff"))
 662    -256L
 663    >>> read_long4(StringIO.StringIO("\x02\x00\x00\x00\x00\x80"))
 664    -32768L
 665    >>> read_long1(StringIO.StringIO("\x00\x00\x00\x00"))
 666    0L
 667    """
 668
 669    n = read_int4(f)
 670    if n < 0:
 671        raise ValueError("long4 byte count < 0: %d" % n)
 672    data = f.read(n)
 673    if len(data) != n:
 674        raise ValueError("not enough data in stream to read long4")
 675    return decode_long(data)
 676
 677long4 = ArgumentDescriptor(
 678    name="long4",
 679    n=TAKEN_FROM_ARGUMENT4,
 680    reader=read_long4,
 681    doc="""A binary representation of a long, little-endian.
 682
 683    This first reads four bytes as a signed size (but requires the
 684    size to be >= 0), then reads that many bytes and interprets them
 685    as a little-endian 2's-complement long.  If the size is 0, that's taken
 686    as a shortcut for the long 0L, although LONG1 should really be used
 687    then instead (and in any case where # of bytes < 256).
 688    """)
 689
 690
 691##############################################################################
 692# Object descriptors.  The stack used by the pickle machine holds objects,
 693# and in the stack_before and stack_after attributes of OpcodeInfo
 694# descriptors we need names to describe the various types of objects that can
 695# appear on the stack.
 696
 697class StackObject(object):
 698    __slots__ = (
 699        # name of descriptor record, for info only
 700        'name',
 701
 702        # type of object, or tuple of type objects (meaning the object can
 703        # be of any type in the tuple)
 704        'obtype',
 705
 706        # human-readable docs for this kind of stack object; a string
 707        'doc',
 708    )
 709
 710    def __init__(self, name, obtype, doc):
 711        assert isinstance(name, str)
 712        self.name = name
 713
 714        assert isinstance(obtype, type) or isinstance(obtype, tuple)
 715        if isinstance(obtype, tuple):
 716            for contained in obtype:
 717                assert isinstance(contained, type)
 718        self.obtype = obtype
 719
 720        assert isinstance(doc, str)
 721        self.doc = doc
 722
 723    def __repr__(self):
 724        return self.name
 725
 726
 727pyint = StackObject(
 728            name='int',
 729            obtype=int,
 730            doc="A short (as opposed to long) Python integer object.")
 731
 732pylong = StackObject(
 733             name='long',
 734             obtype=long,
 735             doc="A long (as opposed to short) Python integer object.")
 736
 737pyinteger_or_bool = StackObject(
 738                        name='int_or_bool',
 739                        obtype=(int, long, bool),
 740                        doc="A Python integer object (short or long), or "
 741                            "a Python bool.")
 742
 743pybool = StackObject(
 744             name='bool',
 745             obtype=(bool,),
 746             doc="A Python bool object.")
 747
 748pyfloat = StackObject(
 749              name='float',
 750              obtype=float,
 751              doc="A Python float object.")
 752
 753pystring = StackObject(
 754               name='str',
 755               obtype=str,
 756               doc="A Python string object.")
 757
 758pyunicode = StackObject(
 759                name='unicode',
 760                obtype=unicode,
 761                doc="A Python Unicode string object.")
 762
 763pynone = StackObject(
 764             name="None",
 765             obtype=type(None),
 766             doc="The Python None object.")
 767
 768pytuple = StackObject(
 769              name="tuple",
 770              obtype=tuple,
 771              doc="A Python tuple object.")
 772
 773pylist = StackObject(
 774             name="list",
 775             obtype=list,
 776             doc="A Python list object.")
 777
 778pydict = StackObject(
 779             name="dict",
 780             obtype=dict,
 781             doc="A Python dict object.")
 782
 783anyobject = StackObject(
 784                name='any',
 785                obtype=object,
 786                doc="Any kind of object whatsoever.")
 787
 788markobject = StackObject(
 789                 name="mark",
 790                 obtype=StackObject,
 791                 doc="""'The mark' is a unique object.
 792
 793                 Opcodes that operate on a variable number of objects
 794                 generally don't embed the count of objects in the opcode,
 795                 or pull it off the stack.  Instead the MARK opcode is used
 796                 to push a special marker object on the stack, and then
 797                 some other opcodes grab all the objects from the top of
 798                 the stack down to (but not including) the topmost marker
 799                 object.
 800                 """)
 801
 802stackslice = StackObject(
 803                 name="stackslice",
 804                 obtype=StackObject,
 805                 doc="""An object representing a contiguous slice of the stack.
 806
 807                 This is used in conjuction with markobject, to represent all
 808                 of the stack following the topmost markobject.  For example,
 809                 the POP_MARK opcode changes the stack from
 810
 811                     [..., markobject, stackslice]
 812                 to
 813                     [...]
 814
 815                 No matter how many object are on the stack after the topmost
 816                 markobject, POP_MARK gets rid of all of them (including the
 817                 topmost markobject too).
 818                 """)
 819
 820##############################################################################
 821# Descriptors for pickle opcodes.
 822
 823class OpcodeInfo(object):
 824
 825    __slots__ = (
 826        # symbolic name of opcode; a string
 827        'name',
 828
 829        # the code used in a bytestream to represent the opcode; a
 830        # one-character string
 831        'code',
 832
 833        # If the opcode has an argument embedded in the byte string, an
 834        # instance of ArgumentDescriptor specifying its type.  Note that
 835        # arg.reader(s) can be used to read and decode the argument from
 836        # the bytestream s, and arg.doc documents the format of the raw
 837        # argument bytes.  If the opcode doesn't have an argument embedded
 838        # in the bytestream, arg should be None.
 839        'arg',
 840
 841        # what the stack looks like before this opcode runs; a list
 842        'stack_before',
 843
 844        # what the stack looks like after this opcode runs; a list
 845        'stack_after',
 846
 847        # the protocol number in which this opcode was introduced; an int
 848        'proto',
 849
 850        # human-readable docs for this opcode; a string
 851        'doc',
 852    )
 853
 854    def __init__(self, name, code, arg,
 855                 stack_before, stack_after, proto, doc):
 856        assert isinstance(name, str)
 857        self.name = name
 858
 859        assert isinstance(code, str)
 860        assert len(code) == 1
 861        self.code = code
 862
 863        assert arg is None or isinstance(arg, ArgumentDescriptor)
 864        self.arg = arg
 865
 866        assert isinstance(stack_before, list)
 867        for x in stack_before:
 868            assert isinstance(x, StackObject)
 869        self.stack_before = stack_before
 870
 871        assert isinstance(stack_after, list)
 872        for x in stack_after:
 873            assert isinstance(x, StackObject)
 874        self.stack_after = stack_after
 875
 876        assert isinstance(proto, int) and 0 <= proto <= 2
 877        self.proto = proto
 878
 879        assert isinstance(doc, str)
 880        self.doc = doc
 881
 882I = OpcodeInfo
 883opcodes = [
 884
 885    # Ways to spell integers.
 886
 887    I(name='INT',
 888      code='I',
 889      arg=decimalnl_short,
 890      stack_before=[],
 891      stack_after=[pyinteger_or_bool],
 892      proto=0,
 893      doc="""Push an integer or bool.
 894
 895      The argument is a newline-terminated decimal literal string.
 896
 897      The intent may have been that this always fit in a short Python int,
 898      but INT can be generated in pickles written on a 64-bit box that
 899      require a Python long on a 32-bit box.  The difference between this
 900      and LONG then is that INT skips a trailing 'L', and produces a short
 901      int whenever possible.
 902
 903      Another difference is due to that, when bool was introduced as a
 904      distinct type in 2.3, builtin names True and False were also added to
 905      2.2.2, mapping to ints 1 and 0.  For compatibility in both directions,
 906      True gets pickled as INT + "I01\\n", and False as INT + "I00\\n".
 907      Leading zeroes are never produced for a genuine integer.  The 2.3
 908      (and later) unpicklers special-case these and return bool instead;
 909      earlier unpicklers ignore the leading "0" and return the int.
 910      """),
 911
 912    I(name='BININT',
 913      code='J',
 914      arg=int4,
 915      stack_before=[],
 916      stack_after=[pyint],
 917      proto=1,
 918      doc="""Push a four-byte signed integer.
 919
 920      This handles the full range of Python (short) integers on a 32-bit
 921      box, directly as binary bytes (1 for the opcode and 4 for the integer).
 922      If the integer is non-negative and fits in 1 or 2 bytes, pickling via
 923      BININT1 or BININT2 saves space.
 924      """),
 925
 926    I(name='BININT1',
 927      code='K',
 928      arg=uint1,
 929      stack_before=[],
 930      stack_after=[pyint],
 931      proto=1,
 932      doc="""Push a one-byte unsigned integer.
 933
 934      This is a space optimization for pickling very small non-negative ints,
 935      in range(256).
 936      """),
 937
 938    I(name='BININT2',
 939      code='M',
 940      arg=uint2,
 941      stack_before=[],
 942      stack_after=[pyint],
 943      proto=1,
 944      doc="""Push a two-byte unsigned integer.
 945
 946      This is a space optimization for pickling small positive ints, in
 947      range(256, 2**16).  Integers in range(256) can also be pickled via
 948      BININT2, but BININT1 instead saves a byte.
 949      """),
 950
 951    I(name='LONG',
 952      code='L',
 953      arg=decimalnl_long,
 954      stack_before=[],
 955      stack_after=[pylong],
 956      proto=0,
 957      doc="""Push a long integer.
 958
 959      The same as INT, except that the literal ends with 'L', and always
 960      unpickles to a Python long.  There doesn't seem a real purpose to the
 961      trailing 'L'.
 962
 963      Note that LONG takes time quadratic in the number of digits when
 964      unpickling (this is simply due to the nature of decimal->binary
 965      conversion).  Proto 2 added linear-time (in C; still quadratic-time
 966      in Python) LONG1 and LONG4 opcodes.
 967      """),
 968
 969    I(name="LONG1",
 970      code='\x8a',
 971      arg=long1,
 972      stack_before=[],
 973      stack_after=[pylong],
 974      proto=2,
 975      doc="""Long integer using one-byte length.
 976
 977      A more efficient encoding of a Python long; the long1 encoding
 978      says it all."""),
 979
 980    I(name="LONG4",
 981      code='\x8b',
 982      arg=long4,
 983      stack_before=[],
 984      stack_after=[pylong],
 985      proto=2,
 986      doc="""Long integer using found-byte length.
 987
 988      A more efficient encoding of a Python long; the long4 encoding
 989      says it all."""),
 990
 991    # Ways to spell strings (8-bit, not Unicode).
 992
 993    I(name='STRING',
 994      code='S',
 995      arg=stringnl,
 996      stack_before=[],
 997      stack_after=[pystring],
 998      proto=0,
 999      doc="""Push a Python string object.
1000
1001      The argument is a repr-style string, with bracketing quote characters,
1002      and perhaps embedded escapes.  The argument extends until the next
1003      newline character.
1004      """),
1005
1006    I(name='BINSTRING',
1007      code='T',
1008      arg=string4,
1009      stack_before=[],
1010      stack_after=[pystring],
1011      proto=1,
1012      doc="""Push a Python string object.
1013
1014      There are two arguments:  the first is a 4-byte little-endian signed int
1015      giving the number of bytes in the string, and the second is that many
1016      bytes, which are taken literally as the string content.
1017      """),
1018
1019    I(name='SHORT_BINSTRING',
1020      code='U',
1021      arg=string1,
1022      stack_before=[],
1023      stack_after=[pystring],
1024      proto=1,
1025      doc="""Push a Python string object.
1026
1027      There are two arguments:  the first is a 1-byte unsigned int giving
1028      the number of bytes in the string, and the second is that many bytes,
1029      which are taken literally as the string content.
1030      """),
1031
1032    # Ways to spell None.
1033
1034    I(name='NONE',
1035      code='N',
1036      arg=None,
1037      stack_before=[],
1038      stack_after=[pynone],
1039      proto=0,
1040      doc="Push None on the stack."),
1041
1042    # Ways to spell bools, starting with proto 2.  See INT for how this was
1043    # done before proto 2.
1044
1045    I(name='NEWTRUE',
1046      code='\x88',
1047      arg=None,
1048      stack_before=[],
1049      stack_after=[pybool],
1050      proto=2,
1051      doc="""True.
1052
1053      Push True onto the stack."""),
1054
1055    I(name='NEWFALSE',
1056      code='\x89',
1057      arg=None,
1058      stack_before=[],
1059      stack_after=[pybool],
1060      proto=2,
1061      doc="""True.
1062
1063      Push False onto the stack."""),
1064
1065    # Ways to spell Unicode strings.
1066
1067    I(name='UNICODE',
1068      code='V',
1069      arg=unicodestringnl,
1070      stack_before=[],
1071      stack_after=[pyunicode],
1072      proto=0,  # this may be pure-text, but it's a later addition
1073      doc="""Push a Python Unicode string object.
1074
1075      The argument is a raw-unicode-escape encoding of a Unicode string,
1076      and so may contain embedded escape sequences.  The argument extends
1077      until the next newline character.
1078      """),
1079
1080    I(name='BINUNICODE',
1081      code='X',
1082      arg=unicodestring4,
1083      stack_before=[],
1084      stack_after=[pyunicode],
1085      proto=1,
1086      doc="""Push a Python Unicode string object.
1087
1088      There are two arguments:  the first is a 4-byte little-endian signed int
1089      giving the number of bytes in the string.  The second is that many
1090      bytes, and is the UTF-8 encoding of the Unicode string.
1091      """),
1092
1093    # Ways to spell floats.
1094
1095    I(name='FLOAT',
1096      code='F',
1097      arg=floatnl,
1098      stack_before=[],
1099      stack_after=[pyfloat],
1100      proto=0,
1101      doc="""Newline-terminated decimal float literal.
1102
1103      The argument is repr(a_float), and in general requires 17 significant
1104      digits for roundtrip conversion to be an identity (this is so for
1105      IEEE-754 double precision values, which is what Python float maps to
1106      on most boxes).
1107
1108      In general, FLOAT cannot be used to transport infinities, NaNs, or
1109      minus zero across boxes (or even on a single box, if the platform C
1110      library can't read the strings it produces for such things -- Windows
1111      is like that), but may do less damage than BINFLOAT on boxes with
1112      greater precision or dynamic range than IEEE-754 double.
1113      """),
1114
1115    I(name='BINFLOAT',
1116      code='G',
1117      arg=float8,
1118      stack_before=[],
1119      stack_after=[pyfloat],
1120      proto=1,
1121      doc="""Float stored in binary form, with 8 bytes of data.
1122
1123      This generally requires less than half the space of FLOAT encoding.
1124      In general, BINFLOAT cannot be used to transport infinities, NaNs, or
1125      minus zero, raises an exception if the exponent exceeds the range of
1126      an IEEE-754 double, and retains no more than 53 bits of precision (if
1127      there are more than that, "add a half and chop" rounding is used to
1128      cut it back to 53 significant bits).
1129      """),
1130
1131    # Ways to build lists.
1132
1133    I(name='EMPTY_LIST',
1134      code=']',
1135      arg=None,
1136      stack_before=[],
1137      stack_after=[pylist],
1138      proto=1,
1139      doc="Push an empty list."),
1140
1141    I(name='APPEND',
1142      code='a',
1143      arg=None,
1144      stack_before=[pylist, anyobject],
1145      stack_after=[pylist],
1146      proto=0,
1147      doc="""Append an object to a list.
1148
1149      Stack before:  ... pylist anyobject
1150      Stack after:   ... pylist+[anyobject]
1151
1152      although pylist is really extended in-place.
1153      """),
1154
1155    I(name='APPENDS',
1156      code='e',
1157      arg=None,
1158      stack_before=[pylist, markobject, stackslice],
1159      stack_after=[pylist],
1160      proto=1,
1161      doc="""Extend a list by a slice of stack objects.
1162
1163      Stack before:  ... pylist markobject stackslice
1164      Stack after:   ... pylist+stackslice
1165
1166      although pylist is really extended in-place.
1167      """),
1168
1169    I(name='LIST',
1170      code='l',
1171      arg=None,
1172      stack_before=[markobject, stackslice],
1173      stack_after=[pylist],
1174      proto=0,
1175      doc="""Build a list out of the topmost stack slice, after markobject.
1176
1177      All the stack entries following the topmost markobject are placed into
1178      a single Python list, which single list object replaces all of the
1179      stack from the topmost markobject onward.  For example,
1180
1181      Stack before: ... markobject 1 2 3 'abc'
1182      Stack after:  ... [1, 2, 3, 'abc']
1183      """),
1184
1185    # Ways to build tuples.
1186
1187    I(name='EMPTY_TUPLE',
1188      code=')',
1189      arg=None,
1190      stack_before=[],
1191      stack_after=[pytuple],
1192      proto=1,
1193      doc="Push an empty tuple."),
1194
1195    I(name='TUPLE',
1196      code='t',
1197      arg=None,
1198      stack_before=[markobject, stackslice],
1199      stack_after=[pytuple],
1200      proto=0,
1201      doc="""Build a tuple out of the topmost stack slice, after markobject.
1202
1203      All the stack entries following the topmost markobject are placed into
1204      a single Python tuple, which single tuple object replaces all of the
1205      stack from the topmost markobject onward.  For example,
1206
1207      Stack before: ... markobject 1 2 3 'abc'
1208      Stack after:  ... (1, 2, 3, 'abc')
1209      """),
1210
1211    I(name='TUPLE1',
1212      code='\x85',
1213      arg=None,
1214      stack_before=[anyobject],
1215      stack_after=[pytuple],
1216      proto=2,
1217      doc="""One-tuple.
1218
1219      This code pops one value off the stack and pushes a tuple of
1220      length 1 whose one item is that value back onto it.  IOW:
1221
1222          stack[-1] = tuple(stack[-1:])
1223      """),
1224
1225    I(name='TUPLE2',
1226      code='\x86',
1227      arg=None,
1228      stack_before=[anyobject, anyobject],
1229      stack_after=[pytuple],
1230      proto=2,
1231      doc="""One-tuple.
1232
1233      This code pops two values off the stack and pushes a tuple
1234      of length 2 whose items are those values back onto it.  IOW:
1235
1236          stack[-2:] = [tuple(stack[-2:])]
1237      """),
1238
1239    I(name='TUPLE3',
1240      code='\x87',
1241      arg=None,
1242      stack_before=[anyobject, anyobject, anyobject],
1243      stack_after=[pytuple],
1244      proto=2,
1245      doc="""One-tuple.
1246
1247      This code pops three values off the stack and pushes a tuple
1248      of length 3 whose items are those values back onto it.  IOW:
1249
1250          stack[-3:] = [tuple(stack[-3:])]
1251      """),
1252
1253    # Ways to build dicts.
1254
1255    I(name='EMPTY_DICT',
1256      code='}',
1257      arg=None,
1258      stack_before=[],
1259      stack_after=[pydict],
1260      proto=1,
1261      doc="Push an empty dict."),
1262
1263    I(name='DICT',
1264      code='d',
1265      arg=None,
1266      stack_before=[markobject, stackslice],
1267      stack_after=[pydict],
1268      proto=0,
1269      doc="""Build a dict out of the topmost stack slice, after markobject.
1270
1271      All the stack entries following the topmost markobject are placed into
1272      a single Python dict, which single dict object replaces all of the
1273      stack from the topmost markobject onward.  The stack slice alternates
1274      key, value, key, value, ....  For example,
1275
1276      Stack before: ... markobject 1 2 3 'abc'
1277      Stack after:  ... {1: 2, 3: 'abc'}
1278      """),
1279
1280    I(name='SETITEM',
1281      code='s',
1282      arg=None,
1283      stack_before=[pydict, anyobject, anyobject],
1284      stack_after=[pydict],
1285      proto=0,
1286      doc="""Add a key+value pair to an existing dict.
1287
1288      Stack before:  ... pydict key value
1289      Stack after:   ... pydict
1290
1291      where pydict has been modified via pydict[key] = value.
1292      """),
1293
1294    I(name='SETITEMS',
1295      code='u',
1296      arg=None,
1297      stack_before=[pydict, markobject, stackslice],
1298      stack_after=[pydict],
1299      proto=1,
1300      doc="""Add an arbitrary number of key+value pairs to an existing dict.
1301
1302      The slice of the stack following the topmost markobject is taken as
1303      an alternating sequence of keys and values, added to the dict
1304      immediately under the topmost markobject.  Everything at and after the
1305      topmost markobject is popped, leaving the mutated dict at the top
1306      of the stack.
1307
1308      Stack before:  ... pydict markobject key_1 value_1 ... key_n value_n
1309      Stack after:   ... pydict
1310
1311      where pydict has been modified via pydict[key_i] = value_i for i in
1312      1, 2, ..., n, and in that order.
1313      """),
1314
1315    # Stack manipulation.
1316
1317    I(name='POP',
1318      code='0',
1319      arg=None,
1320      stack_before=[anyobject],
1321      stack_after=[],
1322      proto=0,
1323      doc="Discard the top stack item, shrinking the stack by one item."),
1324
1325    I(name='DUP',
1326      code='2',
1327      arg=None,
1328      stack_before=[anyobject],
1329      stack_after=[anyobject, anyobject],
1330      proto=0,
1331      doc="Push the top stack item onto the stack again, duplicating it."),
1332
1333    I(name='MARK',
1334      code='(',
1335      arg=None,
1336      stack_before=[],
1337      stack_after=[markobject],
1338      proto=0,
1339      doc="""Push markobject onto the stack.
1340
1341      markobject is a unique object, used by other opcodes to identify a
1342      region of the stack containing a variable number of objects for them
1343      to work on.  See markobject.doc for more detail.
1344      """),
1345
1346    I(name='POP_MARK',
1347      code='1',
1348      arg=None,
1349      stack_before=[markobject, stackslice],
1350      stack_after=[],
1351      proto=1,
1352      doc="""Pop all the stack objects at and above the topmost markobject.
1353
1354      When an opcode using a variable number of stack objects is done,
1355      POP_MARK is used to remove those objects, and to remove the markobject
1356      that delimited their starting position on the stack.
1357      """),
1358
1359    # Memo manipulation.  There are really only two operations (get and put),
1360    # each in all-text, "short binary", and "long binary" flavors.
1361
1362    I(name='GET',
1363      code='g',
1364      arg=decimalnl_short,
1365      stack_before=[],
1366      stack_after=[anyobject],
1367      proto=0,
1368      doc="""Read an object from the memo and push it on the stack.
1369
1370      The index of the memo object to push is given by the newline-teriminated
1371      decimal string following.  BINGET and LONG_BINGET are space-optimized
1372      versions.
1373      """),
1374
1375    I(name='BINGET',
1376      code='h',
1377      arg=uint1,
1378      stack_before=[],
1379      stack_after=[anyobject],
1380      proto=1,
1381      doc="""Read an object from the memo and push it on the stack.
1382
1383      The index of the memo object to push is given by the 1-byte unsigned
1384      integer following.
1385      """),
1386
1387    I(name='LONG_BINGET',
1388      code='j',
1389      arg=int4,
1390      stack_before=[],
1391      stack_after=[anyobject],
1392      proto=1,
1393      doc="""Read an object from the memo and push it on the stack.
1394
1395      The index of the memo object to push is given by the 4-byte signed
1396      little-endian integer following.
1397      """),
1398
1399    I(name='PUT',
1400      code='p',
1401      arg=decimalnl_short,
1402      stack_before=[],
1403      stack_after=[],
1404      proto=0,
1405      doc="""Store the stack top into the memo.  The stack is not popped.
1406
1407      The index of the memo location to write into is given by the newline-
1408      terminated decimal string following.  BINPUT and LONG_BINPUT are
1409      space-optimized versions.
1410      """),
1411
1412    I(name='BINPUT',
1413      code='q',
1414      arg=uint1,
1415      stack_before=[],
1416      stack_after=[],
1417      proto=1,
1418      doc="""Store the stack top into the memo.  The stack is not popped.
1419
1420      The index of the memo location to write into is given by the 1-byte
1421      unsigned integer following.
1422      """),
1423
1424    I(name='LONG_BINPUT',
1425      code='r',
1426      arg=int4,
1427      stack_before=[],
1428      stack_after=[],
1429      proto=1,
1430      doc="""Store the stack top into the memo.  The stack is not popped.
1431
1432      The index of the memo location to write into is given by the 4-byte
1433      signed little-endian integer following.
1434      """),
1435
1436    # Access the extension registry (predefined objects).  Akin to the GET
1437    # family.
1438
1439    I(name='EXT1',
1440      code='\x82',
1441      arg=uint1,
1442      stack_before=[],
1443      stack_after=[anyobject],
1444      proto=2,
1445      doc="""Extension code.
1446
1447      This code and the similar EXT2 and EXT4 allow using a registry
1448      of popular objects that are pickled by name, typically classes.
1449      It is envisioned that through a global negotiation and
1450      registration process, third parties can set up a mapping between
1451      ints and object names.
1452
1453      In order to guarantee pickle interchangeability, the extension
1454      code registry ought to be global, although a range of codes may
1455      be reserved for private use.
1456
1457      EXT1 has a 1-byte integer argument.  This is used to index into the
1458      extension registry, and the object at that index is pushed on the stack.
1459      """),
1460
1461    I(name='EXT2',
1462      code='\x83',
1463      arg=uint2,
1464      stack_before=[],
1465      stack_after=[anyobject],
1466      proto=2,
1467      doc="""Extension code.
1468
1469      See EXT1.  EXT2 has a two-byte integer argument.
1470      """),
1471
1472    I(name='EXT4',
1473      code='\x84',
1474      arg=int4,
1475      stack_before=[],
1476      stack_after=[anyobject],
1477      proto=2,
1478      doc="""Extension code.
1479
1480      See EXT1.  EXT4 has a four-byte integer argument.
1481      """),
1482
1483    # Push a class object, or module function, on the stack, via its module
1484    # and name.
1485
1486    I(name='GLOBAL',
1487      code='c',
1488      arg=stringnl_noescape_pair,
1489      stack_before=[],
1490      stack_after=[anyobject],
1491      proto=0,
1492      doc="""Push a global object (module.attr) on the stack.
1493
1494      Two newline-terminated strings follow the GLOBAL opcode.  The first is
1495      taken as a module name, and the second as a class name.  The class
1496      object module.class is pushed on the stack.  More accurately, the
1497      object returned by self.find_class(module, class) is pushed on the
1498      stack, so unpickling subclasses can override this form of lookup.
1499      """),
1500
1501    # Ways to build objects of classes pickle doesn't know about directly
1502    # (user-defined classes).  I despair of documenting this accurately
1503    # and comprehensibly -- you really have to read the pickle code to
1504    # find all the special cases.
1505
1506    I(name='REDUCE',
1507      code='R',
1508      arg=None,
1509      stack_before=[anyobject, anyobject],
1510      stack_after=[anyobject],
1511      proto=0,
1512      doc="""Push an object built from a callable and an argument tuple.
1513
1514      The opcode is named to remind of the __reduce__() method.
1515
1516      Stack before: ... callable pytuple
1517      Stack after:  ... callable(*pytuple)
1518
1519      The callable and the argument tuple are the first two items returned
1520      by a __reduce__ method.  Applying the callable to the argtuple is
1521      supposed to reproduce the original object, or at least get it started.
1522      If the __reduce__ method returns a 3-tuple, the last component is an
1523      argument to be passed to the object's __setstate__, and then the REDUCE
1524      opcode is followed by code to create setstate's argument, and then a
1525      BUILD opcode to apply  __setstate__ to that argument.
1526
1527      If type(callable) is not ClassType, REDUCE complains unless the
1528      callable has been registered with the copy_reg module's
1529      safe_co…

Large files files are truncated, but you can click here to view the full file