PageRenderTime 23ms CodeModel.GetById 10ms app.highlight 4ms RepoModel.GetById 1ms app.codeStats 0ms

/modules/libjar/appnote.txt

http://github.com/zpao/v8monkey
Plain Text | 1192 lines | 936 code | 256 blank | 0 comment | 0 complexity | a8908b2416f0147a52f20d514010b8ca MD5 | raw file
   1Revised: 03/01/1999
   2
   3Disclaimer
   4----------
   5
   6Although PKWARE will attempt to supply current and accurate
   7information relating to its file formats, algorithms, and the
   8subject programs, the possibility of error can not be eliminated.
   9PKWARE therefore expressly disclaims any warranty that the
  10information contained in the associated materials relating to the
  11subject programs and/or the format of the files created or
  12accessed by the subject programs and/or the algorithms used by
  13the subject programs, or any other matter, is current, correct or
  14accurate as delivered.  Any risk of damage due to any possible
  15inaccurate information is assumed by the user of the information.
  16Furthermore, the information relating to the subject programs
  17and/or the file formats created or accessed by the subject
  18programs and/or the algorithms used by the subject programs is
  19subject to change without notice.
  20
  21General Format of a ZIP file
  22----------------------------
  23
  24  Files stored in arbitrary order.  Large zipfiles can span multiple
  25  diskette media.
  26
  27  Overall zipfile format:
  28
  29    [local file header + file data + data_descriptor] . . .
  30    [central directory] end of central directory record
  31
  32
  33  A.  Local file header:
  34
  35        local file header signature     4 bytes  (0x04034b50)
  36        version needed to extract       2 bytes
  37        general purpose bit flag        2 bytes
  38        compression method              2 bytes
  39        last mod file time              2 bytes
  40        last mod file date              2 bytes
  41        crc-32                          4 bytes
  42        compressed size                 4 bytes
  43        uncompressed size               4 bytes
  44        filename length                 2 bytes
  45        extra field length              2 bytes
  46
  47        filename (variable size)
  48        extra field (variable size)
  49
  50  B.  Data descriptor:
  51
  52        crc-32                          4 bytes
  53        compressed size                 4 bytes
  54        uncompressed size               4 bytes
  55
  56      This descriptor exists only if bit 3 of the general
  57      purpose bit flag is set (see below).  It is byte aligned
  58      and immediately follows the last byte of compressed data.
  59      This descriptor is used only when it was not possible to
  60      seek in the output zip file, e.g., when the output zip file
  61      was standard output or a non seekable device.
  62
  63  C.  Central directory structure:
  64
  65      [file header] . . .  end of central dir record
  66
  67      File header:
  68
  69        central file header signature   4 bytes  (0x02014b50)
  70        version made by                 2 bytes
  71        version needed to extract       2 bytes
  72        general purpose bit flag        2 bytes
  73        compression method              2 bytes
  74        last mod file time              2 bytes
  75        last mod file date              2 bytes
  76        crc-32                          4 bytes
  77        compressed size                 4 bytes
  78        uncompressed size               4 bytes
  79        filename length                 2 bytes
  80        extra field length              2 bytes
  81        file comment length             2 bytes
  82        disk number start               2 bytes
  83        internal file attributes        2 bytes
  84        external file attributes        4 bytes
  85        relative offset of local header 4 bytes
  86
  87        filename (variable size)
  88        extra field (variable size)
  89        file comment (variable size)
  90
  91      End of central dir record:
  92
  93        end of central dir signature    4 bytes  (0x06054b50)
  94        number of this disk             2 bytes
  95        number of the disk with the
  96        start of the central directory  2 bytes
  97        total number of entries in
  98        the central dir on this disk    2 bytes
  99        total number of entries in
 100        the central dir                 2 bytes
 101        size of the central directory   4 bytes
 102        offset of start of central
 103        directory with respect to
 104        the starting disk number        4 bytes
 105        zipfile comment length          2 bytes
 106        zipfile comment (variable size)
 107
 108  D.  Explanation of fields:
 109
 110      version made by (2 bytes)
 111
 112          The upper byte indicates the compatibility of the file
 113          attribute information.  If the external file attributes 
 114          are compatible with MS-DOS and can be read by PKZIP for 
 115          DOS version 2.04g then this value will be zero.  If these 
 116          attributes are not compatible, then this value will 
 117          identify the host system on which the attributes are 
 118          compatible.  Software can use this information to determine
 119          the line record format for text files etc.  The current
 120          mappings are:
 121
 122          0 - MS-DOS and OS/2 (FAT / VFAT / FAT32 file systems)
 123          1 - Amiga                     2 - VAX/VMS
 124          3 - Unix                      4 - VM/CMS
 125          5 - Atari ST                  6 - OS/2 H.P.F.S.
 126          7 - Macintosh                 8 - Z-System
 127          9 - CP/M                     10 - Windows NTFS
 128         11 thru 255 - unused
 129
 130          The lower byte indicates the version number of the
 131          software used to encode the file.  The value/10
 132          indicates the major version number, and the value
 133          mod 10 is the minor version number.
 134
 135      version needed to extract (2 bytes)
 136
 137          The minimum software version needed to extract the
 138          file, mapped as above.
 139
 140      general purpose bit flag: (2 bytes)
 141
 142          Bit 0: If set, indicates that the file is encrypted.
 143
 144          (For Method 6 - Imploding)
 145          Bit 1: If the compression method used was type 6,
 146                 Imploding, then this bit, if set, indicates
 147                 an 8K sliding dictionary was used.  If clear,
 148                 then a 4K sliding dictionary was used.
 149          Bit 2: If the compression method used was type 6,
 150                 Imploding, then this bit, if set, indicates
 151                 3 Shannon-Fano trees were used to encode the
 152                 sliding dictionary output.  If clear, then 2
 153                 Shannon-Fano trees were used.
 154
 155          (For Method 8 - Deflating)
 156          Bit 2  Bit 1
 157            0      0    Normal (-en) compression option was used.
 158            0      1    Maximum (-ex) compression option was used.
 159            1      0    Fast (-ef) compression option was used.
 160            1      1    Super Fast (-es) compression option was used.
 161
 162          Note:  Bits 1 and 2 are undefined if the compression
 163                 method is any other.
 164
 165          Bit 3: If this bit is set, the fields crc-32, compressed 
 166                 size and uncompressed size are set to zero in the 
 167                 local header.  The correct values are put in the 
 168                 data descriptor immediately following the compressed
 169                 data.  (Note: PKZIP version 2.04g for DOS only 
 170                 recognizes this bit for method 8 compression, newer 
 171                 versions of PKZIP recognize this bit for any 
 172                 compression method.)
 173
 174          Bit 4: Reserved for use with method 8, for enhanced
 175                 deflating. 
 176
 177          Bit 5: If this bit is set, this indicates that the file is 
 178                 compressed patched data.  (Note: Requires PKZIP 
 179                 version 2.70 or greater)
 180
 181          Bit 6: Currently unused.
 182
 183          Bit 7: Currently unused.
 184
 185          Bit 8: Currently unused.
 186
 187          Bit 9: Currently unused.
 188
 189          Bit 10: Currently unused.
 190
 191          Bit 11: Currently unused.
 192
 193          Bit 12: Reserved by PKWARE for enhanced compression.
 194
 195          Bit 13: Reserved by PKWARE.
 196
 197          Bit 14: Reserved by PKWARE.
 198
 199          Bit 15: Reserved by PKWARE.
 200
 201      compression method: (2 bytes)
 202
 203          (see accompanying documentation for algorithm
 204          descriptions)
 205
 206          0 - The file is stored (no compression)
 207          1 - The file is Shrunk
 208          2 - The file is Reduced with compression factor 1
 209          3 - The file is Reduced with compression factor 2
 210          4 - The file is Reduced with compression factor 3
 211          5 - The file is Reduced with compression factor 4
 212          6 - The file is Imploded
 213          7 - Reserved for Tokenizing compression algorithm
 214          8 - The file is Deflated
 215          9 - Reserved for enhanced Deflating
 216         10 - PKWARE Date Compression Library Imploding
 217
 218      date and time fields: (2 bytes each)
 219
 220          The date and time are encoded in standard MS-DOS format.
 221          If input came from standard input, the date and time are
 222          those at which compression was started for this data.
 223
 224      CRC-32: (4 bytes)
 225
 226          The CRC-32 algorithm was generously contributed by
 227          David Schwaderer and can be found in his excellent
 228          book "C Programmers Guide to NetBIOS" published by
 229          Howard W. Sams & Co. Inc.  The 'magic number' for
 230          the CRC is 0xdebb20e3.  The proper CRC pre and post
 231          conditioning is used, meaning that the CRC register
 232          is pre-conditioned with all ones (a starting value
 233          of 0xffffffff) and the value is post-conditioned by
 234          taking the one's complement of the CRC residual.
 235          If bit 3 of the general purpose flag is set, this
 236          field is set to zero in the local header and the correct
 237          value is put in the data descriptor and in the central
 238          directory.
 239
 240      compressed size: (4 bytes)
 241      uncompressed size: (4 bytes)
 242
 243          The size of the file compressed and uncompressed,
 244          respectively.  If bit 3 of the general purpose bit flag
 245          is set, these fields are set to zero in the local header
 246          and the correct values are put in the data descriptor and
 247          in the central directory.
 248
 249      filename length: (2 bytes)
 250      extra field length: (2 bytes)
 251      file comment length: (2 bytes)
 252
 253          The length of the filename, extra field, and comment
 254          fields respectively.  The combined length of any
 255          directory record and these three fields should not
 256          generally exceed 65,535 bytes.  If input came from standard
 257          input, the filename length is set to zero.
 258
 259      disk number start: (2 bytes)
 260
 261          The number of the disk on which this file begins.
 262
 263      internal file attributes: (2 bytes)
 264
 265          The lowest bit of this field indicates, if set, that
 266          the file is apparently an ASCII or text file.  If not
 267          set, that the file apparently contains binary data.
 268          The remaining bits are unused in version 1.0.
 269
 270          Bits 1 and 2 are reserved for use by PKWARE.
 271
 272      external file attributes: (4 bytes)
 273
 274          The mapping of the external attributes is
 275          host-system dependent (see 'version made by').  For
 276          MS-DOS, the low order byte is the MS-DOS directory
 277          attribute byte.  If input came from standard input, this
 278          field is set to zero.
 279
 280      relative offset of local header: (4 bytes)
 281
 282          This is the offset from the start of the first disk on
 283          which this file appears, to where the local header should
 284          be found.
 285
 286      filename: (Variable)
 287
 288          The name of the file, with optional relative path.
 289          The path stored should not contain a drive or
 290          device letter, or a leading slash.  All slashes
 291          should be forward slashes '/' as opposed to
 292          backwards slashes '\' for compatibility with Amiga
 293          and Unix file systems etc.  If input came from standard
 294          input, there is no filename field.
 295
 296      extra field: (Variable)
 297
 298          This is for future expansion.  If additional information
 299          needs to be stored in the future, it should be stored
 300          here.  Earlier versions of the software can then safely
 301          skip this file, and find the next file or header.  This
 302          field will be 0 length in version 1.0.
 303
 304          In order to allow different programs and different types
 305          of information to be stored in the 'extra' field in .ZIP
 306          files, the following structure should be used for all
 307          programs storing data in this field:
 308
 309          header1+data1 + header2+data2 . . .
 310
 311          Each header should consist of:
 312
 313            Header ID - 2 bytes
 314            Data Size - 2 bytes
 315
 316          Note: all fields stored in Intel low-byte/high-byte order.
 317
 318          The Header ID field indicates the type of data that is in
 319          the following data block.
 320
 321          Header ID's of 0 thru 31 are reserved for use by PKWARE.
 322          The remaining ID's can be used by third party vendors for
 323          proprietary usage.
 324
 325          The current Header ID mappings defined by PKWARE are:
 326
 327          0x0007        AV Info
 328          0x0009        OS/2
 329          0x000a        NTFS 
 330          0x000c        VAX/VMS
 331          0x000d        Unix
 332          0x000f        Patch Descriptor
 333
 334          Several third party mappings commonly used are:
 335
 336          0x4b46        FWKCS MD5 (see below)
 337          0x07c8        Macintosh
 338          0x4341        Acorn/SparkFS 
 339          0x4453        Windows NT security descriptor (binary ACL)
 340          0x4704        VM/CMS
 341          0x470f        MVS
 342          0x4c41        OS/2 access control list (text ACL)
 343          0x4d49        Info-ZIP VMS (VAX or Alpha)
 344          0x5455        extended timestamp
 345          0x5855        Info-ZIP Unix (original, also OS/2, NT, etc)
 346          0x6542        BeOS/BeBox
 347          0x756e        ASi Unix
 348          0x7855        Info-ZIP Unix (new)
 349          0xfd4a        SMS/QDOS
 350
 351          The Data Size field indicates the size of the following
 352          data block. Programs can use this value to skip to the
 353          next header block, passing over any data blocks that are
 354          not of interest.
 355
 356          Note: As stated above, the size of the entire .ZIP file
 357                header, including the filename, comment, and extra
 358                field should not exceed 64K in size.
 359
 360          In case two different programs should appropriate the same
 361          Header ID value, it is strongly recommended that each
 362          program place a unique signature of at least two bytes in
 363          size (and preferably 4 bytes or bigger) at the start of
 364          each data area.  Every program should verify that its
 365          unique signature is present, in addition to the Header ID
 366          value being correct, before assuming that it is a block of
 367          known type.
 368
 369         -OS/2 Extra Field:
 370
 371          The following is the layout of the OS/2 attributes "extra" 
 372          block.  (Last Revision  09/05/95)
 373
 374          Note: all fields stored in Intel low-byte/high-byte order.
 375
 376          Value       Size          Description
 377          -----       ----          -----------
 378  (OS/2)  0x0009      2 bytes       Tag for this "extra" block type
 379          TSize       2 bytes       Size for the following data block
 380          BSize       4 bytes       Uncompressed Block Size
 381          CType       2 bytes       Compression type
 382          EACRC       4 bytes       CRC value for uncompress block
 383          (var)       variable      Compressed block
 384
 385        The OS/2 extended attribute structure (FEA2LIST) is 
 386        compressed and then stored in it's entirety within this 
 387        structure.  There will only ever be one "block" of data in 
 388        VarFields[].
 389
 390         -UNIX Extra Field:
 391
 392          The following is the layout of the Unix "extra" block.
 393          Note: all fields are stored in Intel low-byte/high-byte 
 394          order.
 395
 396          Value       Size          Description
 397          -----       ----          -----------
 398  (UNIX)  0x000d      2 bytes       Tag for this "extra" block type
 399          TSize       2 bytes       Size for the following data block
 400          Atime       4 bytes       File last access time
 401          Mtime       4 bytes       File last modification time
 402          Uid         2 bytes       File user ID
 403          Gid         2 bytes       File group ID
 404          (var)       variable      Variable length data field
 405
 406          The variable length data field will contain file type 
 407          specific data.  Currently the only values allowed are
 408          the original "linked to" file names for hard or symbolic 
 409          links.
 410
 411         -VAX/VMS Extra Field:
 412
 413          The following is the layout of the VAX/VMS attributes 
 414          "extra" block.
 415
 416          Note: all fields stored in Intel low-byte/high-byte order.
 417
 418          Value      Size       Description
 419          -----      ----       -----------
 420  (VMS)   0x000c     2 bytes    Tag for this "extra" block type
 421          TSize      2 bytes    Size of the total "extra" block
 422          CRC        4 bytes    32-bit CRC for remainder of the block
 423          Tag1       2 bytes    VMS attribute tag value #1
 424          Size1      2 bytes    Size of attribute #1, in bytes
 425          (var.)     Size1      Attribute #1 data
 426          .
 427          .
 428          .
 429          TagN       2 bytes    VMS attribute tage value #N
 430          SizeN      2 bytes    Size of attribute #N, in bytes
 431          (var.)     SizeN      Attribute #N data
 432
 433          Rules:
 434
 435          1. There will be one or more of attributes present, which 
 436             will each be preceded by the above TagX & SizeX values.  
 437             These values are identical to the ATR$C_XXXX and 
 438             ATR$S_XXXX constants which are defined in ATR.H under 
 439             VMS C.  Neither of these values will ever be zero.
 440
 441          2. No word alignment or padding is performed.
 442
 443          3. A well-behaved PKZIP/VMS program should never produce
 444             more than one sub-block with the same TagX value.  Also,
 445             there will never be more than one "extra" block of type
 446             0x000c in a particular directory record.
 447
 448         -NTFS Extra Field:
 449
 450          The following is the layout of the NTFS attributes 
 451          "extra" block.
 452
 453          Note: all fields stored in Intel low-byte/high-byte order.
 454
 455          Value      Size       Description
 456          -----      ----       -----------
 457  (NTFS)  0x000a     2 bytes    Tag for this "extra" block type
 458          TSize      2 bytes    Size of the total "extra" block
 459          Reserved   4 bytes    Reserved for future use
 460          Tag1       2 bytes    NTFS attribute tag value #1
 461          Size1      2 bytes    Size of attribute #1, in bytes
 462          (var.)     Size1      Attribute #1 data
 463          .
 464          .
 465          .
 466          TagN       2 bytes    NTFS attribute tage value #N
 467          SizeN      2 bytes    Size of attribute #N, in bytes
 468          (var.)     SizeN      Attribute #N data
 469
 470          For NTFS, values for Tag1 through TagN are as follows:
 471          (currently only one set of attributes is defined for NTFS)
 472
 473          Tag        Size       Description
 474          -----      ----       -----------
 475          0x0001     2 bytes    Tag for attribute #1 
 476          Size1      2 bytes    Size of attribute #1, in bytes
 477          Mtime      8 bytes    File last modification time
 478          Atime      8 bytes    File last access time
 479          Ctime      8 bytes    File creation time
 480          
 481         -PATCH Descriptor Extra Field:
 482
 483          The following is the layout of the Patch Descriptor "extra"
 484          block.
 485
 486          Note: all fields stored in Intel low-byte/high-byte order.
 487
 488          Value     Size     Description
 489          -----     ----     -----------
 490  (Patch) 0x000f    2 bytes  Tag for this "extra" block type
 491          TSize     2 bytes  Size of the total "extra" block
 492          Version   2 bytes  Version of the descriptor
 493          Flags     4 bytes  Actions and reactions (see below) 
 494          OldSize   4 bytes  Size of the file about to be patched 
 495          OldCRC    4 bytes  32-bit CRC of the file to be patched 
 496          NewSize   4 bytes  Size of the resulting file 
 497          NewCRC    4 bytes  32-bit CRC of the resulting file 
 498
 499          Actions and reactions
 500
 501          Bits          Description
 502          ----          ----------------
 503          0             Use for autodetection
 504          1             Treat as selfpatch
 505          2-3           RESERVED
 506          4-5           Action (see below)
 507          6-7           RESERVED
 508          8-9           Reaction (see below) to absent file 
 509          10-11         Reaction (see below) to newer file
 510          12-13         Reaction (see below) to unknown file
 511          14-15         RESERVED
 512          16-31         RESERVED
 513
 514          Actions
 515
 516          Action       Value
 517          ------       ----- 
 518          none         0
 519          add          1
 520          delete       2
 521          patch        3
 522
 523          Reactions
 524 
 525          Reaction     Value
 526          --------     -----
 527          ask          0
 528          skip         1
 529          ignore       2
 530          fail         3
 531
 532          - FWKCS MD5 Extra Field:
 533
 534          The FWKCS Contents_Signature System, used in
 535          automatically identifying files independent of filename,
 536          optionally adds and uses an extra field to support the
 537          rapid creation of an enhanced contents_signature:
 538
 539              Header ID = 0x4b46
 540              Data Size = 0x0013
 541              Preface   = 'M','D','5'
 542              followed by 16 bytes containing the uncompressed file's
 543              128_bit MD5 hash(1), low byte first.
 544
 545          When FWKCS revises a zipfile central directory to add
 546          this extra field for a file, it also replaces the
 547          central directory entry for that file's uncompressed
 548          filelength with a measured value.
 549
 550          FWKCS provides an option to strip this extra field, if
 551          present, from a zipfile central directory. In adding
 552          this extra field, FWKCS preserves Zipfile Authenticity
 553          Verification; if stripping this extra field, FWKCS
 554          preserves all versions of AV through PKZIP version 2.04g.
 555
 556          FWKCS, and FWKCS Contents_Signature System, are
 557          trademarks of Frederick W. Kantor.
 558
 559          (1) R. Rivest, RFC1321.TXT, MIT Laboratory for Computer
 560              Science and RSA Data Security, Inc., April 1992.
 561              ll.76-77: "The MD5 algorithm is being placed in the
 562              public domain for review and possible adoption as a
 563              standard."
 564
 565      file comment: (Variable)
 566
 567          The comment for this file.
 568
 569      number of this disk: (2 bytes)
 570
 571          The number of this disk, which contains central
 572          directory end record.
 573
 574      number of the disk with the start of the central
 575      directory: (2 bytes)
 576
 577          The number of the disk on which the central
 578          directory starts.
 579
 580      total number of entries in the central dir on 
 581      this disk: (2 bytes)
 582
 583          The number of central directory entries on this disk.
 584
 585      total number of entries in the central dir: (2 bytes)
 586
 587          The total number of files in the zipfile.
 588
 589      size of the central directory: (4 bytes)
 590
 591          The size (in bytes) of the entire central directory.
 592
 593      offset of start of central directory with respect to
 594      the starting disk number:  (4 bytes)
 595
 596          Offset of the start of the central directory on the
 597          disk on which the central directory starts.
 598
 599      zipfile comment length: (2 bytes)
 600
 601          The length of the comment for this zipfile.
 602
 603      zipfile comment: (Variable)
 604
 605          The comment for this zipfile.
 606
 607  D.  General notes:
 608
 609      1)  All fields unless otherwise noted are unsigned and stored
 610          in Intel low-byte:high-byte, low-word:high-word order.
 611
 612      2)  String fields are not null terminated, since the
 613          length is given explicitly.
 614
 615      3)  Local headers should not span disk boundaries.  Also, even
 616          though the central directory can span disk boundaries, no
 617          single record in the central directory should be split
 618          across disks.
 619
 620      4)  The entries in the central directory may not necessarily
 621          be in the same order that files appear in the zipfile.
 622
 623UnShrinking - Method 1
 624----------------------
 625
 626Shrinking is a Dynamic Ziv-Lempel-Welch compression algorithm
 627with partial clearing.  The initial code size is 9 bits, and
 628the maximum code size is 13 bits.  Shrinking differs from
 629conventional Dynamic Ziv-Lempel-Welch implementations in several
 630respects:
 631
 6321)  The code size is controlled by the compressor, and is not
 633    automatically increased when codes larger than the current
 634    code size are created (but not necessarily used).  When
 635    the decompressor encounters the code sequence 256
 636    (decimal) followed by 1, it should increase the code size
 637    read from the input stream to the next bit size.  No
 638    blocking of the codes is performed, so the next code at
 639    the increased size should be read from the input stream
 640    immediately after where the previous code at the smaller
 641    bit size was read.  Again, the decompressor should not
 642    increase the code size used until the sequence 256,1 is
 643    encountered.
 644
 6452)  When the table becomes full, total clearing is not
 646    performed.  Rather, when the compressor emits the code
 647    sequence 256,2 (decimal), the decompressor should clear
 648    all leaf nodes from the Ziv-Lempel tree, and continue to
 649    use the current code size.  The nodes that are cleared
 650    from the Ziv-Lempel tree are then re-used, with the lowest
 651    code value re-used first, and the highest code value
 652    re-used last.  The compressor can emit the sequence 256,2
 653    at any time.
 654
 655Expanding - Methods 2-5
 656-----------------------
 657
 658The Reducing algorithm is actually a combination of two
 659distinct algorithms.  The first algorithm compresses repeated
 660byte sequences, and the second algorithm takes the compressed
 661stream from the first algorithm and applies a probabilistic
 662compression method.
 663
 664The probabilistic compression stores an array of 'follower
 665sets' S(j), for j=0 to 255, corresponding to each possible
 666ASCII character.  Each set contains between 0 and 32
 667characters, to be denoted as S(j)[0],...,S(j)[m], where m<32.
 668The sets are stored at the beginning of the data area for a
 669Reduced file, in reverse order, with S(255) first, and S(0)
 670last.
 671
 672The sets are encoded as { N(j), S(j)[0],...,S(j)[N(j)-1] },
 673where N(j) is the size of set S(j).  N(j) can be 0, in which
 674case the follower set for S(j) is empty.  Each N(j) value is
 675encoded in 6 bits, followed by N(j) eight bit character values
 676corresponding to S(j)[0] to S(j)[N(j)-1] respectively.  If
 677N(j) is 0, then no values for S(j) are stored, and the value
 678for N(j-1) immediately follows.
 679
 680Immediately after the follower sets, is the compressed data
 681stream.  The compressed data stream can be interpreted for the
 682probabilistic decompression as follows:
 683
 684let Last-Character <- 0.
 685loop until done
 686    if the follower set S(Last-Character) is empty then
 687        read 8 bits from the input stream, and copy this
 688        value to the output stream.
 689    otherwise if the follower set S(Last-Character) is non-empty then
 690        read 1 bit from the input stream.
 691        if this bit is not zero then
 692            read 8 bits from the input stream, and copy this
 693            value to the output stream.
 694        otherwise if this bit is zero then
 695            read B(N(Last-Character)) bits from the input
 696            stream, and assign this value to I.
 697            Copy the value of S(Last-Character)[I] to the
 698            output stream.
 699
 700    assign the last value placed on the output stream to
 701    Last-Character.
 702end loop
 703
 704B(N(j)) is defined as the minimal number of bits required to
 705encode the value N(j)-1.
 706
 707The decompressed stream from above can then be expanded to
 708re-create the original file as follows:
 709
 710let State <- 0.
 711
 712loop until done
 713    read 8 bits from the input stream into C.
 714    case State of
 715        0:  if C is not equal to DLE (144 decimal) then
 716                copy C to the output stream.
 717            otherwise if C is equal to DLE then
 718                let State <- 1.
 719
 720        1:  if C is non-zero then
 721                let V <- C.
 722                let Len <- L(V)
 723                let State <- F(Len).
 724            otherwise if C is zero then
 725                copy the value 144 (decimal) to the output stream.
 726                let State <- 0
 727
 728        2:  let Len <- Len + C
 729            let State <- 3.
 730
 731        3:  move backwards D(V,C) bytes in the output stream
 732            (if this position is before the start of the output
 733            stream, then assume that all the data before the
 734            start of the output stream is filled with zeros).
 735            copy Len+3 bytes from this position to the output stream.
 736            let State <- 0.
 737    end case
 738end loop
 739
 740The functions F,L, and D are dependent on the 'compression
 741factor', 1 through 4, and are defined as follows:
 742
 743For compression factor 1:
 744    L(X) equals the lower 7 bits of X.
 745    F(X) equals 2 if X equals 127 otherwise F(X) equals 3.
 746    D(X,Y) equals the (upper 1 bit of X) * 256 + Y + 1.
 747For compression factor 2:
 748    L(X) equals the lower 6 bits of X.
 749    F(X) equals 2 if X equals 63 otherwise F(X) equals 3.
 750    D(X,Y) equals the (upper 2 bits of X) * 256 + Y + 1.
 751For compression factor 3:
 752    L(X) equals the lower 5 bits of X.
 753    F(X) equals 2 if X equals 31 otherwise F(X) equals 3.
 754    D(X,Y) equals the (upper 3 bits of X) * 256 + Y + 1.
 755For compression factor 4:
 756    L(X) equals the lower 4 bits of X.
 757    F(X) equals 2 if X equals 15 otherwise F(X) equals 3.
 758    D(X,Y) equals the (upper 4 bits of X) * 256 + Y + 1.
 759
 760Imploding - Method 6
 761--------------------
 762
 763The Imploding algorithm is actually a combination of two distinct
 764algorithms.  The first algorithm compresses repeated byte
 765sequences using a sliding dictionary.  The second algorithm is
 766used to compress the encoding of the sliding dictionary output,
 767using multiple Shannon-Fano trees.
 768
 769The Imploding algorithm can use a 4K or 8K sliding dictionary
 770size. The dictionary size used can be determined by bit 1 in the
 771general purpose flag word; a 0 bit indicates a 4K dictionary
 772while a 1 bit indicates an 8K dictionary.
 773
 774The Shannon-Fano trees are stored at the start of the compressed
 775file. The number of trees stored is defined by bit 2 in the
 776general purpose flag word; a 0 bit indicates two trees stored, a
 7771 bit indicates three trees are stored.  If 3 trees are stored,
 778the first Shannon-Fano tree represents the encoding of the
 779Literal characters, the second tree represents the encoding of
 780the Length information, the third represents the encoding of the
 781Distance information.  When 2 Shannon-Fano trees are stored, the
 782Length tree is stored first, followed by the Distance tree.
 783
 784The Literal Shannon-Fano tree, if present is used to represent
 785the entire ASCII character set, and contains 256 values.  This
 786tree is used to compress any data not compressed by the sliding
 787dictionary algorithm.  When this tree is present, the Minimum
 788Match Length for the sliding dictionary is 3.  If this tree is
 789not present, the Minimum Match Length is 2.
 790
 791The Length Shannon-Fano tree is used to compress the Length part
 792of the (length,distance) pairs from the sliding dictionary
 793output.  The Length tree contains 64 values, ranging from the
 794Minimum Match Length, to 63 plus the Minimum Match Length.
 795
 796The Distance Shannon-Fano tree is used to compress the Distance
 797part of the (length,distance) pairs from the sliding dictionary
 798output. The Distance tree contains 64 values, ranging from 0 to
 79963, representing the upper 6 bits of the distance value.  The
 800distance values themselves will be between 0 and the sliding
 801dictionary size, either 4K or 8K.
 802
 803The Shannon-Fano trees themselves are stored in a compressed
 804format. The first byte of the tree data represents the number of
 805bytes of data representing the (compressed) Shannon-Fano tree
 806minus 1.  The remaining bytes represent the Shannon-Fano tree
 807data encoded as:
 808
 809    High 4 bits: Number of values at this bit length + 1. (1 - 16)
 810    Low  4 bits: Bit Length needed to represent value + 1. (1 - 16)
 811
 812The Shannon-Fano codes can be constructed from the bit lengths
 813using the following algorithm:
 814
 8151)  Sort the Bit Lengths in ascending order, while retaining the
 816    order of the original lengths stored in the file.
 817
 8182)  Generate the Shannon-Fano trees:
 819
 820    Code <- 0
 821    CodeIncrement <- 0
 822    LastBitLength <- 0
 823    i <- number of Shannon-Fano codes - 1   (either 255 or 63)
 824
 825    loop while i >= 0
 826        Code = Code + CodeIncrement
 827        if BitLength(i) <> LastBitLength then
 828            LastBitLength=BitLength(i)
 829            CodeIncrement = 1 shifted left (16 - LastBitLength)
 830        ShannonCode(i) = Code
 831        i <- i - 1
 832    end loop
 833
 8343)  Reverse the order of all the bits in the above ShannonCode()
 835    vector, so that the most significant bit becomes the least
 836    significant bit.  For example, the value 0x1234 (hex) would
 837    become 0x2C48 (hex).
 838
 8394)  Restore the order of Shannon-Fano codes as originally stored
 840    within the file.
 841
 842Example:
 843
 844    This example will show the encoding of a Shannon-Fano tree
 845    of size 8.  Notice that the actual Shannon-Fano trees used
 846    for Imploding are either 64 or 256 entries in size.
 847
 848Example:   0x02, 0x42, 0x01, 0x13
 849
 850    The first byte indicates 3 values in this table.  Decoding the
 851    bytes:
 852            0x42 = 5 codes of 3 bits long
 853            0x01 = 1 code  of 2 bits long
 854            0x13 = 2 codes of 4 bits long
 855
 856    This would generate the original bit length array of:
 857    (3, 3, 3, 3, 3, 2, 4, 4)
 858
 859    There are 8 codes in this table for the values 0 thru 7.  Using 
 860    the algorithm to obtain the Shannon-Fano codes produces:
 861
 862                                  Reversed     Order     Original
 863Val  Sorted   Constructed Code      Value     Restored    Length
 864---  ------   -----------------   --------    --------    ------
 8650:     2      1100000000000000        11       101          3
 8661:     3      1010000000000000       101       001          3
 8672:     3      1000000000000000       001       110          3
 8683:     3      0110000000000000       110       010          3
 8694:     3      0100000000000000       010       100          3
 8705:     3      0010000000000000       100        11          2
 8716:     4      0001000000000000      1000      1000          4
 8727:     4      0000000000000000      0000      0000          4
 873
 874The values in the Val, Order Restored and Original Length columns
 875now represent the Shannon-Fano encoding tree that can be used for
 876decoding the Shannon-Fano encoded data.  How to parse the
 877variable length Shannon-Fano values from the data stream is beyond
 878the scope of this document.  (See the references listed at the end of
 879this document for more information.)  However, traditional decoding
 880schemes used for Huffman variable length decoding, such as the
 881Greenlaw algorithm, can be successfully applied.
 882
 883The compressed data stream begins immediately after the
 884compressed Shannon-Fano data.  The compressed data stream can be
 885interpreted as follows:
 886
 887loop until done
 888    read 1 bit from input stream.
 889
 890    if this bit is non-zero then       (encoded data is literal data)
 891        if Literal Shannon-Fano tree is present
 892            read and decode character using Literal Shannon-Fano tree.
 893        otherwise
 894            read 8 bits from input stream.
 895        copy character to the output stream.
 896    otherwise              (encoded data is sliding dictionary match)
 897        if 8K dictionary size
 898            read 7 bits for offset Distance (lower 7 bits of offset).
 899        otherwise
 900            read 6 bits for offset Distance (lower 6 bits of offset).
 901
 902        using the Distance Shannon-Fano tree, read and decode the
 903          upper 6 bits of the Distance value.
 904
 905        using the Length Shannon-Fano tree, read and decode
 906          the Length value.
 907
 908        Length <- Length + Minimum Match Length
 909
 910        if Length = 63 + Minimum Match Length
 911            read 8 bits from the input stream,
 912            add this value to Length.
 913
 914        move backwards Distance+1 bytes in the output stream, and
 915        copy Length characters from this position to the output
 916        stream.  (if this position is before the start of the output
 917        stream, then assume that all the data before the start of
 918        the output stream is filled with zeros).
 919end loop
 920
 921Tokenizing - Method 7
 922--------------------
 923
 924This method is not used by PKZIP.
 925
 926Deflating - Method 8
 927-----------------
 928
 929The Deflate algorithm is similar to the Implode algorithm using
 930a sliding dictionary of up to 32K with secondary compression
 931from Huffman/Shannon-Fano codes.
 932
 933The compressed data is stored in blocks with a header describing
 934the block and the Huffman codes used in the data block.  The header
 935format is as follows:
 936
 937   Bit 0: Last Block bit     This bit is set to 1 if this is the last
 938                             compressed block in the data.
 939   Bits 1-2: Block type
 940      00 (0) - Block is stored - All stored data is byte aligned.
 941               Skip bits until next byte, then next word = block 
 942               length, followed by the ones compliment of the block
 943               length word. Remaining data in block is the stored 
 944               data.
 945
 946      01 (1) - Use fixed Huffman codes for literal and distance codes.
 947               Lit Code    Bits             Dist Code   Bits
 948               ---------   ----             ---------   ----
 949                 0 - 143    8                 0 - 31      5
 950               144 - 255    9
 951               256 - 279    7
 952               280 - 287    8
 953
 954               Literal codes 286-287 and distance codes 30-31 are 
 955               never used but participate in the huffman construction.
 956
 957      10 (2) - Dynamic Huffman codes.  (See expanding Huffman codes)
 958
 959      11 (3) - Reserved - Flag a "Error in compressed data" if seen.
 960
 961Expanding Huffman Codes
 962-----------------------
 963If the data block is stored with dynamic Huffman codes, the Huffman
 964codes are sent in the following compressed format:
 965
 966   5 Bits: # of Literal codes sent - 256 (256 - 286)
 967           All other codes are never sent.
 968   5 Bits: # of Dist codes - 1           (1 - 32)
 969   4 Bits: # of Bit Length codes - 3     (3 - 19)
 970
 971The Huffman codes are sent as bit lengths and the codes are built as
 972described in the implode algorithm.  The bit lengths themselves are
 973compressed with Huffman codes.  There are 19 bit length codes:
 974
 975   0 - 15: Represent bit lengths of 0 - 15
 976       16: Copy the previous bit length 3 - 6 times.
 977           The next 2 bits indicate repeat length (0 = 3, ... ,3 = 6)
 978              Example:  Codes 8, 16 (+2 bits 11), 16 (+2 bits 10) will
 979                        expand to 12 bit lengths of 8 (1 + 6 + 5)
 980       17: Repeat a bit length of 0 for 3 - 10 times. (3 bits of length)
 981       18: Repeat a bit length of 0 for 11 - 138 times (7 bits of length)
 982
 983The lengths of the bit length codes are sent packed 3 bits per value
 984(0 - 7) in the following order:
 985
 986   16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15
 987
 988The Huffman codes should be built as described in the Implode algorithm
 989except codes are assigned starting at the shortest bit length, i.e. the
 990shortest code should be all 0's rather than all 1's.  Also, codes with
 991a bit length of zero do not participate in the tree construction.  The
 992codes are then used to decode the bit lengths for the literal and 
 993distance tables.
 994
 995The bit lengths for the literal tables are sent first with the number
 996of entries sent described by the 5 bits sent earlier.  There are up
 997to 286 literal characters; the first 256 represent the respective 8
 998bit character, code 256 represents the End-Of-Block code, the remaining
 99929 codes represent copy lengths of 3 thru 258.  There are up to 30
1000distance codes representing distances from 1 thru 32k as described
1001below.
1002
1003                             Length Codes
1004                             ------------
1005      Extra             Extra              Extra              Extra
1006 Code Bits Length  Code Bits Lengths  Code Bits Lengths  Code Bits Length(s)
1007 ---- ---- ------  ---- ---- -------  ---- ---- -------  ---- ---- ---------
1008  257   0     3     265   1   11,12    273   3   35-42    281   5  131-162
1009  258   0     4     266   1   13,14    274   3   43-50    282   5  163-194
1010  259   0     5     267   1   15,16    275   3   51-58    283   5  195-226
1011  260   0     6     268   1   17,18    276   3   59-66    284   5  227-257
1012  261   0     7     269   2   19-22    277   4   67-82    285   0    258
1013  262   0     8     270   2   23-26    278   4   83-98
1014  263   0     9     271   2   27-30    279   4   99-114
1015  264   0    10     272   2   31-34    280   4  115-130
1016
1017                            Distance Codes
1018                            --------------
1019      Extra           Extra             Extra               Extra
1020 Code Bits Dist  Code Bits  Dist   Code Bits Distance  Code Bits Distance
1021 ---- ---- ----  ---- ---- ------  ---- ---- --------  ---- ---- --------
1022   0   0    1      8   3   17-24    16    7  257-384    24   11  4097-6144
1023   1   0    2      9   3   25-32    17    7  385-512    25   11  6145-8192
1024   2   0    3     10   4   33-48    18    8  513-768    26   12  8193-12288
1025   3   0    4     11   4   49-64    19    8  769-1024   27   12 12289-16384
1026   4   1   5,6    12   5   65-96    20    9 1025-1536   28   13 16385-24576
1027   5   1   7,8    13   5   97-128   21    9 1537-2048   29   13 24577-32768
1028   6   2   9-12   14   6  129-192   22   10 2049-3072
1029   7   2  13-16   15   6  193-256   23   10 3073-4096
1030
1031The compressed data stream begins immediately after the
1032compressed header data.  The compressed data stream can be
1033interpreted as follows:
1034
1035do
1036   read header from input stream.
1037
1038   if stored block
1039      skip bits until byte aligned
1040      read count and 1's compliment of count
1041      copy count bytes data block
1042   otherwise
1043      loop until end of block code sent
1044         decode literal character from input stream
1045         if literal < 256
1046            copy character to the output stream
1047         otherwise
1048            if literal = end of block
1049               break from loop
1050            otherwise
1051               decode distance from input stream
1052
1053               move backwards distance bytes in the output stream, and
1054               copy length characters from this position to the output
1055               stream.
1056      end loop
1057while not last block
1058
1059if data descriptor exists
1060   skip bits until byte aligned
1061   read crc and sizes
1062endif
1063
1064Decryption
1065----------
1066
1067The encryption used in PKZIP was generously supplied by Roger
1068Schlafly.  PKWARE is grateful to Mr. Schlafly for his expert
1069help and advice in the field of data encryption.
1070
1071PKZIP encrypts the compressed data stream.  Encrypted files must
1072be decrypted before they can be extracted.
1073
1074Each encrypted file has an extra 12 bytes stored at the start of
1075the data area defining the encryption header for that file.  The
1076encryption header is originally set to random values, and then
1077itself encrypted, using three, 32-bit keys.  The key values are
1078initialized using the supplied encryption password.  After each byte
1079is encrypted, the keys are then updated using pseudo-random number
1080generation techniques in combination with the same CRC-32 algorithm
1081used in PKZIP and described elsewhere in this document.
1082
1083The following is the basic steps required to decrypt a file:
1084
10851) Initialize the three 32-bit keys with the password.
10862) Read and decrypt the 12-byte encryption header, further
1087   initializing the encryption keys.
10883) Read and decrypt the compressed data stream using the
1089   encryption keys.
1090
1091Step 1 - Initializing the encryption keys
1092-----------------------------------------
1093
1094Key(0) <- 305419896
1095Key(1) <- 591751049
1096Key(2) <- 878082192
1097
1098loop for i <- 0 to length(password)-1
1099    update_keys(password(i))
1100end loop
1101
1102Where update_keys() is defined as:
1103
1104update_keys(char):
1105  Key(0) <- crc32(key(0),char)
1106  Key(1) <- Key(1) + (Key(0) & 000000ffH)
1107  Key(1) <- Key(1) * 134775813 + 1
1108  Key(2) <- crc32(key(2),key(1) >> 24)
1109end update_keys
1110
1111Where crc32(old_crc,char) is a routine that given a CRC value and a
1112character, returns an updated CRC value after applying the CRC-32
1113algorithm described elsewhere in this document.
1114
1115Step 2 - Decrypting the encryption header
1116-----------------------------------------
1117
1118The purpose of this step is to further initialize the encryption
1119keys, based on random data, to render a plaintext attack on the
1120data ineffective.
1121
1122Read the 12-byte encryption header into Buffer, in locations
1123Buffer(0) thru Buffer(11).
1124
1125loop for i <- 0 to 11
1126    C <- buffer(i) ^ decrypt_byte()
1127    update_keys(C)
1128    buffer(i) <- C
1129end loop
1130
1131Where decrypt_byte() is defined as:
1132
1133unsigned char decrypt_byte()
1134    local unsigned short temp
1135    temp <- Key(2) | 2
1136    decrypt_byte <- (temp * (temp ^ 1)) >> 8
1137end decrypt_byte
1138
1139After the header is decrypted,  the last 1 or 2 bytes in Buffer
1140should be the high-order word/byte of the CRC for the file being
1141decrypted, stored in Intel low-byte/high-byte order.  Versions of
1142PKZIP prior to 2.0 used a 2 byte CRC check; a 1 byte CRC check is
1143used on versions after 2.0.  This can be used to test if the password
1144supplied is correct or not.
1145
1146Step 3 - Decrypting the compressed data stream
1147----------------------------------------------
1148
1149The compressed data stream can be decrypted as follows:
1150
1151loop until done
1152    read a character into C
1153    Temp <- C ^ decrypt_byte()
1154    update_keys(temp)
1155    output Temp
1156end loop
1157
1158In addition to the above mentioned contributors to PKZIP and PKUNZIP,
1159I would like to extend special thanks to Robert Mahoney for suggesting
1160the extension .ZIP for this software.
1161
1162References:
1163
1164    Fiala, Edward R., and Greene, Daniel H., "Data compression with
1165       finite windows",  Communications of the ACM, Volume 32, Number 4,
1166       April 1989, pages 490-505.
1167
1168    Held, Gilbert, "Data Compression, Techniques and Applications,
1169       Hardware and Software Considerations", John Wiley & Sons, 1987.
1170
1171    Huffman, D.A., "A method for the construction of minimum-redundancy
1172       codes", Proceedings of the IRE, Volume 40, Number 9, September 1952,
1173       pages 1098-1101.
1174
1175    Nelson, Mark, "LZW Data Compression", Dr. Dobbs Journal, Volume 14,
1176       Number 10, October 1989, pages 29-37.
1177
1178    Nelson, Mark, "The Data Compression Book",  M&T Books, 1991.
1179
1180    Storer, James A., "Data Compression, Methods and Theory",
1181       Computer Science Press, 1988
1182
1183    Welch, Terry, "A Technique for High-Performance Data Compression",
1184       IEEE Computer, Volume 17, Number 6, June 1984, pages 8-19.
1185
1186    Ziv, J. and Lempel, A., "A universal algorithm for sequential data
1187       compression", Communications of the ACM, Volume 30, Number 6,
1188       June 1987, pages 520-540.
1189
1190    Ziv, J. and Lempel, A., "Compression of individual sequences via
1191       variable-rate coding", IEEE Transactions on Information Theory,
1192       Volume 24, Number 5, September 1978, pages 530-536.