<?xml version="1.0"?> <!-- -*- sgml -*- -->
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
  "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"[

<!-- various strings, dates etc. common to all docs -->
<!ENTITY % common-ents SYSTEM "entities.xml"> %common-ents;
]>

<book lang="en" id="userman" xreflabel="bzip2 Manual">

 <bookinfo>
  <title>bzip2 and libbzip2, version 1.0.5</title>
  <subtitle>A program and library for data compression</subtitle>
  <copyright>
   <year>&bz-lifespan;</year>
   <holder>Julian Seward</holder>
  </copyright>
  <releaseinfo>Version &bz-version; of &bz-date;</releaseinfo>

  <authorgroup>
   <author>
    <firstname>Julian</firstname>
    <surname>Seward</surname>
    <affiliation>
     <orgname>&bz-url;</orgname>
    </affiliation>
   </author>
  </authorgroup>

  <legalnotice>

  <para>This program, <computeroutput>bzip2</computeroutput>, the
  associated library <computeroutput>libbzip2</computeroutput>, and
  all documentation, are copyright &copy; &bz-lifespan; Julian Seward.
  All rights reserved.</para>

  <para>Redistribution and use in source and binary forms, with
  or without modification, are permitted provided that the
  following conditions are met:</para>

  <itemizedlist mark='bullet'>

   <listitem><para>Redistributions of source code must retain the
   above copyright notice, this list of conditions and the
   following disclaimer.</para></listitem>

   <listitem><para>The origin of this software must not be
   misrepresented; you must not claim that you wrote the original
   software.  If you use this software in a product, an
   acknowledgment in the product documentation would be
   appreciated but is not required.</para></listitem>

   <listitem><para>Altered source versions must be plainly marked
   as such, and must not be misrepresented as being the original
   software.</para></listitem>

   <listitem><para>The name of the author may not be used to
   endorse or promote products derived from this software without
   specific prior written permission.</para></listitem>

  </itemizedlist>

  <para>THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY
  EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
  THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
  PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE
  AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
  EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
  TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
  ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
  LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
  IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
  THE POSSIBILITY OF SUCH DAMAGE.</para>

 <para>PATENTS: To the best of my knowledge,
 <computeroutput>bzip2</computeroutput> and
 <computeroutput>libbzip2</computeroutput> do not use any patented
 algorithms.  However, I do not have the resources to carry
 out a patent search.  Therefore I cannot give any guarantee of
 the above statement.
 </para>

</legalnotice>

</bookinfo>



<chapter id="intro" xreflabel="Introduction">
<title>Introduction</title>

<para><computeroutput>bzip2</computeroutput> compresses files
using the Burrows-Wheeler block-sorting text compression
algorithm, and Huffman coding.  Compression is generally
considerably better than that achieved by more conventional
LZ77/LZ78-based compressors, and approaches the performance of
the PPM family of statistical compressors.</para>

<para><computeroutput>bzip2</computeroutput> is built on top of
<computeroutput>libbzip2</computeroutput>, a flexible library for
handling compressed data in the
<computeroutput>bzip2</computeroutput> format.  This manual
describes both how to use the program and how to work with the
library interface.  Most of the manual is devoted to this
library, not the program, which is good news if your interest is
only in the program.</para>

<itemizedlist mark='bullet'>

 <listitem><para><xref linkend="using"/> describes how to use
 <computeroutput>bzip2</computeroutput>; this is the only part
 you need to read if you just want to know how to operate the
 program.</para></listitem>

 <listitem><para><xref linkend="libprog"/> describes the
 programming interfaces in detail, and</para></listitem>

 <listitem><para><xref linkend="misc"/> records some
 miscellaneous notes which I thought ought to be recorded
 somewhere.</para></listitem>

</itemizedlist>

</chapter>


<chapter id="using" xreflabel="How to use bzip2">
<title>How to use bzip2</title>

<para>This chapter contains a copy of the
<computeroutput>bzip2</computeroutput> man page, and nothing
else.</para>

<sect1 id="name" xreflabel="NAME">
<title>NAME</title>

<itemizedlist mark='bullet'>

 <listitem><para><computeroutput>bzip2</computeroutput>,
  <computeroutput>bunzip2</computeroutput> - a block-sorting file
  compressor, v1.0.4</para></listitem>

 <listitem><para><computeroutput>bzcat</computeroutput> -
   decompresses files to stdout</para></listitem>

 <listitem><para><computeroutput>bzip2recover</computeroutput> -
   recovers data from damaged bzip2 files</para></listitem>

</itemizedlist>

</sect1>


<sect1 id="synopsis" xreflabel="SYNOPSIS">
<title>SYNOPSIS</title>

<itemizedlist mark='bullet'>

 <listitem><para><computeroutput>bzip2</computeroutput> [
  -cdfkqstvzVL123456789 ] [ filenames ...  ]</para></listitem>

 <listitem><para><computeroutput>bunzip2</computeroutput> [
  -fkvsVL ] [ filenames ...  ]</para></listitem>

 <listitem><para><computeroutput>bzcat</computeroutput> [ -s ] [
  filenames ...  ]</para></listitem>

 <listitem><para><computeroutput>bzip2recover</computeroutput>
  filename</para></listitem>

</itemizedlist>

</sect1>


<sect1 id="description" xreflabel="DESCRIPTION">
<title>DESCRIPTION</title>

<para><computeroutput>bzip2</computeroutput> compresses files
using the Burrows-Wheeler block sorting text compression
algorithm, and Huffman coding.  Compression is generally
considerably better than that achieved by more conventional
LZ77/LZ78-based compressors, and approaches the performance of
the PPM family of statistical compressors.</para>

<para>The command-line options are deliberately very similar to
those of GNU <computeroutput>gzip</computeroutput>, but they are
not identical.</para>

<para><computeroutput>bzip2</computeroutput> expects a list of
file names to accompany the command-line flags.  Each file is
replaced by a compressed version of itself, with the name
<computeroutput>original_name.bz2</computeroutput>.  Each
compressed file has the same modification date, permissions, and,
when possible, ownership as the corresponding original, so that
these properties can be correctly restored at decompression time.
File name handling is naive in the sense that there is no
mechanism for preserving original file names, permissions,
ownerships or dates in filesystems which lack these concepts, or
have serious file name length restrictions, such as
MS-DOS.</para>

<para><computeroutput>bzip2</computeroutput> and
<computeroutput>bunzip2</computeroutput> will by default not
overwrite existing files.  If you want this to happen, specify
the <computeroutput>-f</computeroutput> flag.</para>

<para>If no file names are specified,
<computeroutput>bzip2</computeroutput> compresses from standard
input to standard output.  In this case,
<computeroutput>bzip2</computeroutput> will decline to write
compressed output to a terminal, as this would be entirely
incomprehensible and therefore pointless.</para>

<para><computeroutput>bunzip2</computeroutput> (or
<computeroutput>bzip2 -d</computeroutput>) decompresses all
specified files.  Files which were not created by
<computeroutput>bzip2</computeroutput> will be detected and
ignored, and a warning issued.
<computeroutput>bzip2</computeroutput> attempts to guess the
filename for the decompressed file from that of the compressed
file as follows:</para>

<itemizedlist mark='bullet'>

 <listitem><para><computeroutput>filename.bz2 </computeroutput>
  becomes
  <computeroutput>filename</computeroutput></para></listitem>

 <listitem><para><computeroutput>filename.bz </computeroutput>
  becomes
  <computeroutput>filename</computeroutput></para></listitem>

 <listitem><para><computeroutput>filename.tbz2</computeroutput>
  becomes
  <computeroutput>filename.tar</computeroutput></para></listitem>

 <listitem><para><computeroutput>filename.tbz </computeroutput>
  becomes
  <computeroutput>filename.tar</computeroutput></para></listitem>

 <listitem><para><computeroutput>anyothername </computeroutput>
  becomes
  <computeroutput>anyothername.out</computeroutput></para></listitem>

</itemizedlist>

<para>If the file does not end in one of the recognised endings,
<computeroutput>.bz2</computeroutput>,
<computeroutput>.bz</computeroutput>,
<computeroutput>.tbz2</computeroutput> or
<computeroutput>.tbz</computeroutput>,
<computeroutput>bzip2</computeroutput> complains that it cannot
guess the name of the original file, and uses the original name
with <computeroutput>.out</computeroutput> appended.</para>

<para>As with compression, supplying no filenames causes
decompression from standard input to standard output.</para>

<para><computeroutput>bunzip2</computeroutput> will correctly
decompress a file which is the concatenation of two or more
compressed files.  The result is the concatenation of the
corresponding uncompressed files.  Integrity testing
(<computeroutput>-t</computeroutput>) of concatenated compressed
files is also supported.</para>

<para>You can also compress or decompress files to the standard
output by giving the <computeroutput>-c</computeroutput> flag.
Multiple files may be compressed and decompressed like this.  The
resulting outputs are fed sequentially to stdout.  Compression of
multiple files in this manner generates a stream containing
multiple compressed file representations.  Such a stream can be
decompressed correctly only by
<computeroutput>bzip2</computeroutput> version 0.9.0 or later.
Earlier versions of <computeroutput>bzip2</computeroutput> will
stop after decompressing the first file in the stream.</para>

<para><computeroutput>bzcat</computeroutput> (or
<computeroutput>bzip2 -dc</computeroutput>) decompresses all
specified files to the standard output.</para>

<para><computeroutput>bzip2</computeroutput> will read arguments
from the environment variables
<computeroutput>BZIP2</computeroutput> and
<computeroutput>BZIP</computeroutput>, in that order, and will
process them before any arguments read from the command line.
This gives a convenient way to supply default arguments.</para>

<para>Compression is always performed, even if the compressed
file is slightly larger than the original.  Files of less than
about one hundred bytes tend to get larger, since the compression
mechanism has a constant overhead in the region of 50 bytes.
Random data (including the output of most file compressors) is
coded at about 8.05 bits per byte, giving an expansion of around
0.5%.</para>

<para>As a self-check for your protection,
<computeroutput>bzip2</computeroutput> uses 32-bit CRCs to make
sure that the decompressed version of a file is identical to the
original.  This guards against corruption of the compressed data,
and against undetected bugs in
<computeroutput>bzip2</computeroutput> (hopefully very unlikely).
The chances of data corruption going undetected is microscopic,
about one chance in four billion for each file processed.  Be
aware, though, that the check occurs upon decompression, so it
can only tell you that something is wrong.  It can't help you
recover the original uncompressed data.  You can use
<computeroutput>bzip2recover</computeroutput> to try to recover
data from damaged files.</para>

<para>Return values: 0 for a normal exit, 1 for environmental
problems (file not found, invalid flags, I/O errors, etc.), 2
to indicate a corrupt compressed file, 3 for an internal
consistency error (eg, bug) which caused
<computeroutput>bzip2</computeroutput> to panic.</para>

</sect1>


<sect1 id="options" xreflabel="OPTIONS">
<title>OPTIONS</title>

<variablelist>

 <varlistentry>
 <term><computeroutput>-c --stdout</computeroutput></term>
 <listitem><para>Compress or decompress to standard
  output.</para></listitem>
 </varlistentry>

 <varlistentry>
 <term><computeroutput>-d --decompress</computeroutput></term>
 <listitem><para>Force decompression.
  <computeroutput>bzip2</computeroutput>,
  <computeroutput>bunzip2</computeroutput> and
  <computeroutput>bzcat</computeroutput> are really the same
  program, and the decision about what actions to take is done on
  the basis of which name is used.  This flag overrides that
  mechanism, and forces bzip2 to decompress.</para></listitem>
 </varlistentry>

 <varlistentry>
 <term><computeroutput>-z --compress</computeroutput></term>
 <listitem><para>The complement to
  <computeroutput>-d</computeroutput>: forces compression,
  regardless of the invokation name.</para></listitem>
 </varlistentry>

 <varlistentry>
 <term><computeroutput>-t --test</computeroutput></term>
 <listitem><para>Check integrity of the specified file(s), but
  don't decompress them.  This really performs a trial
  decompression and throws away the result.</para></listitem>
 </varlistentry>

 <varlistentry>
 <term><computeroutput>-f --force</computeroutput></term>
 <listitem><para>Force overwrite of output files.  Normally,
  <computeroutput>bzip2</computeroutput> will not overwrite
  existing output files.  Also forces
  <computeroutput>bzip2</computeroutput> to break hard links to
  files, which it otherwise wouldn't do.</para>
  <para><computeroutput>bzip2</computeroutput> normally declines
  to decompress files which don't have the correct magic header
  bytes. If forced (<computeroutput>-f</computeroutput>),
  however, it will pass such files through unmodified. This is
  how GNU <computeroutput>gzip</computeroutput> behaves.</para>
 </listitem>
 </varlistentry>

 <varlistentry>
 <term><computeroutput>-k --keep</computeroutput></term>
 <listitem><para>Keep (don't delete) input files during
  compression or decompression.</para></listitem>
 </varlistentry>

 <varlistentry>
 <term><computeroutput>-s --small</computeroutput></term>
 <listitem><para>Reduce memory usage, for compression,
  decompression and testing.  Files are decompressed and tested
  using a modified algorithm which only requires 2.5 bytes per
  block byte.  This means any file can be decompressed in 2300k
  of memory, albeit at about half the normal speed.</para>
  <para>During compression, <computeroutput>-s</computeroutput>
  selects a block size of 200k, which limits memory use to around
  the same figure, at the expense of your compression ratio.  In
  short, if your machine is low on memory (8 megabytes or less),
  use <computeroutput>-s</computeroutput> for everything.  See
  <xref linkend="memory-management"/> below.</para></listitem>
 </varlistentry>

 <varlistentry>
 <term><computeroutput>-q --quiet</computeroutput></term>
 <listitem><para>Suppress non-essential warning messages.
  Messages pertaining to I/O errors and other critical events
  will not be suppressed.</para></listitem>
 </varlistentry>

 <varlistentry>
 <term><computeroutput>-v --verbose</computeroutput></term>
 <listitem><para>Verbose mode -- show the compression ratio for
  each file processed.  Further
  <computeroutput>-v</computeroutput>'s increase the verbosity
  level, spewing out lots of information which is primarily of
  interest for diagnostic purposes.</para></listitem>
 </varlistentry>

 <varlistentry>
 <term><computeroutput>-L --license -V --version</computeroutput></term>
 <listitem><para>Display the software version, license terms and
  conditions.</para></listitem>
 </varlistentry>

 <varlistentry>
 <term><computeroutput>-1</computeroutput> (or
 <computeroutput>--fast</computeroutput>) to
 <computeroutput>-9</computeroutput> (or
 <computeroutput>-best</computeroutput>)</term>
 <listitem><para>Set the block size to 100 k, 200 k ...  900 k
  when compressing.  Has no effect when decompressing.  See <xref
  linkend="memory-management" /> below.  The
  <computeroutput>--fast</computeroutput> and
  <computeroutput>--best</computeroutput> aliases are primarily
  for GNU <computeroutput>gzip</computeroutput> compatibility.
  In particular, <computeroutput>--fast</computeroutput> doesn't
  make things significantly faster.  And
  <computeroutput>--best</computeroutput> merely selects the
  default behaviour.</para></listitem>
 </varlistentry>

 <varlistentry>
 <term><computeroutput>--</computeroutput></term>
 <listitem><para>Treats all subsequent arguments as file names,
  even if they start with a dash.  This is so you can handle
  files with names beginning with a dash, for example:
  <computeroutput>bzip2 --
  -myfilename</computeroutput>.</para></listitem>
 </varlistentry>

 <varlistentry>
 <term><computeroutput>--repetitive-fast</computeroutput></term>
 <term><computeroutput>--repetitive-best</computeroutput></term>
 <listitem><para>These flags are redundant in versions 0.9.5 and
  above.  They provided some coarse control over the behaviour of
  the sorting algorithm in earlier versions, which was sometimes
  useful.  0.9.5 and above have an improved algorithm which
  renders these flags irrelevant.</para></listitem>
 </varlistentry>

</variablelist>

</sect1>


<sect1 id="memory-management" xreflabel="MEMORY MANAGEMENT">
<title>MEMORY MANAGEMENT</title>

<para><computeroutput>bzip2</computeroutput> compresses large
files in blocks.  The block size affects both the compression
ratio achieved, and the amount of memory needed for compression
and decompression.  The flags <computeroutput>-1</computeroutput>
through <computeroutput>-9</computeroutput> specify the block
size to be 100,000 bytes through 900,000 bytes (the default)
respectively.  At decompression time, the block size used for
compression is read from the header of the compressed file, and
<computeroutput>bunzip2</computeroutput> then allocates itself
just enough memory to decompress the file.  Since block sizes are
stored in compressed files, it follows that the flags
<computeroutput>-1</computeroutput> to
<computeroutput>-9</computeroutput> are irrelevant to and so
ignored during decompression.</para>

<para>Compression and decompression requirements, in bytes, can be
estimated as:</para>
<programlisting>
Compression:   400k + ( 8 x block size )

Decompression: 100k + ( 4 x block size ), or
               100k + ( 2.5 x block size )
</programlisting>

<para>Larger block sizes give rapidly diminishing marginal
returns.  Most of the compression comes from the first two or
three hundred k of block size, a fact worth bearing in mind when
using <computeroutput>bzip2</computeroutput> on small machines.
It is also important to appreciate that the decompression memory
requirement is set at compression time by the choice of block
size.</para>

<para>For files compressed with the default 900k block size,
<computeroutput>bunzip2</computeroutput> will require about 3700
kbytes to decompress.  To support decompression of any file on a
4 megabyte machine, <computeroutput>bunzip2</computeroutput> has
an option to decompress using approximately half this amount of
memory, about 2300 kbytes.  Decompression speed is also halved,
so you should use this option only where necessary.  The relevant
flag is <computeroutput>-s</computeroutput>.</para>

<para>In general, try and use the largest block size memory
constraints allow, since that maximises the compression achieved.
Compression and decompression speed are virtually unaffected by
block size.</para>

<para>Another significant point applies to files which fit in a
single block -- that means most files you'd encounter using a
large block size.  The amount of real memory touched is
proportional to the size of the file, since the file is smaller
than a block.  For example, compressing a file 20,000 bytes long
with the flag <computeroutput>-9</computeroutput> will cause the
compressor to allocate around 7600k of memory, but only touch
400k + 20000 * 8 = 560 kbytes of it.  Similarly, the decompressor
will allocate 3700k but only touch 100k + 20000 * 4 = 180
kbytes.</para>

<para>Here is a table which summarises the maximum memory usage
for different block sizes.  Also recorded is the total compressed
size for 14 files of the Calgary Text Compression Corpus
totalling 3,141,622 bytes.  This column gives some feel for how
compression varies with block size.  These figures tend to
understate the advantage of larger block sizes for larger files,
since the Corpus is dominated by smaller files.</para>

<programlisting>
        Compress   Decompress   Decompress   Corpus
Flag     usage      usage       -s usage     Size

 -1      1200k       500k         350k      914704
 -2      2000k       900k         600k      877703
 -3      2800k      1300k         850k      860338
 -4      3600k      1700k        1100k      846899
 -5      4400k      2100k        1350k      845160
 -6      5200k      2500k        1600k      838626
 -7      6100k      2900k        1850k      834096
 -8      6800k      3300k        2100k      828642
 -9      7600k      3700k        2350k      828642
</programlisting>

</sect1>


<sect1 id="recovering" xreflabel="RECOVERING DATA FROM DAMAGED FILES">
<title>RECOVERING DATA FROM DAMAGED FILES</title>

<para><computeroutput>bzip2</computeroutput> compresses files in
blocks, usually 900kbytes long.  Each block is handled
independently.  If a media or transmission error causes a
multi-block <computeroutput>.bz2</computeroutput> file to become
damaged, it may be possible to recover data from the undamaged
blocks in the file.</para>

<para>The compressed representation of each block is delimited by
a 48-bit pattern, which makes it possible to find the block
boundaries with reasonable certainty.  Each block also carries
its own 32-bit CRC, so damaged blocks can be distinguished from
undamaged ones.</para>

<para><computeroutput>bzip2recover</computeroutput> is a simple
program whose purpose is to search for blocks in
<computeroutput>.bz2</computeroutput> files, and write each block
out into its own <computeroutput>.bz2</computeroutput> file.  You
can then use <computeroutput>bzip2 -t</computeroutput> to test
the integrity of the resulting files, and decompress those which
are undamaged.</para>

<para><computeroutput>bzip2recover</computeroutput> takes a
single argument, the name of the damaged file, and writes a
number of files <computeroutput>rec0001file.bz2</computeroutput>,
<computeroutput>rec0002file.bz2</computeroutput>, etc, containing
the extracted blocks.  The output filenames are designed so that
the use of wildcards in subsequent processing -- for example,
<computeroutput>bzip2 -dc rec*file.bz2 &#62;
recovered_data</computeroutput> -- lists the files in the correct
order.</para>

<para><computeroutput>bzip2recover</computeroutput> should be of
most use dealing with large <computeroutput>.bz2</computeroutput>
files, as these will contain many blocks.  It is clearly futile
to use it on damaged single-block files, since a damaged block
cannot be recovered.  If you wish to minimise any potential data
loss through media or transmission errors, you might consider
compressing with a smaller block size.</para>

</sect1>


<sect1 id="performance" xreflabel="PERFORMANCE NOTES">
<title>PERFORMANCE NOTES</title>

<para>The sorting phase of compression gathers together similar
strings in the file.  Because of this, files containing very long
runs of repeated symbols, like "aabaabaabaab ..."  (repeated
several hundred times) may compress more slowly than normal.
Versions 0.9.5 and above fare much better than previous versions
in this respect.  The ratio between worst-case and average-case
compression time is in the region of 10:1.  For previous
versions, this figure was more like 100:1.  You can use the
<computeroutput>-vvvv</computeroutput> option to monitor progress
in great detail, if you want.</para>

<para>Decompression speed is unaffected by these
phenomena.</para>

<para><computeroutput>bzip2</computeroutput> usually allocates
several megabytes of memory to operate in, and then charges all
over it in a fairly random fashion.  This means that performance,
both for compressing and decompressing, is largely determined by
the speed at which your machine can service cache misses.
Because of this, small changes to the code to reduce the miss
rate have been observed to give disproportionately large
performance improvements.  I imagine
<computeroutput>bzip2</computeroutput> will perform best on
machines with very large caches.</para>

</sect1>



<sect1 id="caveats" xreflabel="CAVEATS">
<title>CAVEATS</title>

<para>I/O error messages are not as helpful as they could be.
<computeroutput>bzip2</computeroutput> tries hard to detect I/O
errors and exit cleanly, but the details of what the problem is
sometimes seem rather misleading.</para>

<para>This manual page pertains to version &bz-version; of
<computeroutput>bzip2</computeroutput>.  Compressed data created by
this version is entirely forwards and backwards compatible with the
previous public releases, versions 0.1pl2, 0.9.0 and 0.9.5, 1.0.0,
1.0.1, 1.0.2 and 1.0.3, but with the following exception: 0.9.0 and
above can correctly decompress multiple concatenated compressed files.
0.1pl2 cannot do this; it will stop after decompressing just the first
file in the stream.</para>

<para><computeroutput>bzip2recover</computeroutput> versions
prior to 1.0.2 used 32-bit integers to represent bit positions in
compressed files, so it could not handle compressed files more
than 512 megabytes long.  Versions 1.0.2 and above use 64-bit ints
on some platforms which support them (GNU supported targets, and
Windows). To establish whether or not
<computeroutput>bzip2recover</computeroutput> was built with such
a limitation, run it without arguments. In any event you can
build yourself an unlimited version if you can recompile it with
<computeroutput>MaybeUInt64</computeroutput> set to be an
unsigned 64-bit integer.</para>

</sect1>



<sect1 id="author" xreflabel="AUTHOR">
<title>AUTHOR</title>

<para>Julian Seward,
<computeroutput>&bz-email;</computeroutput></para>

<para>The ideas embodied in
<computeroutput>bzip2</computeroutput> are due to (at least) the
following people: Michael Burrows and David Wheeler (for the
block sorting transformation), David Wheeler (again, for the
Huffman coder), Peter Fenwick (for the structured coding model in
the original <computeroutput>bzip</computeroutput>, and many
refinements), and Alistair Moffat, Radford Neal and Ian Witten
(for the arithmetic coder in the original
<computeroutput>bzip</computeroutput>).  I am much indebted for
their help, support and advice.  See the manual in the source
distribution for pointers to sources of documentation.  Christian
von Roques encouraged me to look for faster sorting algorithms,
so as to speed up compression.  Bela Lubkin encouraged me to
improve the worst-case compression performance.  
Donna Robinson XMLised the documentation.
Many people sent
patches, helped with portability problems, lent machines, gave
advice and were generally helpful.</para>

</sect1>

</chapter>



<chapter id="libprog" xreflabel="Programming with libbzip2">
<title>
Programming with <computeroutput>libbzip2</computeroutput>
</title>

<para>This chapter describes the programming interface to
<computeroutput>libbzip2</computeroutput>.</para>

<para>For general background information, particularly about
memory use and performance aspects, you'd be well advised to read
<xref linkend="using"/> as well.</para>


<sect1 id="top-level" xreflabel="Top-level structure">
<title>Top-level structure</title>

<para><computeroutput>libbzip2</computeroutput> is a flexible
library for compressing and decompressing data in the
<computeroutput>bzip2</computeroutput> data format.  Although
packaged as a single entity, it helps to regard the library as
three separate parts: the low level interface, and the high level
interface, and some utility functions.</para>

<para>The structure of
<computeroutput>libbzip2</computeroutput>'s interfaces is similar
to that of Jean-loup Gailly's and Mark Adler's excellent
<computeroutput>zlib</computeroutput> library.</para>

<para>All externally visible symbols have names beginning
<computeroutput>BZ2_</computeroutput>.  This is new in version
1.0.  The intention is to minimise pollution of the namespaces of
library clients.</para>

<para>To use any part of the library, you need to
<computeroutput>#include &lt;bzlib.h&gt;</computeroutput>
into your sources.</para>



<sect2 id="ll-summary" xreflabel="Low-level summary">
<title>Low-level summary</title>

<para>This interface provides services for compressing and
decompressing data in memory.  There's no provision for dealing
with files, streams or any other I/O mechanisms, just straight
memory-to-memory work.  In fact, this part of the library can be
compiled without inclusion of
<computeroutput>stdio.h</computeroutput>, which may be helpful
for embedded applications.</para>

<para>The low-level part of the library has no global variables
and is therefore thread-safe.</para>

<para>Six routines make up the low level interface:
<computeroutput>BZ2_bzCompressInit</computeroutput>,
<computeroutput>BZ2_bzCompress</computeroutput>, and
<computeroutput>BZ2_bzCompressEnd</computeroutput> for
compression, and a corresponding trio
<computeroutput>BZ2_bzDecompressInit</computeroutput>,
<computeroutput>BZ2_bzDecompress</computeroutput> and
<computeroutput>BZ2_bzDecompressEnd</computeroutput> for
decompression.  The <computeroutput>*Init</computeroutput>
functions allocate memory for compression/decompression and do
other initialisations, whilst the
<computeroutput>*End</computeroutput> functions close down
operations and release memory.</para>

<para>The real work is done by
<computeroutput>BZ2_bzCompress</computeroutput> and
<computeroutput>BZ2_bzDecompress</computeroutput>.  These
compress and decompress data from a user-supplied input buffer to
a user-supplied output buffer.  These buffers can be any size;
arbitrary quantities of data are handled by making repeated calls
to these functions.  This is a flexible mechanism allowing a
consumer-pull style of activity, or producer-push, or a mixture
of both.</para>

</sect2>


<sect2 id="hl-summary" xreflabel="High-level summary">
<title>High-level summary</title>

<para>This interface provides some handy wrappers around the
low-level interface to facilitate reading and writing
<computeroutput>bzip2</computeroutput> format files
(<computeroutput>.bz2</computeroutput> files).  The routines
provide hooks to facilitate reading files in which the
<computeroutput>bzip2</computeroutput> data stream is embedded
within some larger-scale file structure, or where there are
multiple <computeroutput>bzip2</computeroutput> data streams
concatenated end-to-end.</para>

<para>For reading files,
<computeroutput>BZ2_bzReadOpen</computeroutput>,
<computeroutput>BZ2_bzRead</computeroutput>,
<computeroutput>BZ2_bzReadClose</computeroutput> and 
<computeroutput>BZ2_bzReadGetUnused</computeroutput> are
supplied.  For writing files,
<computeroutput>BZ2_bzWriteOpen</computeroutput>,
<computeroutput>BZ2_bzWrite</computeroutput> and
<computeroutput>BZ2_bzWriteFinish</computeroutput> are
available.</para>

<para>As with the low-level library, no global variables are used
so the library is per se thread-safe.  However, if I/O errors
occur whilst reading or writing the underlying compressed files,
you may have to consult <computeroutput>errno</computeroutput> to
determine the cause of the error.  In that case, you'd need a C
library which correctly supports
<computeroutput>errno</computeroutput> in a multithreaded
environment.</para>

<para>To make the library a little simpler and more portable,
<computeroutput>BZ2_bzReadOpen</computeroutput> and
<computeroutput>BZ2_bzWriteOpen</computeroutput> require you to
pass them file handles (<computeroutput>FILE*</computeroutput>s)
which have previously been opened for reading or writing
respectively.  That avoids portability problems associated with
file operations and file attributes, whilst not being much of an
imposition on the programmer.</para>

</sect2>


<sect2 id="util-fns-summary" xreflabel="Utility functions summary">
<title>Utility functions summary</title>

<para>For very simple needs,
<computeroutput>BZ2_bzBuffToBuffCompress</computeroutput> and
<computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> are
provided.  These compress data in memory from one buffer to
another buffer in a single function call.  You should assess
whether these functions fulfill your memory-to-memory
compression/decompression requirements before investing effort in
understanding the more general but more complex low-level
interface.</para>

<para>Yoshioka Tsuneo
(<computeroutput>tsuneo@rr.iij4u.or.jp</computeroutput>) has
contributed some functions to give better
<computeroutput>zlib</computeroutput> compatibility.  These
functions are <computeroutput>BZ2_bzopen</computeroutput>,
<computeroutput>BZ2_bzread</computeroutput>,
<computeroutput>BZ2_bzwrite</computeroutput>,
<computeroutput>BZ2_bzflush</computeroutput>,
<computeroutput>BZ2_bzclose</computeroutput>,
<computeroutput>BZ2_bzerror</computeroutput> and
<computeroutput>BZ2_bzlibVersion</computeroutput>.  You may find
these functions more convenient for simple file reading and
writing, than those in the high-level interface.  These functions
are not (yet) officially part of the library, and are minimally
documented here.  If they break, you get to keep all the pieces.
I hope to document them properly when time permits.</para>

<para>Yoshioka also contributed modifications to allow the
library to be built as a Windows DLL.</para>

</sect2>

</sect1>


<sect1 id="err-handling" xreflabel="Error handling">
<title>Error handling</title>

<para>The library is designed to recover cleanly in all
situations, including the worst-case situation of decompressing
random data.  I'm not 100% sure that it can always do this, so
you might want to add a signal handler to catch segmentation
violations during decompression if you are feeling especially
paranoid.  I would be interested in hearing more about the
robustness of the library to corrupted compressed data.</para>

<para>Version 1.0.3 more robust in this respect than any
previous version.  Investigations with Valgrind (a tool for detecting
problems with memory management) indicate
that, at least for the few files I tested, all single-bit errors
in the decompressed data are caught properly, with no
segmentation faults, no uses of uninitialised data, no out of
range reads or writes, and no infinite looping in the decompressor.
So it's certainly pretty robust, although
I wouldn't claim it to be totally bombproof.</para>

<para>The file <computeroutput>bzlib.h</computeroutput> contains
all definitions needed to use the library.  In particular, you
should definitely not include
<computeroutput>bzlib_private.h</computeroutput>.</para>

<para>In <computeroutput>bzlib.h</computeroutput>, the various
return values are defined.  The following list is not intended as
an exhaustive description of the circumstances in which a given
value may be returned -- those descriptions are given later.
Rather, it is intended to convey the rough meaning of each return
value.  The first five actions are normal and not intended to
denote an error situation.</para>

<variablelist>

 <varlistentry>
  <term><computeroutput>BZ_OK</computeroutput></term>
  <listitem><para>The requested action was completed
   successfully.</para></listitem>
 </varlistentry>

 <varlistentry>
  <term><computeroutput>BZ_RUN_OK, BZ_FLUSH_OK,
    BZ_FINISH_OK</computeroutput></term>
  <listitem><para>In 
   <computeroutput>BZ2_bzCompress</computeroutput>, the requested
   flush/finish/nothing-special action was completed
   successfully.</para></listitem>
 </varlistentry>

 <varlistentry>
  <term><computeroutput>BZ_STREAM_END</computeroutput></term>
  <listitem><para>Compression of data was completed, or the
   logical stream end was detected during
   decompression.</para></listitem>
 </varlistentry>

</variablelist>

<para>The following return values indicate an error of some
kind.</para>

<variablelist>

 <varlistentry>
  <term><computeroutput>BZ_CONFIG_ERROR</computeroutput></term>
  <listitem><para>Indicates that the library has been improperly
   compiled on your platform -- a major configuration error.
   Specifically, it means that
   <computeroutput>sizeof(char)</computeroutput>,
   <computeroutput>sizeof(short)</computeroutput> and
   <computeroutput>sizeof(int)</computeroutput> are not 1, 2 and
   4 respectively, as they should be.  Note that the library
   should still work properly on 64-bit platforms which follow
   the LP64 programming model -- that is, where
   <computeroutput>sizeof(long)</computeroutput> and
   <computeroutput>sizeof(void*)</computeroutput> are 8.  Under
   LP64, <computeroutput>sizeof(int)</computeroutput> is still 4,
   so <computeroutput>libbzip2</computeroutput>, which doesn't
   use the <computeroutput>long</computeroutput> type, is
   OK.</para></listitem>
 </varlistentry>

 <varlistentry>
  <term><computeroutput>BZ_SEQUENCE_ERROR</computeroutput></term>
  <listitem><para>When using the library, it is important to call
   the functions in the correct sequence and with data structures
   (buffers etc) in the correct states.
   <computeroutput>libbzip2</computeroutput> checks as much as it
   can to ensure this is happening, and returns
   <computeroutput>BZ_SEQUENCE_ERROR</computeroutput> if not.
   Code which complies precisely with the function semantics, as
   detailed below, should never receive this value; such an event
   denotes buggy code which you should
   investigate.</para></listitem>
 </varlistentry>

 <varlistentry>
  <term><computeroutput>BZ_PARAM_ERROR</computeroutput></term>
  <listitem><para>Returned when a parameter to a function call is
   out of range or otherwise manifestly incorrect.  As with
   <computeroutput>BZ_SEQUENCE_ERROR</computeroutput>, this
   denotes a bug in the client code.  The distinction between
   <computeroutput>BZ_PARAM_ERROR</computeroutput> and
   <computeroutput>BZ_SEQUENCE_ERROR</computeroutput> is a bit
   hazy, but still worth making.</para></listitem>
 </varlistentry>

 <varlistentry>
  <term><computeroutput>BZ_MEM_ERROR</computeroutput></term>
  <listitem><para>Returned when a request to allocate memory
   failed.  Note that the quantity of memory needed to decompress
   a stream cannot be determined until the stream's header has
   been read.  So
   <computeroutput>BZ2_bzDecompress</computeroutput> and
   <computeroutput>BZ2_bzRead</computeroutput> may return
   <computeroutput>BZ_MEM_ERROR</computeroutput> even though some
   of the compressed data has been read.  The same is not true
   for compression; once
   <computeroutput>BZ2_bzCompressInit</computeroutput> or
   <computeroutput>BZ2_bzWriteOpen</computeroutput> have
   successfully completed,
   <computeroutput>BZ_MEM_ERROR</computeroutput> cannot
   occur.</para></listitem>
 </varlistentry>

 <varlistentry>
  <term><computeroutput>BZ_DATA_ERROR</computeroutput></term>
  <listitem><para>Returned when a data integrity error is
   detected during decompression.  Most importantly, this means
   when stored and computed CRCs for the data do not match.  This
   value is also returned upon detection of any other anomaly in
   the compressed data.</para></listitem>
 </varlistentry>

 <varlistentry>
  <term><computeroutput>BZ_DATA_ERROR_MAGIC</computeroutput></term>
  <listitem><para>As a special case of
   <computeroutput>BZ_DATA_ERROR</computeroutput>, it is
   sometimes useful to know when the compressed stream does not
   start with the correct magic bytes (<computeroutput>'B' 'Z'
   'h'</computeroutput>).</para></listitem>
 </varlistentry>

 <varlistentry>
  <term><computeroutput>BZ_IO_ERROR</computeroutput></term>
  <listitem><para>Returned by
   <computeroutput>BZ2_bzRead</computeroutput> and
   <computeroutput>BZ2_bzWrite</computeroutput> when there is an
   error reading or writing in the compressed file, and by
   <computeroutput>BZ2_bzReadOpen</computeroutput> and
   <computeroutput>BZ2_bzWriteOpen</computeroutput> for attempts
   to use a file for which the error indicator (viz,
   <computeroutput>ferror(f)</computeroutput>) is set.  On
   receipt of <computeroutput>BZ_IO_ERROR</computeroutput>, the
   caller should consult <computeroutput>errno</computeroutput>
   and/or <computeroutput>perror</computeroutput> to acquire
   operating-system specific information about the
   problem.</para></listitem>
 </varlistentry>

 <varlistentry>
  <term><computeroutput>BZ_UNEXPECTED_EOF</computeroutput></term>
  <listitem><para>Returned by
   <computeroutput>BZ2_bzRead</computeroutput> when the
   compressed file finishes before the logical end of stream is
   detected.</para></listitem>
 </varlistentry>

 <varlistentry>
  <term><computeroutput>BZ_OUTBUFF_FULL</computeroutput></term>
  <listitem><para>Returned by
   <computeroutput>BZ2_bzBuffToBuffCompress</computeroutput> and
   <computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> to
   indicate that the output data will not fit into the output
   buffer provided.</para></listitem>
 </varlistentry>

</variablelist>

</sect1>



<sect1 id="low-level" xreflabel=">Low-level interface">
<title>Low-level interface</title>


<sect2 id="bzcompress-init" xreflabel="BZ2_bzCompressInit">
<title><computeroutput>BZ2_bzCompressInit</computeroutput></title>

<programlisting>
typedef struct {
  char *next_in;
  unsigned int avail_in;
  unsigned int total_in_lo32;
  unsigned int total_in_hi32;

  char *next_out;
  unsigned int avail_out;
  unsigned int total_out_lo32;
  unsigned int total_out_hi32;

  void *state;

  void *(*bzalloc)(void *,int,int);
  void (*bzfree)(void *,void *);
  void *opaque;
} bz_stream;

int BZ2_bzCompressInit ( bz_stream *strm, 
                         int blockSize100k, 
                         int verbosity,
                         int workFactor );
</programlisting>

<para>Prepares for compression.  The
<computeroutput>bz_stream</computeroutput> structure holds all
data pertaining to the compression activity.  A
<computeroutput>bz_stream</computeroutput> structure should be
allocated and initialised prior to the call.  The fields of
<computeroutput>bz_stream</computeroutput> comprise the entirety
of the user-visible data.  <computeroutput>state</computeroutput>
is a pointer to the private data structures required for
compression.</para>

<para>Custom memory allocators are supported, via fields
<computeroutput>bzalloc</computeroutput>,
<computeroutput>bzfree</computeroutput>, and
<computeroutput>opaque</computeroutput>.  The value
<computeroutput>opaque</computeroutput> is passed to as the first
argument to all calls to <computeroutput>bzalloc</computeroutput>
and <computeroutput>bzfree</computeroutput>, but is otherwise
ignored by the library.  The call <computeroutput>bzalloc (
opaque, n, m )</computeroutput> is expected to return a pointer
<computeroutput>p</computeroutput> to <computeroutput>n *
m</computeroutput> bytes of memory, and <computeroutput>bzfree (
opaque, p )</computeroutput> should free that memory.</para>

<para>If you don't want to use a custom memory allocator, set
<computeroutput>bzalloc</computeroutput>,
<computeroutput>bzfree</computeroutput> and
<computeroutput>opaque</computeroutput> to
<computeroutput>NULL</computeroutput>, and the library will then
use the standard <computeroutput>malloc</computeroutput> /
<computeroutput>free</computeroutput> routines.</para>

<para>Before calling
<computeroutput>BZ2_bzCompressInit</computeroutput>, fields
<computeroutput>bzalloc</computeroutput>,
<computeroutput>bzfree</computeroutput> and
<computeroutput>opaque</computeroutput> should be filled
appropriately, as just described.  Upon return, the internal
state will have been allocated and initialised, and
<computeroutput>total_in_lo32</computeroutput>,
<computeroutput>total_in_hi32</computeroutput>,
<computeroutput>total_out_lo32</computeroutput> and
<computeroutput>total_out_hi32</computeroutput> will have been
set to zero.  These four fields are used by the library to inform
the caller of the total amount of data passed into and out of the
library, respectively.  You should not try to change them.  As of
version 1.0, 64-bit counts are maintained, even on 32-bit
platforms, using the <computeroutput>_hi32</computeroutput>
fields to store the upper 32 bits of the count.  So, for example,
the total amount of data in is <computeroutput>(total_in_hi32
&#60;&#60; 32) + total_in_lo32</computeroutput>.</para>

<para>Parameter <computeroutput>blockSize100k</computeroutput>
specifies the block size to be used for compression.  It should
be a value between 1 and 9 inclusive, and the actual block size
used is 100000 x this figure.  9 gives the best compression but
takes most memory.</para>

<para>Parameter <computeroutput>verbosity</computeroutput> should
be set to a number between 0 and 4 inclusive.  0 is silent, and
greater numbers give increasingly verbose monitoring/debugging
output.  If the library has been compiled with
<computeroutput>-DBZ_NO_STDIO</computeroutput>, no such output
will appear for any verbosity setting.</para>

<para>Parameter <computeroutput>workFactor</computeroutput>
controls how the compression phase behaves when presented with
worst case, highly repetitive, input data.  If compression runs
into difficulties caused by repetitive data, the library switches
from the standard sorting algorithm to a fallback algorithm.  The
fallback is slower than the standard algorithm by perhaps a
factor of three, but always behaves reasonably, no matter how bad
the input.</para>

<para>Lower values of <computeroutput>workFactor</computeroutput>
reduce the amount of effort the standard algorithm will expend
before resorting to the fallback.  You should set this parameter
carefully; too low, and many inputs will be handled by the
fallback algorithm and so compress rather slowly, too high, and
your average-to-worst case compression times can become very
large.  The default value of 30 gives reasonable behaviour over a
wide range of circumstances.</para>

<para>Allowable values range from 0 to 250 inclusive.  0 is a
special case, equivalent to using the default value of 30.</para>

<para>Note that the compressed output generated is the same
regardless of whether or not the fallback algorithm is
used.</para>

<para>Be aware also that this parameter may disappear entirely in
future versions of the library.  In principle it should be
possible to devise a good way to automatically choose which
algorithm to use.  Such a mechanism would render the parameter
obsolete.</para>

<para>Possible return values:</para>

<programlisting>
BZ_CONFIG_ERROR
  if the library has been mis-compiled
BZ_PARAM_ERROR
  if strm is NULL 
  or blockSize < 1 or blockSize > 9
  or verbosity < 0 or verbosity > 4
  or workFactor < 0 or workFactor > 250
BZ_MEM_ERROR 
  if not enough memory is available
BZ_OK 
  otherwise
</programlisting>

<para>Allowable next actions:</para>

<programlisting>
BZ2_bzCompress
  if BZ_OK is returned
  no specific action needed in case of error
</programlisting>

</sect2>


<sect2 id="bzCompress" xreflabel="BZ2_bzCompress">
<title><computeroutput>BZ2_bzCompress</computeroutput></title>

<programlisting>
int BZ2_bzCompress ( bz_stream *strm, int action );
</programlisting>

<para>Provides more input and/or output buffer space for the
library.  The caller maintains input and output buffers, and
calls <computeroutput>BZ2_bzCompress</computeroutput> to transfer
data between them.</para>

<para>Before each call to
<computeroutput>BZ2_bzCompress</computeroutput>,
<computeroutput>next_in</computeroutput> should point at the data
to be compressed, and <computeroutput>avail_in</computeroutput>
should indicate how many bytes the library may read.
<computeroutput>BZ2_bzCompress</computeroutput> updates
<computeroutput>next_in</computeroutput>,
<computeroutput>avail_in</computeroutput> and
<computeroutput>total_in</computeroutput> to reflect the number
of bytes it has read.</para>

<para>Similarly, <computeroutput>next_out</computeroutput> should
point to a buffer in which the compressed data is to be placed,
with <computeroutput>avail_out</computeroutput> indicating how
much output space is available.
<computeroutput>BZ2_bzCompress</computeroutput> updates
<computeroutput>next_out</computeroutput>,
<computeroutput>avail_out</computeroutput> and
<computeroutput>total_out</computeroutput> to reflect the number
of bytes output.</para>

<para>You may provide and remove as little or as much data as you
like on each call of
<computeroutput>BZ2_bzCompress</computeroutput>.  In the limit,
it is acceptable to supply and remove data one byte at a time,
although this would be terribly inefficient.  You should always
ensure that at least one byte of output space is available at
each call.</para>

<para>A second purpose of
<computeroutput>BZ2_bzCompress</computeroutput> is to request a
change of mode of the compressed stream.</para>

<para>Conceptually, a compressed stream can be in one of four
states: IDLE, RUNNING, FLUSHING and FINISHING.  Before
initialisation
(<computeroutput>BZ2_bzCompressInit</computeroutput>) and after
termination (<computeroutput>BZ2_bzCompressEnd</computeroutput>),
a stream is regarded as IDLE.</para>

<para>Upon initialisation
(<computeroutput>BZ2_bzCompressInit</computeroutput>), the stream
is placed in the RUNNING state.  Subsequent calls to
<computeroutput>BZ2_bzCo…