PageRenderTime 43ms CodeModel.GetById 21ms app.highlight 17ms RepoModel.GetById 1ms app.codeStats 1ms

Plain Text | 247 lines | 202 code | 45 blank | 0 comment | 0 complexity | 0e6476ee4e9a308b6c1029d18756e90c MD5 | raw file
Possible License(s): GPL-2.0, LGPL-2.0, AGPL-1.0
  4Squashfs is a compressed read-only filesystem for Linux.
  5It uses zlib compression to compress files, inodes and directories.
  6Inodes in the system are very small and all blocks are packed to minimise
  7data overhead. Block sizes greater than 4K are supported up to a maximum
  8of 1Mbytes (default block size 128K).
 10Squashfs is intended for general read-only filesystem use, for archival
 11use (i.e. in cases where a .tar.gz file may be used), and in constrained
 12block device/memory systems (e.g. embedded systems) where low overhead is
 15Mailing list:
 16Web site:
 21Squashfs filesystem features versus Cramfs:
 23				Squashfs		Cramfs
 25Max filesystem size:		2^64			256 MiB
 26Max file size:			~ 2 TiB			16 MiB
 27Max files:			unlimited		unlimited
 28Max directories:		unlimited		unlimited
 29Max entries per directory:	unlimited		unlimited
 30Max block size:			1 MiB			4 KiB
 31Metadata compression:		yes			no
 32Directory indexes:		yes			no
 33Sparse file support:		yes			no
 34Tail-end packing (fragments):	yes			no
 35Exportable (NFS etc.):		yes			no
 36Hard link support:		yes			no
 37"." and ".." in readdir:	yes			no
 38Real inode numbers:		yes			no
 3932-bit uids/gids:		yes			no
 40File creation time:		yes			no
 41Xattr support:			yes			no
 42ACL support:			no			no
 44Squashfs compresses data, inodes and directories.  In addition, inode and
 45directory data are highly compacted, and packed on byte boundaries.  Each
 46compressed inode is on average 8 bytes in length (the exact length varies on
 47file type, i.e. regular file, directory, symbolic link, and block/char device
 48inodes have different sizes).
 53As squashfs is a read-only filesystem, the mksquashfs program must be used to
 54create populated squashfs filesystems.  This and other squashfs utilities
 55can be obtained from  Usage instructions can be
 56obtained from this site also.
 62A squashfs filesystem consists of a maximum of eight parts, packed together on a byte
 65	 ---------------
 66	|  superblock 	|
 67	|---------------|
 68	|  datablocks   |
 69	|  & fragments  |
 70	|---------------|
 71	|  inode table	|
 72	|---------------|
 73	|   directory	|
 74	|     table     |
 75	|---------------|
 76	|   fragment	|
 77	|    table      |
 78	|---------------|
 79	|    export     |
 80	|    table      |
 81	|---------------|
 82	|    uid/gid	|
 83	|  lookup table	|
 84	|---------------|
 85	|     xattr     |
 86	|     table	|
 87	 ---------------
 89Compressed data blocks are written to the filesystem as files are read from
 90the source directory, and checked for duplicates.  Once all file data has been
 91written the completed inode, directory, fragment, export and uid/gid lookup
 92tables are written.
 943.1 Inodes
 97Metadata (inodes and directories) are compressed in 8Kbyte blocks.  Each
 98compressed block is prefixed by a two byte length, the top bit is set if the
 99block is uncompressed.  A block will be uncompressed if the -noI option is set,
100or if the compressed block was larger than the uncompressed block.
102Inodes are packed into the metadata blocks, and are not aligned to block
103boundaries, therefore inodes overlap compressed blocks.  Inodes are identified
104by a 48-bit number which encodes the location of the compressed metadata block
105containing the inode, and the byte offset into that block where the inode is
106placed (<block, offset>).
108To maximise compression there are different inodes for each file type
109(regular file, directory, device, etc.), the inode contents and length
110varying with the type.
112To further maximise compression, two types of regular file inode and
113directory inode are defined: inodes optimised for frequently occurring
114regular files and directories, and extended types where extra
115information has to be stored.
1173.2 Directories
120Like inodes, directories are packed into compressed metadata blocks, stored
121in a directory table.  Directories are accessed using the start address of
122the metablock containing the directory and the offset into the
123decompressed block (<block, offset>).
125Directories are organised in a slightly complex way, and are not simply
126a list of file names.  The organisation takes advantage of the
127fact that (in most cases) the inodes of the files will be in the same
128compressed metadata block, and therefore, can share the start block.
129Directories are therefore organised in a two level list, a directory
130header containing the shared start block value, and a sequence of directory
131entries, each of which share the shared start block.  A new directory header
132is written once/if the inode start block changes.  The directory
133header/directory entry list is repeated as many times as necessary.
135Directories are sorted, and can contain a directory index to speed up
136file lookup.  Directory indexes store one entry per metablock, each entry
137storing the index/filename mapping to the first directory header
138in each metadata block.  Directories are sorted in alphabetical order,
139and at lookup the index is scanned linearly looking for the first filename
140alphabetically larger than the filename being looked up.  At this point the
141location of the metadata block the filename is in has been found.
142The general idea of the index is ensure only one metadata block needs to be
143decompressed to do a lookup irrespective of the length of the directory.
144This scheme has the advantage that it doesn't require extra memory overhead
145and doesn't require much extra storage on disk.
1473.3 File data
150Regular files consist of a sequence of contiguous compressed blocks, and/or a
151compressed fragment block (tail-end packed block).   The compressed size
152of each datablock is stored in a block list contained within the
153file inode.
155To speed up access to datablocks when reading 'large' files (256 Mbytes or
156larger), the code implements an index cache that caches the mapping from
157block index to datablock location on disk.
159The index cache allows Squashfs to handle large files (up to 1.75 TiB) while
160retaining a simple and space-efficient block list on disk.  The cache
161is split into slots, caching up to eight 224 GiB files (128 KiB blocks).
162Larger files use multiple slots, with 1.75 TiB files using all 8 slots.
163The index cache is designed to be memory efficient, and by default uses
16416 KiB.
1663.4 Fragment lookup table
169Regular files can contain a fragment index which is mapped to a fragment
170location on disk and compressed size using a fragment lookup table.  This
171fragment lookup table is itself stored compressed into metadata blocks.
172A second index table is used to locate these.  This second index table for
173speed of access (and because it is small) is read at mount time and cached
174in memory.
1763.5 Uid/gid lookup table
179For space efficiency regular files store uid and gid indexes, which are
180converted to 32-bit uids/gids using an id look up table.  This table is
181stored compressed into metadata blocks.  A second index table is used to
182locate these.  This second index table for speed of access (and because it
183is small) is read at mount time and cached in memory.
1853.6 Export table
188To enable Squashfs filesystems to be exportable (via NFS etc.) filesystems
189can optionally (disabled with the -no-exports Mksquashfs option) contain
190an inode number to inode disk location lookup table.  This is required to
191enable Squashfs to map inode numbers passed in filehandles to the inode
192location on disk, which is necessary when the export code reinstantiates
193expired/flushed inodes.
195This table is stored compressed into metadata blocks.  A second index table is
196used to locate these.  This second index table for speed of access (and because
197it is small) is read at mount time and cached in memory.
1993.7 Xattr table
202The xattr table contains extended attributes for each inode.  The xattrs
203for each inode are stored in a list, each list entry containing a type,
204name and value field.  The type field encodes the xattr prefix
205("user.", "trusted." etc) and it also encodes how the name/value fields
206should be interpreted.  Currently the type indicates whether the value
207is stored inline (in which case the value field contains the xattr value),
208or if it is stored out of line (in which case the value field stores a
209reference to where the actual value is stored).  This allows large values
210to be stored out of line improving scanning and lookup performance and it
211also allows values to be de-duplicated, the value being stored once, and
212all other occurences holding an out of line reference to that value.
214The xattr lists are packed into compressed 8K metadata blocks.
215To reduce overhead in inodes, rather than storing the on-disk
216location of the xattr list inside each inode, a 32-bit xattr id
217is stored.  This xattr id is mapped into the location of the xattr
218list using a second xattr id lookup table.
2234.1 Todo list
226Implement ACL support.
2284.2 Squashfs internal cache
231Blocks in Squashfs are compressed.  To avoid repeatedly decompressing
232recently accessed data Squashfs uses two small metadata and fragment caches.
234The cache is not used for file datablocks, these are decompressed and cached in
235the page-cache in the normal way.  The cache is used to temporarily cache
236fragment and metadata blocks which have been read as a result of a metadata
237(i.e. inode or directory) or fragment access.  Because metadata and fragments
238are packed together into blocks (to gain greater compression) the read of a
239particular piece of metadata or fragment will retrieve other metadata/fragments
240which have been packed with it, these because of locality-of-reference may be
241read in the near future. Temporarily caching them ensures they are available
242for near future access without requiring an additional read and decompress.
244In the future this internal cache may be replaced with an implementation which
245uses the kernel page cache.  Because the page cache operates on page sized
246units this may introduce additional complexity in terms of locking and
247associated race conditions.