In keeping with the theme of new file systems, let’s take a look at SquashFS. SquashFS is a read-only compressed file system that has a variable block size. The primary intent of the file system is for constrained block device/memory systems. The classic example targets SquashFS for embedded systems but there are other uses for it that fall outside of the embedded world, and could surprise you.

Compressed File Systems

Data compression has been around for a very long time. As everyone knows, the concept behind data compress is to encode data using various techniques saving storage space. Compression also reduces the size of data that is to be transmitted. The most common example in Linux is gzip that is used to compress data files. Here’s a quick example illustrating the change in file size:

The original file was 3.2MB and after using gzip, with the "-9" option that provides maximum compression, the file is 268KB. Also notice the the extension ".gz" to indicate that the file has been compressed with gzip. The compression ratio, which is the ratio of the original size to the compressed size, is 11.9:1. The compression ratio is very dependent on the uncompressed data (how compressible is the data?) and the compression algorithm.

There are generally two types of algorithms - lossless and lossy. Since this article is about data it will focus on lossless algorithms that have an encoding that be reversed to recreate the data exactly. There are a huge number of compression algorithms that take data and find a new encoding of the data that is much smaller. The difference between algorithms focuses on the techniques that create the new encoding. From the user perspective two of the biggest concerns are how much compression can be obtained and how much time and/or CPU usage it takes to perform the compression (or uncompression). However, the goal of all of them remains the same, to reduce the size of data to save space.

The primary trade-offs in compressed file systems is that it takes CPU cycles and time to compress and uncompress data in return for reduced storage space. If you have the cycles and don't need a fast file system then you can save space. Alternatively, if you are severely memory or storage space constrained, then compressed file systems may be the only choice. This is most common in embedded systems that have severe storage restrictions.

There is an older article that discussed the compressed file systems available for Linux at the time, how they could be employed, etc. To date it is one of the better articles around compressed file systems. Overall, there is a very large number of compressed file systems available for Linux. For example, there is, JFFS, JFFS2, LogFS, CramFS, compFUSEd, and SquashFS among others. The file system compFUSEd works in conjunction with existing file systems and FUSE to allow you to do compression with both read and write for your existing file system.

Btrfs also has a compression option. Compression can be turned on as a mount option and provides compression via the zlib function that is in the kernel. While this is a great feature and works on read as well as write (not all compressed file systems do this), the focus of btrfs is not just as a compressed file system. This article will focus on file systems that are primarily compressed file systems. In particular, it will focus on SquashFS, which is the newest compressed file system for Linux, offering lots of new features that can be be exploited by non-embedded devices (even your every day desktop!).

SquashFS

SquashFS has been under development outside the Linux kernel for a number of years. Phillip Lougher started SquashFS because the existing compressed file systems did not have the features that he wanted. He began development around the 2002 time frame and tried to get it merged into the kernel a couple of times. However, the Linux community wanted some changes and features which Phillip accomplished, while adding new features that put SquashFS above other compressed file systems in many respects. So version 4.0 of SquashFS went into the Linux kernel in version 2.6.29.

SquashFS is a read-only compressed file system which means that once you mount a SquashFS file system you can only read from it. However, as will be discussed later, you can combine SquashFS with a union mount file system UnionFS or Aufs to allow you to read and write to the file system (at least from the user perspective). The version of SquashFS currently in the kernel uses gzip for it's compression.

SquashFS is arguably one of the most feature-rich compressed file systems. In the documentation for SquashFS that is included in the kernel is a quick comparison with CramFS. The author has not checked the accuracy or validity of the comparison but it is reproduced here since it at least provides a list of SquashFS features.

Feature

SquashFS

CramFS

Max filesystem size:

2^64

256 MiB

Max file size:

~ 2 TiB

16 MiB

Max files:

unlimited

unlimited

Max directories:

unlimited

unlimited

Max entries per directory

unlimited

unlimited

Max block size

1 MiB

4 KiB

Metadata compression

yes

no

Directory indexes:

yes

no

Sparse file support

yes

no

Tail-end packing (fragments)

yes

no

Exportable (NFS etc.):

yes

no

Hard link support:

yes

no

"." and ".." in readdir

yes

no

Real inode numbers:

yes

no

32-bit uids/gids

yes

no

File creation time:

yes

no

Xattr and ACL support

no

no

The features in SquashFS, particularly the size of the file system and the size of the files and directories, are very appealing for compressed file systems that could use larger amounts of space. The kernel documentation also explains that SquashFS compresses the data as well as the inodes and the directories. The inode and directory data are highly compressed and packed on byte boundaries. Each compressed inode is 8 bytes in length (on average) with the exact length depending upon the file type. For example, a regular file, a directory, a symbolic link, and block/char device inodes all have different sizes.

SquashFS also has a unique feature in that it has a variable block size and the range of block sizes is rather large. By default the block size is 128Kb and the maximum block size is 1Mb. The larger block sizes can help with greater compression ratios (i.e. smaller size file systems). But using block sizes other than the typical 4Kb has created some difficulties for SquashFS.

At this time SquashFS doesn't decompress the blocks into the kernel pagecache. This means that SquashFS has its own cache, one for metadata and one for fragments (a total of two small caches). The cache isn't used for file data blocks which are decompressed and cached in the kernel pagecache in the typical fashion. But for fragment or metadata blocks that have been read as a result of a metadata access or a fragment access, the blocks are decompressed and temporarily placed into the SquashFS cache. SquashFS packs together metadata and fragments into blocks for maximum compression so that when a read access of a fragment or metadata happens, the retrieval also obtains other metadata and fragment data. Rather than discard these additional pieces of information, they are placed into the temporary cache in case a near future access is needed without having to retrieve them and decompress them.

SquashFS 4.0 has some other features that people have been wanting. During the creation of a SquashFS file system file duplicates can be removed. In addition, it can support both big endian and little endian architectures (very useful for embedded systems that use a wide range of processor architectures). Also, during the creation of the file system, SquashFS can create a file system for the endian architecture of the target platform.

To get more insight into SquashFS there is a nice video of the SquashFS developer, Phillip Lougher, discussing the file system. It is well worth the time to watch it.

Using SquashFS

There is a really well written HOWTO that describes how one goes about creating a SquashFS file system. The general approach is to create a file system image using the SquashFS tools. You can create an image of an entire file system, a directory, or even a single file. Then this image can be mounted directly (if it is a device) or mounted using a loopback device (if it is a file).

To get started with SquashFS you have to have either a 2.6.29 kernel or a patched kernel. You also need the user-space SquashFS tools that can be downloaded from the sourceforge site. Then you just build them and install them as directed in the tar file or at the SquashFS website.

Once the tools are installed and the kernel has the ability to use SquashFS file systems, then the next step is to create a file system using mksquashfs. The general form of the command is,

# mksquashfs source1 source2 ... destination [option]

where source1 source2, etc. are files and directories to be added to the SquashFS image. They can be given with absolute or relative paths. There are a large number of options available. You can read the summary here or read the man pages once you have installed the SquashFS tools.

An example of creating a SquashFS file system is to take a subsection of a user's home directory that has data that won't be changed and create a SquashFS image and mount it in the user's home directory. In this case, the directory is /home/laytonjb/Documents. If you are not writing any new data to the Documents subdirectory, then mounting /home/laytonjb/Documents as read-only will be fine for most systems. This directory is a little over 320 MB in size (prior to using SquashFS).

The first step is to create a SquashFS image of /home/laytonjb/Documents. This assumes you have enough space on the device that contains /home/laytonjb/Documents and the resulting SquashFS image, /squashfs/storage/Documents.sqsh.

Notice that mksquashfs discovers and uses both cores and recognizes the system is little endian and creates the image for that system (there is an option for creating images for specific endian systems). After the command is finished it gives you a summary of what it did along with the compression ratios. Notice that for this lowly hard drive and laptop it took a little over 40 seconds to build the image. However, as the file system grows in size and has more files, the longer it will take mksquashfs to create the image.

Comments on "SquashFS: Not Just for Embedded Systems"

One more thing to say is that an online business administration diploma is designed for individuals to be able to effortlessly proceed to bachelor’s degree programs. The Ninety credit certification meets the other bachelor college degree requirements then when you earn your associate of arts in BA online, you will have access to the most recent technologies within this field. Several reasons why students would like to get their associate degree in business is because they are interested in the field and want to obtain the general training necessary prior to jumping right into a bachelor diploma program. Many thanks for the tips you provide with your blog.

Throughout this awesome scheme of things you’ll get a B+ for hard work. Exactly where you misplaced me personally was on your details. You know, they say, details make or break the argument.. And that couldn’t be more correct at this point. Having said that, permit me say to you just what exactly did give good results. The text is actually extremely convincing and this is most likely why I am taking an effort to comment. I do not make it a regular habit of doing that. Secondly, whilst I can easily see a jumps in reasoning you make, I am not really certain of exactly how you seem to connect your details which in turn help to make your final result. For right now I shall yield to your point however trust in the near future you actually connect your dots much better.

I will immediately grasp your rss feed as I can not in finding your e-mail subscription hyperlink or e-newsletter service. Do you have any? Please let me understand so that I may just subscribe. Thanks.

Advertiser Disclosure:
Some of the products that appear on this site are from companies from which QuinStreet receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. QuinStreet does not include all companies or all types of products available in the marketplace.