Union Mount Workshop: Review of Existing Approaches
(Revised on March 22, 2009.)
this is a report from a union mount workshop held from November 3 to 4
in the SUSE/Novell offices in Nuremberg . Back in November, we
discussed our requirements, the different ways how these requirements
can be met, and more specifically union mounts, an approach which we
believe looks promising.
All of the use cases we are interested in basically boil down to the
same thing: having an image or filesystem that is used read-only (either
because it is not writable, or because writing to the image is not
desired), and pretending that this image or filesystem is writable,
storing changes somewhere else.
None of the use cases require to merge arbitrary file systems (*), a
feature which union mounts provide by design. What we need is "only" a
mechanism to "overlay" a read-only image or file system with a writable
one. This means that many approaches other than union mounts may be
(*) One use case is to produce a number of modular images which
can be plugged together in a flexible way: several base OS
images, database images, web server images, etc. It seems
desirable to come up with a solution for plugging multiple
of those images together, resulting in a working system.
This use case requires to solve the problem of somehow
cleverly combining files which exist in more than one image
(like the rpm database). It is unclear how to achieve this,
and I am not convinced that combining images ad-hoc by
simply mounting them together (as opposed to having an
explicit merge step) is of relevance. We consider this use
case out of scope at this point.
At different layers of the storage stack, different solutions exist or
are being worked on, with different advantages and disadvantages. Let's
look at those layers.
THE BLOCK LAYER
At the block layer, copy-on-write could be used to construct a writable
filesystem image out of the read-only image and an exception store:
blocks are read from the read-only image originally. As soon as they
are modified, this fact and the block's new contents are recorded in the
exception store; from that point on, a read will return the new block's
contents from the exception store. This is how device mapper snapshots
- Above the block layer, everything will look like before:
except for the performance degradation, filesystems and
applications will see no difference in behavior.
- Existing files will be modified at the block level, so
working with huge directories and big files will perform
similarly to how these operations perform today (after taking
the block layer slowdown into account).
- Snapshots implemented at the block layer are slow. The
current device mapper implementation scales very poorly. Too
much I/O is necessary because the block layer does not have
any meta-information about disk blocks. (Things are
improving somewhat; particularly, filesystems can pass down
some meta-information to the block layer since recently.
Still, some optimizations available to higher-layer
approaches are not available at the block layer.)
- Mixing different filesystem types, such as a compressed
read-only and an uncompressed writable filesystem, is not
possible with block layer approaches. (The block layer does
not know what a file system is.)
- Because they only record the deltas relative to a read-only
image, exception stores are relatively worthless in
isolation. If the read-only image lost, becomes corrupted,
or is modified by accident, the exception stores based on it
Block device approaches are not limited to a single machine. Network
block devices can be used in distributed environments. (This implies
additional overheads, however.)
THE FILESYSTEM LAYER
Future filesystems based on copy-on-write designs, such as btrfs, do (or
will) support in-filesystem block device management and snapshots.
Conceptually this is similar to block-layer mechanisms, but it allows
optimizations that would be hard to achieve otherwise, which makes this
approach scale much better.
- Transparent to users: everything will behave as if on a
normal, writable file system.
- Faster, because more meta-information is available, changes
can be tracked at a sub-block granularity, and functionality
that the filesystem already provides does not need to be
duplicated at the block layer.
- Efficient compression is easier to implement than at the
- At least in btrfs, snapshots are first-class objects; they do
not merely record the deltas relative to another image. The
filesystem itself guarantees consistency. Despite that, they
may be based on a read-only device, combined with a writable
- These filesystems do not have network transparency built in.
Block-layer network transparency may be used though, and such
filesystems may be exposed to the network via a network
filesystem like NFS, or Zach Brown's CRFS  (in development).
THE VFS LAYER: MOUNTS / BIND MOUNTS
The classical approach for making part of the filesystem namespace
writable is to mount writable and read-only filesystems together "in the
right way". As an extension to ordinary mounts, bind mounts allow to
mount specific directories and even files on top of each other.
- Bind mounts exist today; the technology is well understood,
- The places in the filesystem namespace that shall be writable
need to be known in advance. It may be difficult to adapt to
applications which write to files in unexpected places.
Additional places in the namespace need to be made writable
manually when necessary.
- Bind mounts do not have a concept of unified directories:
files can only be mounted over existing files; new files or
directories can not be created that way. When files need to
be added, as frequently happens in the /etc directory, for
example, the entire directory must be copied, and files in
the directory must either be copied too, bind mounted to
individually, or replaced by symlinks. (In the latter two
cases, the old files referred to must be visible somewhere
else in the namespace.)
- Software updates modify files in many places which are
usually accessed read-only. Our software is rpm based, so
rpm could be modified to bind mount new files over their
previous versions, but over time, the number of union mounts
this would lead to would become excessive, and this would not
scale well: upon system startup, a large number of mounts
would have to be set up, which would take time, and consume a
significant amount of kernel memory. Such systems would
become hard to maintain.
- Some applications may see unexpected error codes when
manipulating bind mounted files and directories: for example,
bind mounted files and directories cannot be renamed (but
neither can files on a read-only filesystem). It could be
argued that applications relying on undefined behavior are
buggy, but this would not help when legacy applications need
to be used.
THE VFS LAYER: UNION MOUNTS
Union mounts are more flexible than bind mounts: file systems are
mounted on top of each other transparently so that at the top layer,
files from the top and bottom layer are visible at the same time.
(Ordinary mounts are opaque, hiding all lower layers.) Files at the top
layer hide files at the bottom layer with the same name. The bottom
layer is never written to. The top layer may be read-only (which is
comparatively boring), or writable.
Union mounts can be stacked on top of each other and it is possible to
"see through" multiple layers. I will describe union mounts in terms of
two layers here to keep things simple.
If a file is deleted which exists at the bottom layer, a so-called
whiteout file with the same name is created at the top layer. Users
never get to see this file; it is not included in readdir results, and
trying to open it fails with errno == ENOENT. If a file with the same
name is later created, this file replaces the whiteout.
When a new directory is created in place of a whiteout and the bottom
layer contains a directory with the same name, the files from the bottom
layer would be visible. This is not desirable as newly created
directories are supposed to be empty. To fix this defect, new
directories are created with a special "opaque" flag set: this flag
indicates that bottom-layer directory contents shall remain hidden.
When files at the bottom layer are about to be modified, since the
bottom layer is accessed read-only, they first need to be copied into
the top layer. This operation is commonly referred to as copy-up. With
union mounts, the decision when to copy files up is made at file open
(Directories do not require a copy-up operation: when a directory that
does not exist at the top layer already is modified, it is sufficient to
create an empty directory at the top layer with the "opaque" flag
- Union mounts are very easy to set up, and they are relatively
- File system types can be arbitrarily mixed.
- Compared to the alternatives, union mounts are "relatively
easy" to implement. (But note the various problems mentioned
below which make it difficult to argue that a particular
implementation is "correct" or "good enough".)
- Even though the result may not be the expected one, the top
layer may be mounted over a bottom layer even after the
bottom layer was modified.
- Multiple filesystems can be mounted together arbitrarily,
fully merging their contents.
- When an process opens a file in read-only mode, it will get a
file descriptor on the bottom-layer filesystem. Another
process may open the same file for writing. The read-only
file is copied up, and the second process will get a file
descriptor to a different file. The first process will not
see any changes done by the second process. (Subsequent
read-only opens will return the top-layer file, of course.)
- When a file is copied up, it changes its device and inode
numbers (the st_dev and st_ino fields returned by stat(2)).
Applications making assumptions about these fields may be
- Union mounts are name based, and hardlinks across mount
layers are not supported. When a file on the bottom layer
with a link count greater than one is opened for writing,
copying it up will drop its link count to one. When another
alias of the same bottom-layer file is opened for writing
afterwards, the top layer will end up with two independent
copies, each with a link count of one.
(It might help to change stat(2) to always return a fake link
count of one for all files on the bottom layer.)
- Operations expected to be cheap may suddenly become very
expensive. This includes changing file timestamps and
creating a hardlink to a bottom layer file, both of which
will trigger a copy-up operation. (Failing link(2) with
errno == EXDEV for files within the same directory may at
first seem acceptable, but many applications make the
assumption that hardlinks will be supported, and do not
implement the appropriate fallback code.)
- Copy-up of lage files may become prohibitively slow. (Think
of fixing a few tags in your music collection.) File system
or block layer approaches scale better in this respect.
- POSIX specifies requirements for readdir that, taken
together, result in having to compute and cache the entire
directory contents, removing duplicates and whiteouts. The
reason is because processes may arbitrarily seek in the
readdir stream (see telldir(3) and seekdir(3)). This is a
cpu and memory intensive operation.
- Union mounts operate on files as a whole, and do not provide
mechanisms for merging files. When multiple filesystems are
union mounted, the file on the top layer will "win", which
may still lead to an inconsistent system. (Think of files
such as the rpm database.)
- Inotify: ?
BETWEEN THE FILESYSTEM AND VFS LAYERS: UNIONFS, AUFS
Unionfs and Aufs try to appear as a filesystem to the VFS, and as the
VFS to filesystems. Some of the namespace handling is done at the
filesystem level, which results in various problems (locking, cache
Both of these implementations include a number of features which our use
cases do not immediately require (multiple writable layers, stable inode
numbers across copy-up, cross-layer hardlinks). These features lead to
a tremendous increase of the complexity of the implementation though.
- Available today.
- The existing implementations are overly complex, and full of
bugs. They are not maintainable.
SOME RANDOM THOUGHTS
In many environments, the underlying read-only image needs to be updated
over time, and overlays based on a previous version of the image need to
be moved to more recent versions. This will require telling system
files from configuration, and configuration from user data. From a
kernel / base OS point of view, we can only provide the mechanisms; the
higher-level issues need to be addressed is systems management software.
One mechanism likely to help is layering, in which a snapshot is based
on another snapshot, or multiple layers are union mounted together.
System updates could go into a different layer than configuration, and
user data could go into yet another layer. All the approaches discussed
support this kind of layering to some degree; all approaches above the
block layer do so in a reasonably scalable way.
Tools will be needed for efficiently analyzing the relative changes
between two layers. This should be possible by mounting the two
versions next to each other and comparing them using traditional tools.
For some approaches, lower-lever tools could be written for achieving
the same results more efficiently. (Such tools may be difficult to
write, and might not be worth the bother, though.)
CONCLUSION (AS OF MARCH 2009)
The union mount/filesystem problem doesn't have a right and several wrong
solutions; each of the discussed solutions has its own strenghts and
weaknesses, even when only considering the union of one read-only and one
writable image. In the long run and where applicable, in-filesystem
snapshots look the most promising to me.
When different types of filesystems must be mixed (and until
in-filesystem snapshots on Linux have matured), union mounts may well
save the day. They could be implemented incrementally starting with the
union of two directories, then extended by whiteouts to allow deletes,
and then recursion and opaque directories could be added. Problems such
as unstable inode numbers and missing support for hardlinks across
layers would remain though; trying to solve them at the VFS layer would
make no sense. This makes union mounts apropriate only in limited
scenarios; when used beyond this, users would trip over the inherent
defects far too often.
Miklos Szeredi has recently posted an early delta filesystem prototype
based on FUSE . While this approach does not unifying filesystems,
it does record the deltas relative to a read-only filesystem. This
allows to record metadata-only changes without triggering full,
expensive copy-up operations, for example. Stacking a delta filesystem
on top of read-only union mounts seems interesting, too. Many of the
details of the delta filesystem design have not been flashed out yet
though, so we will have to see.
20 November 2008 and 22 March 2009
 Union Mount Workshop, http://en.opensuse.org/Union_Mount_Workshop
 Cache Coherent Networked File System (CRFS),
 Miklos Szeredi, delta filesystem prototype,