The Oracle Cluster
Filesystem (ocfs2) is a filesystem for clustered
systems accessing a shared device, such as a Storage Area Network (SAN).
It enables all nodes on the cluster to see the same files; changes to any
files are visible immediately on other nodes. It is the filesystem's
responsibility to ensure that nodes do not corrupt data by writing into each other's
files. To guarantee integrity, ocfs2 uses the Linux Distributed Lock
Manager (DLM) to serialize events. However, a major goal of a clustered
filesystem is to reduce cluster locking overhead in order to improve overall
performance. This article will provide an overview of ocfs2 and how it is
A brief history
Version 1 of the ocfs filesystem was an early effort by Oracle to create a
filesystem for the clustered environment. It was a basic filesystem targeted to
support Oracle database storage and did not have most of the POSIX features due
to its limited disk format. Ocfs2 was a development effort to convert this basic
filesystem into a general-purpose filesystem. The ocfs2 source code was merged
in the Linux kernel with 2.6.16; since this merger (in 2005), a lot of
features have been added to the filesystem which improve data storage
efficiency and access times.
Ocfs2 needs a cluster management system to handle cluster
operations such as node membership and fencing. All nodes must have the same
configuration, so each node knows about the other nodes in the
cluster. There are currently two ways to handle cluster management for ocfs2:
- Ocfs2 Cluster Base (O2CB) - This is the in-kernel implementation of cluster
configuration management; it provides only the basic services needed to have a
clustered filesystem running. Each node writes to a heartbeat file to
let others know the node is alive. More information on running the ocfs2
filesystem using o2cb can be found int the
ocfs2 user guide
This mode does not have the capability of removing nodes from a live cluster and
cannot be used for cluster-wide POSIX locks.
- Linux High Availability - uses user-space tools, such as heartbeat
and pacemaker, to perform cluster management. These packages are complete cluster
management suites which can be used for advanced cluster activities such as
different fail-over, STONITH (Shoot The Other Node In The Head - yes, computing
terms can be barbaric), migration dependent services etc. It is also
capable of removing nodes from a live cluster. It supports cluster-wide
POSIX locks, as
opposed to node-local locks. More information about cluster management tools
can be found at clusterlabs.org and linux-ha.org
ocfs2 separates the way data and metadata are stored on disk. To facilitate this,
it has two types of blocks:
- Metadata or simply "blocks" - the smallest addressable unit. These blocks
contain the metadata of the filesystem, such as the inodes, extent blocks,
group descriptors etc. The valid sizes are 512 bytes to 4KB (incremented in
powers of two). Each metadata block contains a signature that says what
the block contains. This signature is cross-checked while reading that specific
- Data Clusters - data storage for regular files. Valid cluster sizes range
from 4KB to 1MB (in powers of two). A larger data cluster reduces the size
of filesystem metadata such as allocation bitmaps, making filesystem
activities such as data allocation or filesystem checks faster. On the
other hand, a large cluster size increases internal
fragmentation. A large cluster size is recommended for filesystems storing
large files such as virtual machine images, while a small data cluster size is
recommended for a filesystem which holds lots of small files, such as a mail
An inode occupies an entire block. The block number (with respect to the
filesystem block size) doubles as the inode number. This organization may result
in high disk space usage for a filesystem with a lot of small files. To
minimize that problem, ocfs2 packs the data
files into the inode itself if they are small enough to fit. This
feature is known as "inline data." Inode numbers are 64 bits, which gives enough
room for inode numbers to be addressed on large storage devices.
Data in a regular file is maintained in a B-tree of extents; the root of this
B-tree is the inode. The inode holds a list of extent records which may either
point to data extents, or point to extent blocks (which are the
intermediate nodes in the tree). A special field called
l_tree_depth contains the depth of the tree. A value of
zero indicates that the blocks pointed to by extent records are data extents.
The extent records contain the offset from the start of the file
in terms of cluster blocks, which helps in determining the path to take while
traversing the B-tree to find the block to be accessed.
The basic unit of locking is the inode. Locks are granted on a special DLM data
structure known as a lock
resource. For any access to the file, the process must
request a DLM lock on the lock resource over the cluster. DLM offers six lock
differentiate between the type of operation. Out of these, ocfs2 uses only
three: exclusive, protected read, and null locks. The inode
maintains three types of lock resources for different operations:
- read-write lock resource: is used to serialize writes if
multiple nodes perform I/O at the same time on a file.
- inode lock resource: is used for metadata inode operations
- open lock resource: is used to identify deletes of a file. When a
file is open, this lock resource is opened in protected-read mode. If the node
intends to delete it, it will request for a exclusive lock. If successful, it
means that no other node is using the file and it can be safely deleted. If
unsuccessful, the inode is treated as an orphan file (discussed later)
Directory entries are stored in name-inode pairs in blocks known
as directory blocks. Access to the storage pattern of directory blocks is the
same as for a regular file. However, directory blocks are allocated
as cluster blocks. Since a directory block is considered to be a metadata
block, the first allocation uses only a part of the cluster block. As the
directory expands, the remaining unused blocks of the data cluster are filled
until the data cluster block is fully used.
A relatively new feature is indexing the directory entries for faster
retrievals and improved lookup times. Ocfs2 maintains a
separate indexed tree based on the hash of the directory names; the hash
index points to the directory block where the directory entry can be found. Once
the directory block is read, the directory entry is searched linearly.
A special directory trailer is placed at the end of a directory block which
contains additional data about that block. It keeps a track of
the free space in the directory block for faster free space lookup during
directory entry insert operations. The trailer also contains the
checksum for the directory block, which is used by the metaecc feature (discussed
A special system directory (//) contains all the metadata files
for the filesystem; this directory is not accessible from a
regular mount. Note that the // notation is used only for the
Files in the system directory, known as system files, are different from regular
files, both in the terms on how they store information and what they store.
An example of a system file is the slotmap, which defines the mapping of a node
in the cluster. A node joins a cluster by providing its unique DLM name. The
slot map provides it with a slot number, and the node inherits all system files
associated with the slot number. The slot number assignment is not persistent
across boots, so a node may inherit the system files of another node. All
node-associated files are suffixed by the slot number to differentiate the files
of different slots.
A global bitmap file in the system directory keeps a record of the allocated
blocks on the device. Each node also maintains a "local allocations" system
file, which manages chunks of blocks obtained from the global
bitmap. Maintaining local allocations decreases contention over global
The allocation units are divided into the following:
- inode_alloc: allocates inodes for the local node.
- extent_alloc: allocates extent blocks for the local node. Extent
blocks are intermediate leaf nodes in the B-tree storage of the files.
- local_alloc: allocates data in data cluster sizes for the use of
regular file data.
Each allocator is associated with an inode; it maintains allocations in
units known as "block groups." The
allocation groups are preceded by a group descriptor which contains details
about the block group, such as free units, allocation bitmaps etc. The
allocator inode contains a chain of group descriptor block pointers. If this
chain is exhausted, group descriptors are added to the existing ones in the
linked list. Think of it as an array of linked lists. The new group
added to the smallest chain so that number of hops required to reach an
allocation unit is small.
Things get complicated when allocated data blocks are freed because those
blocks could belong to the allocation map of another node. To
resolve this problem, a "truncate log" maintains the blocks which have been
freed locally, but
not yet returned to the global bitmap. Once the node gets a lock on the global
bitmap, the blocks in the local truncate log are freed.
A file is not physically deleted until all processes accessing the file close
it. Filesystems such as ext3 maintain an orphan list which contains a list of
files which have been unlinked but still are in use by the system. Ocfs2 also
maintains such a list to handle orphan inodes. Things are a bit more
complex, however, because a node must check
that a file to be deleted is not being used anywhere in the cluster. This
check is coordinated using the inode lock resource.
Whenever a file is unlinked, and the removed link happens to be the last
link to the file, a check is made to determine whether another node is
using the file by requesting
an exclusive lock over inode lock resource. If the file is being used, it
moved to the orphan directory and marked with a OCFS2_ORPHANED_FL flag. The
orphan directory is later scanned to check for files not being accessed by any
of the nodes in order to physically remove them from the storage device.
Ocfs2 maintains a journal to deal with unexpected crashes. It uses the Linux
JBD2 layer for journaling. The journal files are maintained, per node, for all
I/O performed locally. If a node dies, it is the responsibility of the other
nodes in the cluster to replay the dead node's journal before proceeding
with any operations.
Ocfs2 has a couple of other distinctive features that it can boast
about. They include:
- Reflinks is a feature to support snapshotting of files using
copy-on-write (COW). Currently, a system call interface, to be called reflink() or copyfile()
discussed upstream. Until the system call is finalized, users can access this
feature via the reflink system tool which uses an ioctl()
call to perform the snapshotting.
- Metaecc is an error correcting feature for the metadata using Cyclic
Redundancy Check (CRC32). The code warns if the calculated
error-correcting code is different from the one stored, and re-mounts the
filesystem read-only in order to avoid further corruption. It is also capable of
correcting single-bit errors on the fly. A special data structure,
ocfs2_block_check, is embedded in most metadata structures to hold the CRC32
values associated with the structure.
Ocfs2 developers continue to add features to keep it
up to par with other new filesystems. Some features to expect in
the near future are delayed allocation, online filesystem checks, and
defragmentation. Since one of the main goals of ocfs2 is to support a
database, file I/O performance is considered a priority, making it one of the
best filesystems for the clustered environment.
[Thanks to Mark Fasheh for reviewing this article.]
to post comments)