project was chosen
At the end of 2010,
to replace STGT as the in-kernel SCSI target
implementation. There were two main contenders (LIO and SCST) which tried
to get their code into the Linux kernel tree. This article will compare
the two projects and try to describe what these implementations have to offer.
What are SCSI targets?
The SCSI subsystem uses a sort of client-server model. Typically a computer
is the client or "initiator," requesting that blocks be written to or read
from a "target," which is usually a data storage device. The SCSI target
subsystem enables a computer node to behave as a SCSI storage device, responding
to storage requests by other SCSI initiator nodes. This opens up the possibility
of creating custom SCSI devices and putting intelligence behind the storage.
An example of an intelligent SCSI target is Data Domain's
online backup appliance, which supports de-duplication (thus saving
space). The appliance, functioning as a SCSI target, is a computer node which
intelligently writes only those blocks which are not already stored, and
increases the reference counts of the blocks which are already present, thus
writing only the blocks which have changed since the last backup. On the
other side of the SCSI link,
the initiator sees the appliance as a normal, shared SCSI storage device and
uses its regular backup application to write to the target.
The most common implementation of the SCSI target subsystem is an iSCSI
server, which uses a standard TCP/IP encapsulation of SCSI to export a SCSI
device over the network. Most SCSI target projects started with the idea
targets before supporting other protocols. Since only a network interface is needed to act as both an iSCSI
initiator and an iSCSI target, supporting iSCSI doesn't require any
special hardware beyond a network port, which almost every computer has
these days. However, most SCSI
targets can be supported with existing initiator cards, so if you have a
Fibre, SAS, or Parallel SCSI card, it should be possible to use one of
the SCSI target projects to make your computer into a SCSI target for
the particular SCSI bus supported by the card.
The Linux kernel SCSI subsystem currently uses STGT to implement the
SCSI target functionality; STGT was introduced into the Linux kernel at the
end of 2006 by
Fujita Tomonori. It has a library in the kernel which assists the in-kernel
target drivers. All target processing happens in user space, which may
lead to performance bottlenecks.
Two out-of-tree kernel SCSI target solutions were contenders to replace
STGT: LIO and SCST. SCST has been pushing to
be included in the Linux kernel since at least 2008. It was decided
the STGT project could serve the kernel for a little longer. As time passed, the
design limitations of STGT were encountered and a replacement sought. The
main criteria for a replacement SCSI target subsystem
defined by James Bottomley, the SCSI maintainer, were:
- That it would be a drop in replacement for STGT (our current in-kernel
target mode driver), since there is room for only one SCSI target
- That it used a modern sysfs-based control and configuration plane.
- That the code was reviewed as clean enough for inclusion.
The first condition proved to be too restrictive; it was not possible to
avoid breaking the ABI entirely. So the current goal, instead, is to find
a way to gracefully transition STGT users to the new interface.
Hints of LIO replacing the STGT project came in
the 2010 Linux Storage and Filesystem
Summit. Christoph Hellwig volunteered to review and clean
up the code; he managed to reduce the code-base by around 10,000 lines to
ready to merge into the kernel.
Both projects have drawn comparison charts of their feature lists which are
available on their respective web sites: LIO and SCST. However, before
exploring the differences, lets compare the similarities. Both projects
implement an in-kernel SCSI target core. They provide local SCSI targets
similar to loop devices, which comes in handy for using targets in virtualized
environments. Both projects support iSCSI, which was one of the initial and main
motivations for both projects.
Back-storage handlers are available on both projects in kernel space as
well as for user space. Back-storage handlers allow target administrators
to control how devices are exported to the initiators. For example, a
pass-through handler allows exporting the SCSI hardware as it is, instead
of masking the details of that hardware, while a virtual-disk handler
allows exporting of files as virtual disk to the initiator.
Both projects support Persistent Reservations (PR); a feature for I/O
fencing and failover/retakeover of storage devices in high-availability
clusters. Using the PR commands, an initiator can establish, preempt,
query, or reset a reservation policy with a specified target. During a
failover takeover, the new virtual resource can reset the reservation
policy of the old virtual resource, making device takeover easier and
The main users of the SCSI target subsystem are storage companies providing
storage solutions to the industry. Most of these storage solutions are
plug-and-play appliances which can be attached to the storage network and
used with little or no configuration. SCST boasts of a wider user base, which
probably comes from the fact that they have wider range of transport support.
SCST supports both Qlogic and Emulex fibre channel cards whereas LIO
supports only Qlogic target drives for now, and it is still in its beta stages of
development. SCST supports the SCSI RDMA Protocol
(SRP), and claims to be ahead in terms of
development with respect to Fibre
Channel over Ethernet (FCoE), LSI's
Parallel/Wide SCSI Fibre Channel, and Serial Attached SCSI
(SAS). It already has
support for IBM's
pSeries Virtual SCSI. Companies such as Scalable Informatics, Storewize, and Open-e
have developed PnP appliance products which rely on these target
transports based on SCST.
SCST supports notifications of session changes using asynchronous event
notification (AEN). AEN is a protocol feature that may be used
by SCSI targets to notify a SCSI initiator of events that occur in the target,
when the target is not serving a request. This enables
initiators to be notified of changes at the target end, such as devices added,
removed, resized, or media changes. This way the initiators can see any target
changes in a plug-and-play manner.
The SCST developers claim that their design conforms to more SCSI standards
in terms of robustness
and safety. The SCSI protocol requires that if an initiator clears a
held by another initiator, the reservation holder must be notified about the
reservation clearance or else several initiators could change reservation data,
ultimately corrupting it. SCST is capable of implementing safe RESERVE/RELEASE
operations on devices to avoid such corruption.
According to the SCSI protocol, the initiator and target can communicate with
each other to decide on the transfer size. An incorrect transfer size
communicated by the initiator can lead to target device lockups or a crash.
SCST safeguards against miscommunication of transfer sizes or transfer
directions to avoid such a situation. The code claims to have a good memory
management policy to avoid out-of-memory (OOM) situations. It can also limit the
number of initiators that can connect to the target to avoid resource usage by
too many connections. It also offers per-portal visibility control, which
means that it can be configured in such a way that a target is visible to a
particular subset of initiators only.
The LIO project began with the iSCSI design as its core objective, and
created a generic SCSI target subsystem to support iSCSI. Simplicity has
been a major
design goal and hence LIO is easier to understand. Beyond that, the LIO
developers have shown more willingness to work with the kernel developers
as James pointed out
to SCST maintainer Vladislav Bolkhovitin:
Look, let me try to make it simple: It's not about the community you
bring to the table, it's about the community you have to join when you
become part of the linux kernel. The interactions in the wider
community are critical to the success of an open source project. You've
had the opportunity to interact with a couple of them: sysfs we've
covered elsewhere, but in the STGT case you basically said, here's our
interface, use it. LIO actually asked what they wanted and constructed
something to fit. Why are you amazed then when the STGT people seem to
The LIO project also boasts of features which are either not present in SCST
or are in early development phases. For example, LIO supports asymmetric
assignment (ALUA). ALUA allows a target administrator to manage the access
states and path attributes of the targets. This allows the multipath routing
method to select the best possible path to optimize usage of available
bandwidth, depending on the current access states of the targets. In other
words, the path taken by the initiator in a multipath environment can be
manipulated by target administrator by changing the access states.
LIO supports Management
Information Base (MIB) which makes management of SCSI
devices simpler. The SCSI target devices export management
information values described in SCSI MIB RFC-4455 which is
picked up by an SNMP agent. This feature extends to iSCSI devices and is
beneficial in managing a storage network with multiple SCSI devices.
An error in the iSCSI connection can happen at three different levels: the
session, digest, or connection level. Error recovery can be initiated
at each of these levels, which makes sure that the recovery is made at the
current level, and the error does not pass through to the next one. Error
recovery starts with detecting a broken connection. In reponse, the iSCSI
establishes another TCP connection to the target, then it informs the
target that the SCSI command path is being changed to
the new TCP connection. The target can then continue processing
SCSI commands on the new TCP connection. The upper level SCSI driver remains
unaware that a new TCP connection has been established and that control has
been transferred to the new connection. The iSCSI Session remains active during
the period and does not have to be reinstated. LIO supports a maximum Error
Recovery Level (ERL) of 2, which means that it can recover errors at the
session, digest, or connection levels. SCST supports an ERL of 0, which
it can recover from session-level errors only and that all connection
oriented errors are communicated to the SCSI driver.
LIO also supports "multiple connections per session" (MC/S). MC/S allows the
initiator to open multiple connections between the initiator and target, either
on the same or a different physical link. Hence, in case of a failure of
the established session can use another path without terminating the session.
MC/S can also be used for load balancing across all established connections.
Architectural session command ordering is preserved across those communication
The LIO project also claims that its code is used in a number of appliance products and
deployments though the user base does not seem to be as varied as that of
No comparison can be complete without a performance comparison. SCST developers
have released their performance numbers from time to time. However, all their
numbers were compared against STGT. The SCST comparison
page speaks of SCST performing better than LIO, but the results were drawn on
source-code study and not using real-world tests. SCST blames LIO for not
releasing performance numbers, and there exist no performance data (to my
knowledge) which would compare apples to apples.
The decision has finally been made, though, with quite a bit of opposition.
Now comes the task of getting all the niche features which LIO lacks to
be ported from SCST to LIO. While the decision was contentious, it is yet
another example of the difficulty of getting something merged without
being able to cooperate with the kernel development community.
to post comments)