When a system is compromised, the attackers may try to cover their tracks so
that the administrator is not alerted to the attack. One way for an
attacker to hide is by removing log file entries that might lead an
administrator (or a log file analyzer) to notice. A new feature in the
systemd journal, "forward secure sealing" (FSS) is meant to detect log file
Traditionally, administrators have written log files to external systems
across the network or to a local printer—though paper is notoriously
hard to grep—to defeat log file tampering. As long as the
other system is not compromised, and log file lines are written
immediately, an attacker can't help but leave their "fingerprints" behind.
But FSS provides a way to at least detect tampering using only a single
system, though it won't provide all of the assurances that external logging can.
Systemd developer Lennart Poettering announced
FSS on August 20. The basic idea is that the binary logs handled by the
can be "sealed" at regular time intervals. That seal is a cryptographic
operation on the log data such that any tampering prior to the seal can be
detected. So long as a sealing operation happens before the attacker gets
a chance to tamper with the logs, their fingerprints will be sealed with
the rest of the log data. They can still delete the log files entirely,
but that is likely to be noticed as well.
The algorithm for FSS is based on "Forward Secure Pseudo Random
Generators" (FSPRG), which comes from some post-doctoral research by
Poettering's brother Bertram. The paper on FSPRG has not been published but will
be soon, according to (Lennart) Poettering.
The announcement on Google+ and its long comment thread do give some
details, however. FSS is based on two keys that are generated using:
One key is the "sealing key" which is kept on the system, and the other is
the "verification key" which should be securely stored elsewhere. Using
the FSPRG mechanism, a new sealing key is generated periodically using a
non-reversible process. The old key is
then securely deleted from the system after the change.
The verification key can be used to calculate the sealing key for any given
time range. That means that the attacker can only access the current
sealing key (which will presumably be used for the next sealing operation),
while the administrator can reliably generate any sealing key to verify
previous log file seals. Changing log file entries prior to the last seal
will result in a verification failure.
As a bell—or perhaps a whistle—the key generator can
create a QR code of the verification key, which can be scanned so that the
key doesn't have to be typed in.
after the system is compromised is under control of the attacker, as was
pointed out multiple times in the comments. That
means that local logs cannot be relied on after that point, but it also
applies to remotely stored—or even printed—log files. The
latter two methods do protect against an attacker simply deleting the local
log files, though.
By default, FSS will seal the logs every 15 minutes, but that can be changed
at key generation time with a flag: "--interval=10s" for example.
The system clock time is used in the generation of each new sealing key,
why the interval must be specified when the keys are generated. The
default value surprisingly leaves a rather large window for an attacker who
turns to altering the log file, though. One also
wonders if subtle (or not so subtle) manipulations of the system clock
might be a way to subvert or otherwise interfere with the key generation.
Securely deleting the old sealing
key is handled by setting the FS_SECRM_FL and FS_NOCOW_FL
file attributes, which may or may not be implemented by the underlying
filesystem. That could potentially lead to leaks of previous sealing keys,
which would allow an attacker to make changes to earlier entries.
Obviously, losing control of the verification key means that all bets are
The code is available already in the systemd Git
repository. Poettering notes that it will also be available in Fedora
FSS is an interesting feature that will likely prove useful for some
administrators. It certainly doesn't solve all of the problems with
detecting attackers or compromised systems, but it could definitely help by
red flags. There is more to do, of course, starting with a security audit
of the code—more eyes can only be helpful in ferreting out any holes
in the algorithm or implementation. Once that's done, administrators can
feel more confident that their log files aren't undetectably changing out
from under them—at least if they are using the systemd journal.
to post comments)