At SCALE 11x
in Los Angeles, Gazzang's Casey Marshall presented
his work developing Hockeypuck, an alternative public PGP keyserver.
Although the company developed Hockeypuck to support one of its own
products, the AGPL-licensed server is capable of running a standalone
key service, and is compatible with Synchronizing
Key Server (SKS), the tool used by almost all public key services.
Keyservers are a critical component in the public key
infrastructure (PKI), even though they rarely attract significant
attention. They enable PGP-aware client applications to search for
and retrieve users' public keys, which is what enables parties to
encrypt messages to one another without prior agreement. In addition
to sender's and recipient's keys, PGP relies on a "web of trust" built
up by verifiable signatures from other PGP keys. Aside from secure
private email, PGP encryption is also used in an increasing
number of other tasks, such as verifying software package
signatures. Marshall observed that this system is in essence a
globally distributed social network; the Internet's keyservers share
identities in a distributed fashion across a truly global pool.
Because keyservers distribute the load and can synchronize, it is very
hard for an attacker to tamper with or otherwise undermine the keys.
SKS is by far the most commonly used keyserver, Marshall said, and
it offers a powerful set of features. It uses an efficient "set
reconciliation" algorithm to keep the global database of keys in sync
between remote peers, and it uses Berkeley DB for data storage.
Although there is an older, email-based protocol for querying and
retrieving keys, SKS is built around the HTTP Keyserver Protocol
(HKP), which Marshall described as being RESTful before REST was
Marshall got into keyserver development while working on a Gazzang
product called zTrustee. The product is a closed source
storage service that uses OpenPGP keys to encrypt data. Because the
service relies on generating and storing separate keys for separate
objects, it quite naturally processes keys in heavy volume, which is
not the typical workload for a keyserver. The company has been using
SKS to distribute keys to clients, he said, but SKS is very
write-heavy, and under sufficient load it was found to cause
Hoping to improve on the situation, Marshall started writing
Hockeypuck. It is not yet ready to replace SKS in zTrustee, but
interested developers can test it out. The project is hosted at
Launchpad.net and mirrored on Github.
Binary packages are already available for Ubuntu 13.04, and there is a
publicly accessible instance of the server running at hockeypuck.gazzang.net. GPG
users can query the server by supplying it as a command line switch,
gpg --keyserver hockeypuck.gazzang.net --search-keys Santa
The public server's web interface presents a minimalist
"Google-style" search page (which, as he pointed out, includes an "I'm
Feeling Lucky" button that is not really intended for serious usage).
Hockeypuck does not participate in the global set reconciliation
algorithm of the SKS keyservers, but the public server was initialized
with a dump file provided by an SKS server six months ago, so it
contains a significant subset of the total global key pool.
Hockeypuck is written in Go, which
Marshall said he selected for several reasons. Its simplicity and
modularity make it fun to write in, he said, but it also offers useful
high-performance message passing. The wide assortment of libraries
available included an OpenPGP implementation, which he used, although
he noted that there are not many OpenPGP implementations to choose
from—most PGP development takes the form of additional services
built on top a small set of OpenPGP stacks.
At the moment, Hockeypuck uses MongoDB for storage; Marshall said
he would be adding PostgreSQL support next, and perhaps other database
connectors later. The server architecture is fairly straightforward,
he said. A separate goroutine handles each
HTTP request, opening a channel to a worker that queries the database.
Scaling the system could be as simple as running one worker per CPU,
or more sophisticated techniques could be employed depending on the
Indeed, Marshall said, the choice of MongoDB has come with its
share of problems. It was easy to develop with, he said; Go even has a
library for it. "You give it a struct, and you get a struct back."
But not being a real relational database imposes limitations, starting
with the fact that you cannot index something just because you want
to. The Hockeypuck database is indexed on the uid field
(which contains names and email addresses), thus it cannot also run
searches on other fields (like key-ID); a truly full-text search is
not possible. He also found it necessary to reverse the order of key
fingerprints, placing the shorter key-ID at the beginning of the
record so that it can be read and searched faster. Maintaining
performance has also been tricky, he said; loading data into MongoDB
is very quick, but updates must be aggregated for write efficiency.
Ultimately, he concluded, MongoDB makes it very easy to write database
applications, but it shifts more work onto configuration and
Maintaining the public Hockeypuck server has also imparted its
share of lessons, he said. For example, an unknown user downloaded
Marshall's own key, added new email addresses as uid fields,
then re-uploaded the key to the server. An OpenPGP client application
would not have been fooled by the deception because the grafted-on
fields were not signed by the primary key, but the incident pointed
out to Marshall that Hockeypuck needed to do its part as well. He
quickly added code that checked the signatures on uploads, and
reloaded the SKS key database just to be on the safe side.
Technically, he observed, keyservers themselves are not meant to be
trusted entities—the keys are designed to be verified or
rejected cryptographically—but maintaining a tidy and valid
database is important too.
Keys to the future
Hockeypuck's key loading is quite fast already, Marshall said; it
can load about two million keys in 24 hours. Queries, in turn, are
"reasonably fast," and the database appears to be the bottleneck. But
apart from increasing performance, he has several other important
items on his to-do list. For example, one improvement is support for
OpenPGP version 3 signatures. Version 4 signatures are the
standard today, but version 3 signatures can still be found in the
A far bigger task (which Marshall regards as the missing "killer
feature") is implementing the SKS reconciliation algorithm. This will
allow Hockeypuck to interoperate with the global pool of public keys.
He has started work on an implementation of the algorithm (that he
named conflux), which
he hopes will be general-purpose enough to serve as a synchronization
library outside of the keyserver itself. Conflux is "getting there,"
he said; the mathematical portions are passing unit tests, but he
still has work to do on the network protocol itself.
Further down the line, he speculated that Hockeypuck could serve as
an SSH keyserver as well, and perhaps work with other authentication
schemes like Password
Authenticated Key Exchange by Juggling (J-PAKE) or convergence.io.
Despite the fact that Hockeypuck "competes" with SKS, Marshall said
he has found the SKS community to be very friendly, and many were
excited to hear about Hockeypuck and its implementation of the set
reconciliation algorithm. An independent implementation of the
feature is good news for any project, but especially for
infrastructure projects like SKS, because the "web of trust" that it
implements is so crucial.
Marshall concluded his talk by asking attendees to examine the web
of trust and how it functions. We trust different identity
credentials for very different reasons, he said: we trust PGP keys
because of either the signatures of other PGP users or our
participation in key-signing events; we trust SSH keys because they
are the same key we encountered the first time we connected; we trust
SSL/TLS certificates because they contain a signature from a
certificate authority that our browser trusts. Our trust will have a
stronger foundation if it includes multiple factors, he said; perhaps,
for example, PGP keys need to incorporate notions of identity beyond
email addresses alone.
Keyservers could also play a role in advancing the conversation
about identity management, he suggested. As he noted at the beginning
of the session, the SKS global key pool has functioned as a
decentralized social network for years—perhaps there are ways to
leverage it, such as linking PGP keys to OpenID or OAuth accounts, or
to make SSH user authentication as widely accepted as SSH host
authentication already is.
Of course, PGP is still in use by just a fraction of global email
users; its critics have long argued that public key encryption and the
PKI are too complicated for mass appeal. That is a difficult claim to
prove, particularly since it is hard to disentangle the ideas of PKI
from its client software implementations. But even for those who
understand and use PGP on a regular basis, the accidental monoculture
of SKS keyservers poses its own potential risks. Hockeypuck might
never overtake SKS in popularity, but by offering an additional choice
and by shining new light on HKP and other factors, it may strengthen
critical pieces of PKI anyway.
to post comments)