SCALE: The Hockeypuck key server

At SCALE 11x in Los Angeles, Gazzang's Casey Marshall presented his work developing Hockeypuck, an alternative public PGP keyserver. Although the company developed Hockeypuck to support one of its own products, the AGPL-licensed server is capable of running a standalone key service, and is compatible with Synchronizing Key Server (SKS), the tool used by almost all public key services.

Keyservers are a critical component in the public key infrastructure (PKI), even though they rarely attract significant attention. They enable PGP-aware client applications to search for and retrieve users' public keys, which is what enables parties to encrypt messages to one another without prior agreement. In addition to sender's and recipient's keys, PGP relies on a "web of trust" built up by verifiable signatures from other PGP keys. Aside from secure private email, PGP encryption is also used in an increasing number of other tasks, such as verifying software package signatures. Marshall observed that this system is in essence a globally distributed social network; the Internet's keyservers share identities in a distributed fashion across a truly global pool. Because keyservers distribute the load and can synchronize, it is very hard for an attacker to tamper with or otherwise undermine the keys.

SKS is by far the most commonly used keyserver, Marshall said, and it offers a powerful set of features. It uses an efficient "set reconciliation" algorithm to keep the global database of keys in sync between remote peers, and it uses Berkeley DB for data storage. Although there is an older, email-based protocol for querying and retrieving keys, SKS is built around the HTTP Keyserver Protocol (HKP), which Marshall described as being RESTful before REST was popular.

Enter Hockeypuck

Marshall got into keyserver development while working on a Gazzang product called zTrustee. The product is a closed source storage service that uses OpenPGP keys to encrypt data. Because the service relies on generating and storing separate keys for separate objects, it quite naturally processes keys in heavy volume, which is not the typical workload for a keyserver. The company has been using SKS to distribute keys to clients, he said, but SKS is very write-heavy, and under sufficient load it was found to cause unacceptable delays.

Hoping to improve on the situation, Marshall started writing Hockeypuck. It is not yet ready to replace SKS in zTrustee, but interested developers can test it out. The project is hosted at Launchpad.net and mirrored on Github. Binary packages are already available for Ubuntu 13.04, and there is a publicly accessible instance of the server running at hockeypuck.gazzang.net. GPG users can query the server by supplying it as a command line switch, for example:

     gpg --keyserver hockeypuck.gazzang.net --search-keys Santa

The public server's web interface presents a minimalist "Google-style" search page (which, as he pointed out, includes an "I'm Feeling Lucky" button that is not really intended for serious usage). Hockeypuck does not participate in the global set reconciliation algorithm of the SKS keyservers, but the public server was initialized with a dump file provided by an SKS server six months ago, so it contains a significant subset of the total global key pool.

Hockeypuck is written in Go, which Marshall said he selected for several reasons. Its simplicity and modularity make it fun to write in, he said, but it also offers useful features and high-performance message passing. The wide assortment of libraries available included an OpenPGP implementation, which he used, although he noted that there are not many OpenPGP implementations to choose from—most PGP development takes the form of additional services built on top a small set of OpenPGP stacks.

Lessons learned

At the moment, Hockeypuck uses MongoDB for storage; Marshall said he would be adding PostgreSQL support next, and perhaps other database connectors later. The server architecture is fairly straightforward, he said. A separate goroutine handles each HTTP request, opening a channel to a worker that queries the database. Scaling the system could be as simple as running one worker per CPU, or more sophisticated techniques could be employed depending on the database backend.

Indeed, Marshall said, the choice of MongoDB has come with its share of problems. It was easy to develop with, he said; Go even has a library for it. "You give it a struct, and you get a struct back." But not being a real relational database imposes limitations, starting with the fact that you cannot index something just because you want to. The Hockeypuck database is indexed on the uid field (which contains names and email addresses), thus it cannot also run searches on other fields (like key-ID); a truly full-text search is not possible. He also found it necessary to reverse the order of key fingerprints, placing the shorter key-ID at the beginning of the record so that it can be read and searched faster. Maintaining performance has also been tricky, he said; loading data into MongoDB is very quick, but updates must be aggregated for write efficiency. Ultimately, he concluded, MongoDB makes it very easy to write database applications, but it shifts more work onto configuration and deployment.

Maintaining the public Hockeypuck server has also imparted its share of lessons, he said. For example, an unknown user downloaded Marshall's own key, added new email addresses as uid fields, then re-uploaded the key to the server. An OpenPGP client application would not have been fooled by the deception because the grafted-on fields were not signed by the primary key, but the incident pointed out to Marshall that Hockeypuck needed to do its part as well. He quickly added code that checked the signatures on uploads, and reloaded the SKS key database just to be on the safe side. Technically, he observed, keyservers themselves are not meant to be trusted entities—the keys are designed to be verified or rejected cryptographically—but maintaining a tidy and valid database is important too.

Keys to the future

Hockeypuck's key loading is quite fast already, Marshall said; it can load about two million keys in 24 hours. Queries, in turn, are "reasonably fast," and the database appears to be the bottleneck. But apart from increasing performance, he has several other important items on his to-do list. For example, one improvement is support for OpenPGP version 3 signatures. Version 4 signatures are the standard today, but version 3 signatures can still be found in the wild.

A far bigger task (which Marshall regards as the missing "killer feature") is implementing the SKS reconciliation algorithm. This will allow Hockeypuck to interoperate with the global pool of public keys. He has started work on an implementation of the algorithm (that he named conflux), which he hopes will be general-purpose enough to serve as a synchronization library outside of the keyserver itself. Conflux is "getting there," he said; the mathematical portions are passing unit tests, but he still has work to do on the network protocol itself.

Further down the line, he speculated that Hockeypuck could serve as an SSH keyserver as well, and perhaps work with other authentication schemes like Password Authenticated Key Exchange by Juggling (J-PAKE) or convergence.io.

Trust stuff

Despite the fact that Hockeypuck "competes" with SKS, Marshall said he has found the SKS community to be very friendly, and many were excited to hear about Hockeypuck and its implementation of the set reconciliation algorithm. An independent implementation of the feature is good news for any project, but especially for infrastructure projects like SKS, because the "web of trust" that it implements is so crucial.

Marshall concluded his talk by asking attendees to examine the web of trust and how it functions. We trust different identity credentials for very different reasons, he said: we trust PGP keys because of either the signatures of other PGP users or our participation in key-signing events; we trust SSH keys because they are the same key we encountered the first time we connected; we trust SSL/TLS certificates because they contain a signature from a certificate authority that our browser trusts. Our trust will have a stronger foundation if it includes multiple factors, he said; perhaps, for example, PGP keys need to incorporate notions of identity beyond email addresses alone.

Keyservers could also play a role in advancing the conversation about identity management, he suggested. As he noted at the beginning of the session, the SKS global key pool has functioned as a decentralized social network for years—perhaps there are ways to leverage it, such as linking PGP keys to OpenID or OAuth accounts, or to make SSH user authentication as widely accepted as SSH host authentication already is.

Of course, PGP is still in use by just a fraction of global email users; its critics have long argued that public key encryption and the PKI are too complicated for mass appeal. That is a difficult claim to prove, particularly since it is hard to disentangle the ideas of PKI from its client software implementations. But even for those who understand and use PGP on a regular basis, the accidental monoculture of SKS keyservers poses its own potential risks. Hockeypuck might never overtake SKS in popularity, but by offering an additional choice and by shining new light on HKP and other factors, it may strengthen critical pieces of PKI anyway.