Development

JMAP — reinventing IMAP

The Internet Message Access Protocol version 4 (IMAP4) is a mainstay of remote email access. The vast majority of email clients support the use of IMAP4 to read and manage email. Even web-based email clients that may not really need remote access often use IMAP4, whether to simplify implementation, to improve security through clear separation, or to enhance scalability. IMAP4 recently passed its 21st birthday and, while it is still the king of email access, there is a contender in the wings that appears to be preparing to make a takeover bid: JMAP, the JSON Mail Access Protocol.

JMAP is being developed and promoted by FastMail, a cloud email provider with a history of open-source support, at least of the "open core" variety: it has contributed over half the Git commits to the Cyrus IMAP server in the last five years and is a significant sponsor of the "Roundcube Next" project to build a new webmail client. JMAP, as Bron Gondwana explained in his lightning talk at linux.conf.au 2016, started life as an internal API used by FastMail between its JavaScript in-browser email client and its back-end servers. Positive experiences with open source convinced the company that an open specification would bring more value than keeping the protocol proprietary, so it created a more formal specification and started migrating the internal code base toward it.

Why replace IMAP4?

The first IETF RFC describing IMAP4 was published in December 1994, but it has not been static since then. A quick check of the RFC index shows over 60 RFCs that propose revisions, discuss extensions, or provide clarifications for the specification; the most recent being RFC7377, which was published in October 2014.

This ongoing development and revision could mean that IMAP4 has or will gain all the features that it needs to remain relevant and functional. Or it could mean that it has become a bloated monster with little internal coherence, and multiple implementations each supporting some, but not all, of the extensions. Gondwana was clearly inclined to the latter interpretation, but the specifics he gave were primarily about scope.

A modern email client needs more than just access to messages. It requires the ability to send email (using a protocol like SMTP or LMTP), look up a contacts database (possibly using LDAP or CardDAV), and often to query and update a calendar (perhaps with CalDAV). Managing all of these independently not only increases the configuration burden but is much more likely to run afoul of firewalls or related networking problems. Having a single protocol accessing all this data using a single port is likely to either work smoothly or fail completely, both outcomes seen as preferable to a mix of bits that work and bits that don't.

JMAP basics

JMAP addresses those concerns by defining a single protocol that can access all of the mentioned services and data types, and possibly more in the future, in a fairly uniform way. Configuration is entirely automatic: a DNS lookup for an "SRV" record for _jmaps._tcp.example.com will report a server and port to connect to for someone with an @example.com email address. An https POST request to a well-known URI (/.well-known/jmap) allows JavaScript Object Notation (JSON) messages to be sent and replies received. These messages can create and authenticate a session, which results in an authentication token. This token can then authenticate future connections to any of a small number of URIs that allow different actions to be performed and data to be extracted.

There are upload and download URIs for sending and receiving attachments, raw RFC2822 mail files, and any binary large objects (BLOBs). There are also an event source URI for receiving asynchronous change notifications and an API URI for general request/response interaction. Each API message can contain multiple requests that are processed in sequence, each of which can generate one or more responses, all of which are combined into a single reply. Allowing multiple requests and replies in a single message reduces the number of round-trips to the server to gather complete information, which is important for getting good responsiveness on a high-latency network link.

The JMAP data model identifies a small number of object types, some that exist in their own right and others that are synthesized. The former group includes Mailboxes and Messages, Contact Groups and Contacts, and Calendars and Calendar Events. Objects of these types are each a collection of named attributes that can be requested separately by the client. One particular attribute that is common to all types is an ID, which is a string value, assigned by the server, that is unique within the type and stable for the object's lifetime. Using this ID, objects of a given type (e.g. Foo) can be retrieved (GetFoos), modified if not immutable, and deleted (SetFoos). Each type has a "state" value that is effectively incremented whenever any object of that type is changed, and it is possible to ask for any updates since a particular state (GetFooUpdates). This simplifies the task of keeping a client-side cache up to date with any changes on the server. JMAP also includes the option of a side-band protocol to proactively notify the client of changes so that it doesn't need to poll periodically. This can use a platform-specific "push" mechanism, if available, or the client can make a persistent connection to the event source URI.

For each of the base types (Messages, Contacts, Calendar Events) there is a synthetic List type that is used to represent the results of a search for that type within a particular account. These search results are treated like a separate data type, as they are expected to be cached by the server. The client can request a "window" into a list by indicating a start position and a limit on the number of entries to report. In the case of MessageLists, there is also an Updates function, so getMessageListUpdates will report any changes to the result of a previous search if those results are still cached by the server.

JMAP extras

While this pattern of containers, lists, and objects covers much of the data model, it is not quite complete. There are three exceptions: Accounts, Threads, and SearchSnippets.

The "Accounts" abstraction allows a single user to have access to multiple accounts; there will always be a primary account but there may also be group accounts or accounts that the user been delegated responsibility for. Accounts are a bit like the three container types in that it is possible to get a list of all the accounts in a single request, though no search filtering is possible. Unlike the container, it is not possible to get a list of Updates, though a client can discover if the list of accounts has changed at all.

"Threads" are lists of related messages. Each message is assigned a threadId by a mechanism determined by the server rather than by the protocol and a given thread is all of the messages with that threadId. This could nearly be implemented with the getMessageList function since it can filter based on threadId, but threads are such a central concept in email handling that the protocol can be more efficient if they are a first-class citizen. For example, the getMessageList function can be asked to fetchMessages in which case it returns not just the list of message IDs, but also (as a separate response) the actual messages themselves (or more precisely: a given set of attributes for each message). In a similar way, it can be asked to fetchThreads and will report a list of all threads found by the search, each represented as a flat list of message IDs without any parent/child relationships exposed. These cannot just be a property of the messages since there will likely be a different number of threads than messages. Having a separate object type makes this easy.

"SearchSnippets" are a bit like Threads in that they exist to optimize a particularly valuable part of the user experience. One of the attributes that can be fetched for any message is a "preview", which is a line or two of the most relevant text, possibly skipping quoted text or salutations. For a specific search filter and a specific message, a SearchSnippet contains an alternate preview text that includes sections of the message that match search strings in the filter and has those strings marked for easy highlighting. It also includes an alternate version of the Subject with similar markup. A getMesssageList request can include a fetchSnippets directive so that these snippets are provided to the client.

Messages, mailboxes, and tags

Messages are undoubtedly the richest and most interesting objects managed by JMAP. Messages are mostly immutable, with the only permitted changes being to change the list of mailboxes the message is a member of and to set or clear one of four flags: isUnread, isFlagged, isAnswered, and isDraft. These roughly correspond to the \Seen, \Flagged, \Answered, and \Draft flags supported by IMAP4. The other two IMAP4 flags: \Deleted and \Recent are not supported by JMAP, presumably because they aren't particularly useful. \Deleted is only needed for a two-stage delete, which is adequately handled by moving a message to a "Trash" folder. \Recent has semantics that are not particularly user-friendly: it is quite possible for the server to be required to clear \Recent before the user has had a chance to see any hint of the message at all.

While IMAP4 allows arbitrary user-defined flags to be assigned to a message, JMAP does not. Instead, it allows a message to be attached to an arbitrary number of user-defined mailboxes. This allows the email client to treat mailboxes like folders, tags, or even both. This last suggests a slight weakness in the protocol. It is quite possible that a user would want some mailboxes to behave like tags and others to behave like folders: when dragged to a folder, a message would be removed from the current folder, while dragging to a tag would just add that tag. A client cannot support this distinction without imposing its own interpretation of mailbox names, such as assuming that any child of the "tags" mailbox should be treated as a tag.

A JMAP Mailbox can have a mustBeOnlyMailbox flag set, in which case messages can only be in that mailbox if they aren't in any other. This can force a mailbox to behave like a folder, but then none of the messages in it can belong to any tag-like mailboxes, so it seems rather restrictive.

Mailbox roles and sending email

Only five years ago, IMAP gained the concept of special-use mailboxes thanks to RFC6154. This defined mailbox attributes like \Drafts, \Sent, \Junk, and \Trash so that an IMAP client could use those flags instead of depending on hard-coded (English language) names. JMAP has a matching concept, referred to as mailbox "roles", with some minor variations such as \Junk being replaced by spam. One role that deserves special attention is the outbox role.

When a message is placed in an outbox mailbox, it must have the isDraft flag set; the implication is that it has been queued for delivery. At some future time, which may be immediately or may be when the time in the "Date:" header is reached, the JMAP server should attempt to deliver the message. Once this attempt completes, the message is deleted from the outbox and re-created in the folder with sent role. The specification is not explicit on what happens if the delivery attempt fails. The likely implication is that a delivery status email would appear in the inbox. Given how easy it can be to miss or be confused by such messages, protocol support that would allow the client to highlight and help resolve failures would be quite valuable.

Spam filtering is important in any modern email application and JMAP provides some help in this direction. Aside from the spam role already mentioned, which makes it easy to find spam, there is a dedicated "reportMessages" request that can be used to report that a message is, or is not, spam in the eyes of the user. This is typically used to train the per-user spam filter. Unfortunately, the specification does not spend time justifying various decisions, so we do not know why this request is needed rather then allowing spam status to be set by simply moving messages into or out of the spam mailbox. All we know is that the specification is quite explicit that moving messages like this would be a separate operation from reporting their spam status.

Managing MIME

JMAP is able to handle messages as uninterpreted blobs of RFC2822 text or as completely parsed messages with various headers and details of attachments available as attributes in the JSON encoding. When collecting a message from the server, a client can ask for specific fields and will get the relevant data in pure UTF-8 with no escaping or transport encoding. When uploading a message to the server, it can list the content of various fields and describe attachments that have previously been uploaded as ordinary files, and the server will combine all the parts together using the appropriate encoding. There is even a provision to extract a textBody from a message that only contains an HTML version, or to get an HTML version of plain text.

While this MIME (Multipurpose Internet Mail Extensions) transcoding is useful, it appears to be focused on matching common usage rather than precisely mirroring the MIME specification. MIME supports generic recursive multipart structuring of messages, where any MIME part can itself be multipart, typically multipart/mixed for attachments, multipart/alternative for different renderings of the same message (text or HTML), or multipart/related for HTML combined with some image files. While the standard allows multiple levels of multiparts and a variety of subtypes of multipart, JMAP knows little of this. There is only one pair of alternatives, text or HTML. Together with this, there are attachments, which might be tagged inline in the multipart/related case. One specific type of attachment — an email message — can result in a recursive structure and JMAP handles this correctly, but no other recursion is apparent. This contrasts with IMAP4, which does support reporting of nested structure in the BODYSTRUCTURE response.

While this is theoretically limiting, it does cover the vast majority of email. Since a client can deal directly with the undecoded RFC2822 message if it chooses, it may be an acceptable tradeoff. Unfortunately, there is no easy way for such a client to determine if there might be anything extra to decode, so it cannot know if it needs to fetch the raw message.

One place where this might be a real problem is with signed or encrypted email. The JMAP specification makes no mention of S/MIME, the Secure MIME standard. While there may be difficult issues around deciding whether a server should be trusted to sign or decrypt messages, it would be nice to have the option, and it would be nice if the server told the client when the message might need decrypting or if a signature needs checking.

What else is missing from JMAP?

Email involves such a rich and varied experience that it is probably impossible for any protocol to really cover everybody's favorite feature. There are, however, two missing features worthy of note that both relate to the processing of mail as it arrives — an area currently unaddressed by JMAP. One is the ability to register a "vacation" message to be automatically sent in reply to incoming messages. This was raised on the discussion list and a possible protocol enhancement to cover it was discussed. This suggests that there is an openness to enhance the protocol to meet requirements.

The other feature, one close to my heart, is saved searches. Whether you use procmail, Sieve-based filtering, or a similar mechanism to direct new messages to different mailboxes based on content, or use Notmuch or similar to perform searches at the moment you open a virtual mailbox, there is a clear need to be able to save rules or searches somewhere; having this exposed in JMAP would be quite helpful. This is even mentioned in the JMAP specification, though clearly as an aside. In the section on negotiating extension support, there are some hypothetical example extensions including "com.fastmail.savedSearch:4". Clearly this need has been thought about, but no clear resolution has been found. It may well be a challenge to provide support in the protocol that is useful to clients without being an undue burden on servers. But it would be a poor email access protocol in this day and age that didn't provide for saved searches in some form.

Status

JMAP is clearly something that FastMail wants people to play with. It has released a proxy server, written in Perl, that provides a gateway between JMAP on the client side, and IMAP and SMTP on the server side. The various parts can be downloaded from GitHub and built without too much difficulty, or a hosted version can be used — though, obviously, don't give that the password of an important email account. The downloads include a simple JavaScript email client that talks JMAP and can be used to complete the experience.

While JMAP is certainly interesting, and ticks a lot of the right boxes, but it is far from certain that it can gather sufficient momentum that anyone other than FastMail and a few niche players will invest in it. To achieve that, it would need to be a lot better than "good enough"; it would need to be "compelling", and probably also "exciting". The proxy server should help to reduce the typical chicken-and-egg problem that new protocols face by making it possible to use a JMAP client even if your service provider doesn't support JMAP natively. Whether this will be enough to encourage people to join the adventure is hard to know. We'll just have to wait and see.

Comments (37 posted)

Quotes of the week

— Sean Stephens (Thanks to Paul Wise.)

— Matthew Weaver

Comments (18 posted)

KDE Frameworks 5.20.0 released

Version 5.20.0 of the KDE Frameworks library collection has been released. Among the changes, the Breeze Icons library has been updated (with many new icons added), a number of older dependencies have been removed from KActivities, and many new features have been added to Plasma.

Full Story (comments: none)

Qt 5.6 released

Version 5.6.0 of the Qt framework has been released. Most notably, this is a "Long Term Supported" (LTS) release, which will receive security updates and bug fixes for the next three years. Among the functional changes found in 5.6 are support for high-DPI displays, an updated virtual keyboard with handwriting-recognition support, a new Qt Location module for map and navigation features, and significant updates to Qt WebEngine.

Comments (1 posted)

Development newsletters from the past week

• What's cooking in git.git (March 9)
• Git Rev News (March 16)
• LLVM Weekly (March 14)
• OCaml Weekly News (March 15)
• OpenStack Developer Digest (March 11)
• Perl Weekly (March 14)
• PostgreSQL Weekly News (March 13)
• Python Weekly (March 10)
• Ruby Weekly (March 11)
• This Week in Rust (March 14)
• Tor Weekly News (March 14)
• Wikimedia Tech News (March 14)

Comments (none posted)

A policy statement on open-source software from the White House

The White House has announced a draft policy addressing how the U.S. federal government will share and release custom software. "This policy requires that, among other things: (1) new custom code whose development is paid for by the Federal Government be made available for reuse across Federal agencies; and (2) a portion of that new custom code be released to the public as Open Source Software (OSS)."

The full policy document is available at sourcecode.cio.gov, where it has been made available for public comment. The relevant passage regarding public source releases begins by outlining a pilot program. "Each covered agency shall release at least 20 percent of its newly-developed custom code each year as OSS. Custom code is defined as code for all custom software projects, modules, and add-ons that are self-contained. [...] Although the minimum requirement for OSS release is 20 percent of custom code, covered agencies are strongly encouraged to publish as much custom-developed code as possible to further the Federal Government’s commitment to transparency, participation, and collaboration."

Comments (27 posted)

FSF: A preliminary analysis of High Priority Projects feedback

The Free Software Foundation (FSF) has posted an initial analysis of the public feedback submitted in response to its call for suggestions about what software projects deserve to be on the high-priority projects list. Several of the existing projects on the list are likely to be removed, such as a replacement for Google Earth and an automatic-transcription application. Multiple potential additions to the list are also described, such as a free-software "Siri"-like personal assistant and a free-software implementation of advanced PDF features. Further input is welcome; the page notes that the FSF "will strive to recommend projects that are actionable. Such projects document ways for members of the free software community to get involved and make the project succeed, with any kind of concrete contributions, from money donation, to code patches, advocacy, etc."

Comments (26 posted)

Graber: LXD 2.0: Introduction to LXD

Stéphane Graber provides an introduction to LXD. "LXD focuses on system containers, also called infrastructure containers. That is, a LXD container runs a full Linux system, exactly as it would be when run on metal or in a VM. Those containers will typically be long running and based on a clean distribution image. Traditional configuration management tools and deployment tools can be used with LXD containers exactly as you would use them for a VM, cloud instance or physical machine."

Comments (3 posted)