JMAP — reinventing IMAP
The Internet Message Access Protocol version 4 (IMAP4) is a mainstay of remote email access. The vast majority of email clients support the use of IMAP4 to read and manage email. Even web-based email clients that may not really need remote access often use IMAP4, whether to simplify implementation, to improve security through clear separation, or to enhance scalability. IMAP4 recently passed its 21st birthday and, while it is still the king of email access, there is a contender in the wings that appears to be preparing to make a takeover bid: JMAP, the JSON Mail Access Protocol.
JMAP is being developed and promoted by FastMail, a cloud email provider with a history of open-source support, at least of the "open core" variety: it has contributed over half the Git commits to the Cyrus IMAP server in the last five years and is a significant sponsor of the "Roundcube Next" project to build a new webmail client. JMAP, as Bron Gondwana explained in his lightning talk at linux.conf.au 2016, started life as an internal API used by FastMail between its JavaScript in-browser email client and its back-end servers. Positive experiences with open source convinced the company that an open specification would bring more value than keeping the protocol proprietary, so it created a more formal specification and started migrating the internal code base toward it.
Why replace IMAP4?
The first IETF RFC describing IMAP4 was published in December 1994, but it has not been static since then. A quick check of the RFC index shows over 60 RFCs that propose revisions, discuss extensions, or provide clarifications for the specification; the most recent being RFC7377, which was published in October 2014.
This ongoing development and revision could mean that IMAP4 has or will gain all the features that it needs to remain relevant and functional. Or it could mean that it has become a bloated monster with little internal coherence, and multiple implementations each supporting some, but not all, of the extensions. Gondwana was clearly inclined to the latter interpretation, but the specifics he gave were primarily about scope.
A modern email client needs more than just access to messages. It requires the ability to send email (using a protocol like SMTP or LMTP), look up a contacts database (possibly using LDAP or CardDAV), and often to query and update a calendar (perhaps with CalDAV). Managing all of these independently not only increases the configuration burden but is much more likely to run afoul of firewalls or related networking problems. Having a single protocol accessing all this data using a single port is likely to either work smoothly or fail completely, both outcomes seen as preferable to a mix of bits that work and bits that don't.
JMAP basics
JMAP addresses those concerns by defining a single protocol that can
access all of the mentioned services and data types, and possibly more in
the future,
in a fairly uniform way. Configuration is entirely automatic: a
DNS lookup for an "SRV" record for
_jmaps._tcp.example.com will
report a server and port to connect to for someone with an
@example.com email address. An https POST request to a well-known
URI (/.well-known/jmap) allows JavaScript Object Notation (JSON)
messages to be sent and replies received. These messages can create
and authenticate a session, which results in an authentication token.
This token can then authenticate future connections to any of a small
number of URIs that allow different actions to be performed and data
to be extracted.
There are upload and download URIs for sending and receiving attachments, raw RFC2822 mail files, and any binary large objects (BLOBs). There are also an event source URI for receiving asynchronous change notifications and an API URI for general request/response interaction. Each API message can contain multiple requests that are processed in sequence, each of which can generate one or more responses, all of which are combined into a single reply. Allowing multiple requests and replies in a single message reduces the number of round-trips to the server to gather complete information, which is important for getting good responsiveness on a high-latency network link.
The JMAP data model identifies a small number of object types, some
that exist in their own right and others that are synthesized. The
former group includes Mailboxes and Messages, Contact Groups and
Contacts, and Calendars and Calendar Events. Objects of these types are
each a collection of named attributes that can be requested
separately by the client. One particular attribute that is common to all
types is
an ID, which is a string value, assigned by the server, that is unique
within the type and stable
for the object's lifetime. Using this
ID, objects of a given type (e.g. Foo) can be retrieved
(GetFoos), modified if not immutable, and deleted
(SetFoos). Each
type has a "state" value that is effectively incremented
whenever any object
of that type is changed, and it is possible to ask for any updates
since a particular state (GetFooUpdates). This simplifies the task
of keeping a client-side cache up to date with any changes on the
server. JMAP also includes the option of a side-band protocol to
proactively notify the client of changes so that it doesn't need to
poll periodically.
This can use a platform-specific "push" mechanism, if available, or
the client can make a persistent connection to the event source URI.
For each of the base types (Messages, Contacts, Calendar Events)
there is a synthetic List type that is used to represent the results
of a search for that type within a particular account. These search results are
treated like a separate data type, as they are expected to be cached
by the server. The client can request a "window" into a list by
indicating a start position and a limit on the number of entries
to report. In the case of MessageLists, there is also an Updates
function, so getMessageListUpdates will report any changes to the
result of a previous search if those results are still cached by the
server.
JMAP extras
While this pattern of containers, lists, and objects covers much of the data model, it is not quite complete. There are three exceptions: Accounts, Threads, and SearchSnippets.
The "Accounts" abstraction allows a single user to have access to
multiple accounts; there will always be a primary account but there
may also be group accounts or accounts that the user been delegated
responsibility for. Accounts are a bit like the three container
types in that it is possible to get a list of all the accounts in a
single request, though no search filtering is possible. Unlike the
container, it is not possible to get a list of Updates, though a
client can discover if the list of accounts has changed at all.
"Threads" are lists of related messages. Each message is assigned a
threadId by a mechanism determined by the server rather than
by the protocol and a given thread is all of the messages
with that threadId. This could nearly be implemented with the
getMessageList function since it can filter based on threadId, but
threads are such a central concept in email handling that the protocol
can be more efficient if they are a first-class citizen. For example,
the getMessageList function can be asked to fetchMessages in which
case it returns not just the list of message IDs, but also (as a
separate response) the actual messages themselves (or more precisely:
a given set of attributes for each message). In a similar way, it can
be asked to fetchThreads and will report a list of all threads
found
by the search, each represented as a flat list of message IDs without any
parent/child relationships exposed. These
cannot just be a property of the messages since there will likely be a
different number of threads than messages. Having a separate object
type makes this easy.
"SearchSnippets" are a bit like Threads in that they exist to optimize
a particularly valuable part of the user experience. One of the
attributes that can be fetched for any message is a "preview", which is
a line or two of the most relevant text, possibly skipping quoted text or
salutations. For a specific search filter and a specific message, a
SearchSnippet contains an alternate preview text that includes sections
of the message that match search strings in the filter and has those
strings marked for easy highlighting. It also includes an alternate
version of the Subject with similar markup. A getMesssageList
request can include a fetchSnippets directive so that these
snippets are provided to the client.
Messages, mailboxes, and tags
Messages are undoubtedly the richest and most interesting objects
managed by JMAP. Messages are mostly immutable, with the only
permitted changes being to change the list of mailboxes the message is
a member of and to set or clear one of four flags: isUnread,
isFlagged, isAnswered, and isDraft. These roughly correspond to the
\Seen, \Flagged, \Answered, and \Draft flags supported by IMAP4.
The other two IMAP4 flags: \Deleted and \Recent are not supported
by JMAP, presumably because they aren't particularly
useful. \Deleted is only needed for a two-stage delete, which is
adequately handled by moving a message to a "Trash" folder. \Recent
has semantics that are not particularly user-friendly: it is quite
possible for the server to be required to clear \Recent before the
user has had a chance to see any hint of the message at all.
While IMAP4 allows arbitrary user-defined flags to be assigned to a message, JMAP does not. Instead, it allows a message to be attached to an arbitrary number of user-defined mailboxes. This allows the email client to treat mailboxes like folders, tags, or even both. This last suggests a slight weakness in the protocol. It is quite possible that a user would want some mailboxes to behave like tags and others to behave like folders: when dragged to a folder, a message would be removed from the current folder, while dragging to a tag would just add that tag. A client cannot support this distinction without imposing its own interpretation of mailbox names, such as assuming that any child of the "tags" mailbox should be treated as a tag.
A JMAP Mailbox can have a mustBeOnlyMailbox flag set, in which case
messages can only be in that mailbox if they aren't in any other.
This can force a mailbox to behave like a folder, but then none of the
messages in it can belong to any tag-like mailboxes, so it seems rather
restrictive.
Mailbox roles and sending email
Only five years ago, IMAP gained the concept of special-use mailboxes
thanks to RFC6154. This
defined mailbox attributes like \Drafts, \Sent,
\Junk, and \Trash so that an IMAP client could use those flags instead
of depending on hard-coded (English language) names. JMAP has a
matching concept, referred to as mailbox "roles", with some minor variations
such as \Junk being replaced by spam. One role that deserves
special attention is the outbox role.
When a message is placed in an outbox mailbox, it must have the
isDraft flag set; the implication is that it has been queued for
delivery. At some future time, which may be immediately or may be
when the time in the "Date:" header is reached, the JMAP server
should attempt to deliver the message. Once this attempt completes,
the message is deleted from the outbox and re-created in the folder
with sent role. The specification is not explicit on what happens
if the delivery attempt fails. The likely implication is that a
delivery status email would appear in the inbox. Given how easy it can
be to miss or be confused by such messages, protocol support that would
allow the client to highlight and help
resolve failures would be quite valuable.
Spam filtering is important in any modern email application and JMAP
provides some help in this direction. Aside from the spam role
already mentioned, which makes it easy to find spam, there is a dedicated
"reportMessages" request that can be used to report that a message
is, or is not, spam in the eyes of the user. This is typically used
to train the per-user spam filter. Unfortunately, the specification does
not spend
time justifying various decisions, so we do not know why this request
is needed rather then allowing spam status to be set by simply moving
messages into or out of the spam mailbox. All we know is that the
specification is quite explicit that moving messages
like this would be a separate operation from reporting their spam
status.
Managing MIME
JMAP is able to handle messages as uninterpreted blobs of RFC2822
text or as completely parsed messages with various headers and
details of attachments available as attributes in the JSON encoding.
When collecting a message from the server, a client can ask for
specific fields and will get the relevant data in pure UTF-8 with no
escaping or transport encoding. When uploading a message to the
server, it can list the content of various fields and describe
attachments that have previously been uploaded as ordinary files, and
the server will combine all the parts together using the appropriate
encoding. There is even a provision to extract a textBody from a
message that only contains an HTML version, or to get an HTML version
of plain text.
While this MIME (Multipurpose Internet Mail Extensions) transcoding is
useful, it appears to be focused on matching common
usage rather than precisely mirroring the MIME specification. MIME
supports generic recursive multipart structuring of messages, where
any MIME part can itself be multipart, typically multipart/mixed
for attachments, multipart/alternative for different renderings of
the same message (text or HTML), or multipart/related for HTML
combined with some image files. While the standard allows multiple
levels of multiparts and a variety of subtypes of multipart, JMAP
knows little of this. There is only one pair of alternatives, text or
HTML. Together with this, there are attachments, which might be tagged
inline in the multipart/related case. One specific type of
attachment — an email message — can result in a recursive structure
and JMAP handles this correctly, but no other recursion is apparent.
This contrasts with IMAP4, which does support reporting of nested
structure in the BODYSTRUCTURE response.
While this is theoretically limiting, it does cover the vast majority of email. Since a client can deal directly with the undecoded RFC2822 message if it chooses, it may be an acceptable tradeoff. Unfortunately, there is no easy way for such a client to determine if there might be anything extra to decode, so it cannot know if it needs to fetch the raw message.
One place where this might be a real problem is with signed or encrypted email. The JMAP specification makes no mention of S/MIME, the Secure MIME standard. While there may be difficult issues around deciding whether a server should be trusted to sign or decrypt messages, it would be nice to have the option, and it would be nice if the server told the client when the message might need decrypting or if a signature needs checking.
What else is missing from JMAP?
Email involves such a rich and varied experience that it is probably impossible for any protocol to really cover everybody's favorite feature. There are, however, two missing features worthy of note that both relate to the processing of mail as it arrives — an area currently unaddressed by JMAP. One is the ability to register a "vacation" message to be automatically sent in reply to incoming messages. This was raised on the discussion list and a possible protocol enhancement to cover it was discussed. This suggests that there is an openness to enhance the protocol to meet requirements.
The other feature, one close to my heart, is saved searches.
Whether you use procmail, Sieve-based filtering, or a similar mechanism
to direct new
messages to different mailboxes based on content, or use Notmuch or
similar to perform searches at the moment you open a virtual mailbox,
there is a clear need to be able to save rules or searches somewhere;
having this exposed in JMAP would be quite helpful. This is even
mentioned in the JMAP specification, though clearly as an aside. In
the section on negotiating extension support, there are some
hypothetical example extensions including
"com.fastmail.savedSearch:4". Clearly this need has been thought
about, but no clear resolution has been found. It may well be a
challenge to provide support in the protocol that is useful to
clients without being an undue burden on servers. But it would be a
poor email access protocol in this day and age that didn't
provide for saved searches in some form.
Status
JMAP is clearly something that FastMail wants people to play with. It has released a proxy server, written in Perl, that provides a gateway between JMAP on the client side, and IMAP and SMTP on the server side. The various parts can be downloaded from GitHub and built without too much difficulty, or a hosted version can be used — though, obviously, don't give that the password of an important email account. The downloads include a simple JavaScript email client that talks JMAP and can be used to complete the experience.
While JMAP is certainly interesting, and ticks a lot of the right boxes, but it is far from certain that it can gather sufficient momentum that anyone other than FastMail and a few niche players will invest in it. To achieve that, it would need to be a lot better than "good enough"; it would need to be "compelling", and probably also "exciting". The proxy server should help to reduce the typical chicken-and-egg problem that new protocols face by making it possible to use a JMAP client even if your service provider doesn't support JMAP natively. Whether this will be enough to encourage people to join the adventure is hard to know. We'll just have to wait and see.
| Index entries for this article | |
|---|---|
| GuestArticles | Brown, Neil |
Posted Mar 17, 2016 9:12 UTC (Thu)
by pebolle (guest, #35204)
[Link] (2 responses)
1) This puzzled me:
Is that really correct? If so, JMAP seems to use a definition of draft that is at odds with the common usage of that term.
Posted Mar 17, 2016 9:22 UTC (Thu)
by neilbrown (subscriber, #359)
[Link] (1 responses)
Posted Mar 20, 2016 21:58 UTC (Sun)
by brong (guest, #87268)
[Link]
You can see a pretty horrible implementation of an instantaneous outbox here:
https://github.com/jmapio/jmap-perl/blob/master/JMAP/Imap...
This is the JMAP proxy, which talks IMAP, CalDAV and CardDAV at one side, and JMAP at the other. We're using it for testing client development.
It's pretty much feature complete apart from the authentication flow, and of course the changes to calendaring which will be made alongside the TC-API group at CalConnect, so we can have a shared JSON representation for calendar events. We're hoping to nail down that specification in Hong Kong next month:
https://www.calconnect.org/events/calconnect-xxxvi-april-...
...
The plan for 'outbox' at FastMail once it's finished is that it doesn't actually send immediately. Messages will sit there until they reach their "internaldate" timestamp, and then be sent. A regular append and move without setting internaldate will send within 1 second, but clients will easily be able to set a send time of 30 seconds in the future, and then they can delete the message or move it back to Drafts within that time to abort send.
I blogged about some of the underlying architecture design we'll need for that here:
https://blog.fastmail.com/2015/12/25/a-ghost-of-fastmail-...
Bron
Posted Mar 17, 2016 13:26 UTC (Thu)
by hkario (subscriber, #94864)
[Link] (2 responses)
Posted Mar 17, 2016 18:52 UTC (Thu)
by bronson (subscriber, #4806)
[Link]
Posted Mar 20, 2016 22:07 UTC (Sun)
by brong (guest, #87268)
[Link]
Indeed, the current FastMail API (a more rough-around-the-edges precursor to JMAP) was designed by Neil and I sitting opposite each other at our desks in Oslo with very few other distractions. He was designing the data model in Javascript for Overture:
https://github.com/fastmail/overture
And I had just finished a year and a bit of replacing all Cyrus IMAP's core datastructures with safe locking to guarantee CONDSTORE and QRESYNC accuracy as well as enabling efficient cross-datacentre replication, so I knew those standards inside out.
We've build on CONDSTORE in particular, but also on the concepts underlying GMail's X-GM-THRID and X-GM-MSGID fields. The JMAP proxy, when talking to GMail, will use those values (and the "\Allmail" folder with X-GM-LABELS) to get more efficient entire-account syncing without having to re-calculate threads locally.
FastMail exposes a very similar data model via the totally undocumented "CID" field (equivalent to X-GM-THRID - there's a patch to call it THRID if that ever gets standardised) as well as a bunch of commands XCONVFETCH and XCONVMULTISORT which allow cross folder sort, and thread statistic fetches. Our API is written on top of those.
(we don't yet have labels - there's some more Cyrus internal changes requried, but we have a plan that will give complete standards compliant IMAP _and_ labels, and both of them fast, on the same datastore. I should be writing that rather than talking on the internet - but y'know, it's Monday morning and I haven't had coffee yet)
Bron.
Posted Mar 18, 2016 3:34 UTC (Fri)
by smurf (subscriber, #17840)
[Link]
That's too narrowly geared to a "dumb webmailer" frontend. A server which allows for a JSONish approach to accessing MIME emails, with the "dumb" access layered on top of that instead of "instead of that", would be great.
Hopefully, somebody will take the code in that direction.
Posted Mar 18, 2016 10:08 UTC (Fri)
by miquels (guest, #59247)
[Link] (2 responses)
Mike.
Posted Apr 13, 2016 10:07 UTC (Wed)
by inputmice (guest, #108219)
[Link] (1 responses)
Posted Mar 18, 2016 22:11 UTC (Fri)
by SLi (subscriber, #53131)
[Link]
Posted Mar 26, 2016 20:01 UTC (Sat)
by jengelh (guest, #33263)
[Link]
Why does that sound so much like MAPI/ActiveSync, and xkcd.com/927 .
Posted Mar 26, 2016 21:44 UTC (Sat)
by eduard.munteanu (guest, #66641)
[Link] (24 responses)
Posted Mar 26, 2016 22:28 UTC (Sat)
by neilbrown (subscriber, #359)
[Link] (23 responses)
Posted Mar 27, 2016 0:28 UTC (Sun)
by magila (guest, #49627)
[Link] (22 responses)
This is highly debatable, especially when we have tools like wireshark dissectors. Frankly, if you can't handle debugging binary protocols you have no business implementing network protocols for widespread use. If anything, the history of test-based protocols has been defined by interop clusterfucks like HTTP, SMTP, IRC, etc.
> No need to write new custom, error-prone parsers.
The difficulty of implementing binary protocols has been wildly overstated by text-based advocates. Even aside from that, this argument only applies to protocols with a custom bitstream format. If they're too squeamish for some simple pointer arithmetic then they can use an interchange format like Cap'n Proto or Protocol Buffers and get the same experience as JSON but with a richer set of data types, less complexity, and much better efficiency.
> The difference in speed is likely to be minimal, especially if you GZIP the exchange
It's not really about speed, it's about having a simple and unambiguous protocol. The complexity and pervasive ambiguity of text-based protocols is what leads to the aforementioned interop problems. Also, beware information leakage when sending compressed data over an encrypted channel.
Posted Mar 27, 2016 3:06 UTC (Sun)
by Cyberax (✭ supporter ✭, #52523)
[Link] (21 responses)
Binary encodings are the wrong solution pretty much always. Their only advantage is usually a slightly more efficient encoding, but that also is usually questionable.
First, simple tagged key-value binary encodings (like BSON) bring absolutely nothing, they just help the parser to be a bit more efficient. And if we look at SMTP or HTTP then _parsing_ has never really been a problem, it's interpretation of the parsed data that is tricky.
Second, if we go down the rabbit hole of data schemas and tight specifications then we get ASN.1 as a result or something closely approximating it. With all the problems of versioning, global type identification, type mapping and general awfulness.
Third, anybody who thinks that "dissectors in Wireshark" are enough for debugging is a freaking idiot and should be forced to debug interoperability problems in proprietary SCTP implementations by using partially dumped packets (up to the first \0 symbol) in log files. Until they see the light or kill themselves, whichever comes first.
I'm _really_ glad that the industry settled on mostly-text specifications like JSON (even though I'd like something just a _little_ bit more carefully specified). Unfortunately, HTTP/2 happened but it can be easily ignored for now.
Posted Mar 27, 2016 9:41 UTC (Sun)
by kentonv (subscriber, #92073)
[Link] (9 responses)
I guess you haven't worked with Protobuf or Cap'n Proto, which are binary formats that handle versioning arguably more cleanly than JSON, yet have type-safe schemas unlike BSON?
Posted Mar 27, 2016 20:09 UTC (Sun)
by Cyberax (✭ supporter ✭, #52523)
[Link] (8 responses)
Protobuf doesn't really have a schema. It's more like JSON actually, the structures simply use tag numbers instead of field names and there are slightly more data types available. Versioning support is mostly non-existant - unknown fields are just ignored and conflicting definitions cause havoc.
And there's now a canonical JSON mapping for protobuf as well.
I've read several years ago a post about protobuf from a guy in Google - he wrote that protobuf actually slightly predates JSON and these days they would have just used pure JSON instead.
Posted Mar 28, 2016 17:34 UTC (Mon)
by kentonv (subscriber, #92073)
[Link] (7 responses)
> Protobuf doesn't really have a schema.
That's not true at all. https://developers.google.com/protocol-buffers/docs/proto
The schema is not transmitted on the wire. The schema is used to encode/decode on each end.
The raw wire format is numeric tags and values, but no one actually uses protobuf without schemas.
"Reverse engineering" protobufs usually means:
Or, better yet:
I don't see any .proto files in your repo nor any reference to libprotobuf. Did you reverse engineer from the byte level and write your own decoder?
> Versioning support is mostly non-existant - unknown fields are just ignored
Yes, that's the best way to do versioning, and is exactly the same strategy people use with JSON. Using actual version numbers is a pain as you end up with lots of branchy code to handle every version. That's what Google had before protobuf, and protobuf was explicitly designed to fix it.
Protobuf is better than JSON, though, because with Protobuf you have a schema where you can declare default values to replace things missing on the wire, whereas with JSON you have to check for the existence and type of every single field before accessing it or otherwise risk unexpected exceptions and security bugs.
> and conflicting definitions cause havoc.
In my experience this is not a problem people run into often. You have one owner of the protocol who decides which changes become official. If you want a protocol to be extensible, you use "extensions" which allow third parties to extend the protocol without conflicting (or in proto3 you use the "Any" type).
(Moreover, conflicting definitions would cause an equal amount of havoc for JSON.)
> he wrote that protobuf actually slightly predates JSON and these days they would have just used pure JSON instead.
Sorry, that post was wrong. That's definitely not the opinion held by senior engineers at Google.
1. The importance of type-safety and the implicit documentation provided by schemas is widely recognized inside Google.
2. The vast majority of code at Google is written in statically-typed languages (C++ and Java) where JSON is highly annoying to use due to being dynamic. Protobuf uses schemas to generate classes with pleasant interfaces.
3. Protobuf parsing as-is is responsible for several percent of CPU cycles fleet-wide -- some servers report 30% or more. That equates to many millions of dollars annually. JSON would be an order of magnitude slower, which would cost a massive amount of money.
4. Google stores petabytes of data in Protobuf format. This data would be much larger as JSON. No, compression doesn't magically fix it (compressed protobufs are still much smaller than compressed JSON, particularly for highly-structured (i.e. not textual) data).
5. Similarly, the network bandwidth overhead would be unacceptable.
6. The latency cost of encoding and compressing JSON would be unacceptable for many systems in Google even if there were CPU cycles to spare.
Posted Mar 28, 2016 19:26 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (6 responses)
> Yank the protobuf descriptors out of the app binary, which provide the complete schema.
> I don't see any .proto files in your repo nor any reference to libprotobuf. Did you reverse engineer from the byte level and write your own decoder?
> Yes, that's the best way to do versioning, and is exactly the same strategy people use with JSON.
All the protobuf "smarts" are in the mapping layer which enforces the schema, provides default values and so on. This mapping layer can be built atop JSON just as easily (as many people have done, many times in many companies) to provide nice statically-typed interfaces.
You can _almost_ treat protobufs as an encoding for JSON. Contrast it with ASN.1 PER where you actually have to use the schema to decode raw messages as field types and offsets are not transmitted on the wire.
Posted Mar 28, 2016 20:04 UTC (Mon)
by kentonv (subscriber, #92073)
[Link] (5 responses)
But once you have those generated classes, then it makes no difference to the developer whether the bytes were JSON or Protobuf. Protobuf can produce a textual representation for debugging as needed. Why waste cycles and bytes encoding human-readable text all the time?
It sounds like you're the kind of person who reads raw network dumps a lot, but that you're also the kind of person who doesn't like to use the tools provided to you for this purpose, so I guess that would explain a preference for human-readable messages on the wire. But, I think between using the tools and spending millions of dollars on additional computer hardware, using the tools seems more reasonable.
> And adding .proto files into the mix was just not worth it.
Well, using libprotobuf and a .proto file would have saved you from writing quite a bit of code.
Posted Mar 28, 2016 21:19 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (4 responses)
> Why waste cycles and bytes encoding human-readable text all the time?
> It sounds like you're the kind of person who reads raw network dumps a lot
> but that you're also the kind of person who doesn't like to use the tools provided to you for this purpose, so I guess that would explain a preference for human-readable messages on the wire.
JSON helps in this regard by making the whole stack less opaque.
> Well, using libprotobuf and a .proto file would have saved you from writing quite a bit of code.
Posted Mar 28, 2016 21:47 UTC (Mon)
by kentonv (subscriber, #92073)
[Link] (3 responses)
Uh, what? The absolute fastest JSON parsers top out around a gigabit per second, consuming 100% of CPU time on parsing, in idealized benchmark scenarios. You're asserting that you can do 10gbps and the CPU usage isn't even noticeable?
Posted Mar 28, 2016 22:05 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
Posted Mar 28, 2016 22:32 UTC (Mon)
by kentonv (subscriber, #92073)
[Link] (1 responses)
Generally the fastest JSON parser is RapidJSON. Protobuf beats it handily (3x or more) in most benchmarks, e.g.
https://github.com/erickt/rust-serialization-benchmarks
(Cap'n Proto in turn handily beats Protobuf and can in fact saturate a 10gbps link.)
Posted Mar 29, 2016 0:18 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link]
> Generally the fastest JSON parser is RapidJSON. Protobuf beats it handily (3x or more) in most benchmarks, e.g.
And yeah, even 3x performance difference in _parsing_ is pretty much negligible these days.
Posted Mar 27, 2016 12:43 UTC (Sun)
by smurf (subscriber, #17840)
[Link] (8 responses)
I really doubt that JSON is in any way more debuggable than a binary protocol. If you ever saw a nested multi-page JSON dump without any formatting whatsoever, and needed to start count braces to make heads or tails of it, you'll know what I mean.
Posted Mar 27, 2016 19:42 UTC (Sun)
by bronson (subscriber, #4806)
[Link] (6 responses)
Why wouldn't you install JSON Formatter and then click on arrows to navigate?
Posted Mar 27, 2016 20:05 UTC (Sun)
by zlynx (guest, #2285)
[Link] (1 responses)
Posted Mar 28, 2016 4:26 UTC (Mon)
by bronson (subscriber, #4806)
[Link]
Posted Mar 28, 2016 7:19 UTC (Mon)
by smurf (subscriber, #17840)
[Link] (3 responses)
IMHO, encoding some data into a protocol should be a function in the mathematical sense, i.e. any possible payload shall encode to exactly one datagram. JSON lets you play way too many games with whitespace, escaping and whatnot for that to be realistic.
And yes, computers are fast enough to eat JSON at mind-boggling speed these days, but if messagepack or smile or … can do the exact same thing with even less work (and no ambiguity), why not use that?
Posted Mar 28, 2016 7:32 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
> And yes, computers are fast enough to eat JSON at mind-boggling speed these days, but if messagepack or smile or … can do the exact same thing with even less work (and no ambiguity), why not use that?
Everything, and I mean it, EVERYTHING, in software should be designed with the assumption: "What is going to happen _when_ it breaks?"
Posted Mar 28, 2016 7:59 UTC (Mon)
by neilbrown (subscriber, #359)
[Link] (1 responses)
I can't see how this is at all relevant for JMAP. JMAP is currently only defined over HTTPS, and until wireshark can see inside TLS, it doesn't really matter if it can decode the JSON or not.
Posted Mar 28, 2016 8:01 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Posted Mar 27, 2016 19:51 UTC (Sun)
by Cyberax (✭ supporter ✭, #52523)
[Link]
> hus JSON wrapped in HTTP, instead of _anything_ more sensible. Thus no explicit support for arbitrary MIME structures, or multipart/anything-except-alternate messages
> let alone signing and encryption.
> I really doubt that JSON is in any way more debuggable than a binary protocol. If you ever saw a nested multi-page JSON dump without any formatting whatsoever, and needed to start count braces to make heads or tails of it, you'll know what I mean.
Besides, pretty-printing JSON or extracting well-formed fragments from it is trivial these days.
Posted Apr 6, 2016 15:24 UTC (Wed)
by eduard.munteanu (guest, #66641)
[Link] (1 responses)
Stuff like whitespace trimming, folding multiple lines and so on really have no place in a protocol that's intended to be parsed by machines. Or do you intend to run human servers?
I also doubt text-based protocols are easier to debug. Perhaps that's easier in a "two sticks light a fire easier than a broken lighter" sense. You should be using a proper parser to debug binary formats and I think that's a point of contention. Seeing the many kludgy scripts out there using regexps to parse stuff, it's no wonder people have difficulties and dismiss binary formats as difficult.
Posted Apr 11, 2016 14:25 UTC (Mon)
by jchaxby (subscriber, #63942)
[Link]
Parsing text is not really any different to parsing text. In the example above you've got length counted strings and 0x04 in the middle or you've got strings separated by "=".
Arguing for binary-based mail related protocols is a bit fruitless. At some stage you're going to be parsing RFC5321 and RFC5322. If you can't handle text parsing then you're already in serious trouble :) You'd better be able to parse LDIF and LDAP filter strings as well.
My reaction to JMAP is basically to run around cheering. I wrote an IMAP server and the problems with the protocol are legion. Parsing is the easy bit, getting the protocol to actually work sensibly is not at all easy. I also wrote an LDAP server. Oddly enough, parsing the protocol is the easy bit. Logging was a bit harder: logging binary is not exactly friendly (and yes, you do want to log what you got and what you thought it was). The actual protocol was rather easier to deal with, apart from access control anyway.
Parsing is well understood, it's been well-understood for decades. You can optimize a protocol for speed, but that doesn't mean that binary protocols are faster or more desirable, it means that for particular problem domains an optimized protocol is a good thing. The bottle neck with my mail server(s) and LDAP server(s) was never the protocol.
JMAP — reinventing IMAP
> When a message is placed in an outbox mailbox, it must have the isDraft flag set;
> the implication is that it has been queued for delivery.
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
And the sterling success of ASN.1-based protocols, like H.323 and the OSI stack. Sure.
JMAP — reinventing IMAP
JMAP — reinventing IMAP
I worked with protobuf quite a bit (haven't had a chance to play with Cap'n Proto yet). For example, I reverse engineered the Android Auto protocol recently ( https://github.com/Cyberax/aauto ) which is totally protobuf-based.
JMAP — reinventing IMAP
1. Feed a few messages to protoc --decode_raw.
2. Guess the meaning of each numeric tag.
3. Write a .proto file assigning names and types to the tags.
1. Yank the protobuf descriptors out of the app binary, which provide the complete schema.
JMAP — reinventing IMAP
> The raw wire format is numeric tags and values, but no one actually uses protobuf without schemas.
Well, I do.
That might not be legal, though.
Yep. Protobuf is self-describing so it wasn't complicated. And adding .proto files into the mix was just not worth it.
But that's exactly my point. JSON is semantically very similar to protobuf wire format - the differences are really minor (tags instead of names and more integer types in protobuf).
JMAP — reinventing IMAP
JMAP — reinventing IMAP
But it doesn't do it normally.
The amount of wasted cycles isn't noticeable even at 10G wire speeds.
I don't read network dumps normally, but I do have to do it now and when. Usually during a high-stress breakage situation.
I actually like tools that map a domain object model into messages/database/whatever. But they do have their price in being opaque when you have to diagnose a problem.
Probably not in this case.
JMAP — reinventing IMAP
JMAP — reinventing IMAP
Yes, when compared to parsing protobufs. The fastest possible JSON parser is based on bitslicing and SSE: http://parabix.costar.sfu.ca/ and with it you can get 10G performance.
JMAP — reinventing IMAP
JMAP — reinventing IMAP
They actually have a full XML parser with same performance characteristics. There's a JSON parser there as an example.
I have my own Parabix-based JSON parser that is used in production to do switching for a JSON-based UDP protocol. It saturates multiple 10G links (though it's also multithreaded).
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
Uhm. Wireshark supports JSON out of box. Try it.
Because once it breaks (and it WILL break, no doubt about it) debugging the breakage will be a nightmare.
JMAP — reinventing IMAP
JMAP — reinventing IMAP
JMAP — reinventing IMAP
That hasn't been true for quite a long time. Initially JSON was indeed something that could just be eval()'ed by JS, but nobody does this anymore even in JS these days.
Why should there _be_ such support? JSON can be arbitrarily nested so you don't _need_ anything special to support complex MIME structures and multiparts.
JSON signing and encryption is easy: https://tools.ietf.org/html/rfc7515 Now try that with plain email...
And now imagine that you don't even _have_ braces, instead you have binary offsets. Or worse, no offsets at all and structure information kept separately.
JMAP — reinventing IMAP
JMAP — reinventing IMAP