Matrix: a new specification for federated realtime chat [LWN.net]

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 1:16 UTC (Thu) by wahern (subscriber, #37304)
Parent article: Matrix: a new specification for federated realtime chat

But Matrix may have a key factor working in its favor: no reliance on a centralized server to coordinate conversations. In some circles, certainly, that is a long-requested feature.

But XMPP has that going for it as well. XMPP is even better: Matrix relies on a centralized authority to tie a 3PID (third-party identifier, such as your e-mail address) to your Matrix ID. By contrast, your XMPP handle is almost always your e-mail address, and the only centralized authority involved is DNS.

Otherwise, Matrix works much like XMPP; they're even using DNS SRV records like XMPP. Almost everything works just like XMPP, except with JSON instead of XML and HTTP instead of a regular TCP stream. And MUC is built-in rather than bolted on.

The only novel thing seems to be the idea of a tree of events. I can imagine all kinds of cool things to do with that, including building services like Slack.com, with rich white-boarding features and archiving.

Other than XMPP, Matrix reminds me a little of Cobalt (previously Croquet). Now that is a cool project with very interesting underlying algorithms. Particularly TeaTime, a distributed, transactional, P2P messaging protocol.

to post comments

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 3:53 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (25 responses)

Matrix is nothing like XMPP.

XMPP is a braindead protocol that is now transitioning into a simply 'dead' state. It's made around a concept of permanent "associations" between the endpoints (so a server can store messages while endpoints are offline) and while it has some support for federation, it really works only for simple 1-to-1 messages.

Group chats were a feature bolted-on later, and was supported exceedingly poorly (some clients never really got it).

But the worst shortcoming was the message persistence (or a lack thereof). XMPP has no notion of archiving and message history synchronization. As of 2012 no major XMPP clients implemented it properly, some _servers_ (like Spark) had special server-side support for message storage accessible through their custom web-interface.

Well, also there's the fact that XMPP is built on a totally awesome idea of protocol built on top of an endless stream of XML stanzas. HTTP tunneling was added only very late in the game (and not universally supported either).

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 4:46 UTC (Thu) by pizza (subscriber, #46) [Link] (19 responses)

> XMPP is a braindead protocol that is now transitioning into a simply 'dead' state.

Yet it remains the *only* option there is if you want an IM solution that doesn't require you to sign your soul (and your messages) over to $BigCorp.

> It's made around a concept of permanent "associations" between the endpoints (so a server can store messages while endpoints are offline)

Those associations also act as a whitelist so you aren't massively spammed all the time.

> and while it has some support for federation, it really works only for simple 1-to-1 messages.

Federation was one of the core design principles of XMPP, so I'm not sure where you get this "some support" thing. Or what it has to do (or not) with "1-1 messages".

> Group chats were a feature bolted-on later, and was supported exceedingly poorly (some clients never really got it).

Yes, group chats were added later. And I suggest you blame said clients for their poor support; many others have no problems whatsoever.

> But the worst shortcoming was the message persistence (or a lack thereof). XMPP has no notion of archiving and message history synchronization.

Server-side archiving (and history synchronization) was standardized many years ago. There are multiple implementations, both on client and servers.

> Well, also there's the fact that XMPP is built on a totally awesome idea of protocol built on top of an endless stream of XML stanza

That's the first defensible criticism of XMPP you've made; the rest should be laid at the feet of the implementors (keeping in mind that there's no "XMPP Czar" enforcing feature compliance) In particular, I'd like to single out Google for effectively pulling an IE6 on the XMPP ecosystem. They didn't even properly implement the stuff they themselves specified (Hello, Jingle..) which made it nearly impossible for folks to properly interoperate properly.

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 5:29 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (15 responses)

> Yet it remains the *only* option there is if you want an IM solution that doesn't require you to sign your soul (and your messages) over to $BigCorp.
"It's better than nothing" is a very weak argument, especially since it's now competing with Slack and other similar implementations.

> Those associations also act as a whitelist so you aren't massively spammed all the time.
Sure. But it's not relevant for lots of users, and in any case it should be an optional feature.

>> and while it has some support for federation, it really works only for simple 1-to-1 messages.
> Federation was one of the core design principles of XMPP, so I'm not sure where you get this "some support" thing. Or what it has to do (or not) with "1-1 messages".
Try this:
1) Create a group chat with users from several federated servers.
2) Disconnect the servers.
3) Change each group chat replica (add new users, remove users, stuff like that)
4) Reconnect the servers.
5) Watch the hilarity that will shortly ensue.

> Server-side archiving (and history synchronization) was standardized many years ago. There are multiple implementations, both on client and servers.
Nope. The original XEP-0136 has lots of inadequacies and is difficult to implement. Most of clients and servers simply don't support it: http://en.wikipedia.org/wiki/Comparison_of_XMPP_server_so... .

There's a newer http://xmpp.org/extensions/xep-0313.html which looks a little bit better but it's still a draft.

>> Well, also there's the fact that XMPP is built on a totally awesome idea of protocol built on top of an endless stream of XML stanza
> That's the first defensible criticism of XMPP you've made;
That's actually my least concern. The network layer in XMPP is crazy, but writing a parser for it doesn't take that much time.

> the rest should be laid at the feet of the implementors (keeping in mind that there's no "XMPP Czar" enforcing feature compliance) In particular, I'd like to single out Google for effectively pulling an IE6 on the XMPP ecosystem.
Yep, so the ecosystem rapidly fragmented to the point that you can't depend on ANY feature being present. It's beyond ridiculous. And I can't really blame Google - they tried hard enough to make it work.

That's why a redesign of the protocol which is built on an idea of real distributed workflow is in order.

PS: yes, my company tried to use XMPP for several mission-critical applications and we got horribly scarred as a result.

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 13:40 UTC (Thu) by pizza (subscriber, #46) [Link] (14 responses)

> Yep, so the ecosystem rapidly fragmented to the point that you can't depend on ANY feature being present. It's beyond ridiculous. And I can't really blame Google - they tried hard enough to mke it work.

No.. Google didn't try all that hard in the end, and they were one of the main reasons why it didn't work. They implemented what they cared about, didn't care when others complained about Google's not-quite-spec-compliance. This was aptly demonstrated with the Jingle fiasco; Google released a reference implementation (libjingle) that differed behaviorally from what their own clients used, and never actually fixed their clients (or libjingle) to line up. They they
basically sat on that feature set (ala IE6) for many years until they basically threw Gtalk/XMPP under the bus with their "Everything is now Google+, oh, and f*ck interoperability and federation" announcement in May 2013. Even before that point it was pretty hard meaningfully innovate or improve the XMPP ecosystem as a whole when everyone else had to be bug-for-bug compatible with Google/Gtalk.

The other reason XMPP failed, and any alternatives are also doom to fail, is the old spectre of network effects and the fact that it's mostly impossible to monetize via advertising (not unlike RSS) so the big boys have no incentive to support it, much less actually help improve the standards.

So, as a result, federated XMPP's userbase is a rounding error compared to the likes of Hangouts, AIM, Whatsapp, FB Messenger, Skype, and so forth. It's only advantage at this point is its network effect, and new entrants won't even have that. I just don't see any new F/OSS entrants having the resources to buy enough mindshare/marketshare to make a greenfield design worth the effort.

Google Reader _was_ monetized

Posted Feb 12, 2015 14:50 UTC (Thu) by tialaramex (subscriber, #21167) [Link] (2 responses)

"it's mostly impossible to monetize via advertising (not unlike RSS)"

Google Reader /was/ monetized. A key step to getting rid of Reader was to put engineering time into removing the advertising from Reader, so that its contribution to Google's bottom line would vanish and its termination could be justified as a necessary belt-tightening exercise.

We often read nonsense on LWN about how public corporations are "obliged" to do whatever makes the most profit. Google is an unusual case, but they make it clear how untrue that is. The way to make money was to kill off the me-too social network and keep Reader. But that did not suit the people who actually make the decisions, so that's not what happened. Shareholders can yell and scream and stamp their feet, but they never use their power to actually do anything, so they're irrelevant.

The biggest system-wide scandal is the paying of high salaries or even bonuses to directors who are ineffective. Shareholders are very angry about that. How angry? Well, not angry enough to actually vote those directors out and replace them. So everything continues as before.

Google Reader _was_ monetized

Posted Feb 12, 2015 18:35 UTC (Thu) by pizza (subscriber, #46) [Link] (1 responses)

> Google Reader /was/ monetized. A key step to getting rid of Reader was to put engineering time into removing the advertising from Reader, so that its contribution to Google's bottom line would vanish and its termination could be justified as a necessary belt-tightening exercise.

I wasn't referring to Google Reader; I was referring to those entities that provide, or used to provide, RSS feeds. Monetizing the feed reader is a lot easier than monetizing the feeds themselves.

Google Reader _was_ monetized

Posted Feb 24, 2015 12:36 UTC (Tue) by nye (guest, #51576) [Link]

>I wasn't referring to Google Reader; I was referring to those entities that provide, or used to provide, RSS feeds. Monetizing the feed reader is a lot easier than monetizing the feeds themselves.

Why would it be harder to monetise RSS than a web page? Many of the RSS feeds I read have ads at the top just as they do on their website. What's the difference?

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 15:01 UTC (Thu) by Arathorn (guest, #101018) [Link]

We're not really not targeting the current encumbents (WhatsApp, FB, Skype etc) with Matrix (although if they want to federate up, hooray!) - but instead trying to provide a signalling layer for the next wave of chat apps, and indeed a federated layer between all the new WebRTC services which are popping up, which currently can't interoperate at all. In terms of how to monetise it: well, you can certainly sell value-added services on top of Matrix (e.g. PSTN connectivity; high-SLA hosting; translation services; voice->text transcription; etc) if you so desired. It's not all about advertising.

So hopefully there's still room for another open standard in this space :)

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 18:58 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (9 responses)

It was a little bit different. Libjingle actually worked fine with Google Talk's XMPP implementation, we used it in-house and actually ported its signaling layer to support SIP.

However, none of the major OpenSource projects were interested in adding Jingle - it simply sat there gathering dust for _years_ after Google spent literally millions to make it work. Do you remember the 3-line-text-field Pidgin fiasco? I've heard rumors that this was the trigger for Google to abandon XMPP internally.

And I can't really blame Google here.

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 23:21 UTC (Thu) by jlargentaye (subscriber, #75206) [Link] (7 responses)

> Do you remember the 3-line-text-field Pidgin fiasco?

No? I'd love to have a link. Is it related to this bug: https://developer.pidgin.im/ticket/4986 ?

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 23:33 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (6 responses)

Yep, exactly this one. Its Debian couterpart: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=469283

In short, Pidgin developers limited the text input field to 4 lines (sorry, not 3). This change annoyed lots and lots of people, but developers refused to even make this feature optional.

There were a lot of discussions on various forums about it (Slashdot: http://tech.slashdot.org/story/08/04/30/1822237/pidgin-co... ).

I've heard from a source at Google that this very incident caused a reaction something like: "WTF are we spending millions to support XMPP if the best competing client is a toy managed by idiots?" Remember, this was 3 years after libjingle had been released.

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 23:44 UTC (Thu) by jlargentaye (subscriber, #75206) [Link] (3 responses)

Huh, my initial thought following your previous comment about this being the straw to Google's XMPP efforts was that it was a silly anecdote to make a decision on... but it actually was a good illustration of the state of the XMPP client landscape.

On the other hand, Pidgin was never a good XMPP client, IMHO Psi was always much ahead, but Pidgin definitely benefited from more widespread use for being a multi-protocol client. Also Psi's stagnation and semi-fork in Psi+ didn't help it.

As a former XMPP advocate, I can only sigh.

Matrix: a new specification for federated realtime chat

Posted Feb 20, 2015 21:25 UTC (Fri) by flussence (guest, #85566) [Link] (2 responses)

Pidgin is an ego-stroking exercise, Psi is growing cobwebs and its UI is straight out of the 90s, Jitsi is stereotypical enterprise software... I gave up on desktop clients and now do my IMing from Xabber.

Most of the features I want in an XMPP client are completely missing there (there's space reserved for your avatar, when you can't *set* it), but at least its complexity feels proportional to what it *does* provide. I can send/receive text and it handles Unicode properly. It's even actively developed these days, after considerable pressure from the users.

Matrix: a new specification for federated realtime chat

Posted Feb 23, 2015 5:29 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (1 responses)

> after considerable pressure from the users.

They went through all the work to GPL it and then it seemed to just…die. I've seen updates go by in F-Droid, so I had guessed something picked back up. Good to hear, but I've since moved onto Conversations (which is also Android 5-ish in appearance).

Matrix: a new specification for federated realtime chat

Posted Feb 23, 2015 16:54 UTC (Mon) by flussence (guest, #85566) [Link]

I've tested the Xabber 0.10 versions in F-Droid - the UI has finally caught up, but there's zero new functionality besides that and I also ran into tons of bugs using it. (And then Google's idea of OS package management helpfully destroyed a year of chat history when I tried to downgrade...)

I might switch to Conversations for the time being, or at least keep it around as a backup client.

Matrix: a new specification for federated realtime chat

Posted Feb 20, 2015 13:22 UTC (Fri) by TRauMa (guest, #16483) [Link] (1 responses)

After reading your comment I went and looked at the issue in question just to see what was so terrible that it made Google give up on XMPP.

Boy, this is one very sad issue indeed. And not because the change by the pidgin team, they stated why and how they did it, and that many people saw a bug (tiny text entry area) and didn't know that you could just start typing without needing focus on the entry window.

Then the devs agreed to a larger max size for the window than 4, which is what happened in the next released version (so, at the next possible point in time). But in the meantime the bug report is spammed with plenty of people posting the same things again and again, even if they have been addressed already before (like said tiny field bug).

So let's be real here, this was maybe a mistake in communication, but not the reason XMPP never took on. XMPP, from it's inception, always had the problem that it was never as easy or even discoverable as other IM services, and in all the years I used it, only tech-related people ever were in my contact list, until they started to drop of, one after the other.

IM is one of those things where it doesn't matter if your solution is open and superior to one-vendor proprietary stuff if few of the people you want to message happen to have the client of your choice open. It's just that happened with mobile messaging, I have three clients, I actually like one, and I actually use WhatsApp.

Matrix: a new specification for federated realtime chat

Posted Feb 20, 2015 19:05 UTC (Fri) by pizza (subscriber, #46) [Link]

> So let's be real here, this was maybe a mistake in communication, but not the reason XMPP never took on. XMPP, from it's inception, always had the problem that it was never as easy or even discoverable as other IM services, and in all the years I used it, only tech-related people ever were in my contact list, until they started to drop of, one after the other.

You hit the nail on the head -- XMPP is a protocol, not a service.

Providing a service costs money, and if you don't charge directly for it, you have to make up for it in other ways, eg selling ads via some sort of captive client.

XMPP's open nature directly conflicted with the economic model of providing an IM service, so there was never much incentive (from the service provider's perspective) to use XMPP.

Matrix: a new specification for federated realtime chat

Posted Feb 13, 2015 4:58 UTC (Fri) by drag (guest, #31333) [Link]

I really did have high hopes for Jingle. This makes me sad.

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 16:15 UTC (Thu) by raven667 (guest, #5198) [Link] (1 responses)

> Yes, group chats were added later. And I suggest you blame said clients for their poor support; many others have no problems whatsoever.

I think that any time you see poor support of a protocol across multiple clients like this you have to apply a simple razor when assigning blame, either all the developers of all the clients are unusually stupid or the protocol is unusually difficult to implement. Which do you really think is the more likely explanation?

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 18:32 UTC (Thu) by pizza (subscriber, #46) [Link]

> I think that any time you see poor support of a protocol across multiple clients like this you have to apply a simple razor when assigning blame, either all the developers of all the clients are unusually stupid or the protocol is unusually difficult to implement. Which do you really think is the more likely explanation?

...or, perhaps, both are true? In the case of XMPP, the specs are undeniably complex, but at the same time, so is the problem space it's trying to address.

What I've personally found disappointing is the lack of collaboration among the various client (and server) developers -- Every implementation seems to use a different language. But then there's the Android situation where you have half a dozen forks of the same underlying library, and *still* nobody's sharing. That's not a spec problem.

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 22:41 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

> Yet it remains the *only* option there is if you want an IM solution that doesn't require you to sign your soul (and your messages) over to $BigCorp.

You could hand out accounts to one machine and have everyone use `wall` and `talk` to message each other. You could even get fancy and have people attach to tmux sessions to do group stuff. So, no, XMPP isn't the *only* option. You just need some imagination ;) .

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 5:21 UTC (Thu) by wahern (subscriber, #37304) [Link] (4 responses)

It's made around a concept of permanent "associations" between the endpoints (so a server can store messages while endpoints are offline)

Yet Matrix has a similar setup. Clients connect to "home servers", which are relays for receiving and broadcasting message events. Basically store-and-forward. From section 5.1 of the specification:

A "Client" typically represents a human using a web application or mobile app. Clients use the "Client-to-Server" (C-S) API to communicate with their home server, which stores their profile data and their record of the conversations in which they participate.

I'm not saying that I like XMPP. I just don't see how Matrix is really different, except that it's "web scale" because it ditched XML in favor of JSON over HTTP. Plus IRC has received a resurgence of interest (if the interests of the junior engineers at my company are any indication). So I give them bonus points for mashing together HTTP, JSON, and IRC.

FWIW, I definitely prefer JSON over XML, simply because it's simpler, easier to parse, and maps more closely to language data structures, whether in C, Python, or something else. But tell me how Matrix solves the more serious problems which plague XMPP? Namely, the explosion of competing and complex extensions.

Matrix talks about supporting WebRTC and VoIP as a session initiation channel, but the devil is in the details! Matrix works no better than XMPP in this regard.

Matrix strikes me as the answer to the question, "how do you re-invent XMPP, avoiding design-by-committee, incorporating MUC, and making it easier to implement relays in your preferred language."

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 5:44 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

> Yet Matrix has a similar setup. Clients connect to "home servers", which are relays for receiving and broadcasting message events. Basically store-and-forward. From section 5.1 of the specification:
XMPP is missing that distributed _stateless_ store-and-forward support. In XMPP each server has to store all of the state to remember which messages are yet to be sent to other servers while in Matrix servers can simply use an rsync-like protocol.

Matrix can be thought of as a new incarnation of USENET (now with real-time messaging!) rather than IRC.

> FWIW, I definitely prefer JSON over XML, simply because it's simpler, easier to parse, and maps more closely to language data structures, whether in C, Python, or something else. But tell me how Matrix solves the more serious problems which plague XMPP? Namely, the explosion of competing and complex extensions.
Personally, I don't really care about XML vs. JSON - both work perfectly fine and XML is actually better specified (I still miss XML schemas while using JSON, sigh).

The main feature that I like in Matrix is its use of distributed consensus protocols rather than explicit state storage like in XMPP. This single bit of design can do wonders.

Other than that, I agree that it's really early to tell if Matrix can avoid XMPP's pitfalls. But I have some hope and that's more than I can say about XMPP.

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 12:50 UTC (Thu) by Arathorn (guest, #101018) [Link]

From my perspective, XMPP is all about message passing, whereas Matrix is all about state synchronisation (yes, I know they're two sides of the same coin, but they are opposite ways of addressing a similar problem). Matrix is basically an global EC DB with open federation and pubsub. A good analogy is that XMPP is like SMTP (with some storage extensions like IMAP), whilst Matrix is like NNTP. It's worth noting that XMPP does have some extensions like Buddycloud and FMUC which do provide decentralised persistent messaging - but with Matrix we wanted to see what'd happen if we just built on plain HTTP/JSON rather than the XMPP stack.

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 17:23 UTC (Thu) by jnareb (subscriber, #46500) [Link] (1 responses)

> (I still miss XML schemas while using JSON, sigh)

There is JSON Schema, though

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 18:31 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

Yes, and there are several of them. And none of them are standard or even capture the actual practical JSON use.

Matrix: a new specification for federated realtime chat

Posted Feb 12, 2015 12:39 UTC (Thu) by Arathorn (guest, #101018) [Link] (1 responses)

Matrix absolutely doesn't rely on the logically centralised ID server to function. The ID system is currently in flux (hence not being documented properly in the spec), but we've deliberately kept it layered away from the core communication layer, so that Matrix isn't dependent on it (and so we can swap it out in future). And in fact the current implementations barely use it at all, other than to *optionally* register using an email address if you so desire.

The underlying 'opaque' Matrix IDs look like @matthew:matrix.org - and like JIDs or SIP URIs are indeed DNS backed, and works just as well (or otherwise) as XMPP.

We want to do better than XMPP & SIP (& IRC) by also linking 3PIDs to theses matrix IDs - we have no expectations of foisting yet another username@domain-style namespace on the world. Whilst right now this is logically centralised (but physically decentralised, as it happens) in ID servers, the plan is to adopt a proper decentralised ID system instead - ideally someone else's, so we don't reinvent that wheel. But we haven't got to it yet :)

In terms of novel features - yes, the git-style crypto-signed distributed DAG of conversation history with eventual consistency is the main innovation (and is shamelessly lifted from Git). Decentralised ACLs are also pretty cool. Relying on 3PID mapping for identity (when it lands) will be too.

Well done for spotting the Croquet similarity - needless to say, Matrix is about more than just instant messaging, and we want it to be a general persistent JSON distributed database with EC, pubsub and strong crypto semantics. And that could be for *any* data - be it Slack-style collaboration, or whiteboarding, or indeed Croquet-style virtual worlds & cyberspace. It's called Matrix for a reason ;D

Matrix: a new specification for federated realtime chat

Posted Jul 21, 2015 3:55 UTC (Tue) by sethml (guest, #8471) [Link]

Have you looked at Gale? It was written by a friend of mine, and had a very active community for a while. It featured quite seamless federation and end-to-end public key encryption with surprisingly easy to use key management and discovery. Its downfall was really an over-reliance on the everybody-has-an-account-on-a-*nix-server model, which kept the community tidy for a while, then killed it.