Combating abuse in Matrix - without backdoors (Matrix blog)

[Posted October 20, 2020 by corbet]

This Matrix blog entry describes a planned reputation-management system that, it is claimed, accomplishes some of the same goals as government backdoors without the need to compromise end-to-end encryption. "Just like the Web, Email or the Internet as a whole, there is literally no way to unilaterally censor or block content in Matrix. But what we can do is provide first-class infrastructure to let users (and room/community moderators and server admins) make up their own mind about who to trust, and what content to allow. This would also provide a means for authorities to publish reputation data about illegal content, providing a privacy-respecting mechanism that admins/mods/users can use to keep illegal content away from their servers/clients."

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 20, 2020 14:20 UTC (Tue) by arcivanov (subscriber, #126509) [Link] (8 responses)

Yes, because reputation-based mechanisms are never chronically and severely abused, right?

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 20, 2020 14:40 UTC (Tue) by nix (subscriber, #2304) [Link] (2 responses)

Well, at least if you have multiple reputation sources, any abuser would need to find a way to abuse *all of them at once*. This seems likely to get increasingly difficult the more of them there are.

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 20, 2020 15:20 UTC (Tue) by smoogen (subscriber, #97) [Link] (1 responses)

But the more you have the harder it is to know which ones to trust. Basically how do you know if "TheTruther" or "TheRealTruther" or "TheGuarenteedTruther". It is a low cost method to do deep cons to just set up lots of "reputation" sources and then "mark" each one with high reputations by sock puppets and some 'real' accounts. I expect that the business plans are already in place and by March you will be able to by 1000's of reputation points from various companies for your accounts.

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 20, 2020 16:47 UTC (Tue) by raven667 (subscriber, #5198) [Link]

My read is that this is more useful to server admins who can use the infrastructure to trust blocklists made by other server admins they have a relationship with, so if you run matrix.example.org and matrix.org bounces someone for violating their behavior standards, you can accept that as well so that the person tied to that identity can't just create a new account on your server and keep harassing others. Server admins with broadly compatible community standards can have federated blocklists which are complementary with federated communication, centralized blocklists of harmful content can be subscribed to as well, and end-users can use the same mechanisms to curate their own feed so as to not be exposed to harassment, a formal version of a whisper network.

I'm sure none of this will be perfect, but it doesn't have to be, it just has to be better

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 20, 2020 18:18 UTC (Tue) by martin.langhoff (subscriber, #61417) [Link]

Linkfarms FTW!

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 20, 2020 18:24 UTC (Tue) by Wol (subscriber, #4433) [Link] (1 responses)

If I rely on J Random Joe to say who is reputable, yes.

If I flag the people *I* trust, and then the algorithm preferentially weights the people they trust as "people I trust", then that's hard to game. It's expensive to implement, though.

And it's also susceptible to the "echo chamber" effect - if I trust people I *like*, then I'll only see stuff I agree with. If I trust people I *respect*, then I'll see a far wider spread. There's quite a few people here I've had run-ins with, but whether I like them or not I respect their honesty, skills and integrity. I hope there's people here who feel the same way about me - I'm pretty certain there are some :-)

But the problem with all of this is that the more voices there are out there competing to be heard, it's almost inevitable that those voices that disagree with you are going to be preferentially filtered out, even if their arguments are good and compelling ... the more people who filter on "like" the less impact filtering on "respect" will have :-(

Cheers,
Wol

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 25, 2020 19:34 UTC (Sun) by NYKevin (subscriber, #129325) [Link]

It's even worse than that. If you strongly disagree with an argument, you may perceive it as trolling, or at least as a low-quality argument that doesn't deserve to be taken seriously. You may feel that the author has deliberately failed to take some fact into account, when that fact is perfectly obvious to you. You may feel that the author's position has been debunked many times already, and that continuing to argue the point is a sign of bad faith. If the argument is political, you might feel that the author's position is disrespectful of the people who would (in your view) be harmed by it becoming actual policy. And so on...

As a result, even if everyone filters on "respect" instead of "like," you will still get bubbles or echo chambers, they'll just be a bit broader than you might otherwise see.

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 21, 2020 10:37 UTC (Wed) by k3ninho (subscriber, #50375) [Link]

However let's not dismiss an attempt at a solution because the principle has failed in the past in a different implementation. Your rhetorical approach is flawed.

The problem is hard, they know that. And Element are looking to adopt an approach, learn from it and enable their userbase to protect itself according to their own free choices. We build communities from meaningful human engagement, and this looks to protect those communities with insight as to who or what they should trust. It still remains a hard problem.

K3n.

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 22, 2020 15:22 UTC (Thu) by cyphar (subscriber, #110703) [Link]

On the other hand, systems with no reputation or abuse moderation systems are also chronically and severely abused (usually by a similar set of people). I don't think this is an impossible problem to solve, and I think it's still useful for folks to try to come up with solutions to this problem.

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 21, 2020 2:08 UTC (Wed) by IanKelling (subscriber, #89418) [Link]

This is great. In a communication system like matrix, or email, where reading messages from strangers is normal, it seems like having a common and built-in reputation system with a free software implementation would be a great advancement. In email, adhoc reputation systems are widely deployed and could be so much better.

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 21, 2020 7:53 UTC (Wed) by Sesse (subscriber, #53779) [Link] (2 responses)

How is this meaningfully different from the PGP web of trust, which we pretty much know by now isn't working?

Also, what reputation would you give a completely unknown user on your server? (Assume that user could be a person who's never used Matrix before, _or_ that person you just banned who just created a new identity.)

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 21, 2020 8:01 UTC (Wed) by mjg59 (subscriber, #23239) [Link]

The PGP web of trust is largely set up out of band - I can't say "I trust foo@example.com", I need some way to have a trustworthy conversation with foo@example.com where I obtain their key. This is then exacerbated by there being no commonly supported way to obtain foo@example.com's key, instead forcing me to use a keyserver the provides no guarantee that someone other than foo@example.com uploaded a key (potentially with a whole bunch of fake signatures on it, which I can't verify unless I've managed to bootstrap my way into the web of trust sufficiently well that I have a path to one of those signatures). If the reputation is in-band, I can just say "Based on foo@example.com's behaviour, I trust them" and then that trust propagates in a reasonable way to anyone who's said that they trust me to make good decisions.

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 24, 2020 13:47 UTC (Sat) by gdt (subscriber, #6284) [Link]

I think that's a bit harsh on the PGP Web of Trust. The reason that doesn't work well is because ubiquitous deployment of the web of trust was resisted ferociously by the Wassenaar Arrangement treaty partners. So PGP never made it into the baseline e-mail client Pine, and thus into the feature-set of competing products such as Netscape Communicator or Microsoft Outlook as part of the typical user experience.

We're still paying the price for that decision of the intelligence/defence community. The success of phishing crimes is partly because e-mail clients lack a rigorous notion of trust.

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 22, 2020 1:19 UTC (Thu) by landley (guest, #6789) [Link] (4 responses)

How does this differ from China's https://en.wikipedia.org/wiki/Social_Credit_System ?

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 22, 2020 8:16 UTC (Thu) by anton (subscriber, #25547) [Link] (3 responses)

You decide which reputation sources you use, not China's government. Also, the only effect of ruining your reputation on Matrix is that many people on Matrix don't read you; your can still travel, your children can still go to national universities, etc.

What I wonder about the system is how they deal with new accounts. If someone with a bad reputation can make a new account and start again with a clean slate, the end result may be that many will only read accounts with a good reputation. But then, if you are new and nobody reads you, how do you get a good reputation?

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 23, 2020 7:02 UTC (Fri) by hifi (guest, #109741) [Link] (2 responses)

If the system is truly anonymous a scheme like this just can't work.

Creating an account is free and anonymous and if you create a system that requires some sort of trust level to participate it essentially makes getting in hard or even impossible.

On the other side of the spectrum if you make duplicate account creation almost impossible you can "trust" a person on a server is truly who they claim to be regardless if you agree on their views.

Some sort of verified account system which gives you undisputable identity across the network would be the best way to combat abuse and spam by creating communities that require said verification. Then you can start filtering based on views and behavior.

The problem with all this is that what entity would you trust to verify people? That could be simplified to known trusted servers that are operated by people who claim to verify there's a real person behind an account but that itself is subject to abuse by the administrators.

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 23, 2020 7:59 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

It's more correctly to say that is "pseudonymous". You can stay anonymous by creating multiple identities, but they will have a low reputation.

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Nov 4, 2020 10:55 UTC (Wed) by anton (subscriber, #25547) [Link]

There are some ways to deal with this: postings from zero-reputation or even negative-reputation accounts can be shown to a few people, with a delay, and with the information that the posting is not widely shown, and that answering the posting may give the original posting wider audience than it has now. Or maybe quarantine an answer to such a posting in a similar way as the posting itselves; "Do not feed the troll" has seen limited success in Usenet, but with some audience-limiting measures it might work better.

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 23, 2020 13:50 UTC (Fri) by enkiusz (guest, #142702) [Link]

Most countries who signed the bill are members of the Five Eyes espionage collaboration. They are trying to keep their sigint relevant in a world where all the Chinese agents can use signal and telegram with impunity (while banning working e2e encryption on their own network).

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Oct 25, 2020 17:13 UTC (Sun) by jnxx (guest, #142729) [Link] (1 responses)

Wasn't how to establish a reliable network one of the issue which koro5in.org tried to deal with? And which turned out to be really hard?

https://en.wikipedia.org/wiki/Kuro5hin

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Nov 10, 2020 12:02 UTC (Tue) by ksandstr (guest, #60862) [Link]

The problem with Kuro5hin's "mojo" system wasn't that the problem is hard, it was that the users of that site were technically adept and thus whatever patches mr. Foster would come up with would be immediately exploited and laughed at. In today's world an advanced society such as the United States of America, or the People's Republic of China, should surely use AI to identify similarly disrespectful and/or dangerous subjects and control them effectively, therefore eliminating the issue in the crib. "No man, no problem", as a certain Georgian is quoted.

Combating abuse in Matrix - without backdoors (Matrix blog)

Posted Nov 10, 2020 12:35 UTC (Tue) by ksandstr (guest, #60862) [Link]

That's a solution to one problem. But the other, equally severe, remains: that censorship is always applied in a way where those affected cannot confirm that only the things they disagree with are being censored[0].

Indeed this proposal of what's essentially a government-distributed list of distrusted hashes (media fingerprints, whatever) appears to enable not just censorship, but also the persecution of those who do not subscribe to the Official Naughty List. To wit, the ONL would identify a superficially benign piece of media that'd be monitored by agents (bots) of The Man to surveil nodes where the list is not being obeyed so as to give their operators a discretionary Social Demerit; and not for breaking any law, but for turning Fritz off.

[0] the standard counterargument is that "perfect is the enemy of good". In the case of censorship, the argument goes, irreparably opaque and infinitely tyrant-friendly censorship is better than nothing, so it should be preferred. Subsequently advocates are very surprised at allegations that critique of censorship is among the first things suppressed: certainly it's more likely[1] that everyone trusts censorship and regards it s/h/its friend.
[1] and the argument is always made in terms of seat-of-the-pants handwaving about "likelihood" because censorship cannot be discussed[2] from concrete facts, those having been memory-holed.
[2] woop woop that's the KKK right thur, woop woop