Microblogging with ActivityPub [LWN.net]

November 29, 2022

This article was contributed by Jordan Webb

As of late, concerns about the future of Twitter have caused many of its users to seek alternatives. Amid this upheaval, an open-source microblogging service called Mastodon has received a great deal of attention. Mastodon is not reliant on any single company or central authority to run its servers; anyone can run their own. Servers communicate with each other, allowing people on different servers to send each other messages and follow each other's posts. Mastodon doesn't just talk to itself, though; it can exchange messages with anything that speaks the ActivityPub protocol. There are many such implementations, so someone who wants to deploy their own microblogging service enjoys a variety of choices.

ActivityPub is a W3C Recommendation that describes how servers can exchange messages in the form of JSON Activity Streams. The ActivityPub protocol is highly flexible and is used for all sorts of things, but the scope of this article is limited to software that uses it to provide a Twitter-like microblogging service. The idea of federated microblogging was pioneered by a project called StatusNet, which was the software that originally ran at identi.ca, an early Twitter alternative. Instances of StatusNet, and eventually other software, were federated with each other using a protocol called OStatus.

ActivityPub and OStatus are not interoperable, but they have some parentage in common; Evan Prodromou, the creator of StatusNet, subsequently created pump.io and changed identi.ca to use it. Though development on pump.io seems to have stalled, its API formed the basis of ActivityPub. Some projects continue to support both protocols; Mastodon removed support for OStatus in 2019.

All of the various servers running this software exist in a federation somewhat like email, colloquially known as "The Fediverse"; usernames are similar to Twitter handles, but also include a domain component that identifies the server hosting the account. For example, LWN has an account at @LWN@fosstodon.org; it is hosted on Fosstodon, which is a Mastodon server for people interested in free and open-source software.

Mastodon

Mastodon is a Ruby on Rails application; its source is available on GitHub and it is released under the AGPL 3.0. A working installation of Mastodon has a number of moving parts. Two different instances of the web application are needed; one to handle normal requests and a separate one for the streaming API that serves real-time updates to clients. It also needs a PostgreSQL database and a Redis server for caching. At least one instance of the Sidekiq job scheduler is needed to handle background jobs; busy servers may need to configure several of them in order to remain responsive under load. Elasticsearch is also needed to fully enable Mastodon's search capabilities, although it can be run without Elasticsearch at the cost of degraded functionality. Running all of this requires significant resources; the Raspberry Pi Foundation hosts its own instance on a Pi 4, but it only plays host to a single account. Someone who intends to have more than a handful of users on their server would be wise to budget for something more powerful, as system requirements scale up relative to the number of incoming and outgoing messages that a server needs to process.

Mastodon's web interface appears to have taken most of its design cues from Twitter. It also offers a multi-column "advanced" interface, similar to TweetDeck. Mastodon has an official mobile application but it is oddly incomplete; in particular, it does not offer access to the federated timeline where posts from other servers may be viewed. Alternative clients are available to fill the gap; Metatext, tooot, and Tusky are more fully-featured alternatives. Note that while the ActivityPub standard specifies both client-to-server and server-to-server communications, Mastodon only implements the server-to-server side. Mastodon clients are not generic ActivityPub clients; they communicate with the server using a Mastodon-specific API.

Though it has received a lot of attention lately, Mastodon is not a new project; development began in 2016 and the first stable release was in 2017. Mastodon's development is sponsored by a number of companies along with donations via Patreon, where it currently receives about $31,000 per month.

Mastodon's creator, Eugene Rochko, runs the project on a "benevolent dictator" model and a number of forks have sprung up to add support for features that upstream is not interested in merging. The first feature that many Mastodon forks add is a way to increase the maximum length of posts; upstream Mastodon limits post to 500 characters and offers no way to configure a higher limit, though the ActivityPub protocol imposes no such limitation and Mastodon has no problem displaying longer posts from other servers. Glitch Edition is a fork that tracks Mastodon's main branch closely and includes a completely overhauled user interface. Hometown is a more conservative fork, based on stable releases of Mastodon and focused on adding noninvasive quality-of-life improvements.

Pleroma

Pleroma is written in Elixir and its source is available on the project's own GitLab server; most of it is licensed under AGPL 3.0, but some assets and documentation are covered by Creative Commons licenses instead. Pleroma is known for being more lightweight than Mastodon, and it is significantly less complex from an operational perspective; the only additional service that it needs is a PostgreSQL database. Despite this, it also includes some features that Mastodon lacks; while Mastodon only permits marking a post as a "favorite", Pleroma extends ActivityPub's "Like Activity" to allow users to react to posts with an emoji, similar to Facebook and LinkedIn. It also includes a realtime chat feature. Unlike Mastodon, Pleroma allows the maximum post length to be changed and it defaults to a generous 5000 characters.

Pleroma's default web interface is much less obviously inspired by Twitter than Mastodon's, with a full-window background image and semi-transparent UI elements. It has no official mobile app of its own, but Pleroma implements much of the Mastodon API, so many clients intended for use with Mastodon will work with it. Pleroma's documentation includes a list of working clients.

Perhaps in part due to its ease of deployment, Pleroma has acquired a bad reputation in many communities. This is not for any technical reason, but because it frequently seems to be the software of choice for people aiming to abuse and harass people on other servers over their race, gender, sexual identity, or other characteristic. Many unmoderated "free speech maximalist" servers also run Pleroma, and some of its developers are active in those communities; these types of servers often come into conflict with (and are subsequently blocked by) servers with moderation policies that prohibit abusive content. LWN readers who do not wish to be exposed to such content are advised to steer clear of the "Featured Instances" on Pleroma's website.

Pleroma's commit history dates back to 2017; version 1.0.0 was tagged in 2019. Unlike Mastodon, Pleroma does not solicit donations to fund development, and claims no corporate sponsors. Pleroma has spawned at least one fork; Akkoma was inspired at least in part by differing views about content moderation. Akkoma adds a number of features to Pleroma, like support for search, integration with translation services, and improved domain blocking. Its developers have also added a code of conduct.

Misskey

Misskey is written in TypeScript; its source is available on GitHub and, like Mastodon and Pleroma, uses the AGPL 3.0 license. A deployment of Misskey weighs in somewhere between Mastodon and Pleroma; like Mastodon, it needs PostgreSQL and Redis, and can optionally make use of Elasticsearch, but it does not need separate processes for streaming APIs or background-job processing. Like Pleroma, Misskey allows administrators to configure the maximum post length (3000 is the default), and throws in a few more features that have no equivalent in Mastodon. Misskey originated custom emoji reactions; Pleroma's implementation was built to be compatible with it. Misskey also offers a cloud-storage feature called "Drive" that can be used to share files and images.

Misskey's core developers are from Japan, and it is most popular among Japanese-speaking communities. English translations are available for the user interface and documentation, though. Its web interface is more Twitter-like than Pleroma, but more visually distinct from Twitter than Mastodon. Misskey has no official mobile client, but unlike Pleroma, Mastodon clients can't be used with it because it does not implement the Mastodon API. At the time of this writing, the list of clients was missing from the English documentation, but a list of clients is available in Japanese.

Development of Misskey began in 2014, but the commit history of the current version only goes back to 2016; version 1.0.0 was tagged in 2018. It was initially built as a standalone bulletin board; support for ActivityPub was added on later. Misskey accepts donations on Patreon and currently receives a bit more than $700 a month; it also apparently accepts corporate sponsors, although only one is listed. Misskey also has its share of forks; Calckey adds a long list of user-interface improvements, while FoundKey is focused on fixing bugs and making the code more maintainable.

Others

Mastodon, Pleroma, and Misskey are currently the three most popular choices for ActivityPub microblogging servers, but there are a number of other options. The AGPL 3.0 license seems to be unusually popular in this space; it is used by all of the projects listed below, unless otherwise mentioned.

GNU social is a PHP microblogging server with an interesting history. The initial version of GNU social was originally spun out of the code that powers Libre.fm, but current versions are direct descendants of the original StatusNet codebase, which has been extended to support ActivityPub. Development of GNU social is split into two branches, in two different places; version 2 appears to be in maintenance mode on NotABug.org, while version 3 is receiving frequent commits on Codeberg.
GoToSocial is a lightweight ActivityPub server written in Go. It is still in alpha, but rapidly maturing; source is available on GitHub. While Mastodon, Pleroma, and Misskey require PostgreSQL, GoToSocial also supports SQLite as an option for people who don't want to run a separate database server.
Takahē is another alpha-stage project, which just made its first numbered release. It is written in Python and stores its data in a PostgreSQL database. Takahē's distinguishing feature is its support for virtual hosting; other ActivityPub servers only support a single domain per instance, but Takahē can support multiple domains on a single instance. Source is available on GitHub under the three-clause BSD license.
Socialhome has an unusual user interface which owes more to Pinterest than to Twitter. It is written in Python and stores its data in PostgreSQL; it also needs Redis. Source is available on GitHub.
microblog.pub is a single-user ActivityPub server written in Python that stores its data in SQLite. In addition to ActivityPub, it supports a number of IndieWeb standards, including IndieAuth, Micropub, Webmention, and microformats. Source is available on SourceHut.
honk is another single-user ActivityPub server, with a minimalist aesthetic. It is written in Go and stores its data in SQLite. The license is described as "generally ISC or compatible." Source is available from the project's website, which is also a Mercurial repository.
Friendica is more focused on social networking than microblogging, with an interface and set of features that resemble Facebook more than Twitter. Friendica predates ActivityPub but interoperates with it, along with a number of other protocols. A collection of optional addons is available to connect Friendica to other services; a bidirectional gateway to Twitter is included. It is written in PHP, and stores its data in MySQL or MariaDB; source is available on GitHub.
Hubzilla is another PHP project from the original author of Friendica. While Hubzilla's website and documentation mostly highlight its support for the Zot protocol, it also interoperates with ActivityPub. Source is available on Framasoft's GitLab server. Hubzilla's authors have chosen to publish the source under the MIT license.

The year of the Mastodon?

Many media outlets are covering Mastodon as a "new alternative social network", but it is neither new, nor is it a network unto itself. Mastodon has been around since 2016 and some of the projects it interoperates with are even older. It is true that many of the ActivityPub-speaking servers on the internet are running Mastodon, and indisputable that it has been the primary beneficiary of the current wave of attention, but the collection of servers that make up the Fediverse is far from a monoculture; many thriving alternatives exist.

As a result of the uncertainty around Twitter, public servers have been inundated with new users. It remains to be seen if many of these people will stick around, but ActivityPub was a tried and proven solution with a healthy base of users long before the current spike in interest. There is no reason to think that an established standard with multiple robust implementations is going to go away any time soon, even if its current moment in the sun turns out to be fleeting.

Index entries for this article
GuestArticles	Webb, Jordan

to post comments

Microblogging with ActivityPub

Posted Nov 29, 2022 21:30 UTC (Tue) by mricon (subscriber, #59252) [Link] (4 responses)

Shameless plug: if you are a kernel subsystem maintainer and want a place in the Fediverse, you qualify for an account on https://social.kernel.org.
For details, see: https://korg.docs.kernel.org/social.html

Microblogging with ActivityPub

Posted Nov 29, 2022 21:47 UTC (Tue) by palmer (guest, #84061) [Link]

I just jumped ship from Twitter and the kernel.org setup is pretty great, thanks for setting this up!

Microblogging with ActivityPub

Posted Nov 29, 2022 22:24 UTC (Tue) by jordan (subscriber, #110573) [Link] (1 responses)

There is a very sneaky plug for social.kernel.org in the Pleroma screenshot :)

Microblogging with ActivityPub

Posted Nov 29, 2022 22:45 UTC (Tue) by mricon (subscriber, #59252) [Link]

> There is a very sneaky plug for social.kernel.org in the Pleroma screenshot :)

Ooh, I didn't even notice. Very sneaky! :)

Microblogging with ActivityPub

Posted Nov 30, 2022 18:08 UTC (Wed) by jond (subscriber, #37669) [Link]

I was on the other, older one, kernel.social, which got switched off a year or more ago, and I had to start over somewhere else.

Rebased (formerly Soapbox)

Posted Nov 29, 2022 21:46 UTC (Tue) by mricon (subscriber, #59252) [Link] (1 responses)

Rebased is another notable fork of Pleroma, done for different reasons than Akkoma folks and aiming to go in a different direction (scalable deployments, improved database backend, etc). As it's a recent enough fork, there isn't that much distinguishing it from upstream Pleroma yet, but it's a project to also consider watching if you're evaluating self-hosting options.

https://soapbox.pub/2022/08/19/soapbox-be-is-now-rebased/

Rebased (formerly Soapbox)

Posted Dec 1, 2022 16:06 UTC (Thu) by Gaelan (guest, #145108) [Link]

Note that Soapbox, even more so than Pleroma, has a reputation for being used almost exclusively by the far right.

Microblogging with ActivityPub

Posted Nov 29, 2022 23:06 UTC (Tue) by flussence (guest, #85566) [Link]

I saw this headline and was bracing for the worst after a long month of “blind men and the elephant”-style mainstream and tech media attention. But no - this article has done the homework!

Microblogging with ActivityPub

Posted Nov 29, 2022 23:16 UTC (Tue) by ruscur (guest, #104891) [Link] (2 responses)

I wish Mastodon didn't have such a complex stack. It needs Rails, Node.js, PostgreSQL and Redis just to get off the ground. I'm hosting a small instance for my friends and it really feels like overkill, maybe I should have gone with Plemora et al. It hasn't needed much maintenance after I set it up over the past month, at least.

Performance scaling and content moderation seem to be the two hardest challenges facing server admins - everything is reasonably comfy if your server is only for you, or people you know.

Microblogging with ActivityPub

Posted Nov 29, 2022 23:18 UTC (Tue) by ruscur (guest, #104891) [Link]

Also if anyone's new to the Fediverse and is looking to find more kernel people, you can import accounts to follow from this list: https://github.com/pdp7/mastodon-lists/blob/main/linux-ke... - though there's plenty missing from that list that could be added too.

Microblogging with ActivityPub

Posted Nov 29, 2022 23:45 UTC (Tue) by flussence (guest, #85566) [Link]

If your use case is a web server for a small crowd, and you decide to switch, GoToSocial seems like the most interesting one at present.

It's the only one that seems to be trying anything new w.r.t. robust moderation; Mastodon's system is reviled for its uselessness by most of the site owners I've talked to who've had to use it, and Pleroma's was written by someone who had to leave the project years ago and has had zero upkeep since. Both of the big two fail Postel's Law badly - there is no way to express “do not transact with this remote instance in any way” unless you become a domain expert at nftables and DNS RPZ chicanery, which isn't a burden I'd wish upon my worst enemy.

There's plenty other faults but the lack of access control has been an elephant in the room, so to speak, for going on half a decade.

Microblogging with ActivityPub

Posted Nov 30, 2022 1:29 UTC (Wed) by tau (subscriber, #79651) [Link] (9 responses)

>All of the various servers running this software exist in a federation somewhat like email, colloquially known as "The Fediverse"

And it will need to come up with a solution to the problems that plague "federation[s] somewhat like email". A spammer with some compromised AWS API keys or stolen credit card numbers can easily taste a few hundred domain names, spin up a few hundred EC2 instances to go with them, then inundate popular Fediverse servers with garbage during the brief period before the instances created by the compromised keys are deleted and their speculative domain registrations need to be refunded. Instance operators are overworked and underfunded as it is, a flood of this sort of garbage could quickly render the ecosystem unusable.

I wish this movement the best of luck, but the management of every electronic communication service eventually reduces to a spam filtering problem. Nobody has figured out a way to make blocking spam cheaper than sending spam, the best solution we have found so far is to have large, centralized, for-profit networks that employ a large body of low-wage overseas labor to frantically mop up enough of the sewage that the patrons don't notice the smell too much.

Microblogging with ActivityPub

Posted Nov 30, 2022 2:08 UTC (Wed) by dskoll (subscriber, #1630) [Link]

This is absolutely correct.

However, one way to somewhat mitigate your specific attack scenario is to refrain from sharing posts from newly-seen servers for a week or two (but allow admins to sample them.) This could make the attack too expensive and the bad actors more likely to be shut down before they can spread garbage to the general public.

There's also a trust issue. You have to trust the admins of your ActivityPub server to do their due diligence before accepting content from random instances. (I assume they can control this? Not 100% sure.)

So far, at least, Mastodon is way better than Twitter when it comes to civility. But now that George Takei is on, we'll see how well it scales. 🙂

Microblogging with ActivityPub

Posted Nov 30, 2022 9:17 UTC (Wed) by rqosa (subscriber, #24136) [Link] (7 responses)

> it will need to come up with a solution to the problems that plague "federation[s] somewhat like email".

The problem of email spam-filtering is pretty much a solved problem by now. My personal email account has been using SpamAssassin for a long time, and while not perfect, it seems to be "good enough" at least. (And one good thing about it is that whenever a false-positive occurs, the end user can still see the email message that was flagged as spam, as long as they've configured their MUA's filtering rules to direct spam into a "spam folder" and then read that folder occasionally.)

So it seems like a similar appoach (using Bayesian filtering) could eventually be used for ActivityPub as well. Furthermore, (same as with email) it might also be possible for ActivityPub spam-filtering to be done on the client-side, for end users who don't want their server operators to be in full control of the spam-filtering.

Microblogging with ActivityPub

Posted Nov 30, 2022 9:43 UTC (Wed) by Wol (subscriber, #4433) [Link] (5 responses)

The problem with that approach is that - because spam filtering is so good - users don't check their spam filters.

The combination of Thunderbird filters and server-side spam marking works so well for me that when stuff does get misclassified, it can VERY easily get mass-deleted by mistake ... usually stuff I actively want :-(

Cheers,
Wol

Microblogging with ActivityPub

Posted Nov 30, 2022 11:08 UTC (Wed) by anselm (subscriber, #2796) [Link] (3 responses)

The problem with that approach is that - because spam filtering is so good - users don't check their spam filters.

Having to check a spam filter defies the purpose of using one in the first place. The whole point of having a spam filter is not having to look at the spam.

I don't have a spam folder. I'm running a pre-queue spam checker and stuff that looks like spam is refused by my server while it is still in the process of being submitted, with an SMTP error code. If a legitimate message is misidentified as spam, then at least the sender can potentially do something about it, rather than believing that the message went through while in reality it is sitting in some spam folder that I look at once a month or not at all.

Microblogging with ActivityPub

Posted Nov 30, 2022 12:21 UTC (Wed) by rqosa (subscriber, #24136) [Link] (2 responses)

> Having to check a spam filter defies the purpose of using one in the first place.

I disagree with that; I consider the user-experience of having a spam folder that you check occasionally — but less frequently than your main inbox — to be better than that of having no spam filter at all, and also better than that of blindly trusting the filtering software (or whatever kind of filtering-system that the server operator uses… especially the comment-filtering systems in non-email, publically-visible discussion systems run by huge for-profit companies such as Twitter, YouTube, Reddit, etc., i.e. the kind "that employ a large body of low-wage overseas labor" to manually or semi-manually filter stuff that tau mentioned above) to not produce any false-positives.

(Revisiting what I said earlier about client-side email filtering: if you run SpamAssassin on the client-side, you can take any false-positive messages that end up in your spam folder and pass them to the sa-learn utility as "ham" messages, and also take any false-negative messages that end up in your main inbox and pass them to the sa-learn utility as "spam" messages, in order to train its filtering model to produce fewer incorrect results in the future.)

Microblogging with ActivityPub

Posted Nov 30, 2022 12:39 UTC (Wed) by rqosa (subscriber, #24136) [Link] (1 responses)

(Addendum: that kind of abillity for an end-user to train their own personalized spam-filtering model is a feature that I'd like to see implemented by ActivityPub/Mastodon clients in the future.)

Microblogging with ActivityPub

Posted Nov 30, 2022 14:16 UTC (Wed) by Wol (subscriber, #4433) [Link]

Something I'd also like to see is Thunderbird rules to allow filtering on the BODY of the message. Dunno what's happened, but somehow my email has got associated with someone called "David", so every now and then I get a splurge of spam starting "Dear David". Anything addressed to David can go *straight* in the bin, but I can't configure TB to search on that !!!

Cheers,
Wol

Microblogging with ActivityPub

Posted Dec 1, 2022 16:20 UTC (Thu) by flussence (guest, #85566) [Link]

The difference here is that, unlike email, there's no cultural pressure to pretend to accept everything from strangers in the first place, and the protocol isn't a substrate for important transactions. ActivityPub filtering can afford to be much more aggressive.

Microblogging with ActivityPub

Posted Dec 1, 2022 12:55 UTC (Thu) by taladar (subscriber, #68407) [Link]

If Email spam filtering was a solved problem we wouldn't see so much snake-oil like IP reputation services and grey listing in that area.

bufferbloat in the fediverse

Posted Nov 30, 2022 3:11 UTC (Wed) by mtaht (guest, #11087) [Link] (16 responses)

I have been hoping there was a reasonably active
mastodon server maintainer out there that could take a few packet captures of how it stresses the network and let me tear 'em apart. Just the headers please. (tcpdump -i theinterface -s 128 -w somefile.cap) for an hour or more.

I have been seeing multiple complaints as to what can happen (most recently jwz) and I figure that homeservers, especially, are creating a lot of bufferbloat and network jitter for the end users... there doesn't seem to be a lot of awareness of basic techniques (disabling ecn + enabling bbr), applying sqm, tuning up TCP_NOTSENT_LOWAT, out there.

bufferbloat in the fediverse

Posted Nov 30, 2022 10:08 UTC (Wed) by smurf (subscriber, #17840) [Link]

Those "basic techniques" aren't *that* basic, otherwise they wouldn't be something everybody must painfully re-discover for themselves.

bufferbloat in the fediverse

Posted Nov 30, 2022 12:06 UTC (Wed) by crschmidt (guest, #162445) [Link] (11 responses)

The number of Mastodon instances that have enough traffic to worry about network-level congestion is relatively small. Anything outside the top 100 largest instances likely isn't going to be interesting from that perspective. (For example, jwz's post is not a problem for the Mastodon servers, which are each only fetching one copy of a small amount of data, at all; the problem is that there are 4300 different servers that all make that same request to the _origin_, ie, jwz's webserver. Link fetching in the Mastodon/Plemora part of the Fediverse is effectively a user-controllable botnet. Oops.)

Most of Mastodon's performance problems for medium-size instances over the past month stemmed from the relatively naive configuration of its backend worker queues (where, by default, all tasks are running in the same queue, with no prioritization of handling inbound / outbound posts over lower priority background tasks).

There are exceptions to this: Large instances like Mastodon.social (881k users) and even small-ish but highly active servers like Hachyderm.io (30k users) have sufficient utilization to have real performance constraints ... but most of that isn't in serving the _users_ (which is relatively low cost), but rather in processing the overall flow of information from the broader Fediverse (especially things like caching media) into the server, as far as I have seen.

Put another way: network jitter really doesn't matter when the backend API that I'm fetching from is going to take 10-15 seconds to return 60KB of data; that amount of content could be delivered over a 56k modem in that time window.

bufferbloat in the fediverse

Posted Dec 1, 2022 3:15 UTC (Thu) by mtaht (guest, #11087) [Link] (10 responses)

I really disagree with that. Any sizable upload or download will induce noticable load on the network once that file begins to transfer. Moderating the impact of that with (as one example, disabling ecn and enabling bbr) is three lines in sysctl.conf or more simply, by
propagating a file into /etc/sysctl.d/10-lowerbloat.conf

net.core.default_qdisc=fq_codel
net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_ecn=0

why is it so hard to get folk to do that much?
Sure, measuring what happens via tcpdump, etc takes a bit of time, but nothing compared to how immediately I would hope these lines would help the fediverse.

bufferbloat in the fediverse

Posted Dec 1, 2022 9:30 UTC (Thu) by mbunkus (subscriber, #87248) [Link] (4 responses)

It's hard because it's really not common knowledge. No one talks about it, unlike fq_codel, which was very much a topic picked up by a lot of blog posts, by media outlets, and most importantly, it was made the default by the either the kernel itself or by distros.

Defaults matter so much.

For example, I've never heard of the recommendation of disabling ECN in favor of enabling BBR. I tried a quick Google search, and there wasn't really a lot of good information out there that proposed exactly this together with a easy-to-digest rationale. Sure, there are somewhat scientific explanations of how BBR works, going into details about packet loss & stuff.

Nor are there articles discussing in which type of situation these settings matter (only on machines connected to the internet? all machines? why?).

I'd love to read more about this. Do you have any pointers?

Bufferbloat is a rather complex topic. I'm not surprised that most sysadmins don't know much if anything about it, even less how to fix it properly.

In short: it really, really isn't obvious that this is something that Jane Sysadmin should do.

bufferbloat in the fediverse

Posted Dec 1, 2022 14:57 UTC (Thu) by adobriyan (guest, #30858) [Link]

There is original BBR commit message with links to ACM paper:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/...

bufferbloat in the fediverse

Posted Dec 1, 2022 16:19 UTC (Thu) by mtaht (guest, #11087) [Link] (2 responses)

The BBR folk at google use ECN differently than RFC3168 and disable negotation of it entirely... while apple clients tend to request it, and it's enabled by default across the rest of the internet. There was a patch rejected long ago that did the right thing here, and other patches to just make BBR obey RFC3168, also rejected... I am seeing a lot of BBR uptake (say, 11% of websites), without also disabling ECN negotiation, which means that fq_codel, cake, etc go around marking packets madly instead of dropping them, to no observable effect when BBR is in play.

Mess. My rightest answer would be to make BBRv1 do RFC3168-style ecn, perhaps not dropping the rate by half as per that spec, but by initiating a probe phase. Others differ.

As for recommendations as to good defaults, a lot of the non-controversial fixes have landed in the kernel and become defaults, tsq, bql, pacing, fq_codel, etc.

and, BBR has been shown to be a good step forward for many applications, and I do think it's probably a better transport than cubic for the kinds of long running, periodically bursty, autonomous applications like those in the fediverse, and especially over lte. It's also been shown to be problematic in multiple respects, but BBRv2 is hung up in testing, partially hinged on the non-backward compatible treatment of ECN in it.

It would be nice if more people were aware of these issues, taking packet captures, and worrying about the impacts on the network.

I'm sorry that the amount of "publicity" and discussions these problems have got is not easily visible on today's internet. Perhaps talking about it here will help, and a few more beleaguered sysadmins and users lean in. Google does not index mailing lists like ietf's tsvwg or bufferbloat.net's bloat list all that well, anymore, with over a decades worth of discussion on each.

bufferbloat in the fediverse

Posted Dec 1, 2022 18:09 UTC (Thu) by mtaht (guest, #11087) [Link]

The BBR mailing list.

I am not sure if this BBRv2 status is up to date

bufferbloat in the fediverse

Posted Dec 2, 2022 18:15 UTC (Fri) by intelfx (subscriber, #130118) [Link]

This should be in all the wikis and in form of clearly digestible HOWTOs with rationales for administrators of all skill levels.

With all the immense respect owed to the great work you're doing to tackle the bufferbloat problem — it's a really, really, really obscure problem and, the solutions are not really discoverable unless you make it your explicit objective.

bufferbloat in the fediverse

Posted Dec 1, 2022 10:04 UTC (Thu) by farnz (subscriber, #17727) [Link] (4 responses)

It's hard because it's a change from defaults, and changes are always scary. If those changes are always a win, then they need to be the defaults, so that people use them automatically - I note that with defaults on Fedora Linux KDE spin, I get one of your three preferred changes by default:

$ sysctl net.core.default_qdisc net.ipv4.tcp_congestion_control net.ipv4.tcp_ecn
net.core.default_qdisc = fq_codel
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_ecn = 2

The trouble with asking people to make changes from the defaults is that it's never quite clear why something doesn't work as expected, unless you fully understand the changes. So why make a change when not changing works just fine from your perspective?

bufferbloat in the fediverse

Posted Dec 1, 2022 16:51 UTC (Thu) by mtaht (guest, #11087) [Link] (3 responses)

Well, there was an effort to enable cubic + ecn (vs bbr - ecn) also.

More recently I'm happy to have heard that "backpressure" for microservices, within a machine, has arrived in the kernel as of 6.0, which renders part of the issues debated above moot. ( cilium is bragging about this. ) On the other hand, all the developers of microservices I've met so far, seem to think backpressure exists for other things outside the box, or other protocols outside of tcp, and it *doesn't.

I've tried to engage with the cilium folk a couple times now.

bufferbloat in the fediverse

Posted Dec 2, 2022 14:16 UTC (Fri) by farnz (subscriber, #17727) [Link] (2 responses)

I think, personally, that a core part of the issue is that most of the fixes for bufferbloat involve making metrics worse for a few edge cases (packet loss counters when at saturation, for example), in return for a big improvement that's not hugely visible in metrics for the vast majority of traffic.

And so, where you're asking people to change from the defaults, you're asking them to regress one or two metrics they've "always" paid attention to, without improving other metrics they pay attention to. This is obviously a bad things to do - why make the numbers worse? - and thus you struggle unless your changes are the defaults (at which point, when they look into their tweaks, they find they make the metric better at the expense of something they actually care about).

Basically the traditional confusion between a metric and an outcome :-(

bufferbloat in the fediverse

Posted Dec 2, 2022 14:53 UTC (Fri) by gioele (subscriber, #61675) [Link] (1 responses)

> I think, personally, that a core part of the issue is that most of the fixes for bufferbloat involve making metrics worse for a few edge cases (packet loss counters when at saturation, for example), in return for a big improvement that's not hugely visible in metrics for the vast majority of traffic.

Isn't that addressed by publicizing another contrasting metric?

"Yes packet loss counters went 2.7% up, but 95%-ile latency is down by 84.3%!"

bufferbloat in the fediverse

Posted Dec 2, 2022 15:37 UTC (Fri) by farnz (subscriber, #17727) [Link]

My experience of debloating a network that's not a bottleneck is that that's not actually what you get; packet loss goes up 4%, but 99-%ile throughput and latency are unchanged. And it's hard to explain that this is actually a win - that what I've actually done is get you to a point where instead of needing expensive network upgrades when traffic doubles, you can hold off until traffic triples - because that's something in the far future.

Fortunately, the network I debloated was my home network, so while I can see that no metrics have improved significantly, I'm happy that I'll be able to run on the existing hardware even as demand increases.

bufferbloat in the fediverse

Posted Dec 1, 2022 16:02 UTC (Thu) by flussence (guest, #85566) [Link] (2 responses)

In my experience (not in any way an active server, just a bedroom one) it's like IPFS traffic but in unpredictable, coordinated bursts. “Going viral” has a perverse disincentive because you're liable to get slammed by hundreds if not thousands of remote https connections over the course of a minute in response to any type of remote interaction; almost all of this server software uses grossly misconfigured HTTP client libraries - HTTP/1.1, no compression, no persistent connections. If you've posted a twitter-style long thread that receives attention it's even worse, there's no request batching at any layer of the stack. Mastodon in particular likes to tack on a dozen or two “just-in-case” metadata lookups on users it hasn't seen before (and then proceeds to discard the data only to do it again next time).

My network is as close to an ideal software setup as I can get — to cope with a negligent rural monopoly ISP — but this stuff still manages to knock it down at times. Cake works miracles with other types of traffic but it can't fix lazily-written garbage.

bufferbloat in the fediverse

Posted Dec 1, 2022 17:57 UTC (Thu) by mtaht (guest, #11087) [Link] (1 responses)

Dear flussence:

As a long term cake fan, perhaps your ISP would be interested in a beta of libreqos.io?

It's always been a saner idea to have the ISP shape the down, and the CPE shape the up, but until recently it was very difficult to shape the down at ISP rates, but with XDP, ebpf and a few other tricks, it's now pushing 11Gbit/sec, 10k mixed fiber/FWA users, at 24% of 16 cores, aiming for 40Gbit/sec and 40k users in the next release I'm deleriously happy with where it's been going, you can find the devs at #libreqos:matrix.org.

I would hope that running cake at the isp would mediate far further depredations of the network as you describe, with ipfs and the fediverse as inbound shaping at the CPE simply can't cope, and the per host/per flow fq in cake would really moderate the impact, or so I hope.

I'd love to have a semi-repeatable test case for some of the behaviors everyone is describing, and to see what libreqos can do to help.

bufferbloat in the fediverse

Posted Dec 8, 2022 0:44 UTC (Thu) by flussence (guest, #85566) [Link]

Haha, my ISP is the type from hell that's never heard of IPv6, let alone QoS. There's no way to get hold of a technical contact as an end user, I've looked — because they don't have any. It's all outsourced to Cisco consultants, even actual line faults take weeks/months to get resolved. You could say their chain of command has a severe bufferbloat problem :)

I have a 16Mbit/830kbit DSL connection which hasn't seen a physical upgrade since the late 2000s. They've *started* building out fibre here where they charge obscene rates for less than a 3× increase *each way* (asymmetric FTTP! what a load of BS!), but in any case it doesn't seem like they have many choke points except the last hop they scalp their customers for. My latency's been consistently low even with this fediverse stuff. At this point I just wish they'd stop starving us of bandwidth…

Microblogging with ActivityPub

Posted Nov 30, 2022 3:58 UTC (Wed) by re:fi.64 (subscriber, #132628) [Link] (1 responses)

It's worth noting that Mastodon doesn't just Redis *just* for caching: Redis is also used by Sidekiq as the job queue. This has some important implications for hosting, e.g. the Redis instance used by Sidekiq should have persistence enabled.

Microblogging with ActivityPub

Posted Nov 30, 2022 19:31 UTC (Wed) by riking (subscriber, #95706) [Link]

And unlike Discourse, where a Redis FLUSHALL is a minor data loss incident (undelivered emails), flushing a Mastodon's Redis database is a major data loss incident that you should be declaring to your users.

Microblogging with ActivityPub

Posted Nov 30, 2022 10:09 UTC (Wed) by rqosa (subscriber, #24136) [Link] (2 responses)

Another client for Android devices (which supports both the Mastodon-specific API and the plain ActivityPub client API) is AndStatus. One notable specific feature it supports is the "Conversation view", which displays a tree-view of a discussion thread, similar to "threaded mode" in many email and Usenet clients (as well as comment sections of some blogs/websites, such as LWN itself).

Microblogging with ActivityPub

Posted Nov 30, 2022 10:39 UTC (Wed) by rqosa (subscriber, #24136) [Link]

(Side note unrelated to ActivityPub: vBulletin used to have a "threaded mode" feature like that, but they removed it several years ago, which was a disappointment to users like me who were in the habit of using it.)

Microblogging with ActivityPub

Posted Jan 12, 2023 9:32 UTC (Thu) by rqosa (subscriber, #24136) [Link]

There's also Fedilab, which has a tree-view feature (shown in this screenshot) similar to the one in AndStatus.

Microblogging with ActivityPub

Posted Nov 30, 2022 13:00 UTC (Wed) by Kamiccolo (subscriber, #95159) [Link] (1 responses)

Also, few years ago, the closure of G+ made some spike of traction towards federated networks, like Mastodon. One of the differences being that back then there was more pull from Diaspora (while having G+ like circles, etc). I wonder, where's Diaspora today?

Microblogging with ActivityPub (and Diaspora)

Posted Nov 30, 2022 17:06 UTC (Wed) by Herve5 (guest, #115399) [Link]

I'm on Diaspora, there are various Europeans there, but this is probably rather considered 'local' I presume... Still cool and no spam though.
I have noticed some Mastodon-created posts seem to be automatically copied to D* from time to time, but this is quite scarce and seems not conveying comments back efficiently...

Microblogging with ActivityPub

Posted Nov 30, 2022 18:13 UTC (Wed) by jond (subscriber, #37669) [Link]

Debian have a Pleroma instance at https://pleroma.debian.social/

Slightly jarringly its front page throws an error for public, not-logged-in users.

Existing blogs can also be ActivityPub enabled

Posted Dec 2, 2022 14:33 UTC (Fri) by jejb (subscriber, #6654) [Link]

The point being that in order to join the fediverse you don't have to run a mastadon like server or join a mastodon instance, you can simply activitypub enable your existing communication platform. If you're blogging on wordpress you can do this using the ActivityPub plugin. I will note that the unmodified plugin doesn't allow replies via activitypub, but I've got fixes for that:

https://github.com/jejb/wordpress-activitypub/tree/add-re...

If you want to see it in use, it's on my blog site: https://blog.hansenpartnership.com and you can see what it looks like in my test mastodon account which boosts my blog posts into the fediverse: https://mastodon.online/@jejb

Microblogging with ActivityPub

Posted Dec 2, 2022 18:48 UTC (Fri) by ejr (subscriber, #51652) [Link]

GNU social's origins are in identi.ca / StatusNet. The move to ActivityPub in pump.io somewhat caused the fork. Not an unfriendly one, just different directions.

Microblogging with ActivityPub

Posted Dec 6, 2022 5:06 UTC (Tue) by ssmith32 (subscriber, #72404) [Link]

I'm thoroughly stumped every time I read about Mastadon not having some feature X across instances.

I use the stock Mastadon app, and I have two views:

(1) search
(2) home

Both work fine across Mastadon instances. My home screen shows me a timeline of multiple people I follow across multiple instances. And their boosts of people on yet different instances.

Search defaults to showing me what's happening "in my corner of Mastadon", which by and large shows toots/posts from other instances. Search works fine across all multiple instances. I found Takei by typing "Takei" .. and he is on a different instance.

Yet numerous other reports talk about how you can't search across instances, and this article states I should not have "access to the federated timeline where posts from other servers may be viewed. " in the vanilla Mastadon app. Yet I'm staring at something that really seems to match that description. Unless you want it to include all toots across all servers.. including from folks you don't follow? Which seems.. overwhelming.

What am I missing?