LWN: Comments on "Microblogging with ActivityPub"

Microblogging with ActivityPub

rqosa — Thu, 12 Jan 2023 09:32:16 +0000

There's also Fedilab, which has a tree-view feature (shown in this screenshot) similar to the one in AndStatus.

bufferbloat in the fediverse

flussence — Thu, 08 Dec 2022 00:44:40 +0000

Haha, my ISP is the type from hell that's never heard of IPv6, let alone QoS. There's no way to get hold of a technical contact as an end user, I've looked — because they don't have any. It's all outsourced to Cisco consultants, even actual line faults take weeks/months to get resolved. You could say their chain of command has a severe bufferbloat problem :)

I have a 16Mbit/830kbit DSL connection which hasn't seen a physical upgrade since the late 2000s. They've *started* building out fibre here where they charge obscene rates for less than a 3× increase *each way* (asymmetric FTTP! what a load of BS!), but in any case it doesn't seem like they have many choke points except the last hop they scalp their customers for. My latency's been consistently low even with this fediverse stuff. At this point I just wish they'd stop starving us of bandwidth…

Microblogging with ActivityPub

ssmith32 — Tue, 06 Dec 2022 05:06:37 +0000

I'm thoroughly stumped every time I read about Mastadon not having some feature X across instances.

I use the stock Mastadon app, and I have two views:

(1) search
(2) home

Both work fine across Mastadon instances. My home screen shows me a timeline of multiple people I follow across multiple instances. And their boosts of people on yet different instances.

Search defaults to showing me what's happening "in my corner of Mastadon", which by and large shows toots/posts from other instances. Search works fine across all multiple instances. I found Takei by typing "Takei" .. and he is on a different instance.

Yet numerous other reports talk about how you can't search across instances, and this article states I should not have "access to the federated timeline where posts from other servers may be viewed. " in the vanilla Mastadon app. Yet I'm staring at something that really seems to match that description. Unless you want it to include all toots across all servers.. including from folks you don't follow? Which seems.. overwhelming.

What am I missing?

Microblogging with ActivityPub

ejr — Fri, 02 Dec 2022 18:48:27 +0000

GNU social's origins are in identi.ca / StatusNet. The move to ActivityPub in pump.io somewhat caused the fork. Not an unfriendly one, just different directions.

bufferbloat in the fediverse

intelfx — Fri, 02 Dec 2022 18:15:21 +0000

This should be in all the wikis and in form of clearly digestible HOWTOs with rationales for administrators of all skill levels.

With all the immense respect owed to the great work you're doing to tackle the bufferbloat problem — it's a really, really, really obscure problem and, the solutions are not really discoverable unless you make it your explicit objective.

bufferbloat in the fediverse

farnz — Fri, 02 Dec 2022 15:37:21 +0000

My experience of debloating a network that's not a bottleneck is that that's not actually what you get; packet loss goes up 4%, but 99-%ile throughput and latency are unchanged. And it's hard to explain that this is actually a win - that what I've actually done is get you to a point where instead of needing expensive network upgrades when traffic doubles, you can hold off until traffic triples - because that's something in the far future.

Fortunately, the network I debloated was my home network, so while I can see that no metrics have improved significantly, I'm happy that I'll be able to run on the existing hardware even as demand increases.

bufferbloat in the fediverse

gioele — Fri, 02 Dec 2022 14:53:46 +0000

> I think, personally, that a core part of the issue is that most of the fixes for bufferbloat involve making metrics worse for a few edge cases (packet loss counters when at saturation, for example), in return for a big improvement that's not hugely visible in metrics for the vast majority of traffic.

Isn't that addressed by publicizing another contrasting metric?

"Yes packet loss counters went 2.7% up, but 95%-ile latency is down by 84.3%!"

Existing blogs can also be ActivityPub enabled

jejb — Fri, 02 Dec 2022 14:33:00 +0000

The point being that in order to join the fediverse you don't have to run a mastadon like server or join a mastodon instance, you can simply activitypub enable your existing communication platform. If you're blogging on wordpress you can do this using the ActivityPub plugin. I will note that the unmodified plugin doesn't allow replies via activitypub, but I've got fixes for that:

https://github.com/jejb/wordpress-activitypub/tree/add-re...

If you want to see it in use, it's on my blog site: https://blog.hansenpartnership.com and you can see what it looks like in my test mastodon account which boosts my blog posts into the fediverse: https://mastodon.online/@jejb

bufferbloat in the fediverse

farnz — Fri, 02 Dec 2022 14:16:23 +0000

I think, personally, that a core part of the issue is that most of the fixes for bufferbloat involve making metrics worse for a few edge cases (packet loss counters when at saturation, for example), in return for a big improvement that's not hugely visible in metrics for the vast majority of traffic.

And so, where you're asking people to change from the defaults, you're asking them to regress one or two metrics they've "always" paid attention to, without improving other metrics they pay attention to. This is obviously a bad things to do - why make the numbers worse? - and thus you struggle unless your changes are the defaults (at which point, when they look into their tweaks, they find they make the metric better at the expense of something they actually care about).

Basically the traditional confusion between a metric and an outcome :-(

bufferbloat in the fediverse

mtaht — Thu, 01 Dec 2022 18:09:28 +0000

The BBR mailing list.

I am not sure if this BBRv2 status is up to date

bufferbloat in the fediverse

mtaht — Thu, 01 Dec 2022 17:57:08 +0000

Dear flussence:

As a long term cake fan, perhaps your ISP would be interested in a beta of libreqos.io?

It's always been a saner idea to have the ISP shape the down, and the CPE shape the up, but until recently it was very difficult to shape the down at ISP rates, but with XDP, ebpf and a few other tricks, it's now pushing 11Gbit/sec, 10k mixed fiber/FWA users, at 24% of 16 cores, aiming for 40Gbit/sec and 40k users in the next release I'm deleriously happy with where it's been going, you can find the devs at #libreqos:matrix.org.

I would hope that running cake at the isp would mediate far further depredations of the network as you describe, with ipfs and the fediverse as inbound shaping at the CPE simply can't cope, and the per host/per flow fq in cake would really moderate the impact, or so I hope.

I'd love to have a semi-repeatable test case for some of the behaviors everyone is describing, and to see what libreqos can do to help.

bufferbloat in the fediverse

mtaht — Thu, 01 Dec 2022 16:51:08 +0000

Well, there was an effort to enable cubic + ecn (vs bbr - ecn) also.

More recently I'm happy to have heard that "backpressure" for microservices, within a machine, has arrived in the kernel as of 6.0, which renders part of the issues debated above moot. ( cilium is bragging about this. ) On the other hand, all the developers of microservices I've met so far, seem to think backpressure exists for other things outside the box, or other protocols outside of tcp, and it *doesn't.

I've tried to engage with the cilium folk a couple times now.

Microblogging with ActivityPub

flussence — Thu, 01 Dec 2022 16:20:55 +0000

The difference here is that, unlike email, there's no cultural pressure to pretend to accept everything from strangers in the first place, and the protocol isn't a substrate for important transactions. ActivityPub filtering can afford to be much more aggressive.

bufferbloat in the fediverse

mtaht — Thu, 01 Dec 2022 16:19:16 +0000

The BBR folk at google use ECN differently than RFC3168 and disable negotation of it entirely... while apple clients tend to request it, and it's enabled by default across the rest of the internet. There was a patch rejected long ago that did the right thing here, and other patches to just make BBR obey RFC3168, also rejected... I am seeing a lot of BBR uptake (say, 11% of websites), without also disabling ECN negotiation, which means that fq_codel, cake, etc go around marking packets madly instead of dropping them, to no observable effect when BBR is in play.

Mess. My rightest answer would be to make BBRv1 do RFC3168-style ecn, perhaps not dropping the rate by half as per that spec, but by initiating a probe phase. Others differ.

As for recommendations as to good defaults, a lot of the non-controversial fixes have landed in the kernel and become defaults, tsq, bql, pacing, fq_codel, etc.

and, BBR has been shown to be a good step forward for many applications, and I do think it's probably a better transport than cubic for the kinds of long running, periodically bursty, autonomous applications like those in the fediverse, and especially over lte. It's also been shown to be problematic in multiple respects, but BBRv2 is hung up in testing, partially hinged on the non-backward compatible treatment of ECN in it.

It would be nice if more people were aware of these issues, taking packet captures, and worrying about the impacts on the network.

I'm sorry that the amount of "publicity" and discussions these problems have got is not easily visible on today's internet. Perhaps talking about it here will help, and a few more beleaguered sysadmins and users lean in. Google does not index mailing lists like ietf's tsvwg or bufferbloat.net's bloat list all that well, anymore, with over a decades worth of discussion on each.

Rebased (formerly Soapbox)

Gaelan — Thu, 01 Dec 2022 16:06:26 +0000

Note that Soapbox, even more so than Pleroma, has a reputation for being used almost exclusively by the far right.

bufferbloat in the fediverse

flussence — Thu, 01 Dec 2022 16:02:21 +0000

In my experience (not in any way an active server, just a bedroom one) it's like IPFS traffic but in unpredictable, coordinated bursts. “Going viral” has a perverse disincentive because you're liable to get slammed by hundreds if not thousands of remote https connections over the course of a minute in response to any type of remote interaction; almost all of this server software uses grossly misconfigured HTTP client libraries - HTTP/1.1, no compression, no persistent connections. If you've posted a twitter-style long thread that receives attention it's even worse, there's no request batching at any layer of the stack. Mastodon in particular likes to tack on a dozen or two “just-in-case” metadata lookups on users it hasn't seen before (and then proceeds to discard the data only to do it again next time).

My network is as close to an ideal software setup as I can get — to cope with a negligent rural monopoly ISP — but this stuff still manages to knock it down at times. Cake works miracles with other types of traffic but it can't fix lazily-written garbage.

bufferbloat in the fediverse

adobriyan — Thu, 01 Dec 2022 14:57:12 +0000

There is original BBR commit message with links to ACM paper:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/...

Microblogging with ActivityPub

taladar — Thu, 01 Dec 2022 12:55:07 +0000

If Email spam filtering was a solved problem we wouldn't see so much snake-oil like IP reputation services and grey listing in that area.

bufferbloat in the fediverse

farnz — Thu, 01 Dec 2022 10:04:34 +0000

It's hard because it's a change from defaults, and changes are always scary. If those changes are always a win, then they need to be the defaults, so that people use them automatically - I note that with defaults on Fedora Linux KDE spin, I get one of your three preferred changes by default:

$ sysctl net.core.default_qdisc net.ipv4.tcp_congestion_control net.ipv4.tcp_ecn
net.core.default_qdisc = fq_codel
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_ecn = 2

The trouble with asking people to make changes from the defaults is that it's never quite clear why something doesn't work as expected, unless you fully understand the changes. So why make a change when not changing works just fine from your perspective?

bufferbloat in the fediverse

mbunkus — Thu, 01 Dec 2022 09:30:41 +0000

It's hard because it's really not common knowledge. No one talks about it, unlike fq_codel, which was very much a topic picked up by a lot of blog posts, by media outlets, and most importantly, it was made the default by the either the kernel itself or by distros.

Defaults matter so much.

For example, I've never heard of the recommendation of disabling ECN in favor of enabling BBR. I tried a quick Google search, and there wasn't really a lot of good information out there that proposed exactly this together with a easy-to-digest rationale. Sure, there are somewhat scientific explanations of how BBR works, going into details about packet loss & stuff.

Nor are there articles discussing in which type of situation these settings matter (only on machines connected to the internet? all machines? why?).

I'd love to read more about this. Do you have any pointers?

Bufferbloat is a rather complex topic. I'm not surprised that most sysadmins don't know much if anything about it, even less how to fix it properly.

In short: it really, really isn't obvious that this is something that Jane Sysadmin should do.

bufferbloat in the fediverse

mtaht — Thu, 01 Dec 2022 03:15:38 +0000

I really disagree with that. Any sizable upload or download will induce noticable load on the network once that file begins to transfer. Moderating the impact of that with (as one example, disabling ecn and enabling bbr) is three lines in sysctl.conf or more simply, by
propagating a file into /etc/sysctl.d/10-lowerbloat.conf

net.core.default_qdisc=fq_codel
net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_ecn=0

why is it so hard to get folk to do that much?
Sure, measuring what happens via tcpdump, etc takes a bit of time, but nothing compared to how immediately I would hope these lines would help the fediverse.

Microblogging with ActivityPub

riking — Wed, 30 Nov 2022 19:31:26 +0000

And unlike Discourse, where a Redis FLUSHALL is a minor data loss incident (undelivered emails), flushing a Mastodon's Redis database is a major data loss incident that you should be declaring to your users.

Microblogging with ActivityPub

jond — Wed, 30 Nov 2022 18:13:33 +0000

Debian have a Pleroma instance at https://pleroma.debian.social/

Slightly jarringly its front page throws an error for public, not-logged-in users.

Microblogging with ActivityPub

jond — Wed, 30 Nov 2022 18:08:29 +0000

I was on the other, older one, kernel.social, which got switched off a year or more ago, and I had to start over somewhere else.

Microblogging with ActivityPub (and Diaspora)

Herve5 — Wed, 30 Nov 2022 17:06:35 +0000

I'm on Diaspora, there are various Europeans there, but this is probably rather considered 'local' I presume... Still cool and no spam though.
I have noticed some Mastodon-created posts seem to be automatically copied to D* from time to time, but this is quite scarce and seems not conveying comments back efficiently...

Microblogging with ActivityPub

Wol — Wed, 30 Nov 2022 14:16:03 +0000

Something I'd also like to see is Thunderbird rules to allow filtering on the BODY of the message. Dunno what's happened, but somehow my email has got associated with someone called "David", so every now and then I get a splurge of spam starting "Dear David". Anything addressed to David can go *straight* in the bin, but I can't configure TB to search on that !!!

Cheers,
Wol

Microblogging with ActivityPub

Kamiccolo — Wed, 30 Nov 2022 13:00:39 +0000

Also, few years ago, the closure of G+ made some spike of traction towards federated networks, like Mastodon. One of the differences being that back then there was more pull from Diaspora (while having G+ like circles, etc). I wonder, where's Diaspora today?

Microblogging with ActivityPub

rqosa — Wed, 30 Nov 2022 12:39:09 +0000

(Addendum: that kind of abillity for an end-user to train their own personalized spam-filtering model is a feature that I'd like to see implemented by ActivityPub/Mastodon clients in the future.)

Microblogging with ActivityPub

rqosa — Wed, 30 Nov 2022 12:21:53 +0000

> Having to check a spam filter defies the purpose of using one in the first place.

I disagree with that; I consider the user-experience of having a spam folder that you check occasionally — but less frequently than your main inbox — to be better than that of having no spam filter at all, and also better than that of blindly trusting the filtering software (or whatever kind of filtering-system that the server operator uses… especially the comment-filtering systems in non-email, publically-visible discussion systems run by huge for-profit companies such as Twitter, YouTube, Reddit, etc., i.e. the kind "that employ a large body of low-wage overseas labor" to manually or semi-manually filter stuff that tau mentioned above) to not produce any false-positives.

(Revisiting what I said earlier about client-side email filtering: if you run SpamAssassin on the client-side, you can take any false-positive messages that end up in your spam folder and pass them to the sa-learn utility as "ham" messages, and also take any false-negative messages that end up in your main inbox and pass them to the sa-learn utility as "spam" messages, in order to train its filtering model to produce fewer incorrect results in the future.)

bufferbloat in the fediverse

crschmidt — Wed, 30 Nov 2022 12:06:26 +0000

The number of Mastodon instances that have enough traffic to worry about network-level congestion is relatively small. Anything outside the top 100 largest instances likely isn't going to be interesting from that perspective. (For example, jwz's post is not a problem for the Mastodon servers, which are each only fetching one copy of a small amount of data, at all; the problem is that there are 4300 different servers that all make that same request to the _origin_, ie, jwz's webserver. Link fetching in the Mastodon/Plemora part of the Fediverse is effectively a user-controllable botnet. Oops.)

Most of Mastodon's performance problems for medium-size instances over the past month stemmed from the relatively naive configuration of its backend worker queues (where, by default, all tasks are running in the same queue, with no prioritization of handling inbound / outbound posts over lower priority background tasks).

There are exceptions to this: Large instances like Mastodon.social (881k users) and even small-ish but highly active servers like Hachyderm.io (30k users) have sufficient utilization to have real performance constraints ... but most of that isn't in serving the _users_ (which is relatively low cost), but rather in processing the overall flow of information from the broader Fediverse (especially things like caching media) into the server, as far as I have seen.

Put another way: network jitter really doesn't matter when the backend API that I'm fetching from is going to take 10-15 seconds to return 60KB of data; that amount of content could be delivered over a 56k modem in that time window.

Microblogging with ActivityPub

anselm — Wed, 30 Nov 2022 11:08:30 +0000

The problem with that approach is that - because spam filtering is so good - users don't check their spam filters.

Having to check a spam filter defies the purpose of using one in the first place. The whole point of having a spam filter is not having to look at the spam.

I don't have a spam folder. I'm running a pre-queue spam checker and stuff that looks like spam is refused by my server while it is still in the process of being submitted, with an SMTP error code. If a legitimate message is misidentified as spam, then at least the sender can potentially do something about it, rather than believing that the message went through while in reality it is sitting in some spam folder that I look at once a month or not at all.

Microblogging with ActivityPub

rqosa — Wed, 30 Nov 2022 10:39:12 +0000

(Side note unrelated to ActivityPub: vBulletin used to have a "threaded mode" feature like that, but they removed it several years ago, which was a disappointment to users like me who were in the habit of using it.)

Microblogging with ActivityPub

rqosa — Wed, 30 Nov 2022 10:09:15 +0000

Another client for Android devices (which supports both the Mastodon-specific API and the plain ActivityPub client API) is AndStatus. One notable specific feature it supports is the "Conversation view", which displays a tree-view of a discussion thread, similar to "threaded mode" in many email and Usenet clients (as well as comment sections of some blogs/websites, such as LWN itself).

bufferbloat in the fediverse

smurf — Wed, 30 Nov 2022 10:08:06 +0000

Those "basic techniques" aren't *that* basic, otherwise they wouldn't be something everybody must painfully re-discover for themselves.

Microblogging with ActivityPub

Wol — Wed, 30 Nov 2022 09:43:31 +0000

The problem with that approach is that - because spam filtering is so good - users don't check their spam filters.

The combination of Thunderbird filters and server-side spam marking works so well for me that when stuff does get misclassified, it can VERY easily get mass-deleted by mistake ... usually stuff I actively want :-(

Cheers,
Wol

Microblogging with ActivityPub

rqosa — Wed, 30 Nov 2022 09:17:43 +0000

> it will need to come up with a solution to the problems that plague "federation[s] somewhat like email".

The problem of email spam-filtering is pretty much a solved problem by now. My personal email account has been using SpamAssassin for a long time, and while not perfect, it seems to be "good enough" at least. (And one good thing about it is that whenever a false-positive occurs, the end user can still see the email message that was flagged as spam, as long as they've configured their MUA's filtering rules to direct spam into a "spam folder" and then read that folder occasionally.)

So it seems like a similar appoach (using Bayesian filtering) could eventually be used for ActivityPub as well. Furthermore, (same as with email) it might also be possible for ActivityPub spam-filtering to be done on the client-side, for end users who don't want their server operators to be in full control of the spam-filtering.

Microblogging with ActivityPub

re:fi.64 — Wed, 30 Nov 2022 03:58:12 +0000

It's worth noting that Mastodon doesn't just Redis *just* for caching: Redis is also used by Sidekiq as the job queue. This has some important implications for hosting, e.g. the Redis instance used by Sidekiq should have persistence enabled.

bufferbloat in the fediverse

mtaht — Wed, 30 Nov 2022 03:11:02 +0000

I have been hoping there was a reasonably active
mastodon server maintainer out there that could take a few packet captures of how it stresses the network and let me tear 'em apart. Just the headers please. (tcpdump -i theinterface -s 128 -w somefile.cap) for an hour or more.

I have been seeing multiple complaints as to what can happen (most recently jwz) and I figure that homeservers, especially, are creating a lot of bufferbloat and network jitter for the end users... there doesn't seem to be a lot of awareness of basic techniques (disabling ecn + enabling bbr), applying sqm, tuning up TCP_NOTSENT_LOWAT, out there.

Microblogging with ActivityPub

dskoll — Wed, 30 Nov 2022 02:08:04 +0000

This is absolutely correct.

However, one way to somewhat mitigate your specific attack scenario is to refrain from sharing posts from newly-seen servers for a week or two (but allow admins to sample them.) This could make the attack too expensive and the bad actors more likely to be shut down before they can spread garbage to the general public.

There's also a trust issue. You have to trust the admins of your ActivityPub server to do their due diligence before accepting content from random instances. (I assume they can control this? Not 100% sure.)

So far, at least, Mastodon is way better than Twitter when it comes to civility. But now that George Takei is on, we'll see how well it scales. 🙂

Microblogging with ActivityPub

tau — Wed, 30 Nov 2022 01:29:48 +0000

>All of the various servers running this software exist in a federation somewhat like email, colloquially known as "The Fediverse"

And it will need to come up with a solution to the problems that plague "federation[s] somewhat like email". A spammer with some compromised AWS API keys or stolen credit card numbers can easily taste a few hundred domain names, spin up a few hundred EC2 instances to go with them, then inundate popular Fediverse servers with garbage during the brief period before the instances created by the compromised keys are deleted and their speculative domain registrations need to be refunded. Instance operators are overworked and underfunded as it is, a flood of this sort of garbage could quickly render the ecosystem unusable.

I wish this movement the best of luck, but the management of every electronic communication service eventually reduces to a spam filtering problem. Nobody has figured out a way to make blocking spam cheaper than sending spam, the best solution we have found so far is to have large, centralized, for-profit networks that employ a large body of low-wage overseas labor to frantically mop up enough of the sewage that the patrons don't notice the smell too much.