LWN: Comments on "Free software's not-so-eXZellent adventure"

Free software's not-so-eXZellent adventure

farnz — Thu, 11 Apr 2024 10:00:04 +0000

I don't think you can complacently _rely_ on someone detecting these things quickly, the idea that "many eyes makes bugs shallow" doesn't address the governance model that directs and pays someone to actually look beyond their hobby/curiosity, but the thing with backdoors/bugs in open source is that they are detectable and once they are detected we have the tools to do comprehensive analysis out in the open as well, so they are high-risk for the attacker as it only takes one curious "huh, that's weird" to blow the whole operation.

FWIW, I've always interpreted "many eyes make bugs shallow" as "once a bug has been identified, people will find the root cause and a good fix quickly", and not "all bugs will be found quickly". This meaning of the phrase still applies here - the bug was found (via a mix of chance and Andres Freund's hard work) a long time after it was introduced, but it took very little time to find the root cause and a fix for the bug once it had been identified.

Free software's not-so-eXZellent adventure

raven667 — Wed, 10 Apr 2024 18:35:58 +0000

> This attack increased the CPU used by those logins by about a factor of 3. It's difficult to believe this would not be noticed

That may be true right now, but I'm not sure that the increased CPU usage was a fundamental property of this backdoor or just a bug that could have been fixed had this backdoor persisted for longer. It may have been rushed because the exploit chain they used was about to dry up as the systemd project announced that they were working on using dlopen() for these dependencies to reduce the overhead for tiny environments.

I don't think you can complacently _rely_ on someone detecting these things quickly, the idea that "many eyes makes bugs shallow" doesn't address the governance model that directs and pays someone to actually look beyond their hobby/curiosity, but the thing with backdoors/bugs in open source is that they are detectable and once they are detected we have the tools to do comprehensive analysis out in the open as well, so they are high-risk for the attacker as it only takes one curious "huh, that's weird" to blow the whole operation. With proprietary closed software there may be processes that could catch this at some places but not many and I would expect something as well hidden as this to go undetected for a long time, and if it were detected there is no guarantee that a private company would disclose enough detail to understand the provenance or impact, just take a look at the MS response to having a threat actor take a signing key from Azure to craft login tokens for O365 services and how little actual detail was disclosed, the US DHS Cyber Safety board wrote a whole paper on that themselves. Lack of information sharing leads to the same attacks working against different organizations over and over again because they can't learn from their peers, eg. the explosion of ransomware, like the jackass in every zombie movie who doesn't tell anyone they are bitten.

Free software's not-so-eXZellent adventure

mathstuf — Wed, 10 Apr 2024 17:21:33 +0000

Sure, there aren't data blocks to hold image-viewer-ignored information, but stenography still exists (though that exists for any "list of numbers" format rather than say, SVG or another more algorithmic description of the image).

Free software's not-so-eXZellent adventure

heftig — Wed, 10 Apr 2024 14:57:10 +0000

I guess for icons and pixel art you could check Netpbm images into Git, instead? Since they're ASCII text you can't smuggle in hidden data as in PNG and JPEG images.

Volunteering to be the maintainer

farnz — Tue, 09 Apr 2024 11:15:37 +0000

You seem to be mixing up two sorts of bans:

A ban that happens simply because you tried to observe but not speak.
A ban that happens after you spoke in a disruptive fashion.

Most, if not all, open source projects don't ban you from observing public forums; there may be invite-only forums that you're not part of, but as long as you're not speaking, you can listen, although you may need to use a different identity to your normal identity if you've previously been banned for being disruptive (or rely on external logs - e.g. you can use mailing list archives to read a list you're banned from, or a project's IRC log, or Incognito Mode in your browser to read a web forum).

And it is surely right that society excludes disruptive people from areas where they cause issues; society is, in this situation, valuing rightness over ability to disrupt.

Volunteering to be the maintainer

immibis — Tue, 09 Apr 2024 10:17:55 +0000

Certainly I'm not entitled to talk to people who don't want to listen. However:

- such a ban has a collateral effect as people who do want to listen aren't allowed to listen and often aren't even aware they're not allowed to listen.
- as a society we should probably value rightness, not be indifferent to it.

Volunteering to be the maintainer

rra — Mon, 08 Apr 2024 16:07:20 +0000

> When you allow bans, people get banned incorrectly and this is just as frustrating on the other end.

This is exactly the type of social fallacy (and, frankly, entitlement) that I'm decrying. It does not matter whether you're frustrated about being banned. The scarce resource here is maintainer time and energy, not the frustration level of random people on the Internet. If I don't find someone's communication productive and I haven't agreed to some sort of relationship with them (being formal members of the same project, for example), I am under precisely no obligation to communicate with them as a free software maintainer.

In other words, I do not care in the slightest whether the people who are banned find that frustrating, and, in my opinion, neither should anyone else who is in the maintainer's position. Caring what annoying people think about you or your project is a losing game. Block them and move on. There are more helpful, productive, and positive interactions about free software than one can do justice to in a lifetime. Dropping communication with people the first time they seriously annoy you will free up so much time to do more useful things, will do wonders for your mood, and nothing irreplaceable will be lost.

You do not owe random people on the Internet a debate. Even if they're right!

Free software's not-so-eXZellent adventure

ras — Mon, 08 Apr 2024 10:32:48 +0000

> Yes, there was a massive bit of luck involved in the detection of this attack

I don't want to diminish Andres skill and dedication in following this through because it was a sterling effort, but I suspect the detection of this particular attack was inevitable and it would have happened sooner rather than later.

I guess most of us here run VM's with ssh servers exposed to the internet, and all of those are under continual attack. Normally it takes sshd a relatively small amount of CPU time to reject the an unknown login, but even so the shear number of attacks we get now combined with the tight provisioning of VM's means the resources used are very noticeable. None of the attacks will get through my ssh installations, but regardless the shear number of them has forced me to use fail2ban to keep the OOM killer at bay.

This attack increased the CPU used by those logins by about a factor of 3. It's difficult to believe this would not be noticed. In fact, it Andres picked it up because of this very tell-tale. If it wasn't him, it would have been someone else.

On the downside, I expect that mistake won't be repeated next time around.

Volunteering to be the maintainer

smurf — Mon, 08 Apr 2024 07:00:42 +0000

Not mentioning the specific class of people $asshat wants killed isn't being obtuse, much less deliberately.

It's about highlighting the fact that it doesn't matter which people are his (I assume) problem, or whether the subcategory in question is racial or sexual or environmental or whatnot.

Last but not least, it's about not risking triggering a side discussion about $people here.

Volunteering to be the maintainer

atnot — Sun, 07 Apr 2024 20:47:43 +0000

Based on the how deliberately obtuse you're being about who "$people" are here, I'm just going to assume that they probably made a good call there.

And also failing closed is actually a very good thing. Not only for the safety of the community, but because the contributions of any one bad or abusive person are without exception severely outweighed by the amount of people they put off from joining the project. If anything in my experience we're not banning enough.

Free software's not-so-eXZellent adventure

immibis — Sun, 07 Apr 2024 20:28:39 +0000

If you can run code, you can run code. It could have been in the constructor function which runs when the library is loaded, but it wasn't, but if this ifunc option wasn't available then it probably would have been.

Volunteering to be the maintainer

immibis — Sun, 07 Apr 2024 20:26:52 +0000

When you allow bans, people get banned incorrectly and this is just as frustrating on the other end. Allowing arbitrary bans is probably a different social fallacy. It fails closed instead of failing open, and it's not clear that's much better in some cases.

Recently, I personally noticed that an executive at one of the big OSS foundations happened to also be a racial supremacist (noncontroversially - he basically said "kill all the $people" many times). I did what seemed sensible, and went to a group that seemed equipped to have software politics and said "hey, I noticed this thing, what's the right response?" Because the wrong people were online at the time and in a bad mood, I am now permanently banned from talking to anyone there or reading any messages.

Free software's not-so-eXZellent adventure

immibis — Sun, 07 Apr 2024 20:17:49 +0000

and thirdly, distroa will drop useful packages and people will drop those distros.

Free software's not-so-eXZellent adventure

mathstuf — Sun, 07 Apr 2024 17:36:19 +0000

Seems like this is already the case: https://lwn.net/Articles/968256/

Free software's not-so-eXZellent adventure

mathstuf — Sun, 07 Apr 2024 17:31:51 +0000

> when static constructors are run (when a library is loaded, or when you first access a symbol from the library - both would be fine within the letter of C++).

They are run when a symbol from the *object* containing the static object resides is accessed.

Free software's not-so-eXZellent adventure

mathstuf — Sun, 07 Apr 2024 17:28:33 +0000

I'm shooting into the dark, but couldn't `ifunc` support be limited to touching symbols for the shared object itself and not others? Breaks down for fully static builds though…

Free software's not-so-eXZellent adventure

mathstuf — Sun, 07 Apr 2024 17:19:58 +0000

> It'd be better to take advantage of the fact that, at its strongest, the C++ standard has required statics to be initialized before the first reference by the program to a symbol from the translation unit the static is defined in.

FWIW, macOS does this, so while it is not compatible with the SysV ABI as khim points out, anything cross platform that needs reliable static initialization and is cross-platform needs to handle this case already. I'd be fine with a compile flag to make this doable in ELF with an opt-in compile flag…

Free software's not-so-eXZellent adventure

mathstuf — Sun, 07 Apr 2024 16:20:04 +0000

However, I feel that cryptocurrency mining would have also made the fact that *something* was up very evident and have led to discovery before "much" benefit had been gained without sitting on it to be deployed to LTS distro releases first. Given the pressure to get it into the bleeding edge distros, I suspect that this was more of interest to get "landing pads" for further work deployed. I feel like cryptocurrency mining is better off finding 0-days in existing software as the scam-coin-to-mine changes faster than this deployment process but slower than "people patching their 0-day bugs".

Free software's not-so-eXZellent adventure

mathstuf — Sun, 07 Apr 2024 16:12:43 +0000

FD: CMake developer and have commented in that issue.

Yes, detecting the environment and changing behavior based on its state is an anti-pattern in my mind. This generally means that one should have things be on or off rather than "see if available and enable it if so". *Defaults* for things might make sense behind `--enable-sandboxing-features` so that packaging systems can use the same `configure` command line across multiple versions (with fine-grained disabling where necessary).

My main concern is that the same command line can result in vastly different builds just because some dependency was not made appropriately discoverable at the right time and you end up with a "box of chocolates"[1] build.

There *are* some exceptions to this:

- polyfills for basic platform APIs in order to get compatible behaviors (e.g., libcurl's detection of `send` argument qualifiers to avoid warnings)
- platform knowledge (e.g., never check Landlock on Windows)

Of course, one can also just memoize this information (`libuv` does this) and use the preprocessor to detect feature support with version checks.

[1] https://www.youtube.com/watch?v=bNrNPD1uAII

Free software's not-so-eXZellent adventure

jwakely — Sun, 07 Apr 2024 11:08:36 +0000

Yes! Does the software work? Good, let it work then. Does it need avx512 optimisations, or a Zig port? Probably not. Let somebody else provide it as a separate library or in a fork.

If a vulnerability is found and affects distros relying on the code, the distros should be developing and contributing a fix (maybe while the vuln is still embargoed) and not expecting the original author to do that work for free.

Pressuring unpaid maintainers to make working libraries become less stable isn't progress.

Switching fields

nicku — Sat, 06 Apr 2024 00:51:19 +0000

Thank you Jon.

Free software's not-so-eXZellent adventure

smitty_one_each — Fri, 05 Apr 2024 18:36:15 +0000

> autoconf really has to go

It's one thing to say that $PACKAGE has to go.

It's another thing to say that $PACKAGE can be replaced by objectively superior $ALTERNATIVE.

The real thing is to be Git and be both so "superior" and have enough marketing mass to decimate the alternatives.

Probably what's needed is, e.g. an not only an autoconf => cmake (for example) translator, but a name that can straight up replace the old stuff, beginning with GNU in its entirety.

Maybe the answer is a huge kickstarter at GNU. Doubling down, this might also be a post-RMS marketing campaign for the FSF.

Free software's not-so-eXZellent adventure

karim — Fri, 05 Apr 2024 16:47:19 +0000

Nitpick:
"There is a good chance that our community, made up of people who just want to get something done and most of whom are not security experts, has just been attacked by a powerful adversary with extensive capabilities and resources. We are not equipped for that kind of fight. But, then, neither is the proprietary world."

The proprietary world is probably unlikely to be vulnerable to the "good cop"/"bad cop" routine that seems to have been used here. It doesn't mean it's not vulnerable to other forms of attack. Just saying.

Unrelated, but somewhat comes to mind:
I remember the key signing routines at OLS from the early 2000s. I had always found those dubious. None of the participants were professionally trained at validating government-issued ID -- something even government agencies can struggle with themselves.

We (the open source community) have been operating since forever on implied, distributed trust simply based on each participant's own evaluation of the trustworthiness of the people they're interfacing with.

I have no grand solution or alternative to this. But anyone who, as the article states, is willing and able to play the long game can and likely will be able to score against us.

Maintainer funding is one way of alleviating, although not eliminating, maintainers' feeling of desperation. I've seen several maintainers over the past decades eventually just tire of seeing their work being used for profit by others while they get nothing.

Free software's not-so-eXZellent adventure

bferrell — Fri, 05 Apr 2024 15:54:50 +0000

One other "normalized" "thought" that might bear looking at... "faster! faster!!! Break it, we'll fix it later!!!"
The pressure came from "languishing patches"

Maybe slower is a good thing unless we have a specific incident to address?

Thanks and a comment on tarballs

lisandropm — Fri, 05 Apr 2024 13:39:48 +0000

And part of it is already at "Distributions quotes of the week". Maybe I should read the whole LWN release before commenting :-)

Thanks and a comment on tarballs

lisandropm — Fri, 05 Apr 2024 13:25:32 +0000

First of all: **thanks**. It is a totally insightful article. Also, I can not but feel totally represented in:

> Distributors, as a general rule, would rather carry fewer patches than
> more, and don't patch the software they ship without a reason.
> So, while both complexity and downstream patching should be
> examined and avoided when possible, they are a part of our world
> that is not going away.

Going down that line: indeed the world seems to have gone trough the VCS/git line and abandoned tarballs, and yes, *maybe* we need to change the way certain things work. Should we really abandon tarballs? I don't know, and I also can't say much: I do not currently have the time to help with the required tooling...

But I think Guillem Jover did some very insightful comments on [0], specially at the end. I recommend taking the time to read it in full (it's long, possibly too-Debian related, but it is a good point of view of the current state of affairs).

[0] <https://lists.debian.org/debian-devel/2024/03/msg00367.html>

Free software's not-so-eXZellent adventure

khim — Fri, 05 Apr 2024 11:38:33 +0000

> How much did it cost?

Three years.

> They worked on it part-time for several years.

And that immediately rules out all but largest companies, lone hobbists… and three-letter agencies.

> They were hoping to gain a backdoor to sshd and potentially some more.

They were hoping to get a reusable backdoor.

> This is not beyond what a medium black-hat company could do.

This is beyond what medium black-hat company may plan to do.

> Assuming that there's market for such exploits.

That's precisely the thing: exploits on black market command high price only if they are noncompromised.

Black market favors independent (even if easily detectable and patchable) exploits which you may sell to many buyers. Compromised, well-known exploits go down in price sharply.

This was an attempt to plant reusable exploit which was, presumably, was either planed to be used against one particular extra-high-profile target (which would still be there after three years!) or, alternatively, be reused again and again.

To reuse it again and again you need to “keep it in house” and ensure details wouldn't leak. To even have some high-profile target that is sitting there waiting for your attack you have to large organization which plans that encompass decades.

This, pretty much, rules out everyone but $THREE_LETTER_AGENCY es.

> And I guess they already employ such people.

It's not about abilities to do such an exploit. It's about the use of such exploit. Only $THREE_LETTER_AGENCY (and ironically enough, lone hobbist) may benefit from something like this. And this looks like a work of a team which rules out “lone hobbist” hypnotises.

Free software's not-so-eXZellent adventure

tzafrir — Fri, 05 Apr 2024 10:34:26 +0000

How much did it cost? The team behind this definitely had one "star" programmer and some more team members. They worked on it part-time for several years. They were hoping to gain a backdoor to sshd and potentially some more.

This is not beyond what a medium black-hat company could do. Assuming that there's market for such exploits. They could have hoped to cover the costs of the operation. And I guess they already employ such people.

Free software's not-so-eXZellent adventure

smurf — Fri, 05 Apr 2024 09:38:41 +0000

> In C programming, there are those who argue that header files should not include other header files. Rather, if a header needs something then each user should #include its dependencies explicitly.

There's a reason for this attitude, which is that C is somewhat brain dead and doesn't have namespaces, an explicit list of things included files export, or anything else along these lines. Thus any random macros which X exports to Y are visible in Z, and the programmer working on Z should be aware of that.

Fortunately many public include files of non-system headers use prefixed symbols, and the linker has gotten better at complaining when globals collide. Thus this is not much of an issue.

On the other hand, this practice severely hampers upwards compatibility. When A includes B which includes C which requires a new system header D, should we force A to add some random include file? not really IMHO.

Free software's not-so-eXZellent adventure

smurf — Fri, 05 Apr 2024 09:11:00 +0000

It reads from data that was hidden in / disguised as part of the test suite; the file in question wasn't even used as actual test input AFAICR. That falls squarely into the "subverted anyway" category.

So yes you're right in that in this case the test output didn't actually influence the build. Thus to be safe against "hide an exploit's core in plain sight" attacks we'd have to go a step further and mandate that the builder cannot access its test data, binary or otherwise.

Switching fields

LtWorf — Fri, 05 Apr 2024 07:57:26 +0000

I think you forgot to read the no warranty clause on every free software license.

Volunteering to be the maintainer

rra — Thu, 04 Apr 2024 23:13:31 +0000

> Ideally, people complaining about a lack of activity would be disregarded as background noise, so Collin would have felt less pressure to add a maintainer.

This is a somewhat GitHub-specific point, but since for a long time I didn't realize this option existed, maybe other people don't as well.

If you go into your personal settings on GitHub, and then expand Moderation on the left sidebar, there is a tab called Blocked Users. If you go there, you can block a GitHub account from all of your repositories. This prevents them from opening issues, PRs, interacting on issues, etc. You can also ask GitHub to warn you if any repository you interact with has prior commits from that user.

I haven't checked whether GitLab and other forges have a similar feature. Hopefully so.

People in free software, perhaps due to the Geek Social Fallacies, often seem too reluctant to simply ban people. If I saw even half of the type of malicious attempted bullying in the two bugs referenced earlier in this thread on a project I maintain, I hope that I would have been willing to just ban the user and move on. In fact, I have already proactively banned the two worst offenders in those bugs from all of my repositories, and I recommend others consider doing the same when they see someone behave egregiously, even on repositories they aren't involved in.

It's mentally hard to be subjected to constant ongoing harassment and tell yourself that you should just ignore it. Let the software do it for you.

Free software's not-so-eXZellent adventure

draco — Thu, 04 Apr 2024 19:54:17 +0000

If valgrind were at (or near) the base of the capability hierarchy for the process, it would have the necessary permissions. It probably means Valgrind is built differently than other processes (with a stub before the dynamic linker that retains the process root capability for careful use later) and then launching the program(s) to be analyzed inside the valgrind process, but that seems workable.

I could also imagine that Linux could provide the necessary CHERI capabilities for other processes to those with the appropriate Linux capabilities (i.e., root) if we had no better mechanism.

Free software's not-so-eXZellent adventure

koh — Thu, 04 Apr 2024 19:32:38 +0000

You're right, this isolation could in principle be done using the compartmentalization features the arch does provide. But is it feasible, assuming there was kernel and userland support for the supported base archs?

For instance, I wonder how valgrind and other tools substituting their own symbols into a dynamically linked executable's could work under such a construct. IIUC with the ld.so design you mentioned it couldn't work, as valgrind would be shielded from the child process's libc, is that right? If so, maintaining two different linkers (or worse: via envvar) seems to counteract the gains. Without the valgrind on postgres Andres might not have felt the incentive to dig deeper and find out about the exploit. Also everyone else needs it (I presume).

Free software's not-so-eXZellent adventure

draco — Thu, 04 Apr 2024 18:55:26 +0000

I think you're underselling the power of having memory access controlled by capabilities that can't be created arbitrarily and must be derived from another one of the same or larger bounds (provenance) and same or greater permissions.

The libraries wouldn't be allowed to create a write pointer into the GOT or PLT without being provided an appropriate capability. (And logically, you'd probably do it the other way around.)

And the library couldn't reach into the dynamic linker memory for the write capability without having a capability pointing into its memory that allows read. (One possibility for this architecture is process memory isolation without context switching.)

This also allows the dynamic linker to isolate the libraries from each other—which counters the "liblzma would just manipulate sshd's callback tables directly" attack that was mentioned in response to "eliminate IFUNC entirely". It also ensures that the dynamic linker can enforce that manipulation of the PLT/GOT are sane.

That this changes the ABI is already a given, since using memory addresses as pointers is impossible in enforcing mode.

Free software's not-so-eXZellent adventure

epa — Thu, 04 Apr 2024 18:54:50 +0000

You’re right. Removing the dependency would be a bit tardy. But having to add it explicitly is kind of a feature and what I intended. So if your program starts to link in libxz, that’s because you explicitly okayed it and not just a hidden transitive dependency of some other project.

In C programming, there are those who argue that header files should not include other header files. Rather, if a header needs something then each user should #include its dependencies explicitly. I don’t have a view on the wisdom of that for C development but it’s a similar idea to what I am suggesting at the linking step.

Perhaps given a difference between “safe linking”, which can only define new symbols and not replace them or execute arbitrary code at startup, and the full-fat kind that can do things merely by including a library, transitive dependencies could be included by default if they are “safe”, but if you have a dependency of a dependency which wants to have ifunc hooks or startup code, you must explicitly ask for it or the linker will complain.

Free software's not-so-eXZellent adventure

draco — Thu, 04 Apr 2024 18:28:27 +0000

No, it read from test *input*. There was no need to run the tests.

Here's a picture: https://cdn.arstechnica.net/wp-content/uploads/2024/04/xz...

Free software's not-so-eXZellent adventure

ceplm — Thu, 04 Apr 2024 18:26:46 +0000

Be nice, it could be just an enthusiast who wants to have the latest and greatest. I have filed plenty of such bugs in my life as well.

Free software's not-so-eXZellent adventure

farnz — Thu, 04 Apr 2024 17:27:40 +0000

That opens up a different problem; if only the main binary can indicate dependencies (so no indirect dependencies), how do you remove your liblzma dependency when libsystemd stops depending on it? How do you cope when a future version of libsystemd adds a dependency on libzstd2?

Remember that we're talking about the dynamic linker at this point, not the static linker - so you can't rely on the version of libsystemd you used at compile time being the one used at runtime.

Free software's not-so-eXZellent adventure

epa — Thu, 04 Apr 2024 17:22:33 +0000

I was envisaging that you’d also have to specify all the libraries you link against. So if libsystemd requires libxz, you’d need -lsystemd -lxz on the command line when building something that uses libsystemd. No hidden transitive dependencies.