Cost vs benefit ? [LWN.net]

Cost vs benefit ?

Posted Jun 7, 2026 1:57 UTC (Sun) by Cyberax (✭ supporter ✭, #52523)
In reply to: Cost vs benefit ? by intelfx
Parent article: Moving beyond fork() + exec()

The problem is that glibc's symbol versioning is terrible. If you compile an app on a later version of glibc, you can't deploy it on an earlier version. And there's no easy way to fix this.

user32.dll and macOS libc, in contrast, are extremely stable. I can compile an app on Windows 11, and it will likely work on Windows XP, as long as it doesn't use any missing functions.

I understand that symbol versioning theoretically prevents subtle bugs with slightly different function behaviors between versions. But we're talking about freaking libc!

to post comments

Cost vs benefit ?

Posted Jun 7, 2026 8:27 UTC (Sun) by erincandescent (guest, #141058) [Link] (56 responses)

The problem here is that on Linux we link against /lib/libc.so.6, and not e.g. /usr/lib/sdk/rhel-8.0/lib/libc.so.6.stub

of course working out what such "sdks" should exist and what such a stub file should look like... is a non-trivial question

Cost vs benefit ?

Posted Jun 7, 2026 9:58 UTC (Sun) by malmedal (subscriber, #56172) [Link] (52 responses)

I'm convinced that glibc is a major reason behind both Go's otherwise rather surprising success and the year of the Linux desktop never arriving.

Even in ostensibly well-run computer departments there are almost always some dirty corners with an out of support RHEL box doing something important, can't be upgraded, can't be turned off, don't have any other like it, and then you get a request to just do this specific thing and don't touch anything else. With Go you can sit on a random Mac/Windows/Linux box, write what you need, and just copy it over.

Cost vs benefit ?

Posted Jun 7, 2026 10:14 UTC (Sun) by bluca (subscriber, #118303) [Link] (51 responses)

That just exposes a limitation of the build and deployment system in use. It's really not that hard to target the specific environment one wants to deploy for and automatically build repositories. One doesn't even have to reinvent such a system, as they already exist and are plentiful, like SUSE's OBS.

Cost vs benefit ?

Posted Jun 7, 2026 11:41 UTC (Sun) by malmedal (subscriber, #56172) [Link]

It's not hard, no. At least *I* don't find it hard. Other people seem to be mystified by build systems however, so in previous jobs more than my fair share of these tasks have been pushed over to me.

Nobody has ever asked me to run go build and scp for them...

Cost vs benefit ?

Posted Jun 7, 2026 22:50 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link] (49 responses)

For me it looked like this: I need to compile BlahApp to work on Ubuntu 18.04

So what do I do? I run the Ubuntu 18.04 in a VM/container, and try to compile my app there. Except that 18.04 doesn't have a new enough CMake, so I need to download and build CMake. I also then need to build libsomething, because it's not recent enough. Oh, and of course its pkgconfig is broken, so I need to just hardcode the paths in CFLAGS/LDFLAGS.

...6 hours later...

Ok, now I have a binary. And you want me to automate ALL these steps?!?

The story with Go: `go build ....`, copy the binary over. If the kernel has the required syscalls, it'll work.

Cost vs benefit ?

Posted Jun 7, 2026 23:59 UTC (Sun) by bluca (subscriber, #118303) [Link] (46 responses)

> The story with Go: `go build ....`, copy the binary over. If the kernel has the required syscalls, it'll work.

The story then goes: you vendored 1092302 dependencies, of which one third roll their own crypto. You have no idea of course, because you got no clue, you just run "go build" as you found on StackOverflow and you have no idea what any of that does, but it works, so who cares right? 4 months later half of those have 10.0 severity CVEs that enable RCEs. You still have no clue, as you just have a single binary blob and no idea what's in there. But hey, it's so easy to "scp" around!

Cost vs benefit ?

Posted Jun 8, 2026 2:24 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (45 responses)

And do you think a binary for an old Ubuntu 18 will somehow get updates from the Ubuntu 24 repo? Or that I'm going to re-do all the building shenanigans once one of the libraries gets Mythosed?

> 4 months later half of those have 10.0 severity CVEs that enable RCEs. You still have no clue, as you just have a single binary blob and no idea what's in there. But hey, it's so easy to "scp" around!

On the contrary. I can throw in a Github's dependabot to watch over my binary and scream at me if there are CVEs in my Go library's dependencies. For free.

The solutions to monitor a hodgepodge of randomly wired C libraries exist, but are more complex.

Cost vs benefit ?

Posted Jun 8, 2026 10:07 UTC (Mon) by bluca (subscriber, #118303) [Link] (44 responses)

> And do you think a binary for an old Ubuntu 18 will somehow get updates from the Ubuntu 24 repo? Or that I'm going to re-do all the building shenanigans once one of the libraries gets Mythosed?

Of course you won't, which is the problem: you have no idea what you are doing, and yet you ended up in charge of the entire stack, which is a job you obviously cannot do, instead of letting the professionals (Canonical) handle it for you, as it is proper and sensible and demonstrated to work

> For free.

And then it breaks because you misconfigured it and never noticed, and never fires anyway because you are just "scp" copying around random binaries built on your laptop at random points in time and lost track of them

> The solutions to monitor a hodgepodge of randomly wired C libraries exist, but are more complex.

"apt upgrade" - wow, such complexity!

Cost vs benefit ?

Posted Jun 8, 2026 10:16 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> Of course you won't, which is the problem: you have no idea what you are doing, and yet you ended up in charge of the entire stack, which is a job you obviously cannot do, instead of letting the professionals (Canonical) handle it for you, as it is proper and sensible and demonstrated to work

Ah yes, "don't bother your pretty little head when adults are speaking". This attitude led to classic desktop Linux only _now_ gaining some momentum. And only because distros finally embraced packaging and stable ABIs (and Win32 is the most stable Linux desktop ABI so far).

> And then it breaks because you misconfigured it and never noticed, and never fires anyway because you are just "scp" copying around random binaries built on your laptop at random points in time and lost track of them

Well, and what is YOUR proposal for backporting to older OSes?

Perhaps the good old Soviet Union method? As in, "Today the Great Soviet People have no demand for butter".

> "apt upgrade" - wow, such complexity!

Try to build the most recent LTFS on older RHEL or Debian. Go on, I'll wait.

What? You're still trying to compile libicu and fighting with its message generator?

Cost vs benefit ?

Posted Jun 8, 2026 11:18 UTC (Mon) by bluca (subscriber, #118303) [Link] (1 responses)

> This attitude led to classic desktop Linux only _now_ gaining some momentum.

There is no momentum on Linux desktop, there never was, there never will. It has nothing to do about your whingings, and instead all to do with 99.99999% of desktop hardware available for sale not shipping with it, because vendors simply don't care - and why should they, it's a low-margin business after all, and as for-profit entities in a capitalist society they need to prioritize making more money than they spend or they go out of business, over ideology

> Well, and what is YOUR proposal for backporting to older OSes?

Have you tried not writing shitty software? It's _really_ not that hard to find even mediocre software that just works on any platforms it targets. First you decide what your targets are, and then you write software accordingly. It's really trivial to do, one just has to know what they are doing.

Here's a handy example: https://build.opensuse.org/project/show/network:vpn:openc...
<shock and horror> single source tree that supports targeting centos 8/9/10, debian 11/12/13/14, fedora 43/44/45, rhel 8, sle 15/16, ubuntu 20.04/22.04/24.04/26.04, leap 15/16, thumbleweed. Across 4 or 5 architectures. As if by magic. No golang crap in sight.

> What? You're still trying to compile libicu and fighting with its message generator?

No, I'm not compiling any library, because unlike you I know what I am doing, and I let the OS vendor worry about providing libraries. Not my problem, as it shouldn't be.

Please stop - again

Posted Jun 8, 2026 13:13 UTC (Mon) by corbet (editor, #1) [Link]

Luca, this is the second time in just a few days that I've had to ask you to stop with the personal attacks. Seriously, just stop.

Cost vs benefit ?

Posted Jun 8, 2026 11:15 UTC (Mon) by malmedal (subscriber, #56172) [Link] (40 responses)

> instead of letting the professionals (Canonical) handle it for you, as it is proper and sensible and demonstrated to work
> [...]
> "apt upgrade" - wow, such complexity!

The professionals at Canonical have actually found that apt upgrade is too difficult so they invented snaps which basically is static linking except more heavyweight.

apt works well enough as long as you only use the software that Canonical compiiles for you, but as soon as you want to use something from a third party vendor it starts being complex. It often works for a while with one vendor that gives you a nice ppa, but if you need two or three it gets really annoying to debug the conflicts.

Cost vs benefit ?

Posted Jun 8, 2026 11:22 UTC (Mon) by bluca (subscriber, #118303) [Link] (39 responses)

> The professionals at Canonical have actually found that apt upgrade is too difficult so they invented snaps which basically is static linking except more heavyweight.

That's an answer to flatpak and sandboxing, nothing to do with static linking

> apt works well enough as long as you only use the software that Canonical compiiles for you, but as soon as you want to use something from a third party vendor it starts being complex. It often works for a while with one vendor that gives you a nice ppa, but if you need two or three it gets really annoying to debug the conflicts.

Have you tried not using crappy vendors?

Cost vs benefit ?

Posted Jun 8, 2026 12:27 UTC (Mon) by malmedal (subscriber, #56172) [Link] (38 responses)

> That's an answer to flatpak and sandboxing, nothing to do with static linking

It's an answer to dynamic linking not being a good idea with more than one supplier.

> Have you tried not using crappy vendors?

Vendors are not selected for their skills in packaging software, the criteria is whether the software is useful or not.

Cost vs benefit ?

Posted Jun 8, 2026 12:44 UTC (Mon) by bluca (subscriber, #118303) [Link] (37 responses)

> It's an answer to dynamic linking not being a good idea with more than one supplier.

It still uses dynamic linking, so no

> Vendors are not selected for their skills in packaging software, the criteria is whether the software is useful or not.

If they can't even get the basics right then it is not useful by definition

Cost vs benefit ?

Posted Jun 8, 2026 13:14 UTC (Mon) by malmedal (subscriber, #56172) [Link] (36 responses)

> It still uses dynamic linking, so no

Please read what I wrote, "not a good idea with more then one supplier". If you have e.g. an appimage you have one supplier for all the dlls so it's perfectly fine.

> If they can't even get the basics right then it is not useful by definition

It's useful if the software does something the buyer wants done.

Cost vs benefit ?

Posted Jun 8, 2026 13:33 UTC (Mon) by bluca (subscriber, #118303) [Link] (35 responses)

> Please read what I wrote, "not a good idea with more then one supplier". If you have e.g. an appimage you have one supplier for all the dlls so it's perfectly fine.

The supplier of libraries is the same in both cases - the OS vendor, IE Canonical

> It's useful if the software does something the buyer wants done.

If it doesn't work then it doesn't get anything done by definition

Cost vs benefit ?

Posted Jun 8, 2026 14:44 UTC (Mon) by malmedal (subscriber, #56172) [Link] (34 responses)

> The supplier of libraries is the same in both cases - the OS vendor, IE Canonical

No, in an AppImage it can be anybody, usually whoever was the OS vendor where the packge was made.

> If it doesn't work then it doesn't get anything done by definition

You seem to not bother reading the posts you respond to, as I explained previously, it is often that two apps from different vendors will have confliciting requirements. Each will work just fine by itself. Sometimes they will both work at first and then a later update will introduce a conflict.

Cost vs benefit ?

Posted Jun 8, 2026 14:48 UTC (Mon) by bluca (subscriber, #118303) [Link] (33 responses)

> No, in an AppImage it can be anybody, usually whoever was the OS vendor where the packge was made.

Well sure, but appimage is an entirely different thing from snaps...?

> You seem to not bother reading the posts you respond to, as I explained previously, it is often that two apps from different vendors will have confliciting requirements. Each will work just fine by itself. Sometimes they will both work at first and then a later update will introduce a conflict.

Sure, software sometimes breaks, still not sure what this has to do with shared libraries?

Cost vs benefit ?

Posted Jun 8, 2026 16:00 UTC (Mon) by malmedal (subscriber, #56172) [Link] (32 responses)

> but appimage is an entirely different thing from snaps...?

Thought it would be easier to explain that way since I find snapcraft.io is a bit messy, but lets use snaps then.

Look at snapcraft.io and click on C/C++ and see this listed advantage of snaps: "Snaps install and run the same across Linux. They bundle the exact versions of your app’s dependencies."

This particular advantage is the same as what static linking gives you.

Snaps are in many ways better than static linking since static linking only ensures that the libraries are correct, while snaps can also make sure that other resource-files, icons, images, etc. are also present. There is in no rule that a snap uses shared libraries from Canonical, they can be from anywhere, there was talk of saving memory and use only one copy if two snaps want the exact same library, but I am not sure if that has been implemented.

You can list further advantages of snaps such as automatic updates, but shared libraries out of sync is the item that I have seen stop users being able to use the software at all, so for that reason I focus on just that point as the big issue that needs fixing for Linux to take over the desktop.

> Sure, software sometimes breaks, still not sure what this has to do with shared libraries?

What usually happens is that software from vendor A and vendor B depends on different versions of the same library. In principle this can also happen with any kind of file, but vendors are usually diligent about having their name somewhere in that path, so I've only ever seen it happen with shared libraries.

Cost vs benefit ?

Posted Jun 8, 2026 16:15 UTC (Mon) by bluca (subscriber, #118303) [Link] (31 responses)

> This particular advantage is the same as what static linking gives you.

It is not. It's just a worse version of flatpak, which achieves the same result by having shared runtimes with shared libraries that applications use.

> What usually happens is that software from vendor A and vendor B depends on different versions of the same library. In principle this can also happen with any kind of file, but vendors are usually diligent about having their name somewhere in that path, so I've only ever seen it happen with shared libraries.

Those are bugs. All projects must depend only on what the target vendor provides: et voila, no conflicts.

Cost vs benefit ?

Posted Jun 8, 2026 17:18 UTC (Mon) by malmedal (subscriber, #56172) [Link] (30 responses)

> Those are bugs.

Yes, nobody is disputing that. They are bugs that keep happening on Linux frequently enough that I think it is a major reason for why the year of the Linux desktop hasn't arrived yet. (Not the only reason to be clear)

Cost vs benefit ?

Posted Jun 8, 2026 17:27 UTC (Mon) by bluca (subscriber, #118303) [Link] (26 responses)

That seems wildily unlikely. The "year of the linux desktop" will never arrive because there's no reason for consumer device vendors to spend any serious money to ship such devices. This has nothing to do with shared libraries, they won't even know what they are, and all to do with razor thin margins and a sector which is experiencing serious difficulties as it is, without embarking in hopeless projects with no clear returns. Such is life under capitalism.

Cost vs benefit ?

Posted Jun 8, 2026 18:36 UTC (Mon) by malmedal (subscriber, #56172) [Link] (25 responses)

Linux is cheaper than the competition, in a capitalistic society the cheapest option will most often win, so you would expect Linux to win everywhere. In practice it has won everywhere *except* the *desktop*. So what is special about the desktop compared to the rest of computing?

The reason I think shared libraries is part of the problem is from observing users. Even colleagues whose job it was to actually run Linux would struggle with any shared library issues.

Cost vs benefit ?

Posted Jun 8, 2026 18:48 UTC (Mon) by bluca (subscriber, #118303) [Link] (24 responses)

> Linux is cheaper than the competition, in a capitalistic society the cheapest option will most often win, so you would expect Linux to win everywhere. In practice it has won everywhere *except* the *desktop*. So what is special about the desktop compared to the rest of computing?

No, because the cost of the license is irrelevant and a pittance. What matters is what sales&support agreements large hardware vendors get with large software vendors, what kickback their sales folks get, how much they get wined&dined, etc etc.

> The reason I think shared libraries is part of the problem is from observing users. Even colleagues whose job it was to actually run Linux would struggle with any shared library issues.

The average user doesn't even know what a library is, let alone having any issue with it. They just click on the desktop dashboard to start the browser or mail client or document editor. If they are a bit more advanced, they may also know how to open steam to install and play a few games.
IE, you are falling for https://xkcd.com/2501/

Stop here

Posted Jun 8, 2026 19:03 UTC (Mon) by jzb (editor, #7867) [Link]

It appears that we have strayed far from the original topic; perhaps this debate could be moved elsewhere or at least not conducted here.

Cost vs benefit ?

Posted Jun 9, 2026 8:49 UTC (Tue) by malmedal (subscriber, #56172) [Link] (22 responses)

> you are falling for https://xkcd.com/2501/

I'm not, on windows a normal user can copy a program from one machine to another. On Linux, no.

Cost vs benefit ?

Posted Jun 9, 2026 10:32 UTC (Tue) by bluca (subscriber, #118303) [Link] (21 responses)

Really? So you can copy a UWP from Windows 11 to Windows XP and it will work?

Cost vs benefit ?

Posted Jun 9, 2026 16:50 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (18 responses)

You can copy a Win32 binary, and as long as it doesn't depend on anything new, it'll work.

Nobody is asking for impossible. But glibc has been actively preventing the _basic_ interoperability.

Cost vs benefit ?

Posted Jun 9, 2026 17:14 UTC (Tue) by bluca (subscriber, #118303) [Link] (17 responses)

> You can copy a Win32 binary, and as long as it doesn't depend on anything new, it'll work.

...so exactly as you do on Linux with glibc, then. You target the lowest common denominator among your targets, and then you can ship on all of them, if for some reason you insist on a single build because the build system is use is not adequate.

Cost vs benefit ?

Posted Jun 9, 2026 17:42 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (16 responses)

No. You seem to be remarkably impervious to actual information that people are telling you.

Windows or macOS make it _very_ easy to target earlier OSes. I can open VisualStudio, specify Windows 10 as a target, and build code for it. Even though I'm running on Windows 11. Same for macOS, I can set up macOS BigSur from 2020 as a target, even though I'm running on Tahoe from 2026.

And this behavior has ALWAYS been the case. I used to compile software for Win95 from my workstation on WinXP.

There is NOTHING like this for classic desktop Linux. I can not just install Ubuntu 26.04 and create a binary that would work on Ubuntu 24.04. Even though this binary does not use anything more recent than epoll(), which was added more than 20 years ago. And the main reason for that is the utterly braindead glibc symbol versioning.

That's why Go became so popular. I can make a binary, scp it and then use it immediately.

Cost vs benefit ?

Posted Jun 9, 2026 18:24 UTC (Tue) by bluca (subscriber, #118303) [Link] (15 responses)

> I can not just install Ubuntu 26.04 and create a binary that would work on Ubuntu 24.04.

Of course you can. I regularly build on my Debian 13 x86_64 binaries targeting a Yocto Dunfell aarch64 environment, no sweat.
It's exactly the same as Windows or OSX. The toolchains and dev environments are different - obviously - but the processes are essentially equivalent.

Cost vs benefit ?

Posted Jun 9, 2026 18:34 UTC (Tue) by quotemstr (subscriber, #45331) [Link] (5 responses)

Have you built for other OSes? Down level targeting of glibc is a whole universe of pain that's just absent on other systems. That you need something as enormous as Yocto (built for generating embedded device immutable images, not distribution packages) to get a semblance of downlevel compatibility is a further indictment of the glibc model.

Cost vs benefit ?

Posted Jun 9, 2026 18:55 UTC (Tue) by bluca (subscriber, #118303) [Link] (4 responses)

No, it isn't an indictment of anything, as it's not any different from the OP's "_very_ easy" way to target OSX or Windows: you need a toolchain for the target. That's what XCode or VS give you, wrapped in a nice GUI. Nobody stops anybody from providing that same GUI for Linux targets, but it is absolutely not glibc's bloody job to give you one.

On Linux in fact things are _better_ because you have multiple ways of targeting other systems, while on OSX/Windows you either use the proprietary tools or can get into the sea.

On Linux you can:

- get a toolchain for the target (eg, Yocto with its SDK, debootstrap/dnf bootstrap/whatever else to get a target tree, etc) and use that to compile
- build packages for all your targets natively on a distro build system (eg on SUSE's OBS like the example I already gave upthread which from a single source tree builds for a dozen distros/versions/arches https://build.opensuse.org/project/show/network:vpn:openc... )
- use your favourite container engine, VSCode has extensions to wrap this up in a pretty GUI
- use Flatpak if you are building a desktop app

The demand that building against your _host_'s system should work on _any_ system is what is patently absurd. You don't demand that of OSX or Windows, and yet you demand it of Linux, and then badmouth glibc (why not gcc? why not openssl? etc) when it obviously doesn't work. Why? It makes no sense.

But the FUNNIEST thing is that, you actually CAN build against your host system and target an old one, without any of the above. You need to code it properly of course, and it's not as simple as the above methods, but if you use dlsym to load functions at runtime and handle it gracefully when they are not found, it does work. It is possible, and not that hard either, we do that in systemd now after Daan implemented it some weeks ago.

Cost vs benefit of extended discussion

Posted Jun 9, 2026 18:59 UTC (Tue) by corbet (editor, #1) [Link] (1 responses)

I vaguely recall that, once upon a time, this was an article about fork() and exec(). This discussion has clearly strayed far from that topic. Perhaps more to the point, though, it seems to be becoming increasingly circular. Perhaps it's time for all of the parties involved to let it go?

Cost vs benefit of extended discussion

Posted Jun 9, 2026 19:00 UTC (Tue) by bluca (subscriber, #118303) [Link]

> I vaguely recall that, once upon a time, this was an article about fork() and exec().

Yes, but then it was forked.

...sorry

Cost vs benefit ?

Posted Jun 9, 2026 19:15 UTC (Tue) by quotemstr (subscriber, #45331) [Link] (1 responses)

> The demand that building against your _host_'s system should work on _any_ system is what is patently absurd

No Way to Prevent This, Says Only OS Where This Regularly Happens

> when it obviously doesn't work

Compiling for a downlevel OS works just fine on other systems, including Android with Bionic. And you don't need to connect to some SUSE cloud service to do it.

The problem with downlevel compatibility for GNU/Linux isn't that it's physically impossible, but that in practice it's so fraught (especially since binaries often seem to work until they don't) that it leads people to eschew static linking entirely.

If it were so easy, why would people spend so much time making partial solutions like the glibc polyfill thing (apparently recently abandoned) or the older glibc downlevel header project (also abandoned)?

All glibc has to do is provide a version target preprocessor macro. It would cost them little and benefit the ecosystem massively.

> use dlsym to load functions at runtime and handle it gracefully when they are not found,

dlvsym, since presumably you want to target a specific contract. If you don't, you might call a new (or old) version of a symbol under ABI skew and summon the UB nasal demons.

Under the glibc model, one does not simply call dlsym. dlsym finds the default version of the symbol, which glibc points at the latest. That your "not that hard" advice in fact creates programs with nightmare ABI time bombs embedded in them is kind of emblematic of how the barriers to broader Linux adoption are social, not technical.

The unsafely of dlsym is another charge against the glibc way of managing versions.

Cost vs benefit ?

Posted Jun 9, 2026 19:18 UTC (Tue) by daroc (editor, #160859) [Link]

I will charitably assume you hadn't seen Jon's comment asking for this thread to wind up, but it has in fact wandered far off topic, and we should leave it where it lies.

Cost vs benefit ?

Posted Jun 9, 2026 18:46 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (8 responses)

And how are you doing it? Here's an actual issue that I have: I want to build LTFS to run on Synology. It has build scripts for Debian and it doesn't have to depend on anything external.

> It's exactly the same as Windows or OSX.

It's really really not. The only sane way to do it is to have another glibc in some sort of a container, from a classic chroot to a VM.

You can play around with LD scripts or manually specifying symbols, but this is practical only for no-dependencies plain C binaries. Even just depending on libstc++ makes this infeasible.

Cost vs benefit ?

Posted Jun 9, 2026 18:59 UTC (Tue) by bluca (subscriber, #118303) [Link] (7 responses)

> And how are you doing it?

https://lwn.net/Articles/1077242/

If your vendor doesn't give you what you need, go harass them instead of the glibc maintainers

> It's really really not. The only sane way to do it is to have another glibc in some sort of a container, from a classic chroot to a VM.

...which is the same bloody thing that the "_very_ easy" things you mentioned on Windows/OSX do - they give you a toolchain for your target. You accept it for those, and even call it "_very_ easy", and yet you refuse to accept the same on Linux, which is patently absurd

Cost vs benefit ?

Posted Jun 9, 2026 19:21 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (6 responses)

> If your vendor doesn't give you what you need, go harass them instead of the glibc maintainers

So I should harass Debian? Synology uses old Debian as a base.

> ...which is the same bloody thing that the "_very_ easy" things you mentioned on Windows/OSX do - they give you a toolchain for your target. You accept it for those, and even call it "_very_ easy", and yet you refuse to accept the same on Linux, which is patently absurd

Ok. How do I build on Debian 13 for Debian 12?

Your solution is "run everything in a container and bundle all dependencies". Which for some reason you also seem to be against?

Cost vs benefit ?

Posted Jun 9, 2026 19:41 UTC (Tue) by bluca (subscriber, #118303) [Link] (5 responses)

> So I should harass Debian? Synology uses old Debian as a base.

No, Synology, since that's the vendor

> Ok. How do I build on Debian 13 for Debian 12?

"debootstrap bookworm" and then use the sysroot to build against. No "dependency bundling" needed.

Cost vs benefit ?

Posted Jun 9, 2026 20:02 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

> "debootstrap bookworm" and then use the sysroot to build against. No "dependency bundling" needed.

Yes. And this is literally "run everything in a container". With a full OS installed, a compiler, and a set of include headers.

Microsoft toolchain does NOT require it. MSVS does not have an old copy of Windows hiding inside it. You simply set up a C preprocessor definition that hides symbols from the compiler. The system API .h files have guards like "#if WINVER > WIN7" so that compilation fails if the target version is too old for that symbol.

That's it. It's THAT simple.

Cost vs benefit ?

Posted Jun 9, 2026 20:22 UTC (Tue) by bluca (subscriber, #118303) [Link] (3 responses)

> Yes. And this is literally "run everything in a container".

No. That's the build toolchain, your equivalent of VS or XCode. You _do not_ ship that. It's the build environment. You do not run anything in it.

Cost vs benefit ?

Posted Jun 9, 2026 21:57 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> No. That's the build toolchain, your equivalent of VS or XCode. You _do not_ ship that. It's the build environment. You do not run anything in it.

Sigh. It's not. It's a full containerized system, with full binary copies of glibc, binutils and other libraries. It even requires root privileges to install, fakeroot is not enough. But no matter, suppose that somebody stripped all the binaries out of it and gave it to you as a tar file.

GCC indeed supports systemroot, so you can use it for very simple apps.

But anything more complicated breaks, in practice. For example, pkg-config often can't mix libraries inside and outside the systemroot properly. I recommended you to try and build LTFS ( https://github.com/LinearTapeFileSystem/ltfs ) for an earlier Debian. It's a real project, built with classic autotools. It doesn't use anything special.

And if you want to retain sanity, building it inside a chroot/container is the easiest option. And yes, glibc is absolutely the root of evil here. It does not support easy version selection, so you need to have a full copy of it for the target OS. Same goes for the C++ stdlib.

Cost vs benefit ?

Posted Jun 9, 2026 22:03 UTC (Tue) by bluca (subscriber, #118303) [Link]

> Sigh. It's not.

Yes, it is the same. It doesn't matter that it works differently, it's irrelevant, of course it's different, as it's a different system. XCode and VS are not identical either. What matters is that there are multiple simple and common ways to build for different targets, exactly like on other OSes.

Cost vs benefit ?

Posted Jun 10, 2026 9:14 UTC (Wed) by taladar (subscriber, #68407) [Link]

But the same is true for any other library, you literally need a version of every library you want to use for the target system. So why not just use a copy of the target system inside a container/chroot/... for that purpose instead of some curated set of libraries packaged in a slightly different way a second time?

Cost vs benefit ?

Posted Jun 9, 2026 18:00 UTC (Tue) by Wol (subscriber, #4433) [Link] (1 responses)

Depending on when XP went EOL, then yes if you build a program on Win11 then it would install successfully on XP. (More likely Win7, XP has been EOL for over 10 years.)

But the rule is if you build on the latest Windows, and install on the oldest "live" Windows, it should work (and if you install on EOL windows, it will probably work - "probably" getting less likely the longer it's been EOL).

Cheers,
Wol

Cost vs benefit ?

Posted Jun 9, 2026 18:17 UTC (Tue) by bluca (subscriber, #118303) [Link]

Nope. Not an UWP program. Because the point was, these things work everywhere, _if_ you code and build for the right target.

Cost vs benefit ?

Posted Jun 9, 2026 12:56 UTC (Tue) by mbunkus (subscriber, #87248) [Link] (2 responses)

Yesterday Ubuntu released a security update for nginx, the well-known & widely used web server & reverse proxy, as part of their "security" update channel. All machines that have automatic security updates enabled (usually the "unattended upgrades" mechanism) dutifully installed the update over the last 24h.

Today we arrived at $DAYJOB to a lot of our customers having opened support tickets as their reverse proxies didn't work properly anymore. Judging by the bug report on Launchpad[1] server fleets all over the world face the same issue after upgrading.

Turns out they broke the ABI with their backport of the security fix into the packaged versions. Now that wouldn't be bad if they had also re-built all packages using said ABI & including versions of those packages in the same security update, but they didn't realize they needed to (= didn't realize they broke the ABI) in the first place, so they didn't do that. Hence tons of servers trying to run a mixture of now incompatible nginx ABIs. As of 1 hour ago Ubuntu has acknowledged this, bumped the criticality to "critical", and will address the issue by first reverting the change, releasing another security update (a "regression" update), and then figuring out how to implement the fix without breaking the ABI.

I'm not making a judgement here about static vs. dynamic linking as both prioritize different properties and concerns which simply cannot be fulfilled all at the same time. Would this particular issue have happened with static linking? Obviously not. Would have static linking helped in any way, shape or form with any of the user usual security updates that distros push out daily and which do NOT require re-building the whole world? Also obviously no.

[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/2155992

Cost vs benefit ?

Posted Jun 9, 2026 13:12 UTC (Tue) by bluca (subscriber, #118303) [Link]

Yeah the lack of ABI checks in security updates is problematic. We do have the tools for it, but the glue is just not in place. The vast, vast, vast majority of the time it all works fine, but it very occasionally goes wrong like this, and having abi-compliance-checker or equivalent as part of the autopkgtest pipeline would be beneficial to avoid these occurrences.

Cost vs benefit ?

Posted Jun 10, 2026 9:23 UTC (Wed) by taladar (subscriber, #68407) [Link]

I would guess most of those ABI users were plugins, a pattern fundamentally impossible with static linking anyway as long as you want to select which plugins are included after compile time.

Cost vs benefit ?

Posted Jun 15, 2026 5:48 UTC (Mon) by aleXXX (subscriber, #2742) [Link] (1 responses)

You can download and unpack the binary distribution of CMake, it should work more or less everywhere.

Cost vs benefit ?

Posted Jun 15, 2026 11:15 UTC (Mon) by malmedal (subscriber, #56172) [Link]

The point is not that it's difficult, it's not.

The point is that it is tedious and time-consuming and most people can't be bothered, while they can be bothered to type "go build".

(Though I should mention that it's become a lot less tedious recently, LLMs can now do this with a single prompt)

Cost vs benefit ?

Posted Jun 7, 2026 22:45 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]

This is no different from Windows. Apps link to the same user32.dll in the system32 folder.

Microsoft made sure to NOT change the behavior of published API functions, except for serious bugfixes. Instead, they create new functions as needed with new names. glibc developers decided to keep the function names and transparently change the behavior, depending on the glibc version.

This is the crux of the issue. Back when I was doing Windows-to-Linux ports around 2008, I had to use a virtual machine with an old RedHat to make sure I could compile a binary that would work everywhere.

Cost vs benefit ?

Posted Jun 8, 2026 7:56 UTC (Mon) by kleptog (subscriber, #1183) [Link] (1 responses)

> The problem here is that on Linux we link against /lib/libc.so.6, and not e.g. /usr/lib/sdk/rhel-8.0/lib/libc.so.6.stub

> of course working out what such "sdks" should exist and what such a stub file should look like... is a non-trivial question

Evidently there is demand. If it were easy someone would have done it, right?

Cost vs benefit ?

Posted Jun 8, 2026 9:14 UTC (Mon) by malmedal (subscriber, #56172) [Link]

> If it were easy someone would have done it, right?

It's easy, then glibc does a random change and you have to do the easy thing again and again and yet again. Better to use a language like go or rust where the designers actually understand the problem.

Useful tools include patchelf, chrpath and statifier. You can often, but not always do it with just giving /lib/ld-linux.so some command-line flags and environment variables.

Cost vs benefit ?

Posted Jun 7, 2026 10:34 UTC (Sun) by quotemstr (subscriber, #45331) [Link] (2 responses)

Yes, symbol versioning is terrible and unnecessary. (Function has a new contract? Give it a new name. That's how most systems work and it works fine.)

But the problem is deeper: glibc just refuses to provide a TARGET_VERSION facility of some kind. All other major operating systems let you use new headers to target old systems. Even with symbol versions, glibc could too. They just don't want to. They can't sit there any tell everyone version targeting is infeasible when the rest of the planet does it.

glibc's decades of obstinacy about version targeting has distorted the Linux ecosystem, wasted inestimable developer-years, and, ultimately, has destroyed our ability to coordinate intraprocess resources in userspace.

Thanks, libc-alpha.

Cost vs benefit ?

Posted Jun 7, 2026 22:59 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

And the worst of it, this was done for no real gain in performance or compatibility. glibc still breaks backwards compatibility from time to time, and glibc's allocator is famously middling.

Cost vs benefit ?

Posted Jun 8, 2026 1:25 UTC (Mon) by 0xilly (subscriber, #172315) [Link]

I often wonder if the kernel could create a libkernel and expose that via the VDSO and one could avoid glibc if they wanted outside of direct syscalls or musl etc…

Cost vs benefit ?

Posted Jun 8, 2026 11:46 UTC (Mon) by ballombe (subscriber, #9523) [Link] (3 responses)

> The problem is that glibc's symbol versioning is terrible. If you compile an app on a later version of glibc, you can't deploy it on an earlier version. And there's no easy way to fix this.

Sure there is... Install your newer libc in /bil64, and edit the binaries so that the .interp section says /bil64/ld-linux-x86-64.so.2 instead /lib64/ld-linux-x86-64.so.2

It is not rocket science.

Cost vs benefit ?

Posted Jun 8, 2026 18:20 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> Sure there is... Install your newer libc in /bil64, and edit the binaries so that the .interp section says /bil64/ld-linux-x86-64.so.2 instead /lib64/ld-linux-x86-64.so.2

Try that. It'll explode if you have NSS/PAM modules or any other kind of dynamically loaded code. And yes, I _did_ try that.

Cost vs benefit ?

Posted Jun 8, 2026 18:35 UTC (Mon) by farnz (subscriber, #17727) [Link] (1 responses)

Have you tried the .symver assembly directive to link against older symbol versions with a newer glibc? It should, in theory, work (albeit I haven't tested it), but it'll be painful to use because you need every object file you build to have a consistent set of imported symbols to ensure that you can dynamically link against an older glibc.

If that works, then there's room for glibc to supply a "compat" header for each older version that just sets the symbol versions to link against for every glibc signal, and for your build system to use the -include preprocessor option to force-include the right compat header in your code.

Cost vs benefit ?

Posted Jun 8, 2026 19:15 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

Symvers or linker scripts indeed work. But they are hard to integrate when you have third-party libraries with their own build systems. I used to binary-patch the executables to downgrade the versions.

It really needs to be something like a "target level" support from glibc, like in Windows/macOS, that every library respects.

Cost vs benefit ?

Posted Jun 8, 2026 12:10 UTC (Mon) by bluca (subscriber, #118303) [Link] (4 responses)

> If you compile an app on a later version of glibc, you can't deploy it on an earlier version.

Yeah that's horrible! And also if I compile an app for OSX 12, then I can't run it on Windows 11. Also if I compile it for DOS, then I can't run it on HP Unix. What a shit show, am I right?

Cost vs benefit ?

Posted Jun 8, 2026 18:31 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

Are you saying that two versions of Linux Desktop are as different as Windows and macOS?

...

Actually, you might be right.

Cost vs benefit ?

Posted Jun 8, 2026 18:34 UTC (Mon) by bluca (subscriber, #118303) [Link] (2 responses)

Yes, of course, and that's exactly how it should be, because there's no such thing as "Linux Desktop"

Cost vs benefit ?

Posted Jun 8, 2026 23:28 UTC (Mon) by mjg59 (subscriber, #23239) [Link] (1 responses)

It feels a little bit odd for someone working on a project explicitly attempting to define a consistent set of plumbing shared between distributions to not be concerned about offering a consistent layer above that.

Cost vs benefit ?

Posted Jun 9, 2026 10:28 UTC (Tue) by bluca (subscriber, #118303) [Link]

Vendoring the universe and static linking is not a sensible answer to that problem. It's a cop out with severe drawbacks that get ignored because they are "tomorrow's problems"