Debian still having trouble with merged /usr

Posted Apr 6, 2022 8:26 UTC (Wed) by bluca (subscriber, #118303)
Parent article: Debian still having trouble with merged /usr

While being a good summary of recent events, I think the article missed the most important aspect of all this: the real issue is not about technical design (that debate settled 10 years ago, now it's just a question of the stragglers catching up), or the technical implementation (we know what we got works, Ubuntu does it and it hasn't fallen apart). The question is social, and about project management. The Debian constitution, that all project members have to follow, states that the Technical Committee makes decisions regarding the technical direction of the project, and while it cannot force individuals to work on anything, individual maintainers are not allowed to veto decisions they don't personally like and block the entire project. And yet that's exactly what's happening here.

So the important questions it would have been interesting to see explored from an outsider's perspective are, how does/can a volunteer project based on bureaucracy like Debian deal with dissent or even obstructionism? Can the project make effective decisions and implement then, whatever they might be, given its self-imposed rules and customs and the practical reality of how volunteers take care of specific parts of the projects as their own thing? Are there similar situations in other projects/distros? And so on.

Debian still having trouble with merged /usr

Posted Apr 6, 2022 9:49 UTC (Wed) by ballombe (subscriber, #9523) [Link] (3 responses)

The real issue is that the TC is not a steering committee that is empowered to decide long term project orientation on behalf of the project. This is by design.

The TC should not have decided the usrmerge issue before all the relevant patches were available. Since the TC cannot force anybody to implement anything, what the TC really decided is that bookworm should be delayed indefinitely until someone implement usrmerge. This is not sustainable.

Helmut Grone called it correctly.
If a GR was put to vote whether or not to delay bookworm for usrmerge would likely fail.

Debian still having trouble with merged /usr

Posted Apr 6, 2022 9:55 UTC (Wed) by bluca (subscriber, #118303) [Link] (2 responses)

Changes in dpkg are nice to have but not necessary, so what you are saying is incorrect, there is already an available path forward. The decision could be made today to do the same thing Ubuntu has done successfully last year, make the usrmerge pseudo-essential, and move on from there. We know this works, or Ubuntu wouldn't exist anymore.

Debian still having trouble with merged /usr

Posted Apr 6, 2022 11:00 UTC (Wed) by ballombe (subscriber, #9523) [Link] (1 responses)

But nobody offered to the TC to do this work at the time, so my point stand.
It is not the first TC decision to have been ignored because nobody was willing to do the work.

Debian still having trouble with merged /usr

Posted Apr 6, 2022 11:18 UTC (Wed) by bluca (subscriber, #118303) [Link]

That's also inaccurate, it was proposed multiple times to follow Ubuntu's example and make usrmerge pseudo-essential. It's a really trivial change to do, what's missing is the indication from the TC to do it, not the work to implement it.

Debian still having trouble with merged /usr

Posted Apr 6, 2022 10:59 UTC (Wed) by pebolle (guest, #35204) [Link]

> Are there similar situations in other projects/distros?

This is, as the article says, "Debian, being Debian".

I've come to the view that for a certain number of people things like discussions taking the better part of a decade, endless threads on countless lists, resolutions of thousands of words, voting, and what not are a feature of Debian, not a bug. Some people actually enjoy being part of that. And apparently quite a number of those people ended up running Debian.

(Clearly there are also people in Debian doing all the other stuff needed to keep Debian going. I expect that most of the people in the let's-do-mock-United-nations group are probably in the do-the-work group too.)

Debian still having trouble with merged /usr

Posted Apr 6, 2022 13:14 UTC (Wed) by mgedmin (subscriber, #34497) [Link] (9 responses)

> we know what we got works, Ubuntu does it and it hasn't fallen apart

While it hasn't fallen apart completely, there are minor buglets here and there, e.g. dpkg -S $(which $thing) fails to find the package containing some binaries.

I personally haven't noticed anything else, and I think this will be fixed once Debian drops support for non-usr-merged system and migrates all the buildbots, so the built packages no longer declare that they contain files from /bin or /lib.

Debian still having trouble with merged /usr

Posted Apr 6, 2022 14:45 UTC (Wed) by mbunkus (subscriber, #87248) [Link] (8 responses)

"dpkg -S" has never been perfect:

[0 mbunkus@reliant ~] dpkg -S $(which vim)
dpkg-query: no path found matching pattern /usr/bin/vim

/usr/bin/vim is a symlink to the alternatives system. In my case the actual binary is vim.basic, and that one is found by dpkg. You have to know about this system already, and having one more thing to know isn't really the sky falling in my opinion.

Debian still having trouble with merged /usr

Posted Apr 6, 2022 15:08 UTC (Wed) by ianmcc (subscriber, #88379) [Link] (3 responses)

That seems like it ought to be a fairly simple fix too - if the target isn't found in the database and it is a symlink, then chase the symlink and see if the target is in the database (rinse and repeat as required).

Debian still having trouble with merged /usr

Posted Apr 6, 2022 15:13 UTC (Wed) by mbunkus (subscriber, #87248) [Link] (2 responses)

True, and that it hasn't happened shows that users generally don't really care. My guess is that they'll care about "dpkg -S" not finding /bin/bash (when it's really /usr/bin/bash with /bin → /usr/bin) just as much: not really.

Debian still having trouble with merged /usr

Posted Apr 6, 2022 19:02 UTC (Wed) by sobkas (subscriber, #40809) [Link] (1 responses)

I once wrote a tool to unbreak my system after a long habit of using make install on /usr and or /.
It checked if hash of a file was identical to one in dpkg database and if not it would reinstall package.
It also deleted files that were not part of any package. Surprisingly it didn't brake my system and allowed Gnome to start again.
It was so long time ago, but then I just started to package changes that I needed instead of using make install.

Debian still having trouble with merged /usr

Posted Apr 7, 2022 3:56 UTC (Thu) by pabs (subscriber, #43278) [Link]

That sounds like a reinvention of the cruft and cruft-ng packages. IIRC they are able to explain more files than just the ones in the dpkg database, such as the *.pyc files generated at install time from python3-* package post-install scripts.

Debian still having trouble with merged /usr

Posted Apr 6, 2022 17:30 UTC (Wed) by jond (subscriber, #37669) [Link] (3 responses)

I’d like to see -S’s behaviour wrt symlinks fixed, rather than used as an excuse to ignore other bugs.

Debian still having trouble with merged /usr

Posted Apr 7, 2022 8:52 UTC (Thu) by farnz (subscriber, #17727) [Link] (2 responses)

Note that in this case, it's just a different manifestation of the same bug.

In the usrmerge case, the user has the path /bin/bash, where /bin is a dpkg-managed symlink to /usr/bin. dpkg does not handle the symlink case at all, so it is unaware that /bin/bash and /usr/bin/bash are the same location, despite having all the information it needs to do this.

In the alternatives case, /usr/bin/vim is a symlink to /usr/bin/vim.basic; again, dpkg is unaware that the symlink makes the paths the same from the user's perspective.

If you fixed dpkg to try once with the "raw" path, and then retry with the result of realpath(3) on the user input, it'd work in both cases.

Debian still having trouble with merged /usr

Posted Apr 7, 2022 10:23 UTC (Thu) by mgedmin (subscriber, #34497) [Link] (1 responses)

Not really. realpath /bin/bash produces /usr/bin/bash. dpkg -S /usr/bin/bash finds nothing, because the bash .deb ships the file as /bin/bash, and it is recorded as such in the dpkg database.

You need to call realpath on every path recorded in the dpkg database for things to be consistent, and AFAIU that's what the unmerged and outdated patch was attempting to do.

Debian still having trouble with merged /usr

Posted Apr 8, 2022 10:50 UTC (Fri) by smurf (subscriber, #17840) [Link]

Well if /usrmerge is in force it'd be mostly sufficient to blindly prefix /bin /sbin and /lib with /usr.

Yes that doesn't fix the general problem – but if humanity had always waited for a general solution to appear out of thin air, instead of actually fixing what we can fix Right Now(ish), we'd still be up in the trees somewhere.

Debian still having trouble with merged /usr

Posted Apr 6, 2022 17:27 UTC (Wed) by jond (subscriber, #37669) [Link] (8 responses)

The article does quote Russ refuting the “Ubuntu haven’t had problems” argument.

Debian still having trouble with merged /usr

Posted Apr 6, 2022 17:43 UTC (Wed) by bluca (subscriber, #118303) [Link]

Which I find very unconvincing, as stated <somewhere>. Ubuntu has plenty of downstream packages, and yet this assumed urge to move things around in just the right (or wrong?) way hasn't happened there. If anything, the current TC ban on the specific change that triggers the feared behaviour can be just kept in place until the dpkg maintainer accepts the project-wide change. It's been in place since last year and we seem to be doing fine with it. If fixing dpkg is not possible, and it doesn't look like it is for social reasons more than technical ones, it doesn't seem like a bad solution to finish the transition and move on.

Debian still having trouble with merged /usr

Posted Apr 10, 2022 5:32 UTC (Sun) by rra (subscriber, #99804) [Link] (6 responses)

Luca and I have an architectural disagreement and I'm not sure which of us is correct. It's very much an arguable point.

I assume Luca is correct that Ubuntu hasn't encountered any problems (I don't follow Ubuntu development so have no personal knowledge). The edge cases where there are failures are, well, edge cases, and pretty uncommon ones as well, although they do turn up from time to time. I think it's entirely plausible that Ubuntu just hasn't happened to encounter one of those edge cases but that Debian will in the future; Luca thinks that Ubuntu does enough original packaging including moving files between packages that if the problems were at all common, Ubuntu would have seen them, and thus Debian won't run into them in the future. It's really hard to tell which of us is correct. It's mostly guesswork, and I cede the point that Luca has more concrete data.

My philosophical position rests on a desire for a stronger correctness goal. I think a package manager should be like a file system: short of hardware failure, it should never lose data ever, period, for any reason, no matter how rare the circumstances are. Anything else is a bug that should be fixed, and treated fairly seriously. This is a very strong position. It's entirely legitimate for Luca to not agree and think I'm holding the package manager to too high of a standard.

Philosophically, I think Debian should be a project that pays attention to those sorts of details, and that we should fix this sort of bug even if the incidence is rare. But I'm also not in a position to volunteer to do the work.

Debian still having trouble with merged /usr

Posted Apr 10, 2022 9:19 UTC (Sun) by bluca (subscriber, #118303) [Link] (5 responses)

My main point is that we are blocked, upstream software is starting to plan dropping support for legacy split-usr, and there's no other clear way out. Dpkg can be fixed _eventually_ but it doesn't need to be fixed immediately. There's no urgency. And whatever fix eventually comes has to apply to an already switched system, because Ubuntu exists and cannot be broken, given it's already fully switched. This is what happens when one delays making (and implementing) decisions again and again: eventually others move ahead without you, and you aren't left with many choices.

Debian still having trouble with merged /usr

Posted Apr 11, 2022 0:18 UTC (Mon) by pabs (subscriber, #43278) [Link] (1 responses)

Do you have some examples of upstreams that are dropping support for split-usr systems?

Debian still having trouble with merged /usr

Posted Apr 11, 2022 5:58 UTC (Mon) by johannbg (guest, #65743) [Link]

It's been what more than a decade now since the initial push to clean this artifical mess up on the core/baseOS level which should be way more than enough time for downstream distributions to adapt.

There is literally no technical reason to support systems with split-usr and supporting the split-usr mode makes both code and documentation more complicated for upstreams ( it adds to the CoD for projects ).

Upstream will and should at this point in time close any such issue with wontfix and consumers of components that make up the core/baseOS layers should rather expect that split-usr mode is not supported or no longer supported and upstreams will drop this without hesitation or as much as a warning at this point. ( not that such warning would ever be visible to anyone other than the person building the software anyway like [1] )

1. https://github.com/systemd/systemd/blob/main/meson.build

Debian still having trouble with merged /usr

Posted Apr 11, 2022 2:29 UTC (Mon) by rra (subscriber, #99804) [Link] (2 responses)

The point that one cannot have problems once every system has been switched is a good one. I'm not sure that's 100% correct, but I think the only way it could be incorrect is if someone has held packages from before the /usr merge, and then later upgrades them in a way that involves files moving between packages. This makes the edge case even rarer.

Maybe this really would be a sufficiently rare situation provided that Debian was careful not to move files between / and /usr for one release after usrmerge became mandatory (assuming we do make usrmerge mandatory).

I will say that I am still uncomfortable leaving dpkg with an incorrect understanding of where files are on the file system. That seems like the sort of thing that will cause problems even if we don't know what problems they are and aren't running into them now. But I admit this is philosophical and code smell, rather than me being able to point to a specific issue other than the one we already know about when files move between packages.

Debian still having trouble with merged /usr

Posted Apr 11, 2022 7:29 UTC (Mon) by smurf (subscriber, #17840) [Link] (1 responses)

After /usrmerge, dpkg's understanding is "stuff is in /usr/* instead of /*" (assuming that the database reader is patched to adjust merged paths on the fly). Which is factually correct.

IMHO these oh-so-dangerous edge cases simply do not occur on actual systems. Files from held packages get moved to /usr by the usr-merge transition, they are where dpkg thinks they are (see above) and are reachable under both old and new paths, due to various symlinks.

Debian still having trouble with merged /usr

Posted Apr 11, 2022 17:15 UTC (Mon) by rra (subscriber, #99804) [Link]

> assuming that the database reader is patched to adjust merged paths on the fly

You're assuming exactly the thing that we're discussing, which is not done, and which the dpkg maintainer does not want to do. Yes, if the problem is solved, there isn't much of a problem.

Debian still having trouble with merged /usr

Posted Apr 7, 2022 11:53 UTC (Thu) by kleptog (subscriber, #1183) [Link] (1 responses)

I find the parallels between governance models in Linux distributions (and large software projects in general) and actual forms for governments of countries around the world fascinating. Everywhere you go you get the same problems over and over and similar solutions are proposed.

The specific pattern of having several actors with different mandates and rules that decide who gets to decide what leading to bizarre results is something that happens the world over in many different forms. It remains because the actors generally like the fiefdoms they control and refuse to give them up, even if doing so would lead to better results overall (though what is "better" is often subjective anyway).

This is just humans being humans. I don't imagine this is going to change any time soon. Debian has chosen this model and for good and for ill will have to work it out. In the mean time I guess you can just stock up on popcorn.

Debian still having trouble with merged /usr

Posted Apr 8, 2022 8:11 UTC (Fri) by marcH (subscriber, #57642) [Link]

Reading the article I couldn't help but think about Putin's (and other dictators') opinion about democracies...

I obviously do not wish to live in a dictatorship but are "democratic" ideas the best way to run software projects? I'm not sure. "Free country" and "Free software" sound like two very different things to me.

BTW there is some evidence that political systems depend on family structures:
https://www.cambridge.org/core/journals/journal-of-instit...