Soller: Real hardware breakthroughs, and focusing on rustc

Posted Dec 6, 2019 11:20 UTC (Fri) by nim-nim (subscriber, #34454)
In reply to: Soller: Real hardware breakthroughs, and focusing on rustc by Cyberax
Parent article: Soller: Real hardware breakthroughs, and focusing on rustc

>> None of this works at scale. It’s all local dev-specific workarounds
> I worked in a company that is in the top 10 of the world's companies and has the name starting with an "A" and the second letter not being "p".

When you attain this size you can afford the combination explosion. Most entities (starting with community distros) can not.

So yes it does not scale. With astronomic resources you can brute-force even an inefficient system.

Go modules respect my requirements. I should know, I spend enough months dissecting the module system. They will work as multi-version. Python does not. It's not a multi version language.

Java has not solved anything. Which is why its adoption outside businesses, is dismal. Businesses can afford to pay the not-scaling tax (in app servers, in ops, in lots of things induced by Java software engineering practices). Community distros can not.

>> Therefore, anything manageable at scale must keep semver exceptions as exceptions, not the general case.
> I don't follow. Why?

Because each exception is an additional thing that needs specific handling with the associated costs. That’s engineering 101 (in software or elsewhere).

Rules get defined to bring complexity and costs down. Exceptions exist to accommodate an imperfect reality. A working efficient system allows exceptions without making them the rule.

Soller: Real hardware breakthroughs, and focusing on rustc

Posted Dec 6, 2019 18:56 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (6 responses)

> When you attain this size you can afford the combination explosion. Most entities (starting with community distros) can not.
I maintained several projects there and spent way less time on that than maintaining a package for Ubuntu and coping with it being broken by Python updates.

> So yes it does not scale. With astronomic resources you can brute-force even an inefficient system.
There's nothing inefficient there. It allows to efficiently place the onus of evolving the dependencies on the correct people - package owners.

I.e. if you own a package AppA that uses LibB then you don't care about LibB's rdepends. You just use whatever version of LibB that you need. If you need a specific older or newer version of LibB then you can just maintain it for your own project, without affecting tons of other projects.

This approach scales wonderfully, compared to legacy distro packages. Heck, even Debian right now has just around 60000 source packages and is showing scaling problems. The company I worked at had many times more than that, adding new ones all the time.

> Because each exception is an additional thing that needs specific handling with the associated costs. That’s engineering 101 (in software or elsewhere).
What is "semver exceptions" then?

> Exceptions exist to accommodate an imperfect reality. A working efficient system allows exceptions without making them the rule.
What are "exceptions" in the system I'm describing?

Soller: Real hardware breakthroughs, and focusing on rustc

Posted Dec 6, 2019 19:22 UTC (Fri) by nim-nim (subscriber, #34454) [Link] (4 responses)

In your approach everything is a special case with a special dep list.

It "scales" because you do not maintain anything, you just push code blindly.

Distributions do not only push code, they fix things. When you fix things, keeping the amount of things to be fixed manageable matters.

If you don’t feel upstream code needs any fixing, then I believe you don’t need distributions at all. Just run your own Linux from scratch and be done with it.

Please report to us how much you time you saved with this approach.

Interested? I thought not. It‘s easy to claim distributions are inefficient, when adding things at the fringe. Just replace the core not the fringe if you feel that’s so easy. You won’t be the first one to try.

Soller: Real hardware breakthroughs, and focusing on rustc

Posted Dec 6, 2019 19:56 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

> In your approach everything is a special case with a special dep list.
Correct.

> It "scales" because you do not maintain anything, you just push code blindly.
Incorrect. In that companies libraries are maintained by their owner teams. The difference is that they typically maintain a handful of versions at the same time, so that all their dependants can build it.

There was also a mechanism to deprecate versions to nudge other teams away from aging code, recommendation mechanism, etc.

> Distributions do not only push code, they fix things. When you fix things, keeping the amount of things to be fixed manageable matters.
In my experience, they mostly break things by updating stuff willy-nilly without considering downstream developers.

> If you don’t feel upstream code needs any fixing, then I believe you don’t need distributions at all. Just run your own Linux from scratch and be done with it.
That's pretty much what we'll be doing eventually. The plan is to move everything to containers running on CoreOS (not quite LFS, but close enough).

Soller: Real hardware breakthroughs, and focusing on rustc

Posted Dec 7, 2019 8:58 UTC (Sat) by nim-nim (subscriber, #34454) [Link] (2 responses)

> Incorrect. In that companies libraries are maintained by their owner teams. The difference is that they typically maintain a handful of versions at the same time, so that all their dependants can build it

And that’s exactly what major distributions do, when the language tooling makes it possible without inventing custom layouts like Nix does.

Upstreams do not like distribution custom layouts. The backlash over Debian or Fedora relayouting Python unnilateraly, would be way worse, than the lack of parallel instability in the upstream Python default layout.

> Incorrect. In that companies libraries are maintained by their owner teams.

It’s awfully nice when you can order devs to use the specific versions maintained by a specific owner team.

Of course, most of your complaint is that you *do* not* *want* to use the specific versions maintained by the distro teams.

So, it’s not a technical problem. It’s a social problem.

Soller: Real hardware breakthroughs, and focusing on rustc

Posted Dec 7, 2019 9:05 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> And that’s exactly what major distributions do
Except that it's impossible to use it, because it's not possible to parallel install libraries.

> when the language tooling makes it possible without inventing custom layouts like Nix does.
Then this tooling needs to be implemented for other languages. As I said, I've seen it done at the largest possible scale. It doesn't even require a lot of changes, really.

> Of course, most of your complaint is that you *do* not* *want* to use the specific versions maintained by the distro teams.
Incorrect again. I would love to see a distro-maintained repository with vetted package versions, with changes that are code-reviewed by distro maintainers.

It's just that right now these kinds of repos are useless, because they move really slowly for a variety of reasons. The main one is the necessity to upgrade all versions in the distribution in lockstep.

Soller: Real hardware breakthroughs, and focusing on rustc

Posted Dec 7, 2019 9:47 UTC (Sat) by nim-nim (subscriber, #34454) [Link]

>> when the language tooling makes it possible without inventing custom layouts like Nix does.

> Then this tooling needs to be implemented for other languages. As I said, I've seen it done at the largest possible scale. It doesn't even require a lot of changes, really.

Then don’t complain at distros, write a PEP, make upstream python adopt a parallel-version layout.

Major community distributions will apply the decisions of major language upstreams, it’s that simple. Major community distributions collaborate with major language upstreams. Collaborating implies respecting upstream layout choices.

In a company, you can sit on upstream decisions and do whatever you want (as long as someone finds enough money to fund your fork). That’s ultimately completely inefficient and counter productive, but humans do not like to bow to the decisions of others, so, that’s been done countless times and will continue to be done countless times.

Soller: Real hardware breakthroughs, and focusing on rustc

Posted Dec 7, 2019 10:00 UTC (Sat) by nim-nim (subscriber, #34454) [Link]

>> When you attain this size you can afford the combination explosion. Most entities (starting with community distros) can not.

> I maintained several projects there and spent way less time on that than maintaining a package for Ubuntu and coping with it being broken by Python updates.

That "works" because you do not care about the result being useful to others. But wasn’t your original complaint, that the python3-theano maintainers didn’t care that their package was not useful to your commercial app?

So you want a system that relies, on not caring for others, to scale, to be adopted by distributions, because the distributions should care about you?

Don’t you see the logical fallacy in the argument?

Al the things that you found too much work in the Ubuntu package, exist so the result can be used by others.