Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Posted Jan 17, 2022 20:50 UTC (Mon) by MattBBaker (guest, #28651)Parent article: Brian Kernighan on the origins of Unix
The next 'UNIX' will be multiple users sharing a network of machines. It feels like Plan9 took a stab at this, with the wrong abstraction at the wrong time.
Posted Jan 17, 2022 22:00 UTC (Mon)
by NYKevin (subscriber, #129325)
[Link] (33 responses)
Disclaimer: I work for Google, and while they would probably *like* me to tell you that we have already gotten to that point with GCP, I'm not 100% convinced that we're fully there yet.
Posted Jan 18, 2022 0:37 UTC (Tue)
by mathstuf (subscriber, #69389)
[Link]
Basically, the "one pet that hosts a ranch of cattle" versus some IaaS-based monstrosity that is just not worth my time for what I need.
Posted Jan 18, 2022 5:55 UTC (Tue)
by zdzichu (subscriber, #17118)
[Link] (29 responses)
Posted Jan 18, 2022 7:47 UTC (Tue)
by NYKevin (subscriber, #129325)
[Link] (28 responses)
(Sure, at the low level, Linux-based systems have mostly standardized on "everything is a frontend to cgroups," but orchestration is an entirely different story. There's no single standard way of describing a bunch of generic containers, or machines, their associated resource requirements and constraints, etc., much less a standard piece of software that will take those constraints and schedule your containers for you. As I mentioned, you *can* do this stuff with k8s, but nobody regards k8s as The One True Container Solution in the same way that Unix is The One True Operating System.)
Posted Jan 18, 2022 8:52 UTC (Tue)
by nim-nim (subscriber, #34454)
[Link]
All those new tools are proposed by organisations that do not want to contribute to existing distribution networks, but are unable to set up alternatives (because that’s hard and expensive).
When everyone will be fed up with the current mess things will consolidate violently (like they did at the end of the unix wars, with most of the companies that had refused team play on the losing side).
Posted Jan 18, 2022 15:36 UTC (Tue)
by epa (subscriber, #39769)
[Link] (7 responses)
In theory each user can have an independent set of software, sharing only the kernel; in practice it's damned awkward to run your own software stack from your home directory, and there aren't package management tools to make it easy.
In theory each user can have their own share of CPU time and I/O budget; in practice it's far too easy for one user to take more than their fair share, or to bust global limits with fork bombs and the like.
Users and permissions are themselves not per-user; there's a single global set of users and groups and it does not nest. You can have subdirectories in the file system but you can't have sub-accounts under your user account or make your own groups. It all has to go through a single global administrator. A true multi-user system would let you manage your own permissions within your space and delegate to sub-users.
Can you set up a multiuser Linux system where each user gets their own IP address, so they can run their own httpd on port 80 serving that address, and not interfere with other users? I am sure it's possible with enough hackery, but it's far from being straightforward or manageable.
Posted Jan 19, 2022 17:07 UTC (Wed)
by dgm (subscriber, #49227)
[Link] (6 responses)
Posted Jan 19, 2022 19:55 UTC (Wed)
by epa (subscriber, #39769)
[Link] (4 responses)
We have approximations such as sandboxes used by web browsers to isolate one tab from another. Again, I see those as a kind of workaround or a hazy approximation to the right answer: creating a new user on the fly for each tab, giving it its own memory and CPU budget, and then communicating via signals, Unix domain sockets, or a shared filesystem. But yeah, talk is cheap. The point I want to make is that if it's considered a solved problem, that may be because we're too used to the limitations of the current systems. Before git, software version control might have been seen as a solved problem by some.
Posted Jan 20, 2022 11:16 UTC (Thu)
by farnz (subscriber, #17727)
[Link] (2 responses)
The other thing that would be nice to solve that VMs don't is good resource utilization; a VM's memory allocation is hard to vary dynamically based on overall system load, as is its filesystem size. The only host resource that's demand-shared in a VM setup is CPU.
It would be nice if we didn't have to preallocate resources to processes, but instead could rely on the system sharing all resources fairly and reasonably for the use cases in question. That's a hard problem, but it's one that (e.g.) filesystem quotas and cgroups aim to solve for containerized systems.
Posted Jan 28, 2022 6:05 UTC (Fri)
by wahern (subscriber, #37304)
[Link] (1 responses)
The alternative in each case--VMs and cgroups--is to preallocate memory so that the sum is no more than available backing space. But the Kubernetes clusters I've worked on, for example, all overcommitted cgroups memory--the sum of all pod cgroup memory limits was greater than physical memory (and machines had no swap space, despite my protests). Use of cgroups couldn't prevent the OOM killer from unnecessarily shooting down processes when under high aggregate load; it only helped blunt edge cases where individual pods allocated much more memory than ever expected (e.g. because of a memory leak). This was especially true if a JVM was running in a pod, as for whatever reasons the Java applications people liked to run on those clusters aggressively used anonymous memory. Whenever some other team complained about their pod being OOM killed despite not going over their cgroup budget, the host invariably had a JVM running in another, unrelated pod. (This was especially problematic for the period, 2019-2020, when Linux kernels had a regression that made it *much* more likely that the OOM killer couldn't flush buffer caches faster than processes would dirty them. Worse, the result was the OOM killer spinning, sleeping, and eventually timing out in its flush routine; and often times multiple processes could end up in that routine--or end up trying to take a lock held by that routine--effectively resulting in a situation where the entire system could lock-up for an indeterminate amount of time. Even if you could get an SSH session, poke the wrong process and your session would lock-up, too.)
Posted Jan 28, 2022 14:48 UTC (Fri)
by farnz (subscriber, #17727)
[Link]
Cgroups have two things that a virtual machine setup does not have:
Now, if you're dealing with people who overcommit physical RAM but who've not read and understood In defence of swap, you're in trouble anyway - they're working on a faulty model of how the OS allocates memory to processes to begin with, and both models are going to break because they're expecting "magic" to save them. But with competent people, cgroups already does better than VMs; and because cgroups has more visibility into what the workload is doing, it's more likely that cgroups will improve over time.
As an example, EPOC32 had low memory signals it sent to the applications to let them know that the system was low on memory, and that it was out of memory. Applications were expected to handle these signals by discarding caches, performing an early GC etc - and could also (ab)use them for better performance (e.g. don't run GC until the system is low on memory). You can get similar signals in a cgroup-aware application by having the application monitor memory.events, and treating a breach of the "low" threshold as "memory is running low" and a breach of the "high" threshold as "out of memory". Then, your JVM could be aware of this, and whenever it breaches the "low" threshold (container using too much memory), it does a GC run, while when it breaches the "high" threshold, it also discards any cached JITted sections etc.
This interacts in a good way with the kernel's preference to swap stuff that breaches low more than it swaps stuff that breaches high - when you breach low, you try to come back under control, when you breach high, you try to avoid being killed.
Posted Jan 20, 2022 17:10 UTC (Thu)
by ermo (subscriber, #86690)
[Link]
If all processes used the same form of containerisation/namespacing/layering and the same means of IPC (possibly with some faster-than-TCP-but-still-reliable network transport available), this might function as seamlessly as one could ideally expect.
At least, that's the endpoint my imagination extrapolates to from your problem statement. I could be (very) wrong of course.
I don't know how close Plan 9 comes to this hypothetical scenario or whether it's even close. But perhaps Zircon/Fuchsia might very well become the closest modern approximation of what I outline?
Posted Jan 21, 2022 14:01 UTC (Fri)
by smurf (subscriber, #17840)
[Link]
Posted Jan 19, 2022 0:29 UTC (Wed)
by bartoc (guest, #124262)
[Link] (5 responses)
I've tried to understand k8s multiple times and it's like they refuse to _ever_ document what anything actually does, just some high-level abstract theory of what the results might be. It feels like reading a formal standard but more arbitrary.
Posted Jan 19, 2022 5:36 UTC (Wed)
by jkingweb (subscriber, #113039)
[Link] (4 responses)
Posted Jan 19, 2022 12:10 UTC (Wed)
by cyperpunks (subscriber, #39406)
[Link] (3 responses)
Posted Jan 19, 2022 13:56 UTC (Wed)
by farnz (subscriber, #17727)
[Link]
Technical writing is a very different discipline to software writing, and needs a different set of skills. Unfortunately, too many companies fail to value tech writing, and instead expect that the software authors will write good documentation themselves.
A really good tech writer knows how to take a developer's documentation (written from a place of deep familiarity with the code) and turn it into something useful for people unfamiliar with the code. A software developer just doesn't have that skill.
Posted Jan 19, 2022 17:40 UTC (Wed)
by mpr22 (subscriber, #60784)
[Link] (1 responses)
It sells advertising space (and possibly trademark licences?), and uses a slice of the revenue to pay the salaries of its technology division.
Posted Jan 19, 2022 17:42 UTC (Wed)
by mpr22 (subscriber, #60784)
[Link]
Posted Jan 20, 2022 22:37 UTC (Thu)
by ceplm (subscriber, #41334)
[Link] (12 responses)
I believe that Plan9 was more on the way towards this ideal, but Plan9 unfortunately died.
Posted Jan 20, 2022 22:44 UTC (Thu)
by sfeam (subscriber, #2841)
[Link] (11 responses)
Well, there were VAXClusters. Start a job on the cluster and it would migrate from machine to machine or node to node as they connected or disconnected themselves from the cluster. So far as I know unix/linux clusters never have reached this level of flexibility. Or maybe you were focusing specifically on filesystems?
Posted Jan 21, 2022 11:16 UTC (Fri)
by ceplm (subscriber, #41334)
[Link] (3 responses)
What I was thinking was that all my data are somewhere in The Cloud and whatever machine I use to access them (workstation, my hobby home laptop, tablet, or phone) I get access to the same data. Perhaps I don’t have all applications for all data (thinking that tablet or phone), then I cannot access those, but for those data which I have application for I can access the exact same data. And of course all that over the Internet, so fully secure, with authorizations and all that crap. And all that cached on the local disc, because otherwise latency will kill any user experience. It should be the normal state of everyday affairs not something like pulling your teeth slowly.
If you try something like that today you have NFS or Samba or stuff like that, which is unuseable for everyday work over even slightly non-local network (not mentioning a potential security disaster, so one switches to something even more obscure like sshfs and that’s even slower; NFSv4 may be more secure). Or you have some ultra-high-level stuff like Coda/OpenAFS etc. which requires really high-end hardware (no chance of running it on my tablet, phone could not be even mentioned) and even there I am not sure how well it works. Or you have series of mutually incompatible ad-hoc synchronization hacks (offline-IMAP, similar for CardDAV/CalDAV/etc., some weird in-browser proprietary caching for Google Docs, scripts using rsync, etc.).
http://ninetimes.cat-v.org/ seems to claim that Plan9 was supposed to be able of something like that, but I have my deepest doubts and I will believe it when I see it.
If you have some system which fulfils my requirements than you are either God or a liar, and I be on the latter.
Posted Jan 21, 2022 23:54 UTC (Fri)
by intgr (subscriber, #39733)
[Link] (2 responses)
Perkeep (previously Camlistore) is an attempt to solve this problem.
Posted Jan 22, 2022 19:28 UTC (Sat)
by ceplm (subscriber, #41334)
[Link] (1 responses)
Posted Jan 22, 2022 21:57 UTC (Sat)
by ceplm (subscriber, #41334)
[Link]
Concerning Perkeep, it looks interesting, but more as a secondary story for backup and archiving (something similar to https://github.com/ThinkUpLLC/ThinkUp ?) not as something where you actually work.
Posted Jan 21, 2022 12:01 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (6 responses)
You can do full machine migration easily either via VM checkpoint/restore or via CRIU, and this is useful in virtualization scenarios. You can also do high-availability via Xen where the same code runs on two computers at once, in case one of them goes down.
Posted Jan 21, 2022 13:58 UTC (Fri)
by malmedal (subscriber, #56172)
[Link] (5 responses)
MOSIX however, could move arbitrary processes between cluster machines.
Another option is Condor, which uses CRIU to move simple processes.
Posted Jan 21, 2022 14:01 UTC (Fri)
by corbet (editor, #1)
[Link]
Posted Jan 21, 2022 18:14 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (3 responses)
Posted Jan 21, 2022 18:36 UTC (Fri)
by malmedal (subscriber, #56172)
[Link] (2 responses)
I believe I said Beowulf couldn't but MOSIX could migrate processes, they are different things.
Posted Jan 21, 2022 18:38 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
Yes, of course you're correct. I used a Beowulf cluster with MOSIX back in the day, so that's why I'm confusing two of them.
Posted Jan 22, 2022 13:03 UTC (Sat)
by malmedal (subscriber, #56172)
[Link]
Did MOSIX ever fix that issue?
Posted Jan 18, 2022 17:23 UTC (Tue)
by kleptog (subscriber, #1183)
[Link]
Its main problem was that the implementation was plagued with bugs that made it really annoying to run in production. When its internal state got corrupted (which was often in the beginning) the only fix was to blow away the entire cluster. Which is really annoying when you've got a hundreds of containers spread over dozens of nodes. Bringing everything up at once was a guaranteed way to get more corrupted state.
It's better now, and it's has been pretty stable for a while, but the mind share has all gone to k8s. Swarm is good for smaller set-ups, but if you want to build another layer of automation on top of it, Swarm is a real pain while k8s does this easily. Silly things like secrets that can't be updated are just nails in the coffin.
I've wondered about how the world would have turned out if the Swarm UI could have been merged with a k8s reliable backend. I see there is something called Kompose, but I've never tried it.
Posted Jan 19, 2022 5:45 UTC (Wed)
by nilsmeyer (guest, #122604)
[Link]
I think as a proprietary solution it's a bit of a non-starter anyways.
Posted Jan 18, 2022 0:47 UTC (Tue)
by neilbrown (subscriber, #359)
[Link] (13 responses)
I think this missed the point of the question "Could Unix happen again?"
I think the "Unix" that may or may not happen is again is a "shunkworks" project that appears to come out of nowhere, and takes over the world.
Posted Jan 18, 2022 3:40 UTC (Tue)
by felixfix (subscriber, #242)
[Link] (11 responses)
But this depends on more bandwidth and cleverer algorithms, much more on hardware than software, so it's not a candidate for a 9000 lines of code style revolution. I think those days are long gone, just as no one tinkering in a barn can be the next Henry Ford or Wright Bros. It will be something entirely unexpected, spawning an entirely new field.
Posted Jan 18, 2022 6:16 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
Google doesn't control networking. Start a web server and off you go.
Posted Jan 18, 2022 6:42 UTC (Tue)
by felixfix (subscriber, #242)
[Link]
Posted Jan 18, 2022 7:49 UTC (Tue)
by PengZheng (subscriber, #108006)
[Link] (7 responses)
I guess the performance of that network would be terrible. Let's take bitcoin for example.
Posted Jan 18, 2022 8:29 UTC (Tue)
by felixfix (subscriber, #242)
[Link] (6 responses)
Posted Jan 18, 2022 9:32 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Posted Jan 18, 2022 11:14 UTC (Tue)
by smurf (subscriber, #17840)
[Link] (1 responses)
Please tell us why we'd need that, when we already have high-speed low-latency wired and/or wireless options.
(Physical) mesh networks work well for local IoT applications and similar low-bandwidth, noncritical-latency stuff; who cares whether the AC turns off now vs. in two seconds. But try telling a gamer that their network has >50msec to their favorite server without getting laughed off the stage.
Posted Jan 18, 2022 11:56 UTC (Tue)
by excors (subscriber, #95769)
[Link]
(See also "web3", which is marketed as decentralised but appears to be already heavily centralised in a handful of service providers: https://moxie.org/2022/01/07/web3-first-impressions.html)
Posted Jan 18, 2022 12:18 UTC (Tue)
by rbtree (guest, #129790)
[Link] (1 responses)
Posted Jan 18, 2022 12:59 UTC (Tue)
by smurf (subscriber, #17840)
[Link]
Posted Jan 18, 2022 14:56 UTC (Tue)
by felixfix (subscriber, #242)
[Link]
Posted Jan 18, 2022 14:06 UTC (Tue)
by mtaht (subscriber, #11087)
[Link]
Posted Jan 18, 2022 15:32 UTC (Tue)
by nilsmeyer (guest, #122604)
[Link]
Maybe it is happening right now? Without the benefit of hindsight who would have known that Unix or Linux would have such an impact?
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
We have either local systems (BTRFS, XFS, etc.) or network systems (NFS, Samba, etc.), but nothing in between.Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Yeah...Beowulf clusters still exist, they are just called "data centers" now...
Beowulf
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Though it could be that it just sees a different point to the one that I see - it is a rather vague question.
A key factor in the success of Unix was the lack of effective competition. The OS market today has way too much competition for a similar success story.
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
> Though it could be that it just sees a different point to the one that I see - it is a rather vague question.