Brian Kernighan on the origins of Unix
He began by introducing Bell Labs, which was formed by US telecommunications giant AT&T to carry out research on how to improve telephone services. A lot of inventions came out of Bell Labs, including the transistor, the laser, and fiber optics. Such was the concentration of talent there that, at one point, Claude Shannon and Richard Hamming shared an office. Kernighan joined Bell Labs in 1967, when there were about 25 people engaged in computer-science research.
Early on, Bell Labs joined up with MIT and General Electric to work on a time-sharing operating system known as Multics. As one might have predicted, the attempted collaboration between a research lab, a university, and a profit-making company did not work all that well; Multics slipped later and later, and Bell Labs eventually pulled out of the project. That left two researchers who had been working on Multics — Ken Thompson and Dennis Ritchie — without a project to work on.
After searching for a machine to work on,
Thompson eventually found an old PDP-7, which was already obsolete at that
time, to do some work on filesystem design. The first Unix-like system
was, in essence, a test harness to measure filesystem throughput. But he
and Ritchie later concluded that it was something close to the sort of
timesharing system they had been trying to build before. This system
helped them to convince the lab to buy them a PDP-11/20 for further development.
![[Brian Kernighan]](https://static.lwn.net/images/conf/2022/lca/BrianKernighan-sm.png "Brian Kernighan")
The initial plan was to create a system for document processing, with an
initial focus of, inevitably, preparing patent applications. The result
was "recognizably Unix" and was used to get real work done.
The first Unix versions were entirely done in assembly, but the Multics project had shown that higher-level languages could be used for operating-system development. Starting with the BCPL language used there, Thompson pared it down to a language called B; Ritchie then added niceties like a type system, creating C. The C language hit a "sweet spot" in language design that has proved hard for anybody else to hit since, Kernighan said.
The advent of C brought the development of a wide range of system tools, perhaps the most important of which was Steve Johnson's portable C compiler, which made the operating system itself portable. Around this time pipes were added as well. That was the genesis of one of the core Unix ideas: write small tools that can be combined to solve complicated problems.
Sixth-edition Unix, released in 1975, was the first widely used version of the operating system. It included a number of other core Unix concepts, including the hierarchical filesystem, devices represented as files, a programmable shell, regular expressions, and more. All of this was implemented in about 9,000 lines of C code. The system was small and comprehensible, which led to a lot of interesting things, including the highly influential Lions' Commentary on Unix.
The 1980s were the golden age of Unix, Kernighan said. Unix was everywhere and widely respected. Thompson and Ritchie were given the Turing Award for their work. The absolute peak, he said, was when Unix appeared in the film Jurassic Park. But then the fight between AT&T and Berkeley began, leading to a pointless lawsuit, and the beginning of the fragmentation of the system.
In 1991, Linus Torvalds announced his work on Linux, and "the rest is history".
Kernighan moved into his conclusion by asking what the technical legacy of Unix is. Some significant pieces of that legacy, including the hierarchical filesystem, use of high-level languages, and the programmable shell, have their origins in Multics. Others, including pipes, the whole tools concept, and regular expressions, are a direct result of the Unix work. And most importantly, he said, was that Unix brought a new philosophy on how to create software.
Almost all of this was in place by 1975 and, he said, it may well be true that there have been no great insights into operating-system design since. Certainly Unix has seen a lot of additions, including networking, multiprocessor support, graphical interfaces, Unicode, and more. But it's all built on the foundation created nearly 50 years ago.
The creation of Unix was the result of an accidental combination of factors, starting with the juxtaposition of two exceptionally creative people "with good taste". Kernighan gave a lot of credit to "benign management" at Bell Labs that allowed this work to go forward, naming Doug McIlroy in particular. It was also spurred by the arrival of cheap hardware (in which category he included the $50,000 PDP-11 used to develop the system early on). But a key part was the Bell Labs working environment, which included stable funding and a long-term view, which is something that is hard to find today.
Could something like Unix happen again? There are plenty of talented people around now, he said, and good management does exist. Hardware has never been cheaper, and most of the software is free. But, he said, good environments for the creation of this kind of work are hard to find. Even so, he concluded, think of all the great things that began with one or two people and a good idea; that can certainly happen again. Ritchie once said that Unix was created to provide a community where interesting things could happen; we should try to create such a community again.
In your editor's opinion, Kernighan missed an opportunity to evaluate the free-software community in these terms. Companies may not have long time horizons, but many free-software projects do. It is, still, a place where people with good ideas can come together and see where those ideas lead. It would have been good to hear his thoughts on whether the free-software community has become that place where interesting things can happen and, if not, what we should seek to change to get there.
[The video of this
talk is available on YouTube.]
| Index entries for this article | |
|---|---|
| Conference | linux.conf.au/2022 |
Posted Jan 17, 2022 16:23 UTC (Mon)
by NightMonkey (subscriber, #23051)
[Link] (1 responses)
I agree. Hmm.... could our coach-class-loving Editor perhaps reach out to Mr. Kernighan for a guest article? :) Or even a video interview of Mr. Kernighan?
Thank you for a very nice summary.
Posted Jan 18, 2022 16:20 UTC (Tue)
by ballombe (subscriber, #9523)
[Link]
Creating a new operating system paradigm from scratch that is not a UNIX copycat require a different mindset.
Posted Jan 17, 2022 18:51 UTC (Mon)
by logang (subscriber, #127618)
[Link] (1 responses)
I agree that there's plenty of talent and business incentives to do it again, but I think the unmentioned missing factor is complexity. These days, there are so many open source projects that have tens of millions of lines of code, it's hard to believe that two people could sit down and in a short time come up with something that can compete with any of it. It's hard to even fathom an operating system that is only 9000 lines of code in today's world.
Today's talented engineers and developers all have to make do with incremental improvement projects. Though, that's probably been true for most of history and such improvements often are less known and less appreciated than they deserve to be. Planet Money[1] recently made the point that small incremental improvements to the humble nail eventually radically changed the way construction was done throughout the world.
[1] https://www.npr.org/2022/01/12/1072557499/what-nails-can-...
Posted Jan 17, 2022 23:38 UTC (Mon)
by mtaht (subscriber, #11087)
[Link]
Posted Jan 17, 2022 19:21 UTC (Mon)
by davidbody (subscriber, #123829)
[Link] (2 responses)
Kernighan has also published a book "Unix: A History and Memoir" which goes into all of this in more detail and includes lots of interesting photos. It's an interesting read.
Posted Jan 19, 2022 12:01 UTC (Wed)
by cyperpunks (subscriber, #39406)
[Link]
Posted Jan 21, 2022 15:24 UTC (Fri)
by gwolf (subscriber, #14632)
[Link]
Posted Jan 17, 2022 20:50 UTC (Mon)
by MattBBaker (guest, #28651)
[Link] (48 responses)
Posted Jan 17, 2022 22:00 UTC (Mon)
by NYKevin (subscriber, #129325)
[Link] (33 responses)
Disclaimer: I work for Google, and while they would probably *like* me to tell you that we have already gotten to that point with GCP, I'm not 100% convinced that we're fully there yet.
Posted Jan 18, 2022 0:37 UTC (Tue)
by mathstuf (subscriber, #69389)
[Link]
Basically, the "one pet that hosts a ranch of cattle" versus some IaaS-based monstrosity that is just not worth my time for what I need.
Posted Jan 18, 2022 5:55 UTC (Tue)
by zdzichu (subscriber, #17118)
[Link] (29 responses)
Posted Jan 18, 2022 7:47 UTC (Tue)
by NYKevin (subscriber, #129325)
[Link] (28 responses)
(Sure, at the low level, Linux-based systems have mostly standardized on "everything is a frontend to cgroups," but orchestration is an entirely different story. There's no single standard way of describing a bunch of generic containers, or machines, their associated resource requirements and constraints, etc., much less a standard piece of software that will take those constraints and schedule your containers for you. As I mentioned, you *can* do this stuff with k8s, but nobody regards k8s as The One True Container Solution in the same way that Unix is The One True Operating System.)
Posted Jan 18, 2022 8:52 UTC (Tue)
by nim-nim (subscriber, #34454)
[Link]
All those new tools are proposed by organisations that do not want to contribute to existing distribution networks, but are unable to set up alternatives (because that’s hard and expensive).
When everyone will be fed up with the current mess things will consolidate violently (like they did at the end of the unix wars, with most of the companies that had refused team play on the losing side).
Posted Jan 18, 2022 15:36 UTC (Tue)
by epa (subscriber, #39769)
[Link] (7 responses)
In theory each user can have an independent set of software, sharing only the kernel; in practice it's damned awkward to run your own software stack from your home directory, and there aren't package management tools to make it easy.
In theory each user can have their own share of CPU time and I/O budget; in practice it's far too easy for one user to take more than their fair share, or to bust global limits with fork bombs and the like.
Users and permissions are themselves not per-user; there's a single global set of users and groups and it does not nest. You can have subdirectories in the file system but you can't have sub-accounts under your user account or make your own groups. It all has to go through a single global administrator. A true multi-user system would let you manage your own permissions within your space and delegate to sub-users.
Can you set up a multiuser Linux system where each user gets their own IP address, so they can run their own httpd on port 80 serving that address, and not interfere with other users? I am sure it's possible with enough hackery, but it's far from being straightforward or manageable.
Posted Jan 19, 2022 17:07 UTC (Wed)
by dgm (subscriber, #49227)
[Link] (6 responses)
Posted Jan 19, 2022 19:55 UTC (Wed)
by epa (subscriber, #39769)
[Link] (4 responses)
We have approximations such as sandboxes used by web browsers to isolate one tab from another. Again, I see those as a kind of workaround or a hazy approximation to the right answer: creating a new user on the fly for each tab, giving it its own memory and CPU budget, and then communicating via signals, Unix domain sockets, or a shared filesystem. But yeah, talk is cheap. The point I want to make is that if it's considered a solved problem, that may be because we're too used to the limitations of the current systems. Before git, software version control might have been seen as a solved problem by some.
Posted Jan 20, 2022 11:16 UTC (Thu)
by farnz (subscriber, #17727)
[Link] (2 responses)
The other thing that would be nice to solve that VMs don't is good resource utilization; a VM's memory allocation is hard to vary dynamically based on overall system load, as is its filesystem size. The only host resource that's demand-shared in a VM setup is CPU.
It would be nice if we didn't have to preallocate resources to processes, but instead could rely on the system sharing all resources fairly and reasonably for the use cases in question. That's a hard problem, but it's one that (e.g.) filesystem quotas and cgroups aim to solve for containerized systems.
Posted Jan 28, 2022 6:05 UTC (Fri)
by wahern (subscriber, #37304)
[Link] (1 responses)
The alternative in each case--VMs and cgroups--is to preallocate memory so that the sum is no more than available backing space. But the Kubernetes clusters I've worked on, for example, all overcommitted cgroups memory--the sum of all pod cgroup memory limits was greater than physical memory (and machines had no swap space, despite my protests). Use of cgroups couldn't prevent the OOM killer from unnecessarily shooting down processes when under high aggregate load; it only helped blunt edge cases where individual pods allocated much more memory than ever expected (e.g. because of a memory leak). This was especially true if a JVM was running in a pod, as for whatever reasons the Java applications people liked to run on those clusters aggressively used anonymous memory. Whenever some other team complained about their pod being OOM killed despite not going over their cgroup budget, the host invariably had a JVM running in another, unrelated pod. (This was especially problematic for the period, 2019-2020, when Linux kernels had a regression that made it *much* more likely that the OOM killer couldn't flush buffer caches faster than processes would dirty them. Worse, the result was the OOM killer spinning, sleeping, and eventually timing out in its flush routine; and often times multiple processes could end up in that routine--or end up trying to take a lock held by that routine--effectively resulting in a situation where the entire system could lock-up for an indeterminate amount of time. Even if you could get an SSH session, poke the wrong process and your session would lock-up, too.)
Posted Jan 28, 2022 14:48 UTC (Fri)
by farnz (subscriber, #17727)
[Link]
Cgroups have two things that a virtual machine setup does not have:
Now, if you're dealing with people who overcommit physical RAM but who've not read and understood In defence of swap, you're in trouble anyway - they're working on a faulty model of how the OS allocates memory to processes to begin with, and both models are going to break because they're expecting "magic" to save them. But with competent people, cgroups already does better than VMs; and because cgroups has more visibility into what the workload is doing, it's more likely that cgroups will improve over time.
As an example, EPOC32 had low memory signals it sent to the applications to let them know that the system was low on memory, and that it was out of memory. Applications were expected to handle these signals by discarding caches, performing an early GC etc - and could also (ab)use them for better performance (e.g. don't run GC until the system is low on memory). You can get similar signals in a cgroup-aware application by having the application monitor memory.events, and treating a breach of the "low" threshold as "memory is running low" and a breach of the "high" threshold as "out of memory". Then, your JVM could be aware of this, and whenever it breaches the "low" threshold (container using too much memory), it does a GC run, while when it breaches the "high" threshold, it also discards any cached JITted sections etc.
This interacts in a good way with the kernel's preference to swap stuff that breaches low more than it swaps stuff that breaches high - when you breach low, you try to come back under control, when you breach high, you try to avoid being killed.
Posted Jan 20, 2022 17:10 UTC (Thu)
by ermo (subscriber, #86690)
[Link]
If all processes used the same form of containerisation/namespacing/layering and the same means of IPC (possibly with some faster-than-TCP-but-still-reliable network transport available), this might function as seamlessly as one could ideally expect.
At least, that's the endpoint my imagination extrapolates to from your problem statement. I could be (very) wrong of course.
I don't know how close Plan 9 comes to this hypothetical scenario or whether it's even close. But perhaps Zircon/Fuchsia might very well become the closest modern approximation of what I outline?
Posted Jan 21, 2022 14:01 UTC (Fri)
by smurf (subscriber, #17840)
[Link]
Posted Jan 19, 2022 0:29 UTC (Wed)
by bartoc (guest, #124262)
[Link] (5 responses)
I've tried to understand k8s multiple times and it's like they refuse to _ever_ document what anything actually does, just some high-level abstract theory of what the results might be. It feels like reading a formal standard but more arbitrary.
Posted Jan 19, 2022 5:36 UTC (Wed)
by jkingweb (subscriber, #113039)
[Link] (4 responses)
Posted Jan 19, 2022 12:10 UTC (Wed)
by cyperpunks (subscriber, #39406)
[Link] (3 responses)
Posted Jan 19, 2022 13:56 UTC (Wed)
by farnz (subscriber, #17727)
[Link]
Technical writing is a very different discipline to software writing, and needs a different set of skills. Unfortunately, too many companies fail to value tech writing, and instead expect that the software authors will write good documentation themselves.
A really good tech writer knows how to take a developer's documentation (written from a place of deep familiarity with the code) and turn it into something useful for people unfamiliar with the code. A software developer just doesn't have that skill.
Posted Jan 19, 2022 17:40 UTC (Wed)
by mpr22 (subscriber, #60784)
[Link] (1 responses)
It sells advertising space (and possibly trademark licences?), and uses a slice of the revenue to pay the salaries of its technology division.
Posted Jan 19, 2022 17:42 UTC (Wed)
by mpr22 (subscriber, #60784)
[Link]
Posted Jan 20, 2022 22:37 UTC (Thu)
by ceplm (subscriber, #41334)
[Link] (12 responses)
I believe that Plan9 was more on the way towards this ideal, but Plan9 unfortunately died.
Posted Jan 20, 2022 22:44 UTC (Thu)
by sfeam (subscriber, #2841)
[Link] (11 responses)
Well, there were VAXClusters. Start a job on the cluster and it would migrate from machine to machine or node to node as they connected or disconnected themselves from the cluster. So far as I know unix/linux clusters never have reached this level of flexibility. Or maybe you were focusing specifically on filesystems?
Posted Jan 21, 2022 11:16 UTC (Fri)
by ceplm (subscriber, #41334)
[Link] (3 responses)
What I was thinking was that all my data are somewhere in The Cloud and whatever machine I use to access them (workstation, my hobby home laptop, tablet, or phone) I get access to the same data. Perhaps I don’t have all applications for all data (thinking that tablet or phone), then I cannot access those, but for those data which I have application for I can access the exact same data. And of course all that over the Internet, so fully secure, with authorizations and all that crap. And all that cached on the local disc, because otherwise latency will kill any user experience. It should be the normal state of everyday affairs not something like pulling your teeth slowly.
If you try something like that today you have NFS or Samba or stuff like that, which is unuseable for everyday work over even slightly non-local network (not mentioning a potential security disaster, so one switches to something even more obscure like sshfs and that’s even slower; NFSv4 may be more secure). Or you have some ultra-high-level stuff like Coda/OpenAFS etc. which requires really high-end hardware (no chance of running it on my tablet, phone could not be even mentioned) and even there I am not sure how well it works. Or you have series of mutually incompatible ad-hoc synchronization hacks (offline-IMAP, similar for CardDAV/CalDAV/etc., some weird in-browser proprietary caching for Google Docs, scripts using rsync, etc.).
http://ninetimes.cat-v.org/ seems to claim that Plan9 was supposed to be able of something like that, but I have my deepest doubts and I will believe it when I see it.
If you have some system which fulfils my requirements than you are either God or a liar, and I be on the latter.
Posted Jan 21, 2022 23:54 UTC (Fri)
by intgr (subscriber, #39733)
[Link] (2 responses)
Perkeep (previously Camlistore) is an attempt to solve this problem.
Posted Jan 22, 2022 19:28 UTC (Sat)
by ceplm (subscriber, #41334)
[Link] (1 responses)
Posted Jan 22, 2022 21:57 UTC (Sat)
by ceplm (subscriber, #41334)
[Link]
Concerning Perkeep, it looks interesting, but more as a secondary story for backup and archiving (something similar to https://github.com/ThinkUpLLC/ThinkUp ?) not as something where you actually work.
Posted Jan 21, 2022 12:01 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (6 responses)
You can do full machine migration easily either via VM checkpoint/restore or via CRIU, and this is useful in virtualization scenarios. You can also do high-availability via Xen where the same code runs on two computers at once, in case one of them goes down.
Posted Jan 21, 2022 13:58 UTC (Fri)
by malmedal (subscriber, #56172)
[Link] (5 responses)
MOSIX however, could move arbitrary processes between cluster machines.
Another option is Condor, which uses CRIU to move simple processes.
Posted Jan 21, 2022 14:01 UTC (Fri)
by corbet (editor, #1)
[Link]
Posted Jan 21, 2022 18:14 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (3 responses)
Posted Jan 21, 2022 18:36 UTC (Fri)
by malmedal (subscriber, #56172)
[Link] (2 responses)
I believe I said Beowulf couldn't but MOSIX could migrate processes, they are different things.
Posted Jan 21, 2022 18:38 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
Yes, of course you're correct. I used a Beowulf cluster with MOSIX back in the day, so that's why I'm confusing two of them.
Posted Jan 22, 2022 13:03 UTC (Sat)
by malmedal (subscriber, #56172)
[Link]
Did MOSIX ever fix that issue?
Posted Jan 18, 2022 17:23 UTC (Tue)
by kleptog (subscriber, #1183)
[Link]
Its main problem was that the implementation was plagued with bugs that made it really annoying to run in production. When its internal state got corrupted (which was often in the beginning) the only fix was to blow away the entire cluster. Which is really annoying when you've got a hundreds of containers spread over dozens of nodes. Bringing everything up at once was a guaranteed way to get more corrupted state.
It's better now, and it's has been pretty stable for a while, but the mind share has all gone to k8s. Swarm is good for smaller set-ups, but if you want to build another layer of automation on top of it, Swarm is a real pain while k8s does this easily. Silly things like secrets that can't be updated are just nails in the coffin.
I've wondered about how the world would have turned out if the Swarm UI could have been merged with a k8s reliable backend. I see there is something called Kompose, but I've never tried it.
Posted Jan 19, 2022 5:45 UTC (Wed)
by nilsmeyer (guest, #122604)
[Link]
I think as a proprietary solution it's a bit of a non-starter anyways.
Posted Jan 18, 2022 0:47 UTC (Tue)
by neilbrown (subscriber, #359)
[Link] (13 responses)
I think this missed the point of the question "Could Unix happen again?"
I think the "Unix" that may or may not happen is again is a "shunkworks" project that appears to come out of nowhere, and takes over the world.
Posted Jan 18, 2022 3:40 UTC (Tue)
by felixfix (subscriber, #242)
[Link] (11 responses)
But this depends on more bandwidth and cleverer algorithms, much more on hardware than software, so it's not a candidate for a 9000 lines of code style revolution. I think those days are long gone, just as no one tinkering in a barn can be the next Henry Ford or Wright Bros. It will be something entirely unexpected, spawning an entirely new field.
Posted Jan 18, 2022 6:16 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
Google doesn't control networking. Start a web server and off you go.
Posted Jan 18, 2022 6:42 UTC (Tue)
by felixfix (subscriber, #242)
[Link]
Posted Jan 18, 2022 7:49 UTC (Tue)
by PengZheng (subscriber, #108006)
[Link] (7 responses)
I guess the performance of that network would be terrible. Let's take bitcoin for example.
Posted Jan 18, 2022 8:29 UTC (Tue)
by felixfix (subscriber, #242)
[Link] (6 responses)
Posted Jan 18, 2022 9:32 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Posted Jan 18, 2022 11:14 UTC (Tue)
by smurf (subscriber, #17840)
[Link] (1 responses)
Please tell us why we'd need that, when we already have high-speed low-latency wired and/or wireless options.
(Physical) mesh networks work well for local IoT applications and similar low-bandwidth, noncritical-latency stuff; who cares whether the AC turns off now vs. in two seconds. But try telling a gamer that their network has >50msec to their favorite server without getting laughed off the stage.
Posted Jan 18, 2022 11:56 UTC (Tue)
by excors (subscriber, #95769)
[Link]
(See also "web3", which is marketed as decentralised but appears to be already heavily centralised in a handful of service providers: https://moxie.org/2022/01/07/web3-first-impressions.html)
Posted Jan 18, 2022 12:18 UTC (Tue)
by rbtree (guest, #129790)
[Link] (1 responses)
Posted Jan 18, 2022 12:59 UTC (Tue)
by smurf (subscriber, #17840)
[Link]
Posted Jan 18, 2022 14:56 UTC (Tue)
by felixfix (subscriber, #242)
[Link]
Posted Jan 18, 2022 14:06 UTC (Tue)
by mtaht (subscriber, #11087)
[Link]
Posted Jan 18, 2022 15:32 UTC (Tue)
by nilsmeyer (guest, #122604)
[Link]
Maybe it is happening right now? Without the benefit of hindsight who would have known that Unix or Linux would have such an impact?
Posted Jan 18, 2022 5:19 UTC (Tue)
by alison (subscriber, #63752)
[Link] (2 responses)
Anyone reading the article will enjoy a recent amusing and intriguing piece about Dennis Ritchie's PhD thesis on functional programming: https://computerhistory.org/blog/discovering-dennis-ritch...
Posted Jan 18, 2022 10:11 UTC (Tue)
by larkey (guest, #104463)
[Link]
Posted Jan 18, 2022 13:22 UTC (Tue)
by ema (subscriber, #17750)
[Link]
Posted Jan 18, 2022 15:10 UTC (Tue)
by karim (subscriber, #114)
[Link] (4 responses)
I actually disagree with this. And I hesitate to even write this as a comment because I will likely not want to defend this point of view in response do to anyone -- 1) just not interested in a long-winded debate, 2) this is a very quick post that I haven't taken the time to think through completely, and the rough argument I make likely has many problems. So I'm going to drop this here, and come what may.
For having been involved in OSS since the mid/late-90s it's been my perception that the FOSS community is very good at _commoditizing_ existing technology and very bad at creating new tech. Think of the Linux phones before Android and the "Linux" phones since. Linux now runs on the vast majority of smartphones on the planet, but it's not because of the FOSS community, at least not directly. Here's Andy Rubin's (Android creator) take on OSS (https://techcrunch.com/2011/05/10/andy-rubin-on-androids-...): “Open source is different from a community-driven project. We’re light on community, but everything we do ends up in an open source repository." For him, open source wasn't an enabler, just a by-product.
Similar story with ChromeOS.
Similar story with gcc vs. llvm. In fact the entire GNU project was a commoditization of Unix tools/userspace. And, yet, when evolution came knocking to the compiler world, the GNU project couldn't handle it and llvm has since taken a lot more space.
etc.
Another angle to look at is how many FOSS projects/contributors have struggled with funding. This is something that LWN has covered multiple times. And it's been aptly immortalized by XKCD: https://xkcd.com/2347/ How can you innovate if you can't even pay the bills?
So, apologies, I'm skeptical that FOSS communities are inherently capable of sustaining long-term disruptive innovation. They can however certainly help sustain it by commoditizing the required building blocks that have already been shown to work elsewhere.
Posted Jan 18, 2022 18:36 UTC (Tue)
by mfuzzey (subscriber, #57966)
[Link] (1 responses)
However I think there are other factors at play in your examples.
Android: hardware was a huge factor too. The Android OS would be useless with appropriate hardware and that is still difficult / impossible to fully open source and even harder to develop in a community manner.
ChromeOS: hardware too (though you can run it on a PC so not entirely). But also the chromeOS web centric model relies on huge Google data centres that aren't really reproducible outside of huge companies.
GCC / LLVM: I think this was more a question of the incumbrent being less nimble. GCC, as the dominant OSS compiler saw less need to innovate and, because they had a large existing code base doing so was harder. This happens in the proprietary world too where innovation is often more likely to occur from a new entrant than a large existing player.
But I think where the open source community has really suceeded is in the development model rather than particular technologies.
Sure source availibility isn't everything, and isn't the same thing as community (the point Andy Rubin makes) but, when the source is available with no community it is still possible for a community to form later (by forking if needed). To some extent this happened, albeit temporarilly, with Android with early "community" Android forks like Cyanogen actually bringing significant new features that ended up becoming available in official Android only later (stuff like tethering and permissions modifications come to mind).
Posted Jan 19, 2022 0:37 UTC (Wed)
by bartoc (guest, #124262)
[Link]
Interestingly this happened with iPhones too! Despite most of their software not being open source. All those quick control popouts and a lot of the additional interaction options for stuff like notifications were added by third party mods for "jailbroken" iphones long before apple implemented them. I'm not sure if this is Android/Apple taking ideas from the "community" or just obviously good ideas being implemented by the community first, because a hobbyist doesn't need to polish things up that much before shipping (or go through endless UI redesigns, UX research, etc, etc).
Posted Jan 20, 2022 9:37 UTC (Thu)
by gasche (subscriber, #74946)
[Link] (1 responses)
- Git made distributed version-control widely available and changed our development practices in a radical way.
Many of those projects were born in academia (a natural place to look for radical ideas), but they were also open source projects from the start, and in many cases the free software community contributed greatly to the fact that they became successful enough to spread their ideas to the rest of the software world.
For Git one could argue that this is a commoditization of the proprietary system BitKeeper; but there were also open source ancestors (GNU Arch) and competitors (darcs, Mercurial) that were also influential.
One important area of innovation that is missed here is machine learning, that saw relatively little involving from well-identified open source communities, possibly as the requirements for entering the space (having a *lot* of training data at hand) made distributed development difficult. There are now striving open source projects within the machine learning software ecosystem, but I wouldn't say that the field would be substantially different without the open source community (by which I mean: without large-scale cooperation of hobbyists / benevolent contributors).
Posted Jan 20, 2022 16:55 UTC (Thu)
by karim (subscriber, #114)
[Link]
I somewhat feel git might actually a good example of commoditization. I followed this very much in "real-time" as the BitKeeper drama unfolded and was at OLS in 2005 when Matt Mackall first presented Mercurial -- I loved mercurial btw, and I was sad to see git prevail, but I digress. It was my understanding that BitKeeper was tailor-built by Larry McVoy for Linux development based on direct conversations between Linus and Larry. I don't have a reference for this, so I could be misremembering, but that was my understanding. As such, when the history section of git on Wikipedia (https://en.wikipedia.org/wiki/Git#History) states: "Support a distributed, BitKeeper-like workflow." as one of the goals set out by Linus for git, one can probably easily overlook the work that Larry presumably did to actually and actively listen to Linus and come up with something that fit "the customer's needs". [side bar: I've butted heads with Larry several times in the early 2000s on unrelated topics, but credit should go where it's deserved.]
And maybe this is what it boils down to. "Listening to the customer's needs" and "creating a product the customer will actually use" have nothing to do with the ability of replicating in an open source manor the end result of what those customer-facing steps entail. In fact, I'd venture to say that the open source community generally has had a bad history of being able to listen to customer needs. It's been, on other hand, very effective at replicating what those that have have produced, albeit sometimes in a more sustainable fashion ... because direct monetization wasn't the end goal or even possibly a need.
Again: 1) very rough ideas/arguments off the top of my head, 2) I care not of being right nor making a point. i.e. destroy/demolish at will.
Posted Jan 20, 2022 2:49 UTC (Thu)
by ikitayama (subscriber, #51589)
[Link]
Posted Jan 29, 2022 15:12 UTC (Sat)
by scientes (guest, #83068)
[Link]
Absolutely. I got really excited about Zig-lang's goals to replace C, while still being Turing complete, but it has IMHO since abandoned that goal.
The features of Zig that could feasibly be ported to C (with a great deal of work) would be--
1. Remove the need of C pre-processor, which is a language of its own (the original Go compiler written in C avoided the C pre-processor, but Zig does it better).
2. Better memory allocation. malloc and free are a commonly slow in C applications, leading to Firefix and Chromium using custom allocators (jmalloc and tcmalloc) but these, and also the quite novel Mesh-allocator are not as good as the Zig abstractions. Also, C programs that use malloc and free are generally at the mercy of the OOM killer, and these type of bugs have plagued systemd due to the incredible bug-hunter Evvrx.
3. Make integer overflow part of the language, instead of obscure "undefined behavior" rules which are often over-ruled by an excessive number of non-standard-behavior compiler switches which are even used by Linux.
Posted Feb 1, 2022 13:17 UTC (Tue)
by jdisnard (subscriber, #90248)
[Link]
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
The ability to work in parallel was crucial to the growth the free software.
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
The next 'UNIX' will be multiple users sharing a network of machines. It feels like Plan9 took a stab at this, with the wrong abstraction at the wrong time.
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
We have either local systems (BTRFS, XFS, etc.) or network systems (NFS, Samba, etc.), but nothing in between.Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Yeah...Beowulf clusters still exist, they are just called "data centers" now...
Beowulf
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Though it could be that it just sees a different point to the one that I see - it is a rather vague question.
A key factor in the success of Unix was the lack of effective competition. The OS market today has way too much competition for a similar success story.
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
> Though it could be that it just sees a different point to the one that I see - it is a rather vague question.
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Now that suitable hardware exists there are projects building open source, non AOSP based, operating systems for them. They will probably never become mainstream, just as Linux isn't and will probably never be mainstream on the desktop but they do exist.
I also think Android wouldn't have succeeded as well as it has if it hadn't been open source, even though it definitely isn't a community driven project.
Furthermore Android had specific time to market pressures (to have a chance of avoiding Apple dominating the market and Microsoft taking the rest).
Back in the 90's many people thought the future of computing would be building systems from sets of commercial , closed source, components. That was the model Microsoft technologies like ActiveX, OLE, COM/DCOM etc were supposed to enable.
That failed to materialize and most new languages / frameworks today are non starters unless they are open source.
For many if you can't get it with "git clone" and no jumping through hoops it doesn't exist.
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
- Haskell is a radical programming language that gave a lot of ideas now adopted by other languages
- Nix and Guix are proposing a fresh take on software package management and generally the OS
- QubesOS is also a fairly radical take on OS (although it is a layer on top of existing systems for usability reasons)
- the Coq proof assistant ( https://en.wikipedia.org/wiki/Coq ), and in general Interactive Theorem Provers, are radical projects and all the active ones are open source, with most run as free software projects (with a strong academic bias among contributors).
- I believe that some peer-to-peer projects also had a radical impact (Bitorrent these days), but I don't know enough about their history to tell if the open source / free software community participated actively. (I think there was a fair amount of Windows-based freeware at the time.)
- I would guess that Bitcoin was also operated by a crew of enthusiast hackers from the open source community at the start -- but again I'm not sure.
(I think it would be worth articulating whether we are discussing inventions that were born in the open source community and became successful-enough within it, or inventions that were born anywhere but remained fairly confidential, and became successful thanks to the open source community. I rather have the later in mind, and I think it is different from "commoditization".)
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix
Brian Kernighan on the origins of Unix