|
|
Log in / Subscribe / Register

Crowding out OpenBSD

By Jonathan Corbet
November 13, 2012
Unix as a whole predates Linux by many years, and even the rather younger BSD variant was well into its teens by the time Linus released his first kernel. BSD networking defined and enabled the Internet. This illustrious history notwithstanding, BSD has long since ceded the spotlight to Linux in most settings. As Linux has come to dominate the free software development world, the result has been some occasional pain for other operating system distributions. Now, as a recent discussion on an OpenBSD mailing list shows, BSD developers are feeling that pain in a heightened manner. This situation has some serious implications.

On November 6, longtime OpenBSD hacker Marc Espie complained to the OpenBSD project's "tech" list about behavior from "upstream vendors" that, in his view, is proving harmful to the OpenBSD project. In short, projects like desktop environments are increasingly adding dependencies on changes being made at other levels of the (Linux) systems on which they are developed. That makes it harder for OpenBSD to port and support that code, to the point that "if you don't have tens of people, it becomes more and more of a losing battle". The OpenBSD project doesn't have those people, so it is hurting. Marc continued, saying:

It's also quickly turning Posix and Unix into a travesty: either you have the linux goodies, or you don't. And if you don't, you can forget anything modern...

I'm pretty sure there's a lot of good intention behind the "progress" in recent desktops. But this is turning the field of OS distributions into a wasteland. Either you're a modern linux with pulseaudio and pam and systemd, or you're dying. So much for the pioneer spirit of opensource, where you were free to innovate and do cool things, and more or less have interesting software able to run on your machine...

One could easily poke holes in this complaint; the characterization of PAM as "modern" is somewhat amusing; it is 1990s technology. There is an evident case of cognitive dissonance shown in the simultaneous desire for the comfortable "Posix and Unix" world of decades past and the ability to "innovate and do cool things." It is difficult to simultaneously innovate and stand still, but that is what Marc seems to be asking for here.

In a subsequent message, Marc acknowledged the real source of the problem: OpenBSD simply does not have enough developers to influence the direction of projects like X.org, GNOME, or KDE. Antoine Jacoutot, who works on GNOME for OpenBSD, went further, stating that almost all of the work is being done by "Linux people" with little or no representation from other systems. Why, he asked, should they be concerned about portability in that situation?

In most free software projects, the developers who write the code have the most say over the direction the project takes. The BSD distributions have trouble coming up with enough developers to do the ports to their own systems; finding developers to help push projects forward — and influence their direction — is an even taller order. In the absence of active developers, they are, in a sense, just another user, able to make requests but with no ability to create the changes they would like to see. So big software projects move forward in directions that are not always convenient for systems like BSD.

One could argue that the Linux community is throwing its weight around. But we are really just seeing the way that free software development projects work; in the early 1990s, BSD-oriented developers were equally unconcerned about the difficulty of porting their code to Linux. When developers have enough problems of their own to solve, trying to make life easier for operating systems they do not use tends to end up fairly low on the list of priorities.

Where this is heading seems reasonably clear: without the ability to participate in these projects, or at least to keep up with them, the BSD projects will have an increasingly hard time supporting contemporary desktop environments. Their hardware support will continue to lag. They will not be able to take advantage of the work that is being done to operate well on mobile systems. There will be fewer and fewer settings where BSD-based systems will operate in the way their users want.

That, needless to say, is a recipe for irrelevance and, eventually, disappearance.

It may be tempting to shrug one's shoulders and say that none of this matters anymore. Your editor, whose first kernel hacking experience was on a BSD-running VAX, would not be so sanguine. But, in truth, even the most determined Linux fanboy should be concerned about a development like this. BSD is more than a repository of a great deal of Unix history and the continued home of a great many talented developers. It is an important part of the free software ecosystem.

BSD is a place where developers can experiment with different approaches to kernel design, filesystems, packaging systems, and more. OpenBSD remains a center for security-related work that does not exist to the same degree in the Linux world. The existence of alternative systems gives us resilience in case Linux is undermined by legal issues, security problems, or corporate mismanagement. Monocultures are unhealthy in general; a Linux monoculture may be the ultimate vindication of our approach to development, but it still would not be a good thing for the world as a whole. As in natural ecosystems, diversity is a source of strength.

Even so, a monoculture may be where we are headed, sometime years into the future. Economies of scale and network effects push in that direction; the fact that Linux is the best system for so many purposes helps to ensure that it will continue to get better in the future while alternatives will languish. Few developers will be able to find the time to, in effect, subsidize alternative operating systems by holding back progress in Linux. It is an outcome to anticipate and, possibly, plan for, but it is not one to celebrate or to try to hasten. If other free operating systems start to vanish, we will eventually realize that we were better off when they were still around.

to post comments

The monoculture of meritocracy

Posted Nov 13, 2012 23:09 UTC (Tue) by geofft (subscriber, #59789) [Link] (55 responses)

I've never been sure how to resolve "monocultures are harmful" with a volunteer-based meritocracy. If there is a "better" solution, how do we motivate people to work on the "worse" one? If Linux works on my laptops and has all the features OpenBSD has, why should I install OpenBSD?

I can see why it would be worth someone like Oracle putting money into both Solaris and Linux, both MySQL and Oracle RDBMS, since they can motivate developers with employment. But why should I, as a potential volunteer who's only motivated by impact and personal interest, work for the underdog?

One possible response is that if I have a crazy new idea for how kernels should work, it would be easier for me to get that done in the BSDs, as smaller projects, than in Linux. (But to counter my own counterargument, I'm much more familiar with how the Linux development process works than how the BSDs work.) I'm curious what other motivations we can come up with.

The monoculture of meritocracy

Posted Nov 13, 2012 23:23 UTC (Tue) by pynm0001 (guest, #18379) [Link] (36 responses)

For KDE we try to at least give the token nod to portability, by using facade patterns to try to map high-level concepts to appropriate low level libraries (something for which we take a lot of flak, btw). For instance with KDE 4.10 we will improve support for a shared-memory cache primitive on OpenBSD (and it should have worked even earlier, but then I can't test on OpenBSD... ;).

That doesn't help OpenBSD horribly much since they (or someone) still has to donate the manpower needed to report bugs, sometimes fix them, and test that the bugs are fixed (and stay fixed). But we at least don't try to call OpenBSD unsupported "by fiat" by locking in low-level required dependencies that are highly Linux-specific (that is, as long as it's reasonably doable without, and obviously Linux-specific utility applications will simply not work elsewhere, but the desktop would still be available).

The monoculture of meritocracy

Posted Nov 14, 2012 3:02 UTC (Wed) by aryonoco (guest, #55563) [Link] (29 responses)

Every layer of abstraction that you introduce, every layer of mapping that you put in your code, has a cost. It has a cost in implementation, it has a cost in that it reduces readability and maintanability of code, and it has a cost in that every additional layer is a potential new source of bugs.

Obviously there are benefits to it as well, namely expanded number of users, and the ability to run on "other" Unixes.

Time and resources are finite, and that's certainly true for free software projects. Developer of upstream projects, whether they do it explicitly and methodically, or implicitly and without realising, do a cost/benefit analysis to see whether supporting those other systems is worth it. For a long time, the answer was yes, as the BSDs did offer genuine advantages over Linux, and they had a non-insignificant userbase.

With time, things have changed. Research and development is now increasingly happening on Linux and the percentage of non-Linux users has diminished. To incur all those costs just to make life easy for less than 2% of the userbase is a scenario that is increasingly hard to justify for most projects.

The monoculture of meritocracy

Posted Nov 14, 2012 5:39 UTC (Wed) by dlang (guest, #313) [Link] (10 responses)

maintaining portability to different systems can also end up resulting in better software for every system.

I'll point you to the Linux kernel and it's support of many different CPU types. This requires layers of abstraction, just like supporting different *nix flavors. But, even with this cost, Linus is on record as saying that the kernel is significantly better than it would be if it had remained x86 only. The comment I am thinking of (from several years ago) was that modern kernels are not designed for any existing CPU, they are designed for some theoretical 'perfect' CPU with arch specific helpers to fill the gaps that all existing CPUs have (along with carefully constructed layering violations for the most performance critical sections)

This makes it rather easy for Linux to be ported to a new CPU.

I would like to think that while portability has it's costs, it should also have it's advantages. If the userspace software is modularized properly, just about all of the differences between different *nix (and similar) systems should be out in the periphery of the modules, not in the core logic.

This should not only help them be portable to the BSD family, but also to be better able to take advantage of new features in Linux (and for that matter for people to notice that a lot of programs are implementing similar work-arounds and then propose new features for Linux to avoid the need for them)

I wouldn't be surprised to see that this leads to improved portability to things like Android as well.

The monoculture of meritocracy

Posted Nov 14, 2012 9:45 UTC (Wed) by ortalo (guest, #4654) [Link] (9 responses)

IMHO, the Linux monoculture prevents sometimes from adressing specific problem areas (at least it slows down things enough to miss the target).
Linux is not yet totally ubiquitous - the most recent example is the Android "port" (which involved a lot of domain specific hacks). It will probably never be because it's probably impossible - nothing wrong per se by the way, it has already gone a long way towards the general usefulness goal.

But some areas may benefit from the immediate availability of an alternate (possibly less versatile) free software solution: security box, wifi boxes, GSM phones (look at the "OS" selected for OsmocomBB), home automation devices, networking equipment; possibly: supercomputers, industrial control systems, industrial embedded systems, etc. [1]
Personal computing devices are not the only computers in modern information technology, they may not even be the most important ones - even if they are surely the most *numerous*. (What would you miss more: your smartphone, or your bank central computer?)
IMO, the BSDs should be seen as an opportunity to adress some of those specific concerns that necessitates a full OS too [4]. BSDs developpers seemed to agree as they initially selected some specific target (security, x86 perf., portability, distributed systems). However, up to now, maybe those projects primarily did that by themselves for differenciation or branding purposes [2] while they should be given true opportunities to aim an important target.
And specifically, Linux should *stop* claiming these targets (because they are for the BSDs).

Sounds like political movements are needed: bringing up an alliance between all the "freedom" partners should be on the table, with the objective of settling on a long lasting treaty. It seems to me that this is necessary to refuel some allies again, up to the task.

[1] IIRC, sometime in the 90s, the Fermi lab linear accelerator needed an OS for a VAX. Only a BSD fitted there seriously. That's a single computer in production, but well... only two such scientific equipements have ever been built in human history (Fermi Lab and CERN) so... point gained.
At some point in time, it seemed like OpenBSD was targetting the specific area of an intra-network OS (highly secure, IP/TCP but also BGP, OSPF, NTP, etc.), but I suppose Linux claimed that target too.
[2] Possibly not DragonflyBSD ;-)
[3] I mean, do you really trust Cisco or Huawei to transport your packets on wire? Or Apple and Google your packets in air? Whatabout transporting yourself?
[4] But not as full as Linux...

The monoculture of meritocracy

Posted Nov 14, 2012 10:05 UTC (Wed) by nhippi (subscriber, #34640) [Link] (4 responses)

IMHO, the Linux monoculture prevents sometimes from adressing specific problem areas (at least it slows down things enough to miss the target). Linux is not yet totally ubiquitous - the most recent example is the Android "port" (which involved a lot of domain specific hacks).
I don't see how Linux monoculture prevented anything. Quite the opposite, Google was able to leverage Linux when creating Android. If anything, Android proves Linux is not a monoculture.

The monoculture of meritocracy

Posted Nov 14, 2012 14:42 UTC (Wed) by ortalo (guest, #4654) [Link] (3 responses)

I was speaking of Android modifications only to illustrate the fact that there are no totally ubiquitous OS (or kernel in that case). Alternatives *may* have been a better base.

A BSD was not even considered for projects like Openmoko [1]. They are still not considered for the FreedomBox project even though they share obvious concerns with one of them.
This is in my view a consequence of the Linux domination. With even wider domination, we will see bigger consequences. (Such as: no Linux => no graphics.)

Of course I cannot guess what would have occured for these projects if a BSD had been given a try and if these projects would have been more successful.

[1] However, it seems to me that the BSDs should have been interesting candidates for a smartphone (full networked OS in a single source tree, focus on maintainability, etc.).

The monoculture of meritocracy

Posted Nov 15, 2012 16:59 UTC (Thu) by pboddie (guest, #50784) [Link] (1 responses)

The issue of which platform to use is mostly concerned with what most of the potential contributors are likely to know and which platform is covered best by most of the available documentation and shared knowledge, so if it's easier to find people who know how to write a Linux driver for some hardware, then a project will choose Linux first and foremost. That hasn't stopped other operating systems being ported to such devices, though:

http://wiki.openmoko.org/wiki/Category:Distributions

And it's perfectly possible that other platforms can take advantage of the work done by Linux developers to get driver support for hardware even if they refuse to port the actual drivers. I'm aware of at least one "from scratch" operating system project for the Ben NanoNote that has presumably been able to take advantage of the fully documented hardware and Free Software drivers already written for Linux, so it's not as if the Linux community is denying others opportunities. In fact, it's very much the opposite of that.

The monoculture of meritocracy

Posted Nov 16, 2012 8:35 UTC (Fri) by ortalo (guest, #4654) [Link]

I totally agree with the importance of knowledge you outline.

However, I also think alternative OS do not take so much advantage of Linux work (or cooperation) as they suffer from the fact of being considered second class or second hand implementation (and not the full solution, i.e. "maybe better than Linux").

BSD for a smartphone OS

Posted Nov 16, 2012 10:11 UTC (Fri) by pboddie (guest, #50784) [Link]

Actually, isn't the OS for at least one of the Danger products based on BSD? According to the Wikipedia entry for the Danger Hiptop/Sidekick, they supposedly use NetBSD. Of course, vendors don't have to do anything more than reproduce licensing information for such permissively licensed software, and maybe only in obscure places even then, and so the awareness of a technology in the industry never reaches a point where everyone wants to try it out.

Some people praise permissive licensing and may refer to software licensed in such a way as their "secret weapon" or "competitive advantage", but if they're not willing to share their changes, any wider growth of such technologies is likely to lag behind things like Linux where there is a requirement to share and a resulting culture that does so extensively.

The monoculture of meritocracy

Posted Nov 17, 2012 16:53 UTC (Sat) by Jandar (subscriber, #85683) [Link] (3 responses)

> And specifically, Linux should *stop* claiming these targets (because they are for the BSDs).

If anyone using linux has an itch to scratch in these areas, should it be forbidden for him/her to improve linux? Do we patronazingly say: go away, linux has to be awful in this so that *BSD can shine a little?

The monoculture of meritocracy

Posted Nov 24, 2012 22:02 UTC (Sat) by mfedyk (guest, #55303) [Link] (2 responses)

I think we should call this initiative "no os left behind".

Unions for the developers would be good too.

The monoculture of set theory

Posted Nov 24, 2012 22:28 UTC (Sat) by neilbrown (subscriber, #359) [Link] (1 responses)

> Unions for the developers would be good too.

I prefer intersections. They let you work together without forcing you to.

The monoculture of set theory

Posted Nov 25, 2012 3:50 UTC (Sun) by Trelane (subscriber, #56877) [Link]

Personally, I prefer complements. Because I'm a contrarian.

The monoculture of meritocracy

Posted Nov 14, 2012 9:12 UTC (Wed) by quintesse (guest, #14569) [Link] (4 responses)

> it has a cost in that it reduces readability and maintanability of code

I'm sorry, but this is a myth that always gets thrown about by people who somehow have an allergy to the word "layers". One of the reasons for introducing a layer can be abstraction where you reduce a certain problem to its bare essentials making it *easier* to understand (and implement).

> and it has a cost in that every additional layer is a potential new source of bugs.

Any code you add can improve or reduce readability/maintainability and introduce bugs, it's definitely untrue to say that a layer will always worsen the case.

The monoculture of meritocracy

Posted Nov 14, 2012 11:54 UTC (Wed) by dgm (subscriber, #49227) [Link] (3 responses)

All of this is only true if (and only if) the abstraction is correct, in the sense that nothing relevant is omitted. Sometimes this is not the case, and then they make things more complex instead of simpler. Coming up with good abstractions is hard.

And there's always a price to be paid for the abstraction, be it in performance, flexibility or simplicity. As with many things, there's a trade off that has to be considered.

The monoculture of meritocracy

Posted Nov 14, 2012 12:06 UTC (Wed) by quintesse (guest, #14569) [Link] (2 responses)

Of course, but making things more complex instead of simpler is not something unique to layers. In fact if you just keep adding more and more code to an application without any thought to structure and design you'll probably end up with a mess that nobody but you understands.

The monoculture of meritocracy

Posted Nov 14, 2012 14:35 UTC (Wed) by dgm (subscriber, #49227) [Link] (1 responses)

I agree. The problem is that some think (or at least that's what they say) that abstraction is some kind of panacea that solves all problems, for free. Neither is true.

The monoculture of meritocracy

Posted Nov 14, 2012 14:40 UTC (Wed) by quintesse (guest, #14569) [Link]

You get no beef from me on that point. In fact over-engineering can turn something relatively simple into this hugely complex monster. (I've probably been guilty of that myself a couple of times before learning to KISS)

The monoculture of meritocracy

Posted Nov 14, 2012 9:53 UTC (Wed) by mpr22 (subscriber, #60784) [Link] (10 responses)

If you think abstraction layers always reduce readability and maintainability, I look forward to seeing your explanation of how a GUI application written in raw Xlib calls is more readable and maintainable than one written using a toolkit.

The monoculture of meritocracy

Posted Nov 14, 2012 17:13 UTC (Wed) by alankila (guest, #47141) [Link] (9 responses)

*sigh* I'll bite.

Take a look at the size of some commonly used toolkit, oh, GTK+3. That's 14 megabytes of compressed source, though some part of this is bound to be data. Whatever. 14 MB, I'll run with it.

Now suppose you write a hello world application: that's 14 MB + your own source, probably no more than 1 kB. Now write xlib application: it's probably a bit longer with respect to your own code, but eliminates 14 MB of code you did not need. Now, if you wish to understand the total system, you need to understand all your own code, all of GTK+, and then all of Xlib, and whatever is underneath it. If you don't have GTK+, you eliminated that much abstraction you did not need to understand.

Therefore: if I am allowed to stipulate that complexity can be counted this way, any additional code you add is a net loss, because it makes the total system more complex, no matter how pretty the world constructed in the cocoon of this new abstraction. But is this a valid argument? I don't really think so. GTK+ is probably allowing you to avoid having to deal with Xlib entirely, so you never have to learn any bit of it and yet get applications that run in X. And of course it gives you nice features like themeability practically for free. It's actually something better than Xlib for most programmers (which is why practically nobody writes raw xlib programs).

All of this is besides the point. My personal experience when dealing with abstractions that simply wrap some other functionality, is that they always make things worse. Let's call this the "driver pattern". These drivers are never quite as nice as the abstraction's API is, and the abstraction hides information from me which I generally need to make the application work well. Maybe I don't get good error reporting, or can't access all the functionality, or whatever. So I believe criticism of the driver pattern is often justified, because it rarely actually helps; rather, it seems to me that a lot of people are simply deluded to only count the upsides and wave away the downsides.

The monoculture of meritocracy

Posted Nov 14, 2012 18:14 UTC (Wed) by helge.bahmann (subscriber, #56804) [Link] (3 responses)

I challenge you to write a "correct" hello world Xlib in 1kB. "Correct" means: interacts correctly with modern Window & Compositing managers (think resize, think WM hints, think liveness pings, ...); has anti-aliased rendering of the "Hello World" string. For style points: make the "hello world" string selectable for copy/paste.

Once you have banged you head on the table trying to do that and come up with the required ~8000-16000 lines of xlib code, try the same thing with GTK and appreciate that this is really possible in less than 100 lines of source code. You may or may not want to revisit your POV on abstractions...

The monoculture of meritocracy

Posted Nov 14, 2012 18:38 UTC (Wed) by alankila (guest, #47141) [Link] (2 responses)

You missed the point.

The monoculture of meritocracy

Posted Nov 14, 2012 22:53 UTC (Wed) by helge.bahmann (subscriber, #56804) [Link] (1 responses)

My point is: If you want to get any work done in reasonable time, you have to rely on a lot of leaky abstractions (There are errors that can be caught at the Xlib level, that cannot be caught at the gtk level -- but frankly, no user cares). Prefer to fix an abstraction if it offers almost what you want, but not quite (or file a bug) than working around it.

The second aspect that I did not mention so far: Abstractions help making software better by introducing well-defined interfacing points for testing. A class doing nothing else but wrapping a file descriptor and operations on it is mockable for unit tests. A bare file descriptor is not.

The monoculture of meritocracy

Posted Nov 16, 2012 4:33 UTC (Fri) by alankila (guest, #47141) [Link]

I have nothing against using abstractions, as long as they are good ones. It's just that not all abstractions are good ones. This kind of discussion is really impossible to conduct properly without either drilling down on the features of good vs. bad abstractions, or talking about specific examples and why or why not they might succeed in what they do.

"A bare file descriptor is not."

Indeed. I quite think that libraries are better than file descriptors as abstractions for functionality. For instance, on java side you usually have a stream, and can easily replace it by your own class. C libraries, in contrast, often want file names or file descriptors, which is frankly inconvenient and irritating.

Heretical, I know.

The monoculture of meritocracy

Posted Nov 14, 2012 22:47 UTC (Wed) by jonabbey (guest, #2736) [Link] (1 responses)

See http://blog.aerojockey.com/post/iocccsim for a very short Xlib based flight simulator. ;-)

The monoculture of meritocracy

Posted Nov 15, 2012 8:13 UTC (Thu) by nim-nim (subscriber, #34454) [Link]

And :
1 it's not < 1k
2 the little text rendered does not satisfy modern constrains (could not check the WM interaction from screenshots)

So it's obviously not an exercise in avoiding unneeded abstractions, since it is not feature-comparable to something that would have used those abstractions.

The monoculture of meritocracy

Posted Jan 10, 2013 19:42 UTC (Thu) by zlynx (guest, #2285) [Link] (2 responses)

Missing features in abstractions: definitely true for Xlib.

I worked with a project a long time ago that ran on Solaris. It was nifty, and when the X server shut down for any reason (crash, disconnect, shutdown) the application just kept running and you could just go back to the application's command terminal and type "display :0.0" and it would put the GUI right back where it had been. You could put copies of the GUI on as many displays as you liked.

Try doing THAT with GTK.

The monoculture of meritocracy

Posted Jan 10, 2013 20:20 UTC (Thu) by nix (subscriber, #2304) [Link]

Indeed, Emacs can do this today (it's how I normally use it). But don't try doing that with Gtk, indeed: it'll coredump.

There's an open bug about this.

It's eleven years old.

which is unfortunate

Posted Jan 10, 2013 21:47 UTC (Thu) by renox (guest, #23785) [Link]

That is what is annoying me so much with many criticisms related to X: in fact in many case, they should be directed towards the toolkits not against X..

The monoculture of meritocracy

Posted Nov 14, 2012 23:01 UTC (Wed) by sebas (guest, #51660) [Link] (1 responses)

Yet, we'd be muchos unhappy if work on e.g Firefox or Chrome on Linux would be stalled, while the numbers are probably fairly comparable, relative to Windows...

The monoculture of meritocracy

Posted Nov 16, 2012 2:20 UTC (Fri) by sciurus (guest, #58832) [Link]

I think the number of Chrome users on desktop Linux is very low, but the umber on Linux is very high due to Android. If Google's ChomeOS is successful, the desktop numbers will rise.

The monoculture of meritocracy

Posted Nov 14, 2012 9:22 UTC (Wed) by smcv (subscriber, #53363) [Link] (1 responses)

"""For KDE we try to at least give the token nod to portability, by using facade patterns to try to map high-level concepts to appropriate low level libraries"""

Most projects do, to a lesser or greater extent. GLib wraps a lot of pointlessly different kernel/libc APIs, and the combination of GIO and its plugins wraps libproxy, GNUTLS etc. as well.

One of the problems with that pattern is that there's an art to designing the abstraction so that it has the features you want, and yet can still be implemented on all the OSs you care about. If GLib is anything to go by, it seems that to design such a thing well, you need an in-depth understanding of at least POSIX and Windows, and often Linux-specifics too.

Another is that having the abstraction is no help if you don't have all the implementations. GIO has GFileMonitor, an abstraction for filesystem monitoring (inotify/fam/polling), which should be implementable in terms of *BSD kqueue, but the closest it has got is an unmerged patch: until then, the BSDs get the fam or polling backend. The majority of GIO contributors are on Linux and have no motivation to implement alternative backends that they're never going to run.

GIO kqueue is an interesting case because it seems to be less useful (at least for GLib's use) than inotify: monitoring a filesystem with kqueue prevents it from being unmounted. The proposed patch to implement GFileMonitor adds a configuration file for sysadmins to configure which directories should avoid kqueue and fall back to polling, defaulting to /mnt, /media etc.; the GIO maintainers reviewing the patch, quite understandably, consider this to be a deficiency of the underlying OS, and would rather it be fixed in the kernel.

The monoculture of meritocracy

Posted Nov 14, 2012 13:37 UTC (Wed) by walters (subscriber, #7396) [Link]

The GTK+-and-below stack need to be portable to Windows, so various Unix variants are mostly a rounding error.

I think what's under discussion here has always been what functionality GNOME as a larger system (interface, sysadmin tools, etc.) has.

The monoculture of meritocracy

Posted Nov 14, 2012 13:56 UTC (Wed) by gwolf (subscriber, #14632) [Link] (3 responses)

From the discussions I have participated of (being a Debian Developer, who turned to Debian after being an OpenBSD guy for some years), it's not only that the different projects offer *BSDs the needed hooks for them to fill in, but it's a deeper philosophical issue: Among the points mentioned by Marc Espie were PAM (correctly characterized as 1990s work by the author) and systemd. It's not that they cannot port them — It's rather that their main developer community feels their usual way does not require it. If we in Debian had giant mail threads regarding how we will manage our init, I can perfectly expect the threads to be even more virulent in *BSD land, where they are not only integrators as we are, but authors of the base infrastructure — "You mean for us to throw out all that well-written code we have improved over the years and replace it with n00b stuff‽"

Who would have thought‽

Posted Nov 14, 2012 20:47 UTC (Wed) by man_ls (guest, #15091) [Link] (1 responses)

Nifty, didn't know that the interrobang existed. And in Unicode too!

Who would have thought‽

Posted Nov 15, 2012 11:01 UTC (Thu) by jwakely (subscriber, #60262) [Link]

I was impressed too, it's the first interrobang I've ever seen in the wild :)

The monoculture of meritocracy

Posted Nov 15, 2012 2:21 UTC (Thu) by pynm0001 (guest, #18379) [Link]

> "You mean for us to throw out all that well-written code we have improved over the years and replace it with n00b stuff‽"

But at the same time, what if they're right, and that their own stuff works better with the rest of their integrated software? The point on PAM is well-taken (although my Linux system uses it as well) but being old design doesn't inherently mean it's broken.

You mention being a Debian Developer, I can only assume that Debian would not take it well if e.g. Ubuntu were to switch to using RPM and try to coerce Debian to make the switch as well "because that's how everyone is doing it". And there would be dozens upon dozens of *good reasons* why Debian would not want to switch (backward compatibility, mounds of existing software written against apt, much of which would have to be ported, all of the various *.debian.org services which would need to at least be reviewed, if not modified, etc.)

Even *just in Linux* it has been prudent for us in KDE to avoid tying directly to low-level libraries. Konqueror can now use WebKit in addition to KHTML, Phonon let us transition nicely to PulseAudio for those who use it even though gstreamer-0.10 was current when KDE 4 was released, Solid has helped us survive the HAL/udev/u{disks,power} insanity, we have an authorization framework which has survived PolicyKit and both of its successors, etc.

It is easy to require very low-level dependencies when all you really have to worry about is making sure that Fedora and RHEL still end up working in time for the release. This isn't to say that we at KDE don't want to have very good support for such libraries when present, just that our policy on such libraries so far has actually worked very well for us.

The monoculture of meritocracy

Posted Nov 13, 2012 23:53 UTC (Tue) by ssmith32 (subscriber, #72404) [Link] (5 responses)

Haven't used BSD's too much, especially not lately, but I'd imagine...

Different focus (e.g. security & stability is paramount in OpenBSD), different packaging systems, and different licenses.

Although if security and stability are paramount, I'm not sure why OpenBSD is so worried about pulseaudio, etc. That's stuff you want stripped out if you're building a server anyways..

The monoculture of meritocracy

Posted Nov 14, 2012 0:58 UTC (Wed) by intgr (subscriber, #39733) [Link]

> Although if security and stability are paramount, I'm not sure why OpenBSD is so worried about pulseaudio, etc. That's stuff you want stripped out if you're building a server anyways..

Erm? Some people may want a higher security workstation. And people who develop the OS will certainly want to use it on the desktop too.

The monoculture of meritocracy

Posted Nov 14, 2012 0:58 UTC (Wed) by geofft (subscriber, #59789) [Link] (3 responses)

I don't know much about "stability" and how to quantify that. I thought about the focus on security, in particular because e.g. Capsicum landed in FreeBSD and not yet in Linux, but there's a ton of security research and security hardening happening in Linux, so as an academic researcher I still prefer to work on Linux, and as an academic researcher with hopes of his work being used by real people, working on Linux is even more compelling. By the sheer size of its community and the number of people caring about it, Linux attracts a number of interesting security efforts -- SELinux came before SEBSD, ASLR hit PaX first and was then ported to OpenBSD, yama is happening on Linux, etc. So OpenBSD doesn't seem to have a huge advantage here, in terms of attracting contributors.

I would, admittedly, like to know why Capsicum was done on FreeBSD, since it seems a textbook example of cool things being done somewhere other than the monoculture kernel.

Regarding PulseAudio, I am a little sad that a hardened and stripped-down PulseAudio for servers is not obvious. I'd like to run the network server on a machine in my apartment hooked up to the living room speakers, but it brings in a ton of dependencies I'd rather not put on a server.

The monoculture of meritocracy

Posted Nov 14, 2012 4:21 UTC (Wed) by wahern (subscriber, #37304) [Link] (2 responses)

The security approaches that we know work--code and design simplification, rigorously applying basic security practices like privilege separation, etc--are the areas that Linux fares worst compared to the BSDs. If security could be assessed by name dropping (papers, projects, w'ever), then Windows and other commercial systems would be granite mountains.

The OpenBSD folks don't want to chase features. What they want is to chase minimalism while staying relevant and useful. That's obviously a difficult path. It's made harder because many in the Linux community openly challenge even the pretense of portability, and like evil companies of yore have begun co-opting the standardization process, i.e. POSIX, and adding whatever crap features already in their toolbox regardless of merit.

Also, the idea that the "the BSD folks screwed us in the 1990s with a lack of concern for portability, so it's okay if we screw them now" is a little silly. Regardless of the veracity, it's just plain evil. Linux is not sacred. Not all Linux features are perfect, or worthy of adoption. Innovations which are strong enough to be accepted by other operating systems are likely innovations with far more merit. When you write something which only _you_ think is awesome... that should give you pause.

Monoculture sucks because you lose positive feedback. The bad features soon begin multiplying just as much as the good features, and eventually you end up with a cancerous wreck.

The monoculture of meritocracy

Posted Nov 15, 2012 15:16 UTC (Thu) by lacos (guest, #70616) [Link] (1 responses)

Thank you, William, great comment.

> chase minimalism while staying relevant and useful

This matches my ideals perfectly. Unfortunately, I can't run *BSD as a home user, because (as much as I ignore "modern desktops") I need my consumer electronics crap to work with my computer. For that I need user base behind my desktop OS. I must go with the gnome3-crazed crowd because they cause new drivers to be written too.

(The logical extrapolation would be to run Windows at home, of course, but I simply can't tolerate it.)

I'm already buying only years old (aka "antique") "consumer technology", both for low price and for better support, but Debian Stable *still* screws me regularly.

The monoculture of meritocracy

Posted Nov 15, 2012 18:22 UTC (Thu) by wazoox (subscriber, #69624) [Link]

> Debian Stable *still* screws me regularly.

Slackware, man. Slackware is the way. Antique, battle proven technology (no stinky systemd! no friggin' pam! good ol' *BSD style rc files!) and modern enough stuff. And sbopkg. Even this stupid new phone with mtp storage mode works, thanks dog (new phones don't come with usb-storage anymore, noooo, would be too easy and practical).

The monoculture of meritocracy

Posted Nov 14, 2012 0:56 UTC (Wed) by josh (subscriber, #17465) [Link] (10 responses)

The serious answer: in theory, one system may be better on one axis and worse on another. However, in practice it proves easier to take the overall better (or more popular) system and add the features you want to that system, so that you get the best of both worlds.

The less serious answer: some of the people who run less popular environments do so for the same reason people program in esoteric languages: for fun and challenge.

The monoculture of meritocracy

Posted Nov 14, 2012 1:02 UTC (Wed) by geofft (subscriber, #59789) [Link] (9 responses)

Yeah, the serious answer is what I'm worried about. See my other, more detailed reply, but with things like SELinux and PaX on Linux weakening the "OpenBSD is where security happens" story, I don't know how to avoid a monoculture that is compellingly better on one axis (community, hardware support, etc.) from eventually outpacing its competitors on every other axis.

Or phrased as an engineering problem: if we believe monocultures are bad, without breaking the constraint that meritocracy is good, how do we structure our projects and communities so that people work on e.g. the BSDs instead of making Linux better than the BSDs on all axes?

The monoculture of meritocracy

Posted Nov 14, 2012 1:11 UTC (Wed) by josh (subscriber, #17465) [Link] (8 responses)

I'd rather restate that problem: why do we consider monocultures bad, and based on that, why do we consider Linux a monoculture?

I've heard plenty of arguments against a Windows or OS X monoculture. Most of them amount to lack of alternatives when something doesn't work as desired, and lack of diversity for security purposes. The former simply doesn't apply to Linux: you have an almost excessive number of alternatives even for core OS components, and if you don't like them you can always make more (up to and including forking the kernel). For the latter, I'd personally rather focus on keeping one OS secure rather than two. Any other good reasons?

Let's not repurpose canned arguments summed up by words like "monoculture" without reevaluating those canned arguments against their new targets.

The monoculture of meritocracy

Posted Nov 14, 2012 1:18 UTC (Wed) by geofft (subscriber, #59789) [Link] (3 responses)

So the article says "Monocultures are unhealthy in general; a Linux monoculture may be the ultimate vindication of our approach to development, but it still would not be a good thing for the world as a whole. As in natural ecosystems, diversity is a source of strength."

I'm happy to disagree with that, and say that either Linux isn't a monoculture or that Linux is a good monoculture, and that all the BSD developers should give up and switch. :-) But it's what the article says, so I'm assuming it's representative of our community. Maybe it's not!

That said, to give one possible argument, we might find a class of attacks against a certain way of thinking about things in Linux that wouldn't happen against another OS. This sort of thing is certainly common in low-level crypto design (e.g., side-channel attacks from algorithms that aren't carefully designed to do the same operations regardless of the input), and is one of the many motivations for the AES and SHA competitions involving getting lots of different people to propose different ideas and attack each other's ideas, instead of collaborating on a single algorithm. On the software side, I've heard of cases where, e.g., several separate security bugs were found regarding NT's kernel entry ABI, and you couldn't fix those bugs at once without completely redesigning that ABI and possibly API.

The monoculture of meritocracy

Posted Nov 15, 2012 14:12 UTC (Thu) by khim (subscriber, #9252) [Link] (2 responses)

This sort of thing is certainly common in low-level crypto design (e.g., side-channel attacks from algorithms that aren't carefully designed to do the same operations regardless of the input), and is one of the many motivations for the AES and SHA competitions involving getting lots of different people to propose different ideas and attack each other's ideas, instead of collaborating on a single algorithm.

This is good example. Let me rephrase the question: why do you think Linux is a problem while AES and SHA are Ok? IOW: why even have a contest where one winner is picked and then reused everywhere if monoculture is so bad?

AFAICS Linux fulfills the same role as AES or SHA: different implementations are offered and one it picked, then used everywhere. If it was a bad decision then later it can be changed (see: DES to AES and MD5 to SHAxxx transitions).

We don't support the ability to use some exotic and clever ciphers in our documents and web-servers (only when some cipher wins the contest it's used for "real" programs), why should we support the ability to run "real" programs on experimental OSes like OpenBSD or Haiku?

On the software side, I've heard of cases where, e.g., several separate security bugs were found regarding NT's kernel entry ABI, and you couldn't fix those bugs at once without completely redesigning that ABI and possibly API.

If that happens then you need to redesign said ABI, it makes no sense to cultivate series of OSes each with it's own problems: sooner or later they all will be compromised if you'll not tighten them up.

The monoculture of meritocracy

Posted Dec 1, 2012 19:29 UTC (Sat) by ThinkRob (guest, #64513) [Link] (1 responses)

>This is good example. Let me rephrase the question: why do you think Linux is a problem while AES and SHA are Ok? IOW: why even have a contest where one winner is picked and then reused everywhere if monoculture is so bad?

I think there's a simple answer to that, actually: the impact of bugs.

If a kernel that's been out in the wild for some time has a bug due to a lack of attention, then users might hit crashes or lose data. That's bad, but it's fixable. Users may be able to work around it, and a lot of crash bugs are not *that* hard to fix once they're identified (finding/reproducing them is the hard part.)

If a cipher has been out in a while, and is found to have a bug that, say, reduces the key strength from 128 bits to 50 bits, that's also bad. But unlike a kernel bug which can be fixed or worked around, the impact of the bug is retroactive. All of the data encrypted with that broken cipher is now vulnerable. Worse still, there's no way to recall it. Some bad guy intercepted your traffic protected with $BUSTED_CIPHER? Well if he kept a copy around once the bug is found he can go back and decrypt it.

Unlike a kernel bug, a crypto bug can be devastating for *years* after it's been found and fixed, and there's not always a way to mitigate the damage. So while we want solid, bug-free kernels, there is a much, much higher value placed on getting our encryption/hashing algorithms right the first time.

*That* is why I'm OK with people unifying behind one or two ciphers and one or two hashing algos. Yes, it does have the "eggs in one basket" issue, but the cost of getting it wrong can be so very high that we really want to ensure that we have as many eyeballs on it as possible.

The monoculture of meritocracy

Posted Dec 10, 2012 12:51 UTC (Mon) by ekj (guest, #1524) [Link]

If you're paranoid about it, you nest ciphers (with unrelated keys!)

You use AES( k2, BLOWFISH( k1, plaintext)) which is secure aslong as *either* blowfish *or* AES survives.

You can do the same thing with hashes, but you need to concatenate or interleave them rather than nest them - the result is a hash that is as large as the sum of the two -- and that remains secure aslong as atleast once of the hashes is secure. (and *possibly* secure even if both hashes are broken)

Even someone who -can- find sha1 and md5 collisions *might* have a harder time finding two distinct documents that collide in both md5 and sha1. (yes I'm aware that md5 has been broken)

The monoculture of meritocracy

Posted Nov 14, 2012 11:12 UTC (Wed) by el_presidente (guest, #87621) [Link] (3 responses)

> The former simply doesn't apply to Linux: you have an almost excessive number of alternatives even for core OS components

Nobody needs a separate /usr partition, udev without systemd is a dead end, pulseaudio is the future, sloppy focus is for people caught up in the 90s, those who complain about client side decorations don't know what they are talking about, most users don't write their own init scripts, fallback mode is a maintenance burden, the unix philosophy doesn't apply to the modern desktop.

It's pointless to have alternatives.

The monoculture of meritocracy

Posted Nov 14, 2012 15:30 UTC (Wed) by ortalo (guest, #4654) [Link]

Admittedly, that's an alternative path to world domination, and it does not necessitate any alliance at all (these political discussions are so boring for el_presidente).

But it has already been done as nobody needs something else than M$/Windows.

The monoculture of meritocracy

Posted Nov 14, 2012 21:35 UTC (Wed) by ThinkRob (guest, #64513) [Link] (1 responses)

That's either an excellent troll or -- if coming from a developer -- a deeply depressing expression of a rather dangerous mindset.

The monoculture of meritocracy

Posted Nov 15, 2012 15:29 UTC (Thu) by lacos (guest, #70616) [Link]

> That's either an excellent troll or -- if coming from a developer -- a deeply depressing expression of a rather dangerous mindset.

Hm, neither. el_presidente didn't quote everything he responded to.

josh said:

> lack of alternatives when something doesn't work as desired [...] simply doesn't apply to Linux: you have an almost excessive number of alternatives even for core OS components

to which el_presidente replied,

> Nobody needs a separate /usr partition, udev without systemd is a dead end, pulseaudio is the future, sloppy focus is for people caught up in the 90s, those who complain about client side decorations don't know what they are talking about, most users don't write their own init scripts, fallback mode is a maintenance burden, the unix philosophy doesn't apply to the modern desktop.

It is sarcasm, but not a troll. It argues that alternatives are important, and that their number isn't actually "excessive" in Linux.

That's my take, anyway.

The monoculture of meritocracy

Posted Dec 16, 2012 5:14 UTC (Sun) by hasard (guest, #47410) [Link]

But why should I, as a potential volunteer who's only motivated by impact and personal interest, work for the underdog?
It's funny that you say that, considering that according to 99% of computer users, you are already working for the underdog.

how solid is the Linux "Monoculture"?

Posted Nov 13, 2012 23:23 UTC (Tue) by dlang (guest, #313) [Link] (1 responses)

we see a lot of variation within Linux from the different distros and their different approach to doing thing.

Even at the kernel level, it's more of a family of closely related branches than a single entity.

This is not as diverse as Linux vs OpenBSD, but it's a fairly close analogy to the relationship between the different *BSD systems (not as much divergance, but still quite a bit)

It's amusing to see this viewpoint of Linux being a monoculture at the same time that we see people lamenting that the Linux desktop is so horribly fragmented. And that Linux is so hard for software vendors to support because it's so varied.

We must be doing something right if we're getting criticized for both being too monolithic and being too fragmented at the same time :-)

how solid is the Linux "Monoculture"?

Posted Nov 14, 2012 8:38 UTC (Wed) by marcH (subscriber, #57642) [Link]

> We must be doing something right if we're getting criticized for both being too monolithic and being too fragmented at the same time :-)

It does not require a new culture to break compatibility (I think I've already answered this same comment of yours some time in the past...)

This being said, I agree that Linux does not bear any of the "monoculture" risks: whether it is fragmented today or not the GPL makes sure it could become at any time.

Crowding out OpenBSD

Posted Nov 13, 2012 23:30 UTC (Tue) by rleigh (guest, #14622) [Link] (67 responses)

Though I doubt my opinion will make any difference, I dearly wish that projects such as systemd and gnome would appreciate that their use of Linux-only features is indirectly harmful in a multitude of ways to the free software ecosystem as a whole. Or rather, forcing the use of Linux-only features, and deliberately excluding others, since it's perfectly possible to use platform-specific features while also remaining portable. It just requires a good attitude and a little effort. Unfortunately these seem to be missing entirely in certain quarters in recent years.

Crowding out OpenBSD

Posted Nov 13, 2012 23:41 UTC (Tue) by vonbrand (subscriber, #4458) [Link] (20 responses)

Sorry, but the effort to run, e.g. systemd, over a non-Linux system is definitely not "a little." Writing portable code is very hard. Keeping it portable is a lot of of extra work, requiring many able bodies.

Crowding out OpenBSD

Posted Nov 14, 2012 1:16 UTC (Wed) by mikemol (guest, #83507) [Link] (12 responses)

Are there decent 'lint' or 'valgrind'-style tools for warning about use of nonportable APIs (or APIs without universal behavior)? I could imagine such a thing being immensely useful for those of us who write software and care.

A 'lint'-style tool would obviously be a static analysis thing. A 'valgrind'-style might be able to spot uses of various calls without POSIXLY_CORRECT defined.

Of course, not all warnings are meaningful for a program. Not all platform targets are meaningful for a program. It'd be a matter of defining the testing scope, as with anything else.

Crowding out OpenBSD

Posted Nov 14, 2012 1:34 UTC (Wed) by mpiechotka (subscriber, #83514) [Link] (11 responses)

The problem is not in tools or lack of them. Usually it is that you need to write different code to support different APIs. Think kqueue vs. epool vs. ... - they all sort of solve the same problem but might have different edge cases etc. And this is relatively easy part - if you try to manage the USB driver you need to talk directly to OS. In the end you have a massive code which can be inflexible because you haven't thought about multi-seat at the beginning and you need to add it now. Oh and you need to maintain so better prepare to have multiple VMs to test it on different setups.

As of testing - it might detect bugs but usually a) they are relatively simple bugs not big, large bugs like "you cannot do X on system Y" b) they test one code path and the bug might be in another (I'm not saying that testing is bad but that it is supportive to portable design not a full verification).

Crowding out OpenBSD

Posted Nov 14, 2012 2:07 UTC (Wed) by mikemol (guest, #83507) [Link] (10 responses)

I wasn't looking for full verification, but rather tools to point at likely problems. Similar to how a compiler warns me that some code I wrote may not do what I intended...I'd love to have an API call audit warn me that I'm calling an API not present in all my supported platform targets. Or that I'm calling them with parameters that aren't properly supported.

I wish I had time to respond to all the things you mention. Some of them are going to be true of any software project (Paradigm shifts forcing architectural changes), some are greatly mitigated by avoiding unportable APIs unless there is a strong benefit to using them, and some are mitigated by keeping code modular and unportable portions behind abstractions. But I've got to go catch a bus...

Crowding out OpenBSD

Posted Nov 14, 2012 8:17 UTC (Wed) by mpiechotka (subscriber, #83514) [Link] (9 responses)

No, they don't apply to all programs - I would say they don't apply to at least 99% of programs. There is no point in say graphical calculator to call any unportable API. However graphical calculator-like applications is not the only class of programs.

Modern DE usually at least try to manage devices, do power management, session management etc. - those are the hard problems. As of writing portable abstraction layer - it is usually good move if many components use it. However if you are the only user you moved a problem a bit, make architecture cleaner but haven't solve the ultimate problem - someone have to maintain the portable layer and move it forward on all platforms in sych. That means that say users are limited only to support of things that are on all platforms (and if say NetBSD have some wonderful feature the DE developed by NetBSD people couldn't use it).

I'm not saying that it is impossible but that it is hard (even if you do know something is not portable you don't necessary can solve it in simple way - I run at least once in problem when on system X you had do A while on system Y you mustn't do it). And given that by definition projects are understaffed I can imagine that portability might slip as a goal.

Crowding out OpenBSD

Posted Nov 14, 2012 13:57 UTC (Wed) by mikemol (guest, #83507) [Link] (8 responses)

So we agree that for 99% of programs, the tool I'm asking for makes sense? I'm OK with that.

I disagree that you need to move the portability layer in sync. It makes pragmatic sense to have first-class and second-class platforms. First-class platforms can be in sync, second-class platforms can be a bit behind. As a user of a second-class platform[1], I understand this can be a pain, but I've also seen how it ultimately ends up work out.

At least with the portability layer, you've limited the number of lines of code that a porter needs to focus on.

[1] Gentoo. I'm sure the various BSD groups have it worse, but try running a pure-stable Gentoo. You pretty much can't. And there's no official stance on whether running a mixed stable/unstable setup is supported; ask two developers, you'll get three answers.

Crowding out OpenBSD

Posted Nov 15, 2012 14:27 UTC (Thu) by khim (subscriber, #9252) [Link] (7 responses)

So we agree that for 99% of programs, the tool I'm asking for makes sense?

I think you both are mixing issues. My own experience is radically different: for 99% of programs full portability is a pipe dream (think getopt: pretty popular tool on *nix, but problematic if you need to support Windows, too - you either need to provide your own implementation bundled with your program or use some third-party package, etc).

What is possible is to write, say, 90% of program in a fully-portable way, cover the remaining 10% with portability layer and keep 1% as a system-specific code surrounded by ifdefs.

Sadly even if for typical program just 1-2% code must be system-specific these pieces impose insane amount of maintenance work - significantly more then 1-2%. And then the obvious answer is to use OpenSSH approach (born by OpenBSD, ironically enough): just cover your primary platform and don't think about portability. If someone wants to have portable version - they can do it as a patches on top of clean-yet-unportable code.

Crowding out OpenBSD

Posted Nov 15, 2012 15:04 UTC (Thu) by mikemol (guest, #83507) [Link] (6 responses)

I wouldn't dream of "full portability". To me, that includes Linux, Windows, various BSDs, FreeDOS and places with 12-bit bytes. Even outside of practical considerations, to me "full portability" is a phrase that represents a paradox.

The way I see it, you have a tester program with a database of things to check against, and you give it an operational mode for the classes you care about. I.e. "warn me about anything you know of that's incompatible between Xorg and XFree86. Give me an error for anything that's incompatible between FreeBSD, OpenBSD and GNU." or "Give me an error for anything incompatible with unix-like systems [where that would mean common Linux environments, FreeBSD, OpenBSD, QNX...]"

I'm not suggesting the programmer should be demanded to provide support for all known platforms. I'm suggesting a programmer should be able to decide what platforms he'd like to support, get an advance rough idea at the parts of his program that are incompatible with that set, cull his supported platform set down to something manageable, and manage that.

The tool I'm describing would also be useful for people doing the active porting work, since it can immediately draw their attention to the places which need work.

Crowding out OpenBSD

Posted Nov 17, 2012 22:15 UTC (Sat) by cmccabe (guest, #60281) [Link] (5 responses)

Someone should create a language that you can "write once, run anywhere." Radical to the max, dude!

That was a 90s reference, for you youngsters...

Crowding out OpenBSD

Posted Nov 19, 2012 9:03 UTC (Mon) by ortalo (guest, #4654) [Link] (4 responses)

Hmmm, strictly speaking, it seems that this should even be a reference to the 70s and... C no?

BTW, note that C is the primary unifying factor between Linux and the BSDs in fact... and the main difference with the dominant proprietary OSes.

Crowding out OpenBSD

Posted Nov 20, 2012 9:05 UTC (Tue) by cmccabe (guest, #60281) [Link] (3 responses)

One of Java's slogans was "write once, run everywhere."

I guess the implication was that C programs always required tweaking to run on platforms other than the ones they had been written on, and Java programs would not. Of course it didn't quite turn out that way in practice-- for example, Hadoop is mostly written in Java, and the process of porting it to Windows is still ongoing.

It's hard to find a free lunch in computing. Even harder to find a type of free lunch that someone else hasn't already spent years looking for.

Crowding out OpenBSD

Posted Nov 20, 2012 11:52 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

Uhm. I've been running Hadoop on Windows for _years_. The main stumbling block is setting up an SSH daemon, but even without it you can use Hadoop for certain limited tasks.

Crowding out OpenBSD

Posted Nov 21, 2012 21:33 UTC (Wed) by cmccabe (guest, #60281) [Link]

Yes, there's a branch of hadoop called branch1-win that has been modified to work well on Windows. There's a JIRA tracking its progress here: https://issues.apache.org/jira/browse/HADOOP-8645.

Crowding out OpenBSD

Posted Nov 20, 2012 12:11 UTC (Tue) by man_ls (guest, #15091) [Link]

The idea was actually that a single "binary" (bytecode actually) could run, unmodified, on several platforms. Yes, "write" here is grossly misused.

Given that the bytecode was in fact interpreted by the VM (and later compiled on the fly), the comparison to compiled languages was misdirected: Java had the inconvenience of having to compile and the slowness of interpreted languages, but none of the advantages. It would have been more appropriate to compare with interpreted languages such as Python or Perl, but then Java would have lost big time since VMs were only available for a handful of platforms and the competition can run, unmodified, on tens. Here by "platform" I mean a combination of OS and processor architecture.

Nowadays Java is JITted and fast. But then again so is the humble javascript...

Crowding out OpenBSD

Posted Nov 14, 2012 15:41 UTC (Wed) by cruff (subscriber, #7201) [Link] (6 responses)

Writing portable code is very hard. Keeping it portable is a lot of of extra work, requiring many able bodies.

I have to say that writing portable code is not really that hard, as long as you have a well defined common base, such as POSIX, to target. It just takes discipline and proper abstraction of the features you need that are not part of the base. I've written multithreaded C++ software for a variety of POSIX compliant OSes that only required some straight forward GNU make makefiles and a few OS dependent source files to fill in the missing or OS dependent parts. Didn't even need the mess of autoconf, etc.

Once you stray far from the common base chasing the neat new features, whether for the wow factor or because of need, then it can be hard, but need not be complex if structured correctly.

Crowding out OpenBSD

Posted Nov 14, 2012 18:28 UTC (Wed) by jwakely (subscriber, #60262) [Link] (5 responses)

> a well defined common base, such as POSIX

Which release? I'm not sure all the BSDs are down with some of POSIX-2008 yet (though I have only looked at recent-but-not-current releases)

> only required some straight forward GNU make makefiles

Later in the thread Espie complained [1] that GCC requires GNU make, apparently even that requirement is too much ;)

But then he also complained that GCC major releases are a few years apart (which was true a few years ago, but not now) and that changes only go in the trunk not release branches (so what? push your changes upstream and maintain patches in the ports collection against old releases, then drop the patches when a new release comes out that already has the changes ... surely that's obvious?) The implication in [1] that GCC only builds on Linux is ridiculous, plenty of other OSs manage to keep GCC trunk building without herculean effort, if OpenBSD can't it's not GCC's fault. I've made sure NetBSD does myself, not because I ever use it, but just because I had access to a box.

[1] http://thread.gmane.org/gmane.os.openbsd.tech/30742

Crowding out OpenBSD

Posted Nov 14, 2012 18:49 UTC (Wed) by jwakely (subscriber, #60262) [Link]

Sorry, wrong link, [1] should be http://article.gmane.org/gmane.os.openbsd.tech/30747

Crowding out OpenBSD

Posted Nov 14, 2012 23:54 UTC (Wed) by wahern (subscriber, #37304) [Link] (1 responses)

It doesn't matter whether every Unix supports all of POSIX. What matters is that they everybody aims for common or similar abstractions, and that people test, test, test. "Portability" is a process.

I wrote a C library, wrapped in Lua, which does O(1) polling, async file notification, async DNS, async SSL, wrapped almost all of OpenSSL's X.509 key bindings, async descriptor passing, and async thread management in just about one week's worth of man power. The only dependency is OpenSSL.

Not only that, but I wrote it and maintain it for Linux, OS X, Solaris, FreeBSD, NetBSD, and OpenBSD.

http://25thandclement.com/~william/projects/cqueues.html

I learned one very important thing doing that: the BSDs generally have the better abstractions (particularly in kqueue), though Linux has more comprehensive POSIX API support. But in any event, it simply wasn't that difficult to write all of that portably, and in a small amount of code. And with VMs, it's been trivial to maintain the build for all the present releases of those systems.

The reason why people have a problem with portability is because even the best programmers fall into the habit of cargo cult programming. They're new to some API and scour usenet (or more likely these days, stackoverflow.com) looking for descriptions and usage examples. Then they copy the examples and move on.

But if you take the time to study the APIs, to study POSIX, and to study the implementation code (thank you, open source), then portability just isn't that big of deal. And the end result is that your program is more robust, because not only are more code paths tested, you actually understand what the heck is happening, as well as understand how and in what direction the APIs will evolve in the future.

It's important to constantly test on different platforms, though, and not to let portability issues fester and metastasize. But that's, sadly, what is happening with most open source software these days. People not only just don't care, they openly reject the idea portability. And that's a damn shame.

As for GNU Make, writing portable make is easy, but I allowed myself that single convenience, mostly because GNU Make is readily available everywhere. The BSD makes are almost as powerful, and I've even begun writing a make parser+compiler to translate between them. Mostly because GNU Make is darned ugly, and some of the BSD make syntax features are easier to understand while remaining just as powerful; so I want to write in a subset of of my favorite features from the different makes and then translate to each accordingly.

Crowding out OpenBSD

Posted Nov 14, 2012 23:56 UTC (Wed) by wahern (subscriber, #37304) [Link]

And FWIW, signal notification. Although, Solaris doesn't have a pollable signal facility.

Crowding out OpenBSD

Posted Nov 15, 2012 2:52 UTC (Thu) by nix (subscriber, #2304) [Link] (1 responses)

Later in the thread Espie complained [1] that GCC requires GNU make, apparently even that requirement is too much ;)
Well, that's politics, isn't it. GNU Make is GPLed, therefore bad (even though it doesn't contaminate projects built with it), therefore all projects should work with BSD make as well. (That the program being compiled is both GPLed and a flagship GNU project itself apparently does not affect this policy in its case, though common sense says it probably should.)

Crowding out OpenBSD

Posted Nov 15, 2012 14:57 UTC (Thu) by jwakely (subscriber, #60262) [Link]

Indeed it is politics, and common sense should indeed let the GNU compiler use GNU make.

My point is that espie@ seems to want to have his cake and eat it. If you refuse to work with upstream (for technical or political reasons) then don't be surprised if upstream forgets about you.

On the plus side, as he says he is happy to leave kettenis@ to deal with GCC, and he recently pushed a load of patches from OpenBSD to GCC, which were gladly accepted (although I did forget to apply one until reminded, but I'd hardly call that "a fight".)

Crowding out OpenBSD

Posted Nov 14, 2012 1:05 UTC (Wed) by josh (subscriber, #17465) [Link] (35 responses)

Have you looked at the list of Linux features that systemd requires? Have you ever read through that list and given serious consideration to how much work it would require to duplicate those features on another platform? I have, and doing even a fraction of it in a remotely acceptable and maintainable way would easily make for a full-time job. Do you happen to know a half-dozen OpenBSD developers with the time and inclination to do this work, continuously? If so, why shouldn't they work on the corresponding Linux-compatible kernel features instead, so that non-systemd programs can use them too?

I think this article highlighted the critical problem: Linux development has accelerated to the point that some other OSes don't have enough sparse developer time to keep up. Yelling "wait up" doesn't change that, it just slows Linux down so that other OSes have a better chance of keeping up with it.

I consider the BSDs a valuable part of the community, and I'd rather have multiple awesome OSes to work with rather than one awesome OS. However, I'd rather have one awesome OS than multiple less awesome OSes.

Crowding out OpenBSD

Posted Nov 14, 2012 1:15 UTC (Wed) by dlang (guest, #313) [Link] (33 responses)

picking on systemd for a moment (hopefully without spawning a 1000 post thread)

Part of the question, even on Linux is if systemd should really REQUIRE all those linux specific interfaces, or merely use them if they are available.

Even on Linux, sometimes people don't want to use cgroups and other things that systemd currently requires.

some of the problem that the BSD folks have is due to them just not having the manpower to keep up, but some of it is also due to some developers using new Linux features with no provision for running on a system without those features (be it Linux or *BSD)

Crowding out OpenBSD

Posted Nov 14, 2012 1:41 UTC (Wed) by josh (subscriber, #17465) [Link] (23 responses)

When I said "given serious consideration to how much work it would require to duplicate those features on another platform", that allows for the possibility of providing the same features systemd does without using the platform features it currently requirs. Even allowing for that, I think my comment still stands: doing so would require a prohibitive amount of work.

You don't have to use cgroups; you do have to support the features systemd currently uses cgroups to support, such as supporting daemons that fork or double-fork or have child processes, while still respawning daemons that exit or crash. Upstart handles that "portably" by ptracing daemons that fork and capturing the relevant PIDs; that approach requires a great deal more code and work than using cgroups, because that approach doesn't rely on a platform feature specifically designed to support it.

Now repeat that exercise for the hundreds of systemd features supported by Linux platform features.

Crowding out OpenBSD

Posted Nov 14, 2012 22:01 UTC (Wed) by dlang (guest, #313) [Link] (22 responses)

> You don't have to use cgroups; you do have to support the features systemd currently uses cgroups to support, such as supporting daemons that fork or double-fork or have child processes, while still respawning daemons that exit or crash.

Why does the init system have to do this job?

If a daemon exits or crashes, is it really the right thing to just start a new copy? what damage did the failed instance do that may need to be cleaned up?

If a daemon has a complex set of dependancies (or deliberately does something like a double-fork to distance itself), is it really enough to only consider it 'failed' if they all exit? or should there be something more application aware (i.e. a daemon specific watchdog) that does this?

Crowding out OpenBSD

Posted Nov 14, 2012 22:31 UTC (Wed) by raven667 (subscriber, #5198) [Link] (20 responses)

>> You don't have to use cgroups; you do have to support the features systemd currently uses cgroups to support, such as supporting daemons that fork or double-fork or have child processes, while still respawning daemons that exit or crash.

> Why does the init system have to do this job?

Its in the best position to do this as it's what starts the daemons and has a parent/child relationship with them. Process respawning is also available in SysV init as configured by /etc/inittab but that feature is incomplete.

> If a daemon exits or crashes, is it really the right thing to just start a new copy? what damage did the failed instance do that may need to be cleaned up?

I would think that in the general case the answer is very much YES. If you aren't worried about service outages I suppose you could make failure permanent, maybe wrap the daemon startup in a script which disables the service after the daemon exits. Seems kind of silly though.

> If a daemon has a complex set of dependancies (or deliberately does something like a double-fork to distance itself),

Double forking is more an implementation detail needed by other init systems, you don't need or want to double fork with systemd, although things will still work if you do.

> is it really enough to only consider it 'failed' if they all exit?

I'm not sure what happens if you challenge the init system in this way, what's the Right Thing(tm) to do in this case? If the processes maintain a parent/child relationship then you can rely on signals otherwise if the processes take input you can manage their sockets to detect failures.

> or should there be something more application aware (i.e. a daemon specific watchdog) that does this?

I suppose you could do that too, to handle cases where a process is wedged and needs to be restarted, but you solve 99% of the problem with just basic process monitoring using pretty standard techniques.

Crowding out OpenBSD

Posted Nov 14, 2012 22:46 UTC (Wed) by dlang (guest, #313) [Link] (19 responses)

> I suppose you could do that too, to handle cases where a process is wedged and needs to be restarted, but you solve 99% of the problem with just basic process monitoring using pretty standard techniques.

heh, I think people would argue that the sysV init solved 99% of the problem with even simpler tools.

I think that the simple case is handled well by SysV init, the really complex cases are not handled well by anything short of application specific watchdogs/monitoring/cleanup tools, so the question is if there is enough of a gap between these two to make the cgroups requirement (and overhead) worthwhile.

I've gotten to the point that everything I build is now either a cluster, or at least potentially part of a cluster, so I look at anything with the question in mind of 'how would I do this if it was part of a cluster', and this mindset really makes me question the value of trying to have the init system be more clever in addressing this problem.

One of the most common problems you run into when dealing with clusters isn't "the application exited or crashed", it's "The application is running, but wedged" I just don't see many cases where the cgroups approach to managing the app would have prevented problems (although, I freely admit that it has the 'cool' factor)

Crowding out OpenBSD

Posted Nov 14, 2012 23:44 UTC (Wed) by bronson (subscriber, #4806) [Link] (9 responses)

For every cluster deployed, there must be hundreds or thousands of single hosts deployed to serve up wordpress or static content. Simple stuff written by a weekender, maybe with slashdot effects handled by CloudFlare. I think these guys would find systemd's respawning awesome. Much nicer than configuring and tracking down memory leaks in God, that's for sure! http://godrb.com/

Yes, clusters need more development effort and more powerful tools, but don't let that get in the way of the 80% solution.

Crowding out OpenBSD

Posted Nov 15, 2012 1:13 UTC (Thu) by wahern (subscriber, #37304) [Link] (8 responses)

We've already had respawning with inetd. The fact that few used this almost universal facility should make one wonder whether it's really a feature developers don't enjoy reimplementing themselves.

That might seem illogical at first, until you consider how a developer's environment is setup. When developing a daemon you absolutely _hate_ centralized process management, because it tends to get in the way of debugging, among other things. Instead, you basically have to implement some sort of simple process management yourself, regardless of how your production environment will look. (If you're a web developer for a large website, imagine having to develop, debug, test using your process to upload and install in production... it'd be a nightmare.)

I usually start my development on OS X, and then port to Linux and the BSDs. Not once have I considered using launchd.

Crowding out OpenBSD

Posted Nov 15, 2012 1:45 UTC (Thu) by josh (subscriber, #17465) [Link] (7 responses)

inetd tied process spawning to sockets. systemd doesn't; you can independently choose whether to use socket activation and whether to use respawning.

Crowding out OpenBSD

Posted Nov 15, 2012 1:56 UTC (Thu) by dlang (guest, #313) [Link] (6 responses)

inetd tied process spawning to sockets, but there was also inittab for daemons that you wanted to have restarted if they exited or crashed.

Crowding out OpenBSD

Posted Nov 15, 2012 3:33 UTC (Thu) by josh (subscriber, #17465) [Link] (5 responses)

True, and it seems quite unfortunate to me that distributions stopped using inittab for its intended purpose and started using it to launch scripts instead.

Crowding out OpenBSD

Posted Nov 15, 2012 14:47 UTC (Thu) by khim (subscriber, #9252) [Link] (4 responses)

inittab worked as intended couple of decades ago. But when daemons started spawning processes left-and-right and started depend on other processes it become useless.

You can view systemd group-using core as "inittab done right".

Crowding out OpenBSD

Posted Nov 15, 2012 15:34 UTC (Thu) by raven667 (subscriber, #5198) [Link] (3 responses)

Exactly, I see systemd as a return to unix roots, features that init would have had it continued innovating instead of stagnating and standardizing, not some weird departure against the overall style of the system.

Crowding out OpenBSD

Posted Nov 15, 2012 22:04 UTC (Thu) by dlang (guest, #313) [Link] (2 responses)

parts of systemd could be called implementing init properly, but other parts are far more questionable.

why does device enumeration belong as part of init (udev)?

why does logging belong as part of init (the journal)?

Crowding out OpenBSD

Posted Nov 15, 2012 22:44 UTC (Thu) by raven667 (subscriber, #5198) [Link] (1 responses)

>why does device enumeration belong as part of init (udev)?

It isn't part of PID 1 but hardware initialization is a dependency for services being started. The service manager ends up needing to know about hardware state for dependency purposes and therefore shares a lot of infrastructure with udev which is why they are distributed together now.

> why does logging belong as part of init (the journal)?

This is also not part of PID 1 . Whether system logging should be tackled as part of the systemd project is debatable. It is still an optional component though and might solve some real problems.

Crowding out OpenBSD

Posted Nov 15, 2012 22:50 UTC (Thu) by raven667 (subscriber, #5198) [Link]

I thought of one more thing as well, the init system is in a prime position for logging both because it is connected to the stdout/stderr of processes that it spawn, where they might dump data, and it has its own needs for logging but by definition is started before any userspace logging service could be running. So there are at least a couple of cases that a PID 1 and related tools should identify and solve.

Crowding out OpenBSD

Posted Nov 15, 2012 14:45 UTC (Thu) by khim (subscriber, #9252) [Link] (1 responses)

I've gotten to the point that everything I build is now either a cluster, or at least potentially part of a cluster, so I look at anything with the question in mind of 'how would I do this if it was part of a cluster', and this mindset really makes me question the value of trying to have the init system be more clever in addressing this problem.

Funny that you are trying to use cluster-aware approach to oppose cgroups. You do remember that cgroups were invented by Google engineers to manage clusters, right? SystemD just applies the same logic to local system management.

One of the most common problems you run into when dealing with clusters isn't "the application exited or crashed", it's "The application is running, but wedged" I just don't see many cases where the cgroups approach to managing the app would have prevented problems (although, I freely admit that it has the 'cool' factor)

That's strange because I see tons of places where cgroups solve cluster-management problems. What happens with your cluster if some unimportant task start to eat memory or CPU endlessly? With cgroups answer is obvious: it's bound by cgroup's limit. What happens if some task must be killed and restarted because your node is overloaded? Cgroups make sure you can kill the task reliably. Sure, that means that your tasks must be written to be killable, but it makes sense for clusters anyway since failed PSU can do the same thing suddenly and without warning.

Crowding out OpenBSD

Posted Nov 15, 2012 20:53 UTC (Thu) by dlang (guest, #313) [Link]

I didn't say that cgroups did not have a place or a reason to use them.

I just said that requiring cgroups to properly start and stop a process is using the wrong tool for the job.

Crowding out OpenBSD

Posted Nov 15, 2012 16:34 UTC (Thu) by raven667 (subscriber, #5198) [Link]

I wouldn't call cgroups "cool" but I would say that it's a better solution than what's offered in /etc/init.d/functions and ancillary tools like /sbin/pidof. In fact I recently had an outage due to a SysV script that on restart managed to killall rsyslogd but then only start the one instance causing an outage for the other instance on the same host. That kind of unnecessary outage is depressingly easy to have, we can do better.

Crowding out OpenBSD

Posted Nov 22, 2012 14:53 UTC (Thu) by TRauMa (guest, #16483) [Link] (5 responses)

> people would argue that the sysV init solved 99% of the problem

In 15+ years of using linux I never came across a sysv init based system that could reliably restart mysql under all typical production circumstances. Now mysql used (?) to be a close to worst case behaving deamon, but it was also very popular, so if no distributor could handle it correctly with the tools given (and the init scripts tended to be very, very long and complex and hard to debug), how can one argue that it's a 99% solution?

Crowding out OpenBSD

Posted Nov 22, 2012 19:21 UTC (Thu) by hummassa (guest, #307) [Link] (4 responses)

does systemd (and upstart) actually solve this problem?

Crowding out OpenBSD

Posted Nov 22, 2012 19:36 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

Yep. SystemD can reliably track mysql's state and kill it. Mysql recovery should then take care of unclean shutdowns.

Crowding out OpenBSD

Posted Nov 22, 2012 21:37 UTC (Thu) by paulj (subscriber, #341) [Link] (2 responses)

How does it reliably track mysqls state btw? What would define a bad state requiring restart?

IME just because a process is running doesn't mean it's actually in a good state. You need some kind of heartbeat protocol. Does systemd implement something like that?

Crowding out OpenBSD

Posted Nov 22, 2012 22:45 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

No, the common problem with mysql is that it doesn't terminate cleanly. Its shutdown scripts can hang for minutes and _still_ leave hanging processes behind. These hanging processes later interfere with the startup.

SystemD nicely solves this problem. Hooking it up with an external heartbeat monitor should also be quite easy.

Crowding out OpenBSD

Posted Nov 22, 2012 23:25 UTC (Thu) by cortana (subscriber, #24596) [Link]

systemd can be configured to require that a service regularly notifies that it is still up, or else it'll be killed & restarted. See http://0pointer.de/blog/projects/watchdog.html for the details.

Crowding out OpenBSD

Posted Nov 15, 2012 1:27 UTC (Thu) by josh (subscriber, #17465) [Link]

> Why does the init system have to do this job?

That tangent misses the original point. Given a piece of software (systemd) that implements quite a few features by using Linux-specific platform functionality, I hinted at the amount of additional work required to replace even one of those features with a slightly more portable alternative. Responding to that with "do you really need that feature" misses the point completely.

Crowding out OpenBSD

Posted Nov 14, 2012 1:54 UTC (Wed) by Lennie (guest, #49641) [Link] (8 responses)

The creator of systemd specifically made the choice created an architecture leveraging the best parts of the Linux kernel. So the way the program works, depends on the Linux kernel.

It means that systemd should be a fairly small program when compared if it needed the right abstractions and the architecture would probably have been different too.

Wayland is probably the same, I'm sure the code base is very small in comparison to X. For example keyboard handling is mostly handled by the kernel I believe.

I was already wondering how long the Debian project could keep the kFreeBSD port alive. For now in Debian they will probably let the user choose to use systemd or some other init. Maybe systemd will eventually the default with the Linux kernel.

kFreeBSD had ZFS and PF, what other user visible ? Will btrfs fix the biggest feature ?

It used to be true that academia developed new TCP/IP extension on the FreeBSD kernel first, this isn't true anymore. This is now Linux too.

Crowding out OpenBSD

Posted Nov 14, 2012 4:23 UTC (Wed) by abartlet (subscriber, #3928) [Link] (4 responses)

It reminds me very much of OpenSSH, for which we are very much thankful to the OpenBSD project. It too has a focus on just the bare essentials required to run on one OS - OpenBSD in this case - and a 'portable edition' patch for other platforms.

In my own work keeping Samba portable, we have like most large projects, a portability API (libreplace), mostly trying to emulate the C APIs of other OS versions onto Linux APIs where possible. The Kerberos implementation that we work closely with for the AD DC also has libroken, another portability API, and we link against libbsd for strlcpy etc.

I've done a reasonable amount of work in this area recently, and it is certain the portability doesn't come for free. Indeed, we have two major build systems because (in part) we don't want to drop systems that don't have python!

that smell

Posted Nov 14, 2012 5:17 UTC (Wed) by ncm (guest, #165) [Link] (1 responses)

strlcpy? Please tell me you don't actually use that.

that smell

Posted Nov 14, 2012 11:15 UTC (Wed) by abartlet (subscriber, #3928) [Link]

We do use strlcpy. We moved from safe_strcpy() and the custom wrappers fstrcpy() and pstrcpy() (for fstrings of 256 and pstrings of 1024 bytes) to at least something others have seen before. Natrually, a function called safe_ wasn't really safe (just safe against memory overwrite), but the name made us feel better about it despite having the same issues as strlcpy().

As code is rewritten, most strings are dynamically allocated talloc buffers, but stragglers remain.

Crowding out OpenBSD

Posted Nov 14, 2012 9:43 UTC (Wed) by intgr (subscriber, #39733) [Link] (1 responses)

> two major build systems because (in part) we don't want to drop systems that don't have python!

Huh, there are systems powerful enough to build Samba that don't run Python?

Crowding out OpenBSD

Posted Nov 14, 2012 13:22 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

Yes, old big-iron batshit-crazy OSes in locked rooms without Internet access. Apparantly, Volker Lendecke has to provide support for these kinds of customers.

Crowding out OpenBSD

Posted Nov 15, 2012 2:53 UTC (Thu) by nix (subscriber, #2304) [Link] (2 responses)

The creator of systemd specifically made the choice created an architecture leveraging the best parts of the Linux kernel.
cgroups? Best parts? Only from the perspective that it provides a feature that you can't get any other way. Not from the perspective that it's a crawling horror, nor, indeed that it's considered any good by any of the kernel developers. :)

Crowding out OpenBSD

Posted Nov 16, 2012 21:34 UTC (Fri) by Tet (subscriber, #5433) [Link] (1 responses)

Not from the perspective that it's a crawling horror, nor, indeed that it's considered any good by any of the kernel developers. :)

I've heard that claim many times. But I'm too far removed from what's going on in kernel space these days. So if they're as bad as people are claiming, what alternatives does the kernel provide that give the same functionality as cgroups?

Crowding out OpenBSD

Posted Nov 18, 2012 15:39 UTC (Sun) by nix (subscriber, #2304) [Link]

None. That's why they get used, even if they *are* a crawling horror internally.

Crowding out OpenBSD

Posted Nov 14, 2012 1:51 UTC (Wed) by clump (subscriber, #27801) [Link]

Linux development has accelerated to the point that some other OSes don't have enough sparse developer time to keep up. Yelling "wait up" doesn't change that, it just slows Linux down so that other OSes have a better chance of keeping up with it.
Well said. I report a lot of bugs in Linux, and I can certainly say that the ones I report for RHEL are looked at far more often than the ones I report for Fedora. It's certainly no conspiracy, it's a matter of resources.

To your point, I think the larger issue is that Linux has taken the dominant free OS role. The article rightly points out that certain projects are less concerned with portability than others, though with the success of Linux it's going to be hard for others to keep up.

Crowding out OpenBSD

Posted Nov 14, 2012 12:05 UTC (Wed) by mezcalero (subscriber, #45103) [Link] (9 responses)

rleigh, it's really not as easy as you think. Making the event loop portable to kqueue is complex, but doable, I can agree to that. -- But the trouble starts beyond that. The BSDs don't have anything like cgroups. There's no way to attach a name to a group of processes, in a hierarchal, secure way. And you cannot emulate this. (And no, don't say "BSD jail" now, because that is something very different). But this already is at the very core of systemd. It's how systemd tracks services. And then, a big part of systemd is to track device status, as they come and go. Device management is fundamentally different on all OSes, you get very very different semantics. On Linux we have the udev rules set, which is entirely lacking on other OSes. systemd integrates very closely with that.

Portability of the most basic building blocks of the OS userspace goes way beyond just replacing a few system calls, it massively changes the semantics you expose, the behaviour you can rely on, and even becomes visible in the APIs you export.

Really, you should try porting this stuff, and you will run against a series of walls, and you cannot climb over them, as you will notice. When you start porting, don't start with the easy ones, such as epoll(), start with the hard ones, that are actually at the core, start with cgroups, start with device management. And you will notice that portability of this code is not just a question of caring, it's a massive undertaking.

Crowding out OpenBSD

Posted Nov 14, 2012 12:34 UTC (Wed) by clump (subscriber, #27801) [Link] (2 responses)

I suppose the relevant question to ask you is if anyone has come forward to help systemd get ported to different operating systems?

Crowding out OpenBSD

Posted Dec 5, 2012 1:27 UTC (Wed) by trasz (guest, #45786) [Link] (1 responses)

Not really. The relevant question would be why would anyone want systemd ported to different operating systems.

Crowding out OpenBSD

Posted Dec 18, 2012 13:17 UTC (Tue) by clump (subscriber, #27801) [Link]

Not really. The relevant question would be why would anyone want systemd ported to different operating systems.
Or the relevant question could be exactly what I asked. If you have to ask why, you didn't read the article or the thread.

Crowding out OpenBSD

Posted Nov 14, 2012 16:56 UTC (Wed) by mjthayer (guest, #39183) [Link]

> There's no way to attach a name to a group of processes, in a hierarchal, secure way. And you cannot emulate this.

One could require that administrators should create private users for daemons if they wanted to be able to shoot them down reliably. Not saying you should have gone this way, just that it would be another way of handling it.

> Device management is fundamentally different on all OSes, you get very very different semantics.

Yes, that can be a lot of fun.

Am I right though in thinking that the interfaces through which processes higher up in the stack communicate with systemd are normal DBus ones on the whole? If so then portability issues caused by software on Linux depending on systemd (I mean GNOME panel-like software) could presumably also be solved by making native infrastructure on the target system present those interfaces too.

Crowding out OpenBSD

Posted Nov 14, 2012 17:15 UTC (Wed) by Trelane (subscriber, #56877) [Link] (2 responses)

Perhaps also the comparison with now is shortsighted. Yes, it would take a lot of things to port systemd to a BSD. However, it's not like the features that were developed were developed all at once. The pieces on which systemd relies have been developed over time, and the BSD folks in some cases actively chose not to develop similar or compatible systems. In other cases, it was a more passive decision e.g. deciding to focus developer time in other places. Though of course hindsight is 20/20 and it's hard to predict what features the next cool new thing will rely on, so it is oftentimes also understandable.

But the main message is that the entirety of systemd is not just systemd itself, and the majority of things that make it distinctive and non-portable were developed for Linux and not developed on the BSDs over a large swath of time, not all at once.

It should be noted that similar problems would likely arise from trying to port it to aix, hpux, solaris, OSX, or HURD (I suspect you know a bit more about what that would entail than I ;). But the BSDs are not special in this regard either.

(Incidentally, why the complaint about Linux instead of e.g. OSX's proprietary systems? Why are there not outcries against a lack of e.g. Aqua for BSD? Though perhaps this has been raised and answered elsewhere and I've not seen it.)

Crowding out OpenBSD

Posted Nov 14, 2012 18:44 UTC (Wed) by jwakely (subscriber, #60262) [Link] (1 responses)

> why the complaint about Linux instead of e.g. OSX's proprietary systems?

I thought the same thing. I wonder how many *BSD users and devs were lost to Mac OS X. The BSDs are benefiting from LLVM and Clang, but I don't know if the same is true of other tech Apple have built on top of a BSD kernel.

BSD developers' moves to Apple?

Posted Nov 19, 2012 22:38 UTC (Mon) by dmarti (subscriber, #11625) [Link]

Just what I was going to ask. (Someone asked me what happened to *BSD user group meetings, and I guessed that they're still going on, but they're in a conference room in Cupertino and nobody who goes is allowed to talk to you.)

Crowding out OpenBSD

Posted Nov 15, 2012 0:55 UTC (Thu) by geofft (subscriber, #59789) [Link] (1 responses)

Silly question: has anyone tried adding cgroup support to a BSD kernel (instead of trying to make systemd not use cgroups)?

Crowding out OpenBSD

Posted Dec 5, 2012 1:18 UTC (Wed) by trasz (guest, #45786) [Link]

There is no point in doing it. For simple purposes, you're better off with login classes. For complicated - with jails. There is point in adding third mechanism.

Crowding out OpenBSD

Posted Nov 14, 2012 0:45 UTC (Wed) by smurf (subscriber, #17840) [Link] (26 responses)

There's two rather simple reasons the BSDs can't compute with Linux.

One: There are (at least) three of them.

The BSDs, or rather their coders, spent a lot of time on one-upmanship, internal divisiveness, and whatnot. That effort could have been spent on improving one single code base.

Two: Linux is just a kernel. The BSDs are a whole distribution.

Linux kernel people concentrate on the kernel and let others worry about userland. BSD people do it all by themselves, so their efforts are spread more thinly.

No wonder that they fell behind.

NB: There may be a third reason. Licensing. But I don't want this discussion to turn into yet another GPL-vs.-BSD flame fest.

Crowding out OpenBSD

Posted Nov 14, 2012 4:24 UTC (Wed) by felixfix (subscriber, #242) [Link]

While I like the idea in general of the more kernels, the merrier, part of me can't help but feel that the BSD groups are reaping what they sowed. BSD split for various reasons which seemed more related to being "pure", so to speak, than to fundamental incompatibilities, preferring to be kings of their small ponds rather than working together. Noble in some ways, petty in others. To be jealous (if that is the right word) of Linux, or upset that it's grabbing the lion's share of developer attention, seems more like they are getting what they asked for.

I have a hard time visualizing any of the BSDs being happy at all the changes made in the Linux universe. I rather think they'd turn up their noses and reject the impurities.

I have no personal experience with developers in any camp. But the squabbles from the various BSDs all seemed far more petty and even snobbish, so again, they seem to have what they wanted, and if they don't want it now, maybe it's not too late to be more open to each other. The first step would seem to be unifying the BSD variants, but I don't imagine that's a particularly easy job, either politically or code-wise.

Or maybe the BSD folks are happy in their small camps. It's their code, their choice.

Crowding out OpenBSD

Posted Nov 14, 2012 8:30 UTC (Wed) by marcH (subscriber, #57642) [Link] (24 responses)

> NB: There may be a third reason. Licensing. But I don't want this discussion to turn into yet another GPL-vs.-BSD flame fest.

OK, I'll do it.

BSD licenses are just too generous. BSD-licensed projects give all their code away to all other projects (GPL and not) while not being able to take code from anywhere. Where can this go?

While this will _obviously never happen_, if one BSD project were to switch their license to GPL then they could start cherry-picking all the pieces they need from Linux, "upgrade" them to their supposedly superior technical standards and save themselves from obsolescence. The whole public would benefit.

Crowding out OpenBSD

Posted Nov 14, 2012 10:10 UTC (Wed) by Lennie (guest, #49641) [Link]

That is a silly idea, many BSD developers specifically choose to go with BSD because they don't like the GPL license.

It's probably easier for Linux developers to take code from BSD.

And even that doesn't happen as often as you'd expect.

I wouldn't be surprised if in the Linux kernel there are more "shared development" drivers with FreeBSD (a driver that has a BSD-license or dual license and development on on the same code happend by Linux developers in the Linux kernel and FreeBSD developers on FreeBSD) than BSD-code that was copied.

Crowding out OpenBSD

Posted Nov 14, 2012 10:17 UTC (Wed) by ortalo (guest, #4654) [Link] (5 responses)

One amusing thing is the fact that, currently, I think the BSDs are lacking *paid* developpers, while Linux has (never) enough.
It's astonishing to see that, currently, the licenses used have the opposite of their (supposed) effect.

At the beginning of the eighties, it was opposite (the BSD trial not withstanding). BSD4.4 was surviving via Sun and BSD Inc. thanks to paid development (allowed by the BSDL); while Linux was entirely relying on free or students time.

Crowding out OpenBSD

Posted Nov 14, 2012 13:32 UTC (Wed) by marcH (subscriber, #57642) [Link] (3 responses)

I think this is about GPL project(s) reaching a critical mass.

Companies obviously prefer BSD... until the competing GPL project(s) become so successful they can't avoid it any more.

Every time a company exerts its short term BSD right not to contribute back they indirectly help the GPL competition if any. And since companies' "vision" never sees further than a couple of years...

Crowding out OpenBSD

Posted Nov 14, 2012 16:52 UTC (Wed) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

I'm not sure that companies do prefer BSD licensing. BSD licensing is great if your goal is to make something popular to gain networking effects (e.g. media codecs) or if you're a big player and want to be able to build proprietary systems off the code base. But the GPL seems to be better for the case where you have lots of companies involved, none of which is big enough to dominate the market. The enforced share and share alike provisions of the GPL give companies the reassurance that they aren't going to be taken advantage of by a competitor who uses their contributions while keeping their own improvements proprietary.

Crowding out OpenBSD

Posted Dec 5, 2012 8:23 UTC (Wed) by hummassa (guest, #307) [Link]

The primary examples here being Apple and Google, you really outperform the sum of all counter examples...

Crowding out OpenBSD

Posted Nov 16, 2012 15:07 UTC (Fri) by smurf (subscriber, #17840) [Link]

> until the competing GPL project(s) become so successful they can't avoid it any more.

Unless you're Google. Cf. the largish parts of Linux infrastructure that were rewrittten to be without GPL in Android.

Crowding out OpenBSD

Posted Nov 14, 2012 15:41 UTC (Wed) by ortalo (guest, #4654) [Link]

s/eighties/nineties/

Crowding out OpenBSD

Posted Nov 14, 2012 20:50 UTC (Wed) by Richard_J_Neill (subscriber, #23093) [Link] (13 responses)

You're right that BSD code can be relicensed as GPL, and not vice-versa. But usually when a GPL-minded developer takes significant inspiration from a BSD program, he will either make the whole thing BSD-licensed, or push back his changes under dual-license.

Crowding out OpenBSD

Posted Nov 14, 2012 21:03 UTC (Wed) by dlang (guest, #313) [Link] (12 responses)

usually, but not always (and the BSD crowd really howl when it happens, they are fine with people taking their code and putting it in proprietary codebases, but release it under a GPL license, and just wait for the explosion)

Crowding out OpenBSD

Posted Dec 5, 2012 1:16 UTC (Wed) by trasz (guest, #45786) [Link] (11 responses)

It's simply because commercial vendors using BSD often give something back, and GPL projects don't.

Crowding out OpenBSD

Posted Dec 5, 2012 3:56 UTC (Wed) by rahulsundaram (subscriber, #21946) [Link]

That's a gross over-generalization. There is some amount of hypocrisy about BSD licensed projects complaining about not giving back as well.

Crowding out OpenBSD

Posted Dec 5, 2012 10:14 UTC (Wed) by man_ls (guest, #15091) [Link] (9 responses)

Given that commercial vendors using GPL always give something back, not often, then I would say the GPL is better designed than BSD licenses. BSD is designed so GPL projects have the freedom not to give back, and they sometimes (not always) exercise it. Why complain?

Crowding out OpenBSD

Posted Dec 5, 2012 10:44 UTC (Wed) by njwhite (guest, #51848) [Link] (2 responses)

> commercial vendors using GPL always give something back

Yes, but what they give is very frequently not useful. I'd argue that a culture of contribution to a common core is much more important than the legal requirement, overall. Also GPL is often sidestepped so that the actual useful stuff isn't shared anyway, even when it probably should be (c.f. kernel modules).

Crowding out OpenBSD

Posted Dec 5, 2012 10:54 UTC (Wed) by man_ls (guest, #15091) [Link] (1 responses)

History says otherwise. A few notable examples:
  • Apple has a long culture of non-contribution (they created their own C compiler to avoid GCC's license), but still they adopted KTHML as WebKit and have been improving it all along, even when their mortal enemy Google adopted it too.
  • Many companies that were previously averse to GPL code (and Free software in general) are now contributing actively to the Linux kernel. Just in the 3.7 kernel we find Oracle, AMD (previously ATI), Marvell or Cisco.
For every Nvidia there are many Apples.

Crowding out OpenBSD

Posted Dec 5, 2012 12:38 UTC (Wed) by njwhite (guest, #51848) [Link]

True, the kernel is a pretty strong counterexample. And the requirement to contribute back probably can be a good way of getting good cultures set up, telling a manager "we have to make the code available anyway, we may as well work upstream" is probably helpful.

Crowding out OpenBSD

Posted Dec 5, 2012 11:27 UTC (Wed) by trasz (guest, #45786) [Link] (5 responses)

Commercial vendors using GPL always give back, because if they don't want to give back, they don't use GPL software. So, with both BSD and GPL, commercial vendors give back only when they want to; the difference is that with BSD, they can still use the software.

And the complaint is because commercial vendors _do_ give back to BSD projects, either the code, or money, as donations or salaries. GPL projects reusing BSD code very rarely contribute anything back.

Crowding out OpenBSD

Posted Dec 5, 2012 16:11 UTC (Wed) by nix (subscriber, #2304) [Link]

Er, you can still use GPLed software without giving anything back. What you can't do is redistribute it.

Crowding out OpenBSD

Posted Dec 5, 2012 22:45 UTC (Wed) by marcH (subscriber, #57642) [Link] (3 responses)

> And the complaint is because commercial vendors _do_ give back to BSD projects, either the code, or money, as donations or salaries.

Sure some do. Most of them? [reference needed]

> GPL projects reusing BSD code very rarely contribute anything back.

... to the original, BSD-licensed project.

GPL projects "merely" contribute back to *any* person or company not allergic to the GPL, for free. Not as generous as the BSD license but much more generous than no contribution at all.

Crowding out OpenBSD

Posted Dec 6, 2012 1:13 UTC (Thu) by raven667 (subscriber, #5198) [Link] (2 responses)

>> GPL projects reusing BSD code very rarely contribute anything back.
>... to the original, BSD-licensed project.

That seems a little crazy. IIUC any patches to BSD licensed code in a GPL project can be distributed under the BSD license back to the BSD project by the original patch author. The author can always license code to different parties under any license they wish. What's stopping people from contributing code back to BSD projects?

Crowding out OpenBSD

Posted Dec 6, 2012 1:51 UTC (Thu) by dlang (guest, #313) [Link] (1 responses)

Once the BSD code is part of a GPL project, any new patches are by default GPL (the default license of the project). It takes extra thought and effort to make the patches be BSD

In addition, many GPL people believe that it's a much better license than BSD, and so they _really_ want their work under the GPL.

If either of these are the case, then the patches don't get contributed back under BSD

Crowding out OpenBSD

Posted Dec 6, 2012 2:39 UTC (Thu) by raven667 (subscriber, #5198) [Link]

> Once the BSD code is part of a GPL project, any new patches are by default GPL (the default license of the project). It takes extra thought and effort to make the patches be BSD

I realize that, it seems like a poor reason. Maybe git needs an "export as bsd" option for ones own code 8-)

Crowding out OpenBSD

Posted Nov 15, 2012 22:52 UTC (Thu) by cmccabe (guest, #60281) [Link] (2 responses)

A lot of open source projects have been successful under a BSD-like license. Hadoop, for example. Hadoop is under the Apache 2.0 license, which is basically BSD + patent grant. LLVM is under a BSD-like license, as well as all of Android.

Crowding out OpenBSD

Posted Nov 16, 2012 5:13 UTC (Fri) by marcH (subscriber, #57642) [Link] (1 responses)

Except for LLVM these never had any serious GPL competitors had they?

gcc's technical and governance problems have been extensively discussed here - of course the license is far from the only factor. But i still think it can make a difference in a close competition.

Crowding out OpenBSD

Posted Nov 17, 2012 22:11 UTC (Sat) by cmccabe (guest, #60281) [Link]

I think you may be assuming what you set out to prove. X succeeded; therefore X never had any "serious competitors." I could equally well say that Linux succeeded because it never had any "serious competitors."

During Linux's formative years, the BSDs were hamstrung by a serious of lawsuits. Afterwards, they splintered into many different groups. Neither of these two things had anything to do with GPL vs. BSD. We could easily have had 5 small Linuses rather than one big one. The GPL doesn't prevent forks in the slightest.

The GPL is also less risky in the kernel. If your company writes some userspace code, there is zero chance that it will be considered a derived work of the kernel. (Linux even has a special preface to the GPL clarifying this.) With userspace code, things are a lot less clear. This is one of the reasons why Google banned GPL in userspace on Android. The Oracle vs. Google ruling that APIs are not copyrightable helps a little bit here; however, there are still a lot of ambiguities. What does the LGPL even mean when used in a programming language that doesn't have a linker? etc.

Personally, I mostly work on Apache 2.0 code at work these days. For my own projects I usually use Apache 2.0 or 2-clause BSD. The advantage of the latter is that it is usable in GPLed or other-licenced projects as well.

Crowding out OpenBSD

Posted Nov 14, 2012 1:13 UTC (Wed) by Nahor (subscriber, #51583) [Link] (9 responses)

Weird article. You mention how one can poke holes to Mark's complaint but this article has as many holes.

You talk of the Linux monoculture but what about the Unix/POSIX monoculture? Why is BSD dying worse than the death of CP/M, DOS, AmigaOS, ...?
You say that your humble readers should care about BSD's death, that BSD matters. But if it dies, it is *because* it doesn't matter.
You say that it is an important part of the free software ecosystem. But if it dies, it's because it's not important anymore.
You say that we should care because BSD has many talented developers. Those developers won't die with BSD, they can move on to Linux or Haiku, or even Windows.

And maybe one day, this will happen to Linux too. The world changes, the world evolves. Darwin's theory at its best. The king is dead, long live the king.

Crowding out OpenBSD

Posted Nov 14, 2012 3:59 UTC (Wed) by ncm (guest, #165) [Link]

If it dies, it is because it doesn't seem matter *enough*, *at the moment*. It's easy to imagine circumstances in which one would prefer that it had not died, after it has. Avoiding the pain that would be felt at that time takes some attention today. We lack an effective mechanism to propagate that pain backward, and can only substitute thoughtful foresight.

Crowding out OpenBSD

Posted Nov 14, 2012 10:06 UTC (Wed) by ortalo (guest, #4654) [Link] (3 responses)

BDS4.4 has already died. Its first child 386BSD has died too.
Of the three children of the next generation, it can probably also be said that NetBSD is really not in good shape. OpenBSD is still fighting (that's the warrior of the family... he tries hard to die on the battlefield). Did not have much news from FreeBSD recently (it was at google last time I checked). I hope he's well.
DragonflyBSD is the newest generation IMHO, with lot of brilliant new ideas; but still has to prove itself in the field. (Again, did not check for a couple of years.)

On the darwinist path, Linux is the one which is still single without legitimate children... ;-)

Crowding out OpenBSD

Posted Nov 14, 2012 11:25 UTC (Wed) by hummassa (guest, #307) [Link] (1 responses)

> NetBSD is really not in good shape

[citation needed]

I haven't had it installed recently, but I probably will in a few months (raspberry pi support coming!). So I would like to know why you think that...

Crowding out OpenBSD

Posted Nov 14, 2012 15:49 UTC (Wed) by ortalo (guest, #4654) [Link]

Sorry if I alarmed you as it's only gossip: someone told me (no names of course). I think he will be pretty happy to be proved wrong: please go on with NetBSD on the Pi.

Crowding out OpenBSD

Posted Nov 15, 2012 22:57 UTC (Thu) by cmccabe (guest, #60281) [Link]

But has Netcraft confirmed it?

Fate of the *BSDs

Posted Nov 14, 2012 10:20 UTC (Wed) by man_ls (guest, #15091) [Link] (3 responses)

Those holes are easy to plug, even without being our esteemed editor.
You talk of the Linux monoculture but what about the Unix/POSIX monoculture? Why is BSD dying worse than the death of CP/M, DOS, AmigaOS, ...?
Because BSD is Unix and it is Free software; BSD history is an essential piece of the grand tradition that has brought us Linux, GNU userland and all distros. BSD (or rather the *BSDs) are also "serious" OSs in areas where those others you mention are no competition, such as multiuser support.
You say that your humble readers should care about BSD's death, that BSD matters. But if it dies, it is *because* it doesn't matter.
You have a point, but many worthy software packages are in danger or unmaintained even though they matter a lot -- at least to some people.
You say that it is an important part of the free software ecosystem. But if it dies, it's because it's not important anymore.
That would be like saying that extinct species deserve their fate because they have not adapted to their ecosystems. There are external forces that have great influence on any ecosystem, in our case market pressures or company buyouts.
You say that we should care because BSD has many talented developers. Those developers won't die with BSD, they can move on to Linux or Haiku, or even Windows.
Precisely. Having those developers moving to Windows or Mac OS out of spite would be a loss from our point of view (i.e. Free software).
And maybe one day, this will happen to Linux too. The world changes, the world evolves. Darwin's theory at its best. The king is dead, long live the king.
Having some alternatives is good in case you need them. It is a classic case of generalists vs specialists: the specialist is better suited to its niche. But the niche can disappear at any moment killing the specialist in the process. The generalist can also invade the niche. Looking at evolution there is not one single solution, and neither is there in Free software OSs.

Fate of the *BSDs

Posted Nov 14, 2012 11:56 UTC (Wed) by ortalo (guest, #4654) [Link] (2 responses)

Very interesting. I am not sure the *BSD would like to see themselves as specialists however. They certainly want to be generalist OSes. However, this is certainly a flamewar-like issue, and I do not want to enter into it.

The niche issue is making me think a lot. Are there available niches in computing for the various BSDs? What is the status of their current ones?

OpenBSD selected the security niche. For me, it *seems* that Linux invades it, but that is not actually the case. Linux security is good. But OpenBSD targets paramount security and Linux in general avoids paranoid users. [1]
The real problem with this niche IMHO is that it does not pay. Funding a big development for security is probably doomed (if someone knows a way, please, contact me directly... ;-).
At some time, it seemed that they also targetted a niche that can be called "networking OS": via the addition of OpenNTP, OpenBGP, etc. Maybe it is still the case but it is a difficult niche too: Cisco iOS is there, as well as many of the networking equipment manufacturers. New big players are either developping their own proprietary solution or starting with Linux which has a good reputation (to say the least) in networking. Finally, like with any other big company, the "security" argument is definitely more for marketing than for real so it is not so decisive...
On the smaller scale, OpenBSD does not seem to want to go on small systems (WiFi boxes, SAN, etc) and OpenWRT&co. is already in that niche.
However, I still think this is a nice idea. Networking is really a place where paramount security would be useful, because it would allow us to forget about it, but not in big networks... Personnally, I would select home security as the niche here (networked home of course;-); it may be the only one that really pays (look at the price of that home alarm system...). But that involves porting to small hardware and well, it seems that's a big step.
There is also potential interaction with mobile devices (smartphone, etc.) - a niche to avoid because of all the predators in it... However, in the longer term it opens the path to industrial control systems and there, the "security" thing could have the same effect as in the home and bring back momentum (only in networking if Linux gets there first as he will probably do).

FreeBSD selected the performance on x86 niche last time I checked. It worked for some time, but Linux is a generalist with a passion for performance too so... Hopefully the niche is big enough for several players to exist. (I wouldn't bet on that, but I am no good investor at all.)

DragonflyBSD selected innovation. Well, nice, while you can have new ideas to innovate with. But no need to worry for the short term: computer science is just 40 years old, new ideas are still plenty. Skilled developpers to implement them are available too (and do not get highly paid jobs in the industry anymore). Passion is a good niche after all (without the drawbacks of paranoia... ;-). Getting out of it is the problem (NB: for paranoia too... ;-)).

NetBSD selected portability. Well, it would be interesting to analyze why it was not selected to be on smartphones and tablets then.
I do not know enough of this OS to speak about it however.

So what are we left with? What are the niches that would allow *BSD not only to survive, but to develop and possibly come back competing with other generalists? At least, that's a nice way to look at the issue. Thanks for the comment.

[1] Linux with some specific patches does compete with OpenBSD but probably lack the "full OS" and "security first moto" advantage of OpenBSD. (Ouch, I know some other readers will want react to that paragraph: sending to footnote... Please comment separately. ;-)

Fate of the *BSDs

Posted Nov 14, 2012 16:35 UTC (Wed) by andreasb (guest, #80258) [Link]

> NetBSD selected portability. Well, it would be interesting to analyze why it was not selected to be on smartphones and tablets then.

While it may have the widest hardware support of the BSDs, it certainly can't hold a candle to Linux. They always boasted about the large number of supported platforms aka ports, whereas in Linux you would usually count architectures. Then it doesn't look so good: NetBSD supported platforms list contains 10 architectures including one that Linux doesn't support (VAX), Linux supports 27 if I counted correctly.

Even looking at platforms it doesn't look very exciting. Collected under ARM you find many StrongARM, XScale, and ARM9 but not a single Cortex (though there does appear to exist a port to the ARM11 Raspberry Pi).

Without researching it right now I would say that Linux has eclipsed NetBSD in the portability business at least a decade ago.

Fate of the *BSDs

Posted Nov 14, 2012 16:52 UTC (Wed) by man_ls (guest, #15091) [Link]

Thinking a bit about it, software and life forms are probably very different. Perhaps software is much more adaptable than DNA and the generalist always tends to win, because it can adapt to multiple environments at the same time.

Google, Amazon, Facebook, Twitter... At least on the net there is not an "ecosystem"; almost everything turns to a monoculture with ever-increasing speed. There is no room for competitors, and the generalist tends to win (or perhaps the specialist can generalize without losing adaptative power).

If this is the case, then the *BSDs don't have a future anywhere, any more than the OSs mentioned in grandparent (Amiga, CP/M) or than relics like Multics or Unix System V. Linux (the kernel) is eating their lunch and GNU userland is the future.

But somehow this doesn't seem to be the case, particularly in the Linux userland: KDE and GNOME have prospered for a long time, right now we have llvm, Firefox and Chrome coexist with proprietary variants... So perhaps they do have a future and all they need is even more specialization. Or a different scope. What is clear is that as generalist OSs they have lost the match.

Crowding out OpenBSD

Posted Nov 14, 2012 1:31 UTC (Wed) by geofft (subscriber, #59789) [Link]

I am intrigued by this comment from the "some occasional pain" thread from 2008, by the former DRM maintainer for FreeBSD who had by then had defected to Linux:

What I'm realizing now, is that the process I tried to maintain with shared-core was hopeless. We had this "shared" tree, that actually only ever shipped on one OS -- FreeBSD. Linux distros shipped distro DRM plus a few backports of drm.git stuff, NetBSD ships their own port that isn't in drm.git, OpenBSD ships their own port that isn't in drm.git, and OpenSolaris ships their own port that isn't in drm.git. So this drm.git tree was only ever being shipped wholesale on FreeBSD when I ran my little sed script to import it into FreeBSD.

What we're doing is acknowledging that the drm.git code was only for release on one out of the 5 OSes that DRM exists on. The developers for that OS should bear the burden of porting to their OS. Linux developers aren't too interested, shockingly, in writing code to an API to allow FreeBSD to release their code, when they can't even integrate that code they write into their own kernel for release!

Crowding out OpenBSD

Posted Nov 14, 2012 2:14 UTC (Wed) by davem (guest, #4154) [Link] (2 responses)

The divisive nature of the BSD projects have done more than enough to ensure the eventual irrelevance of these projects. Having limited resources for development is one thing, splitting those resources between at least 4 different forks (Net,Free,Open,Dragonfly) is yet another.

Given that, trying to claim that GNOME, systemd, or PulseAudio not staying portable enough is going to cause their demise is an exercise in absurdity if you ask me.

I really appreciate the nice sockets API we got from the BSDs, but their extinction frankly cannot come fast enough and is long overdue.

Or, maybe they should stick around, so that developers who don't want to work on a system that gains more than a million new users every single day have some place to play.

Crowding out OpenBSD

Posted Nov 14, 2012 9:57 UTC (Wed) by ortalo (guest, #4654) [Link]

I do not consider my preference in the field of security for OpenBSD over a Linux distribution (second place btw, not so bad and possibly closing up) to be of the "play" style. It's even essentially a professional opinion.

And it's not only technical. It has its origin in the design objectives of the respective OSes; and I find *both* policies pretty reasonable (for different use cases of course).

Crowding out OpenBSD

Posted Nov 15, 2012 3:06 UTC (Thu) by nix (subscriber, #2304) [Link]

I really appreciate the nice sockets API we got from the BSDs
What? It's half-baked and non-Unixy, as ugly in its way as SysVIPC and for many of the same reasons, right down to implementing new read and write primitives for non-stream transports (though at least the stream transports use read() and write() like everyone else, the sole good decision in the whole sorry mess). Something properly filesystem-backed would be much more the Unix Way(TM). As in so many other areas, Plan 9 got this right.

IMNSHO, of course.

Crowding out OpenBSD

Posted Nov 14, 2012 3:41 UTC (Wed) by mrshiny (guest, #4266) [Link] (13 responses)

BSD may have been a source of experimentation in the past, when it was a mature and flexible environment for doing so, but is there any experimentation that can't be done on Linux?

* Filesystems: Linux has like a bajillion filesystems. I think experimentation in that department is quite healthy and I'm not sure that another posix system is a better candidate for this kind of work.
* Kernels: There have been many prominent forks of the Linux kernel or experimental features. The Realtime kernels are a good example. Otherwise, for even more drastic experimentation, it seems that even writing a basic kernel which serves as a proof of concept is pretty darn hard. See Hurd, Minix.
* Security: The Linux kernel has tons of security features including some that were subjects of quite a bit of security research, eg, SELinux.
* Packaging: There are a handful of BSDs, but orders of magnitude more Linux distros, all of which experiment with packaging and userland configurations.
* Totally different things: Android, Wi-Fi routers, TVs, BluRay players, GPS devices, servers, workstations, mainframes, basically any form factor and any architecture you want has a fork of Linux. Some of those forks are pretty different from each other.

In some ways, Linux is a sort of monoculture. But in other ways, it certainly is not. I'll be as disappointed when OpenBSD dies as I would be when Gentoo Linux dies, or Slackware, or Mandrake/Mandriva/WhateverItsCalled dies. Every major project that has had a significant number of users at some point has been responsible for advancing things. But they die because they stopped being relevant, and because their mission was picked up by other projects.

Experimenting on the BSDs

Posted Nov 14, 2012 21:53 UTC (Wed) by man_ls (guest, #15091) [Link] (12 responses)

Navarro et al had huge pages working on FreeBSD in 2007. Linux had to wait until 2010. And Alan Cox himself was in the original team that did the work on FreeBSD.

Experimenting on the BSDs

Posted Nov 15, 2012 4:19 UTC (Thu) by rsidd (subscriber, #2582) [Link] (11 responses)

FreeBSD's Alan L. Cox != Linux's Alan Cox. Different continents.

Experimenting on the BSDs

Posted Nov 15, 2012 8:26 UTC (Thu) by man_ls (guest, #15091) [Link] (10 responses)

My mistake. This other Alan L. Cox even appears on the list of FreeBSD contributors; the human namespace is a bit crowded these days. Anyway, even without the irony FreeBSD won by a few years here.

Experimenting on the BSDs

Posted Nov 15, 2012 16:36 UTC (Thu) by khim (subscriber, #9252) [Link] (9 responses)

But what does it change? Decade ago *BSD had tons of advantages over Linux, but they all concentrated on smaller niches and eventually Linux caught up. Linux is still not on par in select few niches, but their number is dwindling.

At some point is stops being feasible to support the whole separate OS just for these few separate narrow niches.

Experimenting on the BSDs

Posted Nov 19, 2012 9:16 UTC (Mon) by ortalo (guest, #4654) [Link] (8 responses)

Unless the dominant OS acknowledges that the alternatives are better and that the situation is going to persist because it does not select that as priorities. It would allow the dominant OS to prioritize other objectives (possibly bigger than those the alternatives could pretend to).

Ok, that's probably a form of subsiding, helping or something like that. And then?
If one has to choose between evolving Linux into a shark or a dolphin; my choice is taken. (And, by the way, there are already plenty of sharks in the sea, may not be so safe in the long run.)

Experimenting on the BSDs

Posted Nov 20, 2012 9:49 UTC (Tue) by mpr22 (subscriber, #60784) [Link] (7 responses)

An analogy that falls somewhat flat given that all dolphins are predators and several dolphin species are apex predators, while there are sharks that only eat plankton :)

Experimenting on the BSDs

Posted Nov 20, 2012 12:01 UTC (Tue) by man_ls (guest, #15091) [Link] (6 responses)

And given that penguins are birds, the only way forward is evolving into mammals, not back to being fish :)

Experimenting on the BSDs

Posted Nov 20, 2012 17:32 UTC (Tue) by viro (subscriber, #7872) [Link] (5 responses)

*blink* WTF is "forward" and why would mammals be that compared to archosaurs? Deeply inferior lung design, lousy eyesight, just to start with[1]...

[1] in fact, human eyesight is amazingly good for mammals - more or less on par with medium-sized eagles; still losing to big ones, but not by much. Most of the mammal species are nowhere near that resolution. Horse is about 4 times worse, cat - about 10, and those are relatively good. Anthropoids are very unusual in that respect - we'd managed to re-evolve fairly decent eyes. We even got part of the colour vision back; still only 3 receptors instead of the normal 4, though ;-/

Mammal evolution, literally

Posted Nov 20, 2012 21:39 UTC (Tue) by man_ls (guest, #15091) [Link] (4 responses)

Good point, and one that I cannot resist. It is hard to compare between different living groups: birds may have a slight advantage in eye resolution, but hardly in nocturnal sensitivity or in fact in many other senses: hearing (bats, dolphins), smell (pigs, dogs) or taste are arguably topped by mammal representatives, owls notwithstanding.

In fact adaptation to different ecosystems is the only measure of success that we may objectively use. We might focus on how our group of previously dwarvish creatures (less than a metre long) have invaded almost all ecosystems, often becoming their apex predators. Birds have not colonized the seas, for example, other than flying above them; and yet they have a whole continent (Antarctica) to them.

If we follow this line of reasoning, we might argue that bacteria are the living champions and that we should devolve to single cell organisms, since we have 10 times more bacterial cells in our bodies than human cells...

But the particular thing I had in mind was that crazy habit of laying eggs around and then abandoning small birds to their luck after at best a few weeks. This is way better than fish just lying eggs around, but much worse than our viviparous system. By "evolving into mammals" I was specifically thinking about breast-feeding their offspring, which allows us mammals to create affective bonds and teach every new generation the old tricks. That is the indubitable advantage of primates which humans have taken to an extreme, by living several decades with their parents.

So our friendly penguin-like operative system should evolve towards the breast-feeding dolphin and not backwards to the offspring-eating shark, even if some of them are viviparous.

Mammal evolution, literally

Posted Nov 22, 2012 7:25 UTC (Thu) by viro (subscriber, #7872) [Link] (3 responses)

"Slight"? A bleeding pigeon has resolution more or less on par with what horses have; compare the eye sizes... For a typical mammal 5 stripes per degree is on the edge of being indistinguishable from grey field. Cats are somewhere about 6--8. Camels are circa 10. Horses - 15 or so. And those are the best outside of anthropoidea. We rate at about 50--75, so do falcons. Large eagles are at about 140 (bigger eye => bigger retina => more sensors in the cell array the image is projected upon => higher resolution). Pigeons, IIRC, are somewhere about 15...

As for vivipary vs. ovipary... Archosaurs use the eggshell as calcium store for embryo, so they are pretty much locked into using hard-shelled eggs. Which has very little to do with the amount of time newborns spend in contact with parents; precocious mammalian species can have the contact ending very fast and "few weeks" is nowhere near the longest you can get with birds.

The only measure of clade being successful is its survival; scala naturae is an artifact of the way we use mnemonics to turn messy and complicated history into something that can be easily remembered, as long as you don't look into details...

Mammal evolution, literally

Posted Nov 22, 2012 9:56 UTC (Thu) by man_ls (guest, #15091) [Link] (2 responses)

The figures you quote are fascinating, thanks.

The resolution each species you have mentioned has is -- what it needs to survive. Or more specifically: the lowest resolution it can get away with. Birds of prey need to locate their victims from a long distance; primates need accuracy to locate tree branches when they jump. Note that horses have better eye resolution than humans overall, but do not have a central fovea with increased accuracy.

The fact is that longer development periods are present in mammals than in oviparous animals. Even marsupials have to support their offspring for a certain time before they are ready -- not so with the more primitive monotremes. I think that breasts and milk are great advancements that have allowed mammals to thrive in many environments and in harsh conditions -- for example allowing the mother to feed their offspring even when the environment is bare.

I am not trying to establish a single scale from bacteria to humans; each species is almost by definition well adapted to its environment, or it will perish. Adaptations are always amazing: albatross have a gland to secrete excess salt so they can drink sea water, condors can reportedly fly up to 10 km high, penguins thrive in the Antarctic.

But there is a measurable degree of change, or divergence from the original form. There are many species which have survived unchanged during hundreds of millions of years. Mammals, primates and humans have evolved a lot from the time of dinosaurs, changing not only morphologically but also biochemically. For a simple example compare elephant trunks, dolphin tails, bat wings and human hands. While most birds are still very similar to their dinosaur ancestors in their basic shape.

Mammal evolution, literally

Posted Nov 22, 2012 15:40 UTC (Thu) by raven667 (subscriber, #5198) [Link] (1 responses)

> There are many species which have survived unchanged during hundreds of millions of years.

Unlike the pop aphorism "Survival of the Fittest" the real world is more like survival of the fit-enough.

Mammal evolution, literally

Posted Nov 23, 2012 11:58 UTC (Fri) by dgm (subscriber, #49227) [Link]

Indeed. It should be "Survival of the Fittests". Sexual reproduction makes survival of a single individual... a bit futile.

Crowding out OpenBSD

Posted Nov 14, 2012 4:00 UTC (Wed) by salimma (subscriber, #34460) [Link] (5 responses)

The BSD distributions have trouble coming up with enough developers to do the ports to their own systems
... or to get Linux news editors to remember that they are mostly operating systems, not distributions (except for PC-BSD, which is a distribution of FreeBSD)

Crowding out OpenBSD

Posted Nov 14, 2012 4:26 UTC (Wed) by neilbrown (subscriber, #359) [Link] (4 responses)

> ... or to get Linux news editors to remember that they are mostly operating systems, not distributions (except for PC-BSD, which is a distribution of FreeBSD)

Do you know what the 'D' in 'BSD' stands for?

Crowding out OpenBSD

Posted Nov 14, 2012 7:18 UTC (Wed) by mgedmin (guest, #34497) [Link] (2 responses)

So is "BSD distribution" like a "PIN number" or "ATM machine"?

Crowding out OpenBSD

Posted Nov 14, 2012 10:23 UTC (Wed) by ortalo (guest, #4654) [Link] (1 responses)

Exactly.
And it is also totally ambiguous nowadays and could mean any of: 4.4BSD, BSD Inc., BSD/OS, {Free,Net,Open,Dragonfly}BSD, and certainly other things I do not know of. Well, maybe that's a sign of celebrity.

Crowding out OpenBSD

Posted Nov 22, 2012 23:16 UTC (Thu) by Wol (subscriber, #4433) [Link]

How many BSD's are actually a BSD nowadays? Do they actually come from Berkeley?

Cheers,
Wol

Crowding out OpenBSD

Posted Nov 14, 2012 7:28 UTC (Wed) by salimma (subscriber, #34460) [Link]

That's back when there was only one BSD. While I've seen references in *BSD project pages to distributions, when referring to major BSD variants the term used is normally operating system.

Worse

Posted Nov 14, 2012 5:05 UTC (Wed) by ncm (guest, #165) [Link]

I'm more worried about increasing interdependence among these frilly subsystems. Systemd is the bees' knees, but isn't needed for many uses of Linux. Pulseaudio is a fine mixer (that doesn't *always* mute for-no-reason whatever channel you had meant to use) but for many purposes ALSA provides all you need. Making software work even though this or that bit of apparatus isn't there benefits the Linux and *BSD communities both, and maybe the Glorious Successor to Linux (which might have better ways to accomplish what those subsystems do) as well.

Crowding out OpenBSD

Posted Nov 14, 2012 5:52 UTC (Wed) by dlang (guest, #313) [Link] (10 responses)

Another thing to remember is that someday Linux is going to be replaced by something that has less baggage and so can be faster and more useful on the hardware of the day.

When that starts to happen, userspace projects are going to have to adapt. Those that 'just work' or only require minor tweaks will become standard on the new system. Those that are a pain because they have no provision for compatibility or the maintainers aren't willing to consider portability will end up being replaced (or forked)

Crowding out OpenBSD

Posted Nov 14, 2012 9:13 UTC (Wed) by airlied (subscriber, #9104) [Link] (5 responses)

But when that day comes, and something attracts enough developers to keep pace with Linux they'll have enough developers to work on the desktop envs of the day.

If you somehow believe a brand new OS will appear and they'll be happy to just have fvwm2 and not whatever DE is the DE of the day at the time, you are probably not thinking things through.

The problem with catching Linux now, is keeping up with the new hw support, doesn't matter how little baggage you have, if you can't boot on common hw.

Crowding out OpenBSD

Posted Nov 14, 2012 10:25 UTC (Wed) by ncm (guest, #165) [Link] (2 responses)

Hardware support is always the stumbling block to OS diversity, and will remain so for a long time. Probably the best that can be done for OS progress would be to build future OSes using Linux as their hardware abstraction layer. As a clearly-better OS architecture emerges, the more performance-limiting drivers will naturally migrate to the new system, ultimately leaving Linux to present a unified view of the zillions of slow and old devices. Two decades after that, we will begin to see machines that don't need a Linux driver subsystem any more -- or that have a Linux in each peripheral, forgotten.

Which parts of Linux will slough off first? Tty, memory management, file systems, networking, program loading, user process management. It will be sad, in a way, but the new OS will keep most of us from looking back.

Crowding out OpenBSD

Posted Nov 14, 2012 12:33 UTC (Wed) by dgm (subscriber, #49227) [Link]

This is already happening. You can run most BSDs (except apparently Darwin) as a KVM guest.

Crowding out OpenBSD

Posted Nov 14, 2012 17:13 UTC (Wed) by mjthayer (guest, #39183) [Link]

> Probably the best that can be done for OS progress would be to build future OSes using Linux as their hardware abstraction layer.

Some people are trying interesting variations on that theme. [1]

[1] http://genode.org/documentation/release-notes/12.05#Re-ap...

hardware + application support

Posted Nov 14, 2012 14:16 UTC (Wed) by tialaramex (subscriber, #21167) [Link] (1 responses)

One of my old CS lecturers taught us that an operating system does exactly two useful things:

• it provides access to the hardware you have
• it runs the software you need

You fastened onto the first, there are plenty of people who claim to have a "better" operating system that doesn't work as well as Linux systems do with the hardware people have (often, with any hardware at all).

But the second is just as important. When Linux was new the various compatibility layers were vital. Everything from the ABI compatibility that later angered SCO through to projects like WINE helped make Linux a good choice for people who, like most of us, don't write everything from scratch.

Any project that thinks it's going to "be the next Linux" needs to handle both these problems well, as well as doing something _better_ than Linux.

hardware + application support

Posted Nov 15, 2012 16:44 UTC (Thu) by khim (subscriber, #9252) [Link]

This may be simpler then you think. Google already is working on this problem (even if I'm not sure it realizes it). Both Android and ChromeOS use abstraction layer which separates programs from Linux to a huge degree.

Crowding out OpenBSD

Posted Nov 15, 2012 12:58 UTC (Thu) by vonbrand (subscriber, #4458) [Link] (3 responses)

Sorry to disappoint you, but the baggage even a minimalistic Linux system carries around today is orders of magnitude larger than what would even have been possible in the VAXen days of yore. Won't happen.

Crowding out OpenBSD

Posted Nov 15, 2012 22:10 UTC (Thu) by dlang (guest, #313) [Link] (2 responses)

my point is that Linux is dragging a lot of baggage around in the form of support for old APIs and so on.

Eventually, someone will create a new OS that does a way with the accumulated baggage, and as a result is able to do things more efficently. It will probably start with some smart person putting together something for their own use 'not intended to be big and professional', just like Linux was.

It's foolish to think that Linux is the ultimate OS (or kernel).

The kernel development model (and rate of adoption in accepting changes) should push this point out a long ways, but eventually the pile of cruft will accumulate to the point that something new (or a fork that throws away a lot of compatibility with the existing kernel) will take over.

Crowding out OpenBSD

Posted Nov 16, 2012 14:42 UTC (Fri) by vonbrand (subscriber, #4458) [Link] (1 responses)

"Those who didn't learn from PentiumPro are doomed to Itanium"...

The "baggage" of keeping old stuff running well is vital for the fledgeling operating system/architecture/whatever. If it isn't there, it won't ever make it to enough popularity to stand on its own.

You'd be surprised at the proportion of shops still running ancient software, often on heroic life support measures. One of the funnier stories from IBM's early days was a machine running an emulator for an older model, under which an emulator for a still older machine ran, just to keep some program from the dawn of computing available. And I remember somewhere where they had a more than 15 year old PL/1 program, to which the source code had been long lost. Their most valuable asset was an old hand, who knew the innards of the code enough to be able to tweak it by patching the executable.

Crowding out OpenBSD

Posted Nov 22, 2012 22:04 UTC (Thu) by tialaramex (subscriber, #21167) [Link]

Indeed, I see this "We will succeed because we lack useless baggage" attitude all the time from little hobbyist systems. It's the same mistake as when someone observes that most MS Office users are using only a small fraction of its features and then mistakenly concludes that a program which implements a small fraction of MS Office's features is adequate for most MS Office users. It would only be true if they all sought the /same/ features but of course everybody is different.

It only takes _one_ missing requirement to rule out your system. It will take hundreds, even thousands of features to "bloat" the system enough to make a measurable difference.

Some people seem to have created a folk history of Linux in which Linus Torvalds slays the bloated dinosaurs of Traditional Unix with his simpler, lightweight OS. In reality from the outset people's gripe about Linux was that it lacked features they wanted, and that's where a lot of early development (and the occasional famous name) comes into the picture. Nobody was shouting "Hooray Linux doesn't have over-complicated PAM", they were shouting "Hooray Linux runs on these incredibly cheap i486 based PCs we got for half the price of an entry-level Sun clone, what a shame it doesn't have PAM, I wonder if we can fix that somehow".

For something different

Posted Nov 14, 2012 7:36 UTC (Wed) by eru (subscriber, #2753) [Link]

If one takes a step back to admire the wider OS landscape, then it is rather clear the Linux and the BSD:s are really variations of the same operating system. The "monoculture" does not really advance that much if one of them dies. For something that is actually different (while still staying in the F/OSS OS world), one should look at the BeOS-inspired operating system Haiku, or even ReactOS, or FreeVMS.

Crowding out OpenBSD

Posted Nov 14, 2012 8:51 UTC (Wed) by rsidd (subscriber, #2582) [Link] (2 responses)

Here's the thing -- at this point I really don't care about desktop environments. I have used BSD in the past (FreeBSD, Dragonfly) and was comfortable with it. If I could run a BSD with a minimal window manager that let me set up icons for a terminal and a few apps, that would be enough (I'd launch other apps from the terminal).

What drove me from FreeBSD to Dragonfly was its incredibly flaky USB stack -- you couldn't use USB peripherals without panicking the system. For example, pulling a USB drive without unmounting it would reliably cause a panic. But I also got panics with USB audio, USB serial, and other USB stuff. Supposedly this is because the BSD kernels were written with the underlying assumption that hardware was always present and does not come and go as it pleases. And this is despite the BSDs boasting that they got USB support before Linux. (Over the years I heard many times that this was fixed in FreeBSD, but when I tried it, it wasn't. Perhaps it is now. It was fixed in Dragonfly, though.)

What drove me back to Linux was, sad to say, hardware support. As I get older I want a system that "just works" without extensive hacking.

I don't mind configuring my network, mounting and unmounting drives, etc, via the command line -- in fact I prefer it to NetworkManager etc which never seem to do what I want. And with GNOME 3, GNOME seems to have abandoned the majority of the Linux community too, so moving to a BSD hardly loses me anything. If the BSDs could cater to us command-line types they'd still have a significant niche. But without adequate support for modern hardware, it's a problem, and it's getting worse with the growth of ARM-based devices.

Crowding out OpenBSD

Posted Nov 14, 2012 10:52 UTC (Wed) by ortalo (guest, #4654) [Link] (1 responses)

Lack of drivers (porting or writing from scratch) is probably the main and most worrying indication of the lack of developpers.
It is said there is a critical mass of programming effort over an OS needed to keep it alive: precisely driver programming. (BTW, remember the title of that first book from our editor.)

BTW, BSD code is probably the only OS code base that actually survived an hibernation period without useful drivers (circa 386BSD). At least, they know they did it once...

Crowding out OpenBSD

Posted Nov 15, 2012 23:09 UTC (Thu) by kleptog (subscriber, #1183) [Link]

At my work there are some people who would like to use OpenBSD but the hardware support problems are chronic. Just today an email got sent around asking for people to collect experiences on systems that actually boot. Basically they're at the level Linux was 10 years ago: you have to be really really careful when selecting hardware.

Fortunately it runs in VMs alright.

Crowding out OpenBSD

Posted Nov 14, 2012 9:32 UTC (Wed) by ortalo (guest, #4654) [Link]

It would be suicide for Linux to let the BSDs disappears.

Linux objective is not to design a single OS to rule them all. Today, even children know that this approach is doomed!

The objective is world domination.
In order to achieve this objective, Linux needs allies, to create an alliance (to rule the world).
Those allies are the BSDs.
So they cannot fail!

Crowding out OpenBSD

Posted Nov 14, 2012 9:36 UTC (Wed) by ortalo (guest, #4654) [Link] (4 responses)

Dear editor, the main reason I see personally for the BSDs diffculties, is the fact that they did not have a BSDWN.net site for the last decade.
Can't you give them some help on that?

Crowding out OpenBSD

Posted Nov 14, 2012 10:13 UTC (Wed) by rsidd (subscriber, #2582) [Link] (3 responses)

Actually they had several. But they didn't have a Corbet.

There used to be a "Daemon News" that had interesting articles by a handful of writers. But if I remember right, they never really went into in-depth debates on the current flamewar in -current and the merits of all sides, the way LWN covers the LKML debates. Probably because the writers were all active in development and couldn't really write impartially.

Crowding out OpenBSD

Posted Nov 14, 2012 10:32 UTC (Wed) by ortalo (guest, #4654) [Link]

You are very right.

Crowding out OpenBSD

Posted Nov 14, 2012 10:42 UTC (Wed) by ortalo (guest, #4654) [Link]

BTW, personal merits aside, knowledge issues seem to be involved in the problem too.
And I don't think one man alone can really try to gather detailed enough knowledge about several of these piece of software... But I really wonder what several persons with similar skills as our editors could say on a general comparison article.

Crowding out OpenBSD

Posted Nov 14, 2012 18:24 UTC (Wed) by jengelh (subscriber, #33263) [Link]

Perhaps LWN could increase its BSD coverage somewhat? LWN is no longer "Linux" only, after all.

Really? GNOME that important?

Posted Nov 14, 2012 12:02 UTC (Wed) by job (guest, #670) [Link]

Isn't the problem framed slightly off?

GNOME and friends turned Linux-only a long time ago (if not from the start), but how problematic is that, really? Lots of Linux users and developers use classic desktops.

If end user software, such as Firefox or Libreoffice, became dependent on GNOME or other Linux-isms, that would be a real problem. I don't see that happening just yet (which may be to some great work put in by the ports maintainers). But I can't see missing out on GNOME as a very important omission for someone who has chosen to use an alternative operating system on their computer.

Crowding out OpenBSD

Posted Nov 14, 2012 14:12 UTC (Wed) by dcantrell (subscriber, #75800) [Link] (1 responses)

The complaint about OpenBSD not having enough developers to maintain ports is their own doing. The barrier to entry to participating in OpenBSD is pretty high and not clear. People go where they can contribute and OpenBSD makes that difficult.

Regarding open source desktop environments like GNOME becoming more and more specific to Linux...who cares? If OpenBSD in particular wants to be as close to a neighboring Linux distribution as it can, that's one thing. But complaining about that in terms of making it difficult to innovate and remain relevant is baloney. The reason I continue to buy OpenBSD releases and follow the mailing lists is because I view it as an interesting and innovative project on its own, not because they are really clever at making a primarily Linux-specific desktop work on OpenBSD. Continue to show us real innovative technologies like OpenSSH and the project will remain relevant.

Personally, I like that OpenBSD doesn't come with all of the modern Linux userland baggage by default.

Crowding out OpenBSD

Posted Nov 15, 2012 0:43 UTC (Thu) by roskegg (subscriber, #105) [Link]

I've never had a problem contributing to OpenBSD. I've found them very helpful and accomodating in getting new developers up to speed.

Crowding out OpenBSD

Posted Nov 14, 2012 15:57 UTC (Wed) by mirabilos (subscriber, #84359) [Link]

PAM is a different way of doing things. It has pros and cons, as usual. In OpenBSD, there’s BSDauth, which does its job also just fine. It’s not as pluggable, but also not as insecure as both PAM on Solaris and PAM on GNU, which are, funnily enough, incompatible with each other.

That being said… after the Wim Vandeputte issue with Theo de Raadt not understanding the concept of European Value-Added-Tax, OpenBSD’s irrelevant in Europe anyway, and recently, twice, developers run away to start their own niche projects. Once upon a time, I’d have loved to join their fun, but they don’t really want that…

Crowding out OpenBSD

Posted Nov 14, 2012 17:18 UTC (Wed) by Trelane (subscriber, #56877) [Link]

Thanks for the article. At least for systems that don't require close OS interaction, it seems like a timely warning against monoculturism. I downloaded FreeBSD to see if I can't ensure my code ports without *too* much effort. :)

Crowding out OpenBSD

Posted Nov 14, 2012 17:38 UTC (Wed) by sorpigal (subscriber, #36106) [Link] (5 responses)

*BSD problems are, I think, a reflection of development model and culture. In Linux land we have the exact same issues, but it just doesn't look the same. In BSD land there are perhaps 4 major forks, in Linux land there are *HUNDREDS*, and developer attention for each individual one is quite limited.

The cultural difference is that when a BSD fork happens the tradition says that one subset of the old group gets access to the new fork, lose access to the original tree, and never talk again. When a Linux fork happens the tradition says that the persons who forked still have the same access to the original tree that they had before and, if they want to, can submit patches to the original tree.

Today you have RHEL kernels which are loosely based on Fedora kernels which are based on Linus kernels, more or less. But you also have Ubuntu kernels based on Debian kernels based on Linus kernels, more or less. The kernel teams for each distribution, except RH, are likely smaller than the kernel team for OpenBSD, but the forks are *very small* because tradition says we put upstream as much as we can, whatever can be agreed on, even if it takes a lot of pain for many years. Imagine if the people working on grsec or realtime had to write all the drivers, schedulers, and filesystems instead of just rebasing against a common core; the difference in effort required is enormous.

If the BSDs had operated this way they'd still have a "main" core being fed by a half dozen sub-projects, even if the sub-project differences were quite large, just as Linux has the realtime fork, distro forks, etc. Now that we have git, which seems to have been designed to facilitate a process that the BSD folks don't remotely follow, the whole "many forks, one core" approach works even better and the pace of the fork-experiment-pullrequest-reject-hack-pullrequest-merge loop has gotten faster.

Crowding out OpenBSD

Posted Nov 14, 2012 17:55 UTC (Wed) by rsidd (subscriber, #2582) [Link] (4 responses)

The cultural difference is that when a BSD fork happens the tradition says that one subset of the old group gets access to the new fork, lose access to the original tree, and never talk again. When a Linux fork happens the tradition says that the persons who forked still have the same access to the original tree that they had before and, if they want to, can submit patches to the original tree.

Not true. In the two most recent forks developers first lost access to the original tree (Theo de Raadt for NetBSD, and Matt Dillon for FreeBSD), and then started their forks (OpenBSD and DragonFly respectively). And these forks are respectively 17 and 9 years old, so it's not like this happens on a constant basis. And at least in the case of FreeBSD and DragonFly, plenty of developers worked (and still work) on both projects.

Crowding out OpenBSD

Posted Nov 14, 2012 20:28 UTC (Wed) by sorpigal (subscriber, #36106) [Link] (3 responses)

> In the two most recent forks developers first lost access to the original tree
The sequence isn't the important part of what I was getting at. That said, have there been any cases of Linux developers who have been blacklisted to the point where their patches are refused out of hand?

> And at least in the case of FreeBSD and DragonFly, plenty of developers worked (and still work) on both projects.
After I re-read what I posted I knew someone would call me out on this. What I mean is that after the fork the two things are thought of as separate and worked in separately, which is very different from the thinking of Linux forks, even ones which are large and have existed for many years. There's still the common tree.

Crowding out OpenBSD

Posted Nov 14, 2012 22:01 UTC (Wed) by man_ls (guest, #15091) [Link]

The sequence isn't the important part of what I was getting at. That said, have there been any cases of Linux developers who have been blacklisted to the point where their patches are refused out of hand?
Con Kolivas? After he was accused of not paying attention to bug reports and regressions. He must be the exception that proves the rule... ducks and makes for the door.

Crowding out OpenBSD

Posted Nov 15, 2012 3:16 UTC (Thu) by nix (subscriber, #2304) [Link]

IIRC there were several, in glibc. Of course that's changed now.

Crowding out OpenBSD

Posted Nov 15, 2012 3:48 UTC (Thu) by rsidd (subscriber, #2582) [Link]

Matt Dillon was never blacklisted. His commit rights were revoked. That means he could not directly commit to the FreeBSD tree (CVS in those days) and would have had to give them to a committer. There is no such concept in the Linux kernel -- only Linus can commit to his tree and he has a human tree that feeds him patches. In other words, Matt was in the same situation as every other Linux developer other than Linus -- except that he could have submitted his patches to multiple people if he chose.

The reason this was done, reportedly, was that he was not following the rules and was treading on too many toes. (Incidentally, his rights had been removed once before, and restored; this time it was permanent.) I don't know the merits of this claim, but obviously when you have a centralised repository you need to play nice with others. One could say that this is a merit of the Linux model. Indeed, Matt's DragonFly BSD project now uses git for source control -- which is a testament to the success of Linus's style, since git was written by Linus for his exact requirements.

Crowding out OpenBSD

Posted Nov 14, 2012 20:03 UTC (Wed) by cyrus (subscriber, #36858) [Link] (2 responses)

I believe that BSD or any other OS for that matter, is important for two reasons:

First, to find performance issues in Linux. A couple of years ago, some guys benchmarked FreeBSD against Linux on MySQL OLTP and Linux scaled badly compared to FreeBSD which turned out to be lock contention in the Linux VM. The result of this comparison was improved Linux performance. Does anyone remember the Mindcraft benchmarks from 1999? These were a wakeup call for the Linux community and eventually resulted in much better scalability.

Second, there was a time (late 90s / early 00s) when the FreeBSD VM *was* superior the one found in Linux. There were even exchanges between Matt Dillon and Rik van Riel on how to improve the Linux VM and what it should borrow from the FreeBSD design. These kinds of exchanges have stopped however. Todays kernel hackers seldom lock at FreeBSD or Solaris (if at all). In some regards, Linux only caught up with FreeBSD with the 2.6 kernel. However, these days, Linux is *much* superior to FreeBSD in most aspects.

Crowding out OpenBSD

Posted Nov 15, 2012 1:18 UTC (Thu) by stressinduktion (subscriber, #46452) [Link] (1 responses)

I think, they haven't adopted the vm subsystem to multicore, yet. Last time, I looked they still used splay trees in FreeBSD for mmap management, which waste a lot of memory bandwidth on todays multicore machines. Projects were underway to change this but never got finished.

Crowding out OpenBSD

Posted Nov 25, 2012 23:49 UTC (Sun) by vsrinivas (subscriber, #56913) [Link]

In DragonFly, we did a fair bit of a work improving the VM on SMP systems recently (in the last year or so); we still use RB-trees w/ sx (rw) locking for vm_maps (mapping from (VA, aspace) -> vm_pages) and there is still only one pagedaemon (kthread handling page queue scans), but the front-end of the VM and page fault handling are SMP-friendly. The page queues are also locked in fairly fine-grained ways.

DFly's recent 3.2 release is pretty competitive with Linux on Postgres/Pgbench workloads; some of the lessons learned while improving Postgres/Pgbench performance would be interesting to the Linux VM team -- for instance, DragonFly moved to implement a limited form of page table sharing; this reduced the overhead of managing pv_entries ((D)FBSD's version of rmap entries; see the pre-objrmap Linux VM). Page-table sharing in Linux would not bring the same sort of wins there ('cause of objrmap), but it is worth exploring, I suspect fork() performance would benefit.

Another neat bit of DFly VM machinery that might be interesting -- 'swapcache' -- basically allows for clean file data/metadata to be written to a hopefully-solid-state swap device. It would have a similar effect to flashcache, except it happens at the file vnode layer first, rather than at the block device layer.

Some other interesting things that we learned while scaling out DFly -- we spent a bunch of time optimizing our spinlocks on a wide AMD (4x-socket 48 core) K10 system; along the way we reinvented Linux's ticket spinlock, except with the two counters on separate cachelines rather than parts of the same word. We found them to be pretty terrible under heavy contention, (see a comment on LWN's discussion "Ticket spinlocks perform terribly for uncontested and highly contested cases.") and had a pretty incredible 175sec -> 90something sec reduction in concurrent buildworld time by just moving to something like while-cmpxchg.

Crowding out OpenBSD

Posted Nov 14, 2012 23:58 UTC (Wed) by amacater (subscriber, #790) [Link]

Why should I care what colour the bikeshed is? :)

Crowding out OpenBSD

Posted Nov 15, 2012 4:05 UTC (Thu) by Arach (guest, #58847) [Link]

"BSD is a place where developers can experiment with different approaches to kernel design, filesystems, packaging systems, and more. OpenBSD remains a center for security-related work that does not exist to the same degree in the Linux world."

There are PaX and Grsecurity projects, and linux distributions which include them by default: Hardened Gentoo, Alpine Linux, Atomic Secured Linux, etc (?). There's Openwall GNU/Linux, with its secure userland base.

So in fact, security-related work does exist in the Linux world. Maybe even to a bigger degree (look at what's done in the last 4 years) than in OpenBSD, which didn't innovate anything on the field of security for a long time.

Crowding out OpenBSD

Posted Nov 15, 2012 12:01 UTC (Thu) by karim (subscriber, #114) [Link] (8 responses)

Isn't it irony incarnate that one of the most successful Linux distributions (Android) is actually made up mainly of BSD-ish-licensed software (ASL, BSD, MIT)?

Crowding out OpenBSD

Posted Nov 15, 2012 16:49 UTC (Thu) by drag (guest, #31333) [Link] (7 responses)

The only irony is that the one Linux system that is widely successful in the consumer-facing personal computer market is the one that choose to completely abandoned all pretense of POSIX compatibility and traditional linux-distribution approach to packaging software and went with a dedicated Java-based userland.

This pretty much breaks all the rules that most of the *nix pundents on LWN and most other forums consider core requirements for success.

The biggest problem faced by Gnome and other folks if they want to remain compatible with the BSDs and Solaris and be able achieve their goals pretty much means writing their own entire OS that runs in a abstraction layer on top of a POSIX API that runs with enough privileges to aggressively bypass everything the BSD/Solaris kernel does.

That means having their own graphics drivers, own hardware discovery, own INIT system that runs on top of the existing init system, etc etc. All of it isolated and restricted from anything else that may happen on the OS.

It would be far easier and cleaner approach to port OpenBSD to be a Xen host and run Linux/Gnome desktop on top of that.

POSIX compliance

Posted Nov 22, 2012 10:54 UTC (Thu) by man_ls (guest, #15091) [Link] (6 responses)

On the other hand, the one BSD system that is widely successful is fully POSIX compliant: Mac OS X.

POSIX compliance

Posted Nov 23, 2012 11:36 UTC (Fri) by njs (subscriber, #40338) [Link] (4 responses)

And they did that by, for example, declaring that if you want to use any of OS X's unique features then you can't also use fork(2): http://mail.scipy.org/pipermail/numpy-discussion/2012-Aug...

Not really a great argument for prioritizing POSIX, sadly.

POSIX compliance

Posted Nov 23, 2012 11:38 UTC (Fri) by man_ls (guest, #15091) [Link]

Apparently there are many hidden ironies in this POSIX business, not just one.

POSIX compliance

Posted Nov 23, 2012 13:32 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

You CAN use fork(2) in OS X. You just can't use fork _and_ Cocoa GUI at the same time.

That's not different from Linux and X11.

POSIX compliance

Posted Nov 23, 2012 14:25 UTC (Fri) by njs (subscriber, #40338) [Link] (1 responses)

The difference is that (1) it's officially unsupported to use *any* Apple-written library that is not part of POSIX in a program which also uses fork, (2) since Apple considers this NOTABUG, the APIs provide no way to work around the problems.

With X11, using fork is fine -- you just have to make sure that you only use that connection in one of the children. The bug report I linked to is Apple explaining that you cannot use their *linear algebra library* (which implements the standard, cross-platform BLAS interfaces) on both sides of a fork, and that the only solution is to use a different linear algebra library.

[NB I don't use OS X or really care about it either way, except that these things are annoying to work around.]

POSIX compliance

Posted Nov 23, 2012 14:35 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

Apple is just conservative. It's easier to say "it doesn't work" then to deal with developers using something that works only accidentally.

In particular, important parts of OS X are fork-safe. Moreover, OS X has support for code confinement and process isolation.

POSIX compliance

Posted Nov 23, 2012 21:39 UTC (Fri) by oak (guest, #2786) [Link]

How much of BSD there's in in Apple's iOS?

Crowding out OpenBSD

Posted Nov 15, 2012 23:40 UTC (Thu) by akeane (guest, #85436) [Link]

"Either you're a modern linux with pulseaudio and pam and systemd, or you're dying."

First they took /dev/audio

I did nothing.

Then they took /etc/password

I did nothing.

Then they took the init scripts.

I did nothing.

Finally, they came for vi...

:-(

Q. How many BSD devs does it take to change a light bulb?
A. Doesn't matter, the hardware's not supported!

Q. How many Gnome devs does it take to change a light bulb?
A. Doesn't matter, the last one to leave remembered to turn if off!!

Q. How many Lennard Poettering's does it take to change a light bulb?
A. Dunno, can't find the man page!!!

Q. How many Lennard Poettering's does it REALLY take to change a light bulb?
A. Light bulbs are old fashioned and no use to anyone, all who use them shall be crushed by the onward march of the NEW, IMPROVED, photon propagation "bulbd" which can only be changed by reading 18 documents, and requires a special bulbd reconfigurator !!!!

Q. How many "haters gonna hate"?
A. Me

Ah, my poor beloved UNIX, ruined by the "youth of today", with their utopian dreams of GUIs where I can't change where the clock goes, installers that can't be automated by a simple text file, or better, a world where whining cry babies ("haters gonna hate, waa, waa, I want my mommy"),seem to get more influence in distributions than that mean and oafish Linus fellow.

----------------------------------------------
Sent by my telnet client

Crowding out OpenBSD

Posted Nov 25, 2012 8:28 UTC (Sun) by gabucino (guest, #72504) [Link]

The only part of my digital life that i regret is that I found that crap linux (1998) before I discovered BSD (2004). And to top that: Linux got a hundred times worse than the disgusting crap it was back then. Just now I spilled my tea because of the nightmaris memories.

I had the dubious honor to read the comments above in which some young and inexperienced linux users fantasized that "BSD has disappeared" etc... Well, if I take all my UNIX acquintances, only 1 (one) is using linux (yo A'rpi ;). The others are either OS/X, NetBSD, or Windows 7. If some of them have to deploy their works on a linux/buguntu garbage dump, they have their minions to do it. I routinely charge +50% above my payment if it involves me touching a linux (IF I even accept the job, and that's damned rare), 'cause that's how much more alcohol I'll have to consume to treat the brain cancer I get.

Oh, and every thread that has "SElinux" written in it - is bogus.

Crowding out OpenBSD

Posted Dec 9, 2012 9:15 UTC (Sun) by sherlockmao (guest, #83435) [Link]

When Electoral College was invented, the balance was believed to have been found. It balanced the weight of each states and prevented possible irrational passion.

But right now, swing states seem to attract all focus, and the rest appear not that important, as everyone knows their choices.

The same, the differences between Linux and *BSD are partially affected by some commercial drivers. Graphic cards and WiFi chips are the two most significant factors for developers as their new laptops can survive in Linux but not *BSD. When PC-BSD looks blur while my Arch Linux with font-patch looks terrific, I already make my choice. The problem is developers will not use the computers for developing only, they will use them for entertainment as well.

Both Linux and *BSD are awesome, and people should feel free to choose. I hope the drivers will not prevent developers/users from one system to the other, because the same sad story happened many years ago, when Linux did not equip enough drivers to compete with Windows.


Copyright © 2012, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds