OpenSolaris governing board threatens dissolution (The H) [LWN.net]

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 10:48 UTC (Wed) by pboddie (guest, #50784) [Link] (18 responses)

Also, why not ask the question the other way around - why people still use Linux? What does Linux do better, apart from hardware support and nicely packaged distributions?

Have you ever used Solaris or is this just another opportunity to take a pop at stuff you don't like? The answer to your question is written all over the various efforts to make OpenSolaris more relevant to a wider audience, like shipping decent versions of common tools (not some limited variants in a bunch of oddly-named directories) and doing package/dependency management.

Oh, and choosing licensing and promoting a community structure that doesn't make it look like everyone is working for some corporate parent might have something to do with it, in case the emphasis of the original article passed you by.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 16:38 UTC (Wed) by trasz (guest, #45786) [Link] (17 responses)

I already mentioned "nicely packaged up distributions", didn't I? One thing I missed is cheaper workforce - more people know Linux, so it's easier to find a sysadmin. However, in technical aspects, from filesystems to things as basic as synchronisation (Linux still uses anachronic spinlocks), Solaris is years ahead. Thus my question.

As for the license and company parents - same thing in Linux, except that GPL is more restrictive (according to Stallman's interpretation it's viral) and you have several corporations instead of one; not much of a difference, I think.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 17:07 UTC (Wed) by pboddie (guest, #50784) [Link]

I already mentioned "nicely packaged up distributions", didn't I?

Yes, but you make it sound like gloss when, in fact, decent package and dependency management is essential, as various other people have already pointed out. It was also a major priority of Project Indiana: you know, the thing where the Debian guy was brought in to add things that people found indispensable in Linux.

One thing I missed is cheaper workforce - more people know Linux, so it's easier to find a sysadmin.

You know, a load of people prided themselves on knowing Solaris and SunOS before it, especially amongst adopters of Linux, so Linux didn't exactly start off with any numerical advantage in this respect.

However, in technical aspects, from filesystems to things as basic as synchronisation (Linux still uses anachronic spinlocks), Solaris is years ahead. Thus my question.

Well, the "years ahead" Solaris isn't seeing that much action in, say, supercomputers, so it doesn't hold all the good cards by any means. Why Linux is used so widely isn't just down to specific technology.

As for the license and company parents - same thing in Linux, except that GPL is more restrictive (according to Stallman's interpretation it's viral) and you have several corporations instead of one; not much of a difference, I think.

You cannot be serious! Firstly, the CDDL is a copyleft licence, but one that isn't compatible with the GPL, which says more about Sun not wanting to relinquish control of their software than it does about the GPL. Secondly, there are corporations developing Linux but they influence the direction of the project mostly on technical merit, not on having a bunch of puppet bureaucracies set up by a single corporation, which have obviously worked out so well for all of Sun's other open source projects, too.

So, apart from various technical essentials that were neglected in Solaris for about a decade or so, plus various community and licensing considerations, what else would make people switch to Linux? Isn't this like asking what the Romans have done for us? Maybe more than you're willing to accept.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 18:54 UTC (Wed) by vonbrand (subscriber, #4458) [Link] (14 responses)

However, in technical aspects, from filesystems to things as basic as synchronisation (Linux still uses anachronic spinlocks), Solaris is years ahead. Thus my question.

Exactly the opposite... on our aging SPARC IPX machines at the time Solaris barely crawled, Red Hat ran just fine, and gave that iron a few extra years of useful life (Solaris wasn't updated on them anymore). Linux was fast and lean, Solaris was bloat at its worst.

BTW, what is "anachronic" about spinlocks? They do work, and have low overhead. Sure, it looks like the in-kernel synchronization primitives for Solaris are less (and simpler to use); but if the cost is that the system runs much slower, no thanks.

Filesystems? Like the in-kernel MS-DOS filessytem they had, that was so incredibly awfully slow (I seem to remember minutes copying a few smallish files) that the very first thing we did on any Sun machine here was to install mtools to get bearable floppies? Like all the filesystems Linux handles today (I believe almost all important disk partitioning schemes are supported, as are most filesystems, even very obscure ones), and which Solaris doesn't understand at all?

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 21:01 UTC (Wed) by farnz (subscriber, #17727) [Link] (9 responses)

AIUI, the Solaris equivalent of a spinlock is a complex thing called an "adaptive mutex"; it's a spinlock if held for a short time period, becoming a sleeping lock if held for a long time period. And they don't have RCU, so many things that are RCU-protected in Linux are mutex protected in Solaris - whereas their Linux equivalents went from sleeping locks to spinlocks to RCU to get scalability up without paying a huge price on small machines.

Spinlocks are probably the fastest primitive for SMP locking - they directly exploit cache coherency protocols for their inter-CPU messaging, so pay less overhead than any sleeping lock type. The only downside is that you lose out if you wait on a spinlock for long periods; this is ameliorated by two things in kernel design:

Long lock hold times are bad for both realtime response (your upper bound on many timings becomes the lock hold time) and SMP scalability (as only one CPU can hold the lock at a time, long lock hold times prevent two CPUs working on related items in parallel - think a scheduler lock, for example, where only one CPU can reschedule at a time if two or more CPUs need the lock), so kernel designers aim to reduce the time any lock is held for, in order to reduce contention. You either do this by introducing finer grained locks (e.g. a per-CPU scheduler lock, and algorithms to rebalance tasks off loaded CPUs), or by redesigning completely to go lock-free.
If you know that a lock's going to be held for a long time, and contended on, you can deliberately choose something else (RCU, which Solaris doesn't use, or a sleeping lock type). If you can't predict lock contention periods, your algorithm probably isn't kernel-ready.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 15, 2010 14:44 UTC (Thu) by trasz (guest, #45786) [Link] (8 responses)

Actually, you missed a small detail, and got the conclusions wrong. The small detail is that because spinlock cannot block, it's often neccessary to disable interrupts (the whole irqsave/irqrestore stuff). In Solaris (same as in FreeBSD or OSX), it's not neccessary, due to use of interrupt threads. Second problem with spinlocks is that they waste CPU time when contention happens - real mutexes just switch to other thread instead (not immediately, though, you mentioned adaptive spinlocks already). In the end, spinlocks are slower then mutexes in other systems (fast path in mutex is pretty much same thing as in spinlock, minus the interrupt stuff) , and they become even slower under load, due to inability to deal with contention.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 15, 2010 16:03 UTC (Thu) by farnz (subscriber, #17727) [Link] (7 responses)

And the thing you are completely ignoring is that the cost of a thread switch outweighs the cost of disabling interrupts for short periods on any CPU you might run Solaris on; any argument against spinlocks that goes "we use threads to avoid needing to disable interrupts" is an instant loser - go measure the difference on (say) a Niagra system, or an UltraSPARC II yourself.

Measurements of spinlocks versus mutexes have shown time and time again that spinlocks are faster than mutexes when the lock hold time is small, regardless of the amount of contention; the whole reason Solaris now has adaptive mutexes (which are spinlocks until you've held them for a while) is that it is now too late for Solaris to go back and change the design decisions that led to mutexes everywhere, even when the lock hold time is small, and thus they need their mutexes to be spinlocks when the lock hold time is small, and proper mutexes when the lock hold time is long.

In contrast, Linux uses mutexes when the lock hold time could reasonably be long, and spinlocks when the protected work is guaranteed to be small; this reduces overhead compared to Solaris, as it doesn't bother spinning if it's unlikely to make progress, instead going straight to sleep. Further, Linux uses RCU and related techniques to avoid taking locks (and thus facing inter-CPU synchronisation issues) where Solaris uses locks.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 16, 2010 11:08 UTC (Fri) by trasz (guest, #45786) [Link] (6 responses)

That's not how it works. When held for a short time, mutex (todays mutexes are always adaptive, at least in Solaris and FreeBSD) doesn't switch threads - it spins, just like spinlock. Thus, the whole "thread switch outweighs the cost of disabling interrupts" argument is invalid - it's not "linux spins, other systems switch"; it's "linux has to disable interrupts while spinning, and other systems don't".

I'm not sure about what measurements you're talking about. I guess you mean some Linux implementation - but then, if they were slower, that just proves that they were badly implemented. Let me repeat: fast path in a mutex (in operating system other than Linux) is pretty much the same as fast path of spinlock in Linux, minus messing with interrupts.

There is one more thing you missed - results. I've already mentioned that IBM doesn't publish high end server workloads benchmarks (Oracle, SAP etc) under Linux if they can use AIX instead. What they do publish for Linux is stuff that doesn't spend much time in the kernel by definition (i.e. HPC), and is thus not so demanding when it comes to scalability - because it doesn't matter if kernel needs to waste twice as much CPU time, if the total kernel time is 1% and the rest is userland.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 16, 2010 22:37 UTC (Fri) by joib (subscriber, #8541) [Link]

In Linux spinlocks are meant for situations were the lock is held for a very short time. This means that code which holds a spinlock may not block, and secondly, by disabling interrupts it ensures that the processor does not go off to do something else with the spinlock held, such as servicing interrupts, or switching to some other thread.

Or to put it another way, since the lock is held for a very short time, and the thread holding the lock can't be bumped off the cpu, a lightweight spinlock is the correct locking primitive.

One way to see the performance impact of mutexes vs. spinlocks is to benchmark with the hard realtime patches (PREEMPT_RT). These patches replace most of the spinlocks with (IIRC priority inheriting) mutexes, since spinlocks cannot guarantee that the highest priority process will always get the lock. Due to this, and some other reasons, the PREEMPT_RT kernel is a bit slower as well as scales much worse than the vanilla kernel.

As an aside, this has nothing to do with threaded interrupt handlers (which FWIW are also available in Linux, although so far not widely used) vs. the traditional bottom/top half+workqueue/whatever interrupt handlers.

As another aside, AFAIK disabling and re-enabling interrupts takes relatively little time; I doubt you can actually measure the performance hit due to this.

As yet another side, Linux also has adaptive mutexes. E.g. the BTRFS file system uses them, although I believe they are otherwise relatively rare.

The argument that fast paths are equally fast in spinlocks as in mutexes, well, duh. Of course. Most locking mechanisms have pretty fast paths. The reason why there are many of them is that we're interested in different behavior under contention. In some cases the appropriate behavior is to spin, in other cases sleeping is better, and in yet other cases some lockless mechanism such as RCU is appropriate.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 18, 2010 8:56 UTC (Sun) by cmccabe (guest, #60281) [Link] (4 responses)

> Thus, the whole "thread switch outweighs the cost of disabling interrupts"
> argument is invalid - it's not "linux spins, other systems switch"; it's
> "linux has to disable interrupts while spinning, and other systems don't".

How useful is a spinlock that doesn't disable interrupts on a uniprocessor system? Think about it carefully.

Another two questions: have you ever written code that used spinlocks, RCU, or seqlocks? Have you ever been the system administrator for a Solaris system?

A simple yes or no will do.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 18, 2010 9:55 UTC (Sun) by johill (subscriber, #25196) [Link] (2 responses)

Err, a spinlock that doesn't need to disable interrupts, say because it is not used from interrupt context will degrade to a no-op on UP systems. Questioning the usefulness of such lock is quite pointless since the UP machine cannot be preempted in that spot by code design.

And something that the start of this thread failed to mention is that Linux doesn't actually always disable interrupts for spinlocks. Only those that are also used from interrupt context need to disable interrupts. The others still make sense on SMP systems. It's why there's spin_lock_irqsave/spin_lock_bh/spin_lock.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 18, 2010 20:04 UTC (Sun) by cmccabe (guest, #60281) [Link] (1 responses)

You're right-- not all spinlocks do disable interrupts. I was just replying to trasz's accusation that disabling interrupts in spinlocks was a crazy, Linux-only idea. I guess I was kind of feeding the troll, sigh.

Traditionally in Linux you used spin_lock_irqsave when you needed to modify the same variable in an interrupt handler as in some other kernel code.

I know that lately there's been a move towards threaded IRQ handlers and less use of CLI / STI. The PREEMPT_RT patchset, which I've used previously at work, replaces most spinlocks with mutexes in order to get better maximum latency performance. It would be interesting to see a comparison between Linux + PREEMPT_RT and Solaris's architecture. As far as I know, the selling point for these architectural changes is reduced latency rather than greater throughput-- which, again, is contrary to what trasz is saying.

OpenSolaris governing board threatens dissolution (The H)

Posted Aug 6, 2010 9:29 UTC (Fri) by trasz (guest, #45786) [Link]

I never said disabling interrupts in spinlocks was crazy - I said systems using interrupt threads and fully functional mutexes don't need to disable interrupts, because there is nothing wrong with interrupt thread blocking on mutex.

As for the speed comparisons - take a look at FreeBSD. It has spinlocks - which disable interrupts - but their use is discouraged, because they _are_ slower. If threaded interrupts handlers in Linux result in worse performance, this might be caused by poor implementation.

OpenSolaris governing board threatens dissolution (The H)

Posted Aug 6, 2010 9:20 UTC (Fri) by trasz (guest, #45786) [Link]

Spinlocks - yes, RCU - no; seqlocks seem to look like a reader/writer locks, so probably yes. Solaris - yes, but for nothing important (i.e. paid).

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 15, 2010 14:59 UTC (Thu) by trasz (guest, #45786) [Link] (3 responses)

WRT the IPX - it's old problem of overhead vs. scalability. Linux always had smaller overhead. So, if you have hardware manufactured in 1994, it's much better choice, I agree. :->

For the spinlocks, see below. In short: they are slow and they don't scale.

As for filesystems - Linux doesn't have anything comparable to ZFS, and this is a huge drawback - snapshots, clones (great for upgrading), fast incremental backups, checksums, not having to size your filesystems by hand and generally fsck with LVM, to mention a few I'm using. But yeah, for handling PC floppies Solaris probably sucked.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 15, 2010 23:59 UTC (Thu) by clump (subscriber, #27801) [Link] (2 responses)

It must burn you up that no matter how highly you regard ZFS it's not stopping Linux from burying Solaris.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 16, 2010 11:18 UTC (Fri) by trasz (guest, #45786) [Link] (1 responses)

It must burn you up that no matter how highly you regard Linux it's not stopping pretty much everyone from ignoring it on desktop ;->

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 20, 2010 14:01 UTC (Tue) by nix (subscriber, #2304) [Link]

While Solaris on the desktop is just *everywhere*, yeah.

(perhaps in a better world...)

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 19:16 UTC (Wed) by mpr22 (subscriber, #60784) [Link]

I am bemused by your apparent belief that having contributions from multiple more-or-less directly competing corporations is "not much different" to pretty much only existing at all in order to sell a single manufacturer's hardware.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 12:17 UTC (Wed) by mpr22 (subscriber, #60784) [Link] (2 responses)

Also, why not ask the question the other way around - why people still use Linux?

"Still use" doesn't make sense when applied to Linux, because Solaris isn't growing at Linux's expense.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 16:47 UTC (Wed) by trasz (guest, #45786) [Link] (1 responses)

Do you have any stats showing that Linux is still growing at Solaris expense? I know this was happening, but it's not quite sure it continues, especially given that Oracle might try to start doing what IBM started a while ago, i.e. again try to push its own unix instead.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 15, 2010 17:10 UTC (Thu) by nix (subscriber, #2304) [Link]

Given what Oracle have just done to Solaris support costs, I'm quite sure it's continuing. I would not be remotely surprised if it were to speed up.

filtering test case?

Posted Jul 14, 2010 12:27 UTC (Wed) by sbishop (guest, #33061) [Link] (2 responses)

I have often times wondered if you're Corbet cooking up test cases for the new comment-filtering functionality. I am going to try it out...

filtering test case?

Posted Jul 14, 2010 12:49 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

No, but feel free to test them! :)

I was genuinely interested and I really have not expected such a big flamewar.

filtering test case?

Posted Jul 14, 2010 17:30 UTC (Wed) by sbishop (guest, #33061) [Link]

Just to be clear, I thought your question was reasonable. My comment was directed toward trasz. And so far the filtering is working great! :)

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 15:27 UTC (Wed) by vonbrand (subscriber, #4458) [Link] (16 responses)

Reasonable hardware support, timely software updates, and a ever growing list of competently packaged software was the reason to move away from Solaris... and was for many other shops around me. No high-end shops, mind you; but Linux is slowly taking over that space too as legacy Solaris boxes retire.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 16:25 UTC (Wed) by SEJeff (guest, #51588) [Link] (15 responses)

Linux does high end too as evidence by:
http://www.top500.org/stats/list/35/osfam

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 16:39 UTC (Wed) by trasz (guest, #45786) [Link] (14 responses)

Except that HPC stuff is very different from high end server workloads. Never wondered why IBM always benchmarks its POWER systems under AIX instead of Linux? ;->

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 20:45 UTC (Wed) by clump (subscriber, #27801) [Link] (13 responses)

"Never wondered"? How about "never bothered to check your facts":

http://www-03.ibm.com/systems/power/hardware/reports/syst...

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 14, 2010 21:11 UTC (Wed) by farnz (subscriber, #17727) [Link]

For those wishing to go directly to the Linux benchmarks, I've looked at the twp May 2010 PDFs from clump's link to IBM: page 19 onwards of the High Performance Computing Performance Report and page 27 onwards of the Systems Performance Report cover Linux performance.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 15, 2010 14:52 UTC (Thu) by trasz (guest, #45786) [Link] (11 responses)

Yeah, nice try. I guess you didn't notice that while there are application benchmarks for Oracle, SAP, Sybase, SAP, Java, DB2 and couple others I don't recognize, it's all under AIX - the only "benchmarks" published for Linux are synthetic ones like SPECcpu, which don't say anything about operating system scalability and performance. See for yourself, it's in the PDF you pasted.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 15, 2010 23:44 UTC (Thu) by clump (subscriber, #27801) [Link] (10 responses)

It might help if you read your own comment, clicked the link to IBM's page, then conveniently clicked the link for the PDF under "IBM Power Systems High Performance Computing Performance Report".

Spoiler alert: you're still wrong.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 16, 2010 10:47 UTC (Fri) by trasz (guest, #45786) [Link] (9 responses)

I've already mentioned three comments above that HPC stuff is very different from high end server workloads. IBM doesn't benchmark high end server workloads under Linux on machines supported by AIX, as demonstrated in your PDFs. (Is this thread really so hard to follow?)

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 16, 2010 10:51 UTC (Fri) by mpr22 (subscriber, #60784) [Link] (8 responses)

Now to my mind, the natural response is "Of course they don't. That might undermine AIX sales."

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 16, 2010 11:16 UTC (Fri) by trasz (guest, #45786) [Link] (7 responses)

I don't think IBM makes big money on AIX licenses. They're selling hardware. If they wanted to sell AIX, they wouldn't publish Linux benchmarks for HPC and instead they would push AIX there as well.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 16, 2010 19:03 UTC (Fri) by giraffedata (guest, #1954) [Link] (6 responses)

I don't think IBM makes big money on AIX licenses. They're selling hardware.

I'm pretty sure you're right about that. If IBM could get Linux to do everything on IBM's hardware that AIX can (in IBM's opinion), IBM would be happy to dump the AIX development cost and leech off others' Linux development.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 16, 2010 20:33 UTC (Fri) by anselm (subscriber, #2796) [Link] (4 responses)

It's not as if IBM isn't taking part in Linux (kernel) development. See, e.g., Jon's stats in last week's issue. As companies go they are fairly high up on the list – not anywhere near Red Hat, to be sure, but certainly ahead of most of the others.

If that is »leeching off others' Linux development« then pretty much everybody is doing it (and their dog, too). Even Red Hat, the leader by a wide margin, has contributed only not quite 12% of the changesets in 2.6.35. There's no point in bashing only IBM about this.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 16, 2010 21:12 UTC (Fri) by giraffedata (guest, #1954) [Link] (3 responses)

It's not as if IBM isn't taking part in Linux (kernel) development.

Right, but I didn't say anything related to IBM's current relationship with Linux. I posed a hypothetical situation where IBM is able to sell its hardware with Linux as developed by others, in order to make a point about whether IBM makes a profit on AIX.

Of course, the hypothetical still works if others do 95% of the development of Linux, which is how it is today.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 17, 2010 0:34 UTC (Sat) by dlang (guest, #313) [Link] (2 responses)

what amount of contribution to linux changes a company from being a 'leech'

if 88% of the development is done by other companies, is that company a leech? if so Redhat is a leech. If that's the line where they are no longer a leech, then only 8 companies in the world can not be a leech.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 17, 2010 1:55 UTC (Sat) by giraffedata (guest, #1954) [Link] (1 responses)

what amount of contribution to linux changes a company from being a 'leech'

If you mean "leech" in a morally negative sense, then I have no opinion on that. I don't much care about morality of business and I didn't mean to say anything about IBM's hypothetical morality if it hypothetically decided to start using Linux instead of AIX to sell its hardware.

I was only talking about A taking advantage of work that B did for some purpose other than to serve A. And I do think that's an honorable way to increase the wealth of the world.

OpenSolaris governing board threatens dissolution (The H)

Posted Jul 17, 2010 4:55 UTC (Sat) by dlang (guest, #313) [Link]

you are the first person I have ever seen use the term leech and mean it to be a positive thing :-)

every other time I've seen it used (in most cases with almost the exact same statement you made), it's being used to say that it's not fair that IBM would bet getting so much benefit and they should be punished (if only by preferring the work of some other company that isn't a leech)

the term 'leech' strongly implies (if not outright states) that you are taking something away from the host that it can't use anymore for your own benefit.

OpenSolaris governing board threatens dissolution (The H)

Posted Aug 6, 2010 9:12 UTC (Fri) by trasz (guest, #45786) [Link]

That was the plan few years ago - when you look at IBM's position on Linux shortly after they got involved (2000, IIRC?), the plan was to phase out AIX. They changed their mind few years ago.