| |||||||||||||
OpenSolaris governing board threatens dissolution (The H)OpenSolaris governing board threatens dissolution (The H)Posted Jul 14, 2010 18:54 UTC (Wed) by vonbrand (subscriber, #4458)In reply to: OpenSolaris governing board threatens dissolution (The H) by trasz Parent article: OpenSolaris governing board threatens dissolution (The H)
However, in technical aspects, from filesystems to things as basic as synchronisation (Linux still uses anachronic spinlocks), Solaris is years ahead. Thus my question. Exactly the opposite... on our aging SPARC IPX machines at the time Solaris barely crawled, Red Hat ran just fine, and gave that iron a few extra years of useful life (Solaris wasn't updated on them anymore). Linux was fast and lean, Solaris was bloat at its worst. BTW, what is "anachronic" about spinlocks? They do work, and have low overhead. Sure, it looks like the in-kernel synchronization primitives for Solaris are less (and simpler to use); but if the cost is that the system runs much slower, no thanks. Filesystems? Like the in-kernel MS-DOS filessytem they had, that was so incredibly awfully slow (I seem to remember minutes copying a few smallish files) that the very first thing we did on any Sun machine here was to install mtools to get bearable floppies? Like all the filesystems Linux handles today (I believe almost all important disk partitioning schemes are supported, as are most filesystems, even very obscure ones), and which Solaris doesn't understand at all?
(Log in to post comments)
OpenSolaris governing board threatens dissolution (The H) Posted Jul 14, 2010 21:01 UTC (Wed) by farnz (guest, #17727) [Link] AIUI, the Solaris equivalent of a spinlock is a complex thing called an "adaptive mutex"; it's a spinlock if held for a short time period, becoming a sleeping lock if held for a long time period. And they don't have RCU, so many things that are RCU-protected in Linux are mutex protected in Solaris - whereas their Linux equivalents went from sleeping locks to spinlocks to RCU to get scalability up without paying a huge price on small machines. Spinlocks are probably the fastest primitive for SMP locking - they directly exploit cache coherency protocols for their inter-CPU messaging, so pay less overhead than any sleeping lock type. The only downside is that you lose out if you wait on a spinlock for long periods; this is ameliorated by two things in kernel design:
OpenSolaris governing board threatens dissolution (The H) Posted Jul 15, 2010 14:44 UTC (Thu) by trasz (guest, #45786) [Link]
Actually, you missed a small detail, and got the conclusions wrong. The small detail is that because spinlock cannot block, it's often neccessary to disable interrupts (the whole irqsave/irqrestore stuff). In Solaris (same as in FreeBSD or OSX), it's not neccessary, due to use of interrupt threads. Second problem with spinlocks is that they waste CPU time when contention happens - real mutexes just switch to other thread instead (not immediately, though, you mentioned adaptive spinlocks already). In the end, spinlocks are slower then mutexes in other systems (fast path in mutex is pretty much same thing as in spinlock, minus the interrupt stuff) , and they become even slower under load, due to inability to deal with contention.
OpenSolaris governing board threatens dissolution (The H) Posted Jul 15, 2010 16:03 UTC (Thu) by farnz (guest, #17727) [Link] And the thing you are completely ignoring is that the cost of a thread switch outweighs the cost of disabling interrupts for short periods on any CPU you might run Solaris on; any argument against spinlocks that goes "we use threads to avoid needing to disable interrupts" is an instant loser - go measure the difference on (say) a Niagra system, or an UltraSPARC II yourself. Measurements of spinlocks versus mutexes have shown time and time again that spinlocks are faster than mutexes when the lock hold time is small, regardless of the amount of contention; the whole reason Solaris now has adaptive mutexes (which are spinlocks until you've held them for a while) is that it is now too late for Solaris to go back and change the design decisions that led to mutexes everywhere, even when the lock hold time is small, and thus they need their mutexes to be spinlocks when the lock hold time is small, and proper mutexes when the lock hold time is long. In contrast, Linux uses mutexes when the lock hold time could reasonably be long, and spinlocks when the protected work is guaranteed to be small; this reduces overhead compared to Solaris, as it doesn't bother spinning if it's unlikely to make progress, instead going straight to sleep. Further, Linux uses RCU and related techniques to avoid taking locks (and thus facing inter-CPU synchronisation issues) where Solaris uses locks.
OpenSolaris governing board threatens dissolution (The H) Posted Jul 16, 2010 11:08 UTC (Fri) by trasz (guest, #45786) [Link]
That's not how it works. When held for a short time, mutex (todays mutexes are always adaptive, at least in Solaris and FreeBSD) doesn't switch threads - it spins, just like spinlock. Thus, the whole "thread switch outweighs the cost of disabling interrupts" argument is invalid - it's not "linux spins, other systems switch"; it's "linux has to disable interrupts while spinning, and other systems don't".
I'm not sure about what measurements you're talking about. I guess you mean some Linux implementation - but then, if they were slower, that just proves that they were badly implemented. Let me repeat: fast path in a mutex (in operating system other than Linux) is pretty much the same as fast path of spinlock in Linux, minus messing with interrupts.
There is one more thing you missed - results. I've already mentioned that IBM doesn't publish high end server workloads benchmarks (Oracle, SAP etc) under Linux if they can use AIX instead. What they do publish for Linux is stuff that doesn't spend much time in the kernel by definition (i.e. HPC), and is thus not so demanding when it comes to scalability - because it doesn't matter if kernel needs to waste twice as much CPU time, if the total kernel time is 1% and the rest is userland.
OpenSolaris governing board threatens dissolution (The H) Posted Jul 16, 2010 22:37 UTC (Fri) by joib (guest, #8541) [Link]
In Linux spinlocks are meant for situations were the lock is held for a very short time. This means that code which holds a spinlock may not block, and secondly, by disabling interrupts it ensures that the processor does not go off to do something else with the spinlock held, such as servicing interrupts, or switching to some other thread.
Or to put it another way, since the lock is held for a very short time, and the thread holding the lock can't be bumped off the cpu, a lightweight spinlock is the correct locking primitive.
One way to see the performance impact of mutexes vs. spinlocks is to benchmark with the hard realtime patches (PREEMPT_RT). These patches replace most of the spinlocks with (IIRC priority inheriting) mutexes, since spinlocks cannot guarantee that the highest priority process will always get the lock. Due to this, and some other reasons, the PREEMPT_RT kernel is a bit slower as well as scales much worse than the vanilla kernel.
As an aside, this has nothing to do with threaded interrupt handlers (which FWIW are also available in Linux, although so far not widely used) vs. the traditional bottom/top half+workqueue/whatever interrupt handlers.
As another aside, AFAIK disabling and re-enabling interrupts takes relatively little time; I doubt you can actually measure the performance hit due to this.
As yet another side, Linux also has adaptive mutexes. E.g. the BTRFS file system uses them, although I believe they are otherwise relatively rare.
The argument that fast paths are equally fast in spinlocks as in mutexes, well, duh. Of course. Most locking mechanisms have pretty fast paths. The reason why there are many of them is that we're interested in different behavior under contention. In some cases the appropriate behavior is to spin, in other cases sleeping is better, and in yet other cases some lockless mechanism such as RCU is appropriate.
OpenSolaris governing board threatens dissolution (The H) Posted Jul 18, 2010 8:56 UTC (Sun) by cmccabe (guest, #60281) [Link]
> Thus, the whole "thread switch outweighs the cost of disabling interrupts"
> argument is invalid - it's not "linux spins, other systems switch"; it's > "linux has to disable interrupts while spinning, and other systems don't".
How useful is a spinlock that doesn't disable interrupts on a uniprocessor system? Think about it carefully.
Another two questions: have you ever written code that used spinlocks, RCU, or seqlocks? Have you ever been the system administrator for a Solaris system?
A simple yes or no will do.
OpenSolaris governing board threatens dissolution (The H) Posted Jul 18, 2010 9:55 UTC (Sun) by johill (subscriber, #25196) [Link]
Err, a spinlock that doesn't need to disable interrupts, say because it is not used from interrupt context will degrade to a no-op on UP systems. Questioning the usefulness of such lock is quite pointless since the UP machine cannot be preempted in that spot by code design.
And something that the start of this thread failed to mention is that Linux doesn't actually always disable interrupts for spinlocks. Only those that are also used from interrupt context need to disable interrupts. The others still make sense on SMP systems. It's why there's spin_lock_irqsave/spin_lock_bh/spin_lock.
OpenSolaris governing board threatens dissolution (The H) Posted Jul 18, 2010 20:04 UTC (Sun) by cmccabe (guest, #60281) [Link]
You're right-- not all spinlocks do disable interrupts. I was just replying to trasz's accusation that disabling interrupts in spinlocks was a crazy, Linux-only idea. I guess I was kind of feeding the troll, sigh.
Traditionally in Linux you used spin_lock_irqsave when you needed to modify the same variable in an interrupt handler as in some other kernel code.
I know that lately there's been a move towards threaded IRQ handlers and less use of CLI / STI. The PREEMPT_RT patchset, which I've used previously at work, replaces most spinlocks with mutexes in order to get better maximum latency performance. It would be interesting to see a comparison between Linux + PREEMPT_RT and Solaris's architecture. As far as I know, the selling point for these architectural changes is reduced latency rather than greater throughput-- which, again, is contrary to what trasz is saying.
OpenSolaris governing board threatens dissolution (The H) Posted Aug 6, 2010 9:29 UTC (Fri) by trasz (guest, #45786) [Link]
I never said disabling interrupts in spinlocks was crazy - I said systems using interrupt threads and fully functional mutexes don't need to disable interrupts, because there is nothing wrong with interrupt thread blocking on mutex.
As for the speed comparisons - take a look at FreeBSD. It has spinlocks - which disable interrupts - but their use is discouraged, because they _are_ slower. If threaded interrupts handlers in Linux result in worse performance, this might be caused by poor implementation.
OpenSolaris governing board threatens dissolution (The H) Posted Aug 6, 2010 9:20 UTC (Fri) by trasz (guest, #45786) [Link]
Spinlocks - yes, RCU - no; seqlocks seem to look like a reader/writer locks, so probably yes. Solaris - yes, but for nothing important (i.e. paid).
OpenSolaris governing board threatens dissolution (The H) Posted Jul 15, 2010 14:59 UTC (Thu) by trasz (guest, #45786) [Link]
WRT the IPX - it's old problem of overhead vs. scalability. Linux always had smaller overhead. So, if you have hardware manufactured in 1994, it's much better choice, I agree. :->
For the spinlocks, see below. In short: they are slow and they don't scale.
As for filesystems - Linux doesn't have anything comparable to ZFS, and this is a huge drawback - snapshots, clones (great for upgrading), fast incremental backups, checksums, not having to size your filesystems by hand and generally fsck with LVM, to mention a few I'm using. But yeah, for handling PC floppies Solaris probably sucked.
OpenSolaris governing board threatens dissolution (The H) Posted Jul 15, 2010 23:59 UTC (Thu) by clump (subscriber, #27801) [Link]
It must burn you up that no matter how highly you regard ZFS it's not stopping Linux from burying Solaris.
OpenSolaris governing board threatens dissolution (The H) Posted Jul 16, 2010 11:18 UTC (Fri) by trasz (guest, #45786) [Link]
It must burn you up that no matter how highly you regard Linux it's not stopping pretty much everyone from ignoring it on desktop ;->
OpenSolaris governing board threatens dissolution (The H) Posted Jul 20, 2010 14:01 UTC (Tue) by nix (subscriber, #2304) [Link]
While Solaris on the desktop is just *everywhere*, yeah.
(perhaps in a better world...)
|
|||||||||||||