LWN: Comments on "OpenSolaris governing board threatens dissolution (The H)"

OpenSolaris governing board threatens dissolution (The H)

trasz — Fri, 06 Aug 2010 09:29:55 +0000

I never said disabling interrupts in spinlocks was crazy - I said systems using interrupt threads and fully functional mutexes don't need to disable interrupts, because there is nothing wrong with interrupt thread blocking on mutex.

As for the speed comparisons - take a look at FreeBSD. It has spinlocks - which disable interrupts - but their use is discouraged, because they _are_ slower. If threaded interrupts handlers in Linux result in worse performance, this might be caused by poor implementation.

OpenSolaris governing board threatens dissolution (The H)

trasz — Fri, 06 Aug 2010 09:20:25 +0000

Spinlocks - yes, RCU - no; seqlocks seem to look like a reader/writer locks, so probably yes. Solaris - yes, but for nothing important (i.e. paid).

Btrfs patent problems??

trasz — Fri, 06 Aug 2010 09:19:01 +0000

See http://www.sun.com/lawsuit/zfs/.

OpenSolaris governing board threatens dissolution (The H)

trasz — Fri, 06 Aug 2010 09:12:15 +0000

That was the plan few years ago - when you look at IBM's position on Linux shortly after they got involved (2000, IIRC?), the plan was to phase out AIX. They changed their mind few years ago.

OpenSolaris governing board threatens dissolution (The H)

nix — Thu, 22 Jul 2010 21:02:14 +0000

Quotas and rquotad for quotas over NFS on the inevitable giant uni NIS/NFS/automounted network.

Those were the days...

OpenSolaris governing board threatens dissolution (The H)

vonbrand — Tue, 20 Jul 2010 21:48:41 +0000

Bah... Solaris 2 (or some such) had disk quotas. That's where I learned about them, in fact.

For high-subscriber-number readers

nye — Tue, 20 Jul 2010 15:13:36 +0000

That was an excellent summary.

OpenSolaris governing board threatens dissolution (The H)

nix — Tue, 20 Jul 2010 14:04:41 +0000

Also, OpenPROM always works. Intel IPMI BMPs lock up all the damn time: more often than not, it's locked up when you want to use it, and to fix it you have to *pull the power cord out*.

OpenSolaris governing board threatens dissolution (The H)

nix — Tue, 20 Jul 2010 14:01:41 +0000

While Solaris on the desktop is just *everywhere*, yeah.

(perhaps in a better world...)

For high-subscriber-number readers

nix — Tue, 20 Jul 2010 13:44:00 +0000

No. Ulrich *can* sometimes be convinced that his position is wrong. It's just very difficult (probably a good attribute for a maintainer of something as critical as glibc), and an extremely unpleasant experience (because he doesn't know what politeness is for), but that doesn't mean he's unpersuadable. He's hard to work with but not impossible. glibc's community is unhealthy but it sort of exists, even if much of it has had to quasi-fork off to stay out of Ulrich's way. Ulrich does not appear to object to the existence of eglibc, so this compromise appears stable.

Jorg Schilling, as far as I can tell, is completely unpersuadable. I've never seen him reverse his position on *anything*, even if it is something that is outright loony or has been rendered completely wrong by the passage of time. I suspect that his much-laughed-at devotion to referring to devices by SCSI IDs is an example of the latter: back in the late 80s this was probably the right decision, as nearly all CD burners worthy of the name were horrifically expensive SCSI-only devices and OSes had few other ways to address them, but when better device frameworks emerged -- in Solaris first! -- his mind was made up and could not change. By the time he had to invent fake SCSI IDs to account for the fact that nearly all CD burners weren't actually SCSI anymore his position had moved, gradually, from reasonable to demented. Plus, his attitude to forks is best described as 'extremely jealous'. It's the last part that makes his software best avoided, in my opinion.

Softpanorama

Trelane — Tue, 20 Jul 2010 12:50:57 +0000

'"unauthorized biography" of Linus Torvalds that's just full of bullshit (I use the term in its technical sense) ?'

you mean "bulshytt"? ;)

(for those that need an explanation: http://anathem.wikia.com/wiki/Bulshytt)

For high-subscriber-number readers

nye — Tue, 20 Jul 2010 12:33:14 +0000

>I have never in my life encountered anyone worse at negotiation or diplomacy.

Ulrich Drepper?

OpenSolaris governing board threatens dissolution (The H)

dlang — Mon, 19 Jul 2010 05:27:41 +0000

linux has clustering, several different variations so that you can pick which one applies to your problem best.

as for it working 'out of the box', that depends on how you define that phrase.

I believe that Linux also has support for GFS

hardware redundancy is not an operating system feature, it's a hardware feature. Put the question another way, why should someone use Solaris instead of Linux on commodity hardware?>

OpenSolaris governing board threatens dissolution (The H)

cmccabe — Sun, 18 Jul 2010 20:04:14 +0000

You're right-- not all spinlocks do disable interrupts. I was just replying to trasz's accusation that disabling interrupts in spinlocks was a crazy, Linux-only idea. I guess I was kind of feeding the troll, sigh.

Traditionally in Linux you used spin_lock_irqsave when you needed to modify the same variable in an interrupt handler as in some other kernel code.

I know that lately there's been a move towards threaded IRQ handlers and less use of CLI / STI. The PREEMPT_RT patchset, which I've used previously at work, replaces most spinlocks with mutexes in order to get better maximum latency performance. It would be interesting to see a comparison between Linux + PREEMPT_RT and Solaris's architecture. As far as I know, the selling point for these architectural changes is reduced latency rather than greater throughput-- which, again, is contrary to what trasz is saying.

switching for virt envs

mcmanus — Sun, 18 Jul 2010 13:16:50 +0000

I understand that VEPA is the likely candidate to deal with some of the shared uplink issues of virtualization.. it basically moves all the switching functionality, including hairpin routing, back onto the switch..

impt for the sr-iov hardware too.

http://lwn.net/Articles/337547/

For high-subscriber-number readers

nix — Sun, 18 Jul 2010 12:26:08 +0000

He's on the OpenSolaris governing board? So much for anything that requires said board to negotiate with anyone, then. I have never in my life encountered anyone worse at negotiation or diplomacy.

OpenSolaris governing board threatens dissolution (The H)

johill — Sun, 18 Jul 2010 09:55:30 +0000

Err, a spinlock that doesn't need to disable interrupts, say because it is not used from interrupt context will degrade to a no-op on UP systems. Questioning the usefulness of such lock is quite pointless since the UP machine cannot be preempted in that spot by code design.

And something that the start of this thread failed to mention is that Linux doesn't actually always disable interrupts for spinlocks. Only those that are also used from interrupt context need to disable interrupts. The others still make sense on SMP systems. It's why there's spin_lock_irqsave/spin_lock_bh/spin_lock.

OpenSolaris governing board threatens dissolution (The H)

tzafrir — Sun, 18 Jul 2010 09:21:20 +0000

Many higher-end x86 servers have similar capabilities. Sadly relatively few of them are supported by coreboot or anything similar, and you're stuck with a slow and bloated proprietary BIOS.

OpenSolaris governing board threatens dissolution (The H)

cmccabe — Sun, 18 Jul 2010 08:56:38 +0000

> Thus, the whole "thread switch outweighs the cost of disabling interrupts"
> argument is invalid - it's not "linux spins, other systems switch"; it's
> "linux has to disable interrupts while spinning, and other systems don't".

How useful is a spinlock that doesn't disable interrupts on a uniprocessor system? Think about it carefully.

Another two questions: have you ever written code that used spinlocks, RCU, or seqlocks? Have you ever been the system administrator for a Solaris system?

A simple yes or no will do.

OpenSolaris governing board threatens dissolution (The H)

fredi@lwn — Sun, 18 Jul 2010 07:04:35 +0000

I forgot, but well, is not related to solaris exactly, it's more about sparc. OBP (http://www.softpanorama.org/Solaris/Startup_and_shutdown/...)

It's a life saviour when you are far from the machine's room.

OpenSolaris governing board threatens dissolution (The H)

fredi@lwn — Sun, 18 Jul 2010 06:58:03 +0000

Features (in short)
IPMP (http://en.wikipedia.org/wiki/IPMP)
MPXIO (http://en.wikipedia.org/wiki/Solaris_Multiplexed_I/O)
GFS (sorry, no quick link here but in short it's a cluster block device)
Zones that work out of the box with separate network stack (paravirtualisation, at least on vanilla linux kernles it's nearly impossible to do it)
Clustering that works out of the box (just few nodes though, max 16 but it's HA)
Hardware fault tolerance.
Dtrace/ZFS/ co, but others pointed that

It is a slow but damn stable system.

As about graphics, it sucks, but i dont think that was the target for them.

disclaimer: I've worked a lot with solaris but i have linux on my laptop from ~ 2001.

OpenSolaris governing board threatens dissolution (The H)

dwkunkel — Sun, 18 Jul 2010 03:47:00 +0000

Ten years ago our large data centers where 80% Solaris. They are now 70% Linux and the remaining Solaris servers are being phased out. Windows servers account for most of the remaining hosts.

OpenSolaris governing board threatens dissolution (The H)

dlang — Sat, 17 Jul 2010 21:22:25 +0000

this is exactly like a normal switch, but if all you want is a normal switch you won't be asking how to dedicate a physical ethernet port to a particular virtual machine, you would just connect all the virtual machines to your virtual switch and have an 'uplink interface' out of your machine.

when you want to dedicate an ethernet port to a particular virtual machine you don't want the host OS to short-circuit traffic between mirtual machines, you want the traffic between virtual machines to go out over the wire.

OpenSolaris governing board threatens dissolution (The H)

cesarb — Sat, 17 Jul 2010 14:31:57 +0000

How is this different to a normal Ethernet switch then? Nowadays, Ethernet switches also avoid sending traffic to where it is not needed. The bridge is just a virtual Ethernet switch, acting like a physical one.

OpenSolaris governing board threatens dissolution (The H)

dlang — Sat, 17 Jul 2010 04:55:47 +0000

you are the first person I have ever seen use the term leech and mean it to be a positive thing :-)

every other time I've seen it used (in most cases with almost the exact same statement you made), it's being used to say that it's not fair that IBM would bet getting so much benefit and they should be punished (if only by preferring the work of some other company that isn't a leech)

the term 'leech' strongly implies (if not outright states) that you are taking something away from the host that it can't use anymore for your own benefit.

OpenSolaris governing board threatens dissolution (The H)

giraffedata — Sat, 17 Jul 2010 01:55:44 +0000

what amount of contribution to linux changes a company from being a 'leech'

If you mean "leech" in a morally negative sense, then I have no opinion on that. I don't much care about morality of business and I didn't mean to say anything about IBM's hypothetical morality if it hypothetically decided to start using Linux instead of AIX to sell its hardware.

I was only talking about A taking advantage of work that B did for some purpose other than to serve A. And I do think that's an honorable way to increase the wealth of the world.

OpenSolaris governing board threatens dissolution (The H)

dlang — Sat, 17 Jul 2010 00:42:01 +0000

the bridging solution doesn't actually work if you have multiple virtual machines on the same network.

or more precisely, it works in that it gets packets from one virtual machine to another, but it doesn't work in terms of making that traffic go through the interface that you bridge to. the host kernel will short-circuit the communication between virtual servers and not send the packets out over the wire, just deliver them to the destination if it's on the same box.

for most people this is the best thing to do, but there are cases where there are requirements for monitoring/controlling the traffic between virtual servers where you really do want to force the traffic out over the wire.

OpenSolaris governing board threatens dissolution (The H)

dlang — Sat, 17 Jul 2010 00:34:43 +0000

what amount of contribution to linux changes a company from being a 'leech'

if 88% of the development is done by other companies, is that company a leech? if so Redhat is a leech. If that's the line where they are no longer a leech, then only 8 companies in the world can not be a leech.

OpenSolaris governing board threatens dissolution (The H)

joib — Fri, 16 Jul 2010 22:37:44 +0000

In Linux spinlocks are meant for situations were the lock is held for a very short time. This means that code which holds a spinlock may not block, and secondly, by disabling interrupts it ensures that the processor does not go off to do something else with the spinlock held, such as servicing interrupts, or switching to some other thread.

Or to put it another way, since the lock is held for a very short time, and the thread holding the lock can't be bumped off the cpu, a lightweight spinlock is the correct locking primitive.

One way to see the performance impact of mutexes vs. spinlocks is to benchmark with the hard realtime patches (PREEMPT_RT). These patches replace most of the spinlocks with (IIRC priority inheriting) mutexes, since spinlocks cannot guarantee that the highest priority process will always get the lock. Due to this, and some other reasons, the PREEMPT_RT kernel is a bit slower as well as scales much worse than the vanilla kernel.

As an aside, this has nothing to do with threaded interrupt handlers (which FWIW are also available in Linux, although so far not widely used) vs. the traditional bottom/top half+workqueue/whatever interrupt handlers.

As another aside, AFAIK disabling and re-enabling interrupts takes relatively little time; I doubt you can actually measure the performance hit due to this.

As yet another side, Linux also has adaptive mutexes. E.g. the BTRFS file system uses them, although I believe they are otherwise relatively rare.

The argument that fast paths are equally fast in spinlocks as in mutexes, well, duh. Of course. Most locking mechanisms have pretty fast paths. The reason why there are many of them is that we're interested in different behavior under contention. In some cases the appropriate behavior is to spin, in other cases sleeping is better, and in yet other cases some lockless mechanism such as RCU is appropriate.

OpenSolaris governing board threatens dissolution (The H)

giraffedata — Fri, 16 Jul 2010 21:12:26 +0000

It's not as if IBM isn't taking part in Linux (kernel) development.

Right, but I didn't say anything related to IBM's current relationship with Linux. I posed a hypothetical situation where IBM is able to sell its hardware with Linux as developed by others, in order to make a point about whether IBM makes a profit on AIX.

Of course, the hypothetical still works if others do 95% of the development of Linux, which is how it is today.

OpenSolaris governing board threatens dissolution (The H)

anselm — Fri, 16 Jul 2010 20:33:41 +0000

It's not as if IBM isn't taking part in Linux (kernel) development. See, e.g., Jon's stats in last week's issue. As companies go they are fairly high up on the list – not anywhere near Red Hat, to be sure, but certainly ahead of most of the others.

If that is »leeching off others' Linux development« then pretty much everybody is doing it (and their dog, too). Even Red Hat, the leader by a wide margin, has contributed only not quite 12% of the changesets in 2.6.35. There's no point in bashing only IBM about this.

OpenSolaris governing board threatens dissolution (The H)

giraffedata — Fri, 16 Jul 2010 19:03:38 +0000

I don't think IBM makes big money on AIX licenses. They're selling hardware.

I'm pretty sure you're right about that. If IBM could get Linux to do everything on IBM's hardware that AIX can (in IBM's opinion), IBM would be happy to dump the AIX development cost and leech off others' Linux development.

OpenSolaris governing board threatens dissolution (The H)

Cyberax — Fri, 16 Jul 2010 13:04:51 +0000

You can give each OpenVZ machine its own network card (they are named veth1, veth2, ...). You can bridge them with your existing card, build vlans or route them using iptables etc.

AFAIR, you can also give full control over a hardware card to an OpenVZ container, but I hadn't tried that.

OpenSolaris governing board threatens dissolution (The H)

trasz — Fri, 16 Jul 2010 11:18:03 +0000

It must burn you up that no matter how highly you regard Linux it's not stopping pretty much everyone from ignoring it on desktop ;->

OpenSolaris governing board threatens dissolution (The H)

trasz — Fri, 16 Jul 2010 11:16:12 +0000

I don't think IBM makes big money on AIX licenses. They're selling hardware. If they wanted to sell AIX, they wouldn't publish Linux benchmarks for HPC and instead they would push AIX there as well.

OpenSolaris governing board threatens dissolution (The H)

trasz — Fri, 16 Jul 2010 11:08:32 +0000

That's not how it works. When held for a short time, mutex (todays mutexes are always adaptive, at least in Solaris and FreeBSD) doesn't switch threads - it spins, just like spinlock. Thus, the whole "thread switch outweighs the cost of disabling interrupts" argument is invalid - it's not "linux spins, other systems switch"; it's "linux has to disable interrupts while spinning, and other systems don't".

I'm not sure about what measurements you're talking about. I guess you mean some Linux implementation - but then, if they were slower, that just proves that they were badly implemented. Let me repeat: fast path in a mutex (in operating system other than Linux) is pretty much the same as fast path of spinlock in Linux, minus messing with interrupts.

There is one more thing you missed - results. I've already mentioned that IBM doesn't publish high end server workloads benchmarks (Oracle, SAP etc) under Linux if they can use AIX instead. What they do publish for Linux is stuff that doesn't spend much time in the kernel by definition (i.e. HPC), and is thus not so demanding when it comes to scalability - because it doesn't matter if kernel needs to waste twice as much CPU time, if the total kernel time is 1% and the rest is userland.

OpenSolaris governing board threatens dissolution (The H)

mpr22 — Fri, 16 Jul 2010 10:51:02 +0000

Now to my mind, the natural response is "Of course they don't. That might undermine AIX sales."

OpenSolaris governing board threatens dissolution (The H)

trasz — Fri, 16 Jul 2010 10:47:49 +0000

I've already mentioned three comments above that HPC stuff is very different from high end server workloads. IBM doesn't benchmark high end server workloads under Linux on machines supported by AIX, as demonstrated in your PDFs. (Is this thread really so hard to follow?)

OpenSolaris governing board threatens dissolution (The H)

cesarb — Fri, 16 Jul 2010 10:27:06 +0000

There is nothing wrong with reporting something you ought to be able to analyze yourself but did not have the time to analyze. Besides, some things might take you hours or even days to fully analyze, but once pointed to a developer of the project, the developer sometimes is able to fully analyze and create a fix in minutes.

OpenSolaris governing board threatens dissolution (The H)

clump — Thu, 15 Jul 2010 23:59:00 +0000

It must burn you up that no matter how highly you regard ZFS it's not stopping Linux from burying Solaris.