LWN: Comments on "SMP alternatives"

Single kernel image for UP & SMP

tyhik — 2006-06-11T16:14:42+00:00

RAM and flash ships get larger and larger all the time. For smaller design companies it may already be cheaper to get 4mb than 2mb flash chip.

But of course the code size matters. Smaller code implies better cache usage.

Single kernel image for UP & SMP

niner — 2006-06-09T23:42:00+00:00

I can't imagine that. Think of embedded applications. When you only have 2MB of flash and a couple of megabytes of RAM, you really start to care about the few kilobytes such a feature costs of both memory footprints.

That's why I love Linux...

cventers — 2006-06-01T16:41:21+00:00

Is it a tester that no one else on Earth owns? :)

I read that GregKH re-emphasized at the recent FreedomHEC that the kernel
has drivers with only a handful of users. Perhaps mainline inclusion
isn't an impossible thing after all.

CPU usage optimisation

xoddam — 2006-03-30T05:20:29+00:00

Exactly. The CPU configuration and speed should be adjusted according to
demand for processing power, not supply of electricity.

There is never a case for consuming more power just because you *can*.
Otherwise we'd all leave all our computers and all our lights switched on
all of the time. Bring on global warming.

SMP alternatives

turpie — 2005-12-27T23:17:54+00:00

He meant that it saved the major cost of shipping different kernels, by allowing them to ship a single kernel for both UP and SMP.

SMP alternatives

cajal — 2005-12-27T17:13:56+00:00

If I'm interpreting your post right, you're saying this is a bad thing. A 2-5% improvement is pretty negligible, and it came at a major cost. Sounds like this is something Linux should avoid.

Single kernel image for UP & SMP

efexis — 2005-12-25T19:59:25+00:00

There's no reason why you would have to do that (I won't repeat my last post). The speed differences would be down to the processor having to deal with the few noop's, eating up slightly more L1/L2 cache, and keeping in memory (and the kernel image file itself) - for those who worry about that - code for both scenario's.

SMP alternatives

efexis — 2005-12-25T19:51:52+00:00

No. This is talking about removing code once you've already switched down to a single processor, and adding code when you want to switch up.

Adding/removing processors at runtime, I believe, is already possible (otherwise this code would be pretty useless)

SMP alternatives

efexis — 2005-12-25T19:46:40+00:00

Um, my guess is that you'd stop processes from running on the second processor -before- switching down to a single processor kernel. As only one processor could then even be holding locks, the locking mechanism becomes irrelevant, so can be NOOPed.

Single kernel image for UP & SMP

renox — 2005-12-22T23:30:57+00:00

Well, as some others have remarked for this to work you have to remember which (spin)locks are taken, even on a UP processor thus loosing some performance in UP.

I think that even a minimal loss would be a hard sell to kernel developpers considering how little this UP<->SMP feature is going to be used..

Single kernel image for UP & SMP

zdzichu — 2005-12-22T15:41:01+00:00

Could this patch cause removing CONFIG_SMP in future? With kernel image identical for SMP and UP machines and runtime patching?

User base

rrw — 2005-12-22T11:59:48+00:00

A dual core system is essentially SMP... and laptops are very power / performance concious. It would not hurt in the slightest to eke out every bit of speed (or efficiency, ie potentially less power use) while on a half-clocked single-core CPU for battery saving, then safely switch to full-bore dual-core mode when on mains.

OK, I have a problem with this. Why would I need to clock down a notebook on batteries and clock it up on mains?

When a Centrino notebook on mains works with full speed it soon gets so hot that the fan works non stop, and the keyboard itself gets annoyingly warm.

On the other hand, when notebook runs on battery and I want to do something cpu intensive it is painstakingly slow. But where's the power consumption advantage? I have to do this anyway, slower or faster, and if I do something with downclocked CPU, power consumption per unit of time in display, gpu, harddisk doesn't decrease, so I actually eat more juice.

Isn't it better to just use sth like powernowd, which dynamically clocks processor depending on system load, wheather you use mains or battery?

Robert

User base

ringerc — 2005-12-22T11:22:12+00:00

The potential user base is larger than hot-plugging SMP users and Xen users. The many of the next generation of laptops will have dual core CPUs, and it's likely to be desirable to shut down a core when on battery (for example).

This patch seems like a really good way to handle that need.

SMP alternatives

csamuel — 2005-12-21T07:08:58+00:00

No.

Xen is still running on a single system and whilst you can migrate a
system between Xen servers (if you've got a shared filesystem that can
cope) it only runs on one or the other with a tiny pause for the actual
transition.

SMP alternatives

bk — 2005-12-18T19:26:09+00:00

Can these processors be on different physical machines? This sounds like an implementation of SSI clustering, which is pretty cool.

Internal-use kernel code

Duncan — 2005-12-16T19:42:15+00:00

Well, I'm aware of the difference, but didn't go to my usual lengths to
specify it. How come other folks can take shortcuts, but every time I
abridge a detail, I get called on it? <g> I suppose it's likely because
folks are used to me being so detailed, tho some might be just
coincidence.

Anyway, yeah, thanks for making the code available, even if it's not
targeted at the kernel tree ATM. That's an important right of Free
source, too, being able to NOT have to go for merge, if desired, tho
because it's Free source, others can take it and go for that merge if they
want to, again, another important right.

Thanks also for filling in those details. It's quite possible the
additional information will be of use to future readers, and I /did/ fail
to mention it, so it's good someone came along to fill the gap. =8^)

Duncan

SMP alternatives

norsk — 2005-12-16T17:38:19+00:00

Back in 1998, while working at Novell on the Netware SMP project, I did the same thing on self-modifying kernel mods. The kernel was built in SMP mode and when installed on a UP system, all the "lock" instructions, SMP and atomics called their respective init routines to determine whether UP or SMP and applied the correct op-codes. Gave us 2-5% improvement and a major cost in shipping of different kernels.

I like the idea of reversing the mods when a CPU hotplug event occurs. Hardware at my time did not have the feature set.

I was wondering why this had not yet happened before in Linux. Still a far better world to work in, then the Netware kernel.

doug "norsk" thompson

Internal-use kernel code

giraffedata — 2005-12-16T16:51:10+00:00

You silently merge the idea of making source code available and of getting it in the kernel.org tree. You also seem to merge, as many people do, the idea of submitting code for inclusion and of that code being included.

There's a significant cost to getting code into kernel.org. I myself write lots of kernel code, and while the world is welcome to all of it and much of it is published, I have never attempted to get any of it into kernel.org.

First, I'd have to translate it to a coding style I don't like and package it according to some pretty specific rules. Then I'd have to run the gauntlet of some mailing list, probably having to rework the code a few times. Some of that rework would be stuff I don't agree with. Some would be stuff I have no use for. At no point would I have any guarantee my work would result in any code going in.

So I suffer the costs of being out of tree (mainly, I can't use the most current kernel.org code), but it beats the cost of getting in tree.

SMP alternatives

captrb — 2005-12-15T20:14:23+00:00

Can anybody speculate whether this functionality will make various
power management easier on SMP machines? It seems (without any
knowledge to back it up) that each CPU could be removed until there
was a single CPU left, then the same procedure that works on
uniprocessor machines could be performed.

It's a really shame, especially with dual core CPU's and
hyperthreading, that there is a choice between fancy power
management and multiple processors.

SMP alternatives

jzbiciak — 2005-12-15T18:59:59+00:00

To enable hotplug SMP -> UP -> SMP, you'd definitely need to retain spinlocks, or at least put a lightweight "take lock" instruction there as opposed to the full spin. Fully NOPping spinlocks out would be a disaster, as you note.

Elsewhere, though, you could nuke/replace LOCK prefixes as needed.

SMP alternatives

Ross — 2005-12-15T18:39:09+00:00

That's not how I read it.

"The main use of SMP alternatives in his patch is with spinlock operations; spinlocks can be patched in or edited out, as dictated by the configuration of the system at boot time."

It sounds like spinlocks will be turned into noops. This may be ok when going SMP->UP, but maybe not the other direction, and I wonder what kind of lock state would be retained when going SMP->UP->SMP...

That's why I love Linux...

sbishop — 2005-12-15T16:18:01+00:00

Yea, I suppose that I should have seen that coming. :)

I work for a manufacturing company. The hardware is a tester that only we'd be interested in. And the driver really isn't far distanced from usb-skeleton.c, so it very interesting either.

SMP alternatives

bronson — 2005-12-15T15:26:40+00:00

Yes, as long as you have more slices per second to give.

Once you've maxed out a single processor, your only recourse is to run the VM on more processors. AFAIK there's no good way of doing this unless the VM is partitioned (parallelized?) as well. Multiple virtual CPUs is the best way of doing this.

But the VM doesn't care! why not always run it in a quad-cpu configuration? Because that wastes cycles if it's just being run on a single CPU.

That's just a guess. It seems an overcomplex solution to me but I haven't looked at the code yet.

Safety

corbet — 2005-12-15T14:55:08+00:00

Changing the functioning of spinlocks could certainly create trouble if parts of the kernel are certainly holding locks! By my reading of the patch, there are a couple of defenses against that problem, though:

A kernel built with the SMP alternatives maintains the counters for spinlocks, so lock state should be preserved in all configurations, and
The hotplug CPU code has to quiesce the system anyway, so no atomic code should be running while alternatives are being applied.

SMP alternatives

kpfleming — 2005-12-15T14:05:24+00:00

You are confusing mutual exclusion/semaphore locks with 'bus' locks. There is no 'unlock' operation involved here; the 'lock' prefix instruction being referred to only affects the instruction it precedes, and there is an automatic 'unlock' of the bus when that instruction completes.

Switching from uni- to multi-processor mode won't even require holding all the kernel threads/processes in an idle state while this happens, it would just have to complete all the instruction patching before any threads could be allowed to run on the new CPU.

This is a very, very cool idea :-)

SMP alternatives

NAR — 2005-12-15T12:35:25+00:00

Virtualization systems (and Xen in particular) are implementing the ability to configure the number of (virtual) CPUs in each running instance on the fly, in response to the load on each. So it may really be that a busy, virtualized server will have CPUs hot-plugged into it, and that those processors will go away when the load drops.

Why bother with hotplugging virtual processors? Isn't it simpler to add more resources (e.g. more CPU splices per second) on the host to the particular virtualized server?

Bye,NAR

That's why I love Linux...

Duncan — 2005-12-15T10:35:52+00:00

I'm certainly no MSWormOS fan, and I like your point, but in actuality,
from what I've read, MS is one of the better proprietaryware venders in
regard to multicore, at least. While the likes of Oracle continue to bill
per core, MS has been looking pretty flexible by comparison, saying it
will continue to license per CPU, even as the number of cores per CPU
begins to climb.

As to the USB driver, if it's useful for you, unless you are developing it
for as yet unreleased hardware, it's almost certainly going to be useful
to others as well. The GPL doesn't require publishing code for internal
work, but consider the benefits of either /not/ having to do that internal
maintenance, as it's done for you by the kernel crew, or submitting it for
kernel inclusion and becoming the maintainer yourself, thus gaining
visibility and favorable press throughout the Linux world, /plus/ having a
bit of the work still done for you, when someone changes out kernel code
you depend on from under you.

Of course, you likely have your reasons for not doing so, but come on, you
gotta admit such a posting to LWN is just /begging/ someone to ask why you
haven't yet made source available! <g>

Duncan

SMP alternatives

dw — 2005-12-15T09:44:18+00:00

Please excuse my ignorance of kernel development, but I don't understand how this can be made to work safely. I am not sure under which conditions CPU hotplug events are processed, but I would imagine that at that time numerous locks in the kernel would be in the locked state.

If the CPU hotplug code causes the unlock functions to become a NOOP, then after the event has been processed, any scheduled tasks that perform an unlock of an existing lock will not cause any change in the state of the kernel.

If the kernel is then patched for SMP again, and numerous locks which were 'unlocked' by tasks running under the non-SMP kernel are actually still marked in memory as locked, would that not cause severe system instabilty or a deadlock condition?

Again, I only know the kernel by concept, and have never written a line of code for it. Excuse my ignorance. :)

That's why I love Linux...

sbishop — 2005-12-15T03:19:57+00:00

The difference between this idea and the way Windows works is incredible. Self-modifying/optimizing code versus "Another processor? Do you have a license for that?" Wow.

I've realized recently that Linux often benefits technically from its license. This idea is an example of that.

I've seen another example at work, where I maintain a Linux USB driver used internally. It's based off the example USB driver that's included in the Linux kernel code, and it's about 600 lines long. The corresponding example Windows driver that comes with Microsoft's DDK (Driver Development Kit) consists of multiple files and around 8,000 lines total. Why the difference? Because the Linux developers aren't tied to an ABI, or API, for that matter. They can adjust the interface between the kernel and drivers (because they have the source for those too) until they get it right. The Windows driver contains much that really ought to be built-in. A +10x difference in code size--that's huge.

And guess which is more than twice as fast than the other? :)