kGraft — live kernel patching from SUSE

[Posted February 3, 2014 by corbet]

SUSE has announced the existence of kGraft, a mechanism for applying kernel patches without the need to reboot the system. It is similar to ksplice in functionality, but the implementation appears to be rather different and the developers plan to try to get it merged into the mainline kernel. "kGraft builds on technologies and ideas that are already present in the kernel: ftrace and its mcount-based reserved space in function headers, the INT3/IPI-NMI patching also used in jumplabels, and RCU-like update of code that does not require stopping the kernel. A kGraft patch is a kernel module and fully relies on the in-kernel module loader to link the new code with the kernel. Thanks to all that, the design can be nicely minimalistic." The first code release is planned for March.

kGraft — live kernel patching from SUSE

Posted Feb 3, 2014 19:37 UTC (Mon) by SEJeff (guest, #51588) [Link] (18 responses)

Ksplice has some patents on this (before and after the acquisition) and Oracle is a known patent troll:
https://www.google.com/?tbm=pts#q=ininventor%3A%22Jeffrey...

I do wonder how this will end up working out, but I seem to expect a legal fight ie: if SUSE ships this to customers, Oracle legal will be calling.

kGraft — live kernel patching from SUSE

Posted Feb 3, 2014 19:50 UTC (Mon) by k8to (guest, #15413) [Link] (9 responses)

Patents on the functionality, or the specific implementation tools used here?

kGraft — live kernel patching from SUSE

Posted Feb 3, 2014 19:58 UTC (Mon) by SEJeff (guest, #51588) [Link] (4 responses)

Look at a few of these. They are just the ideas:

https://www.google.com/patents/US20100269106?dq=ininvento...
https://www.google.com/patents/US8612951?dq=ininventor:%2...
https://www.google.com/patents/US8607208?dq=ininventor:%2...

Just saying these are awfully similar and I'd hate to see it shut down by patent trolling / FUD. This is something that would be fantastic to see for production quality kernel upgrades (for telcom style uptime requirements).

kGraft — live kernel patching from SUSE

Posted Feb 3, 2014 20:16 UTC (Mon) by oshepherd (guest, #90163) [Link] (3 responses)

It is little known, but Windows Server 2003 contains hotpatching support. Functions in the kernel and core system DLLs is compiled with a special prologue, so as to avoid the need to determine whether the existing code is in use. Space for a "long" jump is placed immediately before the function header (5 bytes on i386, JMP + 4 byte immediate), and the first instruction of the function is a "mov edi, edi" 2 byte NOP in order to permit a 2 byte short jump backwards to be atomically inserted (Jumping backwards to the start of that aforementioned long jump)

Safe hot patching is done by first setting up the long jump, then doing an atomic replacement of the 2 byte NOP with the backwards jump. No need to detect if the code is in use; anybody already executing the old code will successfully continue to do so.

This would require compiling the kernel with such prologues. The need to insert a 2 byte NOP can be obviated in cases where the first instruction is already >=2 bytes long.

On the one hand, this would permit working around the Oracle patents. On the other hand, there is a need to watch out for Microsoft patents..

kGraft — live kernel patching from SUSE

Posted Feb 3, 2014 20:34 UTC (Mon) by amacater (subscriber, #790) [Link]

Both SUSE and Oracle are part of the OIN patent pool. SUSE may also remain covered by the Novell-Microsoft patent agreements - unlikely that a full-on spat with Oracle will ensue.

kGraft — live kernel patching from SUSE

Posted Feb 3, 2014 22:16 UTC (Mon) by SEJeff (guest, #51588) [Link]

I was indeed ignorant of this functionality. That is outstanding. Any idea what uses this functionality?

kGraft — live kernel patching from SUSE

Posted Feb 4, 2014 1:02 UTC (Tue) by stressinduktion (subscriber, #46452) [Link]

Linux kernel is also compiled with -pg since some time when CONFIG_DYNAMIC_FTRACE is enabled which leaves some bytes (actually a call to mcount) free on the function prologue. ftrace hooks up there and can redirect the function. I guess all those patching could also be achieved by just writing a rather small script in systemtap with embedded-c.

kGraft — live kernel patching from SUSE

Posted Feb 3, 2014 20:43 UTC (Mon) by bokr (guest, #58369) [Link] (3 responses)

I thought Ericsson worked out that functionality in the 80's with Erlang.

kGraft — live kernel patching from SUSE

Posted Feb 4, 2014 3:30 UTC (Tue) by k8to (guest, #15413) [Link]

Ericsson definitely implemented a system where dynamic code patching was possible. It wasn't designed for typical C ABIs though.

kGraft — live kernel patching from SUSE

Posted Feb 5, 2014 17:13 UTC (Wed) by drag (guest, #31333) [Link] (1 responses)

Patents don't really cover ideas or concepts.

A patent covers a specific set of steps to accomplish something.

So unless Erlang did it's code injection in the same/similar enough manner (very arbitrary) as this new implementation then it's not really relevant to the patent situation.

kGraft — live kernel patching from SUSE

Posted Feb 6, 2014 7:11 UTC (Thu) by smurf (subscriber, #17840) [Link]

> A patent covers a specific set of steps to accomplish something.

Looking at a few software patents out there, I beg to differ …

NB: *its code injection.

kGraft — live kernel patching from SUSE

Posted Feb 3, 2014 22:16 UTC (Mon) by jhhaller (guest, #56103) [Link] (7 responses)

The biggest improvement to the state of the art in the ksplice-related tools was the ability to determine what actually changed in the kernel so that a minimal patch could be built automatically.

Live patching was a feature of telecom systems back in the 80's on a Unix base. Patching the kernel was the easy part. The hard part was patching shared libraries. Patches with data structure changes caused problems as well. Spare areas in structures and arrays were manually maintained and kept zeroed, so that they could be reallocated for a new piece of data. There was also a performance penalty, as all subroutine calls were indirect through a transfer vector, an array of jump tables. The entire array could be replaced, so that any particular process would only see the new functions when the process reentered its event loop.

Being able to live-patch the kernel without being able to live-patch shared libraries and long-running executables only addresses part of the problem. I expect that it would be valuable to clouds if the qemu process and it's libraries could be live-patched. But there will still be firmware to patched, and it's best to boot onto the new firmware when installed to be sure it works.

kGraft — live kernel patching from SUSE

Posted Feb 4, 2014 4:10 UTC (Tue) by joyuh (guest, #95216) [Link] (6 responses)

You can just use live migration to migrate a VM to a new version of QEMU.

kGraft — live kernel patching from SUSE

Posted Feb 4, 2014 8:56 UTC (Tue) by pbonzini (subscriber, #60935) [Link] (4 responses)

That doesn't help with upgrading the kernel inside the VM.

kGraft — live kernel patching from SUSE

Posted Feb 4, 2014 9:54 UTC (Tue) by dgm (subscriber, #49227) [Link] (3 responses)

You could use something like LinuxPMI or OpenMOSIX to migrate processes to the new kernel.

In fact, an alternative method to update the kernel would be to boot a second kernel and migrate processes to it using this mechanism.

Maybe one day.

kGraft — live kernel patching from SUSE

Posted Feb 4, 2014 13:37 UTC (Tue) by ledow (guest, #11753) [Link] (2 responses)

So long as you don't mind using 2.6 kernels and/or not having 9 things that a process could do that might stop it migrating (such as using I/O directly, having a memory-mapped file, etc.).

There's plenty of reasons that something like ksplice would be superior under certain workloads than trying to revive a (dead) concept / software project.

kGraft — live kernel patching from SUSE

Posted Feb 4, 2014 22:11 UTC (Tue) by dgm (subscriber, #49227) [Link]

There are hurdles to doing it right now, but no fatal flaws in the concept. ksplice may be "superior under certain workloads", but process migration is a good design idea that will be implemented anyway on the long term.

kGraft — live kernel patching from SUSE

Posted Feb 6, 2014 17:54 UTC (Thu) by cov (guest, #84351) [Link]

CRIU can do this and is very much alive.

kGraft — live kernel patching from SUSE

Posted Feb 4, 2014 14:22 UTC (Tue) by jhhaller (guest, #56103) [Link]

Live migration has its limits, including a short period of time between when the VM stops on the old host, the last few memory updates are forwarded to the new host, and the VM starts again. For some applications, that time may be long enough to be considered downtime, even if it's a fraction of a second. Those are the applications which benefit from live patching and reduction in the number of events where software must be updated.

kGraft — live kernel patching from SUSE

Posted Feb 4, 2014 7:45 UTC (Tue) by paravoid (subscriber, #32869) [Link] (3 responses)

Finally, someone reviving this, years after Oracle killed a project as useful as ksplice was (like they usually do...).

The timing is interesting though: this past year we've seen effort towards the (so-called) "secure boot" functionality and I can only assume from the description it'd be directly incompatible.

kGraft — live kernel patching from SUSE

Posted Feb 4, 2014 7:57 UTC (Tue) by mchapman (subscriber, #66589) [Link]

> The timing is interesting though: this past year we've seen effort towards the (so-called) "secure boot" functionality and I can only assume from the description it'd be directly incompatible.

I can't see why it would be incompatible if the updates themselves were signed.

kGraft — live kernel patching from SUSE

Posted Feb 4, 2014 17:11 UTC (Tue) by SEJeff (guest, #51588) [Link] (1 responses)

They never killed it. They made is "commercial" and available to OEL subscribers just like they are slowly doing with MySQL.

kGraft — live kernel patching from SUSE

Posted Feb 5, 2014 9:44 UTC (Wed) by cyperpunks (subscriber, #39406) [Link]

> They made is "commercial" and available to OEL subscribers just like they are slowly doing with MySQL.

Please stop this FUD.

Watch this to get a clue:

http://www.mysqlperformanceblog.com/2014/01/22/past-prese...