Lua in the kernel?

Posted Sep 9, 2020 17:37 UTC (Wed) by smurf (subscriber, #17840)
Parent article: Lua in the kernel?

I wonder whether it'd be possible to run a benchmark of something like this test in both userspace interpreted-Lua and LuaJIT, in order to get a handle on the actual performance difference after optimization. Comparing BPF to "plain" Lua is hardly fair, after all, and 500x slowdown somewhat unacceptable.

On the other hand, if nothing else running Lua is easily two orders of magnitude more accessible than BPF, to people who want to actually get things done instead of fighting with a BPF compiler and its limitations. (Been there, done that, thrown in the towel in favor of more satisfactory pursuits.) This alone is a rather powerful argument in favor of adding Lua IMHO.

Lua in the kernel?

Posted Sep 10, 2020 6:24 UTC (Thu) by amarao (guest, #87073) [Link] (2 responses)

500kpps is not kernel scale number. You can have the same performance on the usual proxy servers in userpsace. Lua in the kernel sounds like 'oh, I have a hammer. Isn't it a nice nail shines out there?'. Why in the kernel? Userspace was created for those things, wasn't it?

Lua in the kernel?

Posted Sep 10, 2020 7:30 UTC (Thu) by smurf (subscriber, #17840) [Link]

You can say the same thing about BPF. A BPF interpreter should be about as fast as non-JITted Lua.

Transitioning each and every message to userspace requires a context switch (and/or a cache transfer between CPUs) at minimum. That kills performance.

Lua in the kernel?

Posted Sep 10, 2020 7:30 UTC (Thu) by edeloget (subscriber, #88392) [Link]

Doing it in the kernel means that you still use the kernel network stack. If you plan to do it in user space as afast as possible then you have to extract the network stack (for example using DPDK) which is a burden of its own (but possible: we're doing it ; it's not pretty and we still have rare bugs we do not understand).

(but for the record, I also think this is a hammer-based solution; there is a limit to what the kernel should do "in the name of performance"; does this announce the return of the HTTP server module?)

Lua in the kernel?

Posted Sep 12, 2020 13:11 UTC (Sat) by foom (subscriber, #14868) [Link] (1 responses)

If someone is willing to add Lua, they could also eliminate the artificial restrictions of bpf.

The current state of things is really unfortunate, because you cannot _actually_ compile C to bpf. Well, rather, you can compile, but most nontrivial programs will be rejected by the bpf verifier when you try to load it into the kernel.

The verifier requires that all programs be statically analyzable to determine that the program will terminate, and that it is statically impossible for the program to ever access out of bounds memory. There is, by design, no runtime "failure" -- every bpf program that passes the verifier is guaranteed to terminate with a valid return value from the end of the program.

This is certainly an interesting property! But, this makes bpf very much not a generic architecture, and furthermore, impossible to reliably target with a C compiler. Even if you've carefully written your code to be verifier-safe, normal compiler optimizations may foil the static analysis, and break your program.

Yet, these verifier restrictions are not necessary -- *if* you're willing to admit the possibility of a program which can (safely!) fail to return a result when run.

Supporting Lua necessarily admits such a possibility of failure (e.g. by execution timeout, excessive memory allocation, etc etc). If that's acceptable, the same could surely be allowed for bpf, and result in a more generally usable system.

Lua in the kernel?

Posted Sep 18, 2020 9:29 UTC (Fri) by HelloWorld (guest, #56129) [Link]

The way the BPF verifier ensures termination appears to be rather simplistic. More sophisticated mechanisms do exist, such as these:

http://docs.idris-lang.org/en/latest/tutorial/theorems.ht...
http://ats-lang.sourceforge.net/DOCUMENT/INT2PROGINATS/HT...

However while this does allow for more complex programs, I'm not sure how useful it is in practice at this time. For one, even if you prove that a program will run for a finite amount of time, that amount may well be longer than the age of the universe. These approaches also require additional training and education for programmers to use them effectively.