I guess it's been considered already, but something like iptables or the Linux socket filter make sense to me. Provide unprivileged userspace with a small handful of operations for testing syscall number, doing comparisons and jumps, and looking strings up in a set, then leave the rest to a userspace library (actually I guess underneath this is probably what OS X' sandbox looks like too). That way extending the interface later is only a matter of adding extra operations.
If the overhead for running the pseudocode on each syscall was too high, then perhaps a declarative approach would be possible, where the kernel could transform the supplied rules into lookup tables, or some hybrid combination of both.
Posted Oct 20, 2011 8:09 UTC (Thu) by dw (subscriber, #12017)
[Link]
Grumble, slightly incomprehensible comment. By mention of the library and "unprivileged userspace", I meant something like how BPF or iptables works, where complexity of parsing some expression (or rule set) is handled by a library, which produces easily verifiable byte code, which is then handed off to the kernel.