Accessing new kernel features from Python
Every release of the Linux kernel has lots of new features, many of which are accessible from user space. Usually, though, the GNU C Library (glibc) and tools that access the Linux user-space API lag behind the kernel releases. Geoffrey Thomas showed how Python programs can access these new kernel features as soon as the kernel is released in his "What's New in the Linux Kernel... from Python" talk at PyCon US 2025. While he had two examples of accessing new kernel features, the real goal of the talk was to demonstrate how to go about connecting Python to the Linux kernel.
He began by noting that the kernel and its interfaces are written in C, so
there would be a "tiny bit
" of C in the talk. He would be
explaining any of that, so "as long as you can
read Python, you'll be okay
". In addition, all of the code, his
slides, and more are available from his GitHub repository for
the talk.
Since the presentation, the YouTube video of it
has been released as well.
Creating files
So, he asked, "what are these new kernel features and why would you even
want to use them?
" Opening a file in Python looks something like this:
open("file.txt", "w")
That will open a file named "file.txt" for writing; if the file
exists, it will get overwritten, but if it does not, it will be created.
It turns out that the systemd developers needed to figure out which of
those two things happened, so they added a way to find out to the kernel.
He pointed to the first 6.12 merge window
article here at LWN as the place where he found out about the change.
![[Geoffrey Thomas]](https://static.lwn.net/images/2025/pycon-thomas-sm.png "Geoffrey Thomas")
The commit spells out the change, which added a new fcntl() command called F_CREATED_QUERY that programs can use to find out if the file was created when it was opened or not. fcntl() is a Linux system call, which is a way for an unprivileged user-space program to request something from the kernel; it is, effectively, a function call, he said, but also kind of like making an API call to a server. As with network-server APIs, languages like Python often have wrappers around system calls. The fcntl module in the Python standard library serves that purpose for fcntl().
But, when looking at the Python documentation, there is no mention of the new command. The biggest reason for that is because it is new, but it also has not really been documented anywhere other than in the kernel source. He sympathizes with the kernel developers, since he also skips documenting new features at times, but it does mean that those who want to use the feature need to dig for information on it.
"Being able to search source code and figure out just enough of a large
project to track down the info you need and not get lost is a valuable
skill
". He listed a few code-search options, starting with GitHub code
search, which searches its public repositories, though you need to be logged into the site to access it. Sourcegraph search searches
GitHub, GitLab, and a few other sites. Debian code search will look in
anything that is part of the Debian distribution, including projects that
are not in Git at all. And, of course, the "git grep" command is
invaluable if you are searching in a local Git clone. One that he did not
mention, but that is quite useful for searching the kernel (and its various
versions) is Elixir.
Since he has a clone of the Linux kernel repository, he used "git grep" to look for F_CREATED_QUERY. It appears in two locations, fs/fcntl.c, which looks like the implementation, and include/uapi/linux/fcntl.h, which is a definition in the user-space API (uapi) of the kernel. The latter file contains:
#define F_CREATED_QUERY (F_LINUX_SPECIFIC_BASE + 4)
Another "git grep" shows that F_LINUX_SPECIFIC_BASE is
1024, so the value of the constant for the new command is 1028. Thomas did note that the new command
shows up in a few spots in the kernel's tools directory,
which can be ignored for the purposes of this exercise, though that can sometimes
be useful for examples and tests using a feature.
He then turned to the do_fcntl() function in fcntl.c. That function matches fairly closely with the fcntl.fcntl() described in the Python documentation, so he tried using it in the Python REPL:
>>> F_CREATED_QUERY = 1028
>>> import fcntl
>>> a = open("foo.txt", "w")
>>> fcntl.fcntl(a, F_CREATED_QUERY)
1
>>> a.close()
>>> b = open("foo.txt", "w")
>>> fcntl.fcntl(b, F_CREATED_QUERY)
0
As can be seen, the file was created with the first open, and not on the
second. The code could then be wrapped up into a tiny library as follows:
import fcntl
F_CREATED_QUERY = 1028
def was_created(file):
return bool(fcntl.fcntl(file, F_CREATED_QUERY))
Even though it is only four lines long, that library does something that
nothing else in the Python API can do. It might even be worth uploading to
the Python Package Index (PyPI), he said;
beyond its immediate utility, it could perhaps serve as an example for
someone trying to support some other new fcntl() command in the
future.
Memory maps
A longstanding feature of Unix-like operating systems is to try to treat everything as a file; the tools and interfaces for working with files already exist, which makes it easy to access new features that way. Linux has long had the /proc filesystem, which has files and directories but is not anything that is stored on disk. Instead, it provides textual information about processes and other kernel objects, which Python programmers can access using the standard mechanisms for reading files. Every process in the system has a directory in /proc with its process ID as its name; various files in that directory give information about the process. For example:
$ ls -l /proc/4067/exe
[...] /proc/4067/exe -> /usr/bin/bash
Various utilities, such as ps and lsof, use
information from /proc.
The /proc/self directory refers to the directory for the current process, so one can look at the memory mappings for their running Python with the following:
>>> print(open("/proc/self/maps").read())
00400000-009d8000 r-xp 00000000 00:20 3872 \
/usr/bin/python3.13
[...]
f08ceae80000-f08ceaeba000 r-xp 00000000 00:20 8872 \
/usr/lib/aarch64-linux-gnu/libncursesw.so.6.5
[...]
That produces voluminous output about all of the memory mappings, which
includes things like the Python executable and all of the libraries used by
it. One way to (mostly) just look at the filenames that appear at the end
of the line would be:
>>> libs = {l.strip().split()[-1] for l in open("/proc/self/maps")}
>>> print("\n".join(libs))
That splits each line into fields and puts the last one into a set
using a set comprehension; it then prints each set element on its own line.
Thomas said he uses some variant of that frequently in order to check
library versions. Its output might look something like this:
[...]
/usr/lib/aarch64-linux-gnu/libexpat.so.1.10.0
/usr/lib/aarch64-linux-gnu/libz.so.1.3.1
[heap]
/usr/bin/python3.13
/usr/lib/aarch64-linux-gnu/libstdc++.so.6.0.34
[...]
But that code is a bit of a hack; it shows things that are not files (e.g. "[heap]") and will not work correctly if there are spaces in the filename, for example. In general, it seems somewhat circuitous for the kernel to have to turn its internal data structures into textual output that the Python code then needs to parse in some fashion to extract what it wants.
It turns out that some high-performance tracing programs for Linux needed a
way to get that information directly, so a new interface was added
for the 6.11 kernel; it uses a new PROCMAP_QUERY ioctl()
command on a file descriptor for the /proc/PID/maps file. It
provides mechanisms to filter the output in various ways and get the
information in a binary format. "So the motivation was not Python
users, but that doesn't mean we can't use it from Python.
"
In a bit of an aside, he noted that C is sort of the "lingua franca
"
of computing, but that Linux (and other operating systems) could have been
written in other languages. At PyCon 2018 in Cleveland, he and Alex Gaynor
wondered how hard it would be to get Rust code working with the Linux
kernel, so they began that effort at the sprints after the conference.
That is where Rust in the kernel got its start. He strongly recommended
attending the sprints, even for non-Python projects like the one they
worked on.
Complications
The new PROCMAP_QUERY is a "much more complicated
interface
" than his other example; it starts with the struct
procmap_query that is used with the new ioctl() command. He
showed the lengthy structure on a slide without all of its comments so that
it fit; he noted that C structures are akin to Python data classes.
One of the reasons C is used for system-level programs, though, is that its data types are simple,
Thomas said. It does not have classes, dictionaries, inheritance, and so on
like Python does. "C has very few data types
", just numbers, integer
and floating point, pointers (addresses, which are ultimately numbers),
structures, and not much more. Its arrays are fixed length and its strings
are simply arrays of bytes. For the most part, C completely specifies the
format of its data, down to the level of bits and bytes; a __u32,
for example,
is a 32-bit unsigned integer, and it will take up exactly four
bytes in the structure.
Turning that into a data class is fairly straightforward, but all of
the types will simply be int, because Python does not care about
the number of bits in a value of that sort. There will need to be a way to
convert between the two representations, but the difference makes it clear
that the data-class representation "doesn't tell you anything about what
the bits and bytes are
".
@dataclasses.dataclass
class ProcmapQuery:
size: int
query_flags: int
query_addr: int
vma_start: int
vma_end: int
vma_flags: int
[...]
The Python representation is great for doing calculations and not having to worry about the number of bytes involved, but it does not work so well when you need to store the data in an interoperable way. That's where C's rigid specification helps. The Python struct module can convert between Python types and the bits and bytes needed by C. So, using struct, he changed the ProcmapQuery class to automatically generate the size field:
@dataclasses.dataclass
class ProcmapQuery:
_STRUCT = struct.Struct("@9L4I2L")
size: int = _STRUCT.size
[...]
The format string ("@9L4I2L") specifies nine 64-bit values, then four 32-bit values, ending with two 64-bit fields, all of which are encoded in the native byte order ("@"). The size field is used to allow the kernel to extend the API by adding more entries to the structure, thus increasing its size.
The structure for the query consists of both parameters that are passed into the call, others that are returned from the call, and two that have both roles. The idea is that the programmer fills in the input fields and the kernel responds by filling in the output fields. Given that he wants to extract the names of the files from the query, he focused on two pairs of fields in struct procmap_query:
__u32 vma_name_size; /* in/out */
__u32 build_id_size; /* in/out */
__u64 vma_name_addr; /* in */
__u64 build_id_addr; /* in */
Those fields highlight some big differences between C and Python.
In both cases, for the virtual memory area (VMA) name and the build ID (which he did not use), the kernel is being passed the address of a buffer in the *_addr field and the length of the buffer in *_size. The size fields are both input and output parameters because they tell the kernel how many bytes are available in the buffer and the kernel returns how many of those bytes it filled in. If the returned size is the same as the passed-in size, Thomas said, the code should probably retry the query with a larger buffer, but he planned to leave that as an enhancement that could be added.
The query_flags field uses values from the enum procmap_query_flags, most of which are uninteresting for getting a list of the file mappings. But two are of use: PROCMAP_QUERY_FILE_BACKED_VMA asks for VMAs that are backed by a file, while PROCMAP_QUERY_COVERING_OR_NEXT_VMA allows code to step through all of the VMAs in a process by asking for the next VMA starting at address zero.
He then showed a few member functions added to the ProcmapQuery class:
def pack(self) -> bytes:
return self._STRUCT.pack(*dataclasses.astuple(self))
@classmethod
def unpack(cls, packed: bytes) -> Self:
return cls(*cls._STRUCT.unpack(packed))
def ioctl(self, fd: int) -> Self:
return self.unpack(fcntl.ioctl(fd, PROCMAP_QUERY, self.pack()))
The first two simply convert the data class from its Python form to the
binary form expected by C—or the reverse. For pack(), which
converts to the C representation, the pack()
member function of the struct.Struct
class expects a tuple, which dataclasses.astuple()
creates. unpack() turns the C structure into a
ProcmapQuery object by using the @classmethod
decorator and the struct.Struct
unpack() method. The ioctl() function ties it all together by
packing up the data class, passing it to the ioctl() system call,
and unpacking the result.
PROCMAP_QUERY
One important thing that is needed, though, is a value for the PROCMAP_QUERY constant, which is more difficult to determine than F_CREATED_QUERY was, he said. It is a #define in fs.h to:
_IOWR(PROCFS_IOCTL_MAGIC, 17, struct procmap_query)
That definition creates a constant value that encodes the type of the file the command should operate on
along with other information that the ioctl() machinery uses to try to
fend off invalid calls. It is not at all
straightforward to work all of that out and turn it into an integer,
especially within the time constraints of his talk, so he simply took the
path of least resistance and wrote a little C program to print it out.
With the appropriate C header files, a simple printf() of
PROCMAP_QUERY gives the magic integer needed: 3228067345. That
number may be architecture-specific, he cautioned, so production code would
need a better mechanism to obtain it, possibly following the lead of his procmapquery-cffi example.
The last thing needed before this can all be put together into a program, Thomas said, is to figure out how to get a buffer to pass in. There are a number of different ways to get a buffer in Python, but some of them do not provide a mechanism to get its address, which is not useful for Python, generally, but is needed here. For this, he is using the ctypes module for its create_string_buffer() and addressof() functions:
>>> buf = ctypes.create_string_buffer(1024)
>>> ctypes.addressof(buf)
4806710272
With all of that in hand, he put up the 15 lines of Python code that
constitute the guts of his "what libraries are being used?" tool:
libraries = set()
query = ProcmapQuery(
query_flags=PROCMAP_QUERY_COVERING_OR_NEXT_VMA |
PROCMAP_QUERY_FILE_BACKED_VMA,
query_addr=0,
vma_name_addr=ctypes.addressof(buf),
vma_name_size=1024,
)
while True:
try:
response = query.ioctl(proc_maps)
except FileNotFoundError:
break
libraries.add(os.fsdecode(buf[:response.vma_name_size]))
query.query_addr = response.vma_end
It creates a set to hold the libraries, sets up the query object, and then
loops over the addresses, adding filenames from the response to the set
(using os.fsdecode()
for the filename), and then using the vma_end address returned as
the next address to query. The standard FileNotFoundError
exception is what fnctl.ioctl() raises when there are no more VMAs to
process, which he found by trial and error. The procmapquery
directory in his repository contains the code, which has a get_libraries() function
to return the set of libraries for a file descriptor or that can be run
from the command line to print the libraries from /proc/self/maps.
Some wrap-up
There were two web sites that he recommended using to find out about new
kernel features. As readers presumably already know, LWN.net is a great
place to discover new kernel features in the merge-window summaries,
feature articles, and so on. While Thomas is "a happy subscriber
",
he noted that the paywall only lasts a week or so; all of the content he
mentioned in the talk was freely available to anyone.
Another site he pointed to was KernelNewbies, which has "really
detailed changelogs of just about everything that goes into the kernel
"
(example). That summary
will have links to articles and blog posts about those features, as well as
links to the commits.
He also had suggestions on some areas to explore for those who are
interested in digging in further, including investigating the
ctypes library he used and the C Foreign Function
Interface (CFFI) module that he mentioned for retrieving the
PROCMAP_QUERY constant. Both of those are useful for connecting
Python to C programs, including the kernel. Cython is a Python-like language that
compiles to C, which is another avenue for exploration. Writing Python
extensions in C will give even more power to Python programs; various
modules that he used, such as fcntl and os were written
that way. "If you need that much control and you don't want to
write C
", he suggested investigating PyO3, which allows writing Python
extensions in Rust.
In the brief Q&A, an audience member wondered about how programmers could be sure that the Python version of the C procmap_query structure was correct. Thomas said that could definitely be a problem, which might be hard to track down. His longer example using the CFFI module would actually properly ensure that, but it is more involved to use; CFFI is a third-party module and requires a C compiler to use. If he was working on a production version of the query, though, he said he would likely base it on the CFFI version.
[Thanks to the Linux Foundation for its travel sponsorship that allowed me
to travel to Pittsburgh for PyCon US.]
| Index entries for this article | |
|---|---|
| Conference | PyCon/2025 |
| Python | Linux kernel |