Leading items

Welcome to the LWN.net Weekly Edition for July 29, 2021

This edition contains the following feature content:

• Python gets a "Developer-in-Residence": a supported position will help the Python project catch up with its backlog.
• The core of the -stable debate: two views of what the stable kernel updates should be.
• Using DAMON for proactive reclaim: a mechanism for proactively freeing memory under user-space control.
• Hastening process cleanup with process_mrelease(): a proposed new system call to speed the freeing of memory when a process is killed.
• A look forward to Linux Plumbers 2021: what's coming at this year's LPC event.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Python gets a "Developer-in-Residence"

Backlogs in bug triage, code review, and other elements of the development process are nothing new for free-software projects; there is clearly a lot more interest in creating new features (and the bugs that go with them, of course) than in taking on the less-satisfying bits. For a large project like CPython, though, the backlog can seriously impede progress—potentially chasing off contributors whose work falls through the cracks. In order to address that, the Python Software Foundation (PSF) has raised some funds to hire Łukasz Langa as the CPython "Developer-in-Residence". Langa will be working to help clear the backlog, while also looking into other areas of interest to the PSF and the Python steering council.

Langa is a longtime CPython core developer and the release manager for Python 3.8 and 3.9; he is also the creator of the Black code formatter for Python. But, beyond all of that, he has been advocating for more full-time Python developers for a while now, so this is something of a dream come true for him personally. He described the goals for the position in a July 12 blog post, which was his first day on the job:

When the PSF first announced the Developer in Residence position, I was immediately incredibly hopeful for Python. I think it's a role with transformational potential for the project. In short, I believe the mission of the Developer in Residence (DIR) is to accelerate the developer experience of everybody else. This includes not only the core development team, but most importantly the drive-by contributors submitting pull requests and creating issues on the tracker.

He noted that Python is largely volunteer-driven, though there are a few folks being paid by their employer to work on the language full-time; for example, there is a team of three people (including Python creator Guido van Rossum) who are working on speeding up CPython. Full-time people make a difference: "Just by the sheer force of sitting at a desk for a given number of hours they can achieve big things." But the role of the DIR is somewhat different; he sees it as a way to bring in even more contributors to the project:

Now, what can the DIR do as one person? I believe I can multiply the impact of the hundreds of contributors who are not core developers. The DIR can do it by:

• providing a steady review stream which helps dealing with PR backlog;
• triaging issues on the tracker dealing with issue backlog;
• being present in official communication channels to unblock people with questions;
• keeping CI and the test suite in usable state which further helps contributors focus on their changes at hand;
• keeping tabs on where the most work is needed and what parts of the project are most important.

There is an important side effect to providing this service, and that is a good first developer experience for an external contributor. Great contributor experiences lead to future contributions, and a stream of contributions leads to an occasional contributor becoming a core developer. I've seen this through leading the Black project. It's now maintained by 9 people including me.

The DIR position is patterned after a successful program for the Python-based Django web framework; since 2014, the Django fellowship program has been paying people to handle "some of the administrative and community management tasks" for the project. One aspect of that program that Langa has adopted is its weekly progress reports; transparency is one of the key elements of the job and in measuring its impact. He noted that metrics like the number of pull requests and bug reports handled are useful, but there is more that he (and the community) would like to know:

It would be awesome to have some insight into whether having a DIR improves the health of the Python community, has any concrete impact over Python adoption, or Python runtime's performance, and so on, and so on. How to measure this is admittedly unclear to me at this point.

He will be undertaking some larger-scale research to try to help determine what kinds of work the project's volunteers are doing and how the CPython project is functioning overall. For example, the Python standard library is often under discussion within the project, but there are unresolved questions about it that he will be seeking answers on:

Research to understand the project better can be done by capturing user interest in particular libraries within the standard library, the amount of work given libraries require, and who the active experts behind the libraries are. This can be done through open issue analysis, the amount of pull requests for a given library, but also through surveying core developers and users as some libraries might require little maintenance but be critical. The reason to do this research is to determine which standard library modules need help and what the maintainer cost is for standard library modules.

So far, Langa has posted two admirably detailed status reports for the first two weeks. Each includes a daily log of his activities, which show an eye-opening amount of progress with closing pull requests and bug reports. In addition, there is quite a bit of other information about the things that he looked into. For example, he observed a test failure on Windows in week one:

Since Azure Pipelines just released an update to their Windows image and I identified that the problem only happens with the new image, I was ready to report a regression on their GitHub project. But I noticed that somehow the problem only happens to me.

I joked that this is probably because of my first name… and it turned out to be true. Maybe I shouldn't be surprised. I have a long history of Unicode-related issues affecting me in the real world.

Long story short: the new Azure Pipelines image includes the full name of the PR author in an environment variable. The HTTP_ACCEPT tests tried to serialize all of os.environ to an ASCII string in the CGI script. This crashed the CGI script but since the tested server is fork-based, this crash was only seen on the test side as truncated output.

Knowing this, it was easy to fix. But this [easily] took 10 hours of work for me this week.

That same kind of problem manifested itself (more benignly) during his 2018 Python Language Summit talk: "[...] he noticed his name in Larry Hastings's schedule-display application started with the dreaded ▯ rather than Ł. That, he said with a grin, is the story of his life."

Meanwhile, in week two, he tackled some problems with tests that intermittently fail in the continuous integration (CI) system. While there are efforts to reduce the number of such tests, or to reduce their volatility, tests for areas like networking or that use timeouts will likely always be somewhat problematic; he has improved things substantially:

Up until now, re-runs triggered re-running all tests in the affected test file. As you can imagine, the test files for the most flaky tests tend to also be the largest. Now with a change I made (see BPO-44708 for details), we are only re-running affected test methods which speeds up the occasional re-run dramatically.

[...] As you can see, re-running concurrent_futures tests now takes 2.5s instead of over 82s.

Interestingly, this not only increases speed of the re-runs but also makes it more likely that they will succeed. You see, the fewer tests we need to re-run, the less likely we are to hit another intermittent failure.

One guesses that these reports may become more terse over time, but they definitely reflect a high level of excitement with his new role. Langa is also cognizant of his responsibility both to CPython and to the DIR program. He strongly believes in the need for this position—it could perhaps lead to other paid positions within the Python ecosystem for one thing—so he intends to leave a lasting impression of its value:

On the other hand, this year will prove whether this position is worth keeping for future years. I deeply believe it is, therefore it all comes down to my performance over the next twelve months.

A recent thread on the python-dev mailing list highlights the impact the DIR position could have. In the thread, a Python user pointed to a bug that was filed in 2017, had a pull request associated with it, and still remained unfixed, They wondered if that was evidence that the Python review process is flawed. As Van Rossum replied, Python is no exception to the rule that "*maintainer attention* is actually the scarcest resource in many open source projects". In addition, while it may look like a bug fix "just" requires review, often that is actually the most difficult part:

I think there's a misunderstanding here -- you seem to imply that producing a bugfix is work that takes somebody's time, while reviewing a bugfix is not work and doesn't cost anything. But realistically, for most issues, things are the other way around -- writing the code is easy (at least to a core dev :-) but reviewing code is a gut-wrenching process that takes up emotional energy and a lot of time. Given the discussion in the issue corresponding to the PR it is clear that that is what is going on here.

As Barry Warsaw pointed out, unblocking this kind of problem "is exactly what the Developer-in-Residence program was designed to address". That specific problem was resolved by Van Rossum before the DIR came on board, but one might guess that these kinds of problems will be reduced over the next year.

There are other examples of people being hired by foundations to work on open-source projects, and some companies are supporting those contributors, both full- and part-time, as well. The Linux Foundation fellows are a prominent example; they often take on duties that might not be done by volunteers or those solely focused on their company's development priorities.

Hiring a Developer-in-Residence is a big step for the PSF, but one that seems likely to pay outsized dividends. Bringing in more contributors, which will hopefully lead to more core developers, is important for any open-source project—Python is no exception there either. Providing a smoother path for those contributors in the early going, and trying to ensure that their efforts do not languish, will help a lot in that effort. Finding a way to hire for this kind of role is a step that more projects should consider—many probably will.

Comments (29 posted)

The core of the -stable debate

Disagreements over which patches should find their way into stable updates are not new — or uncommon. So when the topic came up again recently, there was little reason to expect anything but more of the same. And, for the most part, that is what ensued but, in this exchange, we were also able to see the core issue that drives these discussions. There are, in the end, two fundamentally different views of what the stable tree should be.

The 5.13.2 stable update was not a small one; it contained an even 800 patches. That is 5% of the total size of the mainline 5.13 development cycle, which was not small either. With the other stable kernels going out for consideration on the same day, there were over 2,000 stable-bound patches in need of review; that is a somewhat tall order for even a large community to handle in the two days that are allowed. Even so, Hugh Dickins was able to raise an objection over the inclusion of several memory-management patches that had not been specifically marked for inclusion in the stable releases. Those patches, he thought, were not suitable for a stable kernel and should not have been selected.

Stable-kernel maintainer Greg Kroah-Hartman responded that the size of the update was due to maintainers holding onto fixes until the merge window opens. Once the -rc1 release comes out, those fixes all land in the stable updates, which are, as a result, huge. But it is clear that the amount of change going into the stable kernels has been growing over time. If one looks at the number of changes going into the first five updates for each release (enough updates to include the merge-window fixes), the result is:

Release	Updates	Changes
		First 5	Total
4.19	198	754	19,682
5.0	21	408	2,387
5.1	18	353	1,747
5.2	21	779	2,429
5.3	18	561	2,178
5.4	134	435	14,414
5.5	19	653	2,516
5.6	19	364	1,864
5.7	19	581	2,984
5.8	18	899	2,755
5.9	16	1,244	2,339
5.10	52	841	7,295
5.11	22	948	3,588
5.12	19	1,446	3,843
5.13	4	1,416	1,416

While 5.13 has not yet reached five stable releases as of this writing, it seems safe to predict that it will collect more changes in its first five stable updates than any of its predecessors. The number of patches going into the stable updates is increasing in general, at a rate that would seem to exceed the growth in the rate at which changes are applied during mainline kernel development cycles. Long-term stable kernels now receive more patches during their "stable" period than during the development cycle leading up to their "final" release. In other words, the development cycle is not even close to being finished when Linus Torvalds applies a tag and leaves the building.

Developers and maintainers can indicate that a mainline patch should be backported to the stable releases by including a "CC: stable@vger.kernel.org" tag, but that is not how most patches get there; of the 1,416 commits in 5.13.4, only 259 — 18% — contained such a tag. The stable maintainers have increasingly aggressively sought out mainline patches that look like fixes and put them into the stable releases. The Fixes tag found on many patches is used as a cue that a patch is a fix, but machine learning is also being used to select patches. The result is a lot of commits going into stable updates without having ever been explicitly marked for that treatment.

This work has always been controversial, especially when regressions slip through. Regressions are inevitable regardless, but it is hard to imagine a way to add over 3,000 changes to a kernel during a three-month short-term stable cycle without a few of them being bad. The patches singled out by Dickins this time around are not responsible for any regressions (that anybody knows about yet), but the memory-management developers are clearly worried about the possibility.

To avoid any such outcome, memory-management maintainer Andrew Morton has requested that patches carrying his Signed-off-by tag not be included in stable updates in the absence of a specific request: "At present this -stable promiscuity is overriding the (sometime carefully) considered decisions of the MM developers, and that's a bit scary". Kroah-Hartman asked how the decision to mark a patch for stable backporting is made and, specifically, why a number of clear fixes were not selected; Morton explained the thinking this way:

Broadly speaking: end-user impact. If we don't have reports of the issue causing a user-visible problem and we don't expect such things to occur, don't backport. Why risk causing some regression when we cannot identify any benefit?

Kroah-Hartman's position, instead, is that, if a patch fixes a bug, it should be included in the stable updates:

But it really feels odd that you all take the time to add a "Hey, this fixes this specific commit!" tag in the changelog, yet you do not actually want to go and fix the kernels that have that commit in it. This is an odd signal to others that watch the changelogs for context clues.

Sasha Levin (who also works on the stable updates) added that a lot of important fixes are missed if only the explicitly tagged patches are backported into the stable kernels. Some of those fixes then find their way into distributor kernels via other paths, which doesn't seem ideal.

In the end, this is what the disagreement comes down to: a difference of opinion on what is the best way to create stable updates that are truly stable and free of problems.

• Many developers see the stable updates as a carefully curated collection of hand-selected fixes, each of which has been extensively reviewed for importance and the lack of regression potential. These kernels should be safe to update to, since they should have a minimal chance of introducing problems not seen in their predecessors. This position tends to be taken by the developers of complex, core subsystems that have a high potential for subtle regressions.

  The memory-management subsystem is a classic example; there was also a similar discussion with the XFS filesystem developers in late 2020. Memory management requires predicting the future; as a result, the code is a large collection of complicated heuristics that have to work with a huge variety of workloads. It is not uncommon for an innocent-seeming change to create a performance regression for some private customer workload that won't surface until years have passed. Memory-management developers have learned that their lives run much more smoothly in the absence of such regressions, so they go out of their way to avoid making unnecessary changes to stable kernels.

• Others, including the stable maintainers, feel that the best kernel is to be had by including every fix that can be reasonably backported. Many bugs have user impacts — including security problems — that are not obvious to the developers when those bugs are being fixed; including all of the fixes will head off a lot of problems before they are discovered.

  Some distributors have taken this position and, as a result, are happy with how things are working; Justin Forbes wrote that "the current stable process has fixed more bugs than it has introduced". The Android kernel is increasingly tied to the stable updates as a way of getting as many fixes as possible; based on this experience, Kroah-Hartman said: "I have numbers to back up the other side, along with the security research showing that to ignore these stable releases puts systems at documented risk".

Which position is "correct" is not entirely clear, but there is no doubt about which position is "winning". As always in the Linux world, the people who are doing the work will decide how the work is to be done, and the stable maintainers have opted for the "promiscuous" approach. It would be interesting to see if there would be a user community for ultra-stable kernels maintained using the more restrictive approach, but it is doubtful that anybody has the time to create such a thing.

There is always room for tweaking around the edges and opting out certain subsystems; this seems likely to happen with regard to memory management. More testing would also certainly help; the testing picture for stable releases has improved considerably over the years but could still get better. Ted Ts'o suggested that there could be a role for a "perfbot" system that looked for performance regressions in particular, if the resources could be found to create that sort of facility. Performance regressions are particularly difficult to test for, though; the resource requirements are large and it is nearly impossible to simulate every type of workload.

In any case it seems that large stable updates will continue to be the rule going forward. With luck they will continue to become less regression-prone, but they will never be completely regression-free. So it is safe to predict that the debates over what should and should not go into stable updates will continue indefinitely.

Comments (69 posted)

Using DAMON for proactive reclaim

The DAMON patch set was first covered here in early 2020; this work, now in its 34th revision, enables the efficient collection of information about memory-usage patterns on Linux systems. That data can then be used to influence the kernel's memory-management subsystem; one possible way to do that is to more aggressively reclaim memory that is not being used. To that end, DAMON author SeongJae Park is proposing a DAMON-based mechanism to perform user-controllable proactive reclaim.

The core idea of DAMON is to use a sampling technique to determine which memory is in active use and which is sitting idle. A process's virtual address space is broken down into regions which vary in size depending on activity; the busiest regions are then subdivided over time for more fine-grained monitoring. Within each region, a randomly selected page is watched for activity, with the results being considered representative of the whole region. On demand, DAMON will produce a report in the form of a histogram informing the reader of how busy each memory region is.

Putting DAMON to work

From the beginning, the motivation for this work has been to use DAMON data to improve system performance by tweaking the operation of the memory-management subsystem. The initial example is the DAMON operation schemes (DAMOS) patch set, which adds a mechanism that can make various madvise() calls in response to usage patterns. Thus, for example, a request could be made to page out little-used memory, or to use huge pages for large, active regions.

The proactive-reclaim patches build on DAMOS to take a more active role in memory management. This work adds a new kernel module that will monitor data out of DAMON, find any regions of memory that have not been accessed within a given time period, and actively reclaim the pages within those regions for other uses, even if the system is not feeling memory pressure at the moment. The results, seemingly, can be impressive, with significant improvements to the overall performance of the system in the form of fewer latency spikes and less time spent in direct reclaim. Getting there requires a number of changes, though.

The first step is to add a "pageout" scheme to DAMOS, allowing the direct invocation of reclaim operations. This differs from the previously implemented madvise() mechanism in that it operates on physical pages, while madvise() works within the virtual address space of a single process. As suggested by its name, madvise() is advisory, while the new pageout scheme simply makes the reclaim happen. It thus will be far more effective at actually pushing pages out of memory.

Making proactive reclaim efficient

It's worth noting that proactive reclaim is not a new idea; it was discussed at the 2019 Linux Storage, Filesystem, and Memory-Management Summit, for example. At that time, the developers present agreed that there could be value in proactively reclaiming memory that appears to be unused; that is, after all, what the memory-management subsystem is meant to be doing in general. The problem is the cost; monitoring the system closely enough to figure out which pages are truly idle requires repeatedly scanning all of memory, which is expensive. That expense will, in many (if not most) cases, outweigh the benefits of accurate proactive reclamation of pages, so this work has not made its way into the mainline kernel.

The whole idea behind DAMON is to make this kind of monitoring cheap; Park described it in the proactive-reclaim cover letter this way:

Its adaptive monitoring overhead control feature minimizes its monitoring overhead. It also let the upper-bound of the overhead be configurable by clients, regardless of the size of the monitoring target memory. While monitoring 70 GB memory of a production system every 5 milliseconds, it consumes less than 1% single CPU time.

The granularity of the monitoring is configurable to get the overhead within whatever bounds the user finds acceptable; lower overhead, of course, leads to less accurate results. The efficiency of DAMON, it is hoped, can help to overcome the objections to the cost of proactive reclaim, but that is not a full solution to the problem; the reclaim operations themselves have a cost, and that cost cannot be allowed to run out of control.

To avoid that problem, the proactive-reclaim patch set adds a number of knobs allowing the configuration of the new mechanism. One knob simply controls the amount of memory (expressed in bytes) that can be operated on in each (configurable) unit of time. Once that quota has been hit, proactive reclaim will take a proactive break until the next time period begins. To avoid getting stuck in a single memory region, proactive reclaim will move on to a new region in the next time slot if it runs into the limit.

Beyond that, there is a knob setting a time quota, being the amount of CPU time that proactive reclaim is allowed to use per time period. Once again, when that time is exceeded, reclaim will pause until the next time slot opens up. Yet another knob allows the entire mechanism to be paused if the amount of free memory in the system exceeds a given watermark. There is, after all, little value to be had in using system resources to free memory when there is already more of it than is needed. Interestingly, proactive reclaim can also be paused if the amount of free memory is too low; in such situations, the memory-management subsystem will already be actively reclaiming memory and it is likely better for the proactive mechanism to just stay out of the way.

Finally, there is a prioritization mechanism. Depending on the settings of the quota parameters, proactive reclaim may well be unable to work through all of memory in any reasonable time, so it makes sense to focus that work on the regions where the most benefit will be gained. Users can tweak yet another set of knobs to direct reclaim to specific regions based on their size, age, and access frequencies.

Not for everybody

All of this work leads to the creation of a new mechanism with a lot of parameters to tweak and, probably, some real potential for harming the performance of the system rather than helping it. This documentation patch from the series describes the operation of this new module in detail. It seems unlikely to be a feature for casual users to play with.

For administrators who need to get the best performance out of their systems and who have the time to tune it, though, proactive reclaim may offer some significant benefits. Benchmark results posted with the patch set show that an unconstrained reclaim module can free up nearly half of the available memory, but at a cost: nearly 12% of one CPU dedicated to the reclaim work, and a workload slowdown of just over 5% (presumably resulting from increased paging when reclaimed memory turns out to be needed after all). Applying some quotas can cut that overhead considerably while still gaining most of the free-memory benefit, which can be used to run other workloads. It seems likely, though, that the optimal settings will depend heavily on the nature of the workload.

The proactive-reclaim work is in its third round on the mailing lists, but DAMON itself has now been posted 34 times, with no clear indication of when (or even if) it will be merged into the mainline. Park would clearly like to see some progress on that front; the cover letter begins with a request for reviews of the DAMON work. Those reviews — and an indication of wider interest in this work — will clearly be necessary if DAMON and the mechanisms built on it are to find their way into the mainline.

Comments (2 posted)

Hastening process cleanup with process_mrelease()

One of the fundamental invariants of computing is that, regardless of how much memory is installed in a system, it is never enough. This is especially true of systems with tight performance constraints, where every page of memory is allocated and in use, making it difficult to find more when it is badly needed. One way to make more memory available is to kill one or more processes, freeing their resources for other users. But that often does not work as quickly or reliably as users would like. In an attempt to improve the situation, Suren Baghdasaryan has proposed the addition of a system call named process_mrelease().

Systems running mixed workloads, where some tasks are more important than others, are not uncommon. If the system is being run near its maximum capacity, the relatively unimportant tasks may end up using memory that is needed by the more important work, at which point it might be better if the unimportant processes went away. Such systems often run process managers that will kill off the low-priority processes in these situations; perhaps the most widespread example of this pattern is Android, which will kill background apps if the available memory is insufficient for whatever is running in the foreground. Cloud-computing systems will also kill low-priority, best-effort workloads if their memory is needed by more important work.

Killing a process should, in principle, make its memory immediately available for other users. In the real world, though, things are not so simple. The killed process is, itself, responsible for cleaning up and freeing its resources, a task that is carried out in kernel context. If, however, the killed process finds itself blocked in an uninterruptible sleep, that cleanup work could be delayed indefinitely. There are other factors that can slow down the freeing of memory, including how busy the relevant CPU is and whether that CPU is running in a slow, low-power state.

When this happens, the system has paid the cost of killing the process (which was presumably doing something useful) without receiving the benefits from that action. Unfortunately, those benefits tend to be needed urgently; the system would not be killing processes otherwise. Delays in process cleanup can have immediate and visible effects on the higher-priority workloads; these can include jerky response on a handset or a delay in the delivery of a cat video to an impatient viewer.

This problem was encountered years ago in the context of the system's out-of-memory (OOM) killer, which is the kernel's last-resort response when memory runs out. Back in 2015, the development of the OOM reaper addressed this problem by taking the memory cleanup work out of the dying process's hands and making it the responsibility of a separate kernel thread. That made OOM killing significantly more robust, with the ability to free memory quickly even if the chosen process is not able to exit immediately.

That work did not address one other unfortunate characteristic of the OOM killer, though: its opinion of what is the least important process on the system tends to differ from that of the system's users. Invoking the OOM killer may allow the system as a whole to continue functioning, but the user whose window-system server was just killed may be forgiven for not being fully enthusiastic in their celebration of that feat.

For this reason, systems developers have tended to take the business of killing processes to rob them of their memory into their own hands. An out-of-memory handler running in user space can take more proactive steps to prevent the system from going into the OOM state to begin with, and it probably has a better idea of which processes will cause the least pain should they encounter an untimely demise. The oomd daemon released by Facebook is one example of this kind of utility; there are others as well.

User-space OOM killers, though, are not in the same position as the kernel's OOM killer; they must rely on the kill() system call (or, more recently, pidfd_send_signal()) to implement the sharp end of their memory-freeing decisions. Killing a process that way does not bring the OOM reaper into play, so user-space daemons are back in the situation of having to wait for the targeted processes to release their own resources.

Baghdasaryan's answer to this problem is a new system call:

    int process_mrelease(int pidfd, unsigned int flags);

The pidfd argument is a pidfd identifying the process of interest; that process must be exiting (presumably as the result of a previous kill() operation) when the call is made. The flags argument must be zero for now. This call will have the same effect as setting the OOM reaper on the indicated process, stripping away as much of its memory as possible.

One of the reasons behind the creation of a separate call for this work is to give the system a context in which to do it. The task of going through the process's address space and freeing up all that memory will be done by the process that calls process_mrelease(), which may or may not be the process that killed the target in the first place. The kernel can then do this work with the priority of the calling process, and with its CPU assignments, allowing the cleanup work to be contained where it will not interfere with the (remaining) system workload.

An alternative that was discussed with an earlier attempt to solve this problem was to just unconditionally reap the memory of a process when it is killed, without requiring a separate system call to make that happen. In that case, though, the work would be done in the context of the process sending the signal, which might not be welcome. A process that kills a lot of other ones — a killall command, for example — could be significantly slowed if that policy were to be adopted. Adding a separate system call gives user space more control over when and how that work is done.

In the previous posting of this work, the main topic of discussion was the name of the system call itself — process_reap() at that time. That is a reasonably clear sign that the more significant issues have been addressed and that the work may be about ready to move forward. The number of callers of process_mrelease() is likely to be small, but it seems there will be some situations where it will be a useful tool to have.

Comments (7 posted)

A look forward to Linux Plumbers 2021

The annual Linux Plumbers Conference (LPC) is a gathering of a relatively small subset of the developers working on the low-level (plumbing) details of Linux systems. It covers topics from below the kernel through the user-space components that underlie the interfaces and applications that most Linux users interact with. This year's event will be held virtually September 20‑24; it is shaping up to be another great edition of one of the premier open-registration Linux technical conferences on the calendar.

LPC is known for its "microconferences" that typically feature face-to-face discussions, planning, and development work in a wide variety of topic areas. As with most of the rest of the conferences over the last year or so, it was turned into a virtual event last year; that continues this year. With any luck, the COVID-19 pandemic will subside enough to resume more in-person events—cautiously—in 2022.

In the meantime, the BigBlueButton-based videoconferencing system that was used in 2020 will return, but there are plans to use Matrix for the text chat piece. A request for quotes (RFQ) for some of the work to make that happen was posted in March; the LPC committee was looking to fund projects to improve BigBlueButton and to integrate Matrix with it. Unfortunately, all of the companies that might have been able to do so were already fully occupied by their existing customers, who had increased needs for videoconferencing due to the pandemic, LPC committee member Guy Lunardi said in an email exchange. So committee members will be working on parts of those plans as time allows, he said.

While microconferences are a big piece of LPC (and, in some sense, its claim to fame), there is more to the conference. There are several other tracks, some of which are similar in format to microconferences, though they all generally last longer than the usual half-day allotted to microconferences. The refereed presentation track contains more traditional 45-minute talks on relevant topics. The web page for the various CFPs for LPC describes those topics as follows:

Examples of Linux plumbing include core kernel subsystems, core libraries, windowing systems, management tools, device support, container run-times, media creation/playback, and so on. The best presentations are not about finished work, but rather problems, proposals, or proof-of-concept solutions that require face-to-face discussions and debate.

At this point, the CFP for refereed talks is closed, but the accepted presentations have not yet been announced. A look at the schedule from LPC 2020 should give an idea of how the refereed track will be scheduled in parallel with the rest of LPC.

Another track that will be "co-located" with LPC is the traditional Kernel summit, which is discussing its topics on the ksummit-discuss mailing list. It will have 45-minute slots "to discuss specific technical issues that would be easier to resolve in person than over e-mail"; as usual, roughly half of the slots will be reserved for topics that crop up close to the time of the summit.

Like the Kernel summit, both the Networking and BPF track and the GNU Tools track will be spread over multiple days. Each will have its own format that is more suited to the participants for those topic areas. Networking/BPF will consist of 40-minute presentations, including questions and discussion, optionally accompanied by a short paper, on advanced Linux networking and BPF topics. The GNU Tools track is targeting developers "working in the GNU tool chain: GCC, GDB, binutils, runtimes, etc."; it will have shorter sessions to try to accommodate as many as possible:

The purpose of this virtual meeting is to gather all GNU tools developers, discuss current/future work, coordinate efforts, exchange reports on ongoing efforts, discuss development plans for the next 12 months, developer tutorials and any other related discussions.

Microconferences

The CFP for microconferences is closed at this point; 13 have been accepted at the time of this writing. According to LPC microconference chair Steven Rostedt, there are another six in-progress that are likely to be accepted and announced soon. Once a microconference has been accepted, it goes into its own CFP process to solicit topics, though sessions are not supposed to be traditional presentations. Microconference sessions may have a few slides (the guideline is three or less) to lay the groundwork for a discussion, but the focus should be on working together to find and then develop a solution to a particular problem. The CFP page puts it this way:

A microconference is a collection of collaborative sessions focused on problems in a particular area of the Linux plumbing, which includes the kernel, libraries, utilities, UI, and so forth, but can also focus on cross-cutting concerns such as security, scaling, energy efficiency, or a particular use case. Good microconferences result in solutions to these problems and concerns, while the best microconferences result in patches that implement those solutions.

Many of the microconferences have appeared in multiple LPCs, but there are a few new ones that have already been accepted this year. The Confidential Computing microconference is a brand new entry; it is looking at various technologies for memory encryption that can protect the data in virtual machines (VMs) from other guest VMs and even from the host system itself. The microconference, which is being led by Joerg Roedel, will discuss those processor features and more:

In this microconference we will discuss how Linux can support encryption technologies which protect data during processing on the CPU. Examples are AMD SEV, Intel TDX, IBM Secure Execution for s390x and ARM Secure Virtualization. These are recent additions compared to technologies which protect data while in transit (SSL, VPNs) and at rest (disk encryption).

The Linux kernel recently gained support for SEV-ES and support for Intel TDX is upcoming. AMD SEV will be further enhanced by Secure Nested Paging (SNP). Support for these technologies requires intrusive changes to the Linux kernel for memory integrity and secure interrupt delivery to virtual machines. Designing these changes in a way that works for different confidential computing technologies is one goal of this microconference.

While there were previous editions, it has been a number of years since a File System microconference has been a part of LPC; it returns this year. There are four leads for it, Josef Bacik, Amir Goldstein, Jan Kara, and Ted Ts'o, and they are targeting sessions in a number of areas: "enabling new features in the file system ecosystem as a whole, interface improvements, interesting work being done, really anything related to file systems and their use in general". A number of specific topic ideas were suggested: whether DAX is ready for prime time, page folios, writeback throttling, performance testing, and more.

After taking a few years off, the Performance and Scalability microconference has returned to LPC; it is being led by Davidlohr Bueso, Daniel Jordan, and Pavel Tatashin. As the name would imply, the focus is on performance, particularly in the context of getting the most from multiple processors in the system. The announcement looks at some of the results since the last edition of the microconference in 2018, along with ideas of what might be discussed this time around.

The Toolchains and Kernel microconference is related to the Toolchains microconference from last year, but is more focused on the integration of LLVM, Clang, GCC, and friends with the kernel. It is being led by Nick Desaulniers and Jose E. Marchesi. One major area that will be discussed is the ongoing effort to bring Rust to the kernel, but there are plenty of other topics as well, including support and integration for DWARF, BPF type format (BTF), and compact type format (CTF) in the toolchains, security feature requests, locking annotations in Clang, post-link optimization, and more.

Some of the microconferences that have yet to be announced are also new to LPC. The Diversity, Equity & Inclusion microconference is aimed at discussions on ways to improve the mix of people involved in kernel development. The intersection of new hardware accelerators for GPUs, media, and neural-network processing presents a number of challenges to DRM and other subsystems; those issues will be the subject of the GPU/media/AI Buffer microconference. More information on both of those will be forthcoming when they are announced, as with the other returning microconferences that are pending: Android, System Boot and Security, Apps Ecosystem, and Kernel Dependability & Assurance.

There are, of course, lots of other microconferences that are returning for repeat engagements, including Containers and Checkpoint/Restore, Tracing, Real-time, Testing and Fuzzing, Open Printing, and more—nearly 20 in all. The award for the most confusingly named microconference clearly goes to the IOThree's Company microconference, which is not sponsored by a company, as one might be forgiven for surmising, but is, instead, highlighting the third incarnation of the IoT microconference by way of a reference to a (bad) 1970s TV sitcom.

There are lots of ways for folks to participate in LPC, even though some of the CFPs have closed. The microconferences and most of the tracks are still taking suggestions for topics, so if the theme of one strikes a chord, proposing a session to solve or at least discuss a difficult problem, especially one that involves multiple layers of the Linux system, might be in order. Registration is still open, as well; the virtual nature of the conference has removed much of the stress of the tightly limited attendance that existed for the in-person events. One advantage of videoconferences is a lessening of the logistical challenges that limited the number of attendees. With a registration fee of just $50, LPC is likely affordable to most as well.

Comments (6 posted)