Leading items

Welcome to the LWN.net Weekly Edition for March 26, 2020

This edition contains the following feature content:

• Helping FOSS conferences in the face of a pandemic: FOSS Responders forms to give aid to community projects suffering from the pandemic.
• Video conferencing with Jitsi: hosting video gatherings without proprietary software or servers.
• Working-set protection for anonymous pages: simple memory-management changes with big benefits.
• Automatic buffer selection for io_uring: yet another feature for this asynchronous I/O subsystem.
• Django changes its governance: the Django project moves away from the "core committers" model.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Helping FOSS conferences in the face of a pandemic

The effects of the Coronavirus disease 2019 (COVID-19) pandemic are horrific and far-reaching; we really do not yet know just how bad it will get. One far less serious area that has been affected is conferences for and about free and open-source software (FOSS). On the grand scale, these problems are pretty low on the priority list. There are a fair number of non-profit organizations behind the gatherings, however, that have spent considerable sums setting up now-canceled events or depend on the conferences for a big chunk of their budget—or both. A new organization, FOSS Responders, has formed to try to help out.

Over the last few weeks, FOSS Responders has picked up steam. The effort was "started as an organic movement led by Duane O’Brien (Indeed.com), Megan Byrd-Sanicki (Google), and others". The goal is to try to counteract some of the effects of the pandemic on FOSS communities, with a particular focus on events:

There are some resources listed on the web site, including organizations focused more widely on the problems that COVID-19 is bringing to our world (e.g. the World Health Organization response fund). There are also pointers to web pages covering virtual event organization and the mechanics of running an online event. The FOSS Responders web site makes it clear that many of the items posted there are simply meant as starting points; those with ideas are encouraged to get in touch with the project.

FOSS Responders has also set up a site for donations at Open Collective:

Open Collective provides a mechanism to collect money and fund various things in a transparent fashion. The income and expense line items are available so that contributors can see what has been collected and where it has gone. At the time of this writing, the FOSS Responders collective has received $5400 and has not spent any of that. The platform fees that Open Collective normally deducts for its handling of the funds have been donated to the project, so all of the contributions will be spent furthering the FOSS Responders mission.

The organization is helping maintain a calendar of canceled tech events (LWN is tracking some of those too). Those calendars make for some eye-opening reading. The FOSS Responders calendar lists events well outside of the usual FOSS fare, but it is truly amazing how many different tech events there are (or would have been) just in the next few months. One guesses that many organizers of conferences in the northern-hemisphere summer are crossing their fingers and hoping for the best; some in June and July have already been canceled or postponed at this point and it seems likely that more, perhaps many more, will follow.

A Discourse Instance was set up on March 24; unsurprisingly, so far posts concern organizational and administrative issues. The project has a GitHub repository, for the web site and as a place for organizations to request support, as well as a Slack channel; email can be sent to foss-responders@googlegroups.com. There is also a weekly meeting for anyone interested on Thursdays at 1700 UTC; video and telephone options for attendees are both available. FOSS Responders is a brand new project so there is a need for more people to get involved:

The search for funders (and funds) will presumably get a boost from an upcoming virtual event:

The problem our communities are facing is real; without assistance there are organizations that may go completely under due to the financial upheaval from a canceled conference. Before PyCon US was canceled the Python Software Foundation (PSF), which puts the conference on, was quite candid about the problems it faced in continuing to support Python "through grants, sponsorships, and funding core infrastructure that the community relies on". The PSF was concerned that penalties and a lack of conference income could eat into its reserves, which might make it difficult to continue funding Python development at current levels. Many smaller projects and the organizations behind them may not even have a reserve to fall back on.

Some companies have been maintaining their sponsorship for conferences even if they are canceled, rescheduled, or recreated as virtual events. That will undoubtedly help cushion the blow, but even in the absence of penalties and such, the logistics of un-planning a conference are likely to be costly. The Linux Foundation (LF) has often stepped in to help out FOSS projects along the way; one would guess it is (or will be) helping out with FOSS Responders as well, but perhaps not in a financial sense. The LF has presumably had to absorb quite a chunk of unexpected expense from undoing its busy schedule of conferences in March and April, with more likely to come.

FOSS Responders is certainly a worthy effort and something that will help our communities continue as we get past COVID-19 over the next year or more. The main focus of philanthropic efforts will be elsewhere, as they should be, but it is nice to see our community finding ways to help itself out internally. One suspects the community as a whole is helping out in many other ways that go well beyond FOSS, conferences, computers, or any of that. Humanity is going to need a whole lot of healing as we find our way through this pandemic.

Comments (none posted)

Video conferencing with Jitsi

Spring is coming to the northern hemisphere, and one's thoughts naturally turn to ... being locked up inside the house and not allowed to go anywhere. That has, in turn, led to an increasing interest in alternative mechanisms for keeping up with family and coworkers, especially video conferencing. There are a number of proprietary video-conferencing services out there; your editor decided to look into what solutions exist in the free-software realm. It turns out that there are a few; the first to be looked at is Jitsi.

Jitsi is, in fact a collection of components, written mostly in Java (and JavaScript) and released under the Apache license. At the core is Jitsi Videobridge, which implements multi-participant video conferences, and Jitsi Meet, which implements the client side. Various other components live under the hood and are likely to only come to one's attention if something goes wrong with them. There is also a Jitsi Desktop application, but that has been superseded by the browser interface and is considered "legacy" at this point.

Getting it going

Brave folks can certainly install the whole thing from source; for the rest of us, pre-built packages are limited to the Debian and Ubuntu distributions. Your editor installed that version on a Debian 10 machine sitting out there in the cloud somewhere.

That installation did not go as easily as one might have liked. The (sparse) documentation suggests that the packages will perform the needed web-server setup, but a number of details were left as exercises for the reader. There is support for automating the setup of a Let's Encrypt certificate, but your editor did not try that. Once the server configuration was properly tweaked, it was possible to connect to the newly established bridge with an ordinary Firefox browser.

Running a single connection to a Jitsi server is a great way to admire one's own image on the screen, but it lacks something from the full conference experience. Unfortunately, when the second user connected, both users were immediately disconnected with a helpful "something went wrong" error, and an even more helpful "error 143" message on the server side. Some searching revealed this useful post stating that the Debian package installs the server in a misconfigured manner; once the suggested change was made, everything worked as expected.

The user experience

The Jitsi client runs fine in a standard web browser, with no need to install any plugins. In the default setup, a new user connecting to the server will see a screen inviting them to create a new meeting; typing any string into the field provided creates a "room" with that name (or joins a room with that name if it already exists). The curious can see this screen in action on the demo site set up to allow anybody to run a free conference.

After one joins a conference, the behavior is much like the proprietary services out there. It is possible to see a tiled view with all participants, or just the person who is speaking at any given time. There are buttons to mute audio and/or video, a separate area for text chat, and a button to "raise your hand" for attention. One cute feature is the ability to blur the background of one's outgoing video, though the result is somewhat ethereal.

Video and audio quality both seem to be quite good, though video suffers somewhat when there is a low-bandwidth connection involved. Jitsi claims that all data is encrypted between clients and the video bridge, though it goes through the bridge itself in the clear.

There is a screen-sharing option that can transmit the contents of a single window or the screen as a whole — though the latter leads to amusing effects if the conference itself is on-screen. The web site describes a "presenter mode" that mixes both the screen contents and the stream from the camera, but that doesn't appear to have made it into the stable builds yet.

The first person to join a room is designated the moderator, with some extra privileges. It's possible to mute participants, for example, or to kick them out entirely. There's a little window available that displays how much time each participant has spent speaking. It's also possible to set a password needed for entry into the room. It does not seem possible to delegate moderation to others, though.

Apps and more

If one connects to a Jitsi server from a mobile device, the response is a cheery message that "You need the Jitsi Meet mobile app to join this meeting on your phone". If, however, one selects the "desktop site" option helpfully provided by Chrome on Android, everything works just fine; there is no real need for the system to push the app that firmly. If this system is kept around for a while, your editor may just go in and find a way to disable that behavior.

That said, the app is not without its utility. It, too, is free software; it can be installed from F-Droid as well as from the Google Play Store. It provides most of the same features as the web version with a more finger-friendly layout. There is also an app for Apple devices, but your editor was unable to experiment with that.

A word to the wise: if the app is refusing to connect to a Jitsi server with obscure and unhelpful error messages, the problem is almost certainly with the server certificate. The app seems to be much more fastidious than browsers about things like getting any intermediate certificates in the right place. Not that your editor would have ever made any such mistakes.

There are a number of other features that weren't tried in this experiment. It is possible, for example, to stream a conference to YouTube, from whence it can be broadcast to the world. "Stream to YouTube" also seems to be the answer, for now, for anybody who wants to record a conference. If there is an Etherpad instance running on the same server, Jitsi can integrate with it. There is also SIP integration allowing people to dial into a conference using a telephone, but that requires all of the usual painful SIP setup outside of Jitsi.

In testing with a handful of participants, your editor found that the demands on the server are relatively light, but not zero — about 6-7% of a low-level cloud server per participant. Video conferencing will always require a certain amount of bandwidth, but routing that data doesn't appear to require a lot of CPU time. The web client will definitely make a laptop's fans run, though.

One thing that seems to be missing, or at least not easy to find, is the ability for the server administrator to see which conferences are running and what their resource usage is. Also lacking is central control over the outgoing video stream, which might be wanted for those trying to use Jitsi for events like virtual conferences, where they may want to keep the focus on the speaker and mix in slides, for example.

Virtual conferences, of course, are one of the areas of interest here at LWN. If current conditions persist, as they well might, it is going to be hard for the community to have face-to-face gatherings for some time. We depend heavily on those gatherings to solve difficult problems and, in general, to come to know the human beings that exist on the other side of (most) email addresses. Both of those roles are crucially important; we will need to find a way to replace them until real conferences again become possible.

There are a number of groups looking for solutions to these problems. Our community as a whole would, of course, be most comfortable with solutions built on free software. Jitsi seems like a good base to work from, if not a complete solution. Your editor will continue to investigate the alternatives in the coming weeks; stay tuned.

Comments (37 posted)

Working-set protection for anonymous pages

The kernel's memory-management subsystem goes to great lengths to keep the pages that are actually in use in memory. But sometimes it gets things wrong, leading to reduced performance or, in the worst cases, flat-out thrashing. We may be about to see a significant improvement, though, thanks to a patch set from Joonsoo Kim changing how anonymous pages (those containing data not backed by files on disk) are managed. As it turns out, all that had to be done was to make use of some work that already exists in related parts of the memory-management code.

LRU lists

A bit of background may be helpful for understanding how this patch set works; we'll start with a highly oversimplified picture, then add some details as we go.

Virtual-memory systems allow applications to address far more memory than can actually fit into the physical memory installed in the system, so a significant part of any given process's address space is likely to exist only on secondary storage at any given time. Obviously, the pages that are in physical memory should be the ones that the process is going to use in the near future. The kernel cannot know for sure which pages will be useful, though, so it must fall back onto a set of heuristics that allow it to guess as well as it can.

Some of those heuristics are reasonably straightforward. For example, if a process is observed to be working through a file sequentially, chances are pretty good that it will soon be wanting the pages of the file immediately after those it is accessing now. Another heuristic, found at the core of almost any virtual-memory implementation, is that pages that have been used recently are likely to be used in the future, while those that have languished unused for a while may not be worth keeping around.

To implement that last approach, the kernel maintains a "least-recently used" (LRU) list; all user-space pages in physical memory are kept on that list. The kernel occasionally checks the pages on the LRU list and moves those that have been accessed recently to the head of the list. When more memory is needed, to bring in pages from secondary storage, for example, pages at the tail end of the list are reclaimed.

In truth, the kernel maintains more than one LRU list. To begin with, the "LRU list" is actually two lists: the "active" and "inactive" lists. The active list functions mostly as described in the previous paragraph, except that, when pages fall off the tail of the list, they are put onto the inactive list instead. At that point, the protections on those pages are set to disallow all user-space access. Should some process access one of those pages, a "soft" page fault will result; the page will be made accessible again and returned to the active list. When memory is needed, pages will be reclaimed from the inactive list.

The inactive list thus serves as a sort of second chance for pages that are on their way out. But it also plays another role: managing pages that are highly likely to only be used once. A classic example is a process reading through a file; the pages read will probably be processed and not be needed again. These pages probably should not push out memory that is likely to be useful in the future. The kernel handles this case by putting newly faulted, file-backed pages directly onto the inactive list; they will only move to the active list if they are accessed again before being reclaimed.

In truth, things are more complicated that that; among other things, there are more than two LRU lists. Relevant here is the fact that there are separate active and inactive lists for file-backed and anonymous pages. It is common to reclaim file-backed pages before anonymous pages, since the former often need not be written back (while anonymous pages must always be written to swap) and can be easier to get back if need be. It is also worth noting that, in the case of file-backed pages, the kernel maintains "shadow entries" to remember (for a while) the existence of pages that have been reclaimed off the inactive list. If those pages are "refaulted" back in, the kernel knows that it's pushing out useful pages and can make adjustments to try to avoid doing that.

Improving anonymous LRU-list behavior

Kim's patch set addresses two significant differences between how anonymous and file-backed pages are handled. One of those is that, while file-backed pages are faulted into the inactive list as described above, anonymous pages go directly to the active list. If an application faults in a lot of anonymous pages, it will likely push other useful pages off the active list onto the inactive list. If the newly faulted pages are only used once, though, they will push aside other, more useful pages needlessly. To address this, Kim's patch set causes anonymous pages to be faulted into the inactive list, just like file-backed pages are. If those pages are truly useful, they will be promoted to the active list when the soft fault happens; otherwise they can be reclaimed relatively quickly.

The other change addresses the fact that refault tracking, in current kernels, is only done for the file-backed LRU list. Once an anonymous page is reclaimed, the kernel forgets about its history. As it turns out, the previous change (faulting pages into the inactive list) can exacerbate some bad behavior: adding new pages to the inactive list can quickly push out other pages that were just faulted in before they can be accessed a second time and promoted to the active list. If refault tracking were done for the anonymous LRU list, this situation could be detected and dealt with.

So the patch set adds this tracking for anonymous pages. In a sense the work was straightforward, since the infrastructure for refault tracking already exists and can be reused; it simply needs to be extended to track more than one LRU list. There are some added details, though. Since anonymous pages are written to swap when they are reclaimed, the shadow LRU entry used to track refaults can be written there as well rather than being kept in RAM, for example.

Kim included a number of benchmarks showing how these patches improve memory-management behavior in various situations. What really got the attention of memory-management maintainer Andrew Morton, though, was this automated test result showing a 400% improvement in a virtual-memory scalability test. He asked: "One wonders why on earth we weren't doing these things in the first place?" Kim replied with copies of the patches adding the current behavior in 2002 — written by a certain Andrew Morton, who acknowledged that it may well be time to revisit some of that work.

There does not appear to be any opposition to this work in the memory-management community. That does not necessarily mean that it will be merged soon; memory-management patches often require a lot of testing and review before developers become confident enough to apply them. That is doubly true of patches affecting heuristics, since they can often cause unexpected problems in surprising places. These patches have not even made it into the -mm tree yet, a step that would increase both testing and review. So, even though Morton has said that "given all the potential benefits, perhaps I should be more aggressive here", this work doesn't look like 5.7 material. It may well find its way upstream shortly thereafter, though.

Comments (3 posted)

Automatic buffer selection for io_uring

The io_uring subsystem has, in the last year, redefined how asynchronous I/O is done on Linux systems. As this subsystem grows in both capability and users, though, it starts to run into limitations in the types of operations that can be expressed. That is driving a number of changes in how operations are programmed for io_uring. One example is the mechanisms considered for carrying a file descriptor between operations that was covered here in early March. Another has to do with how I/O buffers are chosen for operations.

As io_uring developer Jens Axboe describes in this patch set, the usual mode for programs that handle large numbers of file descriptors is to use poll() to find out which descriptors are ready for I/O, then making separate calls to actually perform that I/O. One could use io_uring in this mode, but it defeats one of the purposes of the whole exercise: avoiding system calls whenever possible. The io_uring way of doing things is to just queue an asynchronous operation on every file descriptor, then react to the resulting events whenever one of those operations is executed.

Working that way can indeed reduce system calls — all the way to zero if the request ring is kept full. But it also requires allocating a separate I/O buffer for each of those queued operations, even though many of them may not execute for an indefinite period of time. The poll() method, instead, allows an application to defer buffer allocation until a buffer is actually needed. Losing that flexibility can result in significantly higher memory use for applications that keep a large number of operations outstanding.

What is needed here is some sort of mechanism that allows buffers to be allocated to operations after they have been queued in the ring. The answer is, of course, obvious: add a hook for a BPF program that can perform buffer management in the kernel at the moment that any particular operation is able to go forward. Why even try anything else? Unfortunately, Axboe said, "I had a hard time imagining how life times of the buffer could be managed through [BPF]", so that idea went by the wayside.

Fortunately, there was another idea waiting in the wings: have the application provide one or more buffer pools to io_uring, which would then select a buffer from one of those pools whenever one is needed. That is what Axboe ended up implementing.

To use this mechanism, an application starts by queuing one or more IORING_OP_PROVIDE_BUFFERS operations to give a set of I/O buffers to the kernel. Each operation includes the base address of the buffer(s), a count of buffers, the size (the same for all buffers in this operation), a base buffer ID, and a group ID. If more than one buffer is included in the request, the buffer ID will be incremented by one for each after the first. There is no requirement that all buffers in a given group be the same size, but that seems to be the way that the mechanism is intended to be used.

Subsequently, operations can be queued without providing buffers at submission time; instead, the special value IOSQE_BUFFER_SELECT is used. The new buf_group field in the queue entry should be set to the ID of the group from which a buffer should be obtained when needed. When an operation unblocks and can proceed, the kernel will grab a buffer from the indicated group and use it. The size of the buffer is not considered during the selection process so, if the buffer is too small, the operation will not be able to complete properly. The ID of the selected buffer is returned with the operation's completion status.

If the requested buffer group is empty, the operation will fail with an ENOBUFS error. Once a buffer has been consumed by an operation, the kernel will not use it again until it has been given back with another IORING_OP_PROVIDE_BUFFERS request.

Only some operations support buffer selection in the current patch set; it is limited to read(), readv(), recv(), and recvmsg(). Earlier versions of the patch set supported write(), though your editor will freely admit to being baffled with regard to how that was supposed to actually work even after looking at the code; that support was removed in version 3.

This work has not yet found its way into linux-next, but there is still some time before the 5.7 merge window opens. So there is a chance that the buffer-selection feature could yet land in the next development cycle. That will increase the flexibility of io_uring operations, and no BPF hooks are required.

Comments (10 posted)

Django changes its governance

The Django web framework has come a long way since it was first released as open source in 2005. It started with a benevolent dictator for life (BDFL) governance model, like the language it is implemented in, Python, but switched to a different model in 2014. When Python switched away from the BDFL model in 2018, it followed Django's lead to some extent. But now Django is changing yet again, moving from governance based around a "core team" to one that is more inclusive and better reflects the way the project is operating now.

Django actually started out with co-BDFLs; Adrian Holovaty and Jacob Kaplan-Moss filled that role until they retired in early 2014, which motivated the change to the core-team model. By 2018, some problems with that new model were being felt, so James Bennett spearheaded an effort to change it, which resulted in Django enhancement proposal (DEP) 10 ("New governance for the Django project")—adopted on March 12. Perhaps unsurprisingly, some of the problems identified for Django are sometimes heard in Python circles as well; the changes for Django could be a harbinger of Python's governance down the road.

The core team model relied on a select group of developers (the committers) to make all of the high-level decisions for the project. The BDFL "deadlock-avoidance" role was filled by a technical board that was elected by the committers; the core team of committers was responsible for all of the changes to Django. Around the same time as the governance change, the Django Software Foundation (DSF) started hiring "Fellows" to manage a lot of the day-to-day maintenance of the software and project; it turned out that the Fellows ended up doing most of the actual commits. As Bennett put it back in 2018:

So the proposal is to eliminate the core team entirely, which will have minimal to no effect on the project, while recognizing all of the existing and former core team members under the honorary "Django Core Developer" title. The technical board will still be there to break ties and such, but it will be elected differently. From the announcement of the change, also authored by Bennett:

To replace the job of committers, at least as it was originally envisioned, there will be two new roles: merger and releaser. Both roles will be filled by the current DSF Fellows, Carlton Gibson and Mariusz Felisiak; the technical board will appoint a few more people to those roles. The role names are fairly self-explanatory; mergers will merge pull requests that have achieved rough consensus, while releasers will be empowered to actually create Django releases and make them available in the Python Package Index (PyPI) and on the project's home page.

For the most part, the changes are, in fact, not really changes at all. They are simply recognizing how the development work has been done of late and formalizing it. But removing the perceived barrier of needing to be a member of the core team—thus opening up Django development to more people—is clearly the main driver for the change. The development process was working just fine, but the formal governance structure had not quite kept up; in effect, the core team idea simply outlived its usefulness:

The reaction has been somewhat quiet, perhaps because the idea has been floating around without any real opposition for more than a year. It did require that four-fifths of the core team, the technical board, and the DSF board all approve the measure, so it is pretty clearly uncontroversial overall. Kaplan-Moss was certainly in favor of the change, noting that a more democratic process was needed when he and Holovaty stepped down, but there are other considerations now. "This time our main obstacle is inclusion; to grow further we need a broader talent pool. I believe this change will get us there (or at least take some good steps in that direction)."

The belief that it requires "superhuman" powers to become a core team member has reared its head in the Python project along the way as well. Committers or core developers have evolved over time into a much more bureaucratic role; in the early days of many projects, a "commit bit" meant that one could add code to a project with impunity, or nearly so. These days, for Django, Python, and lots of other projects, committer simply means "the one who has to actually make the merge happen" in a mechanical sense. The decision-making process for those projects is handled separately.

It is also true that both Django and Python are mature projects at this point; there is much less of a "wild west" feel to their development these days. That may also be contributing to less interest in their "core development" in general. But both projects (and plenty of others) need an ongoing maintenance crew, for features, bug fixes, security problems, and so on, so it makes sense to try to ensure that those with an interest realize that they can actually fill those roles. People with a long-term outlook that will keep the best interests of the project to the fore, not necessarily some massive amount of technical know-how, is what will keep the projects going over the next few decades—or far beyond.

Comments (none posted)