Leading items

Welcome to the LWN.net Weekly Edition for March 27, 2025

This edition contains the following feature content:

• Lessons from open source in the Mexican government: a SCALE presentation on the challenges that come with increasing open-source usage in government.
• OSI election ends with unsatisfying results: the election is over, but the questions continue.
• MM medley: huge page allocation, page promotion, KSM, and BPF: an overview of several current memory-management topics.
• Multiple memory classes for address-space isolation: speculative-execution vulnerabilities cannot expose data that is not mapped, so isolating address spaces has the potential to protect against these problems, including the ones we don't know about yet.
• The guaranteed contiguous memory allocator: another attempt to solve the old problem of large, physically contiguous memory allocations in the kernel.
• Development statistics for 6.14: where the changes in the latest kernel release came from.
• The 2025 Linux Storage, Filesystem, Memory-Management, and BPF Summit: the beginning of the flood of coverage from this annual core-kernel conference, including:
  • The state of the page in 2025: how the folio transition is going and what to expect next.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Lessons from open source in the Mexican government

The adoption of open-source software in governments has had its ups and downs. While open source seems like a "no-brainer", it turns out that governments can be surprisingly resistant to using FOSS for a variety of reasons. Federico González Waite spoke in the Open Government track at SCALE 22x in Pasadena, California to recount his experiences working with and for the Mexican government. He led multiple projects to switch away from proprietary, often predatory, software companies with some success—and failure.

González Waite began by noting that he is a Mexican/Kiwi ("there's not many of us", he said with a grin), who spent nine or ten years in high-level roles in the Mexican government "promoting open-source implementations". Among other things, he was the CTO for the Ministry of Foreign Affairs; "I am actually responsible for Mexicans having an electronic passport today." That was one of the projects that he led and part of it was done with open-source software, which is something that people find to be amazing, González Waite said. He served in the office for national strategy under President Andrés Manuel López Obrador, eventually moving into the CEO role at the National Research and Innovation Center for Mexico.

In all of his roles, he advocated using open source within the government; it is never easy to be a "change-maker" of that sort, he said. He left the government at the recent change of presidents (to Claudia Sheinbaum) and is now working on "helping people do their own transformations" to open source, while keeping an eye out for the next big thing. He noted that he would take questions at the end and he might not be able to answer them all due to confidentiality responsibilities, but he is "no longer a public official, so that gives me a lot more leeway to talk freely".

Why?

"So why open source for the government?" One reason is to cut costs; Mexico is a financially small, developing country that is always looking to reduce costs, he said. Paying for licensing was costing a lot of money that could be used to do other things. The López Obrador administration passed an austerity plan into law, so that various actions (e.g. government officials traveling out of the country for work) needed presidential-level approval. That also affected purchasing licenses and the plan pushed the use of open source.

Another major reason is to work toward Mexico having "IT sovereignty". One major problem that the country has had is that its IT leadership is not technical, which means that people were signing off on projects and licenses without understanding what they were buying. "They really didn't know if they are were getting good value for [the] money".

There is a need to build up the talent within the government to support open-source software; "there's no point in building a whole infrastructure on open source if you don't have the talent to keep it up". There has been a lot of effort to bring in new people and to train existing workers to that end. One goal is to go from being an IT-consumer nation to being an IT-producing nation. It is surprising to many who think of Mexico as a manufacturing and assembly nation that the country does actually produce a lot of technology, he said. Much of what was being produced within the government, though, was being siphoned off by the private sector; more recently, that is changing, so that the government can package and even sell its IT development.

Another goal is for Mexico to become more self-sufficient, so that it is not locked-in by vendors of various sorts. In his most recent role, he was able to see what was happening all across the government. One common thread is that when agencies were asked why they were spending so much on a particular service, they claimed they had no choice, even though there are lots of other companies offering the same services. It turned out that various contracted companies had corruptly put the software licenses they bought for the government into their own names, leading to a lock-in for their services. Moving to open source can break those and other kinds of locks.

There are multiple agencies within the government to handle various pieces of the technology puzzle, González Waite said. He headed INFOTEC, which is the IT service provider to the rest of the government. It managed the largest telecommunications project in the world at that time to place 6,000 BTS antennas, while deploying over 30,000km of fiber-optic cable, in various places throughout the country. That was part of the CFE Internet Para Todos project, which is aimed at the democratization of access to the internet; prior to that, a large region in Mexico had no internet because it was not commercially viable.

All of the technology projects within the federal government require approval from the president's office; that has been part of the country's laws for quite a while, González Waite said, but had not been enforced. Under the López Obrador administration, reviews of those projects often found that they could use open source, rather than the expensive proprietary solution being proposed, so the projects needed to switch.

Using open-source software has been enshrined in Mexican law since 2021, when it was specifically authorized for government agencies. González Waite went briefly through a few pieces of legislation to show that the government is serious about cutting costs and gaining control of its infrastructure through the use of open-source software.

Foreign affairs

He led a project for the foreign affairs agency; it started with separating the data that was being stored so that some of it was put into the cloud, while the most sensitive data was still stored in the government's data center in Mexico. The agency was resistant to moving any of its data into the cloud, because it was supposedly all "national security data", but there was no data classification that specified which data was actually sensitive. When pressed, the agency claimed that anything ever produced in or by a consulate qualified; he pointed out that meant that consulates sending party invitations and holiday greetings were violations of the national security act. That led to an effort to classify the data and, thus, to foreign affairs being the second government agency to store data in the cloud.

Another part of the project was to move away from Oracle and to PostgreSQL. That change led to various threats and intimidation from the company when it learned of the change, González Waite said. "They told me that the entire passport system of the country was going to fall down" and that it would be his fault that Mexico could not let anyone into or out of the country. "Guess what? That didn't happen."

It turned out that the employees working on the database had heard of PostgreSQL, but never used it. So his team reached out to the Mexican PostgreSQL community for assistance and advice. It took around three months to migrate the information out of Oracle into PostgreSQL. The team took advantage of the shift to restructure the database "because we found that our storage provider was being a little bit naughty", storing the data three or four times in order to charge more money. "Mexico is still a very corrupt country, like a lot of the countries in the world", he said. When you make decisions that take money out of people's pockets, they "start getting really nasty".

The "biggest game-changer" from the switch was that the agency now had direct access to its data and was not reliant on partners and contractors to provide applications and reports. That led to the ability to develop an electronic passport for Mexico. The platform for that was developed in-house, "so we are not paying any licensing"; various open-source software libraries were used in the platform, including for handling biometric information, he said.

Education

In addition to the need for retraining technical staff on open-source topics, his team discovered that students at universities and technical schools were not getting exposed to open-source software either. INFOTEC started a large internship program to place students into government agencies, but found that the students were unable to perform their duties due to a lack of training. That led to creating three-month boot camps for software development and cybersecurity for the interns. The interns were getting paid for that time, which made the training expensive for the agencies, but it brought the interns up to speed so that they could be productive going forward.

Along the way, he visited the communication lab at a technical university and saw that all of the equipment there was from Cisco. The students there did not know how to, say, configure a firewall for any other type of equipment or using open-source software tools—and they were fearful about learning anything else. This is, González Waite said, part of the strategy of the companies, "where they were pretty much giving away the equipment to the universities, but creating a handicap for the students".

To combat that, INFOTEC developed four online educational programs, adding cloud computing and data center operations to the two boot camp subjects, all based on open-source software. Those were two-year courses that included practical work within the government at various levels (local, state, or federal). There was also a need for a "fast track", so the two boot camps were translated into 22-week training courses for those subjects. These open-source education projects reached across borders, he said, which was "a very cool thing"; El Salvador has adopted them and more than 5,000 people have enrolled. Beyond that, the cybersecurity course has been modified to target non-technical government employees in Mexico and more than 3,000 have taken advantage of it.

A different form of education came about in a project to use Mifos for banks in Mexico. Mifos is an open-source platform for handling financial transactions of various kinds. One of the big barriers to getting the project off the ground in Mexico was in educating the regulatory body (National Commission on Banking) about open-source software. His team spent around six months in meetings with the commission to do that education. That paid off and, as the project gained momentum, the regulators had the context needed, which smoothed the path. The project partnered with the Mifos community; some members of the community traveled to Mexico to meet with the regulatory bodies as well.

The other big challenge from this project was in being able return code that was developed for Mexican banking to the Mifos project. Banking institutions have security concerns about releasing code, but the Mifos community wanted to add the code to its repository and tout the deployment, which was one of the largest ever. That was a conversation that played out over three years and eventually resulted in a complicated scheme where older versions of the code could be added to Mifos after two subsequent releases were rolled out to production.

The banking project had gotten its start in part because of a study showing the enormous amount of money being spent to run the existing infrastructure for handling certain types of loans. But, of the two different deployments that came out of the project, the one directly targeting saving a big chunk of that money ended up being canceled—after it had already been proven to work. Another deployment, which basically took the code from the first and reworked it to handle different kinds of loans, was smaller in size, but was switched over to successfully.

The main difference is that the successful project was done for a recently restructured agency that lacked the bias and entrenched interests that the other agency had. The leadership in that organization was better, he said, and also a bit desperate to fairly rapidly show the president's office that it was doing its job effectively.

Lessons

The regulations around open source helped a lot, "because it gave a legal framework to explain to people why we were doing stuff", but it was not enough. Whenever the idea of using open-source software was raised, there was fear because of a lack of knowledge about it, but, because the decision-makers were government officials, there was legal fear on top of that. "That liability could get you into jail, so don't change the technology, leave whatever there is, pay those millions, and nothing will happen, you will be fine."

Technology is often seen as the problem, he said, but he generally found that the problems were due to using obsolete technology and a lack of knowledge about the data being handled. There is often no documentation of the data and its structure, coupled with no understanding of that by the people in charge of it. Poor leadership in the agencies is another barrier; there needs to be a champion for a change of this sort, who understands what needs to be done and properly assigns people to work on it. Many Mexican government officials are political appointees who do not want to get involved in managing a project, so they hand it off to someone else who is already doing too many other things.

Another reason that projects would fail was because the leadership decided that switching to open source meant that no money would be needed in the future. But servers, developers, system administrators, and so on still require a budget if the project is going to succeed. Beyond that, some of the government employees did not have any interest in acquiring the skills needed to switch to open-source software. Many were happy to hand the whole job off to a vendor due to lack of knowledge or "other personal motives".

The biggest lesson that he learned was that projects for switching to open-source software solutions was that either "you win big or you lose big". He and his team never encountered a technological or regulatory problem that could not be fixed; both technology and regulations can be rewritten, "it just takes time and it takes effort". They were able to anticipate most of the loss scenarios in advance because they saw that logical arguments were not prevailing; when that happens, "there's only one explanation and that's an alternative motive". His "number one recommendation" is to ensure, even before the project gets started, that it has the right champion and backing inside the agency or organization; that is the real determiner for whether a project will succeed or fail.

In answer to a question about license audits (to enforce compliance with proprietary-software terms) that was asked by a Mexican citizen who used to work at INFOTEC as a contractor, González Waite said that all of the large proprietary software companies "are big bullies". He has been called into the US embassy and been threatened because Mexico was using technology that was not from the US; those threats were dialed back when he explained that the government also used software and services from Amazon, Google, and Microsoft. Various companies use the US government to bully other countries, but they also use license audits as a reaction to projects that move to open-source software. Every time a successful switch happened, "six months later there was an audit"; having the right legal team helps defend against those tactics, he said.

Earlier, González Waite had noted that the new presidential administration had not specified the use of open-source software in any of the legislation that has been enacted so far. Another Mexican asked him whether he was concerned that the new administration was turning away from open source. "I think it's too early to know that", he said. For now, the relevant legislation from the previous administration has not been undone, but that will come sometime soon, and he is hopeful that the open-source efforts will continue.

The final question that he took was about handling obsolescence in the libraries and other dependencies within government systems; many of those projects will run for decades, but the dependencies may not be maintained for that long. González Waite agreed that it is a major problem; many governments are run by politicians with a short attention span. The politicians are looking for recognition, which leads to votes, and then they move onto the next thing. In Mexico, it is particularly problematic because of the presidential change every six years; most contracts and plans do not stretch past that transition point. It is important to address that problem, because technology is changing so quickly, though it would take major legislation to do so in Mexico—for now, that problem remains unsolved.

A video of just the talk should appear soon on SCALE's YouTube page, but the full set of talks from the room is available with this talk as the first of the day.

[I would like to thank the Linux Foundation, LWN's travel sponsor, for funding to travel to Pasadena for SCALE.]

Comments (12 posted)

OSI election ends with unsatisfying results

The Open Source Initiative (OSI) has announced the results of its recent board of directors election. Ruth Suehle and McCoy Smith are new to the board, while Carlo Piana will serve another term. The results, however, seem tainted in the eyes of some participants and observers. The election has been plagued by missteps from the beginning. It has culminated with the exclusion of three candidates for failing to meet a requirement to sign the OSI board agreement, which was added after the election was over and before results were tallied or announced.

OSI board of directors

OSI is a 501(c)(3) non-profit founded in 1998. It, of course, published the Open Source Definition (OSD) which is the litmus test that open-source licenses are judged by. Its primary function these days seems to be around approving licenses via its license review process, open-source advocacy, as well as policy and standards work.

OSI's board has 12 directors. Four of the board members are directly appointed by the board itself. According to its elections page, this is "to ensure that the resulting Board has an optimal balance of skills, knowledge, and demographics". Board-appointed directors serve two-year terms. Four of the board seats are affiliate seats, which are nominated and voted on by OSI's affiliate members, which are the organizations "engaged in and with the open source community". That includes projects like the Apache Software Foundation, Debian, GNOME, KDE e.V., the Linux Foundation, and many others. Affiliate directors serve three-year terms. Finally, the other four board members are nominated and voted on by the individual members of OSI. That would be OSI supporters who pay a yearly membership fee (starting at $50). Directors nominated by individual members serve two-year terms. No director is allowed to serve more than six consecutive years before being required to sit out a year before serving again.

OSI has elections annually, to replace directors whose terms are ending, with the timeline determined by the board in December and publication of the timeline for the election in January. This year, the elections were announced on January 22, by OSI head of community Nick Vidal, with nominations running through February 17 for one affiliate director and two individual directors. That was later corrected to two affiliate directors and one individual director, after nominations had closed, with an update inserted into the blog post:

Thankfully, a candidate caught the mistake in time, and we have corrected the ballots before elections opened. To prevent similar errors in the future, we have updated our procedures to improve accuracy. We appreciate your understanding and your participation in the process!

The elections, however, are non-binding. OSI's board makes the final determination whether to accept the results and seat the winners.

Election missteps

Typically, OSI elections draw little attention. However, the OSI has received more attention than usual in the past year over the Open Source AI Definition (OSAID), which LWN covered in October. Several people critical of OSAID decided to stand for the board, including Debian developer Luke Faraone, who initially wanted to be nominated as an affiliate candidate. However, since there was apparently only one affiliate seat open, they said that they decided to run for an individual seat instead.

Since Bradley Kuhn is also running for an Affiliate seat with a similar platform to me, especially with regards to the OSAID, I decided to run as part of an aligned "ticket" as an Individual Member to avoid contention for the 1 Affiliate seat.

Faraone submitted their self-nomination "around 9pm" US Pacific Time on February 17. Their candidacy was rejected, though, due to missing a poorly-advertised deadline of 11:59 p.m. UTC on that day. While the date was well communicated, the time was not. It was only present in one of two emails about the election that were sent to members. (I received copies of these emails from Faraone and a member of the OSI board.) It was apparently also mentioned during information sessions run by OSI for potential directors, but Faraone did not attend those. (Update: Faraone's post made reference to post-nomination meetings, rather than pre-nomination meetings, which of course they could not attend as their nomination was not accepted.)

Faraone said that it seemed arbitrary and capricious to govern elections by UTC when the organization is based in California. They urged OSI to reconsider the policy and allow them to stand for election, but the organization declined to do so.

Kuhn, and Richard Fontana, ran on an "OSI reform" platform that sought to repeal the Open Source AI Definition (OSAID) and remove the "code of silence" from the board agreement. Specifically, the agreement requires board members to "support publicly all Board decisions, especially those that do not have unanimous consent".

On March 3, Kuhn published a lengthy blog post about problems with the OSI elections which said that this was unfair to candidates:

If we had known there were two Affiliate seats and just one Member seat, Debian (an OSI Affiliate) would have nominated Luke a week early to the Affiliate seat. Instead, Debian's leadership, Luke, Fontana, and I had a complex discussion in the final week of nominations on how best to run as a "ticket of three". In that discussion, Debian leadership decided to nominate no one (instead of nominating Luke) precisely because I was already nominated on a platform that Debian supported, and Debian chose not to run a candidate against me for the (at the time, purported) one Affiliate seat available.

He argued that this impacted several affiliate organizations. Kuhn said he was nominated by four affiliates, because "if they nominated someone else, that candidate would be effectively running against me". At least one of those would have nominated a second person if it had been known at the time that there were two available seats and not only one. He called for OSI to reopen nominations and reset the election, but to no avail.

Kuhn has detailed a number of what he terms irregularities with the OSI election in his post. Some might be considered minor, such as a complaint about the order in which candidates are presented on the ballot, or the fact that only one nominating organization was listed for each candidate instead of listing all nominating organizations. One, in particular, seems fairly substantive and worth noting. Kuhn had reached out, by email, to likely voters in the affiliate category to promote his candidacy. In an apparent reaction to this, OSI sent out an email to affiliate voters that said it had learned a candidate was reaching out to voters "without their consent":

We do not give out affiliate emails for candidate reachouts, and understand that you did not consent to be spammed by candidates for this election cycle.

Candidates can engage with their fellow affiliates on our forums where we provide community management and moderation support, and in other public settings where our affiliates have opted to sign up and publicly engage.

Kuhn said that he spent between 12 and 14 hours researching to find the contacts at affiliate organizations to reach out to. The text of the email Kuhn sent to 55 people at affiliate organizations is included in his blog post. Kuhn complained that OSI had not given candidates any specific guidance about reaching out to voters, and said that it is not spam to "contact one's 'FOSS Neighbors' to learn their concerns when in a political campaign for an important position". What is considered "spam", of course, varies greatly from person to person. However, in my experience, Kuhn's lobbying affiliates for votes is not at all unusual. While I have never held an affiliate vote for an organization in OSI elections, I have been a voter for similar positions in other open-source foundations and can attest that other people have lobbied for my vote. Merely reaching out to voters once and asking for consideration does not seem beyond the pale in this type of election, nor unwelcome. If an affiliate voter is put off by this sort of campaigning, the obvious result would be that they vote for another candidate.

In the same post, Kuhn said that OSI should reopen the nominations or just forget elections entirely, since the elections are only advisory in any case.

OSI's halfway solution (i.e., a half-heartedly organized election that isn't really binding) seems designed to manufacture consent. OSI's Affiliates and paid individual Membership are given the impression they have electoral power, but it's an illusion. Giving up on the whole illusion would be the most transparent choice for OSI, and if the OSI would rather end these advisory elections and just self-perpetuate, I'd support that decision.

Board agreement

On March 19, Fontana, who was running for an individual seat, posted on the OSI Discourse forum that he had received an email asking him to sign the OSI board agreement before the results were announced. Kuhn wrote about this requirement on his blog the same day. Fontana and Kuhn objected to this, saying that it was explained during the orientation sessions that "signing the Board Agreement was a requirement to be seated as a director, but it was never stated that mere candidates were expected to sign".

OSI executive director Stefano Maffulli replied:

While the polls were open, we've heard that there may be candidates with no intention to sign the board agreement, which has become a mandatory requirement since you last served. We need to know who the actual candidates are before we run the STV calculation to determine the outcome of the vote. So for process efficiency, the board asked all candidates to confirm their good faith intention to serve on the board, so we can tell the software.

The board agreement only applies to seated members.

If the agreement applies only to seated members, it is puzzling that the OSI would require signing the agreement before a candidate has been elected. It seems akin to requiring political candidates to take an oath of office before an election, rather than doing so in order to be seated or asking job candidates to sign non-disparagement agreements before being offered a job instead of as part of onboarding paperwork.

Kuhn refused to use the proprietary Docusign service to sign the agreement, but returned an annotated PDF that struck out the requirement to publicly support all board decisions. Fontana attached an addendum that would allow and encourage public dissent "involving matters central to OSI's core mission and the public interest", as long as the member makes it clear that they are speaking in their personal capacity. It also would free directors to comment on decisions made prior to their tenure, and ensure that nothing in the board agreement would silence whistleblowing or contravene fiduciary duties to the OSI.

Thierry Carrez, a member of the board, defended the "code of silence" as standard practice in non-profit boards that he has served on. He asked Fontana how publicly complaining would help an organization like OSI, instead of hurting it. Fontana replied that it should be possible for a board member to express dissent in a respectful way rather than a false show of public unanimity or resigning from the board. Smith defended the requirement and said that disagree and commit is a "pretty common standard in the corporate world". Pamela Chestek, also a board member, and Casey Valk, who ran but did not win a seat, added support in the conversation for the board agreement's requirement to support board decisions.

Results and aftermath

Both Fontana and Kuhn said that they had returned a signed version of the board agreement prior to the stated deadline, but their signed versions were altered and apparently not delivered via OSI's preferred method of using Docusign. The OSI does not seem to have recognized those as valid, and votes for Kuhn and Fontana are not included in the public results. The announcement said that three candidates were excluded from the results, two for not signing the agreement and one for failing to meet the deadline, though it did not name the candidates who were excluded. The other candidate who seems to have been excluded is Bentley Hensel. His platform doesn't indicate any desire for radical change at OSI.

Chad Whitacre, who had run for the board in this election, wrote on the OSI forum that it was "unfortunate that OSI had to deal with such adversarial trolls in this election", though he acknowledged that the OSI had not helped itself by misstating the number of seats available for the election.

On social media, Kevin P. Fleming said that the entire process was confusing and poorly communicated, as well as placing requirements on candidates after votes had been cast. He said the people elected are good people, but "that's not the only measure of the process 'going well'". Maffulli replied that the candidates "excluded themselves" by not signing the board agreement. He allowed that there were bumps in the election process, and pointed to a decision by the board to create a retrospective on the election by April 19. Fleming followed up to suggest that the retrospective should be performed by independent members of the organization because the board members should not be analyzing their own behavior. Currently, it appears that board member Josh Berkus will be leading the retrospective.

Aaron Wolf pointed out that it isn't merely the candidates who have lost out: those who voted for them have been disenfranchised. He said that the only way forward with integrity would be to release the vote totals. Josh Triplett said that the board should have tallied the results, announced the winners, and then enforced the requirements. Failing to do that presents the appearance that the rule changes specifically targeted candidates that the current board does not like.

It doesn't *matter* if that was the real reason or not, it's the default and reasonable assumption anyone should make when the process of an election is changed on the fly in ways that *just so happen* to affect specific candidates. The process itself is now rightfully suspect, when it had no need to be.

The conversation has continued on social media, but nothing new has emerged to date. The OSI board seems content with its handling of the election, and it may never be known whether OSI's members and affiliates wanted Kuhn or Fontana on the board. That is deeply unfortunate, and undermines trust and goodwill in an organization that depends heavily on both. It is worth remembering that the OSI's authority to define what is, or isn't, open source is purely based on the willingness of others to follow its definition. It has no teeth to enforce the use of the term "open source". The respect for the OSD comes from a large community of people willing to police the use of the term, and to recognize the OSI's authority to approve (or not) new licenses.

It may well be that Fontana and Kuhn would have lost anyway, but we have no way of knowing. A cynical person might conclude that the last-minute requirement to sign the agreement was to disqualify one or both because they would have won otherwise; and that the OSI leadership was unwilling to have even a minority number of board members who might seek to steer the ship in a different direction. If that was the case, the board has the authority to simply treat the results as non-binding and it could have refused to seat unwelcome candidates without disqualifying them or using other machinations to pretend there was a fair election where there was not.

Comments (55 posted)

MM medley: huge page allocation, page promotion, KSM, and BPF

As the 2025 Linux Storage, Filesystem, Memory-Management, and BPF Summit (LSFMM+BPF) approaches, the density of memory-management patches on the mailing lists has increased. Included among those are patches aimed at improving the reliability and performance of huge-page allocation, implementing page promotion on tiered-memory systems, adding a different approach to deduplicating memory, and replacing the BPF memory allocator. Read on for an overview of each.

Reliable huge-page allocation

The use of huge pages — of various sizes — is increasingly important to the overall performance of many, if not most, systems. If memory fragmentation prevents the kernel from allocating huge pages, performance will suffer. While quite a bit of progress in avoiding and fixing fragmentation has been made over the years, the ability to allocate huge pages still decreases over the lifetime of the system. Johannes Weiner has been slowly working on a patch series to improve the situation for a couple of years now; the changes made are relatively small, but their impact may be quite a bit larger.

The kernel's page allocator tracks the "migration type" of each request; that type describes how easily a given page of data could be moved elsewhere in physical memory if it turns out to be in the way. The migratetype enum describes these types: whether pages can be moved or easily reclaimed, for example, or whether they are fixed in place. The kernel divides physical memory into contiguous groups called "page blocks", typically holding 512 pages; each page block has an associated migration type, and only compatible allocations are meant to happen from each page block. The intent is to group all of the unmovable pages together, preventing situations where one unmovable page prevents the defragmentation of a large block of memory.

While the page allocator tries to avoid allocating unmovable pages in movable page blocks, it also tries to avoid performing reclaim and compaction while a caller is requesting memory. An allocation request that starts either direct reclaim or compaction will do a lot of work before returning the asked-for memory, creating latency where it might not be welcome. In current kernels, the page allocator will give up and allocate from a page block with the wrong migration type before trying more expensive remedies.

Weiner has concluded that those remedies are only more expensive in the short term, though, since misplaced allocations create fragmentation that may last until the system reboots. So his series starts by introducing a new "defrag mode" that can be enabled via a sysctl knob. When this mode is enabled, reclaim and compaction will be attempted before falling back to allocating from a page block with the wrong migration type. Benchmark results shown in this patch demonstrate that, with this mode enabled, the success rate for huge-page allocation no longer degrades over time.

They also show, though, that the success rate starts at a much lower level than with current kernels and stays there. Two more changes are needed to bring the success rate to a level above what a freshly booted current kernel can achieve, and to keep it there. The first is a small change to instruct the kswapd and kcompactd kernel threads to try to create page-block-sized chunks of free memory rather than being content with smaller successes; that makes more full page blocks available for allocation as huge pages. The other change raises the bar for the number of free page blocks that those kernel threads aim to create.

The end result is a much higher success rate for the allocation of transparent huge pages, and a lower allocation latency as well. The results look good, but changes like these have the potential to create performance regressions in other types of workloads. So rather wider testing of this work is likely to be needed before the memory-management community will feel confident about merging it.

Hot-page promotion

In tiered-memory systems — those with memory having different performance characteristics — the kernel must constantly strive to keep the right pages in the right types of memory. Usually, that means placing frequently accessed ("hot") data in the fastest memory that is closest to the workload using it, while less-frequently accessed ("cold") data can be relegated to slower, more distant memory.

One problem with this approach is that it can be hard to determine how hot a given page is. Detecting memory that has not been accessed for a while is relatively easy, so demoting cold pages to slower memory can be made to work fairly well. The promotion problem, which requires determining which data in slow memory is sufficiently actively used to merit migration to faster memory, is somewhat harder.

The latest attempt at solving the promotion problem is this series from Bharata B Rao, which is built on the idea that the number of sources of information on data hotness is growing over time. Classic methods, such as scanning memory and checking the "accessed" bit maintained by the hardware remain. But newer technologies, including AMD's Instruction Based Sampling (IBS) and monitoring provided by CXL memory, are coming as well. What is needed is a way to gather and use all of this information to promote the hottest pages to faster memory.

Rao's patch set adds a new function that can be used by any subsystem that knows something about memory access:

    int kpromoted_record_access(u64 pfn, int nid, int src, unsigned long time);

This call tells the kernel the memory indicated by the page-frame number pfn has just been accessed from the NUMA node indicated by nid. The source of the information is provided by src, and time is the time of the access, in jiffies. Sources are represented by macros like KPROMOTED_HW_HINTS for hardware-provided access information, or KPROMOTED_PGTABLE_SCAN, for access information obtained by scanning the page tables. The return value is either zero or an error code indicating whether the call was able to record the access information. What the caller is meant to do with that information is not clear; the IBS driver included in the patch set ignores it.

Access data is stored in a hash table, keyed by the page-frame number. This storage looks expensive, with a fair amount of data being maintained for each page in the system. The consumer of this data is a new kernel thread called kpromoted; it uses an algorithm that Rao describes as "pretty primitive" to promote pages that appear to be hot — those that have been accessed a minimum number of times (twice in the current patch) within the last 5ms.

Much of the code in this series is, by Rao's admission, fairly rudimentary. The purpose at this point is not to provide a polished subsystem for merging; instead, it is to try to get a sense for whether the overall approach is viable. Most of the comments so far have focused on details, but Jonathan Cameron pointed out that CXL memory providers will be able to provide much of the information that this patch set is working to create by aggregating individual events. Using that data, he suggested, may give good-enough results without needing all of the storage required by Rao's approach.

Simpler KSM

The kernel samepage merging (KSM) mechanism was designed to improve memory utilization by detecting pages of memory with identical contents, then arranging for a single page to be shared while freeing the duplicates. Those pages may be shared across processes that are entirely unaware of each other; the shared memory is (if writable) marked copy-on-write, causing the sharing to be broken should one process write to a shared page. KSM can free a fair amount of memory for other uses, but it has never been used as heavily as might have been imagined. Its page scanning requires CPU time, it requires some fiddly tuning to match the workload, and it raises security concerns since it can be used to determine whether a page with a given content exists elsewhere in the system.

Mathieu Desnoyers has recently posted a simplified KSM-like functionality that he calls "Synchronous KSM" (SKSM). It is aimed at a specific, seemingly niche use case, though it should be applicable more widely. Desnoyers is working to make run-time code patching more common in user space. Code patching is heavily used in the kernel to take advantage of the optimal instructions for the detected CPU, enable or disable features without the need for run-time tests, enable or disable tracepoints, and more. Desnoyers is looking to bring more of these techniques to user space.

The problem with run-time code patching is that it breaks the sharing that normally happens with executable code. If many processes are running the same program, they will normally share a single copy of its text; code patching will break that sharing, even if every process patches its code in the same way. That increases memory use and decreases cache locality, taking away the performance that these techniques are meant to bring in the first place.

SKSM is meant to allow processes, with an explicit opt-in, to restore the sharing of these pages after they have patched them. A call to madvise() with the MADV_MERGE operation will set up this sharing for the indicated memory range. Significantly, that range is scanned immediately, and any sharing established, in the madvise() call itself. There is no kernel thread scanning memory looking for sharing opportunities after the fact. As a result, the overhead imposed by SKSM happens entirely at initialization time; after that, its job is done.

Linus Torvalds responded to the posting by saying that he had no interest in allowing a second KSM implementation into the kernel, but "if the feeling is that this can *replace* the current horror that is KSM, I'm a lot more interested". He admitted that he didn't know who was using KSM now, and worried that it might work well in specific cloud environments and, as a result, cannot be removed. David Hildenbrand said that KSM is mostly used in private clouds, where it can be "very efficient", but that its use in public clouds is strongly discouraged due to the associated security concerns.

As a result, it is not clear that SKSM is going anywhere. The existing KSM functionality, for all its faults, does seem to work for some users, so its removal would not be greeted with universal acclaim. Unless SKSM can somehow be made to fill in for KSM, there may simply not be a place for it in the mainline kernel.

Edging out the BPF allocator

BPF programs can run in almost any context, including within interrupt handlers and even in handlers for non-maskable interrupts (NMIs), where the ability to take locks is severely constrained. That can make allocating memory in those programs challenging. The BPF memory allocator was introduced in the 6.1 development cycle as a way for BPF programs to (attempt to) allocate memory in any possible execution context. It works by maintaining its own pool of memory that can be dipped into, without taking locks, when the need arises. This allocator works, but it operates independently of the memory-management subsystem and is counter to the ongoing effort to reduce the number of memory allocators in the kernel. One of the goals discussed at the 2024 LSFMM+BPF gathering was the desire to bring allocation for BPF programs back into the memory-management core.

Alexei Starovoitov has been working on that project; his work so far was posted in its ninth revision in late February. It adds a new allocation function:

    struct page *try_alloc_pages(int nid, unsigned int order);

This function is safe to call in any context. In more constrained contexts, this function will attempt to grab the number of desired pages (indicated by order) from the per-CPU free list maintained for the given NUMA node ID (nid). Failing that, it will attempt to take the suitable per-zone lock and acquire the memory directly from the buddy allocator. It is only an attempt, though; if the lock is not available, the allocator will fail the request rather than wait for it. As a result, try_alloc_pages() has a relatively high likelihood of failure, and callers must be prepared to do without the requested memory.

There is also a new free_pages_nolock() function that can release pages back to the system without taking any locks in contexts where that is not possible. This patch set does not go as far as removing the BPF-specific allocator, though it does provide much of the infrastructure that will be needed to complete that step.

Andrew Morton initially resisted this work, worrying that the extra maintenance burden it would impose on the core page allocator was not justified by the benefits it would provide. Starovoitov reminded him of last year's discussion, adding that moving this functionality back into the memory-management subsystem would eliminate the wasted memory sitting in the BPF allocator. Slab maintainer Vlastimil Babka added his support. Morton resisted no further, and these changes are currently sitting in linux-next, with a probable merge into the mainline happening for the 6.15 release.

Stay tuned

The 2025 LSFMM+BPF gathering begins on March 24 in Montreal, Canada. Many, if not all, of the above topics are likely to see further discussion there. LWN, of course, will be there; keep an eye on these pages for our reporting from the conference.

Comments (6 posted)

Multiple memory classes for address-space isolation

Brendan Jackman has been working to try to get ahead of the next hardware CPU vulnerability before it gets discovered. In January, he posted the second version of a patch set that introduces address-space isolation (ASI) as a way of preventing future CPU vulnerabilities from leaking important information. The core concept is to ensure that data that is not currently needed is not present in memory, so that speculative execution cannot leak it. The work is nowhere near ready to be incorporated into the mainline kernel — not least of all because it has a large performance impact in its current form — but it is likely to once again be a topic of discussion at the 2025 Linux Filesystem, Memory Management, and BPF Summit.

Jackman's patch set introduces different classes of memory. The classes are effectively isolated from one another, in order to avoid leaking information between them. The first version only differentiated between memory mappings for KVM virtual machines and everything else, but reviewers wanted a demonstration that his approach could handle more than just two classes. Therefore, the most recent version of the patch set has a separate class for kernel code that handles certain system calls that don't require access to sensitive data as well. Each class of memory has its own independent address space, intended to contain only information relevant to a particular part of the system. When the kernel needs to access user-space memory, or run code for a virtual machine, it needs to switch to using that class (and that address space). Speculative execution cannot leak information that is not mapped into memory, and the system keeps track of switches between classes in order to flush any other microarchitectural state that could be used to leak information.

If this idea sounds similar to kernel page-table isolation (KPTI), that's because it is. KPTI ensures that kernel and user space have almost entirely separate address spaces, with separate page tables. This prevents user-space code from using speculative execution to leak memory from the kernel. ASI is essentially taking the idea one step further, by carving up the kernel into (for now) separate sections for KVM operations, the parts needed for responding to certain non-sensitive system calls, and everything else. Somewhat confusingly, the patch set calls the address space for responding to system calls that don't need sensitive information the user-space ASI class, even though it is used in the kernel, not in user space.

Jackman's patch set does not interfere with KPTI; systems where both KPTI and ASI run together will currently end up with four separate address spaces per task: the unrestricted kernel address space, the user address space (added by KPTI), and one restricted kernel address space for each ASI class (KVM and ASI's user-space class). There are some fundamental drawbacks to this proliferation of address spaces, however.

Ideally, this design would make it trivial to respond to the next discovered CPU vulnerability; in the most likely case of another speculative-execution-based vulnerability, the kernel would already be protected. If some other way of leaking information with CPU state is discovered, kernel developers would just have to make sure that the relevant CPU state was included in the list of things to flush when changing classes. That list currently includes the translation lookaside buffer, the level 1 CPU cache, and branch-predictor state. Leaking information within a class are not a concern, because the idea is to only put the minimum necessary information in each class, so code running in an address space should already have access to it anyway.

As ever, the problem is performance. The impact of the patches varies greatly between different workloads, but the greatest impact that Jackman observed during testing was a 70% reduction in throughput on a particular disk I/O benchmark. Because each class of memory needs to have its own, separate set of mapped pages, the patch set removes the page cache from the kernel's direct map. File operations — which almost always need to touch the page cache — require changing ASI class for each operation, which is slow. Theoretically, such a change isn't necessary, if non-sensitive parts of the page cache can be added to the relevant ASI class. This is one of several potential improvements that Jackson called out in his commit messages. He hopes to implement them in the future, but right now, he's focused on validating the approach and seeing whether the kernel development community would support implementing it.

Borislav Petkov thought that Jackman's patch set presented a "weird API". It adds three different functions that parts of the kernel can use to manipulate the current ASI class: asi_enter(), which enters a restricted class; asi_relax(), which signals the end of untrusted code; and asi_exit(), which does the actual work of exiting a restricted class. Petkov thought the naming was unintuitive — calls to asi_enter() don't necessarily match one-to-one with calls to asi_exit().

The design, Jackman said, was something of a holdover from when he had only implemented support for KVM. Calls to asi_enter() are actually balanced with calls to asi_relax(), as he explained with this pseudocode description of that support:

    ioctl(KVM_RUN) {
        enter_from_user_mode();
        while !need_userspace_handling() {
            asi_enter();
            vmenter();
            asi_relax();
        }
        asi_exit();
        exit_to_user_mode();
    }

Since asi_exit() does the expensive work of leaving a restricted ASI class, arranging the API so that the kernel needs to call it as little as possible is necessary for performance. Petkov thought it would make more sense to ensure that the functions that need to be balanced with each other have corresponding names, and suggested renaming asi_exit() to asi_switch() and asi_relax() to asi_exit().

Jackman was receptive to the criticism, but pointed out that there were other complications: eventually, he would like to completely remove asi_exit(), and have the kernel automatically switch ASI classes when a need is detected at run time. This could potentially be another way to avoid unnecessary ASI class switches, for better performance. So any API design should make it easy to eventually do away with whatever equivalent of asi_exit() the kernel developers end up committing to. Also, the API needs to ensure that there's no period during which an interrupt can arrive and accidentally return to code that should run in a restricted ASI class without applying the restriction.

The second concern didn't make sense to Petkov, but Junaid Shahid shared a simpler example that made it click. Petkov suggested an API that separates out the operation of telling interrupt handlers that they may need to worry about returning to a restricted address space, actually switching to that space, and tracking whether the CPU was currently executing restricted code.

Jackman didn't see why updating interrupt handlers and switching address spaces needed to be separate operations, and proposed using functions called asi_start()/asi_end() to handle those and functions called asi_start_critical()/asi_end_critical() to demarcate untrusted code. Shahid and Petkov were generally happy with that design.

At the time of writing, the review and discussion are ongoing. While Jackman's updated patches seem to demonstrate that ASI could flexibly handle multiple different classes of memory, there are still a lot of questions that need to be answered before the work will be ready. Getting ahead of hardware vulnerabilities with a generic protection like the one Jackman proposes is a daunting task, but a worthy one. It remains to be seen whether other kernel developers will agree with this approach.

Comments (1 posted)

The guaranteed contiguous memory allocator

As a system runs and its memory becomes fragmented, allocating large, physically contiguous regions of memory becomes increasingly difficult. Much effort over the years has gone into avoiding the need to make such allocations whenever possible, but there are times when they simply cannot be avoided. The kernel's contiguous memory allocator (CMA) subsystem attempts to make such allocations possible, but it has never been a perfect solution. Suren Baghdasaryan is is trying to improve that situation with the guaranteed contiguous memory allocator patch set, which includes work from Minchan Kim as well.

In the distant past, Dan Magenheimer introduced the concept of transcendent memory — memory that is not directly addressable, but which can be used opportunistically by the kernel for caching or other purposes. Most of the transcendent-memory work has since gone unused and been removed from the kernel, but the idea persists, and this patch series makes use of it to provide guaranteed CMA.

Specifically, the patch set includes a subsystem called "cleancache", which is a concept that was proposed by Magenheimer in 2012. If the kernel has to dump a page of data, but would like to keep that data around if possible, it can put it into the cleancache, which will stash it aside somewhere. Should the need for that data arise, the kernel can copy it back out of the cleancache — if it is still there. Meanwhile, the page that initially contained that data can be reclaimed for other uses.

Guaranteed CMA then builds on cleancache by allocating a region of physically contiguous memory at boot, when such allocations are relatively easy. That memory is then turned into a cleancache and made available to the kernel. Whenever the memory-management system reclaims pages of file-backed memory, it can choose to place the data from those pages into the cleancache. Should that data be needed, an attempt will be made to retrieve it from the cleancache before rereading it from disk. The memory reserved for CMA is thus available to the kernel when not allocated to a CMA user, but in a restricted way.

At some point, some kernel subsystem will need a large, physically contiguous buffer. Requesting that buffer from the guaranteed CMA subsystem will result in an allocation from the reserved memory, after dropping any cached data that happens to be in the allocated region. This allocation can happen quickly, since that data has been cached with the explicit stipulation that it can be dropped at any time. This approach was proposed by Seongjae Park and Kim in 2014.

This new subsystem is integrated with the existing CMA API, so CMA users need not change to make use of it. The reserved region is set up by way of a devicetree property explicitly requesting the "guaranteed" behavior.

The end result is a version of CMA that is guaranteed to succeed as long as the total allocations do not exceed the size of the reserved area; existing CMA has a higher likelihood of failure. Since CMA usage is often restricted to a problematic device or two with known needs, sizing the reserved area for a specific system should be straightforward.

The other advantage of guaranteed CMA is latency; if the memory is available, it can be allocated quickly. CMA in current kernels may have to migrate data out of the allocated region first, which takes time. The downside is that the memory reserved for guaranteed CMA can only be used for data that can be dropped at will; that will increase the pressure on the rest of the memory in the system.

This patch series was posted just ahead of the 2025 Linux Storage, Filesystem, Memory-Management, and BPF Summit, where it is currently scheduled for a discussion in the memory-management track. There will probably not be a lot of comments on it ahead of that discussion. The patches are relatively small, though, and do not intrude into the memory-management subsystem on systems where CMA is not in use, so we might just see a transcendent-memory application actually go forward, some 15 years after the idea was first proposed.

Comments (7 posted)

Development statistics for 6.14

By the time that Linus Torvalds released the 6.14 kernel, 11,003 non-merge changesets had been pulled into the mainline, making this one of the smallest releases we have seen in some time. Indeed, one must go back to the 4.0 release, which happened almost exactly ten years ago, to find a release with fewer changesets than 6.14. Even so, "small" is relative, and 6.14 contains a lot of significant changes.

This kernel release included contributions from 1,897 developers; that, too, is a relatively small number — though rather larger than the 1,457 who participated in the 4.0 release. One only needs to go back to 5.15, released in 2021, to find a kernel release made with the participation of fewer developers. Of the developers contributing to 6.14, 228 were first-time contributors.

The most active developers this time around were:

Most active 6.14 developers

By changesets Kent Overstreet 275 2.5% Thomas Weißschuh 183 1.7% Jani Nikula 123 1.1% Sean Christopherson 120 1.1% Dmitry Baryshkov 106 1.0% Krzysztof Kozlowski 105 1.0% David Howells 97 0.9% Russell King 94 0.9% Darrick J. Wong 93 0.8% Jakub Kicinski 91 0.8% Christoph Hellwig 84 0.8% Dr. David Alan Gilbert 80 0.7% Eric Biggers 72 0.7% Mario Limonciello 68 0.6% Eric Dumazet 67 0.6% Christian Brauner 66 0.6% SeongJae Park 60 0.5% Steven Rostedt 58 0.5% Claudiu Beznea 58 0.5% Ian Rogers 57 0.5%

By changed lines Richard Fitzgerald 17946 3.5% Darrick J. Wong 10807 2.1% Eric Biggers 10622 2.1% Karan Tilak Kumar 10174 2.0% Lizhi Hou 9252 1.8% Kent Overstreet 9075 1.8% David Howells 8366 1.6% Taniya Das 7181 1.4% Dmitry Baryshkov 6827 1.3% Patrick Rudolph 6089 1.2% Charlie Jenkins 5961 1.2% Even Xu 5169 1.0% Jani Nikula 4965 1.0% Melody Olvera 4841 0.9% Jie Gan 4781 0.9% Krzysztof Kozlowski 4769 0.9% Nuno Das Neves 4579 0.9% Piotr Kwapulinski 4460 0.9% SeongJae Park 4306 0.8% Lukas Bulwahn 4256 0.8%

The developer with the most changesets this time around is Kent Overstreet, who merged the usual set of fixes and improvements to the bcachefs filesystem; that count was increased by the fact that Overstreet missed the 6.13 development cycle. Thomas Weißschuh contributed fixes and hardening patches throughout the kernel. Jani Nikula worked extensively on the Intel i915 GPU driver, Sean Christopherson continued a long-running series of KVM improvements, and Dmitry Baryshkov contributed a set of graphics and devicetree changes.

Looking at the "lines changed" column, Richard Fitzgerald added a set of unit tests for the Cirrus firmware drivers. Darrick Wong added a long list of improvements to the XFS filesystem. Eric Biggers did some extensive refactoring in the crypto subsystem and removed some unused drivers. Karan Tilak Kumar contributed extensively to the Cisco FNIC driver, and Lizhi Hou added a new driver for the AMD "amdxdna" AI engine.

The percentages of patches with Reviewed-by tags dropped a little from 6.13 (to 49.2%) in this release, but the number with Tested-by tags increased to 9.3%. The top reviewers and testers this time around were:

Test and review credits in 6.14
Tested-by Daniel Wheeler 113 9.6% Mark Pearson 65 5.5% Arnaldo Carvalho de Melo 40 3.4% Sergey Senozhatsky 24 2.0% Aaron Ma 22 1.9% Rui Zhang 21 1.8% Manivannan Sadhasivam 18 1.5% Fuad Tabba 18 1.5% Choong Yong Liang 18 1.5% James Clark 17 1.4% Rick Wertenbroek 16 1.4% Dirk Behme 16 1.4% Johan Hovold 15 1.3% Antony Antony 15 1.3%	Reviewed-by Christoph Hellwig 185 2.5% Ilpo Järvinen 152 2.1% Dmitry Baryshkov 134 1.8% Konrad Dybcio 124 1.7% Krzysztof Kozlowski 121 1.6% Jeff Layton 117 1.6% Andrew Lunn 113 1.5% Simon Horman 112 1.5% Geert Uytterhoeven 108 1.5% AngeloGioacchino Del Regno 104 1.4% Mario Limonciello 83 1.1% Mark Pearson 71 1.0% Johannes Thumshirn 69 0.9% Jani Nikula 68 0.9%

Daniel Wheeler has been the top credited tester for every kernel release since 6.3; if one discounts a disappointing second-place finish in 6.2, though, he has actually held that position since the 5.12 release in 2021. Christoph Hellwig routinely shows up at the top of the Reviewed-by column. Curious LWN subscribers can learn more about the review, test, and bug-reporting credits for this release on this KSDB page.

Work on 6.14 was supported by 220 employers that we know of — a few more than contributed to 6.13. The most active employers supporting 6.14 were:

Most active 6.14 employers

By changesets Intel 1177 10.7% (Unknown) 1049 9.5% Google 959 8.7% (None) 658 6.0% Red Hat 620 5.6% AMD 583 5.3% Linaro 471 4.3% Qualcomm 465 4.2% Meta 346 3.1% Oracle 327 3.0% SUSE 231 2.1% Renesas Electronics 221 2.0% IBM 213 1.9% NVIDIA 208 1.9% Huawei Technologies 187 1.7% (Consultant) 185 1.7% Amadeus IT GmbH 167 1.5% NXP Semiconductors 155 1.4% Microsoft 153 1.4% Arm 139 1.3%

By lines changed Intel 46050 8.9% Google 44221 8.6% (Unknown) 43909 8.5% Qualcomm 43055 8.4% AMD 37831 7.3% Red Hat 30166 5.9% (None) 25576 5.0% Cirrus Logic 18639 3.6% Linaro 18412 3.6% Oracle 15375 3.0% Meta 14161 2.7% Cisco 10296 2.0% Microsoft 9719 1.9% NVIDIA 9452 1.8% Bootlin 7888 1.5% Microchip Technology Inc. 7534 1.5% IBM 6872 1.3% NXP Semiconductors 6770 1.3% 9Elements 6099 1.2% Rivos 6007 1.2%

As usual, there are few surprises here. It is worth noting that contributions from developers working on their own time (and those with unknown employers, many of whom will also be volunteers) is up slightly over 6.13. That may be normal variation, but it may also reflect the holidays, where volunteers may have found more time to write patches while developers working on the kernel as their job took time off.

Whether the holidays are the real reason for the relative slowness of 6.14 is not entirely clear, though. As of this writing, there are just over 12,100 non-merge changesets sitting in linux-next, most of which will be destined for the 6.15 kernel. That number exceeds the approximately 9,000 that were staged for 6.14 by a significant margin, suggesting that 6.15 will be a busier release, more in line with recent history.

Comments (none posted)

The 2025 Linux Storage, Filesystem, Memory-Management, and BPF Summit

The Linux Storage, Filesystem, Memory-Management, and BPF Summit is an annual, invitation-only event where about 140 developers gather to address core-kernel problems. The 2025 event was held March 24 to 26 in Montreal, Canada. As usual, LWN was there to report on the discussions that were held.

Reporting on those discussions has been done over the last few months and is now complete:

Storage, Filesystem, and Memory-Management sessions

• Updates on storage standards: a look at progress in the standards for NVMe, SCSI, and ATA.
• The state of the page in 2025: the annual update on how the transition to folios is going.
• The future of ZONE_DEVICE: some thoughts on how to bring an "ugly stepchild" of the memory-management subsystem a bit closer to the fold.
• A process for handling Rust code in the core kernel: introducing a new taxonomy of approaches to Rust code, and how to make the merging of that code work better.
• Device-initiated I/O: a discussion on providing a mechanism for NVMe devices to go beyond peer-to-peer DMA and actually be able to initiate I/O operations themselves.

Storage and Filesystem sessions

• Supporting untorn buffered writes: untorn (or atomic) writes are now supported in the kernel for direct I/O, should that support be extended to buffered I/O?
• An update on torn-write protection: changes and plans for untorn direct I/O writes for XFS and other filesystems.
• Atomic writes for ext4: supporting atomic (or untorn) writes on the ext4 filesystem.
• Flexible data placement: a discussion on providing hints to SSDs to improve performance and endurance.
• A new DMA-mapping API: a new API for DMA in the kernel has stalled and its developers are trying to figure out the right path forward.
• Zero-copy for FUSE: a discussion on ways to eliminate data copies for user-space filesystems.
• Improving iov_iter: kernel buffer-handling data structures need improvement, though the path forward is not clear.
• Parallelizing filesystem writeback: a discussion on handling writeback with multiple threads.
• Supporting NFS v4.2 WRITE_SAME: adding support for an NFS command that is used to write bulk data to the server is discussed.

Filesystem and memory-management sessions

• Improving FUSE writeback performance: finding a way to eliminate expensive allocation and copying operations when doing writeback for FUSE filesystems.
• A FUSE implementation for famfs: the famfs filesystem, which targets CXL and other disaggregated memory as a backing store, has been ported to FUSE.

Filesystem sessions

• Topics from the virtual filesystem layer: a handful of issues in the VFS to discuss—or at least introduce.
• Parallel directory operations: finding a way to handle multiple file operations in a directory in parallel.
• Tracepoints for the VFS?: a discussion on possibly adding tracepoints to the virtual filesystem (VFS) layer.
• VFS write barriers: a feature proposed for the VFS that would give user space the ability to recognize (and log) filesystem changes before they happen; it would allow the creation of a filesystem change journal, similar to what NTFS provides.
• Freezing filesystems for suspend: the system suspend/resume path lacks the ability to freeze and thaw filesystems; what would it take to add that functionality?
• Long-duration stress-testing for filesystems: a discussion on filesystem testing aimed at finding more bugs before they are discovered in a production setting.
• Getting Lustre upstream: a discussion on what is needed to get the Lustre filesystem into the kernel.
• A distributed filesystem for archival systems: ngnfs: a new filesystem, ngnfs, was introduced; it is used for large data sets where the files are stored offline.

Memory-management track sessions

• A herd of migration discussions: the ability to move folios around in physical memory is crucial for system performance, but that ability cannot always be taken for granted. Three sessions discussed migration and how to make it faster and more reliable.
• Improving the merging of anonymous VMAs: properly managing adjacent memory areas improves performance and reliability, but is not always easily done.
• Memory persistence over kexec: part of the thorny problem of updating the kernel on a cloud server without disturbing the guests running there.
• Slab allocator: sheaves and any-context allocations: a pair of slab-allocator discussions focusing on the new "sheaves" caching layer and ways of allocating memory safely in any execution context.
• Approaches to reducing TLB pressure: making better use of the CPU's translation lookaside buffer is often the key to better performance; what can the kernel do to help?
• Better hugetlb page-table walking: a discussion on making hugetlb look a bit more like the rest of the memory-management subsystem.
• Page allocation for address-space isolation: unmapping sensitive kernel data whenever possible has the potential to increase security, but it will place some interesting demands on the memory-management subsystem.
• The state of guest_memfd: protecting guest pages from the host can be a difficult business.
• Three ways to rework the swap subsystem: swapping in Linux is a complex business; three sessions look at ways to both simplify the swap subsystem and make it more effective.
• Per-CPU memory for user space: per-CPU variables are an effective performance optimization in the kernel; how can similar benefits be brought to user space?
• Using large folios for text areas: the kernel's readahead code works well for data, but tends to break up executable code (text) into small chunks. Some minor changes could improve that situation.
• Two approaches to better kernel samepage merging: a pair of sessions on how the system can do more focused and synchronous deduplication of memory.
• Improving hot-page detection and promotion: the ongoing challenge of figuring out which memory is in the most active use at any given time.
• Preparing DAMON for future memory-management problems: an update on the DAMON monitoring subsystem and what is likely to happen next.
• Management of volatile CXL devices: CXL memory stresses the kernel's view of hardware in a number of interesting ways.
• Managing multiple sources of page-hotness data: more data about data-access frequency is good, but how can the kernel best make use of it?
• The state of the memory-management development process, 2025 edition: the annual look at how the development community is doing and what can be improved.
• Automatic tuning for weighted interleaving: how can weighted interleaving be made to just work on dynamic systems?
• Improvements for the contiguous memory allocator: two sessions on better ways to ensure the success of large, physically contiguous allocations in the kernel.
• Memory controller performance improvements: progress in the perennial challenge of reducing the overhead of the memory controller.
• The problem of unnecessary readahead: the concluding session in the memory-management track looked at preventing the kernel from reading data that nobody will use.

BPF track sessions

• An update on GCC BPF support: GCC has been slowly catching up to Clang for compiling to BPF. This session goes into the details of that work.
• A strange BPF error message: Yonghong Song explains how undefined behavior can turn a simple uninitialized variable into a puzzling error from the BPF verifier.
• An update on pahole: Arnaldo Carvalho de Melo gives an update on how development of the pahole debugging-information manipulation tool has gone over the past year.
• A new type of spinlock for the BPF subsystem: Deadlocks are a recurring problem for BPF, one that complicates the handling of locks in BPF programs. A new lock design may help prevent them.
• Inlining kfuncs into BPF programs: A way to make calling select kernel functions from BPF programs more efficient.
• In search of a stable BPF verifier: Two approaches to supporting BPF programs across multiple stable kernels.
• Taking BPF programs beyond one-million instructions: Alexei Starovoitov explains how to write BPF programs that take advantage of improvements in the verifier to handle large loops.
• Indirect calls in BPF: Anton Protopopov is continuing to work to enable jump tables, function pointers, and static keys in BPF.
• Code signing for BPF programs: Blaise Boscaccy and Cong Wang present two different visions for how BPF programs could be authenticated.
• Inline socket-local storage for BPF: BPF developers discuss changing the layout of BPF map information in the networking subsystem to reduce cache misses.
• Better debugging information for inline kernel functions: Selectively inlined functions can be difficult to debug; perhaps enriching the kernel's BTF debugging information could help.
• Filtering fanotify events with BPF: a joint session on improving filesystem access monitoring in the kernel.
• Hash table memory usage and a BPF interpreter bug: Anton Protopopov presents a plan to reduce BPF map memory usage.
• A look at what's possible with BPF arenas: a report on real-world use of BPF arenas with some feedback on the feature and thoughts on possible enhancements.
• Formally verifying the BPF verifier: a look at using Agni to prove parts of the BPF verifier correct.
• Verifying the BPF verifier's path-exploration logic: Srinivas Narayana shares a plan to extend Agni to cover more of the verifier.
• Allowing BPF programs more access to the network: new functions to allow BPF programs to send data over the network, and cleanly disconnect TCP connections.
• Two sessions on faster networking: Cong Wang and Daniel Borkmann each present ways to remove unnecessary networking operations.

Group photo

The group assembled at the end of the first day for this photo:

Support

Once again, we thank the Linux Foundation, LWN's travel sponsor, for supporting our travel to this event.

Comments (2 posted)

The state of the page in 2025

The folio transition is one of the most fundamental kernel changes ever made; it can be thought of as being similar to replacing the foundation of a building while it remains open for business. So it is not surprising that, for some years, the annual Linux Storage, Filesystem, Memory-Management, and BPF Summit has included a session on the state of this transition. The 2025 Summit was no exception, with Matthew Wilcox updating the group on what has been accomplished, what remains to be done, and where some of the significant problems are.

The initial idea behind folios, Wilcox began, was to manage pages in larger blocks; the experience of the last few years shows that it works. Later on, the goal of shrinking the page structure, which represents a single page in memory was added. Even later came objectives like enabling filesystems with block sizes larger than the page size and improving the debugability and clarity of the memory-management subsystem. There has been the accumulation of a lot of cruft in that subsystem over the years, he said; the folio transition is an opportunity to clean some of it out.

There are two understandings of what a folio is. The first, which he called the "Ottawa interpretation", is what he intended initially; it was, in essence, just the head page of a compound page. Over time, though, the conception of folios has shifted toward the "New York interpretation", much of which is the work of Johannes Weiner. In that view, folios are an opportunity to shrink struct page to a single u64 memory descriptor. Progress is being made toward that goal, but it will not be achieved this year.

Since a folio is an independent structure, it can grow as needed. The size of struct page, instead, is strictly constrained; since there is one per page, it must be as small as possible. Even though struct folio is getting larger, there will be a lot fewer of them, so the overall memory-management overhead will decrease. Getting to the point where struct page can be replaced will require quite a bit of work, still.

In 2025, the objective is to get to the point where struct folio is indeed a separate structure from struct page and can be allocated independently. Then, data can be removed from struct page, shrinking it, but not yet all the way.

Wilcox noted that he is getting tired of converting filesystems to folios, which is a necessary step on the path to a smaller page structure. So he is considering adding a new kernel configuration option, CONFIG_SEPARATE_FOLIO, that would compile out any code that is not yet prepared for a separate folio structure. That would allow the creation of a kernel where the separate-folio changes can be tested, even if it isn't yet a kernel that supports all of the important features (networking, say) that users might actually want.

The upcoming work involves removing all references to a number of struct page fields, including mapping, index, and lru. The networking subsystem's pagepool mechanism will need to be separated out. There is also the page pointer in struct folio, which points to the underlying page structure, and a lot of "more interesting" casts scattered through the kernel code that perform a similar function; those will have to be fixed. For example, the buffer_head structure has page and folio pointers in a single union and will need to be fixed. Another near-future change will be adapting the slab allocator to use the separate slab structure.

Wilcox then reviewed some of the goals he had covered in the 2024 update. Many of them remained incomplete; in his defense, he said, it had only been ten months since last year's conference. Given those two months — essentially one whole development cycle — he would have checked off more of them.

Some things were definitely accomplished, though. The zpdesc memory descriptor was added to replace struct page use in the zswap subsystem. It is currently an overlay on struct folio, but Wilcox thinks it could be made more space-efficient. The ability to use a filesystem block size larger than the page size has been "quite the journey", but that ability now exists for the XFS and bcachefs filesystems. It should be easy to add to other filesystems as well — at least, once those filesystems are able to support large folios.

Another achievement is the ability to allocate and free frozen pages, which have no reference count. The adoption of this feature, he said, is "borrowing pain from the future". Recently, the network stack found this pain; happily, this problem turned up and was fixed before the 6.14 release. Wilcox would like to see more use of frozen pages in general, but he pointed out that there will always be some places where reference counts will be needed.

Another important step forward is imprecise mapping-count tracking for large folios, which changes the kernel to track the number of processes mapping a folio, rather than the number of mappings, once the number of processes exceeds two. This work enables precise tracking for the common cases while maintaining correctness in the more exotic cases.

It is now possible to create large folios in generic_perform_write(), which, he said, is a big deal; it can double write performance in some tests. That result wasn't surprising, he said, since using large folios frees the kernel from having to manage large numbers of base pages. Meanwhile, the bh_page pointer in buffer heads is now unused, all that remains is to actually delete it. There has also been a lot of work removing the wrapper functions around the various page flags, further reducing the role of struct page.

There is currently some use of page types, which will eventually be stored in the memory descriptor. The type is stored in the page structure, but it is overlaid by the mapcount field, so it cannot be used in mapped pages. There is some trickery being used to distinguish mapping counts (positive numbers) from the page types, which are indicated by a negative mapping count. Various types of pages, including hugetlb, slab, zsmalloc, unaccepted, and large-kmalloc pages, are identified by page types now.

Page flags have long been in short supply; there is exactly one of them available at the moment. The PG_slab flag is gone now, as is PG_error, which turned out to just be the inverse of PG_uptodate. Those changes freed two flags, but then PG_dropbehind (which might be renamed PG_reclaim) was added. The PG_uncached flag is now an alias for PG_arch_2, while PG_mappedtodisk overlays PG_owner_2. The PG_private_2 flag is almost unused, but the Ceph and NFS filesystems still need it. PG_private needs more work before it can be removed, since it is used for a lot of different things in different places. Often it is used to indicate that there is something stored in the private field of struct page, but a test for NULL should be usable instead (some members of the group disagreed with that claim). Most of the existing page flags will eventually become folio flags, he said, while PG_hwpoison will become a page type.

Wilcox concluded with some suggestions for anybody who wants to help with the folio transition. At the top of his list was working to make more filesystems support large folios; that, he said, is good for the system as a whole. The "bitmap" MD target needs to stop using buffer heads, but he hasn't looked at MD in many years and is afraid of it. Removing any uses of the page member of struct folio and the lru page field would also be useful.

There have been many developers who have helped with this work so far, and he thanked them all. It has been a fun project; he was looking forward, he said in jest, to next year's Summit where he will be able to say that it is complete.

Comments (none posted)