Leading items

Welcome to the LWN.net Weekly Edition for October 3, 2024

This edition contains the following feature content:

• The WordPress mess: conflicting business interests lead to serious uncertainty for WordPress users.
• Sched_ext at LPC 2024: the launch of a new kernel-development community focused on CPU-scheduler development.
• The rest of the 6.12 merge window: more changes for the next kernel release.
• More Kangrejos coverage:
  • Getting PCI driver abstractions upstream: What is required to have a completely safe PCI driver?
  • Coccinelle for Rust: next steps toward being able to use Coccinelle with Rust code.
  • BTF, Rust, and the kernel toolchain: how evolutions in BTF could impact the Rust-for-Linux project.
• Debian changes OpenSSH packaging: splitting some functionality out of the core Debian OpenSSH is not as easy as it seems.
• An update on gccrs development: progress toward a second compiler for the Rust language.
• Inkscape 1.4 coming soon: what to expect in the next release of this vector-drawing application.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

The WordPress mess

WordPress is the world's most popular open‑source blogging and content‑management platform. In its 20‑plus years of existence, WordPress has been something of a poster child for open source, similar to Linux and Firefox. It introduced the concept of open source to millions of bloggers, small‑business owners, and others who have deployed WordPress to support their web‑publishing needs. Unfortunately, it is now in the spotlight due to an increasingly ugly dispute between two companies, Automattic and WP Engine, that has spilled over into the WordPress community.

Background

WordPress is a PHP‑based, GPLv2-licensed, content‑management system (CMS). It was forked from b2 by Matt Mullenweg and Mike Little in 2003, after b2's development stalled. As blogging became mainstream, WordPress was the tool of choice for many aspiring bloggers due to its ease of use and the fact that it was free as in beer and speech.

In 2005, Mullenweg founded Automattic, which started out offering WordPress hosting via WordPress.com and comment-spam protection via the Akismet service. Its portfolio of services and holdings (including the Tumblr micro‑blogging platform) has grown substantially since then; it includes the WooCommerce online-store platform, Gravatar web-profile service, Newspack news-publishing platform, and others. WordPress hosting, however, remains at the core of its business.

Foundation and trademark

Automattic's press page lists 11 investors in the company and describes it as a "later‑stage growth phase" company. In 2010, Mullenweg started the non‑profit WordPress Foundation to manage WordPress development and hold the WordPress trademarks. The open‑source project is often referred to as WordPress.org to distinguish it from Automattic's WordPress.com hosting business. When the marks were transferred, Mullenweg wrote that "the most central piece of WordPress's identity, its name, is now fully independent from any company".

The reality is that the name is still under the control of Automattic, and thus still under Mullenweg's control. The WordPress Foundation trademark policy says that any commercial use of the WordPress trademark is controlled by Automattic, which has an exclusive license. The foundation only controls non-commercial uses, and the only active member of the foundation's board seems to be Mullenweg. According to the most recent tax filing by the foundation, there were only three board members in 2022: Mullenweg, Mark Ghosh, and Chele Chiavacci Farley.

One might assume that the foundation is involved in running WordPress.org, but it is not. Donations to the foundation primarily support events and education about WordPress. Its financials for 2023 detail expenditures for events and publishing videos on WordPress.tv. Individuals and organizations that want to support the WordPress project itself are encouraged to contribute resources to "Five for the Future". This program was introduced in 2014 by Mullenweg for others in the WordPress ecosystem to help "grow the WordPress pie".

The dispute

The other major player in this drama is WP Engine. That company was founded in 2010 by Jason Cohen and Cullen Wilson, who described themselves as "power users" who wanted to provide hosting by WordPress experts. The company now bills its offerings as the "most trusted WordPress platform". Like Automattic, WP Engine's business is based on hosting and supporting WordPress. In 2018, WP Engine took funding from private-equity firm Silver Lake. The upshot is that both companies compete for the same customers, have substantially similar offerings, and have investors to please.

On September 20, at WordCamp US 2024, Mullenweg used his talk to air some grievances against WP Engine and Silver Lake. He followed up the next day with a blog post titled "WP Engine is not WordPress" that criticized WP Engine's lack of contributions to WordPress, and its disabling of WordPress's revisions feature that saves each saved draft or published version of a post or page. A standard WordPress installation will have no limit on the number of revisions stored, or how long the revisions will be stored. Users can compare revisions to see what has changed, and restore an older revision if desired. WP Engine turns this feature off by default, though users can contact support to turn on as many as five revisions that will be stored up to 60 days.

Mullenweg described the feature as being a core piece of the promise to protect user data, and claimed that WP Engine wanted to turn off revisions to avoid paying for storage. "They are strip-mining the WordPress ecosystem, giving our users a crappier experience so they can make more money."

Mullenweg claimed that the companies enjoyed similar revenue, but complained that WP Engine's contributions to the WordPress project averaged 40 hours per week compared to Automattic's nearly 4,000 hours per week. He had choice words for WP Engine, calling it a "cancer to WordPress" and encouraged customers to "consider literally any other WordPress host". He argued that because of WP Engine's modifications, customers were not getting WordPress. "And they're profiting off of the confusion. WP Engine needs a trademark license to continue their business."

The foundation's policy page used to say that the WP abbreviation "is not covered by the WordPress trademarks and you are free to use it in any way you see fit". It has been recently updated, sometime after September 19, to say that the abbreviation is not covered, "but please don't use it in a way that confuses people":

For example, many people think WP Engine is "WordPress Engine" and officially associated with WordPress, which it's not. They have never once even donated to the WordPress Foundation, despite making billions of revenue on top of WordPress.

Mullenweg's blog post focuses heavily on making the point that WP Engine's offering is "not WordPress" because it turns off revisions—but the trademark policy is silent on what modifications are or are not permitted to describe something as WordPress. It should be noted at this point that WordPress.com does not offer unadulterated WordPress to all of its customers either. Another core feature of WordPress, the ability to install third‑party plugins and themes, is restricted to customers who pay for the Business tier of hosting or above (about $300 a year).

Stop it

On September 23, WP Engine sent a cease-and-desist letter to Automattic, which it also made public. In the letter, the company claimed that Mullenweg had "suddenly began demanding that WP Engine pay Automattic large sums of money", or he would "go 'nuclear' on WP Engine" during the keynote at WordCamp on September 20. The letter includes what are alleged to be text messages from Mullenweg that describe the way he will attack WP Engine and Silver Lake during the keynote, unless the company was willing to negotiate a payment of a percentage of its revenues for use of the trademark.

Automattic also sent (and published) its own cease-and-desist letter the same day, that demanded WP Engine stop using an assortment of trademarks, including WordPress. It demanded an accounting of profits from use of service offerings that were marketed under the trademarks, and compensation for unauthorized use. As examples of infringement it specifically called out "many references to WP Engine being 'WordPress Engine'". The letter was also accompanied by pages of examples highlighting instances of the phrase "WordPress Engine" as well as social-media posts where apparently confused users ask questions about WordPress Engine.

Automattic's cease-and-desist put a tentative figure on what the company was seeking from WP Engine: a "mere 8% royalty" on WP Engine's estimated $400 million in annual revenue, or more than $32 million per year in "lost revenue". It left open the potential for amicably resolving the matter "including through a licensing relationship for use of its trademarks".

In a post about the dispute on his personal blog, Mullenweg said that Automattic had been trying to make a licensing deal with WP Engine for a long time, but had been strung along. It was not a money grab, he said, but an attempt to make WP Engine pay its fair share, and that the company had been offered the options of "either pay a direct licensing fee, or make in‑kind contributions to the open source project".

Escalation

Because Mullenweg posted the "WP Engine is not WordPress" post to WordPress.org's news page, it also showed up in the WordPress administrative dashboard in a "news and events" widget. That meant the message was immediately visible to WP Engine customers if they happened to have that widget enabled and were using the dashboard. WP Engine apparently removed the widget from its hosted customers' WordPress installs on September 24. On September 25, Mullenweg posted on WordPress.org announcing that WP Engine was "banned from WordPress.org", meaning that users on WP Engine could no longer use built‑in WordPress update features and more:

WP Engine wants to control your WordPress experience, they need to run their own user login system, update servers, plugin directory, theme directory, pattern directory, block directory, translations, photo directory, job board, meetups, conferences, bug tracker, forums, Slack, Ping‑o‑matic, and showcase. Their servers can no longer access our servers for free.

Why now?

The obvious question on many people's minds is "why now?", which was not really addressed in any of the public statements. Mullenweg discussed the timing in a live-streamed interview by Michael B. Paulson on September 26. Paulson asked "at what point did this become something that must be addressed, and in this way?" Mullenweg responded that other companies in the WordPress ecosystem contribute or pay for trademark agreements "even though they make way less revenue" than WP Engine:

They kept saying they were going to do something for years, and there's a lot of friendly people there [...] I guess I didn't realize how, like, capricious and evil private equity could be, so that is totally my fault and I apologize for that.

Paulson followed up and said that, "as an outsider" it was unclear why this happened now and not a year ago or another point in time. Mullenweg said that he had been "trying to connect" with WP Engine for years but the company had been unwilling to negotiate: "they were just like, nope, not anything, you know? Let's delay, delay, delay". Mullenweg said that WP Engine had "tried to make it look like I was extorting them last minute" but he had wanted to avoid the fight. "In fact, they could sign a trademark license today and this would all go away".

Another bone of contention that Mullenweg said he had failed to communicate to the community was over transactions made via Stripe with the WooCommerce plugin. Mullenweg said that WP Engine had "hacked" the plugin to change the attribution code so that "tens of millions of dollars" in fees go to WP Engine rather than Automattic. Note that the plugin is licensed under the GPLv3, and the information page makes no mention of any referral codes or expectations that the referral codes should remain unchanged. The privacy policy that is linked from the WooCommerce information page mentions several reasons that information is collected, but none of them address affiliate fees or attribution codes that generate revenue for Automattic.

Reactions

The ban of WP Engine from using WordPress.org services has sent WordPress users and developers scrambling. Cullen Whitmore, a WordPress contributor and owner of an agency that provides WordPress services, wrote an open letter that made an appeal to Mullenweg to remove the ban on WP Engine and provide a timeline to reach a deal. The only brand confusion, he said, is for new users confused about where to get started with WordPress, WordPress.com or WordPress.org. He also defended WP Engine's contributions, saying that it contributes back "through other methods", such as supporting users and developers in adopting and implementing WordPress features.

He found the ban troubling, but he said "my biggest concern was raised by your response to a user in the Make WordPress Slack". (The conversation is here but requires creating or having a WordPress.org account and then signing up for the Slack channel.) A user asked what they should tell their customers who need updates. Mullenweg told the user to contact WP Engine and "be mad at the person you're paying". He added that WP Engine "can fork WordPress, too, and publish their own version. I'd love that". Mullenweg, Whitmore said, should "rethink your position on speaking about those caught in the middle. You didn't sound like a leader here".

User "FeralRobots" on mastodon.social said that, without an easy way to specify alternative repositories for WordPress, "Matt Mullenweg is basically a living supply chain vulnerability in the WordPress ecosystem".

Part of Automattic's urgency in trying to force WP Engine to the table may be the increasing commoditization of WordPress hosting as a business. Eric Mann, a "power user" of WordPress who used to work with it professionally, wrote that all of the companies exhibiting at WordCamp US were "commodity plays". Every one, he said, claims to be the fastest or most innovative, but "I can't tell you what makes them different other than the color of the socks they were giving away as swag". He later wrote that Mullenweg had raised "valid concerns" around "WP Engine's failure to contribute back to the open source community". Mann also blamed WP Engine for failing to have protections in place "for an unexpected outage of WordPress.org API services".

The value of infrastructure

Brian Coords, who is a technology director at a WordPress development agency, wrote on September 27 that "Matt's intentions in waging a battle of FOSS vs PE [private equity] are good, but to be clear: I don't agree with his tactics." (Emphasis his.) He said he never expected Mullenweg to "compromise individual WordPress installations and act in such a user-hostile way" but that what made WordPress successful was more than the code, it was "the seamless distribution of core, theme, and plugin updates that are hosted and deployed by WordPress.org". If users still had to manually upload PHP files and SFTP into servers, as they had to do in the early days, the project wouldn't be nearly as successful. That distribution network, he said, is subsidized by Automattic and is "Matt's to do what he pleases with it":

If you've been relying on one-click core updates and the plugin directory to manage the distribution of open source plugins to/from websites (which of course we all have), then you have been playing in Matt's sandbox. This sandbox is not "open source WordPress" or the WordPress Foundation. It is WordPress.org, which is a wonderful thing that Matt Mullenweg owns and has stewarded for the last two decades. But it is not "ours".

Paolo Belcastro, an Automattic employee who has been with the company since 2011, wrote that the costs of maintaining WordPress.org and its infrastructure are "tens of millions of dollars every year", which is sponsored by Mullenweg personally and Automattic. He notes that WP Engine actually charges customers for the automatic updates ($3 per month) that are provided free to others via WordPress.org.

A brief reprieve

On September 27, Mullenweg announced a reprieve for WP Engine until October 1 to allow the company to "spin up their mirrors of all of WordPress.org's resources that they were using for free while not paying". He blamed WP Engine and Silver Lake for the impact on customers:

I've heard from WP Engine customers that they are frustrated that WP Engine hasn't been able to make updates, plugin directory, theme directory, and Openverse work on their sites. It saddens me that they've been negatively impacted by Silver Lake's commercial decisions.

WP Engine was well aware that we could remove access when they chose to ignore our efforts to resolve our differences and enter into a commercial licensing agreement. [...] You could assume that WPE has a workaround ready, or they were simply reckless in supporting their customers. Silver Lake and WP Engine put their customers at risk, not me.

Calls for change

Joost de Valk wrote that he would like to see "big WordPress companies" contribute, but there was a need for transparency and community-based governance. He had founded Yoast, a company that provides search-engine optimization (SEO) tools and plugins for WordPress (as well as other platforms). He wrote that if there are requirements to contribute, "we can call that taxation", and that should come with representation:

In my opinion, we all should get a say in how we spend those contributions. I understand that core contributors are very important, but so are the organizers of our (flagship) events, the leadership of hosting companies, etc. We need to find a way to have a group of people who represent the community and the contributing corporations.

Just like in a democracy. Because, after all, isn't WordPress all about democratizing?

Josh Collinsworth, a front-end developer who has worked at WP Engine in the past, called for Mullenweg to be removed. He said that Mullenweg "has, for far too long, enjoyed unchecked powers at the top of WordPress" and that those powers were "all too often a direct and flagrant conflict of interest". Collinsworth said that he was not defending his former employer, and he would happily discuss "how much WP Engine sucks" but that "the ability to block an entire hosting provider from accessing the plugins repository is a power that nobody should have":

I don't care about Automattic giving 5% to WordPress anymore. I want it to give up Matt's unchecked, unilateral power. Because it's clearer than ever he can't be trusted with it.

Uncertainty continues

The tit‑for‑tat between Automattic and WP Engine may continue for some time, but the impact of the conflict has already sent tremors throughout the WordPress community. Users on Mastodon, Twitter, and other social-media sites are voicing complaints with the way that Mullenweg has handled the situation and the uncertainty it has caused. It seems likely that "are we next?" is a popular topic of conversation in the hallways (or Slack channels...) at other companies providing WordPress hosting and services.

No matter how long the two companies have been negotiating behind the scenes, this incident came as a surprise to far too many WordPress users caught in the middle. And, if one takes a moment to think about Mullenweg's argument that "WP Engine isn't WordPress" for lack of the revisions feature, it's unclear how paying Automattic a licensing fee for use of the name is going to make anything better for anyone but Automattic. Presumably, if WP Engine forks over a big enough pile of cash, Automattic is willing to let the company continue to skimp on revisions, and contributions.

It is apparent that the foundation is simply a fig leaf to allow Automattic continued control of the WordPress marks, while giving the appearance of independence. There is no benefit to the WordPress community to have a non-profit holding the marks if the entity simply boomerangs the exclusive rights back to the for‑profit that turned them over in the first place.

As Coords wrote, the infrastructure around WordPress.org is wholly controlled by Mullenweg and funded by Automattic. That is a burden that shouldn't be carried by one person or company, and it gives too much power over the larger community. A week ago that argument was a hypothetical, but Mullenweg has demonstrated willingness to wield that power unilaterally, without warning, and with no checks or balances aside from public outcry. The arrangement that places Automattic as the sole benefactor providing hosting and other services for WordPress.org, and Mullenweg as the sole decider about its policies, is a weak point for the larger community and ecosystem around WordPress.

The WordPress mission is to "democratize publishing". The project has enabled millions of users and organizations to easily publish content to the web. I include myself as one of them, since I've been using WordPress (off and on) for almost its entire history. The project itself, unfortunately, seems to be leaning toward the autocratic. One can find WP Engine's lack of contributions to be problematic while still being concerned that Mullenweg has too much power over a project—and its infrastructure—that powers a significant portion of the world's web sites.

Comments (40 posted)

Sched_ext at LPC 2024

The extensible scheduler class (sched_ext) enables the implementation of CPU schedulers as a set of BPF programs loaded from user space; it first hit the mailing lists in late 2022. Sched_ext has engendered its share of controversy since, but is currently slated to be part of the 6.12 kernel release. At the 2024 Linux Plumbers Conference, the growing sched_ext community held one of its first public gatherings; sched_ext would appear to have launched a new burst of creativity in scheduler design.

An overview

The sched_ext microconference began with Tejun Heo, one of the authors of this new subsystem. He introduced sched_ext as a new scheduling class that sits in the hierarchy along with the EEVDF and realtime schedulers. It serves as a bridge between the scheduling core and the BPF virtual machine, where all of the interesting decisions are made. BPF maps are used as a high-performance interface to user space.

Work on sched_ext is proceeding on several fronts, starting with the merge of the sched_ext core into the mainline kernel, which was still pending at the time of this talk. Basic scheduling is part of that and works now. There are ongoing efforts to support features like CPU-frequency scaling, CPU hotplug, and control groups (basic support for which landed in 6.12); those can be expected to be added in future mainline kernel releases.

There is a repository available with the current sched_ext work, including a number of example schedulers. The scx_lavd scheduler, for example, is focused on interactivity and, specifically, consistently getting higher frame rates out of games. The scx_bpfland scheduler, instead, is aimed at minimizing response times. There is scx_rustland, which simply forwards scheduling events to user space, where the decisions are made. Heo admitted that he had been skeptical of that idea at the outset, but it has turned out to be "quite usable". The repository also contains scx_rusty for load balancing on complex CPU topologies, and scx_layered, which is a partitioning scheduler.

One of the best things about sched_ext, he said, is that it cannot crash the machine. All of the usual BPF safety checks apply here. Additionally, if the kernel detects a scheduling problem, it will simply revert the system to the EEVDF scheduler and life goes on. That makes experimenting easy and the development cycle short.

It is, he said, still the early days of sched_ext development, and he is focused on getting some practical wins. One of those appears to be scx_lavd (about which more was heard later), which is headed for shipment in Steam Deck gaming systems. scx_bpfland is showing promising results for personal machines, while scx_layered has been deployed in over one million machines and is delivering significant performance gains.

The sched_ext developers are also working on building the development community. Support for sched_ext is now shipping in a number of distributions, including CachyOS, Arch Linux, Ubuntu, Fedora, Nix, and openSUSE. On those distributions, running a new scheduler is just a matter of installing a package and running a program.

Work in the future, Heo said in conclusion, is focused on composability — making it possible for multiple schedulers to work together. That will allow different developers to focus on independent layers; one scheduler could be concerned with time slices, while another would focus on load balancing or partitioning. The plan is also to eventually allow the stacking of schedulers down the control-group hierarchy, so that different schedulers at each level could handle a part of the overall scheduling problem.

User-space scheduling

While sched_ext is meant to put CPU-scheduling decisions into users' hands, it was still expected that those decisions would be made by BPF programs running within the kernel. So Andrea Righi's scx_rustland scheduler, which defers all of those decisions to user space, came as a bit of a surprise. Righi started his session by saying that scx_rustland began as just a fun project, with no expectation that something useful would result. He mostly wanted better observability of scheduling decisions and a faster development cycle, where installing a new scheduler is just a matter of restarting a program.

What he came up with is a new Rust crate providing the scheduling interface; it is licensed under GPLv2. Schedulers are thus written in Rust, but the BPF code, which mostly just communicates events and decisions between the kernel and user space, is still compiled from C code. A pair of ring buffers is used for communication; initially BPF maps had been used, but the ring buffers are much faster. The API for schedulers has been deliberately kept simple, with the idea that anybody should be able to use it to write a new scheduler.

Righi admitted, though, that scx_rustland "is not all rainbows and unicorns". One significant problem is that the scheduler program cannot block for any reason (such as a page fault), or scheduling as a whole comes to a halt. So a custom memory allocator is used to keep the scheduler running in locked-in-RAM memory. Multithreading in the scheduling program is "tricky" but mostly solved. Even with the ring buffers, the communication overhead with the kernel is significant, but not a huge issue. There are some possible sched_ext changes that would help there.

Righi's future plans include standardizing and locking down the user-space API for schedulers. He would also like to create a concept of "scheduling domains", each of which is made up of a set of CPUs. The ability to attach a task to one of these domains would make scheduling easier and improve performance.

Higher frame rates

Changwoo Min took over via a remote link to talk about scx_lavd, which is a "latency criticality aware virtual deadline" scheduler aimed at gaming applications. It uses latency criticality (described later) as the primary scheduling decision, handles heterogeneous cores well, and adapts its scheduling decisions to the load pattern on the system.

The goal behind this scheduler was to provide the best gaming experience on Linux in general — not just on the Steam Deck. That requires getting high performance (and high video frame rates) without stuttering (short-term performance loss due to load in the system). The scheduler should deliver reasonable performance across a wide range of CPU configurations, but it is not intended to be the best server or general-purpose scheduler.

A key aspect of gaming workloads is that tasks tend to run quickly, typically no more than 100µs at a time. There are a lot of tightly linked tasks, though, and performance depends on the most critical of those tasks running in the necessary sequence; that is the critical path. Every task has a latency criticality that is determined by its place in this path; tasks that wait on others, and are waited on in turn, have a large impact on overall performance and are thus "latency critical". Detecting these tasks requires observing which tasks wait for which others, and ensuring that the tasks being waited for are run with low latency.

Each task has a virtual deadline calculated for it, which is a function of both its waking and waiting frequencies — its latency criticality, in other words. Tasks that both wait often for others and are often waited upon are seen as the most critical, so their deadline is the shortest. Time slices are then assigned in a manner similar to how the completely fair scheduler does it; slices are fixed, but get shorter as the number of runnable tasks increases.

Care is also taken to chose CPUs properly on heterogeneous systems. At times of low load, with a simple workload, the low-power cores can get the job done while minimizing power use. If the load is heavy, though, then performance becomes the primary goal, and the fast cores must be used. The in-between case is trickier; some tasks can be put on smaller cores, but some will need the faster ones.

In the scx_lavd "autopilot" mode, the scheduler looks at the current CPU utilization. For light loads, a power-saving mode is chosen; for heavy loads, the fast cores are used in a race-to-idle strategy. In between those extremes, the scheduler tries to minimize the number of cores in use, but takes care to put the latency-critical tasks onto the large cores.

Min concluded by saying that, for gaming applications, scx_lavd consistently enables higher frame rates than the EEVDF scheduler while using (slightly) less power and with fewer stutters.

A lot of activity

The sched_ext microconference included a number of other presentations, some from people who had been working on out-of-tree schedulers for years. Barret Rhoden and Josh Don talked about the use of pluggable scheduling within Google, a project that has been underway since 2019. Once again, this effort was able to obtain better performance, but also highlighted the fact that different workloads benefit from different scheduling policies. Himadri Chhaya-Shailesh discussed using sched_ext for paravirtualized scheduling, where host and guest schedulers communicate to optimize the overall result. Masahito Suzuki and Alfred Chen have both been working on out-of-tree schedulers for desktop use. Peter Jung discussed the CachyOS distribution, which has been shipping a range of out-of-tree schedulers for years; developers there have created a whole infrastructure allowing users to switch schedulers on the fly.

The kernel project has long had a policy that it would support one general-purpose CPU scheduler, and that scheduler had to provide good service for all workloads. This policy has, beyond a doubt, resulted in a sophisticated scheduler that is able to run on everything from small embedded systems to massive data-center machines. It has ensured that all users benefit from scheduler improvements.

What was made abundantly clear at the sched_ext microconference, though, is that this policy has also led to the marginalization of a lot of creative work in this area. A scheduler that cannot regress for any workload leaves little room for developers wanting to optimize a specific class of applications, and who cannot even test many other workloads. This is a hard area in which to scratch an itch; developers have been discouraged from trying, and those who have ventured into this area have rarely seen their work enter the mainline kernel.

Sched_ext has removed many of the barriers to entry in the area of scheduler development, and the result has been an immediate increase in the number of developers playing with ideas and seeing where they lead. There is a new community that is quickly forming here, and it seems likely to come up with some novel (and sometimes crazy) approaches to CPU scheduling. This will be an interesting space to watch in the coming years.

[ Thanks to the Linux Foundation, LWN's travel sponsor, for supporting our travel to this event. ]

Comments (3 posted)

The rest of the 6.12 merge window

Linus Torvalds released 6.12-rc1 and closed the 6.12 merge window on September 29; at that point, 11,260 non-merge change sets had been pulled into the mainline for the 6.12 release. That is the lowest number of merge-window changes since 5.17-rc1 in January 2022, which brought in 11,068 changesets. Nonetheless, 6.12 brings a number of interesting changes, many of which were included in the roughly 4,500 changes merged since the summary of the first half of the 6.12 merge window was written.

Significant changes entering the kernel in the second half of the merge window include:

Core kernel

• Several features of the version-1 control-group interface, including TCP accounting, the v1 soft limits, out-of-memory control, and pressure level, have been deprecated. As was discussed earlier this year, the first step is to emit warnings to see how many users (if any) are affected at this point.
• After a year of controversy, the extensible scheduler class has been merged. This feature allows the writing of CPU schedulers as a set of BPF programs; it has already resulted in a number of interesting, specialized schedulers being written. This commit contains some introductory documentation.
• It is now possible to set up the tracing ring buffer in a way that allows it to persist across a reboot, perhaps saving information that would otherwise be lost in a crash. See this changelog for some details.

Filesystems and block I/O

• The virtual filesystem layer has gained the ability to work with storage devices that have a block size larger than the system's page size. Getting to this point has required a multi-year effort, the last stages of which were discussed at the Linux Storage, Filesystem, Memory-Management, and BPF Summit earlier this year. At this point, only the XFS filesystem is able to take advantage of this capability.
• The filesystems in user space (FUSE) subsystem had gained support for ID-mapped mounts.
• The NFS filesystem now supports the "LOCALIO" protocol, which allows the client and server to determine whether they are running on the same host; there are some significant performance improvements that can be had in that case. See this documentation commit and this one for more information.

Hardware support

• Clock: Rockchip RK3576 clock controllers, Qualcomm SM8150 camera clock controllers, Qualcomm SM4450 display clock controllers, Qualcomm SM4450 camera clock controllers, Qualcomm SM4450 graphics clock controllers, and DFRobot SD2405AL I2C realtime clocks.
• GPIO and pin control: Mobileye EyeQ5 pin controllers and Sophgo CV1800B, CV1812H, SG2000, and SG2002 SoC pin controllers.
• Industrial I/O: TYHX HX9023S SAR sensors, Analog Devices ADXL380 3-axis accelerometers, Analog Devices LTC2664 and LTC2672 digital-to-analog converters, Analog Devices AD4695 and AD4000 analog-to-digital converters, ROHM BH1745 color sensors, Microchip Technology PAC1921 power/current monitors, Sensirion SDP500 differential pressure sensors, ScioSense ENS210 temperature and humidity sensors, Sophgo CV1800B analog-to-digital converters, and Awinic AW96103/AW96105 proximity sensors.
• Input: PixArt PS/2 touchpads.
• Media: Extron DA HD 4K Plus CEC HDMI distribution amplifiers.
• Miscellaneous: native PCIe enclosure LEDs, Renesas RZ/V2H(P) watchdog timers, Nuvoton MA35 USB2.0 PHYs, AMD Queue-based DMA engines, Loongson1 APB DMA controllers, NXP LPC32xx DMA multiplexers, BlinkM I2C RGB LED controllers, Theobroma Systems Mule I2C device multiplexers, KEBA I2C controllers, TI K3 M4 remote processors, Qualcomm MSM8976 and MSM8937 interconnects, and ON Semiconductor LC824206XA microUSB switches.

Miscellaneous

• The user-space perf tool has gained a number of new features; see this commit for a summary.
• Rust code now works with a number of kernel development and security features, including the KASAN sanitizer, Spectre mitigations, and kernel control-flow integrity. Most GCC plugins can now be used with Rust as well, with the only exception being structure-layout randomization.
• There are new Rust modules for doubly linked lists and red-black trees.

Security-related

• The Landlock security module now supports the concept of "scoping" for Unix-domain sockets, allowing the administrator to limit communications between sandboxes. This documentation commit has more information. The scoping concept also applies to signals; see this commit for (a little) more information.
• The Tomoyo security module can now be built as a loadable kernel module. This is a bit of a surprising change, since making security modules loadable has been resisted by the relevant maintainers for years. The main commit also includes a novel mechanism for bypassing the normal restrictions on access to (some) non-exported symbols by loadable modules. [Update: this change was subsequently reverted.]

Internal kernel changes

• The new xdrgen tool exists for "translating XDR specifications into XDR encoder and decoder functions written in the Linux kernel's C coding style". See this commit for more information.
• Torvalds added a new mechanism for user-space address masking that is intended to regain some of the performance that was lost to Spectre mitigations. It only works on x86 systems for now; it works through the application of a data-dependent mask that prevents speculation, eliminating the need to use more expensive barriers.

The 6.12 release is feature-complete at this point; all that remains is to find and fix all of the bugs that were inevitably added as well. That work will be carried out through the upcoming series of -rc releases, culminating in the 6.12 release in the second half of November.

Comments (2 posted)

Getting PCI driver abstractions upstream

Danilo Krummrich gave a talk at Kangrejos 2024 focusing on the question of how the Rust-for-Linux project could improve at getting device and driver abstractions upstream. As a case study, he used some of his recent work that attempts to make it possible to write a PCI driver entirely in Rust. There wasn't time to go into as much detail as he would have liked, but he did demonstrate that it is possible to interface with the kernel's module loader in a way that is much harder to screw up than the current standard approach in C.

To give context to the discussion, he started by explaining that his goal was to make development of Nova (the new NVIDIA driver he has been working on) go smoothly. He opined that Nova would probably be the first "more complex" thing to go upstream. Luckily, other in-progress efforts to write kernel components in Rust need some of the same abstractions as Nova, including rvkms, rnvme, Apple AGX, and rcpufreq-dt. Ultimately, he would like to provide driver infrastructure that integrates with the kernel, but that takes advantage of Rust's capabilities where possible.

Initially, there was confusion on the mailing list, Krummrich said, about what the Rust abstractions he wanted to discuss were meant to represent. The patch set that spawned the discussion is not really about abstracting drivers he explained, but about permitting drivers to access bus types safely — abstractions needed for drivers. A fact which had proved difficult to communicate via email. At the end of the discussion, Greg Kroah-Hartman had called for writing the driver-registration code itself separately, in C.

At Kangrejos, Kroah-Hartman took responsibility for half of the misunderstanding, but said that he did still think it made sense to implement driver registration itself in C. He said that it would be a small amount of "safe C" — a comment that prompted good-natured chuckles from the attending Rust programmers. Krummrich was sympathetic, but wanted to go through an example of what he was trying to achieve with the driver abstractions and why.

Krummrich showed the code for a simple PCI driver written in C, and then went through a series of modifications to incrementally rewrite it in Rust. The simplest change was just to replace the probe and remove functions with versions written in Rust:

    #[no_mangle]
    unsafe extern "C" fn rust_pci_driver_probe(
        _pdev: *mut bindings::pci_dev,
        _ent: bindings::pci_device_id,
    ) -> core::ffi::c_int {
        pr_info!("Probe Rust PCI driver sample.\n");

        0
    }

    #[no_mangle]
    unsafe extern "C" fn rust_pci_driver_remove(_pdev: *mut bindings::pci_dev) {
        pr_info!("Remove Rust PCI driver sample.\n");
    }

The code was fairly simple, but after showing it, Krummrich explained that there was actually already a bug in this example that he hadn't noticed until later: the signature of the Rust functions was incorrect. It didn't cause problems, because the incorrect parameter was unused, but it still served to illustrate why he wanted to make interfacing with the kernel's driver system less error-prone.

He also mentioned an issue that didn't show up in this simple example — binding object lifetimes to driver and module lifetimes, instead of device lifetimes, such as is sometimes necessary for complex drivers. Kroah-Hartman thought that it was good for that to be hard, because "we don't want you to do that". He explained that the kernel tried hard to ensure that data is bound to data, not to code, and that drivers should use per-device storage. Krummrich said that the same problem applied to trying to set up per-device storage, in this case. Ultimately, they both agreed that what they wanted was simple lifetime handling that bound data (such as cache data) to other data (such as a specific device), and that the simplest way to integrate Rust and C code did not make that easy.

Krummrich did include the caveat that Nova had some buffers that needed to live for the lifetime of the module itself, because of how NVIDIA's GPU system processor handles debugging information. Kroah-Hartman replied that GPUs were "the most crazy complex hardware out there". So he was fine with GPU drivers needing to do something out of the ordinary — but normal drivers shouldn't need to.

This prompted some discussion of which subsystems might need to do things differently. Kroah-Hartman agreed that some of the core infrastructure would need more complicated lifetime management as well. Most drivers are simple I2C or PCI drivers, however, he said. So it's only a handful of complicated drivers that need more complex lifetime management.

Krummrich then showed how the bindings could be made safer by moving some of the implementation into Rust; he demonstrated this in stages. The first stage was to use a macro to declare the module in Rust instead of C. That still left unsafe functions, the possibility that the driver could forget to unregister correctly, and some awkwardness with constructing the PCI device ID table.

Next, he showed how the uses of unsafe could be centralized to a single helper library, such that each driver only needs safe functions, with signatures type-checked by the compiler. The new version involves creating a type that implements the pci::Driver trait, which then takes care of the details of registering and unregistering the driver, and storing device-specific information in a type-safe way.

There were some more questions about the specific details. Kroah-Hartman asked why Krummrich had reimplemented a structure in Rust instead of using the version defined in C; Krummrich said that he didn't want to expose the structure directly to drivers. Benno Lossin suggested using a type alias, so that it's still not defined in two places.

The most involved part of the code concerned the machinery to make the PCI device ID table automatically add an appropriate sentinel value to the end at compile time, so that the whole device ID table can be stored in the binary's data section — a much more elegant solution than creating a copy of the table at run time. That ended up requiring the use of some additional intermediate traits, and made the whole code more than twice as large.

Kroah-Hartman said that the code was great, but that the complexity needed to add the sentinel "sounds crazy". He asked whether there wasn't a simpler way to do it, but nobody present volunteered a way.

Krummrich's example did show that it is definitely possible to build a driver-binding abstraction that requires no unsafe code in drivers. Such a driver cannot accidentally forget to unregister, automatically has type-safe storage associated with each device that it is registered for, does not need to touch raw pointers (instead receiving bounds-checked smart pointers for the appropriate PCI buses), and should generally be quite difficult to misuse.

The session ran out of time before Krummrich could finish his presentation. The overall conclusion from Kroah-Hartman was that he wants to see the full example in a patch set, but that it was a promising start. Krummrich wants this driver abstraction to be available for Nova and the other Rust drivers that use the same features, so it seems likely that he'll provide that example soon.

Comments (1 posted)

Coccinelle for Rust

Tathagata Roy has been working to make the Coccinelle tool that is used (among other things) to automate the refactoring of C code work on Rust code as well. Roy gave a presentation at Kangrejos about that work, including the creative approaches necessary to work with Rust's more complicated control flow and syntax.

Roy opened by describing the purpose of Coccinelle within the kernel. Linux's code is huge, and frequently a change to some internal API will need to be reflected across a large number of drivers or other kernel components; Coccinelle allows maintainers to write patches in a special language that can be automatically applied across the entire code base, making it easier to make changes that have a broad impact. This kind of change requires a specialized tool since parsing C source code is not trivial.

Although there is much less Rust code in the kernel, it would be nice if the same tool that existing kernel maintainers are familiar with also worked on Rust code. Roy presented an example of what that might look like; in short, authors would follow the same process as for C code. To construct a semantic patch for Coccinelle, the programmer picks a typical diff for the desired change, and then makes it more generic by adding "expression variables". An example of a patch for a Rust API change might look like this:

    @@
    expression tcx, arg;
    @@

    - tcx.type_of(arg)
    + tcx.bound_type_of(arg).subst_identity()

That patch finds uses of .type_of() and rewrites them to call a different function, even if the method is being invoked on or with complex expressions.

Under the surface, however, correctly supporting Rust requires a lot of thought. The biggest issue Roy highlighted is more complex control flow. In order to apply patches correctly, Coccinelle needs to match patch expressions to the abstract syntax tree (AST) of the program. This requires knowledge of the program's control flow, Roy explained. In Rust, this includes wrinkles such as if expressions. It's valid to write code like this in Rust:

    println!("{}", if boolean { "string 1" } else { "string 2" })

This can lead to some counterintuitive control-flow graphs. The center of Roy's recent work has been getting the representation of Rust programs into a form that can be processed by Coccinelle's existing pattern-matching algorithm. His approach works by doing a preorder traversal of the AST, converting it into a representation of the control flow in the program, and then matching on that, instead of on the raw AST. This approach works, but still has some problems, he said. It results in "huge" graphs, which take time for Coccinelle to process, and make debugging difficult.

The graphs can be compressed by 4-5 times by eliminating intermediate nodes with only one child, but that does not present a complete solution. Another issue with this representation is dealing with metavariables (such as tcx and arg, above), Roy explained. Metavariables can stand in for any expression, which makes matching them to parts of the control-flow graph somewhat tricky. So, to make matching easier, the graph is also decorated with extra edges that point to the next child (or closest cousin) of a node. This makes it much easier to efficiently determine where an expression ends.

This new internal representation isn't the only work-in-progress for Coccinelle. Roy explained that Coccinelle for Rust had recently added an ellipsis operator that can stand in for any control-flow path connecting two nodes. This makes it easier to write rules that match two different call sites separated by some arbitrary code.

Matching a pattern that contains an ellipsis requires considering different potential matches, which means considering multiple alternatives, so it is a form of disjunction. Previously, Coccinelle only permitted disjunctions of expressions; now, Roy is working on generalizing disjunctions to include arbitrary statements. This has proved particularly difficult for Rust code, because of Coccinelle's approach to parsing. Coccinelle's C parser is hand-written, and can therefore include extensions for parsing Coccinelle-specific constructs in semantic patches into the same structure as the adjacent C code. Coccinelle's Rust parser, on the other hand, is the stock parser provided by the rust-analyzer project — because Roy did not want to rewrite an entire Rust parser and worry about keeping it up to date. This makes parsing Coccinelle's syntax for disjunctions tricky.

The solution Roy has selected is to convert disjunctions into Rust macros by simple textual substitution, parse the resulting Rust code with rust-analyzer, and then recognize the macro name and treat it specially. This has the advantage that it permits easier parsing, but the disadvantage that it doesn't allow disjunctions of attributes, enum alternatives, or other constructs that can't be wrapped in macros. Roy emphasized that support for disjunctions was still a work in progress.

There are two other problems that Roy wants to see resolved before calling the work complete: allowing Coccinelle to work on actual macros (which he described as "a pain in the AST") and processing patches in parallel. When these are finished, Coccinelle will hopefully prove to be as useful a tool for Rust programmers as it is for C programmers. People wishing to use the current (experimental) support for Rust can find Roy's development version in the CoccinelleForRust repository — although the linked branch does not include the in-progress support for disjunctions.

Comments (13 posted)

BTF, Rust, and the kernel toolchain

BPF Type Format (BTF), BPF's debugging information format, has undergone rapid evolution to match the evolving needs of BPF programs. José Marchesi spoke at Kangrejos about some of that work — and how it could impact Rust, specifically. He discussed debug information, kernel-specific relocations, and the planned changes to kernel stack unwinding. Each of these will require some amount of work to fully support in Rust, but preliminary signs look promising.

BTF

Marchesi described BTF as a format to denote the compiled form of C types. He said that it was similar to DWARF, but "way, way simpler". BTF is designed for a particular use case: efficient, online operations on C types and functions as they exist in memory. DWARF information is concerned with mapping debugging information to the source-level constructs of a programming language; BTF is concerned with what is in the compiled object and "not much related to the source language". At run time, this information is used by BPF programs to access kernel structures correctly, among other uses.

The process of generating BTF for a given kernel is somewhat tortured. When the kernel is compiled with BTF support, it is built with DWARF information. Then pahole converts the DWARF to BTF. One consequence of this approach is that BTF can only include information that is also present in DWARF — a problem for some of the kernel's structure attributes that aren't properly represented, so Marchesi is working toward being able to generate BTF directly. This is already mostly working in GCC, but the kernel is not yet built that way.

When the C compiler does start producing BTF directly, though, it will cause problems for the parts of the kernel written in Rust: the Rust compiler will also need to generate BTF. There are benefits to having Rust generate it as well — BTF could be used for genksyms, the tool that generates lists of kernel symbols to check loadable module compatibility — but it will certainly require some work as well.

The Rust compiler will not have to start from scratch, Marchesi said. People do already write BPF programs in Rust, and LLVM emits "correct-enough BTF". "But that's not by design," he warned, just a result of supporting BTF for C. Properly supporting BTF for Rust will mean making sure it lines up with the BTF generated for the rest of the kernel, that it works even for obscure corner cases, and that it can fully capture the richness of Rust types.

Right now, pahole is sidestepping the issue by just ignoring DWARF generated for Rust code, not creating BTF from it. This has already caused problems for some users. Carlos Bilbao asked whether anyone had tried generating BTF from a program written in a mix of C and Rust, and seen what the problem is. Marchesi explained that Rust generates DWARF with some structures that pahole doesn't support. Miguel Ojeda expanded on that, saying that Rust uses some DWARF types that were originally introduced for C++ support, and that therefore pahole doesn't have existing support for.

Björn Roy Baron and Gary Guo listed some problems with Rust enums that might apply to BTF. In particular, Rust enums are more like tagged unions in C — they have a discriminant and then a set of fields. The Rust compiler doesn't guarantee any particular representation, however; it uses this freedom to optimize some types to take less space. For example, Option<T> is an enum that contains either None or a value of type T. When values of type T can never be zero, the compiler can save the space needed by the enum tag by using zero to represent None.

This means that unlike structures, which can be annotated with #[repr(C)] to instruct the compiler to lay them out exactly like C structures, native Rust enums can't be forced to have a stable layout. The Rust compiler can, each time it is run, choose a different layout for each enum. In practice, a given version of the compiler always uses the same layout, but that isn't guaranteed. If BTF needs to refer to enum types, that freedom could complicate the implementation.

Marchesi also highlighted the difficulty that link-time optimization (LTO) poses. DWARF distinguishes between different compilation units, whereas BTF does not. So name clashes across compilation units are potentially a problem for using BTF in an LTO build of the kernel. Alice Ryhl raised a different problem — LTO can inline Rust code into C compilation units, meaning that the DWARF info can be mixed. That causes a problem for LTO builds today, since pahole can't handle the mixed DWARF info.

CO-RE

After laying out his basic concerns, Marchesi raised the topic of compile once - run everywhere (CO-RE), the approach that lets the kernel load BPF programs without requiring an exact match between the kernel headers the program was compiled against and the running kernel. In order to make this work, the compiler for the BPF program needs to take some special steps. In C, an attribute called preserve_access_index causes the compiler to generate loads and stores in a way that can be patched, and a relocation entry that tells the loader how to patch the program if the layout of the structure has changed from a different version of the kernel. Both GCC and LLVM have support for CO-RE; Marchesi wanted to know if the same approach made sense for Rust, given that the compiler can reorder fields of Rust structures (that aren't marked as using the C layout).

Andreas Hindborg thought that support like that would be great to have in Rust, since it could potentially allow for linking object files from different compilers — something that currently requires explicitly using the C calling convention, since Rust lacks a stable ABI of its own. He did have some questions about how it could work in practice, however, including what happens if a BPF program is built against an incompatible version of the kernel headers.

"Nothing good", Marchesi answered. But in the case of BPF, the verifier would complain about any bad accesses. After some discussion, during which Ojeda and Guo clarified some details of Rust's layout semantics, Marchesi suggested that perhaps a good first step would be generating CO-RE relocations only for #[repr(C)] structures. Guo questioned how that would interact with the offset_of!() macro, which can be used to find the offset of a field within a structure. Marchesi explained that the value would have to change with the relocation, but that this meant that any math that depended on the offset would be broken. Baron suggested that this might require an opaque wrapper type to prevent things from breaking.

Unwinding

Marchesi had one last topic: the potential switch from ORC to SFrame for stack unwinding in the kernel. He wanted to check that the switch would not cause problems for the Rust parts of the kernel. Guo assured him that Rust does support unwinding, currently with the same DWARF-based methods that C programs largely use. The important part is that compiled functions have unwinding information that matches what the C code does, so any potential compiler change might work out of the box. Marchesi called that "very good news", and wrapped up the session on a positive note.

Overall, BTF is unlikely to pose insurmountable challenges to the inclusion of Rust in the Linux kernel, but there are some areas that will need additional work. At the least, there will need to be testing for LLVM's BTF support, for applying CO-RE to the Rust parts of the kernel, and for ensuring that Rust's unwinding support remains working. Some of those areas may also need additional attention to ensure that the kernel can continue working smoothly as a conglomerate of C, BPF, and Rust.

Comments (15 posted)

Debian changes OpenSSH packaging

In the wake of the XZ backdoor, the Debian project has revisited some of the patches included in its OpenSSH packages to improve security. The outcome of this is that the project will be splitting out support for Kerberos key exchange into a separate set of packages, though not until after the Debian 13 ("trixie") release expected next year. The impact on Debian users should be minimal, but it is an interesting look into the changes Linux distributions make to upstream software as well as some of the long-term consequences of those choices.

Patches revisited

In April, Colin Watson (one of the OpenSSH package maintainers for Debian) started a discussion on the debian-devel mailing list, saying that he was "reconsidering some choices in Debian's OpenSSH packaging" because he hoped to configure the package "more wisely within more or less our current constraints". Watson did not elaborate much beyond that, but it was patching OpenSSH to link to libsystemd that made Debian and Fedora's packages vulnerable to the XZ backdoor, which was still very fresh in everyone's mind at the time. See LWN's coverage of how the XZ backdoor works for more detail.

Debian carries nearly 40 patches against OpenSSH, ranging from fairly large patches like the one that enables GSS-API and changes OpenSSH functionality to small documentation patches. Out of all of the patches Debian applies to OpenSSH, Watson identified four for discussion: the patches for libsystemd, SELinux, TCP wrappers, and Generic Security Service Application Program Interface (GSS-API), which is used to provide a vendor-independent method of authenticating against different Kerberos implementations.

At the time, he said that the libsystemd patch (actually called the systemd-readiness patch) was "the obvious thing on everyone's mind right now" due to its role in the XZ backdoor. OpenSSH was patched so that it could notify systemd when it was ready to accept connections, using the libsystemd library, which in turn had a dependency on the liblzma library that was compromised.

Watson said it had seemed reasonable to merge the patch when he did it, but "we do seem to have ended up on the wrong side of history with that one". However, he noted that there was work to remove the dependency on libsystemd itself, and to move the readiness protocol upstream, "so we'll get rid of that linkage and reduce our patch load a bit".

Debian also compiles OpenSSH with support for SELinux, which OpenSSH developer Damien Miller said should be in an alternate package (along with Kerberos/GSS-API) to reduce exposure for the bulk of users who may not need the feature. However, Watson said, "my inclination is to leave this [patch] be", but invited SELinux experts to weigh in. No SELinux experts turned up to voice an opinion one way or another, and that patch remains.

TCP wrappers

The idea of dropping support for TCP wrappers drew more discussion. TCP wrappers is a system for filtering access to services, such as OpenSSH, using host-based access-control lists (ACLs) placed in /etc/hosts.deny and /etc/hosts.allow. The upstream OpenSSH project dropped support for TCP wrappers in 2014, just ahead of the Debian 8 ("jessie") freeze. Watson said that he had patched it back in "temporarily" and then "dropped the ball on organizing a proper transition". He also noted that Fedora had dropped the use of TCP wrappers in 2018.

A few users were reluctant to lose TCP wrappers, even if the project was considered outdated. Marc Haber said it was much easier to use TCP wrappers to protect his system than to use a packet filter. Marco d'Itri objected to removing support for TCP wrappers since it supports use cases like DNS-based ACLs that are not supported by layer 3 firewalls.

Watson suggested d'Itri could replace the functionality with tcpd as Fedora suggested when dropping TCP wrappers, and pointed out that OpenSSH supports DNS-based ACLs without requiring a separate library. That did not convince d'Itri, who complained that the tcpd approach would require socket activation of sshd "which is not the default for good reasons". He did not elaborate on this, but there are reports that socket activation can be exploited to cause a denial of service, and recent changes that split sshd into multiple binaries have made Debian's non-default support for socket activation more fragile.

Watson wanted to get rid of the TCP wrappers dependency because its library, libwrap, pulls in a heavy dependency chain, including support for the little-used Network Information Service (NIS). He suggested refactoring libwrap to reduce its dependency chain, which d'Itri seems to have done. TCP wrappers support does not appear to be on the chopping block at this point—though Watson hasn't explicitly confirmed that as of yet.

GSS-API key exchange

OpenSSH added initial support for GSS-API in version 3.7 back in 2003. But OpenSSH's GSS-API support still requires SSH host keys to authenticate the server to the user, as Simon Wilkinson pointed out in announcing a version of his GSS-API patch for OpenSSH 5.0p1. (The "p" in the release version indicates it is an OpenSSH portable release for systems other than OpenBSD.) Many users of OpenSSH did not want to maintain both a Kerberos infrastructure and OpenSSH host keys, so the patch allows organizations to use a single security architecture rather than managing both.

In 2004, Matthijs Mohlmann requested that Debian add support for Kerberos to OpenSSH. Ultimately, after some discussion and waiting for the patch to mature, Watson merged the patch in 2005. A number of attempts were made over the years to persuade the OpenSSH developers to accept the patch, but they officially turned it down in 2010. Miller said that none of the OpenSSH developers were in favor of the patch:

This is not a slight on Simon's patch, which is of fine quality, but just that a) we don't trust GSSAPI implementations that much and b) we don't like adding new KEX [key exchange] since they are pre-auth attack surface. This one is particularly scary, since it requires hooks out to typically root-owned system resources.

The patch is now maintained on GitHub by contributors from Debian and Fedora, and both distributions carry a version of the patch in their OpenSSH packages.

In his initial message to debian-devel, Watson said that the GSS-API patch was the largest patch applied to Debian OpenSSH "by an order of magnitude, and easily the most intrusive in terms of complexity and exposure". He added that he had come to regret merging the patch a few times over the years, but he was also aware that people depend on the functionality in the patch. "We've had it for nearly 20 years now, so we can take the time to do a proper job that at least tries not to leave users in the lurch."

His proposal was to create special ("dependency-only") packages for Debian trixie that would have names to reflect GSS-API support. For the trixie release, those packages would effectively be aliases for Debian's OpenSSH client and server packages, which would still contain GSS-API support. However, that functionality will be split out into packages with the same name in the Debian 14 ("forky") release, so users who have the special packages installed would automatically get the new packages when they upgrade. The patch would be dropped from the regular OpenSSH packages in forky. The OpenSSH package NEWS.debian file and trixie's release notes will also inform users of the changes.

Michael Stone suggested speeding things up by doing things the opposite way: creating "nogss" packages that don't contain the GSS-API patch for Debian trixie, so users could move to the new, non-GSS-API, version quickly. "Pushing back the ability to install the unpatched version for a few more years seems suboptimal." Watson said that he worried about churn and inviting more bugs by approaching it that way, but he'd consider it.

Split

In the end, though, he went with his original approach. On August 30, he announced that he'd created the special openssh-client-gssapi and openssh-server-gssapi packages with the intent to split out GSS-API key-exchange support and authentication in forky. For now, Debian does not have an OpenSSH package without the GSS-API patch. Users who want those features in forky will need to use the -gssapi packages, and users who want to run OpenSSH without the patch will want the openssh-client and openssh-server packages.

Miller said this was excellent news: "this substantially reduces the amount of pre-authentication attack surface exposed on your users' sshd by default".

I emailed the maintainers of the OpenSSH packages for Fedora, and Dmitry Belyavskiy said that Fedora will be keeping the patch. He added that the maintainers had discussed following Debian's approach, but they do not plan on getting rid of GSS-API support for now or breaking it out to separate packages.

The transition should be painless for Debian users. Those who depend on the GSS-API patch will have years to ensure that their systems are ready for the transition. Meanwhile, it's an interesting case to ponder for Linux distribution packagers thinking about carrying a patch against an upstream project. A patch applied today may still need tending for years, if not decades.

[Thanks to LWN reader A. Karl Kornel who suggested this story.]

Comments (24 posted)

An update on gccrs development

One concern that has often been expressed about the Rust language is that there is only one compiler for it. That makes it hard to say what the standard version of the language is and restricts the architectures that can be targeted by Rust code to those that the available compiler supports. Adding a Rust frontend to GCC would do much to address those concerns; at the 2024 GNU Tools Cauldron, Pierre-Emmanuel Patry gave an update on the state of that work and what its objectives are.

The GCC frontend goes by the name "gccrs"; the trademark rules around the language prevent it from being called a Rust compiler. Since the project restarted in 2019, Patry began, it has been targeting the 1.49 release of the language, which came out in 2020. It has been included in GCC since GCC 14, though that release did not include the gccrs documentation.

At the 2023 GNU Tools Cauldron, Patry had said that the project had over 800 commits in its repository that needed to be upstreamed into the GCC mainline — a significant pile of work. This year, that number has reduced to 40 — and those are new work, not leftovers from the previous year. So some progress has been made. The goal had been to push changes upstream every two weeks, but that has proved to be too much work; they are still aiming to get their code upstream as quickly as they can, though they have backed off on that specific goal.

Over the last year, gccrs development was helped by three Google Summer of Code (GSoC) interns. One of them (Jasmine Tang) worked on support for inline assembly code, a feature that the Rust-for-Linux project needs; most of the frontend work for that is now done. Inline assembly in Rust looks like:

    unsafe {
        asm!("assembly code here", "other info")
    };

While the frontend work is in place, there had been some difficulties getting that assembly code through the later compilation stages intact. Those problems have been fixed, but there are still various details to be dealt with.

The second GSoC student (Muhammad Mahad) worked on a test-suite adapter, with the goal of helping gccrs pass the Rust test suite. There is an impedance mismatch to work around here; GCC uses DejaGnu for its tests, so some sort of conversion of Rust's tests is needed. The final GSoC student (Kushal Pal) worked on borrow checking in the GCC intermediate representation and getting proper error messages to users.

Looking forward to the upcoming GCC 15 release, Patry said that the primary focus is on implementing features that are needed to compile kernel code with gccrs. The most important task there is getting to a point where gccrs can compile the Rust core library, which turns out to not be an easy task. The developers had thought they had implemented enough to do that, but were surprised by many problems in need of fixing. The Rust project's habit of using experimental features in the core library does not help there.

Those problems are being worked through, Patry said, but even when it works, the compiler is currently quite slow. That, though, is a problem to be worried about later, once the feature and correctness work is done and the core library compiles.

Closing out with longer-term goals, Patry started with, once again, compiling the core library — but this time without making any changes to the library itself. The developers would like to catch up to the 1.78 release that is currently used by Rust for Linux. That is not as big a leap as might be thought, he said; many of the 1.78 features had been experimental long before and are already implemented in gccrs. Beyond that, he said, the project will eventually address building the std library and, someday, work to catch up with the rustc compiler.

The impression left by this session is that the gccrs developers are working hard to fill the second-compiler gap for Rust, but that there are far too few of them. This is an important project that really should attract some significant company support but, for the most part, that is not happening. Unless more in the industry step up to push this work forward, Rust may be fated to remain a language with a single compiler for a long time to come.

Comments (101 posted)

Inkscape 1.4 coming soon

The open-source vector-graphics editor, Inkscape, is expected to release version 1.4 in October. The release represents an evolutionary step for the program, which brings new features, user-interface improvements, new and improved file-format support, and important changes to the code base. The changes in this release should improve the user experience for both casual and professional designers, and make Inkscape more compatible with proprietary vector-graphics software, including Adobe Illustrator and Affinity Designer.

New and improved dialogs

Dialogs (also referred to as panels) are a core component of the Inkscape user interface. Implemented as GTK+ tabs and housed in a custom panel widget (typically on the left side of the application window), dialogs organize commonly used tools and settings into groups. For example, the "Fill and Stroke" dialog contains settings for object colors, gradients, patterns, and swatches, as well as settings for strokes (whether by themselves or as outlines for objects).

Thanks to this approach of grouping features and their settings, the developers can introduce incremental improvements, or add new features altogether, without seriously compromising the overall layout and familiarity of the application. In this release, emphasis has been placed on polishing existing dialogs, while also introducing new ones, such as the galleries for extensions and filters, which highlight a number of the standard tools and extensions Inkscape provides.

Inkscape is shipped with lots of extensions, written in Python, that provide features developed separately from the core project. In previous releases, extensions were only available through the "Extensions" menu, which does not have features such as searching or previews. However, with Inkscape 1.4, comes a new "Extensions Gallery" — a more modern and familiar interface for browsing Inkscape's extensions. Benefits of this new dialog include a dedicated search box for extensions, flexibility through customization of the dialog, and thumbnails associated with each extension. Some extensions do not have thumbnails and will use shared placeholders for now. The Inkscape team has extended an invitation to the community for contributions to this effort.

Debuting in this release, the "Filter Gallery" dialog presents all of the SVG filters included with Inkscape in a searchable format. SVG filters are W3C-standard, image effects that can be applied to any object, including raster graphics. In previous releases, filters were only available from the "Filters" submenu, which organizes them into categories but offers no means of searching or previewing the effect. Happily, the new dialog provides a way to find, preview, and select filters. The previews approximate the action of the filter, but do not present a live view of the filters' effects.

The search feature in the "Filter Gallery" allows for finding individual filters by name. This serves as a more intuitive solution compared to using Inkscape's command palette (accessed through "?" on the keyboard). The command palette is akin to Ubuntu Unity's HUD, in that it provides a means of typing any action Inkscape supports, pulling actions from menus, toolbars, or even extensions. In the "Filters" menu, filters are listed by their common names, but there is no search functionality. In contrast, the command palette accesses filters by their more complex internal names such as "doc.org.Inkscape.effect.filter.f088", making them harder to find. The new dialog simplifies this by allowing users to search for filters using their common names.

The "Swatches" dialog, which manages color palettes and recently used colors, has been completely overhauled. Users can now select palettes, search for colors by name, and load palette files in various formats, including GIMP Color Palette (.gpl), Adobe Swatch Exchange (.ase), and Adobe Color Book (.acb). It is also possible to load palettes using CEILAB colors, and, to a limited degree, CMYK colors. When loading palettes with CYMK, the colors are converted to RGB more effectively than before, even without an ICC profile installed. This brings Inkscape closer to full CMYK support. The gradient editor has also been improved, and gradients can now be rotated from the "Fill And Stroke" dialog, with the benefit of a full 360° slider.

Working with layers has long been a bit cumbersome in Inkscape, but, with successive releases, the layer workflow has been improved. The trend continues in this release, bringing two user-experience improvements to the layer workflow. When adding a new layer from the "Add Layer" dialog, users now choose where to place the layer using radio buttons instead of a drop-down list. This reduces the number of clicks needed and is more discoverable for new users. In the combined "Layers and Objects" dialog, adding a new layer no longer brings up the "Add Layer" dialog. Instead, new layers are added on top of the currently selected layer.

Inkscape's "Text and Font" panel combines basic text editing, font selection, and advanced font settings in one interface. However, the existing panel has problems with spacing and cutting off certain fonts from being displayed correctly. In Inkscape 1.4, a new, experimental "Unified Font Browser" has been introduced, which fixes those problems, allowing users to preview and choose fonts more intuitively. This feature must be activated through the preferences, at "Edit > Preferences > Interface > Windows", and requires a restart of Inkscape to take effect.

Once activated, the unified browser replaces the old font browser in the "Text and Font" panel. The new browser brings live previews, search functionality, and extensive customization options that are not present in its predecessor. On the first run, the new interface may take some time to initialize, especially with many fonts installed on the system. However, subsequent uses of the tool only experience a minimal delay to allow for previews to be rendered. As this feature is still experimental, it is likely to be further optimized in future releases.

Updated core features

Several of Inkscape's core features have seen significant changes in this release as well. The "Shape Builder" tool, introduced in Inkscape 1.3, now supports raster graphics as well as vectors. This tool provides a streamlined interface to the use of boolean operations to create more complex shapes by combining or removing elements from overlapping objects. Since these operations are not limited to SVG's primitives, such as ellipses and rectangles, designers can create complex shapes, for things like puzzles and cutouts. In the past, achieving the same result would have required a lengthy process of duplicating, clipping, and arranging both images and objects — or an extension.

Images sliced with the "Shape Builder" are automatically clipped to whatever shape the designer requires. Even shapes built within the tool itself can be used for this purpose. In addition to the new raster-image support, clones (copies) now preserve their visibility when the original path is being edited in the shape builder. In Inkscape 1.4, modifying a parent shape with the "Shape Builder" tool will turn any clones into independent objects, rather than affecting the clones directly.

It is now possible to add internal document links and preserve them when exporting to the PDF format. Previously, only hyperlinks to external sources were supported. All links are set up in the "Object Properties" dialog, and can be entered manually, or by clicking on the target object. Also improved in this release is the text-editing workflow, featuring a new toolbar layout that places text-alignment buttons directly on the toolbar as opposed to being in a popup menu as before. Text spacing has been moved into a popover component, along with kerning and rotation settings. With these changes, Inkscape is more able to handle desktop-publishing duties.

Interface and internal changes

This update brings several user-interface changes, including some that follow the ongoing work to improve Inkscape's user experience by simplifying and streamlining the interface to improve user workflow. In the world of digital-graphics software, user experience is of paramount interest, and goes beyond the actual interface itself, building upon considerations for users' needs. Inkscape has long charted its own path with how canvas controls are displayed, not following more familiar conventions that are common in the industry, which has negatively impacted some workflows. However, in Inkscape 1.4, new infrastructure is in place to customize the appearance of canvas controls. This work is the culmination of a Google Summer Of Code project by Sanidhya Singh, and allows controls to be customized using CSS.

Inkscape features a few icon themes out of the box, and version 1.4 introduces a new icon theme, called "Dash", designed to cover Inkscape's extensive library of features while simplifying the visual style of the application. Borrowing ideas from other applications in the graphic-design industry, this theme should be more familiar to users transitioning from other applications. On the merge request for this icon theme, there's been some discussion about replacing the legacy Tango icon theme with Dash once it is matured. However, for now, no official decision has been announced, but it has been included among the default themes in Inkscape 1.4.

There are also several new features and changes available via the command line that are worth noting. Inkscape exposes various actions through the command-line interface, including features and functions typically accessed through the GUI. This allows using Inkscape via scripts, and for batch processing, without the GUI. A full list of these command-line options is available by running "inkscape –help-all". Additional features can be accessed via Inkscape actions, which tap into the application's internals and provide more robust control of advanced features. Actions (and their options) can be listed by running "inkscape –action-list".

In version 1.4, Inkscape gains new actions for editing and exporting SVG files, including setting the SVG version, removing both proprietary and Inkscape-specific data, removing transforms from shapes, and various other settings and adjustments. It's also now possible to rotate pages by 90° via a new action. Additionally, Inkscape gains new options ("--export-png-compression" and "--export-png-antialias") for setting the compression level and antialiasing settings of exported PNG images.

Finally, Inkscape can now trace raster images from the command line via a new action, with many of the same options available from the GUI (in the "Trace Bitmap" dialog). Tracing is a process that makes an approximate reproduction of a bitmap image in vector format. This work is based on a project called "Inkscape For Trace", created by "nullmastermind" on GitHub.

What's next?

Features for Inkscape are chosen according to their maturity, after having been agreed upon between developers, and listed in GitLab milestones. This release sets the next stage for a transition to GTK+ 4, something which is made evident in the GUI by the use of popovers for context menus. According to Inkscape developer Martin Owens, the transition to GTK+ 4 was intended to occur with this cycle, but the development team was unable to land it in time for 1.4. Instead, the transition to GTK+ 4 is slated to occur with Inkscape 1.5, which will also bring a new color-management system, though true support for CMYK output is still in the works at this time. Instead, the new system will have support for "DeviceCMYK", which still uses the RGB color space underneath.

Inkscape 1.4 is set to be a significant milestone for the vector-graphics editor. Anyone interested in trying out the beta, or upcoming release candidate for Inkscape 1.4 can visit the download page and grab the Appimage file, which should run on almost any Linux distribution.

Comments (1 posted)