|
|
Subscribe / Log in / New account

Leading items

Welcome to the LWN.net Weekly Edition for December 5, 2024

This edition contains the following feature content:

This week's edition also includes these inner pages:

  • Brief items: Brief news items from throughout the community.
  • Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

NonStop discussion around adding Rust to Git

By Daroc Alden
November 22, 2024

The Linux kernel community's discussions about including Rust have gotten a lot of attention, but the kernel is not the only project wrestling with the question of whether to allow Rust. The Git project discussed the prospect in January, and then again at the Git Contributor's Summit in September. Complicating the discussion is the Git project's lack of a policy on platform support, and the fact that it does already have tools written in other languages. While the project has not committed to using or avoiding Rust, it seems like only a matter of time until maintainers will have to make a decision.

Taylor Blau opened the discussion in January by listing some of the advantages of Rust for the Git project: memory safety, lack of data races, easier refactoring (in the absence of unsafe code), and making it easier for more people to contribute to Git. "Given the allure of these benefits, I think it's at least worth considering and discussing how Rust might make its way into" Git's mainline. Blau also pointed out some questions that would need answering before the project could consider using Rust: how the language would affect platform support, which parts of Git would be easiest to migrate or extend (as is nature of sprawling mailing list discussions, the participants later touched on both), and how the change would interact with ongoing maintenance efforts.

Elijah Newren was excited about the idea, but suggested that one alternative would be to allow Rust only in optional components, in the same way that the kernel currently does. He volunteered to rewrite git-replay in Rust as a test case, noting that since (at the time of posting) it hadn't been included in a Git release, it wouldn't cause a regression. Newren was in favor of adopting Rust for different reasons than the ones Blau had listed in his message, however. He said that the Git developers often avoid making parts of the code execute in parallel, because of the difficulty of doing that correctly in C. Using Rust might allow for performance improvements from adding parallelism, and from switching to more efficient algorithms and data structures that are "quite onerous" in C. He specifically called out hash maps as a more performant alternative to the use of flat arrays in core Git code.

Other contributors were also open to the possibility. For example, brian m. carlson said "I'm very much in favour of this". He did think that there were serious obstacles to adopting Rust, however. He mentioned Rust's rapid release cycle as one such problem, but saw platform support as "probably the biggest stumbling point".

His main concern was with NonStop, a relatively niche platform primarily used in the financial sector for high-reliability computing. Earlier NonStop systems are based on Intel's Itanium architecture, but more recent systems have used x86_64. Both architectures run a proprietary operating system. The platform has its own C compiler, but many common tools, such as GCC, have not been ported. Git, however, has supported NonStop since Git 1.8.0 in 2012. That initial support was added by Joachim Schmitz, but Git doesn't have a policy for when and how that support should continue — as Emily Shaffer pointed out: "As far as I can tell, the support policy now is 'if you run make test on it and breaks, and you let us know, we'll try to fix it' - without much in the way of additional caveats."

Randall Becker, one of the most currently active NonStop maintainers, said that he thought Git's wide portability was actually one of its most important strengths. Git's policy of not relying on Linux-specific tools means that "we can take any developer from any platform and move them anywhere else without having to design new processes or teach them new processes for their workflows". In Becker's view, adopting Rust will "rapidly limit (or eliminate) git's long-term viability." He went on to suggest that the Git project should consider the impact of limiting platform support carefully, since it is such a fundamental piece of infrastructure:

What is the broad impact of dropping support for a platform that has a significant investment in git, where loss of support could have societal impact. There are platforms with 100 million to 1000 million lines of code managed by git today. This type of investment-related impact is specific to git compared to other Open-Source products. Leaving debit or credit card authorizers without a supported git would be, let's say, "bad".

Junio Hamano, the leader of the Git project since 2005, wasn't convinced, saying:

It depends but "investment-related impact" is unlikely cause for us to personally care. But those $CORPS who will feel the "investment-related impact" are welcome to hire quality developers and to these employed developers, the "impact" might become an issue they care more deeply about.

In another message, he did suggest that the project might consider treating niche platforms in the same way as platforms that package binary releases of Git — that is, requiring the platform to carry its own patches on top of stock Git. Newren listed the many problems he saw with establishing a platform-support policy that could guide the project in cases like this. Among other problems, he said that "we don't even know which platforms Git is used on". Newren also thought that any platform policy would need to address optional build components, including the several optional components that Git already has, such as Tcl support.

The idea of treating items written in rust as optional components, to be used only for non-essential uses, caused some doubts. Patrick Steinhardt expressed the opinion that once Rust had been permitted into any part of the Git code, it was likely to spread. He said that if Rust was accepted, "it will only be a matter of time before we want to start using it in central code paths".

Ultimately, the January discussion petered out with the topic unresolved. Blau implicitly reopened the topic in September when he posted the notes from the Git Contributor's Summit. Becker responded by noting that — while he does "not fundamentally object to Rust" — the contributors who are discussing it still have not answered whether or how Git could continue supporting NonStop as a platform if Rust were allowed. Porting Rust to NonStop is non-trivial, but beyond that, he said that depending on a single toolchain to determine which platforms are supported is "not responsible, in my view". Sean Allred pointed out that any platform Git supports already depends on outside toolchains. "Any platform-specific bugs in the various C compilers used are not in this project's control, either", he said.

Allred also thought that refusing to allow any new languages or libraries to be used for the project would essentially be a commitment to stop adding new features to Git:

It is hard to write correct, safe C. I worry that this difficulty will eventually translate to significant, safe contributions coming only from those with the resources to create them and not from a more diverse pool of contributors. As grateful as I am to large companies who fund Git contributions, I don't think those companies, given the choice in and of themselves, would think twice about dropping support for a platform they don't use — and that is probably a much smaller set than 'does the platform have GCC'.

Introducing Rust would be a pain for NonStop users, though, Allred acknowledged. Becker was not satisfied with that line of thinking. For one thing, he only objected to adding non-portable dependencies. For another: "Telling 10-20000 users that their core bit of infrastructure is insecure and not fixable is not a tenable position", he said. The fact that old versions of Git would remain usable on NonStop isn't good enough, when there's no guarantee or policy for how long it would take to backport security fixes, nor how long a version will remain supported.

Phillip Wood thought that it would always be possible for communities of interested users to backport their own security fixes, but agreed that adding Rust to the project was a tradeoff:

I believe that those advocating for using rust are doing so because they believe it will benefit both contributors and users. The problem we have to wrestle with is whether those benefits outweigh the cost to the relatively small proportion of users who do not have access to rust on their platform.

Becker has raised the possibility of porting Rust to NonStop with the platform vendor and promised to keep the mailing list updated, but any potential port will be at least a year away, assuming one is even possible. With this discussion failing to reach a firm conclusion either, it appears that the Git project is stuck with an ongoing tension between a small group of users, its legacy of broad support, and its role as core infrastructure on one hand, and increased security, performance, and ease of contribution on the other. Which way the project (and specifically Hamano, as its leader) will decide remains to be seen.

Comments (224 posted)

Arch Linux finally starts licensing PKGBUILDs

By Joe Brockmeier
November 26, 2024

Arch Linux is popular as a base for other Linux distributions; examples of Arch-derivatives include EndeavourOS, Manjaro, Parabola, and SteamOS. There's one small problem: the control files used to describe how to build packages for Arch Linux have no stated license. That creates a bit of uncertainty about the rights and responsibilities for the downstream derivatives. So far, that doesn't seem to have been a problem, nor has it stopped other projects from assuming that reuse is allowed. However, the Arch project is looking to add some clarity by explicitly assigning a liberal license to its package sources. Currently the project is in the process of reaching out to contributors to see if they have any objections.

An Arch Linux package is described using a PKGBUILD file, which is a Bash script used by the makepkg utility to compile the package source and make it ready to install using the pacman package manager. The Arch wiki has a good overview of the components and steps needed to create an Arch package.

RFC

What Arch Linux packages have not had in the 22 years since Arch was first released, is explicit licensing for the PKGBUILD files or other incidental scripts that are used to create packages. In August, Sven-Hendrik Haase took the first steps to correcting this by following the Arch RFC process and submitting an RFC that proposed licensing all Arch Linux package sources under the terms of the Zero-Clause BSD license. (The license is erroneously named, as it was modeled after the ISC License and not one of the BSD licenses.) This would include things like the PKGBUILD file, systemd files, and other files for installation on Arch and Arch-derived systems, but not change any licensing for the packaged software.

The RFC to clarify PKGBUILD licensing does not seem to have been prompted by a recent event, but the text of the RFC does note that people have raised concerns over the lack of license in the past.

In 2011, there was a lengthy discussion on Arch's aur-general mailing list about the legal status of files uploaded to the Arch User Repository (AUR), including PKGBUILDs, but that did not lead to any action. In 2020 Denis "GNUtoo" Carikli opened a bug report in the Parabola bug tracker looking to clear up the copyright status of Arch package sources. Activity in the bug report stalled until November 19, when it was updated about the RFC.

It is somewhat unusual for a major Linux distribution to have no policy regarding its package sources. Debian does not have an overarching policy on which license should be assigned to files used for building each package, but its packaging guidelines require a debian/copyright file included with each package (see example). That file describes the license assigned to every file in a package, including the packaging-related files under the debian directory.

Fedora considers the RPM spec files to be code, which are covered by the Fedora Project Contributor Agreement (FPCA). Unless declared otherwise, spec files for Fedora packages are given the MIT license by default, as noted in the FPCA for code contributions. Contributors can choose other licenses if they wish. For example the spec file for the php-smbclient package is explicitly licensed under the Creative Commons CC BY-SA 4.0. But in the absence of a declared license, there is a default license that is assigned.

The openSUSE project requires spec files to have a license header. It assigns the same license to its RPM spec files as the license of the package itself. So, if a package is GPLv3-licensed, the spec file is also considered GPLv3. The exception is if a package does not have an open-source license, then the spec file is assigned the MIT License. Gentoo's ebuild repository contains a header file with the GPLv2. Individual packages may also have copyright assignments to "Gentoo Authors" and list the GPLv2 as the license (example).

Is this really necessary?

Even though it is a common practice, some Arch contributors were skeptical of the need to license package-build files. The initial draft of the RFC would have had the project try to identify every developer who had contributed PKGBUILD files and any other build files, and ask their permission to license the contributions. The project would email each contributor and ask for a reply with this language: "I agree to provide all my contributions to the above packages under the terms of the Zero-Clause BSD license." If the developer did not agree, or they could not be reached within six months, then the project would decide if the contribution needed to be removed and rewritten. That, obviously, would entail a great deal of work on behalf of the project without dramatic benefits—and some contributors were not convinced the effort was necessary or desirable.

Christian Heusel asked why the Arch terms of service (ToS) did not apply when developers contributed PKGBUILD files. The ToS states that contributions to Arch projects are assumed to be under the GPLv3 if no other license is stated. Morten Linderud replied that the ToS is less than a decade old, and would not clear up the legal status of PKGBUILDs overall. Though he did not say so explicitly, many contributions may predate the ToS by several years.

User "Toolybird" said that they did not understand "all the fuss" and complained that the attempt to declare licenses for PKGBUILD files "smacks of corporate influence". They expressed disagreement that even complex build scripts should be subject to a license or copyright.

Allan McRae was more diplomatic, but still wondered whether it was a project worth doing. He noted that the only gain would be for distributions based on Arch Linux, all of which seemed to be working fine without the clarification. "Given the lack of contribution heading back upstream, is it worth 1,100 hours to address?" He suggested instead that the project contact all contributors to give them the opportunity to disagree, but not try to get an affirmative response from each one. Haase liked that idea, and incorporated it to the proposal.

Accepted

Once Arch RFC's are accepted, they are merged into the rfcs repository and made available on the Arch Linux RFCs page. After discussion, an RFC is finalized and an announcement is sent to the arch-dev-public list to start a 14-day final comment period. Haase announced the comment period on September 3, and the RFC's final draft merged on September 23. It calls for the project to scan all packages for #Maintainer and #Contributor tags to come up with a list of contributors to contact.

Rafael Epplée said that the project had gathered about 5,000 unique contributor emails. The project announced that emails would start going out, over the course of a week, on November 19. Contributors who are fine with the change need do nothing, the project will assume their consent to the license assignment and drop a LICENSE file into the package repositories after four months. In the event a contributor objects, the project will decide on a case-by-case basis what to do next.

Arch's approach may not be quite what a lawyer would recommend, but it seems like a reasonable and pragmatic approach to solve a longstanding problem. It seems unlikely that many, if any, Arch contributors will have strong objections to a permissive license being applied to package sources even if they don't believe it's wholly necessary. It will be interesting, four months from now, to see if any have objected at all, and (if so) what their reasons are for doing so.

Comments (52 posted)

The kernel's command-line commotion

By Jonathan Corbet
November 27, 2024
For the most part, the 6.13 merge window has gone smoothly, with relatively few problems or disagreements — other than this one, of course. There is one other exception, though, relating to the kernel's presentation of a process's command line to interested user-space observers when a relatively new system call is used. A pull request with a simple change to make that information more user-friendly ran afoul of Linus Torvalds, who has his own view of how it should be managed.

When one looks at a running process, often the first item of interest is which program that process is running. The kernel makes that information available in two different places, both found within a process's /proc directory:

  • comm holds the "command" that the process is running. When a program is launched with execve(), the base name of the executable file is placed in comm. So if the user runs /usr/bin/rogue, comm will contain "rogue".
  • cmdline contains the entire command line passed to the running program via the argv array given to execve(); it traditionally holds the program name, which is passed as the first argument in the command line.

The two files contain similar but different information, and have different access characteristics. comm will contain the actual file name that was passed to execve() (but keep on reading); it is also stored in a kernel data structure and can be accessed quickly. Instead, the command name in cmdline is whatever was passed to execve() as argv[0], which may or may not be the actual name of the command. It is stored in the relevant process's address space, meaning that the process itself can change it, and accessing it from another process is a relatively expensive operation. For these reasons, programs like ps or top will use comm instead of cmdline when possible.

As it happens, execve() is not the only way to launch a new program within a process on Linux. There is a library function called fexecve() that takes an open file descriptor for the program to execute rather than its path name; under the hood, it is implemented with execveat(). There is interest in using fexecve() because it allows the target program to be opened and checked (looking for a signature, for example), then executed in a race-free way. Tools like systemd have support for running programs this way, and its developers would prefer to use that mode.

There is just one little problem. While execve() can initialize comm from the name of the file passed to it, fexecve() only has an open file descriptor that no longer has any path-name information associated with it. That file descriptor may be marked "close on exec", meaning that even any information that may have been found in /proc/PID/fd will be lost. The result, in current kernels, is that, when a program is run with fexecve(), the comm is simply set to the file-descriptor number of the program. If rogue is run with fexecve() from file descriptor five, comm will contain "5" rather than "rogue".

Users, being the irascible creatures that they are, have expressed the unreasonable opinion that replacing the command names of processes in their system-management tools with small integers is an unwelcome change. They have been spoiled by being able to see which program each process is running and feel entitled to that ability in the future. Kernel programming would be so much easier without users, but that is not the world we live in. So the search for a better way to set comm when fexecve() is used was begun.

In a mid-November pull request, Kees Cook included a patch from Tycho Andersen that tried to restore some useful information to /proc/PID/comm in the fexecve() case. In the absence of the file name, the kernel would simply use the information from argv[0] instead, causing the information from comm and cmdline to be essentially the same. That patch had been through a few iterations, and seemed like a good solution to everybody involved.

Torvalds, though, disagreed, saying that "this horrible hack is too broken to live". Over the course of an extended and not always courteous discussion, he argued that argv[0] is under user-space control and can contain any sort of information; the kernel uses comm for its own purposes, and letting user space control it could help attackers to hide the actual executable being run. Copying argv[0] into comm will slow program start, he said. The right solution, according to Torvalds, is to use the file name stored in the directory entry ("dentry") associated with the file to be executed. That information is always present and is reliably under the kernel's control.

The problem with the dentry-based approach, as explained by Eric Biederman, Zbigniew Jędrzejewski-Szmek, and Cook, is that it would change the name of the executable as seen by user space. As an example, a user may run rogue, but some helpful distributor may have long since turned /usr/bin/rogue into a symbolic link to /usr/bin/nethack. On current systems, a tool like ps will show this user, busily at work, running rogue, but a comm value derived from the dentry would use the actual file being executed, so it would show as nethack instead. On some systems, like Debian with its "alternatives" mechanism, the visible names of quite a few commands would change. That could break programs that are looking for specific command names. Setting comm from argv[0], instead, would preserve the original name.

Torvalds, though, was unmoved by this argument. The dentry-based name, he said, is "THE TRUTH", and any program that wants to see argv[0] should just be using cmdline instead. Anybody who wants the behavior of execve() should just not use fexecve(), he added. The patch, as written, would not be pulled into the mainline. Cook tried one more time to explain why using argv[0] was desirable. It is the "friendlier" choice, he said, but if Torvalds was adamant that the dentry-based name must be used, that was going to be the result of the discussion. Torvalds responded: "no. THERE IS NO WAY I WILL ACCEPT THE GARBAGE THAT IS ARGV[0]".

This response seemed to indeed be somewhat adamant, so Cook has subsequently resent his pull request without the controversial patch, saying that the dentry-based approach would be implemented for the 6.14 merge window. Jędrzejewski-Szmek said that this approach could be worked with, "but we'll need to make an effort to warn users and do it much more visibly". There will, he said, inevitably be complaints from users whose scripts have broken.

In the end, this disagreement comes down to a small piece of the kernel's user-space interface that has existed almost since the beginning, but which has never been precisely specified. As with any user-visible behavior, programs have developed a reliance on the way things have traditionally worked, making newer approaches (such as execve() from a file descriptor) hard to implement without breaking things. There may be no ideal solution in this case, but it would have been nice if a workable solution could have been found with less shouting.

Comments (66 posted)

The beginning of the 6.13 merge window

By Jonathan Corbet
November 21, 2024
As of this writing, just over 1,800 non-merge changesets have been pulled into the mainline kernel for the 6.13 release. That number may seem small, given that a typical merge window brings in at least 12,000 commits, but the early pulls this time around have focused on significant core changes, and there are quite a few of them. The time has come to summarize the changes pulled so far, including lazy preemption, multi-grained timestamps, new extended-attribute system calls, and more.

Some of the most significant changes pulled in the first part of the 6.13 merge window include:

Architecture-specific

Core kernel

  • The PIDFD_GET_INFO ioctl() operation, which will fetch information about a process represented by a pidfd, has been merged.
  • The io_uring subsystem has a new command, IORING_REGISTER_RESIZE_RINGS, that allows on-the-fly resizing of the submission and completion rings. This allows applications to start with a pair of relatively small rings, and grow them later should the need arise.
  • The lazy preemption patches have been merged. This work greatly simplifies the kernel's preemption logic (and configuration options) while maintaining performance for throughput-oriented configurations. It is a significant change that should, eventually, reduce the amount of scheduling-related logic scattered through the non-scheduler parts of the kernel.
  • Some preliminary work needed to implement proxy execution, an improved approach to the avoidance of priority inversion, has been merged. The proxy execution feature itself, though, has not yet landed.

Filesystems and block I/O

  • There have been a few tries to implement fine-grained timestamps for file metadata; the last one ran aground at the end of 2023. Another attempt is being made for 6.13; this merge message contains the details of how it works.

    In short: as before, filesystems only need to track fine-grained change times for a given file if the time is being actively queried; most of the time, low-resolution timestamps are good enough. That is important, since lower-resolution timestamps do not need to be written back to persistent storage as frequently. The previous implementation ran into problems, though, where a low-resolution timestamp could appear to be earlier than a high-resolution timestamp, even though the actual changes happened in the opposite order.

    In the new implementation, the kernel remembers the last fine-grained timestamp that was given out and ensures that any coarse-grained timestamps assigned for file modifications are later than the that last fine-grained value. This technique avoids the above-mentioned problem, ensuring that timestamps always correctly reflect the order in which files were modified.

    See this documentation commit for more information.

  • There is a new sysctl knob, fs.dentry-negative, that controls whether the virtual filesystem (VFS) layer deletes a file's kernel-internal directory entry ("dentry") when the file itself is deleted. It seems that some benchmarks do better when dentries are removed, while others benefit from having a negative dentry left behind, so the kernel developers have put the decision into the system administrator's hands. The default value (zero) means that dentries are not automatically deleted, matching the behavior of previous kernels.
  • The statmount() system call has gained options to return the filesystem subtype, superblock source, and security mount options. There is also a new flag, STATMOUNT_OPT_ARRAY, that returns filesystem options as a series of NUL-separated strings and without the usual "\000" escaping.
  • There have been some deep reference-counting changes within the VFS layer that yield a 3-5% performance improvement on highly threaded workloads; see this merge message for some details.
  • It is now possible to assemble an overlayfs stack using file descriptors rather than path names; see this merge message for details.
  • The tmpfs filesystem can now be mounted in a case-folding mode where file names are no longer case-sensitive. See this documentation commit for the relevant mount options.
  • Limited support for atomic write operations has been added to the ext4 and XFS filesystems.
  • There is a new set of system calls for the management of extended attributes: setxattrat(), getxattrat(), listxattrat(), and removexattrat(). They are variants of setxattr(), getxattr(), listxattr(), and removexattr() that include a directory file descriptor as the starting point for the path-name search.
  • The new BTRFS_IOC_SUBVOL_SYNC_WAIT ioctl() command for the Btrfs filesystem will wait for the cleaning of one or more subvolumes. It is an unprivileged operation, and is intended to allow the "btrfs subvolume sync" command to work without privilege.
  • Btrfs now supports performing encoded reads (reading of compressed extents directly, without decompression) via io_uring.

Hardware support

  • Hardware monitoring: Renesas ISL28022 power monitors and Nuvoton NCT7363Y fan controllers.
  • Miscellaneous: Marvell PEM performance-monitoring units, Airoha true HW random number generators, Broadcom BCM74110 random number generators, Renesas RZ/V2H(P) interrupt control units, and THEAD C9XX ACLINT S-mode IPI interrupt controllers.

Miscellaneous

  • There is a new user-space API allowing administrators to set thermal thresholds on specific devices; notifications will be generated when a threshold is crossed. This commit gives an overview of the functionality, but the actual (netlink-based) API is uncompromisingly undocumented.

Security-related

  • The SELinux security module can now manage policies for individual netlink operations; see this commit message for a terse overview.
  • The /sys/fs/selinux/user configuration knob has been deprecated and will be removed in a future release.

Internal kernel changes

  • There are now Rust abstractions for a number of VFS data structures and interfaces — enough to support the Rust implementation of binder.
  • The bulk of the file-descriptor memory-safety work has been merged.
  • The kernel's cryptographic subsystem has gained a new internal API for signature generation. There is some kerneldoc documentation available.
  • There is a new variant of the sleepable RCU ("SRCU") API that makes the read side cheaper at the cost of more expensive write operations. Documentation for the new functions — srcu_read_lock_lite() and srcu_read_unlock_lite() — can be found by reading through this commit.
  • The debugobjects subsystem has been massively reworked for better performance and robustness; see this merge message for details.
  • The venerable dontdiff file has been removed from the documentation tree.

The 6.13 merge window can be expected to remain open through December 1. That closing date is immediately after a significant holiday weekend in the US, but past experience suggests that the 6.13-rc1 release will come out on schedule regardless. LWN will have an update of the remaining changes from this merge window once it closes.

Comments (5 posted)

The rest of the 6.13 merge window

By Jonathan Corbet
December 2, 2024
The 6.13 merge window closed with the release of 6.13-rc1 on December 1. By that time, 11,307 non-merge commits had been pulled into the mainline repository; about 9,500 of those landed after our first-half merge-window summary was written. There was a lot of new material in these patches, including architecture-support improvements, new BPF features, an efficient way to add guard pages to an address space, more Rust support, a vast number of new device drivers, and more.

The most interesting changes merged during the latter half of the 6.13 merge window include:

Architecture-specific

  • Support for PowerPC "Maple" platforms has been removed. According to this commit: "Sadly all those machines have passed into memory, and there's been no evidence for years that anyone is still using any of them".
  • The Loongarch architecture has gained support for realtime preemption and lazy preemption.
  • The kernel now supports the "Smmpm", "Smnpm", and "Ssnpm" RISC-V pointer-masking extensions; this commit contains some documentation.

Core kernel

  • There is a new kmem_cache iterator that enables BPF programs to obtain information about the state of the slab allocator. There is no documentation, but this merge message gives an overview, and this commit contains a test program using this feature.
  • The new bpf_send_signal_task() kfunc allows a BPF program to send a signal to another process.
  • There is a new BPF map feature that allows threads to share some memory with a BPF scheduler (implemented with sched_ext). The idea is to share hints about the workload that will let the scheduler do a better job. The feature is undocumented; see this merge message for an overview, and this commit for a test program.
  • BPF programs can now have a private stack, reducing the risk of overflows in long call chains. This merge message contains an overview of the feature and its motivation. Only x86-64 systems support this feature for now.
  • Support has been added to allow system-call entry and exit tracepoints to incur page faults in order to read user-space parameters.
  • There is a new madvise() operation, MADV_GUARD_INSTALL, that installs "guard" pages in the indicated address range; an attempt to access a guard page will result in a SIGSEGV signal. These guard pages are a more efficient way to prevent code from running off the end of an allocation than PROT_NONE mappings (the only current way to do that) because it does not require the allocation of a new virtual memory area. MADV_GUARD_REMOVE will remove existing guard pages. See this commit for a bit more information.
  • The new transparent_hugepage_shmem= command-line parameter controls the policy for the allocation of transparent huge pages in the shmem/tmpfs filesystems; see this commit for some more information.

Filesystems and block I/O

  • The EROFS filesystem has gained support for the SEEK_HOLE and SEEK_DATA lseek() options.
  • As expected, the reiserfs filesystem has been removed from the kernel.
  • The F2FS filesystem has a new feature, called "device aliasing", that allows a portion of a block device to be set aside. "Essentially, this mechanism allows a portion of the f2fs area to be temporarily reserved and used by another filesystem or for different purposes. Once that external usage is complete, the device aliasing file can be deleted, releasing the reserved space back to F2FS for its own use." See this commit for a documentation update.

Hardware support

  • Clock: Virtual machine PTP clocks, MediaTek MT6735 clocks, Nuvoton Arbel BMC NPCM8XX clock controllers, Mobileye EyeQ clock controllers, Renesas VBATTB clock controllers, Qualcomm SA8775P camera, display, and video clock controllers, Qualcomm SAR2130P global and graphics clock controllers, Qualcomm IPQ5424 global clock controllers Qualcomm QCS8300 global clock controllers, Renesas RTCA-3 realtime clocks, Marvell 88PM886 realtime clocks, and Amlogic A113L2(A4) and A113X2(A5) SoC realtime clocks.
  • GPIB (general purpose interface bus) driver support was added to the staging tree; new drivers support National Instruments PCI/ISA compatible boards, Agilent 82357a/b USB dongles, Capital Equipment PCI-488 and Keithly KPCI-488 boards, Measurement Computing CPCI_GPIB, ISA-GPIB, ISA-GPIB/LC, PCI-GPIB/1M, PCI-GPIB/300K and PCMCIA-GPIB boards, fmhess FPGA-based devices, Raspberry Pi GPIO adapters, HP Agilent HP82335, HP27209 and HP27209 boards, numerous PC2 and PC2A-compatible adapters, TMS 9914 GPIB interfaces, and NEC 7210 GPIB interfaces.
  • GPIO and pin control: Congatec GPIO and I2C controllers, FTDI MPSSE GPIO interfaces, Microchip FPGA GPIO controllers, T-Head TH1520 SoC pin controllers, Qualcomm IPQ5424, QCS8300, SM8750, SAR2130P, and QCS615 pin controllers, Canaan Kendryte K230 pin controllers, SpacemiT K1 SoC pin controllers, and Airoha EN7581 pin controllers.
  • Graphics: TI TDP158 HDMI/TMDS bridges, Samsung S6E3HA8 DSI video mode panels, Samsung AMS639RQ08 and AMS581VF01 panels, Samsung AMS427AP24 panels with S6E88A0 controllers, Sharp Memory LCD panels, and ITE IT6263 LVDS/HDMI bridges.
  • I2C: AMD ASF I2C controllers and Realtek RTL9300 I2C controllers.
  • Industrial I/O: Bosch BMI270 6-Axis IMUs, Analog Devices AD8460 and AD3552R digital-to-analog converters, GE HealthCare PMC analog-to-digital converters, Vishay VEML6035 ambient light sensors, Analog Devices AD7625/AD7626 high-speed analog-to-digital converters, Bosch Sensor SMI240 IMUs, Analog Devices AD7779 analog-to-digital converters, Vishay VEML3235 ambient light sensors, and Allegro MicroSystems ALS31300 3-D linear Hall-effect sensors.
  • Input: Corsair Void headsets.
  • Media: Raspberry Pi RP1 Camera Front End (CFE) video capture interfaces.
  • Miscellaneous: Realtek SPI-NAND flash controllers, Apple SoC SPI controllers, Congatec watchdog timers, UHS2-capable devices on SDHCI controllers, AMD 3D V-Cache performance optimizers, MediaTek dynamic voltage and frequency scaling resource collectors, Microchip LAN966x PCIe controllers, Marvell 88E6xxx network interface LEDs, RISC-V I/O memory-management units, T-head TH1520 mailboxes, Marvell OCTEON PCI hotplug controllers, STMicroelectronics STM32MP25 COMBOPHYs, NXP PTN3222 1-port eUSB2 to USB2 redrivers, TI OMAP TWL6030 battery charge interfaces, Renesas R-Car Gen4 E-FUSEs, and Qualcomm QCS8300, SAR2130P, and QCS615 interconnect providers.
  • Networking: Hisilicon BMC gigabit Ethernet interfaces, Microchip Lan969x switches, NXP i.MX95 ENETC interfaces, Marvell RVU E-Switch switches, Realtek 8821AU/8811AU and 8812AU USB wireless network adapters, and KEBA CP500 LAN9252 interfaces.
  • Sound: NeoFidelity NTP8918, NTP8835, and NTP8835C amplifiers, Realtek RT721 SDCA codecs, Everest Semi ES8323 codecs, NXP UDA1342 codecs, Cirrus Logic CS42L84 codecs, Awinic aw88081 amplifiers, and Iron Device SMA1307 audio amplifiers.
  • USB: TI TUSB1046 Type-C crosspoint switches.

Miscellaneous

Networking

  • Networking performance in many cases is constrained by contention on the RTNL lock — a problem that has been understood for years. The 6.13 kernel contains work turning the RTNL lock into a per-network-namespace lock as a way of reducing that contention in namespace-heavy workloads (running containers, for example). It is just one step in a long process, and the regression potential is considered to be significant, so the per-namespace behavior is disabled by default. The DEBUG_NET_SMALL_RTNL configuration variable can be used to turn it on.
  • There is a new, more flexible API for the control of traffic shaping; this merge message gives an overview of how it works.
  • There is a new netlink API for the control of NAPI (polling) behavior in network drivers; this merge message provides an overview.
  • The long-obsolete wireless extensions emulation within the kernel will no longer export a set of symbols to loadable modules. No in-tree drivers need that feature at this point; it is possible that out-of-tree drivers might break.
  • There is a new mechanism allowing network drivers to suspend interrupts during busy-traffic periods; see this merge message for more information.

Virtualization and containers

  • The IOMMUFD subsystem has gained a number of new operations aimed at better support for virtual machines; see this merge message for an overview.
  • The sound subsystem has a new pass-through mode for audio-related accelerators; this commit includes an overview document.
  • The new virtual CPU-frequency driver allows CPU-frequency coordination between hosts and guests; see this commit for a little more information.

Internal kernel changes

  • There is a new memory allocator for Rust code that removes the dependency on the external Alloc module. Internal documentation for the associated Allocator trait can be found in this commit. There are also kernel-specific implementations of Box and Vec that use this allocator; see this commit for some internal Box documentation and this commit for Vec.
  • There were 231 exported symbols removed in 6.13, while 443 new exports were added; there are also 11 new kfuncs. See this page for the full list.

The 6.13 kernel development cycle now moves into the stabilization phase where, with luck, the bulk of the bugs that have inevitably been introduced will be fixed. The 6.13 release can be expected on January 19 or 26.

Comments (10 posted)

The return of RWF_UNCACHED

By Jonathan Corbet
December 4, 2024
Linux offers two broad ways of performing I/O to files. Buffered I/O, which is the usual way of accessing a file, stores a copy of the transferred data in the kernel's page cache to speed future accesses. Direct I/O, instead, moves data directly between the storage device and a user-space buffer, avoiding the page cache. Both modes have their advantages and disadvantages. In 2019, Jens Axboe proposed an uncached buffered mode to get some of the advantages of both, but that effort stalled at the time. Now, uncached buffered I/O is back with some impressive performance results behind it.

By saving data in the page cache, buffered I/O can accelerate many I/O operations. Cached data need not be reread from the storage device, and multiple write operations can be combined in the cache, reducing the number of writes back to persistent storage. But that caching comes at a cost; the page cache is typically the largest user of memory on a Linux system, and the CPU must spend time copying data to and from the cache. Direct I/O avoids this memory use and copying, but it is inherently synchronous, adds complexity to a program, and provides a number of inconvenient pitfalls, especially in cases where a file is accessed concurrently. Developers normally only reach for direct I/O if they really need it.

Still, as Axboe describes in the patch-set cover letter, users are often driven toward direct I/O, despite its challenges. That pressure is especially acute in cases where the data being transferred will not be needed again. Storing unneeded data in the page cache costs memory, but the problem is worse than that. Even though once-accessed data is put on the kernel's inactive list, meaning that it will be the first to be reclaimed when free memory runs low, the kernel must still make the effort to process that list and reclaim pages from it. With the right sort of I/O load (randomly accessing a set of files much larger than the system's RAM, for example), much of the available CPU time can be taken by the kernel's kswapd threads, which are working simply to reclaim memory from the page cache.

The solution that he came up with in 2019 was to add a new flag, RWF_UNCACHED, for the preadv2() and pwritev2() system calls. When that flag is present, those calls will perform I/O through the page cache as usual, with one exception: once the operation is complete, the relevant pages are immediately deleted from the page cache, making that memory available to the system without the need to go through the reclaim process. In 2019, the work then wandered into an attempt to avoid the page cache entirely, to get closer to direct-I/O performance, before coming to a stop.

The new series picks things up again, returning to transferring data by way of the page cache and removing it afterward. For read operations, the data will be removed from the page cache as soon as it is copied into the user-space buffer (with the exception that, if it was already resident in the page cache prior to the operation, it will be left there afterward). Going through the page cache in this way avoids the coherency pitfalls that come with using direct I/O.

Writes work like buffered writes do now; the data will be written to the page cache, and the pages will be marked for eventual writeback to persistent storage. Once that writeback completes, the pages will be removed from the page cache (except, again, in cases where they were resident there prior to the operation starting). In the meantime, though, multiple writes can be combined into a single writeback operation, maintaining I/O performance.

Since the last time around, the kernel's file-I/O infrastructure has improved somewhat, to the point that much of the work of supporting RWF_UNCACHED can be performed in the kernel's iomap layer. Filesystems that use iomap fully will get RWF_UNCACHED support almost for free. Filesystems that use less generic code, including ext4, require a bit more work. The patch series includes the needed changes for ext4; XFS and Btrfs are supported as well.

The effect of these changes can be seen in the associated benchmark results. For the read side, Axboe included results showing how, in the absence of RWF_UNCACHED, a system performing a lot of random reads will bog down once memory fills and reclaim begins. At that point, nearly 28 cores on the 32-core test system are busy just running kswapd full time. With RWF_UNCACHED, that bogging-down no longer happens, and kswapd does not appear among the top CPU-using processes as all. In summary: "Not only is performance 65% better, it's also using half the CPU to do it". The write-side results are almost the same.

Most of the responses to this work have been positive; XFS developer Darrick Wong, for example, said that "there's plenty of places where this could be useful to me personally". Dave Chinner (also an XFS developer) is less convinced, though. He argued that, rather than adding a new flag for preadv2() and pwritev2(), Axboe should add the desired behavior to existing features in the kernel. Specifically, he said, the POSIX_FADV_NOREUSE flag to posix_fadvise() is meant to provide that functionality. Axboe, though, disagreed, saying that it is better to specify the desired behavior on each I/O operation than as an attribute of an open file:

Per-file settings is fine for sync IO, for anything async per-io is the way to go. It's why we have things like RWF_NOWAIT as well, where O_NONBLOCK exists too. I'd argue that RWF_NOWAIT should always have been a thing, and O_NONBLOCK is a mistake. That's why RWF_UNCACHED exists.

In other words, the O_NONBLOCK flag to open() puts the election of non-blocking behavior in the wrong place. Rather than attaching that behavior to the file descriptor, it should be selected for specific operations. RWF_UNCACHED is a way to easily get that asynchronous behavior when needed.

The discussion has since wound down for now, doubtless to be revived once the next version of the series is posted to the mailing lists. There would appear to be enough interest in this feature, though, to justify its merging into the mainline. It is too late to put uncached buffered I/O support into 6.13, but the chances of it showing up in 6.14 seem reasonably good.

Comments (27 posted)

Rust's incremental compiler architecture

By Daroc Alden
December 3, 2024

The traditional structure of a compiler forms a pipeline — parsing, type-checking, optimization, and code-generation, usually in that order. But modern programming languages have requirements that are ill-suited to such a design. Increasingly, compilers are moving toward other designs in order to support incremental compilation and low-latency responses for uses like integration into IDEs. Rust has, for the last eight years, been pursuing a particularly unusual design; in that time compile times have substantially improved, but there's still more work to be done.

In a traditional compiler, normally each step of the pipeline needs to complete before the next is started. For example, optimization must complete before the compiler can begin emitting machine code. There are ways to improve this — working on translation units in parallel, caching some kinds of analysis, etc. — but, at a certain point, the architectures of the compilers themselves make it difficult to make the code support incremental compilation.

Language design can influence how much of a problem this is in practice. For example, while the GCC developers did look into making the compiler incremental, the fact that C defines each file as a stand-alone translation unit makes it easier to add incrementalism at the level of the build system. Many modern languages don't observe the same separation between files, however. In Rust, an entire crate is a single translation unit; cargo, Rust's build system, handles only recompiling changed crates, but adding incrementalism within a crate requires special support from the compiler.

So, in response to wishes for faster compile times, Rust has been exploring an alternative to the traditional structure of a compiler by switching to a query-based model: instead of having a fixed pipeline, the compiler has a set of queries for different properties of the program. Each query can either retrieve the answer from a persistent compilation database, or, if the answer has not yet been cached, calculate it from scratch by making other queries and combining their results. For example, the compiler uses the type_of() query to infer the types of local bindings. The query takes the ID of a local definition, and then calls other queries to determine whether it is an impl Trait type, what the abstract syntax tree of the definition is, what the types of the variables it refers to are, and so on.

Suppose that the programmer changes the type of a constant in their crate. When they run the compiler, it will evaluate a top-level query to produce an output binary. That query will in turn query for the final versions of each function in the crate (after type-checking and optimization). Most of those will not have changed, so the answers are provided from the on-disk compilation database. The queries for functions that depend on the changed constant, however, will get re-run — and will, themselves, call other queries until the whole process ends up at the query corresponding to the parsed representation of the single changed definition. Only the minimum amount of analysis and compilation needed to accommodate that change is performed.

This approach is reminiscent of a build system — and that's deliberate. Build systems, while not always perfect, usually do a good job of tracking the dependencies between components and correctly caching partial results. A compiler that has the architecture of a build system can take advantage of the research that has been done on build systems to do the same.

Rust's approach

The Rust project began the process of rewriting its compiler to support incremental compilation in 2016. While that redesign is not yet fully complete, large portions of the compiler now use a query-based approach. LWN recently covered Sparrow Li's talk that discussed what remains to be done.

The Rust compiler organizes queries using a structure called TyCtxt. When the compiler starts up, the various modules all register functions called "providers" that implement each query. Then, depending on how the compiler was invoked, it parses the code (because the parser has not yet been integrated into the query system) and runs a set of top-level queries to compile an artifact, format the code, produce warnings, or whatever else was requested.

Providers need to be pure functions, since the compiler may not always need to run them, so the programmer can't depend on any side effects. This also has the effect of forcing those parts of the compiler that have been converted to the query system to avoid global state. The results of queries could in theory be any type, but in practice the compiler requires that they be values that can be cheaply copied, in order to simplify the ownership of cached values. Not every query is saved to disk, but most queries return types that can be serialized and hashed, so that they can be.

There are some important differences in structure between Rust's queries and static build systems like Make, not just in the form that their results take. In Rust's implementation, the relationship between queries is implicit. Queries are always invoked via a TyCtxt; that structure can track the dependencies between queries by seeing which queries are invoked while another is executing. Also, unlike some build systems, the compiler tracks the order in which dependencies are invoked. This is important because the results of one query can change which other queries get executed — therefore, the order impacts the calculation of which queries need to be rerun when doing incremental compilation. For example, this query depends on different things depending on the result of subquery1(): 

    fn example_query<'tcx>(tcx: TyCtxt<'tcx>) -> SomeType {
        if tcx.subquery1() {
            tcx.subquery2()
        } else {
            tcx.subquery3()
        }
    }

During the initial run of the compiler, when there are no saved results, the program collects the graph of which queries refer to each other. When the compiler is rerun, it checks that graph to see which queries need to be rerun because of changed inputs. As an additional optimization, if the inputs to a query have changed, but the result of the query has not, the compiler can detect that and cut off the process of recalculating queries early.

The process of figuring out whether an intermediate result has changed is complicated by the fact that any information about the incremental build needs to be shared between runs of the compiler by saving it to disk. The compiler uses a lot of internal ID numbers to organize different components that differ from run to run, and serializing every intermediate result in the compiler would be expensive. So actually saving the results of cached queries is more complex.

In order to work around the problem of changing internal IDs, structures that are saved to disk are given "stable" forms of each ID. For example, function and type definitions are identified by path (e.g. std::collections::HashMap) instead of by internal ID. In order to avoid the overhead of deserializing intermediate results to detect changes, the compiler calculates and stores hashes for each item. Hashes are much more compact, and don't require complex deserialization, so this is a substantial performance improvement. The downside is that computing so many hashes actually causes significant overhead. The compiler's internal documentation calls it "the main reason why incremental compilation can be slower than non-incremental compilation."

These performance improvements introduce their own complications, of course. For example, if a result is never loaded from disk, the compiler won't necessarily be able to include it in the incremental-compilation database for the next run of the program. So, after generating results, the compiler does another pass through the query graph to fetch and update the results that it skipped earlier, in order to make sure that the size of the cache doesn't unnecessarily shrink. This increases the compiler's overall runtime — but what users usually care about is the time it takes to produce warnings and errors, not how long it takes the compiler to do cleanup before exiting. There are various other optimizations and tweaks applied to the largest queries to make the overall system more performant. The Rust Compiler Development Guide's section on the query system has more details.

Overall, the query-based approach introduces a lot of complexity compared to a simple pipeline-based approach. But any large, performance-sensitive codebase accumulates complexity over time — and having a single, well-understood system for incremental compilation may well win out over gradually introducing ad-hoc improvements to a pipeline-based system. In any case, Rust's incremental compilation has been stable enough to become the default for development builds. Release builds still avoid incremental compilation by default, however, out of an abundance of caution. Whether the query-based compiler design will ultimately be adopted by other languages remains to be seen.

Comments (16 posted)

GIMP 3.0 — a milestone for open-source image editing

November 28, 2024

This article was contributed by Roland Taylor

The long-awaited release of the GNU Image Manipulation Program (GIMP) 3.0 is on the way, marking the first major update since version 2.10 was released in April 2018. It now features a GTK 3 user interface and GIMP 3.0 introduces significant changes to the core platform and plugins. This release also brings performance and usability improvements, as well as more compatibility with Wayland and complex input sources.

Modernized interface

GIMP 3.0 is the first release to use GTK 3, a more modern foundation than the GTK 2 base of prior releases. GTK 4 has been available for a few years now, and is on the project's radar, but the plan was always to finish the GTK 3 work first. Moving to GTK 3 brings initial Wayland compatibility and HiDPI scaling. In addition, this allows for GIMP users to take advantage of multi-touch input, bringing pinch-to-zoom gestures to the program, and offering a better experience when working with complex peripherals, such as advanced drawing tablets. These features were not previously possible due to the limitations of GTK 2.

[Welcome screen]

A secondary result of the transition to GTK 3 is a refreshed user interface (UI), now with support for CSS themes included. In this release, four themes are available by default, including light, dark, and gray themes, along with a high-contrast theme for users with visual impairments. Additionally, this release has transitioned to using GTK's header bar component, typically used to combine an application's toolbar and title bar into one unit. To maintain familiarity with previous releases, however, GIMP 3.0 still supports the traditional menu interface.

When GIMP 3.0 is launched, users will be greeted with a new welcome screen, seen on the right. This dialog presents various useful links for the GIMP project, such as tutorials, documentation, and donation options. It also provides users with quick functions for setting up GIMP as needed before starting their work. This includes the option to choose GTK and icon themes, and for placing the menu in the title bar, saving vertical space. In the "Create" tab of the dialog, users can create a new image, select from their recent work, or open an existing image from the filesystem. The welcome screen can be disabled if desired.

Better workflow, performance, and color management

A major focus of this release is greater integration with the Generic Graphics Library (GEGL), first introduced in 2000 for the purpose of improving GIMP's image-processing capabilities through a scene-graph-based architecture. As part of this effort, there have been numerous optimizations to GIMP's core and to its standard plugins. In tandem with memory management and multi-threading improvements, these changes should bring significant speed boosts when applying filters and effects, even on larger images.

GEGL allows image-processing operations to be chained together in such a way that the original image data is preserved, along with a record of every edit. This is referred to as non-destructive editing, and GIMP 3.0 is the first stable release of the project to make this workflow available, though there is still more work to be done. Users can apply filters and effects to any layer without altering the original image. As a result, effect parameters can be changed even after they've been applied. This change removes the need to perform an undo any time a filter or effect does not have the desired result. Filters, and any plugins that use GEGL operations, now offer real-time previews.

[Split preview]

GIMP 3.0 continues efforts to improve color management under the initiative referred to as "space invasion", and delivers significant color-correctness results. This will improve the consistency of color reproduction across different devices and workflows. The foundation of these improvements is the babl library, responsible for handling pixel-format conversions and color-space management. Color profiles are now automatically managed when opening files with an included color profile. In the 2.10.x series, manual intervention was required when loading these files.

GIMP 3.0 supports palettes outside of the "Standard Red Green Blue" (sRGB) range, such as "Cyan Magenta Yellow Key" (CMYK) and (CIELAB). This expanded color support, especially for CMYK, is essential to those who work with print and desktop publishing. However, GIMP continues to use sRGB, grayscale, and indexed colors for storing color information internally for now. Conversion to other color spaces is done on output, where necessary.

Version 3.0 delivers CMYK output for some file formats, specifically JPEG, TIFF, and Photoshop (PSD). When working with these color formats, users often need to use soft-proofing, a digital preview of how colors will look when printed, helping ensure color accuracy in the final output. GIMP's 3.0 soft-proofing workflow has been improved, and it saves this information in its XCF format, preserving settings between sessions.

Layers and file-format improvements

The layer workflow has been upgraded in GIMP 3.0, with new features that bring greater parity with other advanced photo editors such as Adobe Photoshop and Affinity Photo. Layer operations can now be applied in bulk, and multiple layers can be selected and grouped. Layers can even be moved, reordered, duplicated, merged, and deleted en masse, whether or not they are contiguous within the layer dialog. Each layer now displays a distinct "Fx" icon to represent when filters and effects are in use. Filters and effects can be managed from a popover, which is opened up by clicking on this icon.

[Layers dialog]

The developers intend to further improve the layer-effects workflow with tighter integration of these features and a better UI. In future releases, GIMP's non-destructive functionality will be extended to layer masks, and layer channels as well. For users who wish to use a destructive workflow (where filters are applied to the original image data), there is a "Merge Filter" option available in the filter dialog.

Another new feature in GIMP 3.0's layer workflow is auto-expanding layers. Layers may need to extend outside of the canvas in order to preserve parts of an image that need to later be moved or transformed, or within animations, where part of the animation needs to be made invisible temporarily. An auto-expanding layer will extend beyond its borders when painting outside of the canvas, though the canvas itself will retain its size. To enable this feature, users will need to check the "Expand Layers" option on any brush tool. Layer masks can be set to expand with the layer; the "Align And Distribute" tool has been completely reworked to recognize the layer's contents and not just its boundaries. Improvements to guides, snapping, and the text editor also help make GIMP's layer workflows more appealing in a way that will be familiar to users of other tools in this space.

This update brings improved support for many image formats. PSD images now preserve their layer order on import, and Paintshop Pro images (PSP) preserve several features, including International Color Consortium (ICC) profiles, grids, and guides. In a crucial bug fix for working with GIFs, GIMP 3.0 detects whether these files contain animations, and correctly saves animated GIFs when overwriting existing files.

New image formats have been added, with import and export supported for each format. The highly versatile SwatchBooker palette format, SBZ, is now supported, bringing parity with Scribus and Krita. Unlike most other palette formats, SBZ files can save complex details such as layouts, textures, gradients, named colors, color-space information, and even multiple palettes in a single file. For processing raw images (formats that preserve unprocessed image data), GIMP has long relied on external applications such as darktable and RawTherapee. However, recent changes in darktable's API broke this integration, causing GIMP to be unable to detect and recognize if it was installed. To rectify this issue, dedicated darktable integration is now available thanks to a collaborative effort between the two projects.

This update introduces a new extension system and file format, GEX, allowing for easier distribution of plugins, themes, brushes, and other means of extending and supplementing the core application. This extension system even allows for multiple features, such as plugins, brushes, and themes, to be packaged and managed simultaneously. For example, a project such as PhotoGIMP could use this system to completely transform GIMP's standard functionality with a single package.

In addition, an extension manager has been developed and is available in this update. It's not immediately exposed to users, however, because the backend infrastructure for distributing extensions is still under development. To access it, users will need to use GIMP's command search, which can be found in the menu under "Help>Search and Run a Command". Once extension support has matured, it will allow managing other features on the fly.

Future developments

Apart from the myriad user-visible features, several under-the-hood changes will either land in this release or come to fruition in near future releases. GIMP 3.0 contains the infrastructure for non-destructive layer types (such as vector layers and link layers), animations in the core application, multi-page files, and native support for working in color spaces outside of the RGB range.

The changes in GIMP 3.0 have necessitated an API break so it is no longer compatible with plugins from older releases. To lessen the impact, the developers have decided to couple this shift with the incompatible changes of switching to GTK 3. Because of this, the development of the 2.10.x branch has continued, with some features being backported from GIMP 3.0 during this development cycle. Future releases within the 2.10.x branch are still possible, but unlikely, with the foundation of the 3.x series reaching maturity. Do note, however, that most of GIMP's existing plugins will need to be updated to support the new API, which is expected to provide developers with the benefit of increased functionality and improved performance.

These are just some of the changes coming to GIMP in version 3.0. Exactly when GIMP 3.0 will be released remains to be seen, but users interested in trying the new features before release can download development builds; updates are regularly posted on the GIMP web site. The first release candidate has already landed on November 6, following the project reaching string freeze on the main branch. So long as there are no major bugs or regressions, the final release will be built from the release candidate. However, additional candidates may be necessary if any major problems are found. The developers have already announced that subsequent releases should land faster, with smaller, feature-focused releases providing more stability. Overall, GIMP 3.0 represents a big step forward for the free photo and graphics editor, and sets the stage for even greater improvements to come.

Comments (42 posted)

Page editor: Jonathan Corbet
Next page: Brief items>>


Copyright © 2024, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds