Leading items

Welcome to the LWN.net Weekly Edition for October 1, 2020

This edition contains the following feature content:

• LVFS tames firmware updates: after 20 million downloads, the Linux Vendor Firmware Service has come into its own.
• Toward a "modern" Emacs: what will it take to bring new users to this venerable editing platform?
• OpenWrt and SELinux: bringing advanced security to a router distribution.
• Saying goodbye to set_fs(): a long-term kernel improvement project reaches its end.
• Mercurial planning to transition away from SHA-1: Git is not the only source-code management system with SHA-1 problems.
• New features in the fish shell: an update on seven years of fish shell development.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

LVFS tames firmware updates

Keeping device firmware up-to-date can be a challenge for end users. Firmware updates are often important for correct behavior, and they can have security implications as well. The Linux Vendor Firmware Service (LVFS) project is playing an increasing role in making firmware updates more straightforward for both end users and vendors; LVFS just announced its 20-millionth firmware download. Since even a wireless mouse dongle can pose a security threat, the importance of simple, reliable, and easily applied firmware updates is hard to overstate.

Red Hat's Richard Hughes started LVFS in 2016, and in 2019 the Linux Foundation took the project under its wing. Since its inception, LVFS has grown to provide firmware updates for over 2,000 devices from approximately 100 different vendors.

In a 2019 presentation [YouTube] (slides [PDF]), Hughes discussed firmware updates in terms of human, organizational, and technical complexity. As he explained, end users generally don't know what exact hardware they have in their machines, whether its firmware can be updated, if that firmware needs to be updated, where to get that update, or how to apply it. Additionally, users often do not understand the importance of these updates; as Hughes pointed out, "updating your mouse firmware when your mouse is working fine seems ridiculous."

Further, one vendor's hardware often contains updatable components from another vendor, and each only provides firmware for the hardware that it is directly responsible for. Taken together, as Hughes said, "users have no chance of getting this right." LVFS addresses this complexity by providing a centralized repository of firmware and the associated metadata, bringing vendors and end users together. LVFS, however, is more than a centralized firmware distribution site; in the words of Hughes, the LVFS provides "a pipeline right from the firmware author, all the way to the end user."

The project consists of two open-source components: the LVFS web site, which allows vendors to manage their respective firmware (GitHub repository), and the fwupd project (GitHub repository), which provides the command-line fwupdmgr tool and the fwupd daemon to interact with the LVFS web site. A separate project, GNOME Software, provides a user-friendly graphical interface to perform updates; it and fwupd are available by default in most distributions that support GNOME.

At the core of the LVFS project are the binary firmware blobs, provided by vendors as Cabinet files. It is an odd choice for a Linux-focused project, but Hughes explained that it "allows vendors to use the same deliverable for Windows Update as they do for LVFS." Inside the Cabinet file is a cryptographically signed binary blob containing the firmware and metadata in formats consumable by both LVFS and Windows Update. After processing, these Cabinet files are re-signed by LVFS to ensure fwupd clients can verify the integrity of the firmware. LVFS public keys that are used for this verification are distributed as part of fwupd.

Hughes described the LVFS web site, which provides the vendor interface, as "just a Python Flask project." It serves as the central firmware repository and management portal for vendors. Cabinet files uploaded by vendors to the web site are processed and, once validated, added to a master XML document called AppStream, signed by the site. This master list provides the data fwupd needs to notify users of any firmware updates available for their system. For security, Cabinet files are served over SSL by the site directly; the AppStream document, however, is distributed via a content distribution network.

The web site can also be hosted locally, giving organizations privacy and complete control over the update process. For a local instance, the same workflow described above is applied using an organization's own keys, signatures, and firmware packages; the fwupd installations for the organization's machines would then be configured to use the local instance. Hughes said there are a number of undisclosed organizations that use LVFS behind their firewalls for various reasons.

Firmware processing

The processing that is done on firmware provided to LVFS is extensive. In a private email, Hughes explained that the site can process various firmware payload formats into a collection of individual executables and resources. Two types, Pre-EFI Initialization modules (PEIM) and Driver Execution Environments (DXE) are extracted, hashed, and stored in the LVFS database; the project calls them "shards." These shards can be common across firmware when, as Hughes explained, a hardware vendor gives two OEMs the same driver. By collecting and hashing them, they can later be queried against. For example, a query to determine which hardware is the most popular across OEMs by looking for the same hash value in firmware blobs.

That's not the end of the project's firmware analysis, either; Hughes says that DXE shards are analyzed as well. He explained that the project looks for expired certificates, private keys, and other indicators that shouldn't exist in a production firmware like the strings "DO NOT SHIP" or "To Be Filled by O.E.M."

The process of extracting shards is something Hughes says helps "keep vendors honest." Sometimes, vendors will upload firmware described in the release notes as a "low-priority enhancement-only update" that turns out to also include security updates. While there is a limit to the information one can glean from binaries in firmware blobs, enhancement-only updates don't typically need to interact with (for example) the UEFI Secure Boot, explained Hughes; when a discovery like that is made, it allows him to reach out to the vendor to discuss it.

The ability to break firmware blobs down into base, traceable binaries is a significant benefit of the project; especially when it comes to security. For example, LVFS provides a means to block a shard with known security issues, and in his presentation, Hughes showed an example of a rule to block an object known to contain "the Dual EC backdoor for the NSA [National Security Agency]". Hughes said in my correspondence with him that it was "trivial" to query the shard database for firmware that includes the compromised Computrace module. Blocklist rules are defined in the blocklist plugin for the LVFS website. It is also worth noting that security researchers can get a free account on LVFS to issue YARA queries against the available (public) shard data.

As LVFS has gained more support from the industry, the project is imposing increasingly strict restrictions and expectations on vendors in order to participate. In a discussion during Hughes's presentation of firmware signatures, he told the audience that the project had a requirement that "the signature couldn't be more than three years out of date, and half the firmware failed. Over the last few months, we have been increasing the number of checks and making them more and more strict." While the checks can be ignored by vendors, Hughes points out "it's way less work" to do things correctly.

There is information that is only available to vendors on the LVFS web site, including firmware statistics such as installation success and failure rates. These statistics are enabled by another feature Hughes described as "very interesting," a bidirectional feedback mechanism to communicate with users applying updates. According to Hughes, "traditionally when you have a firmware update on an FTP site, you don't get any information back from the end user themselves." fwupdmgr, however, can provide the history of firmware updates made on a system to LVFS. This feedback mechanism can be used by users via the report-history sub-command of fwupdmgr; it is not accessible from GNOME Software. Limiting access to the feature is intentional; Hughes argues that users upgrading via the command line "understand the ramifications of uploading things like your kernel version" to a third party. Hughes has a blog post that details what is transmitted as part of the feedback mechanism.

The feedback not only helps vendors identify who is experiencing a problem with their firmware so they can respond accordingly, but is also valuable to the LVFS. For example, firmware updates that have a high failure rate can be removed from distribution automatically until the issue is fixed.

Working with vendors

While the most common firmware-update protocols are already available in fwupd (39 are currently supported), some vendors use proprietary protocols with implementations that they are unwilling to release under the LGPLv2+, as required by the project. Hughes described vendor secrecy concerns as "an order of magnitude more paranoid" than he experienced working for national defense contractors. Working with those vendors can be a challenge, one that the project is still trying to overcome. In his presentation, Hughes described how Logitech would not share the code for its flashing protocol. Still, the developers were able to answer any questions he had about it. To create the protocol implementation, Hughes engaged in what amounts to a game of Twenty Questions with them. Hughes says that most vendors can be convinced to release their protocols without such games, once it is explained to them that the intellectual property at question has little value; fundamentally, all flashing protocols operate identically.

Wrapping up

LVFS looks to be a useful project and service which is working to push hardware vendors in the right direction. GitHub reports around 120 contributors for the two projects. Of the two, fwupd is the more active project — it has 85 releases and near-daily commits. The LVFS web site only has nine releases with bi-weekly commits. Hughes made it clear that more contributors are welcome; he doesn't have enough time to do all of the things he'd love the project to take on.

As for what is next, Hughes hopes to further build the automated firmware analysis tooling, add more features to the LVFS web site to support vendors, and get more industry players like ASUS and Microsoft to participate. For readers who would like to keep informed on the project, the announcement mailing list is a good way to do so. The project also provides a developer page with information on the process of contributing or opening issues. With more vendors, support for additional update protocols, and so on, LVFS seems poised to strengthen the firmware-update landscape for years to come.

Comments (17 posted)

Toward a "modern" Emacs

It has only been a few months since the Emacs community went through an extended discussion on how to make the Emacs editor "popular again". As the community gears up for the Emacs 28 development cycle, (after the Emacs 27.1 release in August) that discussion has returned with a vengeance. The themes of this discussion differ somewhat from the last; developers are concerned about making Emacs — an editor with decades of history — seem "modern" to attract new users.

The May 2020 discussion focused on restoring the popularity that Emacs is felt to have enjoyed in the past. It could well be that there are more Emacs users now than at any time in the past but the editor's share of the total computing user base has clearly shrunk over time. The current discussion has a similar but different focus: attracting new users to Emacs, an editor that is widely seen as being outdated and as having a difficult and intimidating learning curve.

A more modern Emacs

The initial thread was started by "Ergus" on September 6 as a request to consider some changes for the Emacs 28 release. Rather than focus on new features, though, Ergus proposed making some existing features more accessible:

Many of the suggestions reverberated through the following discussions, starting with the idea that the default theme for Emacs should be improved. In particular, Ergus suggested the adoption of a dark theme by default "to make Emacs feel more modern". The discussion often returned to the idea that Emacs needs a more "modern" look and feel, though there was, understandably, some difference of opinion over what "modern" means.

A default dark theme may not be in the future, leading one to think that there may yet be hope for the world in general. But there does seem to be general agreement that Emacs could benefit from a better, more centralized approach to color themes, rather than having color names hard-coded throughout various Elisp packages. From that, a proper theme engine could be supported, making dark themes and such easily available to those who want them.

There was some discussion of adopting the Solarized color palette in particular. As Dmitry Gutov pointed out, though, Solarized makes for a rather low-contrast experience; a look at this screenshot of Emacs with Solarized colors makes that clear enough.

Another area where Emacs is insufficiently "modern", it seems, has to do with keyboard and mouse bindings. On the keyboard side, users have come to expect certain actions from certain keystrokes; ^X to cut a selection, ^V to paste it, etc. These bindings are easily had by turning on the Cua mode, but new users tend not to know about this mode or how to enable it. Many participants in the discussion said that this mode should be on by default.

That, of course, would break the finger memory of large numbers of existing Emacs users, who would be unlikely to appreciate the disruption. Or, as Richard Stallman put it:

For better or worse, though, the "newer editors" appear to have won out in this regard. The default bindings in Emacs may not change anytime soon, but that doesn't mean Emacs can't provide a way for users to get that behavior without needing to learn Lisp first.

The situation with mouse behavior is similar; as several participants in the discussion pointed out, users of graphical interfaces have come to expect that a right-button click will produce a menu of available actions. In Emacs, instead, that button marks a region ("selection"), with a second click in the same spot yanking ("cutting") the selected text. Many experienced Emacs users have come to like this behavior, but it is surprising to newcomers. The right mouse button with the control key held down does produce a menu defined by the current major mode, but that is evidently not what is being requested here; that menu, some say, should present global actions rather mode-specific ones.

Stallman suggested offering a "reshuffled mode" that would bring the context menu to an unadorned right-button click, and which would add some of the expected basic editing commands there as well. This would be relatively easy to do, he said, since mouse bindings are separate from everything else. Besides, as he noted, the current mouse behavior was derived from "what was the standard in X Windows around 1990"; while one wouldn't want to act in haste, it might just be about time for an update.

Discoverability

In general, Emacs has all of the capabilities that new users might wish to have (and more), but those capabilities can be difficult to discover and use. So a number of the proposed changes had to do with easing discovery. This came as an apparent surprise to some Emacs developers, who have long thought of the menu bar at the top of each window as being an easy way for users to discover features. But evidently it is common to disable the menu bar for Emacs; as Gregory Heytings put it, "menus are not 'modern'". To be "modern", an application must either attach its menus to the right mouse button or, at most, provide a "hamburger menu" from a single button.

Emacs could certainly move to that mode, at least until menu bars are fashionable again. But there are other ideas for improving discoverability circulating as well. One of those would be to offer a "guided tour" to new users upon the first invocation of the editor that would quickly introduce core Emacs features; this would be an enhancement of the tutorial that exists now. There was also a discussion of producing a series of videos, or perhaps just making use of the many videos that exist now. Stefan Monnier suggested short videos to match the perceived attention span of young users, "ideally funny and sexy, maybe with a cat".

A step beyond the guided tour would be some sort of "configuration assistant" that would run on the first invocation of Emacs. It could offer tours, but it would also give the user the opportunity to configure the editor in ways that might better match their expectations. Features like Cua mode could be enabled, for example, mouse bindings adjusted, and a depressing dark theme selected without the need to type a single parenthesis. That might be enough to overcome the initial shock of exposure to Emacs for many new users.

One other suggestion that was raised frequently is to enable various modes that are not turned on by default in Emacs; indeed, several of them are not shipped with Emacs at all. For example, Ibuffer is generally seen as superior to the standard Emacs buffer list, so many think it should replace the standard list. The Emacs "undo" mechanism is more capable than what is provided in many other editors, but it is also seen as difficult to use; some users think that the undo-tree mode makes undo easier and should be shipped with Emacs and enabled by default. There are several modes that enhance the editor's command-search and autocompletion capabilities, including Ivy, Helm, and Ido, among others. Many of these, it is argued, provide behavior more in line with current expectations and improve discoverability; one of them should be adopted and made the default.

Attracting developers

One problem with many of these packages is that they are not actually a part of GNU Emacs. Changing that would often require the author to sign copyrights over to the Free Software Foundation, which is not something all authors are willing to do. Similar problems arise with many of the derivative Emacs "distributions", such as spacemacs or Doom, which have clearly eased the path into Emacs for some users. Some of the ideas found in these distributions may well merit inclusion in Emacs, but that does not happen. Emacs maintainer Eli Zaretskii complained that the creators of these distributions do not contribute their work back.

This points to one idea that did not really come up in the discussion, but which should be considered: part of the key to making Emacs more attractive to users might be making its development community more attractive to contributors. Adopting some of the more forge-oriented tools favored by younger developers might help in this regard, as might a less centralized governance model. Emacs remains an old-style GNU project, a fact which clearly grates on its developers at times. Increasing the relevance of Emacs in the coming years is going to require more than new key bindings and some cat videos; it is likely to require attention to improving the experience at all levels.

Comments (198 posted)

OpenWrt and SELinux

SELinux is a security mechanism with a lot of ability to restrict user-space compromises in various useful ways. It has also generally been considered a heavyweight option that is not suitable for more resource-restricted systems like wireless routers. Undeterred by this perception, some OpenWrt developers are adding SELinux as an option for protecting the distribution, which targets embedded devices.

A mid-September blog post from Paul Spooren gives some of the background and status. In July, W. Michael Petullo picked up some older patches from Thomas Petazzoni. Petullo updated some of that work, including switching the tools from Python 2 to Python 3; he submitted the result as a pull request (PR) for OpenWrt. On August 30, that PR was merged; a related PR from Petazzoni adding some packages to support SELinux on the distribution was merged September 12.

The SELinux feature adds a mandatory access control (MAC) mechanism to the Linux kernel. Discretionary access control (DAC) is typified by the Unix read-write-execute permissions for files based on user and group IDs. While users can make changes to DAC permissions (e.g. chmod 777 ~), MAC policies generally cannot be overridden at run time.

The MAC policies used by SELinux allow administrators to specify which applications can have access to various objects in the system, primarily files and network resources. Instead of user-based permissions, these objects get assigned security contexts describing their scope. The policies then map which objects can access other objects that have specific contexts, thus they could allow a web server to only be able to access files under a certain directory, to only be able to bind to specific network port numbers, and to be unable to initiate outbound connections. In order to change those restrictions, the root user would need to install different policy at boot time or relabel objects to have contexts that allow them to be accessed. Absent a kernel compromise, services and other processes restricted in this way should not be able to reach outside of the places where their access is confined to.

Security labels are the way that SELinux determines the context of a file object; that information is stored as an extended attribute on the files themselves. Because they live in the protected security extended attribute namespace, they cannot be changed by regular users. So the refpolicy package will label the files with their context as the image is being built.

Switching a distribution to use SELinux involves more than just enabling a kernel feature, however. As the blog post describes, the build system needed to be configured so that kernels supporting SELinux can be created. Turning on options to enable SELinux (CONFIG_KERNEL_SECURITY_SELINUX) and to allow security labels in the root filesystem (CONFIG_TARGET_ROOTFS_SECURITY_LABELS) will automatically select the refpolicy package, which will put the security labels on the files in the squashfs (or F2FS) root filesystem.

In order for SELinux to apply its policies, processes started on the system need to have a security context associated with them. That is the responsibility of the init process, but, currently, the init used for OpenWrt, procd, does not set contexts on the processes that it starts. Work on that is planned. Administrators also need to be able to examine the context associated with files, processes, and other objects, which is done using the -Z switch to utilities like ls and ps. Like most embedded distributions, OpenWrt uses BusyBox; a custom busybox-selinux package was created that configures BusyBox to add the needed support.

The initial policy being used is the reference policy from the upstream SELinux project; "it has not been tuned specifically for OpenWrt", Petazzoni said. There are plans to tweak that policy based on the specific needs of OpenWrt. There is still more to be done, of course, but much of the underlying infrastructure has been merged into OpenWrt at this point. As the blog post shows, it is possible to launch a virtual machine with an SELinux-enabled OpenWrt image and see contexts, labels, and so on. Adding support to procd for setting contexts will provide for much of the remaining needed pieces for basic support in OpenWrt.

In general, SELinux has not been that much of a factor in the embedded Linux world. An obvious counterexample, though, is SELinux on Android, which started small in the Android 4.x series but has since grown to be an integral part of the security landscape for the phone distribution. But embedded SELinux has been investigated for longer still, though the resource requirements have generally kept it to higher-end devices. OpenWrt supports a wide range of devices, so SELinux might just find a home in some of them.

The security landscape for home routers is generally pretty bleak; adding SELinux protections for OpenWrt can only help with that picture. Also, SELinux is perhaps a better fit for a router distribution than it is for, say, Fedora systems, where the software being run is changing more frequently and needs more tweaks to the SELinux policy in order to continue to function. As elsewhere, of course, SELinux is no panacea, but it could certainly improve the security for home routers and the like.

Comments (14 posted)

Saying goodbye to set_fs()

The set_fs() function dates back to the earliest days of the Linux kernel; it is a key part of the machinery that keeps user-space and kernel-space memory separated from each other. It is also easy to misuse and has been the source of various security problems over the years; kernel developers have long wanted to be rid of it. They won't completely get their wish in the 5.10 kernel but, as the result of work that has been quietly progressing for several months, the end of set_fs() will be easily visible at that point.

This 2017 article describes set_fs() and its history in some detail. The short version is that set_fs() sets the location of the boundary between the user-space portion of the address space and the kernel's part. Any virtual address that is below the boundary set by the last set_fs() call on behalf of a given process is fair game for that process to access, though the memory permissions stored in the page tables still apply. Anything above that limit belongs to the kernel and is out of bounds.

Normally, that boundary should be firmly fixed in place. When the need to move it arises, the reason is usually the same: some kernel subsystem needs to invoke a function that is intended to access user-space data, but on a kernel-space address. Think, for example, of the simple task of reading the contents of a file into a memory buffer; the read() system call will do that, but it also performs all of the usual access checks, meaning that it will refuse to read into a kernel-space buffer. If a kernel subsystem must perform such a read, it first calls set_fs() to disable those checks; if all goes well, it remembers to restore the old boundary with another set_fs() call when the work is done.

Naturally, history has proved that all does not always go well. It's thus not surprising that the development community has wanted to rid itself of set_fs() for many years. It's also unsurprising that this hasn't happened, though. The kernel project does not lack for developers, but there is always a shortage of people who are willing and able to do this sort of deep infrastructural work; it tends to not feature highly in any company's marketing plan. So the task of removing set_fs() has languished for years.

Recently, though, Christoph Hellwig has stepped up to this task and the kernel-wide cleaning-up that is required to get it done.

For example, one might be surprised to find set_fs() calls in the core networking code, and even more surprised to learn that they were added in 2019, during the 5.3 development cycle. The patch in question added the ability for BPF programs to invoke the setsockopt() and getsockopt() system calls. Those calls are normally invoked from user space, so they apply the usual access checks on any parameters passed to them; calls originating from BPF programs, though, will supply buffers in kernel space. Putting in a call to set_fs() in that case allowed those calls to work without further modification.

Hellwig's plan for taking that set_fs() call back out involved the creation of a new sockptr_t type that can hold an address pointing into either kernel or user space:

    typedef struct {
	union {
	    void	*kernel;
	    void __user	*user;
	};
	bool		is_kernel : 1;
    } sockptr_t;

Code that initializes a sockptr_t variable must specify whether the address is meant to refer to kernel or user space; a set of helper functions can then be used to copy data to and from that address without needing to worry further about where the destination buffer is — or to call set_fs(). As it turns out, setsockopt() and getsockopt() offer a lot of different options, so a long patch series was required to convert the relevant functions to sockptr_t addresses. At the end of the series, the set_fs() calls were removed. This series entered the mainline during the 5.9 merge window.

Something that was not merged was an earlier version of this idea, which was meant to be used throughout the kernel. Hellwig proposed the creation of a "universal pointer" type (uptr) that functioned like sockptr_t; it was accompanied by a pair of new file_operations methods that would work with those pointers. Then, any kernel subsystem that might need to perform I/O on both kernel-space and user-space pointers could be converted to use these new methods rather than calling set_fs().

Linus Torvalds vetoed that idea; he objected to the addition of the new type and file_operations methods, which he saw as temporary and unnecessary workarounds for the real problem. If somebody was going to bother to convert ordinary read() and write() calls to the new read_uptr() and write_uptr(), he asked, why wouldn't they just convert to the existing read_iter() and write_iter() methods instead? Those methods already handle the different address spaces just fine (through yet another union in struct iov_iter that tracks which type of address is in use); indeed, much of the work to remove set_fs() calls in various parts of the kernel has involved switching to iov_iter. So the uptr type fell by the wayside, but the sockptr_t was able to overcome Torvalds's opposition and was merged.

Then, there is the set_fs() call that isn't actually there. In current kernels, the boundary between kernel and user space is established fairly late in the boot process (but before the init process is started). Before that happens, kernel functions that operate on user-space pointers will happily use kernel-space pointers instead; parts of the initialization code (dealing with the initial ramdisk, for example) depend on this behavior. Eliminating that implicit set_fs() call required another patch series creating a set of special helpers that is discarded once the bootstrap process is complete. This series, too, was merged for the 5.9 release.

The final step, for the x86 and PowerPC architectures at least, is this patch series removing set_fs() entirely. Getting there requires tidying up a number of loose ends. It adds iov_iter support to the /proc filesystem, for example. This patch converts kernel_read() and kernel_write() (yet another way to perform I/O on kernel-space buffers) to iov_iter, removing the set_fs() calls previously used there. The splice() implementation is changed in a way that might break existing users: it simply no longer works if the data source is a device that does not support the splice_read() method. Hellwig said that the affected users all appear to have working fallbacks in place, but that specific devices can gain splice_read() methods if the need turns out to exist.

After a few more patches to remove the last uses of set_fs() from the x86 and PowerPC architectures, support for set_fs() itself is disabled and the task is complete. These patches are currently in linux-next, and thus should be merged for the 5.10 release. Hellwig has also posted a patch set for RISC-V, and Arnd Bergman has a patch set for Arm, but those have not yet been applied. Hellwig intends to work through the remaining architectures, removing set_fs() from each.

The patches described above are only a small portion of the effort that has gone into making it possible to finally get rid of set_fs(). The end result of all this work is the near elimination of a kernel interface that has been deemed dangerous for almost as long as it has existed — and it has been around for a long time. It is an example of a form of kernel development that tends not to create headlines, but which quietly keeps the kernel maintainable in the long term. Tasks like this often suffer from a lack of attention, but they do tend to get done in the long run, which is a good thing; even after nearly 30 years, there is a lot of cleaning up still to be done in the kernel.

Comments (7 posted)

Mercurial planning to transition away from SHA-1

Recently, the Mercurial project has been discussing its plans to migrate away from the compromised SHA-1 hashing algorithm in favor of a more secure alternative. So far, the discussion is in the planning stages of algorithm selection and migration strategy, with a general transition plan for users. The project, for the moment, is favoring the BLAKE2 hashing algorithm.

In July 2020, Joerg Sonnenberger started the conversation on moving away from SHA-1. Sonnenberger focused on four major aspects of the transition: which hash function to use, updating the test suite, updating the code base, and backward compatibility.

Picking a new hash

For obvious reasons, choosing a new hashing algorithm is one of the most important choices facing the project regarding the transition from SHA-1; it is also where Sonnenberger focused the bulk of his post. He presented the community with a variety of options, describing the benefits and drawbacks of each. SHA2-256 was "the most obvious choice" to Sonnenberger, given its wide support (including hardware acceleration on some CPUs). That said, he had a lot of reservations when it came to adopting it for the project:

SHA2-256 is the algorithm that the Git project selected for its migration from SHA-1, but that project doesn't have to worry about the same potential vulnerabilities to length extension attacks, unlike Mercurial. The Mercurial wiki page on transitioning from SHA-1 includes a conversation with an unknown Git contributor (likely brian m. carlson) who explains why SHA2-256 length extension attacks are not relevant to Git; it hashes the type and length as a prefix of its objects.

Among the other choices Sonnenberger suggested were BLAKE2, KangarooTwelve (K12), and BLAKE3. According to him, BLAKE2 was the "second most widely supported hash function." Sonnenberger analyzed two other algorithms, K12 and BLAKE3; each offer their own benefits and drawbacks:

Sonnenberger provided a set of "basic benchmarks" of how each algorithm performed in his tests using a variety of implementations and configurations against a ~7GB file. For each, he tested the algorithm with implementations in assembler and C; for BLAKE3 he tested both the standard version of the algorithm and a variant that uses the same number of rounds as BLAKE2. The results indicated that, of the 11 tests, BLAKE3's (normal) assembler implementation was the fastest, followed closely by SHA2-256 in assembler. SHA2-256 written in C was the slowest. Josef 'Jeff' Sipek provided some additional benchmarks that were more exhaustive than the ones provided by Sonnenberger.

Augie Fackler responded to Sonnenberger, agreeing with his assessment of SHA2-256. Fackler did, however, question Sonnenberger's selection of the BLAKE2s variant (optimized for 8- to 32-bit platforms) instead of BLAKE2b (optimized for 64-bit platforms). Fackler stated that he has been "strongly favoring blake2b for years now," adding, "a crypto expert at Google was who pushed me to blake2b, so if we go a non-blake2 direction I'd want to run it by them." In Sonnenberger's benchmarks, BLAKE2s ranked eighth out of the 11 algorithms and was more than 3 times slower than the best performer (BLAKE3 in assembler).

Sonnenberger responded that the choice to use BLAKE2s was driven by the performance concerns on 32-bit systems; BLAKE2b performs poorly on 32-bit architectures. Fackler, however, questioned whether 32-bit platforms should be the priority:

In the end, Fackler and Sonnenberger reached an agreement to use BLAKE2b by default, but to also provide "opt-in" support for BLAKE2s. In Fackler's opinion, the decision is not only a reasonable compromise, but "will probably help us have a more generalized hash-selection mechanism in hg so that when the _next_ hash crisis hits we just push a button and we're done."

Implementation

With the selection of a hashing algorithm appearing to reach a consensus, the challenge to the project will be implementing the changes. Both the test suite for the project and the code base itself will need updating — not to mention addressing the problem of backward compatibility for existing repositories.

In Sonnenberger's opinion, the biggest pain point is the test suite. According to him, "every single test in the tree has to be adjusted" to at least replace SHA-1 changeset identifiers. This appears to be more complicated than a simple search and replace, as some tests check aspects like output size that will change with a new algorithm. Sonnenberger suggested approaching the problem by creating a duplicate test suite based on the current one, keeping both moving forward. Fackler, however, thought that getting the changeset hashes out of the tests was a better approach, writing "I think we should have a handful of tests that care about hashes, and most shouldn't." At the moment how to address the test suite remains an open issue.

Updating the core code itself, described as "mercurial.nodes" by Sonnenberger, is "the biggest point of pain after test cases." This may seem counterintuitive, but Mercurial revlogs already support 32-byte hashes, unlike Git, which had been hard-coded for the 20-byte length of SHA-1. Mercurial does, however, assign certain predefined hash values to identify special nodes in the repository; they are used to mark new or empty nodes (examples of these hashes can be found here). In this respect, the transition to a new hashing algorithm for Mercurial seems to be more a matter of effort than technical difficulty.

To end users, the most important part of the project's plans will be how it addresses backward compatibility; Mercurial does not currently support the storage of multiple changeset hashes in parallel, but the developers do appear to want to head in that direction. The project also wants to make sure that any transition is left up to the repository owner to do, rather than being a requirement of a specific version of Mercurial. Sonnenberger proposed that repositories could be upgraded by creating a static map of hashes for existing changesets, mapping the legacy SHA-1 hashes to their new equivalent. This map could then be used by a compatibility layer when referencing changesets by the SHA-1 hash value. For changesets created after the upgrade of a repository, all hashes would be calculated using the new algorithm. Fackler was not a fan of the idea, but seemed willing to accept the plan, writing "this sucks, because it means migration implies re-cloning rather than just an incremental pull. But if you're going to do the work, it's probably good enough."

According to the project's SHA-1 transition wiki page, the plan will be to use a truncated version of whichever hashing algorithm is ultimately selected. This truncation (currently a byte or two) is to make room for flags in the changeset needed to identify the type of hash algorithm used. This will be accompanied by capabilities negotiation on the protocol level, although there are few details currently on how that will work. The general plan is for servers to reject old clients attempting to access upgraded repositories.

What is next

For now, Mercurial's transition away from SHA-1 is still in the planning phases. The last time the subject of a transition from SHA-1 was raised in the mailing list was August 15. However, overall, the project remains active so it's likely the transition will ultimately be completed. The project still has a lot of outstanding questions it is trying to address and contributors are making reasonable efforts to determine the best path forward. Hopefully, this planning translates into both a more secure hashing algorithm for repositories now and in the future.

Comments (16 posted)

New features in the fish shell

Fish (the "friendly interactive shell") has the explicit goal of being more user-friendly than other shells. It features a modern command-line interface with syntax highlighting, tab completion, and auto-suggestions out of the box (all with no configuration required). Unlike many of its competitors, it doesn't care about being POSIX-compliant but attempts to blaze its own path. Since our last look at the project, way back in 2013, it has seen lots of new releases with features, bug fixes, and refinements aimed at appealing to a wide range of users. Some of the biggest additions landed in the 3.0 release, but we will also describe some other notable changes from version 2.1 up through latest version.

The first release of fish was made by Axel Liljencrantz in February 2005. Version 2.0 was released in September 2013, and 3.0 in December 2018. The latest version, 3.1.2, was released on April 29, 2020.

Logical operators

Prior to the 3.0 release, fish lacked a special syntax to express logical AND (&&) and OR (||) operations. Instead, the and and or commands were used for such operations. These commands verify the previous command's exit status before acting accordingly. For example, command1; and command2 expresses a logical AND operation in which command2 is executed if, and only if, command1 returns an exit status of zero.

This syntax proved to be unpopular among converts from other shells where operators such as &&, ||, and ! were used to express logical operations and it was the subject of several discussions on the project's issue tracker. Due to popular demand, these logical operators are now supported as of fish 3.0 to make it easier to migrate from other shells. It is now possible to enter command1 && command2 in the shell or in a script to express the logical AND operation.

History and incognito mode

When a command is executed in the shell, it is inserted at the end of the history file located at ~/.local/share/fish/fish_history, accompanied by the timestamp of its entry. It is also possible to save a command to the history without executing it using the Alt+# binding (introduced in version 2.3.0). It toggles the current command line between commented and uncommented states. As with Bash and other shells, the up or down arrow keys can be used to search forward or backward in the history.

One nice feature for fish history is the ability to filter through the history based on what is typed into the shell. Entering a few characters and then hitting the up arrow key performs a history search for the commands that include the typed characters. For example, typing git and hitting the up arrow will display only the commands that contain the string git.

Fish does not enter any command that begins with a space into its history file, essentially treating it like an incognito command. This often surprises users who are transitioning to fish from Bash or other shells that do not have this behavior; it is also easy to activate it unintentionally when pasting commands to the terminal from other sources. There have been several discussions in GitHub issues on whether to retain this behavior in future releases and how to make it more discoverable. As of fish 3.1.2, no changes have been made to this aspect of the shell behavior.

A new addition in fish 3.0 is the --private flag that can be used to start fish in private mode; it stops all subsequent commands from being logged in the history file. This is handy for when a user wants to enter sensitive information in a command. The $fish_private_mode variable was also added for the purpose of detecting if a script is being run in private mode so that its behavior may be modified accordingly to respect the user's wish for privacy.

Aliases and abbreviations

Like many other shells, fish provides the ability to define memorable alternate names (known as aliases) for commonly used commands. Aliases can help to eliminate a great deal of typing when working in the command line. There are two ways to create aliases in fish. The first involves creating a function wrapping a command:

    function gc
      git commit $argv
    end

The second option is to use the alias keyword, which is essentially a shell wrapper for the function syntax above. When using this syntax, the $argv arguments string will be appended automatically.

    alias gc="git commit"

The release of fish 2.2 brought support for abbreviations to the shell. These are similar to standard aliases, but expand inline to the full command when typed. For example, a command can be shortened to yul using the following:

   abbr yul "yarn upgrade --latest"

Once yul is entered into the terminal followed by the space or enter key, it will be expanded into the full command. By default, abbreviations are stored to the universal scope, so they immediately become available in all current fish sessions and subsequent ones as well. To make an abbreviation visible in the current session only, use the (surprisingly named) --global flag when creating it. This places the abbreviation in the global scope which is local to the current shell. Abbreviations are generally better than aliases because the full command can be seen before executing it, which makes it easier to edit for a one-off change. They are also more suitable for interactive demos or presentations because the instructor can use a shortcut without obscuring the full command.

Vi mode

The default key bindings used by fish for moving the cursor around on the command line are from the Emacs editor. Examples are Ctrl+A to move the cursor to the start of a command, Ctrl+E to move to the end of a command, and so on. Fish 2.2 introduced a Vi mode for those who prefer bindings from the Vi editor. It supports command mode, insert mode, and visual mode bindings. Fish also provides shared bindings that are accessible regardless of your preferred mode.

If neither mode is sufficient for a particular edit, an external editor may be summoned using the Alt+E or Alt+V shortcuts. The two commands are synonymous; the editor is chosen from the first available of the $VISUAL or $EDITOR environmental variables.

Clipboard integration

Fish features an Emacs-style kill ring for blocks of text that were previously killed. For example, Ctrl+U cuts from the cursor position to the start of the line and inserts the text into the kill ring while Ctrl+Y pastes the latest value from the kill ring at the cursor position.

A change in Fish 2.4 is that any text cut with kill ring commands no longer overrides the system clipboard. This behavior was changed because it was a common source of frustration when pasting to the shell from external sources because cutting some text would override the previously copied text meaning that it would have to be copied again. Working with the system clipboard is still supported on Linux (X11 and Wayland) and macOS. Ctrl+X is used to copy the whole line to the clipboard (regardless of cursor position) and Ctrl+V pastes from the clipboard to the shell.

Web configuration tool

The web configuration tool, launched with fish_config, provides a nice way to customize the prompt style, color theme, and so on. An important security enhancement landed in 2.1.1 which prevents a remote-code-execution attack when running this web interface by using an authentication token to protect requests and responding only to requests that include this token.

In addition, the behavior of this tool was changed slightly in version 3.1 by causing it to display a list of the commands it is executing, for debugging purposes. Other improvements include new theme presets, base-16 color options, prompt styles, and support for viewing, editing, and adding abbreviations.

Wrap it up

Fish was originally implemented in C but is now primarily written in C++. It does not follow any particular release cycle but major versions usually take a few years to come out. The best way to follow the development of the shell is through its GitHub repository and official mailing list. The changes that are being planned for upcoming releases are detailed on the milestones page on GitHub. An official Gitter channel also exists for community discussions.

Overall, the refinements being made continue the process of making fish easier and more convenient to use compared to other shells. It strikes a good balance between having useful defaults to get started with immediately and leaving room for extensibility and customizability. A tutorial is available for those looking to quickly get up to speed with the precise differences between fish and more traditional shells.

Comments (40 posted)