LWN.net Weekly Edition for December 24, 2020

Welcome to the LWN.net Weekly Edition for December 24, 2020

This edition contains the following feature content:

• LWN's 2020 Retrospective: a look back at the year that was and what we thought it might be in January.
• PureOS: freedom, privacy, and security: a company to create a secure Linux product.
• 5.11 Merge window, part 1: the first set of patches merged for the 5.11 release.
• Managing multifunction devices with the auxiliary bus: a new virtual bus for devices that do more than one thing.
• Doing symbolic math with SymPy: Lee Phillips reviews a capable mathematical tool.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

This is the final LWN Weekly Edition for 2020; as is traditional we will be taking the last week of the year off for rest, recovery, and preparation for the year to come. Thanks to all of you for supporting LWN though this challenging year. Best wishes to all of you for the holidays and the new year, and we look forward to seeing you all back here in 2021.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

LWN's 2020 Retrospective

Predictions are hard, as they say, especially when they are about the future. So perhaps your editor can be forgiven for not anticipating that 2020 would be the sort of year that makes one think nostalgically about trips to the dentist, waiting in a crowded motor-vehicle office, or crossing the Pacific in a row-47 middle seat. If only we had known how good we had it. Be that as it may, this year is finally coming to an end. Read on for a look back at the year, starting with the ill-advised predictions made in January.

A look back at a look forward

The first prediction had to do with the Python project finding its path in a post-Guido world. In truth, founder Guido van Rossum has not been as absent as one might have thought, and he continues to have a strong say in the development direction of the language. His championing of the PEP 622 pattern-matching mechanism shows that he still has strong ideas about where the language should go. Python governance may be happening without Guido, but the project's leaders still listen to him.

The direction of the GNU Project was the subject of another prediction that did not really pan out as expected. As far as the public can see, at least, the discussion about the future of the GNU project appears to have faded away. Richard Stallman remains in charge (as expected), but the project gives every indication of being mostly on autopilot.

The prediction about increasing retirements in the community explicitly said that little might happen in 2020, and that appears to be the case. If anything, things went the other way, with Van Rossum ending his retirement to take a new job in the industry.

Did the kernel workflow effort begin to bear fruit, as predicted? That work has continued, and the community has gained some new infrastructure for dealing with patches outside of email. The ripest fruit at this point, certainly, is the b4 tool, which continually gains capabilities and has become an increasingly indispensable part of many developers' workflow. One wonders how we did without it, but it only showed up (under a different name) in January.

Your editor predicted that the next LTS kernel would be 5.9, released on November 1. There were some weasel words about how, if there were fewer ten-week cycles, that kernel would actually be 5.10, released in December. And, in fact, kernel releases got a little faster in 2020, 5.9 was released on October 11, and the long-term-support release for 2020 will be 5.10, which came out on December 13. So this prediction is a win — as long as the fine print is taken into account.

A prediction that there will be conflict within the Debian community is relatively safe in any year, 2020 included. Attacks on Debian from outside the community may have fallen off a bit, but Debian developers are good at creating their own heat. This can be seen, for example, in ongoing disagreements about vendoring and, of course, init scripts.

As predicted, the kernel community did come close to finishing its work to address the year-2038 problem, though some loose ends remain. It just goes to show that if you predict something often enough you may eventually be able to claim victory. Another prediction was that BPF API issues would not go away. They arguably have not, but neither did they play a large role in the discussions of 2020. The one exception was perhaps this libbpf issue which isn't really an issue in BPF itself. Meanwhile, BPF development continues at a furious pace.

Were there "perturbations in the employment market" as predicted? Obviously, labor markets as a whole have had a difficult year, but it seems that the market for free-software developers was mostly shielded from all of that. Social distancing and working from home come naturally to us, and the technology industry as a whole had a good year.

What was missed

Overall, it is fair to say that your editor's record as a mediocre predictor of future events has not improved in 2020. But failure here isn't just a matter of predicting things that did not happen; there's also the little problem of not predicting things that did come about. Needless to say, there's a few of those to make fun of as well.

One surprise, despite being arguably predictable, was Red Hat's decision to end support of CentOS as we know it in favor of "CentOS Stream". There have been worries about what Red Hat would do with CentOS ever since the acquisition, given that there are clear incentives to end the free ride that CentOS has been for so many users. The hard part with such predictions is always in the timing; it's hard to tell which straw will break the camel's back.

A similar story played out in the LibreOffice community, where Collabora moved LibreOffice Online into a separate, Collabora-branded project. Once again, a company putting significant resources into a free-software project has felt the need to make changes in order to get a sufficient return on that investment. Happily, Collabora Online remains free software, but it lives outside of the LibreOffice community, for now at least.

The death of email in 2020 was not predicted, and it did not happen. But the movement of projects away from email-centered processes was predictable and did continue. Debian debated a move to Discourse, and Fedora continues to move discussions in that direction, for example. Not everybody agrees that these web-based systems are an improvement over email, but there is pressure to move in that direction regardless.

That other story

Then, there is that one other little surprise. Some comfort can be taken in the fact that few others predicted that the world would be thoroughly disrupted by a global pandemic, even though the first reports were showing up by the time the predictions article was written. Over the year, the pandemic has brought a few interesting things to light.

It is worth pointing out, for example, that as soon as discussion of contact-tracing apps took off, a consensus formed that these apps should be open source. Given the sensitive nature of the information handled, the need to be able to inspect the operation of contact-tracing apps and ensure that privacy was protected was easy to see. We have, over the years, raised some awareness of that advantage of free and open-source software, it seems.

Of course, weather and solitaire apps can still follow your every move and report everything back to the mothership. Oh well.

One other thing is worth a mention. Free software famously built the Internet. When in-person meetings became an unwise thing to do, there was an immediate, pressing need for the Internet to support activities like video conferencing, and the free-software community had no real answer to give. There are good video-conferencing applications, and many of us put them to good use this year, but there was nothing that could work on the scale needed. Thus, even free-software companies — and free-software events — ended up depending on highly proprietary platforms, some of which are not necessarily designed with privacy or freedom in mind. This was a huge missed opportunity; free software, it seems, is still far from world domination.

We were certainly hurt by the end of in-person conferences; they are necessary for the smooth functioning of our global, electronically connected community. That situation seems likely to continue well into 2021. But how did the pandemic affect the development of the code? One way of answering that would be to point out that two of the three busiest kernel development cycles ever happened in 2020. Another way would be this set of plots:

The upper plot shows a count of patches merged into the mainline kernel by the date they were posted; the lower one counts by commit date instead. The shaded areas indicate the merge windows. If there is any signal to be seen there at all, it suggests that activity may have picked up ever so slightly during the peak of the lockdowns in March and April. Certainly there is no decline in activity over the course of the year. The pandemic continues to cause untold grief and harm worldwide, but it has been relatively gentle with the development community.

In the same vein, the community has been gentle with LWN. At the beginning of the pandemic it was hard indeed to predict how things would go and easy to fear the worst. At the end of the year, LWN's staff is healthy, and so is its financial footing. We can only offer our deepest thanks to all of you who have supported us over the years, and who continued to support us through this difficult year as well, many at higher levels than before. You all are the reason LWN has been here since 1998.

We wish the best of the holidays for all of our readers and a happy beginning to what will hopefully be a better year than 2020.

Comments (16 posted)

PureOS: freedom, privacy, and security

A recent blog post from Purism—the company that developed PureOS to run on its security-focused hardware—celebrates three years of FSF endorsement of the Linux distribution. While this endorsement is an achievement that is not as highly valued by our communities as one might think, the work done to obtain and maintain that endorsement is useful even to those who disdain the FSF or disagree with its definition of what makes a distribution "free". While Purism and PureOS have been on our radar for a few years now, it seems worth a look at where things have gone with the distribution—and the company behind it.

The blog post notes that PureOS and Purism "sit on a three-legged stool of Freedom, Privacy and Security". The three are intertwined, of course, since PureOS consisting of only free software allows users to ensure there are no antifeatures being slipped into the OS or applications that would impact their privacy or security. Beyond that, free software is an excellent defense against various software supply-chain attacks; in addition the scrutiny of the code afforded to free software, it can also be built in a manner that provides more security:

PureOS is a Debian derivative that consists of a stable "Amber" release, as well as a rolling "Byzantium" release. Amber is based on Debian 10 ("Buster"), while Byzantium pulls packages from Debian testing. Because PureOS only includes free software, it only pulls from the "main" archive, not "contrib" or "non-free" because they contain packages that do not comply with the Debian Free Software Guidelines (DFSG).

The system is customized to make various tweaks, including adding kernel patches for security, enabling AppArmor, and defaulting to a Wayland-based GNOME desktop. It also installs a browser that is configured for better privacy and security; originally it was Firefox-based, but that has changed to GNOME Web (formerly known as Epiphany) more recently. It also comes with DuckDuckGo as the default search engine, rather than alternatives that hoover up vast amounts of information about searches and clicks to enable "better" advertising.

PureOS will run on most desktops and laptops that will run Linux, which is not really a surprise. Some hardware may not work (e.g. laptop WiFi) because it needs a proprietary binary blob, but users can install those pieces from elsewhere if desired. But the mobile version of PureOS is not likely to run on existing phone hardware, which, as the PureOS FAQ notes, generally requires binary blobs. Those blobs typically only work with specific older kernels that are not supported by Mobile PureOS, which uses recent mainline kernels.

For PureOS on phones, Purism now offers its Librem 5 phone. It was originally crowdfunded, and has taken a somewhat circuitous route to mass production (leaving some rather unhappy with Purism), but it is designed with the three legs of the stool in mind. For example, it has hardware kill switches to disconnect various peripherals, such as the cellular modem, microphone, camera, and WiFi. Naturally, it does not need any binary blobs for its functionality either.

Other hardware, such as laptops (Librem 14 and 15), mini-PC, and servers, have also been designed with privacy and security in mind. The laptops feature hardware kill switches for the camera and microphone, for example. Any of the hardware can be ordered with the company's anti-interdiction service that provides customized mechanisms to enable recipients to detect hardware tampering during shipping. These include tamper-evident tape on the system and its box, glitter nail polish on screws, and pictures of all of that sent separately, encrypted with GPG.

Beyond that, users can also order the PureBoot Bundle that couples the PureBoot security-oriented firmware with a pre-installed Librem Key, which is a tamper-resistant USB OpenPGP smart card. The Key will come with a GPG key that will be installed as the secure boot key for the system; it will be shipped separately, perhaps to a different address, to the new owner before the system is shipped. The Librem Key is configured such that it will blink its LED to indicate if the firmware has been tampered with en route.

PureBoot is based on coreboot and has neutralized and disabled the Intel Management Engine (IME), which is an intrusive part of the firmware that has had a number of security flaws identified in it over the years. Users wanting to fully control their systems will want to get rid of as much of IME as possible. The Heads boot software is used to detect tampering with the firmware as well.

It all adds up to a pretty impressive story for those who are concerned about their security and privacy. That story, painted via the huge number of blog posts and other documentation available from Purism, may be somewhat off the mark, however. There have been other complaints about the company, its products, and its behavior, beyond those that were mentioned here as well. There are clearly some problems to be addressed, but the ideas and concepts behind the hardware and software seem sound.

As might be guessed, security and privacy features do not come for free—or even inexpensively. The Purism hardware products are generally quite a bit more expensive than their less secure competitors, but the availability of the systems and services is a boon for those who need that level of assurance.

To a large extent, we humans have sacrificed our freedom, privacy, and security on the altar of convenience—and low cost. Over the years, LWN has looked at various aspects of these problems, including the recent efforts by Mozilla to "take back" the internet from the forces of surveillance capitalism (inspired, in part, by The Social Dilemma movie). In early December, we also looked at the movement away from allowing general-purpose computing on our devices; hardware like that provided by Purism is a way around that problem—at least for now.

But the bottom line is that these options will only exist if at least some consumers are interested in buying them. Purism looks to have a lot of the right answers, but, with any luck, the market will be large enough to support multiple options for hardware and software of this sort. PureOS and PureBoot are all free software that can be adopted and improved by others as needed. In order for that to go anywhere, though, people are going to have to start changing their thinking and prioritize freedom, privacy, and security over convenience and price. In truth, that all seems rather unlikely, sadly.

Comments (28 posted)

5.11 Merge window, part 1

When Linus Torvalds released the 5.10 kernel, he noted that the 5.11 merge window would run up against the holidays. He indicated strongly that maintainers should send him pull requests early as a result. Maintainers appear to have listened; over 10,000 non-merge changesets were pulled into the mainline in the first three days of the 5.11 merge window. Read on for a summary of the most significant changes in that flood of patches.

Architecture-specific

• Support for Intel's software guard extensions (SGX) finally made it into the kernel after 41 revisions on the mailing lists. Some information can be found in this documentation commit.
• In the ongoing effort to crack down on user-space access to x86 model-specific registers (MSRs), writes to MSR_IA32_ENERGY_PERF_BIAS are no longer allowed. There is a document being constructed with a list of tools that perform writes to MSRs, with the eventual goal of finding better solutions for all of them.
• The arm64 architecture can now expose tag bits on addresses (the uppermost bits that are not used as part of the virtual address) to signal handlers if the SA_EXPOSE_TAGBITS option has been set with sigaction(). This provides access to, among other things, memory type extension keys in pointers.
• Support for Microblaze systems without a memory-management unit has been removed; it would seem that there are no longer any users of such systems.
• The MIPS architecture has gained support for coverage testing with gcov.

Core kernel

• There is a new system-call interception mechanism, based on prctl(), that allows user space to trap and emulate system calls. The target use case is running Windows games, but other applications seem likely as well.
• The userfaultfd() system call now provides the ability to disable handling of kernel-mode faults as a security-enhancing measure.
• The BPF subsystem has gained support for task-local storage — data that lives with a given task. The first user is the BPF Linux security module (formerly KRSI).
• The accounting of memory used by BPF programs has changed completely. There is now a control-group controller providing fine-grained management of memory use; see this merge commit for (some) information.
• The BTF mechanism, providing information about kernel types for BPF programs (and more), has been extended to kernel modules.
• The io_uring subsystem has gained support for the shutdown(), renameat2(), and unlinkat() system calls.
• Calls to io_uring_enter() can now specify a timeout value. Adding this feature requires changing the API of that system call; this was done by adding a new flag (IORING_FEAT_EXT_ARG) to indicate the presence of the timeout argument.

Filesystems and block I/O

• The Btrfs filesystem has a couple of new mount options intended to help with the unwelcome task of rescuing data off a corrupted filesystem. Using rescue=ignorebadroots will cause a mount to try to proceed with a corrupted extent root, while rescue=ignoredatacsums turns off data checksum verification.
• Re-exporting a filesystem mounted over NFS is now considered to be a supported operation.
• The close_range() system call has gained a new CLOSE_RANGE_CLOEXEC option. When that option is specified, the indicated file descriptors are marked close-on-exec rather than closed immediately.

Hardware support

• Crypto: Intel QAT_4xxx crypto accelerators and Intel Keem Bay OCS AES/SM4 accelerators.
• Graphics: Novatek NT36672A DSI panels, TDO TL070WSH30 DSI panels, Analogix Anx7625 MIPI to DP interfaces, AMD "Van Gogh" and "Dimgrey cavefish" graphics processors, Lontium LT9611UXC DSI/HDMI bridges, Samsung sofef00/s6e3fc2x01 OnePlus 6/6T DSI cmd mode panels, and ABT Y030XX067A 320x480 LCD panels.
• Hardware monitoring: Corsair power-supply HID controllers, Maxim MAX127 12-bit 8-channel data acquisition systems, STMicroelectronics pm6764 voltage regulators, Delta Q54SJ108A2 power supplies, and Linear Technology LTC2992 I2C system monitors.
• Media: OmniVision OV9734 sensors, OmniVision OV02A10 sensors, and Amlogic 2D graphic acceleration units.
• Miscellaneous: Modtronix lcd2s 20x4 character displays, Arm DMC-620 memory controllers, Samsung Exynos generic interconnects, Intel Keem Bay USB PHYs, MediaTek MT7621 PCI PHYs, Ingenic USB PHYs, Mediatek MT6360 analog-to-digital converters, Dialog Semiconductor DA9121 regulators, NXP PF8100/PF8121A/PF8200 regulators, Mellanox BlueField performance monitoring counters, Dell Wyse 3020 power buttons, Dialog Semiconductor DA7280 haptic interfaces, TI PRU remote processors, Intel LGM SoC NAND controllers, and AMD sensor fusion hubs.
• Networking: Hirschmann Hellcreek TSN switches, Samsung S3FWRN82 UARTs, and OpenCompute TimeCard clocks.
• Pin control and GPIO: Qualcomm LPASS LPI, 8953, SC7280, and SDX55 pin controllers, Intel Lakefield, Elkhart Lake, and Alder Lake-S pin controllers, and Microsemi/Microchip serial GPIO controllers.
• Sound: NXP audio transceivers, Mediatek MT8192 audio interfaces, Nuvoton NAU8315 class-D amplifiers, Analog Devices ADAU1372 codecs, and Realtek RT715 SDCA codecs.
• It's also worth noting that there has been more than the usual number of obsolete drivers removed during this merge window. Quite a bit of cleanup has been happening across the driver subsystem.

Miscellaneous

• Support for the auxiliary bus, a virtual bus for multi-function devices, has been added.

Networking

• The 802.1Q "connectivity fault management" mechanism is now supported. See this merge message for (a bit) more information.
• Support for the WiMAX protocol has been moved to staging with the intent of removing it altogether in the near future. It would appear that this support has not actually worked for some time, so the number of users is thought to be zero.
• RFC 6951 — UDP encapsulation of the SCTP protocol — is now supported.
• Zero-copy TCP receives have seen a number of performance improvements, making this feature worthwhile for much smaller payloads; see this merge message for more information.
• There is a pair of new ioctl() calls to facilitate the bridging of PPP channels; see this commit for some documentation.

Security-related

• The seccomp() system call has gained support for constant-action bitmaps. This is a mechanism allowing seccomp() to determine that specific system calls are always allowed or denied and short out much of the processing work for those calls.

Internal kernel changes

• The arm64 and s390 architectures have removed set_fs().
• The migration disable functionality has been merged. The realtime tree has had this capability for years, but there is increasing need for it in the mainline as well.
• One user of migration disable is the kmap_local() API, which has also been merged.

By the normal schedule, the 5.11 merge window should close on December 27, but Torvalds has indicated that he might delay the 5.11-rc1 release if he falls behind on pull requests due to the holidays. The pace of merging thus far suggests, though, that nobody should count on the merge window lasting any longer than usual. As always, we'll post another summary once the merge window closes, whenever that may be.

Comments (7 posted)

Managing multifunction devices with the auxiliary bus

Device drivers usually live within a single kernel subsystem. Sometimes, however, developers need to handle functionalities outside of this model. Consider, for example, a network interface card (NIC) exposing both Ethernet and RDMA functionalities. There is one hardware block, but two drivers for the two functions. Those drivers need to work within their respective subsystems, but they must also share access to the same hardware. There is no standard way in current kernels to connect those drivers together, so developers invent ad-hoc methods to handle the interaction between them. Recently, Dave Ertman posted a patch set introducing a new type of a bus, called the "auxiliary bus", to address this problem.

Complex devices

Linux already includes a number of drivers for multi-function devices. One of the ways to support them is the Multi-Function Devices (MFD) subsystem. It handles independent devices "glued" together into one hardware block which may contain some shared resources. MFD allows access to device registers either directly, or using a common bus. In this second case, it conveniently multiplexes accesses on Inter-Integrated Circuit (I2C) or Serial Peripheral Interface (SPI) buses. As the MFD sub-devices are separate, MFD drivers do not share a common state.

The devices Ertman addresses do not fit well into the MFD model. Devices using the auxiliary bus provide subsets of the capabilities of a single hardware device. They do not expose separate register sets for each function; thus they cannot be described by devicetrees or discovered by ACPI. Their drivers need to share access to the hardware. Events concerning all sub-functionalities (like power management) need to be properly handled by all drivers. These devices will often be specialized processors running firmware and communicating with the host system (and the Linux drivers) by messaging. The available functions may not be known in advance, and thus must be discovered at run time.

The documentation patch in the auxiliary bus series cites a number of examples. The Sound Open Firmware (SOF) driver interacts with a single device exposing interfaces like HDMI output, microphones, speakers, testing, and debug hooks. NICs implementing both Ethernet and RDMA may need a driver supporting a common part of the functionalities, and then the specific Ethernet and RDMA drivers can implement specific parts on top of that.

Current kernels do not have a generic way to describe dependencies between drivers for this kind of device. A solution to the problem could be to have a way to attach secondary drivers to the primary one; this is exactly what the auxiliary bus implements.

Auxiliary devices and drivers

The patch set introduces two main concepts: The "auxiliary device" and "auxiliary driver". These implement the relationship between the main and the secondary drivers. The main driver maintains the device state, allocating and managing all shared data. It also unregisters all secondary drivers when shutting down. Secondary drivers, instead, handle the interactions with the specific subsystem they are implementing a device for.

Each main driver may expose a number of functionalities (devices) for secondary drivers. Only one secondary driver can attach to each of those functionalities.

The main driver creates an auxiliary device, represented by struct auxiliary_device:

    struct auxiliary_device {
        struct device dev;
        const char *name;
        u32 id;
    };

The combination of name and id must be unique; the complete device name is a combination of the module name and those two fields, connected by dots (.). That yields a result like modname.device_name.id.

The developer embeds this structure in the device structure of the main driver, with all shared data necessary for the communication between the main driver and secondary drivers. They may also add supplementary callbacks.

The sequence to initialize the main driver contains two steps. The first one is to call auxiliary_device_init():

    int auxiliary_device_init(struct auxiliary_device *auxdev);

It verifies the arguments and returns an error code if need be; in such case the initialization of the device should be aborted. If the first call succeeds, the second step is to call the macro auxiliary_device_add() with the initialized device; this will set up the device name and register the device itself.

The unregistration procedure also has two steps, consisting of calls to auxiliary_device_uninit() (necessary from the point when auxiliary_device_init() has succeeded) and auxiliary_device_delete(). Those functions have the following prototypes:

    void auxiliary_device_uninit(struct auxiliary_device *auxdev);
    void auxiliary_device_delete(struct auxiliary_device *auxdev);

This two-step approach was implemented in response to comments on earlier versions of the patch set. It allows the driver to allocate its own data between auxiliary_device_init() and auxiliary_device_add() with a possibility to free it correctly in the case of a failure.

The secondary devices, which will connect to the main driver, are represented by struct auxiliary_driver:

    struct auxiliary_driver {
        int (*probe)(struct auxiliary_device *auxdev,
		     const struct auxiliary_device_id *id);
        int (*remove)(struct auxiliary_device *auxdev);
        void (*shutdown)(struct auxiliary_device *auxdev);
        int (*suspend)(struct auxiliary_device *auxdev, pm_message_t state);
        int (*resume)(struct auxiliary_device *auxdev);
        const char *name;
        struct device_driver driver;
        const struct auxiliary_device_id *id_table;
    };

This structure includes a number of callbacks to manage the device's life cycle, and the id_table containing names of the devices the driver can bind with. All callbacks receive pointers to the parent's auxiliary_device, allowing access to the shared data.

The secondary devices are set up with auxiliary_driver_register():

    int auxiliary_driver_register(struct auxiliary_driver *auxdrv);

This function requires the probe() callback and the id_table to be filled in. When successful, it causes a probe() callback call for any matching devices. The secondary devices can access the shared data using container_of() and the auxiliary_device structure.

When unregistering a driver, the developer should call auxiliary_driver_unregister():

    void auxiliary_driver_unregister(struct auxiliary_driver *auxdrv);

First users

Together with the auxiliary bus implementation, Ertman posted changes to the SOF driver. The modified driver uses this infrastructure to implement a test driver, and a probes driver, allowing the creation of a new virtual audio device that can tap into the pipeline and allow listening in at any point.

Another user can be found in the networking subsystem; Leon Romanovsky posted a conversion of the mlx5 driver to use the auxiliary bus. The updated driver creates network, VDPA, and RDMA drivers for one physical device. Those changes allow the removal of a bunch of custom driver code. Parav Pandit followed up by using this functionality to implement device sub-functions.

The patch set has come to its fourth iteration in its current form, and witnessed a number of earlier ones under the names of ancillary and virtual bus. The development of the auxiliary bus patch set took time, and it created dependencies in other work. This caused a fair amount of pressure to get it upstream, and that led to some pushing on the list. In an attempt to push things forward, Dan Williams reposted the patch set, stating that "it looks good to me and several other stakeholders". After a review from Greg Kroah-Hartman, the auxiliary bus code was merged into the mainline for the 5.11 kernel release.

Comments (none posted)

Doing symbolic math with SymPy

On November 29, version 1.7 of SymPy, a Python library for symbolic mathematics, was released. The new version brings a large number of enhancements and bug fixes, and some minor backward incompatibilities. While these are enumerated in detail in the release notes, we will take advantage of this opportunity to look at some of the things that can be done with SymPy and explore its interface options through several detailed examples.

What is SymPy?

SymPy performs symbolic mathematical manipulations using Python. Like others of its kind, it can solve algebraic and differential equations, simplify expressions, apply trigonometric identities, differentiate, and integrate; SymPy knows things about sets, manifolds, tensors, and many other mathematical objects. It offers a convenient interface to Matplotlib, the Python plotting library that I looked at back in 2015; this allows a seamless graphical exploration of results.

Although they are not limited to algebra, members of this category of software are universally called computer algebra systems (abbreviated CAS), so I will adopt this term here. Computer algebra systems are not used primarily for numerical work, as is a program such as Octave, which I reviewed recently. They combine the encoded knowledge of specialists in diverse mathematical fields with strategies of expression transformation to perform actual symbolic mathematical manipulations. These programs do not replace a human mathematician in creating new mathematical knowledge, but they can solve problems that can be reduced to a set of rules and mechanical procedures, such as finding an integral or factoring a polynomial.

Who uses SymPy?

Anyone who needs to carry out non-trivial mathematical manipulations can potentially benefit from a computer algebra system. It can be used to check work done by hand, to replace weighty tomes of formulas, to solve large systems of equations that cannot (or should not) be done with pencil and paper, or to explore, symbolically and graphically, a mathematical landscape.

There are many computer algebra systems available, both of the free-software variety and proprietary offerings. The most well-known of the commercial programs is certainly Mathematica, while SymPy is one of the most popular free-software computer algebra systems. Their predecessors go back decades. The first system of this kind was Macsyma, which arrived in the 1960s, and is still offered as a commercial product, though apparently only for Windows. I wrote about its free-software offshoot, Maxima, in a 2017 LWN article where I surveyed programs for symbolic mathematics available for Linux.

Maxima was for many years the standard general-purpose, free-software computer algebra system. Although it is still the preferred such program for many practitioners, the comparatively young SymPy has become a prominent choice over the past decade. This is due to several factors, such as its inclusion in other packages, in particular Sage, which is a large system that bundles and provides a unified interface to many numerical, graphical, and symbolic software packages. Another reason behind SymPy's wide adoption is that it is a Python library, so the large population familiar with Python's syntax will not be tripped up when using SymPy. In contrast, Maxima is written in Lisp and has its own syntax that the user is obligated to master.

Many who use SymPy do so within Sage; however, for those who don't need the other things that come with Sage, installing standalone SymPy makes more sense. For one, Sage takes up over a gigabyte of storage space, due to the large number of packages that it bundles, while my SymPy installation takes up just 63 megabytes, including its 45 standard modules. SymPy can be included in other Python code with a single import statement, and its liberal BSD license presents no obstacles to any form of distribution.

SymPy's only dependency is mpmath, which is a library for arbitrary precision arithmetic that is also BSD-licensed. Users installing from source will have to install mpmath before installing SymPy, but pip will install it automatically. SymPy and mpmath are included in projects such as the Anaconda data-science platform. In addition, there is a symbolic math package for Octave that implements SymPy inside of that numerical toolbox.

Using SymPy

Since SymPy is a Python library, it can be used wherever Python can be used. It can be imported into scripts, incorporated into applications, or used from any of Python's interactive environments, including the plain read-eval-print loop (REPL), the IPython enhanced REPL, or Jupyter.

Let's first try the IPython interface. After typing ipython in a terminal and getting the IPython startup message, the user types two commands, shown in the figure below, to start up the SymPy environment. The first command imports one function from SymPy, which is then run to bootstrap the rest. This function, init_session(), imports the rest of SymPy and then invokes the SymPy symbols() function three times.

The purpose of the calls to symbols() is to define some names for variables that can be used in mathematical expressions. This is necessary because, in Python, you may not use undefined symbols in expressions, but in a computer algebra system you need to refer to variables somehow—symbols that, by definition, have not been assigned a value. init_session() establishes a handful of variables with conventional names, and of course the user is free to define as many more as required. Finally, init_printing() sets up any defaults for printing results to the terminal.

If the terminal in use supports Unicode, SymPy will form mathematical output from Unicode glyphs. To see what this looks like, we'll ask it to simply represent an integral for us, using the Integral() function, which returns an unevaluated integral. The figure below shows how the integral sign is built up to the required height using line segment glyphs, how Greek letters are used for mathematical constants, and the infinity symbol, which is entered on the command line using a double "o".

Now let's do a little math. As mentioned above, SymPy comes with a good number of specialized modules. In the following figure we import the geometry module, which knows about a variety of planar geometrical objects. We define a point at the origin, and a circle centered there, with a radius of one. Then we define two more points, and a line connecting them. This line should be horizontal and touch the circle at its highest point. The next two commands check on that.

First we ask SymPy whether the line is tangent to the circle, and it confirms that it is. Then we ask where the line and circle intersect. The answer, at x = 0 and y = 1, is what we expected.

The qtconsole interface for SymPy has several advantages over the IPython interface. The most significant is that it can typeset math using LaTeX, which helps immensely in reading complicated expressions in the output. Other nice features are embedded graphs and automatic function documentation: when typing an open-parenthesis after a function name, the qtconsole will immediately pop up a box with the function signature, if it recognizes the name, and some documentation about the function, as shown in the figure below:

Hitting escape or clicking anywhere in the window dismisses the documentation box. Tab-completion works in qtconsole, as does accessing command history with the arrow keys, which is the same as it is in the IPython interface. At least on my machine, qtconsole is deaf to control-C, but I was able to cancel any pending input with the escape key.

The disadvantages of using the qtconsole are that it involves extra installs; on my machine I had to install the PyQt5 package with pip. It also did not, at first, handle my screen resolution correctly. Those with screen resolutions larger than 72 dots per inch will need to enter the command:

    init_printing(scale=2, forecolor='Black')

That will create larger LaTeX output. The scale factor can be set as desired, and the color setting creates more legible output than the default medium grey.

If using inline plots, which you get with the "%matplotlib inline" command, Matplotlib also needs to know about the screen resolution:

    matplotlib.rcParams['figure.dpi'] = 220

The number need not match the actual resolution, but can be set as a convenient way to get the desired figure size. All of the above commands are entered at the interactive prompt within the qtconsole session. After this is done, the interface performs well and is a pleasure to use.

The following figure is designed to convince the user of the advantage of LaTeX output. Simple-looking problems often lead to hairy expressions; the result of solving this innocent-looking integral would be difficult to parse using ASCII or even SymPy's well-executed Unicode output.

The next example includes an embedded line graph and shows off SymPy's knowledge of special functions and integrals. It asks for the integral of a Gaussian, which is called the error function, written as erf(). The plot command uses _ in the argument, which stands for the most recently returned result. The basic plotting functions are imported automatically with SymPy.

Our last example demonstrates multiple integration and surface plotting. Before using this plotting function, plot3d must be imported from sympy.plotting. The plot command again refers to the previous result, and defines limits for the x and y coordinates.

The new release

The 1.7 release brings a few backward-incompatible changes. Some of these involve the naming of modules, so the import statements in existing scripts may need to be updated. Another possible disruption for some users is the dropping of support for Python 3.5; the minimum version now required by SymPy is Python 3.6.

The new version brings a handful of improvements to SymPy's understanding of trigonometric functions, absolute values, and other mathematical objects, which improve its ability to simplify expressions. Translation of the output to LaTeX now works better in some situations, and it is more convenient to substitute another plotting library for Matplotlib.

New capabilities were added to several modules, including physics, statistics, and the ordinary differential equation solver; the relevant sections of the release notes contain links to the issue tracker entries for each of these, which users of these packages may want to consult.

Documentation and assistance

The starting place to learn how to use SymPy is the documentation based on the current release, which covers the various methods of installation, includes a nicely done tutorial aimed at people who might not know what a computer algebra system is, has a detailed reference to all the standard SymPy modules, and contains information about the SymPy project. This manual is mostly up-to-date and quite good, although it contains a few corners with obsolete information.

One wonderful feature of the documentation is that every code block comes with a button that opens a window where a live notebook interface called SymPy Live is running; the code in the block is copied to the window and executed.

There are some glitches, however. The documentation is up to date, covering, as of this writing, version 1.7.1; but the SymPy Live instance is still on 1.5. The consequence of that is that some code blocks in the documentation produce errors in the SymPy Live window. Another problem, unavoidable in a service such as this, is that the response is much slower in general than what a user will experience with a locally installed SymPy. Direct access to SymPy Live is also provided here, free for anyone to experiment with the software.

Users seeking help from others may want to turn to the general mailing list, hosted on Google Groups, or the Gitter channel; there is also an IRC channel called #sympy.

SymPy Gamma is an interesting project with SymPy at its core. It aspires to offer something similar to Wolfram Alpha, which is an online mathematical knowledge engine. SymPy Gamma presents several example topics in mathematics and the user can type in problems or questions. It has made impressive progress, but it not yet in the same league as Wolfram Alpha, in general failing to answer open-ended questions about mathematical objects, but presenting a nice variety of information in response to a more focused query.

If the user simply types a mathematical expression, the engine will reply with its typeset form, graph (only of single-variable expressions), roots, derivative, antiderivative, series expansion, and sometimes more, depending on the type of expression entered. This is all provided without the user needing to know the syntax for calculating these things, so this is a way to make some use of SymPy without having to learn anything about it, or even about Python, aside from the basic syntax related to mathematics.

Anyone in need of a general environment for computer-assisted mathematics will be well-served by either Maxima or SymPy. Which one to use depends on several factors, which include the availability and maturity of modules for any needed specialized areas of math or science. Lisp experts interested in possibly adding to the system or improving existing routines, while occasionally dropping down into the Lisp subsystem, will naturally gravitate towards Maxima. SymPy is a good choice for those already experienced with Python or who are interested in adding mathematical intelligence to a Python project.

Comments (5 posted)