Leading items

Welcome to the LWN.net Weekly Edition for September 14, 2023

This edition contains the following feature content:

• The bogus CVE problem: the Common Vulnerabilities and Exposures mechanism, which has not lacked for problems over the years, is challenged by low-quality (and false) entries.
• Replacing openSUSE Leap: changes at SUSE are depriving the openSUSE Leap distribution of its stable base; what will come after Leap?
• Prerequisites for large anonymous folios: a discussion on what is needed to enable the kernel to use large folios for anonymous memory.
• The rest of the 6.6 merge window: the changes pulled for 6.6 in the latter part of the merge window.
• Arduino: open source for microcontroller boards: catching up on the current and future state of Arduino.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

The bogus CVE problem

The "Common Vulnerabilities and Exposures" (CVE) system was launched late in the previous century (September 1999) to track vulnerabilities in software. Over the years since, it has had a somewhat checkered reputation, along with some some attempts to replace it, but CVE numbers are still the only effective way to track vulnerabilities. While that can certainly be useful, the CVE-assignment (and severity scoring) process is not without its problems. The prominence of CVE numbers, and the consequent increase in "reputation" for a reporter, have combined to create a system that can be—and is—actively gamed. Meanwhile, the organizations that oversee the system are ultimately not doing a particularly stellar job.

A recent incident highlights some of the problems inherent in the system. CVE-2020-19909, which is an integer-overflow bug in the curl tool and library for URL-based data transfers that was only reported to the project in 2023. In a blog post describing the mess, curl maintainer Daniel Stenberg said that a message to the curl-library mailing list on August 25 alerted the project that the CVE had become public the week before.

The year in the CVE number (2020 in this case) is meant to indicate when the bug was reported to one of the more than 300 CVE numbering authorities (CNAs) that hand out CVE numbers. Under normal circumstances, a new bug showing up with a CVE number would have 2023 in it, but sometimes CVEs are given out for older bugs that somehow slipped through the cracks. That appears to be what happened in this case, as Stenberg was able to track the problem back to a bug report from Jason Lee in mid-2019.

The report was for a legitimate bug, where the ‑‑retry‑delay option value was being multiplied by 1000 (to milliseconds) without an overflow check. But what it was not was a security bug, Stenberg said; giving insanely large values for the option might result in incorrect delays—far shorter than requested—but it is not a security problem to make multiple requests in a short time span. If it were, "then a browser makes a DOS [denial of service] every time you visit a website — and curl does it when you give it two URLs on the same command line", he said in a followup post.

The problem was duly fixed, a test case was added, and Lee was credited with the report in the commit message. In September 2019, curl 7.66.0 was released with fix, which was mentioned in the announcement; also notable are the two CVEs mentioned at the top of the bug fixes listed. As Stenberg noted, the curl project works hard to ensure that it fully documents the (real) CVEs that get filed for it; his exasperation with CVE-2020-19909 coming out of the blue is evident:

In the curl project we work hard and fierce on security and we always work with security researchers who report problems. We file our own CVEs, we document them and we make sure to tell the world about them. We list over 140 of them with every imaginable detail about them provided. We aim at providing gold-level documentation for everything and that includes our past security vulnerabilities.

That someone else suddenly has submitted a CVE for curl is a surprise. We have not been told about this and we would really have liked to. [...]

The National Vulnerability Database (NVD) tracks CVEs and "scores" them using the Common Vulnerability Scoring System (CVSS), which is a ten-point scale that is meant to give an indication of the severity of a vulnerability. For the curl bug, which should probably not be scored at all, NVD initially came up with a "9.8 critical", scoring an integer overflow in a delay parameter as one of the most severe types of vulnerability possible. Stenberg, who has tangled with NVD over scoring before, is even further exasperated:

It was obvious already before that NVD really does not try very hard to actually understand or figure out the problem they grade. In this case it is quite impossible for me to understand how they could come up with this severity level. It's like they saw "integer overflow" and figure that wow, yeah that is the most horrible flaw we can imagine, but clearly nobody at NVD engaged their brains nor looked at the "vulnerable" code or the patch that fixed the bug. Anyone that looks can see that this is not a security problem.

In fact, the pull request for the fix was attached to the report, but that apparently made little difference in the assessment from NVD. Back in March, Stenberg decried the NVD scoring process and, in particular, the NVD practice of re-scoring CVEs that have already had severity levels attached to them. NVD uses CVSS, but the curl project long ago rejected that scoring system:

In the curl project we decided to abandon CVSS years ago because of its inherent problems. Instead we use only the four severity names: Low, Medium, High, and Critical and we work out the severity together in the curl security team as we work on the vulnerability. We make sure we understand the problem, the risks, its prevalence and more. We take all factors into account and then we set a severity level we think helps the world understand it.

His example in that case is a double-free in curl that the project determined had a "medium" severity, while NVD scored it as "9.8 critical", as can be seen in the GitHub advisory database. Since then, NVD apparently had a change of heart after Stenberg's complaint as the CVE is now scored "8.1 high". In a followup on NVD "brokenness", Stenberg gave another example of a CVE that was initially scored "9.8 critical", but eventually was reduced to "5.9 medium" after complaints—though the curl project rates it as "low". He also noted that there is a set of projects that never report low or medium CVEs that they find, in order to avoid these kinds of scoring woes.

One could perhaps wonder if this is all just a problem for the curl project and not more widespread, but there is a fair amount of evidence of a variety of problems in CVE-land. For example, the PostgreSQL project had a similar problem with a CVE "from" 2020 that appeared recently—and is not a security vulnerability at all, according to the project. In June, the KeePassXC password manager project had a bogus CVE filed for the tool; there are other examples as well.

Each of these bogus CVE filings, which can apparently be made anonymously and without much in the way of backing information, require that the project notice its existence, analyze the problem (or "problem"), and, if necessary, dispute the existence or score of the CVE. As noted, several curl CVEs have had their scores reduced rather substantially due to requests from the project. The delay parameter overflow that was initially scored 9.8 has been reduced to "3.3 low", marked as "disputed", and had a link to Stenberg's blog post added to the NVD entry.

Keeping up with all of that is a lot of work, which Stenberg said he is going to try to avoid in the future by applying to become the CNA for curl. Several other open-source projects are CNAs, which gives them some notification of a reported problem along with ways to try to ensure that the problem is handled sanely. He mentioned Apache, OpenSSL, and Python as some of the projects that are already CNAs; Python was just granted CNA status at the end of August.

Meanwhile, though, CVEs are used in ways that elevate their importance well beyond the level that makes sense given the amount of scrutiny that is apparently applied to them. Service-level contracts and governmental requirements mean that a critical CVE needs to be addressed in short order, so non-critical bugs that get marked that way can cause real problems. It does provide incentives for companies and others to try to downplay the severity of bugs, as well, of course, which makes for something of a "Pushmi-Pullyu" in CVE-land.

As was alluded to in our mid-August look at kernel security reporting for distributions, the CVE system is generally included in the "security circus" that kernel developers largely disdain. A 2019 talk by Greg Kroah-Hartman described multiple problems that he sees with the system as well.

All in all, the CVE system seems to be broken in various ways. It also seems to be getting more and more entrenched into "cybersecurity" handling at various levels. Given that it is effectively run by—and now for—governmental agencies, the ability to replace it with something more sensible has likely already passed us by. CVE, warts and all, will be with us for a long time to come; FOSS projects and organizations are simply going to have to figure out how to coexist with it.

Comments (28 posted)

Replacing openSUSE Leap

OpenSUSE Leap is a hybrid distribution; it is based on SUSE's enterprise distribution (SLE), which follows the "slow and stable" approach, but adds a number of newer packages on top. Leap is intended to be a desktop-oriented distribution with a stable and reliable base. As SUSE transitions away from its traditional enterprise distribution toward its "Adaptable Linux Platform" (ALP), though, the stable base upon which openSUSE Leap is built is going away. The openSUSE community is currently discussing how the project should respond.

ALP is, by design, an amorphous project; it is intended to facilitate the creation of focused distributions for a number of different use cases. The initial targets appear to be server workloads, with a focus on containers and confidential-computing uses. While it is somewhat hard to get a grasp on what future SUSE products will look like, the company does not appear to be heading toward the creation of another traditional enterprise distribution as part of the ALP transition.

Rolling slowly

This change poses both a challenge and an opportunity for the openSUSE project. The loss of the existing base means that it will not be possible to continue to create Leap in its current form, but the flexibility of ALP means that a number of options may open up in the future. Before pursuing one or more of those options, though, the community must decide what it wants to do — and what it has the resources to do. To that end, a survey of openSUSE users and contributors was recently run; the results have been posted, as has an analysis of those results by Richard Brown. The latter has sparked the beginning of what may be a long conversation in the openSUSE community.

The results are somewhat ambiguous. On the one hand, there seems to be a significant portion of the community that would prefer to keep Leap going in something close to its current form. On the other, though, there is also a lot of interest in rolling distributions, which are almost the opposite of how the core of Leap is managed. Respondents were asked about two possible distributions that the community could create to take Leap's place:

• "Linarite" would be "a regular old fashioned release desktop distribution", though with a smaller package selection than is provided by Leap now. It would be based on a SUSE project known as "Granite"; there is little public information available regarding this project, but it seems to be the ALP-derived product that will be the closest to a traditional enterprise server distribution.
• "Slowroll", instead, is a rolling distribution based on the existing openSUSE Tumbleweed offering (which is not built on enterprise packages and can be expected to continue without big changes), but with a stronger focus on stability. Packages would move from Tumbleweed to Slowroll when they have shown, over a period of time, that they are unlikely to introduce new problems. Slowroll, Brown said, is "an attempt to provide something less scary than full speed Tumbleweed".

After looking at the survey responses, Brown came to a few conclusions regarding where, in his opinion, the openSUSE community should focus its efforts. By his reading, the respondents "overwhelmingly" support rolling releases; as a result, he said, any Leap replacement should look like Slowroll rather than Linarite. "It is the most popular with our users, and the option more closely aligned to what our contributors use themselves."

Beyond that, he said, openSUSE should focus firmly on desktop use cases rather than trying to build something that might be suitable for server deployments as well. Given that openSUSE will be able to package and ship its own copies of the SUSE products coming out of ALP, and those products will "be awesome for folk who want Enterprise-like server distros", Brown does not see much point in openSUSE expending its resources in that direction. Instead, since SUSE has little interest in the desktop area at this point, that is where openSUSE can make a real contribution.

An openSUSE Slowroll can only work, though, if the community can pull together the resources to build it, and Brown is "concerned that we do not have enough contributors to make any Leap replacement viable". Creating and maintaining Slowroll would require more resources than the current Leap effort, and he is not convinced that those resources will be available.

Michal Suchánek questioned the idea that building and maintaining Slowroll would require more work, given that it would mostly just pull packages from the existing Tumbleweed distribution. Brown answered that it would require a lot of work that is not being done (by the openSUSE community, at least) now:

Because, unlike Leap where maintenance for SLE packages is effectively 'automatic' (ie. taken care as part of the daily business of SLE), and unlike Tumbleweed where it's also effectively 'automatic' ('just throw a new version at it), Slowroll will likely require old-fashioned maintenance (CVE bumps, backports, narrow-fixes) for packages in Slowroll but not-yet-ready to be copied from Tumbleweed.

If the project is going to do Slowroll at all, he added, it needs to do the job properly so that it is truly a more secure and stable offering than Tumbleweed.

Matěj Cepl, instead, suggested that ALP might be Slowroll, and that openSUSE could have it for free. Brown's response was that, if ALP ends up producing a rolling release at all, it will advance at a rather slower rate than the community would like. Slowroll, he said, is an opportunity to create a rolling release that rolls at a speed chosen by the openSUSE community.

Taking Leap for Granite

Not everybody in the community has given up on continuing to produce a distribution that, even if it is not Leap, would have many of the same characteristics as Leap. Simon Lees expressed interest in building such a distribution on top of SUSE's Granite, whatever that turns out to be. Brown pointed out that Granite is not planned for release before 2025; given that there is currently no plan within openSUSE to create a Leap 15.7 release, basing a successor on Granite would likely leave Leap users without a supported alternative for at least a year. Lees, though, is convinced that a Granite-based Leap replacement is a viable option, and seems determined to prove it as Granite takes shape:

Well i'm the sort of person that will take the goal of having a distro based on Granite before the end of 15.6 and push my hardest to resolve the issues to get there and if come September / October 2025 that is looking unlikely i'd start to look at temporary workarounds to bridge the gap.

Suchánek asked why there was no Leap 15.7 release planned; Brown answered that creating that release would take resources — specifically the SUSE developers tasked to work on openSUSE — away from helping with a Leap replacement. He later added that: "if I were to persuade SUSE to sponsor the work for a 15.7, that 15.7 will not just be the last Leap release, but we'll almost certainly have no replacement going forward". That, he said, would be the end of a stable, desktop-oriented openSUSE release:

openSUSE Leap is a very Desktop-centric distribution.

None of SUSE's planned ALP products have any intention of being a Desktop-centric offering.

If openSUSE doesn't start the work now, then ALP will never be ready for the Desktop use cases we currently enjoy with Leap.

In April, Brown had looked at where ALP was headed and concluded that "if we the community want to build something like old Leap, that should be totally technically feasible". Now, though, he seems to be arguing that openSUSE only has one viable option, Slowroll, going forward. Lees, instead, appears to still be in favor of doing something more along the lines of an "old Leap" with an upcoming ALP product as the base. What the community as a whole feels is the best approach is yet to be determined.

It is clearly an important decision point for one of the oldest existing Linux distributions. Perhaps openSUSE's biggest problem, though, is one that only appeared at the edges of this conversation: how can the project attract more contributors so that its options might be less limited in the future? According to Brown's initial message, only 61 people contributed to the last Leap release. Without people to do the work, the direction that openSUSE would like to go will only be so relevant.

Comments (12 posted)

Prerequisites for large anonymous folios

The work to add support for large anonymous folios to the kernel has been underway for some time, but this feature has not yet landed in the mainline. The author of this work, Ryan Roberts, has been trying to get a handle on what the remaining obstacles are so he can address them. On September 6, an online meeting of memory-management developers discussed that topic and made some progress; there is still some work to do, though, before large anonymous folios can go upstream.

Folios are, at their core, a simple mechanism to group physically contiguous pages into larger units that can be managed more efficiently. Over the last couple of years, many parts of the memory-management subsystem have been converted to use folios, but the management of anonymous pages (pages representing data that is not backed up by a file) has proved to be somewhat tricky.

Large anonymous folios

Roberts started the discussion with a review of the work. Whenever it handles a page fault on an anonymous virtual-memory area (VMA) that requires allocating a page of new memory, the kernel will attempt to allocate a group of pages as a folio rather than a single page. This folio will normally be between 16KB and 64KB in size. It will be mapped into the process's address space at the page-table-entry (PTE) level — there is no alternative to doing that, since these folios are smaller than the huge-page size.

The addition of large anonymous folios is driven by performance concerns, he said. Allocating and mapping an entire folio will reduce page faults, since the process will not need to fault in the other pages in that folio. A lot of per-page operations, such as the management of reference counts, become per-folio operations instead, so there are correspondingly fewer of them. Using folios reduces the length of the memory-management subsystem's least-recently-used (LRU) lists, making the scanning of that list more efficient. On some architectures (specifically 64-bit Arm), zeroing a 64KB folio is faster than zeroing individual pages.

Turning to specific use cases, Roberts said that this feature provides better performance on Android systems running with 4KB base pages. It allows the kernel to use the CPU's "contiguous-PTE" bits, which indicate that a contiguous set of PTEs maps to contiguous pages of memory; that, in turn, allows the hardware to optimize the translation lookaside buffer (TLB) entry for those pages. On a 64-bit Arm system with 4KB base pages, using the contiguous-PTE requires 16-page (64KB) contiguous mappings, which large anonymous folios can provide. Support for the contiguous-PTE feature is implemented in a separate patch set.

There is a separate use case on Arm systems running with 64KB base pages. On such systems, the PMD size (the size of the smallest huge pages) is 512MB; that much physically contiguous memory is difficult to allocate, so huge pages tend not to be used when running with 64KB pages. The contiguous-PTE size, though, is 2MB, which is much easier to manage. Large anonymous folios can be used to provide a sort of "2MB transparent huge page" feature that improves performance.

Using a larger base-page size has a similar effect to using large folios everywhere, so it is not surprising that Roberts was asked whether switching Android systems to 64KB base pages was out of the question. The answer was "yes", but that using 16KB pages might be an option; even then, though, there are problems with app compatibility and wasted memory due to internal fragmentation. That question led naturally to Roberts's next point, though, which was illustrated with benchmark results.

Roberts did a number of tests with the industry-standard kernel-compilation benchmark, and with Speedometer 2.0 as well. In both cases, enabling large anonymous folios improved performance significantly, on a scale similar to what can be achieved using 16KB base pages. The increase in memory usage, though, was tiny compared to the situation with 16KB base pages (and 64KB base pages was far worse yet). Large anonymous folios, thus, look like a way to gain many of the performance advantages provided by a larger base-page size without most of the associated cost.

The patch series is currently in its fifth revision, posted on August 10. It has had a lot of reviews, and a number of prerequisite changes have been identified. There are a few open issues that he wanted to discuss: naming, runtime controls, and which statistics to measure and report. He did not manage to get through even that short list in the time allotted, though.

Starting with naming, Roberts said that the feature should have a name that tells users what it does. "Large anonymous folio" might work, but he is not sure that it is the best option. Others he suggested were "transparent large page", "flexible transparent huge page", and "small transparent huge page". It quickly became clear, though, that Roberts would not succeed in getting the assembled developers to disagree over the name; they had other concerns instead.

How to turn it off

David Hildenbrand said that the real question to answer is the extent to which large anonymous folios should be used automatically. The biggest problem with the existing transparent huge page feature is memory waste, and the same situation will arise with this feature. If large anonymous folios are added to the kernel without a way to disable them, he said, the result will certainly be problems for some applications. The feature is reasonable, he said, but it needs a way to avoid and/or recover from over-allocation of large folios.

The problem of memory waste comes about when applications create sparse mappings — regions of memory where the actively used pages are scattered and surrounded by unused pages. If a large folio is allocated in a sparse mapping, most of it will likely consist of unused pages, which are wasted. Roberts questioned the use cases for such mappings, but it seems that they are reasonably common. Memory allocators often create them; highly threaded applications with a lot of thread stacks are another example.

Roberts said that he saw a couple of approaches for dealing with this problem, preferably in a way that lets him get a minimal patch into the kernel to better characterize how large anonymous folios affect performance. It seems that there needs to be a way to turn the feature on or off at a minimum. Even better, though, would be some sort of dynamic mechanism that would check on page usage and break apart folios that are not fully utilized. He is not sure how to do that without scanning the pages in question, though.

Hildenbrand said that, if Roberts wants large anonymous folios to be enabled by distributions, there has to be a way to toggle it on or off. The best way, of course, would be to have the kernel tune this behavior automatically and not add any new knobs at all. But some way must be found, he said, to deal with the memory-waste problem, whether it's reclaiming incorrectly allocated folios, giving user space a way to tell the kernel about sparse mappings, or some other sort of mechanism.

Roberts concluded that the kernel will have to get better at discovering and reclaiming unused pages in folios. That is not necessarily a small job, though, and he would really like to start with a minimum patch that people can play with. Hildenbrand said that Roberts should not worry too much about introducing a new toggle to just disable the feature; if the kernel gets to a point where it can tune its own behavior effectively, it can just ignore the setting of the knob.

As it turns out, Roberts already had a proposal for a control knob, called laf_order, that would be found in either debugfs or sysfs (there is some disagreement over which place is better). Setting this knob to zero disables large anonymous folios entirely, while setting it to "auto" (the proposed default) leaves the management of anonymous folios to the kernel. It can also be set to an integer indicating the allocation order to use for anonymous folios — setting laf_order=4, for example, would set the size to 16 pages, or 64KB. Yu Zhao argued that the knob needs to accept a list of orders rather than a single one, with the kernel falling back through the list if the larger orders cannot be allocated.

Matthew Wilcox, instead, pointed out that different applications will want different behavior. A system-wide laf_order knob is good for expressing what the system administrator wants, but the administrator likely does not know what the software wants. Hildenbrand, though, said that there should not be too much effort put into optimizing this feature; it should be possible to disable it when it is not useful, and to play with it otherwise. He reiterated that automatic tuning would be best; user space should not be expected to know better than the kernel when it comes to memory management.

John Hubbard said that there is a need for per-application tuning of the feature, or it will be disabled by users; David Rientjes said that it was looking like there would be a need for applications to tell the kernel which folio size to use for each VMA. Wilcox protested that, "when applications tell the kernel that they know what they are doing, they don't". Instead, the kernel should watch what an application does, ramp up the use of large anonymous folios conservatively, and track how those folios are used.

Roberts said that he had thought about a "per-VMA slow start" feature where, if page faults within a VMA form in contiguous blocks, the folio size for that VMA would be increased. Wilcox agreed that this idea was worth looking into. Roberts said that the universal toggle still felt like the first step to take, though.

Out of time

At that point, the hour was done; Roberts lamented that he had come with three topics to discuss and had never gotten past the first. He is still looking for a roadmap to get him to a bare-minimum patch that could get into the kernel. Other improvements can be made after that. There was not much agreement on what that minimum should be, though. Hildenbrand suggested a simple toggle to disable the feature; Zhao, instead, argued for a separate toggle attached to each memory control group — and that a per-VMA toggle would be better yet. Hubbard suggested a set of controls that mirrors the knobs that have been developed to control transparent huge pages.

In a sense, the meeting came to an end without having resolved much of anything. But it did shine a light on some of the problems that will have to be solved before large anonymous folios can be a part of a mainstream kernel release. The experience with transparent huge pages shows that introducing a feature before it is truly ready can cause long-term difficulties of its own, with users disabling the feature long after its problems have been solved. With luck and some patience, large anonymous folios will eventually enter the kernel without such troubles.

Comments (15 posted)

The rest of the 6.6 merge window

Linus Torvalds released 6.6-rc1 and closed the 6.6 merge window on September 10. At that point, 12,230 non-merge changesets had been pulled into the mainline repository, which is exactly 500 more than were pulled for 6.5 at this stage in the cycle. Over 7,000 of those changes were pulled after our first-half summary was written; they brought a fair amount of new functionality with them. Read on for an overview of those changes.

Significant changes pulled in the second half of this merge window include:

Architecture-specific

• After years of work and discussions, the Intel shadow-stack patches have finally been merged. This hardware feature hardens user-space code against stack-overwrite attacks.
• The PowerPC architecture now supports "hotplug SMT", where some or all of the simultaneous multithreading CPUs in a given core can be enabled or disabled at run time.
• Changes merged for RISC-V include support for user-space access to performance counters, the ability to locate crash kernels above the 4GB boundary, support for LLVM-implemented forward-edge control-flow integrity, and support for kernel address-space layout randomization.
• The s390 architecture has removed the noexec command-line parameter, on the reasoning that there is no point in being able to disable non-executable mappings. This change mirrors a similar removal for x86 in 5.19.
• The LoongArch architecture has gained support for a number of kernel-development tools, including KDB, KGDB, kcov, KFENCE, and KASAN.

Core kernel

• The new "eventfs" subsystem significantly improves the memory efficiency of the tracing subsystem. There are thousands of tracepoints in the kernel, each of which requires inodes and dentries for its representation in the filesystem. Those structures exist even if tracing is not used, wasting memory. Eventfs will allocated the needed structures only when they are needed, eliminating that waste.
• The /proc/pid/smaps file now includes information on how well kernel samepage merging (KSM) is working to reduce memory usage.

Filesystems and block I/O

• The overlayfs filesystem has gained the ability to use fs-verity data to ensure the integrity of data supplied by lower-level filesystems; this is part of a larger effort to get overlayfs to provide the features needed by the composefs filesystem.
• The reiserfs filesystem has been marked as being obsolete, a step toward its eventual removal.
• The process of merging the online repair functionality for XFS has begun, though it may be a while yet before this code is all upstream and ready for use.
• The NFS server now supports NFSv4 write delegations. This mechanism allows for more caching of data and metadata for a file being written to, reducing network traffic and increasing performance.
• The Ceph filesystem has gained fscrypt support.
• The ksmbd server (providing in-kernel SMB3 filesystem support) is no longer considered "experimental".

Hardware support

• Clock: StarFive JH7110 clocks, Renesas VersaClock 3 clock controllers, and Qualcomm IPQ5018 global clock controllers.
• GPIO and pin control: Amlogic C3 SoC pin controllers, Qualcomm SM6115 and SM8350 LPASS LPI pin controllers, and Intel Tangier pin controllers.
• Graphics: Visionox R66451 panels and Loongson display controllers.
• Industrial I/O: Allwinner D1/T113s/T507/R329 analog-to-digital converters, Murata IRS-D200 PIR sensors, and Microchip MCP4728 digital-to-analog converters.
• Input: Azoteq IQS7222D touch controllers and Azoteq IQS7210A/7211A/E trackpad/touchscreen controllers.
• Media: Generic I2C address translators, TI DS90UB960 FPD-Link III deserializers, TI DS90UB913 and DS90UB953 FPD-Link III serializers, Dongwoo DW9719 lens voice coils, and Intel visual sensing controllers.
• Miscellaneous: Huawei HCCS interconnect controllers, Loongson-2 SoC power management controllers, Schweitzer Engineering Laboratories SEL-3350 LED controllers, Siemens Simatic CMOS batteries, Nuvoton NPCM PECI controllers, IBM I2C responders, NXP QorIQ eFuses, Qualcomm secure QFPROMs, Google Stadia force-feedback controllers, Starfive JH7110 PCIE 2.0 PHYs, StarFive JH7110 D-PHYs, Qualcomm M31 HS PHYs, Maxim MAX735x/MAX736x I2C mux/switches, NXP PCA995x I2C-controlled LEDs, Siemens Simatic IPC BX-21A LEDs and Marvell GTI watchdog timers.
• Sound: TI TAS2781 codecs, Cirrus Logic CS35L56 amplifiers, Realtek RT1017 SDCA codecs, Awinic aw88261 amplifiers, Intel Lunarlake audio interfaces, AMD ACP5x audio interfaces, and Cirrus Logic CS42L43 codecs. Also: the sound layer can now manage devices provided by the industrial I/O subsystem as audio devices.
• USB: Realtek RTD USB2 and USB3 PHY transceivers, USB MIDI 2.0 gadget interfaces, and Starfive JH7110 USB 2.0 PHYs.

Miscellaneous

• The tools/workqueue directory includes a new tool, wq_dump.py, that can be used to examine the current configuration of the system's workqueues and how they map onto worker pools. See this commit for some more information.
• Another set of enhancements to the perf tool has been merged; see this commit for details.

Security-related

• The NFS server no longer supports the (long-deprecated) DES and triple-DES Kerberos encryption types.

Virtualization and containers

• The VFIO subsystem provides a new, char-device interface for controlling VFIO devices.
• The kernel can now support SEV-SNP and TDX guests on Hyper-V.

Internal kernel changes

• Unbound workqueues have gained a number of options aimed at improving performance on larger systems; see the latter part of this article and the "affinity scopes" section of Documentation/core-api/workqueue.rst for the details.
• Kernel builds with W=1 set now enable the -Wformat-overflow, -Wformat-truncation, -Wstringop-overflow, and -Wrestrict warnings. The -Wenum-conversion warning is now enabled unconditionally.
• There was an increase of 326 exported symbols during this merge window; see this page for the full list of changes. One kfunc (bpf_map_sum_elem_count()) was added as well.
• The support files for the GitLab continuous-integration tests used by the graphics subsystem have been merged. See Documentation/gpu/automated_testing.rst for more information on how this testing works.

One significant change that was not merged was, once again, the bcachefs filesystem. Curious readers can see the associated discussion; in short, Torvalds was not happy that the code had never appeared in linux-next (a situation that is being rectified) and had some problems that wider testing should have found. Even Torvalds, though, said that "any further changes are better done in-tree"; it should be possible to get this code into 6.7.

Meanwhile, the time has come to fix as many bugs as possible and to get this code into shape for the 6.6 release. Assuming the usual schedule holds (and it always does anymore), that release can be expected on October 29 or November 5. While it is possible that the next release, 6.7, could happen on December 31, that seems unlikely, meaning that 6.6 is almost certainly the final kernel release for 2023. Should that prove to be the case, it will probably be the long-term-support release for this year as well.

Comments (9 posted)

Arduino: open source for microcontroller boards

Arduino has emerged as one of the prime success stories of the open-hardware movement. In recent years, the company has shifted its focus toward Internet of Things (IoT) applications. As part of this transformation, it has completely redesigned its open-source integrated development environment (IDE), adding a more professional feature set for its hobbyist target audience. If you have experimented with Arduino in the past, but have lost track of its progress, now might be a good time to give it another try.

The Arduino project originated in 2005 at the Interaction Design Institute Ivrea in Italy as an educational tool for teaching students how to create and program interactive devices with sensors and actuators. Gradually, the project outgrew its academic origins and became the go-to platform for hobbyists interested in programming microcontroller boards. After a trademark dispute between the project's founders that lingered for years, the Arduino ecosystem now seems to have found a new life.

What we commonly refer to as Arduino is actually a combination of several components. First, there's the Arduino hardware, which consists of microcontroller boards with open-hardware designs. Then, there's the open-source software: an IDE that supports C++ programming using some libraries provided with the tool, coupled with an Arduino "core" that supports a specific board family. In addition, users can extend the basic functionality with third-party Arduino libraries. Finally, there's the Arduino company, which oversees the development and possesses the Arduino trademark. This article takes a look at the latest developments in each of these areas.

Open hardware

Over the years, the Arduino company has released more than a hundred hardware products, including microcontroller boards, shields (expansion boards), kits, and accessories. The hardware schematics are distributed under the Creative Commons Attribution Share-Alike 2.5 license and published on each product's web page. This means that anyone is allowed to copy and further develop the hardware design in their own project. However, they can't put the Arduino brand name and logo on their products. As a result, several Arduino-compatible products are available on the market using names ending in -duino, like Seeed Studio's Seeeduino product line of microcontroller boards.

The original Arduino boards, such as the Arduino UNO, were based on Atmel's 8-bit AVR microcontroller family. Later, more powerful, 32-bit, Atmel SAM-based boards were introduced. An example is the Arduino Due, which has gained popularity in robotics projects. Initially, Arduino boards had no network connectivity. However, due to increasing competition from Espressif's ESP8266 and ESP32 WiFi chips, Arduino started incorporating its own boards equipped with radio modules. The MKR family is based on a Cortex-M0 32-bit SAMD21 processor and comes with WiFi, Bluetooth, LoRa, Sigfox, or NB-IoT communication options. The most recent addition is the Arduino Nano ESP32, which incorporates Espressif's ESP32-S3 WiFi and Bluetooth chip.

Arduino IDE

The Arduino IDE has played a significant role in Arduino's success. While professional developers may dismiss the official development environment for its limited capabilities, it's perfectly suited for hobbyist developers. Because it doesn't come with too many bells and whistles, it's quite easy to learn.

However, the Arduino IDE has started showing its age. It was a cross-platform application written in the Java programming language, with a somewhat inflexible user interface, and it appeared that little was done to improve it. In 2019, the Arduino developers started to work on a new IDE, which was released as Arduino IDE 2.0 in September 2022. The previous version, the Arduino IDE 1.8 series, is now called the "Legacy IDE".

Arduino IDE 2.0 represents a complete overhaul of the development environment. The application's frontend is built on the Eclipse Theia framework, written in TypeScript, and uses Electron, which is a technology for building desktop applications using JavaScript, HTML, and CSS. The result is a modern, fully-featured, AGPL-3.0-licensed development environment with a flexible, responsive user interface.

For example, the new IDE allows developers to view both the serial monitor and the serial plotter data-tracking tool at the same time, while the previous version forced a choice between viewing text or a graph. The editor now includes autocompletion for variables and functions, as well as navigation shortcuts to jump to the place where a variable or function is defined. The new IDE also incorporates a debugger for stepping through a program's execution, at least on SAMD boards. While these may seem like basic features for a development environment, they were absent in the legacy IDE.

Subsequently, Arduino IDE 2.1 introduced the possibility of backing up Arduino projects to the cloud. And, more recently, Arduino IDE 2.2 added support for IDE extensions. For example, users can now install the ESP Exception Decoder, which provides human-readable stack traces and backtraces for ESP8266 and ESP32 chips within the Arduino IDE.

Anatomy of an Arduino sketch

In the Arduino ecosystem, programs are referred to as "sketches", each saved as a file with the .ino extension. An Arduino sketch always has two functions: setup() and loop(). Here is a typical example that blinks the built-in LED of a board:

    void setup() {
      pinMode(LED_BUILTIN, OUTPUT); 
    }

    void loop() {
      digitalWrite(LED_BUILTIN, HIGH);
      delay(1000);
      digitalWrite(LED_BUILTIN, LOW);
      delay(1000);
    }

The setup() function initializes variables, sets pin modes, and performs any other necessary code that should run once after the board powers up or resets. It is the initial entry point; in this example, the LED_BUILTIN pin is configured as a digital output.

Once the setup() function completes, the loop() function is executed repeatedly until the board is powered off or reset. This is comparable to the contents of an infinite while(1) loop that is often used in the main() function of a program running on a microcontroller. In this example, the pin of the built-in LED is set to HIGH to turn on the LED, followed by a delay of 1000 milliseconds. The pin is then set to LOW to turn off the LED and then it delays again. Since the loop() function is called over and over again, this simple program causes the LED to blink.

Arduino code is written in C++, but with some library functions, such as pinMode(), digitalWrite(), and delay() in the above example. Most boards also define the LED_BUILTIN macro as the pin number to which an on-board LED is connected.

The predefined functions offered by Arduino are not particularly innovative. However, Arduino provides an abstraction layer this way that ensures the above code works on all supported microcontroller boards, regardless of the underlying architecture. And that's the big advantage of using Arduino code. Once you've learned Arduino on an Arduino UNO, you can reuse much of your knowledge on an ESP32 development board or a Raspberry Pi Pico. In contrast, if you were to use a board's native SDK or I/O registers, your LED blinking code would differ significantly among different boards, as can be observed in example C code for the Raspberry Pi Pico and for an Atmel AVR.

Under the hood, Arduino's abstraction layer is implemented by libraries that provide the same software API for different boards. Such an implementation for a specific group of chips is called an Arduino core. For example, there's a core for the traditional AVR-based Arduino boards and for the SAMD21-based Arduino boards. These cores can be installed from the Arduino IDE. Furthermore, third-party cores allow Arduino code to run on boards that are not developed by the Arduino company. Espressif, for example, maintains Arduino cores for its ESP32 and ESP8266 chips. Looking at the main.cpp file in such a core provides insight into what occurs behind the scenes of the main() function that calls the setup() and loop() functions of an Arduino sketch. Many Arduino cores are LGPL-2.1-licensed, although the AVR core has some missing license information.

From the command line

Another recent development is the introduction of Arduino CLI, a command-line interface for various Arduino-related tasks. Although its API is officially considered unstable until a 1.0 release, it's already at the heart of the Arduino IDE. The command-line interface is used by the IDE as a backend for detecting boards, building sketches, uploading firmware to the boards, installing cores and libraries, and more.

Arduino CLI is developed in the Go programming language and is released under the GPLv3 license. Anyone can use the arduino-cli command from the shell to perform all of the non-editing tasks available in the IDE.

An important advantage of this command-line approach is that this makes it possible to automatically check whether an Arduino sketch builds correctly. I used the Arduino CLI this way in the GitHub Actions workflow in the repository of my book about developing Bluetooth Low Energy applications. With this setup, every time I update an Arduino sketch in the repository, arduino-cli automatically installs the necessary core and libraries and builds the sketch.

Myriads of applications

There are thousands of libraries that can be used to extend the Arduino environment. A tutorial about writing an Arduino library is available; a library can be as simple as a header and code file implementing a C++ class. There's also a style guide to ensure usability of your library for beginners. For example, the style guide discourages the use of pointers in an API, since they can be daunting for novice users. To make a library installable from the Library Manager in the Arduino IDE, the repository needs some metadata that follows the library specification.

As a result of this beginner-friendliness, Arduino's abstraction layer, the vast amount of libraries and the extensive documentation, the Arduino ecosystem has a diverse set of projects. From robotics, home-automation systems, and wireless sensor receivers to model railway command stations and CNC routers, Arduino offers a lot of versatility.

The Arduino company

When evaluating the sustainability of any open-source ecosystem, it's crucial to consider its development model. In the case of Arduino, the company seems to have found a way to fund the open-source development of its tools and hardware by offering additional services.

The sales of official Arduino products in the Arduino Store and through authorized distributors have historically contributed to the project's development. Due to the open-source hardware designs, Arduino clones can be found at lower prices, as any manufacturer can produce them (without using the Arduino name and logo). However, purchasing an official Arduino board ensures support for further development of the Arduino ecosystem.

Over time, the company broadened its focus. Although Arduino initially targeted students, for a long time it was really focused on "makers" working at the intersection of electronics, design, and programming. The Education sector steadily gained focus, though, resulting in the creation of hardware kits and educational content for middle schools, high schools, and universities. Moreover, the company introduced a Pro line, offering high-performance microcontroller boards for industrial control, robotics, and on-device artificial intelligence. Arduino also offers a dual licensing program to include its hardware and software in commercial products. Additionally, it offers the Arduino Cloud, a web-based platform with a web editor and dashboard.

There are a few problems in and around the Arduino ecosystem, however. Because of its use of an abstraction layer, the resulting firmware running on a board isn't as efficient as possible. While this isn't a problem for many hobbyist projects, this could lead to problems for applications that need the highest performance or precise timings for communication with other chips. Another issue is Arduino's focus on non-professional developers and non-experts in electronics. As a result, a lot of Arduino sketches, libraries, and suggestions in forums and elsewhere online vary greatly in their quality. In addition, the company's increasing focus on its cloud services is something to keep an eye on; the revenue may be needed to keep developing the Arduino ecosystem, which is fine as long as using those services remains voluntary.

Conclusion

Arduino has undoubtedly found its sweet spot in the space of embedded development for makers and in education. It has lower hardware requirements and stays "closer to the hardware" than MicroPython, but some smart API decisions (like discouraging pointers) make its use of C++ quite user-friendly. It doesn't offer a full-fledged real-time operating system such as Zephyr, yet it manages to offer a basic hardware abstraction layer for various architectures. As for the new Arduino IDE 2 series, it represents a significant improvement over the legacy IDE, without descending into the complexity of Visual Studio Code with PlatformIO, which is a commonly touted combination for embedded development. Whether you're just starting out with electronics or are ready to tap into a large library ecosystem to build and program some specialized hardware, Arduino has much to offer.

Comments (6 posted)