Welcome to the LWN.net Weekly Edition for October 15, 2020

• Python and the infinite: does the Python language need a firmer concept of infinity?
• Further analysis of PyPI typosquatting: how malware lurks behind misspelled package names.
• The ABI status of filesystem formats: does rejecting an invalid filesystem image that previously worked constitute a regression?
• NAPI polling in kernel threads: a performance improvement for high-traffic networking.
• Some 5.9 kernel development statistics: where the code for the 5.9 kernel release came from.
• A PHP syntax for discardable assignments: what's the best way to tell PHP that a value will not be used?

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Python and the infinite

A recent proposal on the python-ideas mailing list would add a new way to represent floating-point infinity in the language. Cade Brown suggested the change; he cited a few different reasons for it, including fixing an inconsistency in the way the string representation of infinity is handled in the language. The discussion that followed branched in a few directions, including adding a constant for "not a number" (NaN) and a more general discussion of the inconsistent way that Python handles expressions that evaluate to infinity.

In general, Python handles floating-point numbers, including concepts like infinity, following the standards laid out by IEEE 754. Positive and negative infinity are represented by two specific floating-point values in most architectures. Currently, representing a floating-point infinite value in Python can be done using a couple of different mechanisms. There is the float() function, which can be passed the string "inf" to produce infinity, and there is the inf constant in the math library, which is equivalent to float('inf'). Brown provided several reasons why he believed a new, identical, and built-in constant was necessary. One of his reasons was that he felt that infinity is a "fundamental constant" that should be accessible from Python without having to call a function or require a library import.

To highlight the issue, Brown provided an example using the repr() and eval() functions. The repr() function converts a data structure to a printable string that in many cases can be evaluated back into the original data structure using eval(). Brown noted that, unlike other floating-point values, using eval() on the printable representation of inf results in an exception unless inf has been imported from math:

    >>> eval(repr(float('inf')))
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "<string>", line 1, in <module>
    NameError: name 'inf' is not defined

Brown's post created a mega-thread around how floating-point infinity values should be handled in Python. To some, it wasn't clear why importing inf was a bad solution to Brown's problem. Additionally, Greg Ewing noted that using the eval() function in the way suggested by Brown can be "a serious security problem in some contexts." The eval() function evaluates a string as Python code, which is dangerous if the string contains user-supplied data. Ewing suggested that the safer function to use would be literal_eval() from the ast library, which only evaluates literal values. Using that function would only work, however, if inf and nan were built-in constants.

Christopher Barker agreed, but noted another issue: "there is no way to use literal_eval that gets you an inf (or NaN value). Which is a real, though maybe not important, use case." Later in the thread, Barker further refocused the conversation on the fundamental point that Brown's post was leading to:

Barker did not think that it was a "critically important" issue, but he did feel that extending the concept of a valid literal for the float type to include inf and nan was useful. He also claimed that it shouldn't require a new keyword. Barker admitted that if a new keyword were the only way to implement Brown's idea, it would be "dead in the water." Ewing suggested a possible implementation using a "special bytecode that looks it up as a name, and if that doesn't work, returns float('inf')," but noted that it was more straightforward and functionally the same to move math.inf and math.nan to __builtins__.

The problem with adding these constants to __builtins__, as Ewing noted, was that Guido van Rossum was not in favor of that idea. To that point, Barker responded by stating that, while Van Rossum has a "very well respected opinion", he is no longer the Benevolent Dictator for Life who could make this decision unilaterally.

Barker clarified that his goal was "to do that little bit more to make Python better support the float special values out of the box." He noted that, while PEP 754 ("IEEE 754 Floating Point Special Values") was rejected due to inactivity, almost all of the suggestions it made had ultimately been implemented, except for a version of the built-in literals for infinity and NaN now being proposed.

Van Rossum responded by expressing his willingness to sponsor a PEP for the Python steering council to consider. While he did not personally agree with the idea, he committed to remaining neutral so that Barker and Brown could receive "a fair hearing" with that body. With Van Rossum's sponsorship support, the duo put together a draft PEP that is now available on GitHub. Reflecting the thread, the PEP proposes "the introduction of built-in variables for floating point 'Infinity' (inf) and 'Not a Number' (nan)." Also included in the draft is an example summarizing the impact of the proposed change:

>>> inf
inf
>>> inf > 1e9
True
>>> eval('inf')
inf
>>> assert inf == eval('inf')
>>> inf == eval(repr(inf))
True
>>> import ast
>>> inf == ast.literal_eval('inf')
True

The discussion that evolved from Brown's post did more than lead to a new PEP; it also sparked an interesting conversation around how Python handles expressions that evaluate to mathematical infinity. Paul Bryan provided a relevant example of some quirks of the IEEE specification, specifically how subtracting infinity from itself results in NaN, but comparing the NaN from that operation doesn't equal math.nan:

>>> math.inf - math.inf
nan
>>> (math.inf - math.inf) == math.nan
False

Steven D'Aprano explained this strange behavior. According to IEEE 754, NaN values are, by definition, never equal to each other. To be otherwise, D'Aprano explained, would result in nonsensical expressions like sqrt(-2) == sqrt(-3) evaluating to True.

The conversation prompted a question from Stephen J. Turnbull, who asked if "base" Python could "create an infinity." Van Rossum had suggested earlier in the thread that you could produce inf from the value 1e1000 "in a pinch". However, Turnbull noted that a value like 1e1000, which cannot be represented in IEEE floating point, is technically an overflow rather than an infinity. While IEEE 754 permits mapping an overflow to infinity, that is not the same thing as the abstract mathematical concept of infinity. In another message, he expanded on this idea:

Turnbull explained that arithmetic involving infinite values only makes sense if it is carefully analyzed in the context of a specific calculation. The problem being that Python is inconsistent on the matter: some expressions should evaluate as the IEEE infinity but do not, some currently evaluate to inf when they are really overflows, and still others raise exceptions. As Turnbull said:

Ben Rudiak-Gould provided some examples of Python's inconsistencies when dealing with various mathematical expressions that represent an infinity. Each one should ideally evaluate to inf, but most raised exceptions like ValueError or OverflowError instead; "I get the impression that little planning has gone into this", he added.

Brown agreed that "a look should be taken at the structure of math-based errors and exceptions," adding that "the exceptions make little sense." Since the types of changes required to make Python consistent in this regard would represent a significant backward-compatibility break, it seems unlikely that they will be addressed soon.

As for the PEP, the next step is for Brown and Barker to submit a pull request to the steering council for acceptance, be assigned a PEP number, and bring it up again for further discussion. Before the steering council or its delegate will be able to accept or reject it, though, the PEP will need to be discussed on the python-dev mailing list. Time will tell if Brown's wish to move inf and nan into __builtins__ is accepted, and what, if anything, comes of the inconsistent way that Python handles concepts like infinity throughout the language.

Comments (18 posted)

Further analysis of PyPI typosquatting

We have looked at the problem of confusingly named packages in repositories such as the Python Package Index (PyPI) before. In general, malicious actors create these packages with names that can be mistaken for those of legitimate packages in the repository in a form of "typosquatting". Since our 2016 article, the problem has not gone away—no surprise—but there has been some recent analysis of it, as well as some efforts to combat it.

On the IQT blog, John Speed Meyers and Bentz Tozer recently posted some analysis they had done to quantify PyPI typosquatting attacks and to categorize them. They started by looking at the examples of actual attacks against PyPI users from 2017 to 2020; they found 40 separate instances over that time span. The criteria used were that the package had a name similar to another in PyPI, contained malware, and was identified and removed from the repository.

They identified two types of package typosquatting: misspelling and confusion. The first type relies on package names that are slightly misspelled, djanga instead of django or urlib3 instead of urllib3. The confusion attacks rely upon changing the order of the "words" in the name (e.g. nmap-python rather than python-nmap), removing or changing separators (e.g. easyinstall vs. easy_install), or otherwise changing the elements of the name (e.g. crypt/crypto, python-sqlite/pysqlite). Of the 40 attacks identified, 18 were of the misspelling variety, while 24 were confusing—two were both, which accounts for the overlap.

The blog post noted that William Bengston had done some research on one type of confusion attack in particular: separator changes. In July 2018, Bengston registered around 1,100 package names on PyPI by eliminating any separators (i.e. - or _) in the names the top 10,000 packages on PyPI. The packages registered would simply cause an error when they were installed; that error would redirect the user to the correct package name.

The IQT researchers found that separator attacks made up a small portion of the typosquatting incidents they looked at, though. So the number of potential systems that have fallen prey to the typosquatting problem is likely far higher. "Separator attacks account for only three of the 26 confusion attacks in this dataset though, suggesting that Bengston’s already frightening estimate of PyPI user susceptibility to typosquatting is a lower bound of overall user susceptibility to typosquatting attacks."

As might be guessed, typosquatters concentrated their efforts on the most popular PyPI packages. The IQT researchers found that 28% of their instances were typosquatting the top 50 most popular PyPI packages and more than half of the attacks (63%) were against the top 500.

Finding typosquatting attacks, or preventing those kinds of packages from being created in the first place, is obviously worth doing. The misspelling attacks are relatively easy to detect using the Levenshtein distance between two names. That distance measures the number of one-character changes that are needed to turn one string into another; the misspellings that the researchers found have a Levenshtein distance of one (15 attacks) or two (3 attacks). While there may be perfectly valid reasons for a package to have a name that is only slightly different than an existing package, it could be used as a reason for administrator scrutiny. The confusion category, on the other hand, generally had distances that were three or more (17 of 24), making them more difficult to (automatically) detect.

Python has made some efforts to help reduce the typosquatting problem. In 2017, code was added to block new PyPI packages with names that are the same as those of standard library modules. Existing PyPI packages that conflict are not being removed—some are backports of newer functionality to older versions of the language, for example—but they are being audited to determine their validity. Some malware checking has also been added to Warehouse, which is the web application behind PyPI.

The researchers also noted several different papers about ways to detect and stop malware from being distributed from package repositories such as PyPI. Other language repositories, npm for JavaScript and RubyGems for Ruby, are also considered in these papers. A team largely from the University of Kansas looked specifically at typosquatting defenses [PDF] for npm and PyPI, while a Georgia Tech team "built a sophisticated anti-malware analysis pipeline that repositories could employ to find malicious software, including typosquatters, hiding in repositories" (paper [PDF]). The amusingly titled "Backstabber’s Knife Collection: A Review of Open Source Software Supply Chain Attacks" [PDF] analyzes nearly 200 malicious packages from npm, RubyGems, and PyPI to try to extract information that can be used to detect new malware based on the characteristics of the existing malicious packages.

There is, of course, a more draconian solution to the problem: connecting packages with the real-life identities of their maintainers. It is the semi-anonymous nature of the repositories that makes these kinds of attacks easy to perform with little risk of personal repercussions for perpetrating them. There are lots of advantages to the free-for-all nature of repositories like PyPI (as well as GitHub and friends), but there are some downsides to it as well. On the other hand, of course, it is hard to imagine that attackers would not find ways around some ID requirement—or some way to pin their attack on an innocent bystander. Finding ways to thwart these attacks without resorting to that kind of policing is important.

Better vetting for packages is another potential solution, but it is not really practical. The number of changes that goes into these repositories is so enormous that it rapidly outpaces the limited bandwidth of administrators. Even large commercial companies are unable to handle this problem—the various app stores have been known to provide malware—so projects like Python cannot even begin to keep up. It would seem that automated efforts are slowly getting somewhat better, but it would be unsurprising to see more instances of malicious typosquatting in PyPI and elsewhere in coming years.

Comments (39 posted)

The ABI status of filesystem formats

Users reporting kernel regressions will receive a varying amount of sympathy, depending on where the regression is. For normal user-space programs using the system-call API, that sympathy is nearly absolute, and changes that break things will need to be redone. This view of the ABI also extends to the virtual filesystems, such as /proc and sysfs, exported by the kernel. Changes that break things are deemed a little more tolerable when they apply to low-level administrative tools; if there is only one program that is known to use an interface, and that program has been updated, the change may be allowed. On the other hand, nobody will be concerned about changes that break out-of-tree kernel modules; the interface they used is considered to be internal to the kernel and not subject to any stability guarantee.

But those are not the only places where user space interfaces with the kernel. Consider, for example, this regression report from Josh Triplett. It seems that an ext4 filesystem bug fix merged for 5.9-rc2 breaks the mounting of some ext4 filesystems that he works with.

These filesystems are created for some unspecified internal purpose with an unreleased internal tool. They are meant to be read-only, so there will never be any need (or ability) to create new files on them. As a space-saving measure, this tool overlays all of the block and inode bitmaps onto a single block, set to all ones, indicating that all blocks and inodes are already allocated. To indicate that this has been done, this tool marks the filesystem with a special flag (EXT4_FEATURE_RO_COMPAT_SHARED_BLOCKS). This flag is defined by the ext4 tools, but is not used by the kernel in any way. It is placed in the set of read-only compatibility flags, though, meaning that a kernel that sees it in a filesystem will know that said filesystem can be safely mounted, but only in read-only mode.

Until 5.9-rc2, mainline kernels were happy to mount these filesystems. The commit highlighted by Triplett changed that situation by adding some checks to the mount-time ext4 verifier that ensures that the filesystem image is valid. As a result of these new checks, the overlapping bitmaps are detected, the kernel complains, and any attempt to mount the filesystem fails. Something that used to work no longer does — the definition of a regression. Triplett included in his report a small patch that disables the validity check when the EXT4_FEATURE_RO_COMPAT_SHARED_BLOCKS flag is present, rendering his filesystems mountable again.

That change is likely to be merged, but it has not brought great joy to the filesystem developers, who see it as a sign of things going wrong. XFS maintainer Darrick Wong argued that "unofficial" filesystem variants should not be supported:

Triplett responded that filesystem images should fall within the realm of the kernel ABI:

Ext4 maintainer Ted Ts'o said that he was not opposed to Triplett's patch, but suggested that further patches to support this tool might receive a chillier reception. He then told the story of the make_ext4fs tool, an independently written ext4 filesystem creator used for years by Android. It created a long list of compatibility and corruption problems, he said, that took years to iron out. Third-party filesystem tools are prone to such problems, he said:

Filesystem developers have a certain natural aversion to "mischief", so it is unsurprising that Ts'o would prefer that these outside tools simply not exist. At a minimum, he suggested, future policy should say that filesystem-image regressions would only need to be addressed for images that were created and managed by the designated official tools. He requested that Triplett find a way to get rid of his custom tool.

Triplett disagreed with that policy suggestion, saying that a more reasonable approach would be that any filesystem images that pass the e2fsck checker should be supported. Not all tools that work with the ext4 format should have to live in the e2fsprogs repository; to say otherwise, he added, would be tantamount to saying that the FreeBSD kernel, which has an ext4 filesystem driver, should live in e2fsprogs too. The tool in question here, which Triplett finally described late in the conversation, appears equally unsuited to inclusion in the e2fsprogs repository.

Triplett concluded that message with a restatement of his request:

Ts'o's response made it clear that he is uninclined to grant that wish; that would, he said, make it impossible to fix security-related problems related to invalid filesystem images, of which there are many. Many of these invalid images — often generated by fuzzing tools or attackers — pass e2fsck until the problems are found and fixed. Grandfathering in any image that passes e2fsck would thus, he said, require invalid filesystems to be supported forever. He concluded with some suggestions for other ways to solve Triplett's problem and requested that Triplett work more closely with the ext4 developers in the future.

As of this writing, that is where the discussion has stopped. Ts'o's willingness to apply the fix for the immediate problem means that there is no pressing need to resolve the larger issue of regressions involving filesystems; that, in turn, means that the issue is likely to come up again at some point. The kernel ABI is a large and amorphous thing, and many of the boundaries are fuzzy at best. Filesystems are one area where that boundary has not yet been fully explored; somebody is likely to inadvertently end up on the wrong side of it sooner or later.

Comments (34 posted)

NAPI polling in kernel threads

Once a packet arrives on a network interface, the kernel must usually perform a fair amount of protocol-processing work before the data in that packet can be delivered to the user-space application that is waiting for it. Once upon a time, the network interface would interrupt the CPU when a packet arrived; the kernel would acknowledge the interrupt, then trigger a software interrupt to perform this processing work. The problem with this approach is that, on busy systems, thousands of packets can arrive every second; handling the corresponding thousands of hardware interrupts can run the system into the ground.

The solution to this problem was merged in 2003 in the form of a mechanism that was called, at the time, "new API" or "NAPI". Drivers that support NAPI can disable the packet-reception interrupt most of the time and rely on the network stack to poll for new packets at a frequent interval. Polling may seem inefficient, but on busy systems there will always be new packets by the time the kernel polls for them; the driver can then process all of the waiting packets at once. In this way, one poll can replace dozens of hardware interrupts.

NAPI has evolved considerably since 2003, but one aspect remains the same: it still runs in software-interrupt mode. These interrupts, once queued by the kernel, will be processed at either the next return from a hardware interrupt or the next return from kernel to user mode. They thus run in an essentially random context, stealing time from whatever unrelated process happens to be running at the time. Software interrupts are hard for system administrators to manage and can create surprising latencies if they run for a long time. For this reason, kernel developers have wanted to reduce or eliminate their use for years; they are an old mechanism that is deeply wired into core parts of the kernel, though, and are hard to get rid of.

Wang's patch set (which contains work from Paolo Abeni, Felix Fietkau, and Jakub Kicinski) doesn't eliminate software interrupts, but it is a possible step in that direction. With these patches applied, the kernel can optionally (under administrator control) create a separate kernel thread for each NAPI-enabled network interface. After that, NAPI polling will be done in the context of that thread, rather than in a software interrupt.

The amount of work that needs to be done is essentially unchanged with this patch set, but the change in the way that work is done is significant. Once NAPI polling moves to its own kernel thread, it becomes much more visible and subject to administrator control. A kernel thread can have its priority changed, and it can be bound to a specific set of CPUs; that allows the administrator to adjust how that work is done in relation to the system's user-space workload. Meanwhile, the CPU scheduler will have a better understanding of how much CPU time NAPI polling requires and can avoid overloading the CPUs where it is running. Time spent handling software interrupts, instead, is nearly invisible to the scheduler.

There aren't a lot of benchmark results posted with the patch set; those that are available indicate a possible slight increase in overhead when the threaded mode is used. Users who process packets at high rates tend to fret over every nanosecond, but even they might find little to quibble about if these results hold. Meanwhile, those users should also see more deterministic scheduling for their user-space code, which is also important.

The networking developers seem to be generally in favor of this work; Eric Dumazet indicated a desire to merge it quickly. This feeling is not unanimous, though; Kicinski, in particular, dislikes the kernel-thread implementation. He believes that better performance can be had by using the kernel's workqueue mechanism for the polling rather than threads. Dumazet answered that workqueues would not perform well on "server class platforms" and indicated a lack of desire to wait to see a new workqueue-based implementation at some point in the future.

So it appears that this work will be merged soon; it's late for 5.10, so landing in the 5.11 kernel seems likely. It's worth noting that the threaded mode will remain off by default. Making the best use of it will almost certainly require system tuning to ensure that the NAPI threads are able to run without interfering with the workload; for now, administrators who are unwilling or unable to do that tuning are probably well advised to stick with the default, software-interrupt mode. Software interrupts themselves are still not going away anytime soon, but this work may help in the long-term project of moving away from them.

Comments (15 posted)

Some 5.9 kernel development statistics

Of the 1,914 developers contributing to 5.9, 306 made their first contribution for this release. This is the largest number of new contributors we have seen since 4.12 (which had 334 first-time contributors) was released in 2017 and, indeed, the second-highest number ever seen. All together, the 5.9 contributors added just over 730,000 lines of code and removed nearly 262,000 for a net growth of 468,000 lines of code. The busiest developers this time around were:

Most active 5.9 developers

By changesets Lee Jones 520 3.5% Christoph Hellwig 292 2.0% Randy Dunlap 261 1.8% Alexander A. Klimov 187 1.3% Ben Skeggs 137 0.9% Chris Wilson 135 0.9% Laurent Pinchart 135 0.9% Evan Quan 113 0.8% Pierre-Louis Bossart 113 0.8% Gustavo A. R. Silva 110 0.7% Likun Gao 109 0.7% Thomas Zimmermann 105 0.7% Thierry Reding 102 0.7% Colin Ian King 97 0.7% Pavel Begunkov 96 0.6% Kuninori Morimoto 95 0.6% Andy Shevchenko 91 0.6% Krzysztof Kozlowski 88 0.6% Kees Cook 83 0.6% Edward Cree 80 0.5%

By changed lines Jerry (Fangzhi) Zuo 92950 11.2% Likun Gao 77897 9.4% Bhawanpreet Lakha 28787 3.5% Mike Rapoport 18531 2.2% Edward Cree 13146 1.6% Ben Skeggs 10761 1.3% Christoph Hellwig 9286 1.1% Leo Liu 9056 1.1% Tzu-En Huang 8521 1.0% Hans Verkuil 8487 1.0% Evan Quan 8428 1.0% Laurent Pinchart 6438 0.8% Alexander Lobakin 6129 0.7% Rob Clark 5992 0.7% Chris Wilson 5934 0.7% Hyun Kwon 5839 0.7% Dmitry Osipenko 5728 0.7% Jesse Brandeburg 5335 0.6% Leon Romanovsky 5134 0.6% Jakub Kicinski 4774 0.6%

The largest number of changesets going into 5.9 came from Lee Jones, who contributed many cleanups across the device-driver subsystem. Christoph Hellwig made substantive changes across a number of kernel subsystems; this work includes the removal of set_fs() among many other things. Randy Dunlap removed duplicated words words from files all over the kernel tree, Alexander Klimov converted vast numbers of "HTTP" links to "HTTPS", and Ben Skeggs contributed a lot of improvements to the Nouveau graphics driver.

The "lines contributed" column starts with Jerry (Fangzhi) Zuo, Likun Guo, and Bhawanpreet Lakha, all of whom added more code to the massive amdgpu graphics driver. Zuo only contributed two patches, but one of them was large (and consisted only of definitions of preprocessor symbols). Mike Rapoport removed the unloved unicore32 architecture and Edward Cree did a lot of work on the Solarflare SFC network driver.

Overall, 207 employers supported work on the 5.9 kernel, a number that is consistent with previous releases. The most active employers were:

Most active 5.9 employers

By changesets (None) 1377 9.3% Intel 1336 9.0% Red Hat 1006 6.8% (Unknown) 895 6.0% AMD 848 5.7% Linaro 842 5.7% Google 662 4.5% SUSE 554 3.7% (Consultant) 504 3.4% IBM 478 3.2% Huawei Technologies 471 3.2% Facebook 385 2.6% Renesas Electronics 323 2.2% NXP Semiconductors 313 2.1% Mellanox 303 2.0% Oracle 245 1.6% NVIDIA 221 1.5% Arm 207 1.4% Code Aurora Forum 203 1.4% Texas Instruments 189 1.3%

By lines changed AMD 243874 29.4% Intel 56635 6.8% Red Hat 39347 4.8% IBM 35658 4.3% (None) 30232 3.7% Google 29715 3.6% (Unknown) 29421 3.6% Mellanox 24149 2.9% Facebook 22410 2.7% Linaro 19271 2.3% (Consultant) 18151 2.2% NVIDIA 17985 2.2% Renesas Electronics 14974 1.8% SUSE 14409 1.7% Texas Instruments 13508 1.6% Solarflare Communications 13146 1.6% Marvell 11284 1.4% NXP Semiconductors 10900 1.3% Code Aurora Forum 10817 1.3% Realtek 10260 1.2%

This is the first time in years that the largest number of changesets came from people who were working on their own time; it seems unlikely to be a change to the long-term trend toward smaller volunteer participation, but one never knows. Otherwise the employer numbers look about the same as they do in most months.

The busiest testers and bug reporters for 5.9 were:

Test and report credits in 5.9
Tested-by Andrew Bowers 71 7.5% Aaron Brown 38 4.0% Nicolas Saenz Julienne 28 2.9% Arnaldo Carvalho de Melo 28 2.9% Sedat Dilek 24 2.5% Stan Johnson 21 2.2% 周正 (Zhou Zheng) 18 1.9% John Donnelly 17 1.8% Dmitry Baryshkov 16 1.7% Alexei Starovoitov 16 1.7%	Reported-by kernel test robot 169 17.1% Syzbot 91 9.2% Hulk Robot 67 6.8% Dan Carpenter 23 2.3% Stephen Rothwell 17 1.7% Naresh Kamboju 16 1.6% Randy Dunlap 16 1.6% Lars-Peter Clausen 13 1.3% Qian Cai 12 1.2% Colin Ian King 8 0.8%

Automated testing systems continue to be the most prolific source of bug reports; they were responsible for just over one-third of the total in the 5.9 cycle.

The developers with the most review credits this time around were:

Review credits in 5.9
Rob Herring	195	3.0%
Alex Deucher	162	2.5%
David Sterba	131	2.0%
Lyude Paul	130	2.0%
Hawking Zhang	121	1.9%
Christoph Hellwig	107	1.7%
Florian Fainelli	103	1.6%
Andy Shevchenko	95	1.5%
Jiri Pirko	82	1.3%
Darrick J. Wong	80	1.2%

A total of 5,235 commits for 5.9 contained Reviewed-by tags; that is 35% of the total, which is up slightly from 5.8.

All told, 5.9 looks like another fairly routine development cycle that is a little busier than the average. Once again, it seems that the ongoing global pandemic has not hurt the pace of Linux kernel development; it might even possibly have encouraged the larger-than-normal number of changes from first-time and volunteer developers. The kernel community is clearly running at full speed as it heads into the 5.10 development cycle.

Comments (2 posted)

A PHP syntax for discardable assignments

Recently, John Bafford revived a years-long conversation on expanding the syntax of the PHP foreach statement to include iterating solely over keys. Bafford, who wrote a patch and request for comments (RFC) on the matter back in 2016, hopes to update his work and convince the community to adopt the abbreviated syntax in PHP 8.1. The community took Bafford's general idea and expanded it into other areas of the language.

The PHP foreach statement is designed to iterate over arrays and objects that implement the Traversable interface. PHP arrays are analogous to Python dictionaries. Two varieties of syntax are currently available to foreach: the first extracts the value of each item, while the second extracts both the key and the value:

    $traversable = [
        'A' => 1,
        'B' => 2
    ];

    foreach($traversable as $value) { /* ... */ }

    foreach($traversable as $key => $value) { /* ... */ }

The purpose of Bafford's proposal is to provide an efficient way for the foreach statement to extract only the keys while ignoring the values; he decided on using the void keyword to indicate that the value portion should be discarded:

    foreach($traversable as $key => void) { /* ... */ }

It's not the first time this idea has been suggested; Chris London had a similar proposal back in 2013. At the time, London's suggestion didn't gain much traction, and contributor Nikita Popov characterized the use case for the syntax as "not particularly common." Moreover, as Popov wrote, PHP already provides multiple different ways of addressing this need without adding syntax to the language. One solution is to provide a variable for the value in foreach statements and not use it when only the keys are needed:

    foreach($array as $key => $_) { /* ... */ }

Note that, in the above, $_ is just a variable to PHP without any special meaning; developers, by convention, sometimes use it to signify an "unused" value. Another alternative that was suggested by Popov is to use the array_keys() function to similar effect. Using this approach, a second, integer-indexed array is created from the original array that contains the keys to iterate over, eliminating the need for an unused variable:

    foreach(array_keys($array) as $key) { /* ... */ }

The arguments against the new syntax have not changed significantly from seven years ago; Popov effectively repeated them in response to Bafford's proposal. In his opinion, it added unnecessary complexity to the language, would make foreach "marginally slower," and there are multiple valid alternatives already available to address the problem the RFC looks to solve. Not everyone in the community felt that Popov's alternatives were ideal, however. Dik Takken noted that the first option creates an unused variable, adding that "code inspection currently complains about that, and for good reason, which is annoying." As for using array_keys(), Markus Fischer wrote that this solution isn't ideal, either; the array_keys() function needs to create a new array, which could be a performance issue.

Some felt that Bafford's choice to use void wasn't the best option. Olle Härstedt suggested that the underscore character might be a better alternative to void:

foreach($iterable as $key => _) { ... }

As justification, Härstedt noted that OCaml uses the underscore symbol when the value of an assignment can be discarded. Unfortunately, _() is also an alias to the gettext() function, making it a poor choice for this purpose. Several participants in the thread preferred the keyword null to void, although Bafford was opposed to that suggestion. He argued that, because null is a value in PHP whereas void is a type, void is the better choice. With the RFC still under discussion, what keyword might ultimately be used is undecided.

Disagreements on the specific keyword aside, Takken suggested that the scope of Bafford's proposal was too narrow, writing:

There are at least two areas where this syntax would be beneficial to the language. The first is captured in Bafford's RFC, and the second is when assigning variables from an array as shown in the example below:

    [, , $a] = ['val1', 'val2', 'val3'];

To many programmers, the code above might look more like a syntax error than a valid expression. It is, however, valid PHP code, which assigns "val3" to $a; the seemingly out-of-place commas in the example above indicate that $a should be assigned the third value in the source. Härstedt noted that "anything would be better" than the current comma-only syntax; generalizing Bafford's proposal to include this aspect of PHP certainly clarifies what is happening in such an assignment operation:

    [void, void, $a] = ['val1', 'val2', 'val3'];

Takken's suggestion to generalize the syntax was positively received by Härstedt and others, including PHP 8.0 release manager Sara Golemon, who described both the proposed changes to foreach and the suggested improvements to array-value assignments "great, expressive syntaxes."

Bafford believes his proposal shouldn't hurt performance. In fact, while he was clear that "any performance benefits are secondary" to the syntactical benefits, he did make a case arguing that his proposal would benefit the run-time performance of foreach:

While Bafford may be correct that there is a theoretical performance gain, he offered no benchmarks to demonstrate that and it is unlikely to be of much practical value. Longtime project member Stanislav Malyshev explained what he felt was faulty in Bafford's analysis. In Malyshev's opinion, eliminating an opcode to retrieve the value wouldn't move the performance needle in a meaningful way:

As things stand now, Bafford's RFC needs to be updated to capture the substance of the thread and address any outstanding issues raised in it. One of the key issues is whether Bafford will broaden his RFC to a more generalized syntax, or whether he will keep it limited to his original scope. The wider scope appears to have fairly broad support in the community, so it seems likely the RFC will be updated to reflect that. In an email linked above, Bafford summed up where things stand; he also let the community know he would be taking a break from computers "for a few weeks." Since this is all planned for PHP 8.1, there isn't any rush; PHP 8.0 isn't scheduled to be released until November 26.

Comments (10 posted)

BleedingTooth: critical kernel Bluetooth vulnerability

Comments (28 posted)

Security quotes of the week

Comments (9 posted)

Kernel release status

Some of the significant features in this release are: x86 FSGSBASE support, capacity awareness in the deadline scheduler, the close_range() system call, proactive compaction in the memory-management subsystem, the rationalization of kernel-thread priorities, and more. See the KernelNewbies 5.9 page for more details.

Stable updates: 5.8.15, 5.4.71, 4.19.151, 4.14.201, 4.9.239, and 4.4.239 were released on October 14.

Comments (none posted)

Wishing David Miller well

Comments (8 posted)

Quote of the week

Comments (3 posted)

Distribution quotes of the week

Comments (4 posted)

An open letter to Apache OpenOffice

Comments (63 posted)

Krita 4.4.0 released

Comments (none posted)

LLVM 11.0.0 released

Comments (4 posted)

Plasma 5.20 released

Comments (3 posted)

Plausible relicenses to AGPL

Comments (4 posted)

Development quotes of the week

Comments (3 posted)

The Open Invention Network's expanded Linux System Definition

Full Story (comments: none)

• DistroWatch Weekly (October 12)

• Lunar Linux Weekly News (October 9)

• openSUSE Tumbleweed Review of the Week (October 9)

• Tails Report (September)

• Ubuntu Weekly Newsletter (October 10)

• Emacs News (October 12)

• These Weeks in Firefox (October 13)

• What's cooking in git.git (October 9)

• LLVM Weekly (October 12)

• LXC/LXD/LXCFS Weekly Status (October 12)

• OCaml Weekly News (October 13)

• Perl Weekly (October 12)

• Python Weekly Newsletter (October 8)

• Weekly Rakudo News (October 12)

• Ruby Weekly News (October 8)

• This Week in Rust (October 7)

• Wikimedia Tech News (October 12)

• Document Foundation Board of Directors minutes (October 9)

• openSUSE board meeting minutes (September 29)

Call For Presentations - SQLite & Tcl 2020 (Virtual Event)

Full Story (comments: none)

CFP Deadlines: October 15, 2020 to December 14, 2020

If the CFP deadline for your event does not appear here, please tell us about it.

Events: October 15, 2020 to December 14, 2020

If your event does not appear here, please tell us about it.

Netdev 0x14: Videos posted

Full Story (comments: none)

Alert summary October 8, 2020 to October 14, 2020