Leading items

Welcome to the LWN.net Weekly Edition for August 31, 2023

This edition contains the following feature content:

• Mastering Emacs: reviewing a book on how to use the Emacs editor well.
• Python is (mostly) made of syntactic sugar: a conference exercise on how to pare Python down to a minimal core.
• A more dynamic software I/O TLB: a kernel subsystem once created for substandard hardware is finding a number of new uses and must evolve to match.
• Development statistics for the 6.5 kernel: where the contributions to the 6.5 kernel came from.
• The OpenSprinkler controller: an Internet-of-things device that tries to do it right.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Mastering Emacs

A series of rabbit holes, some of which led to unshaved yaks, recently landed me on a book called Mastering Emacs. Given that I have been using Emacs "professionally" for more than 16 years—and first looked into it a good ways into the previous century—I should probably be pretty well-versed in that editor-cum-operating-system. Sadly, for a variety of reasons, that is not really true, but the book and some concerted effort have been helping me down a path toward Emacs-ian enlightenment. Mastering Emacs may also help others who are struggling in the frothy sea that makes up Emacs documentation.

The backstory of how I got here is kind of goofy—some days rabbit holes look like so much fun ... and they definitely can be, but the lost "productivity" may be problematic. In any case, a Hacker News item on "Elixir for cynical curmudgeons" caught my eye a few weeks back since I certainly qualify. After reading that and poking at Elixir (and Erlang) documentation some, I remembered that I always wanted to understand Lisp macros better—or at all, in truth. That led me back to a project that I started (and promptly drifted away from) after a talk at linux.conf.au about the history of Lisp that I really enjoyed.

That project was, naturally, a Lisp interpreter. It is written in Python and is, at best, 5% complete—and likely not to go too much further. But I dusted it off, made it work for some more Lisp constructs, and suddenly realized that I did not know enough Lisp to even be setting off on such a journey. That should be a solvable problem, so I sought out some more Lisp wisdom and came across "A Road to Common Lisp". It seemed like it contained sensible advice, some of which I started to follow, but then ran into some annoyances using Lisp from the command line. That, inevitably, led to SLIME.

Of course, given that SLIME is also known as the "Superior Lisp Interaction Mode for Emacs", that leads to Emacs, which is an editor I use in anger on a daily basis. It is not, however, my editor of choice; I have been using vi (and its variants) since the early 1980s and it is the editor I reach for—except for work-related tasks. A day-one surprise when I started full-time at LWN in 2007 was that Emacs was closely intertwined with the site operations (article editing, in particular). That led to a crash course in just-enough-Emacs to get by, which has mostly served—for good or ill—since.

In truth, I am not a "power user" of any editor; I suspect that I could still function perfectly well in the Berkeley Unix version of vi that I learned on, lo these many years ago. I may well have only scratched the surface of its capabilities as I am not averse to brute-force solutions to editing problems. But, over the years, I have picked up a few more Emacs "tricks" and been impressed with all that it can do. It has always intrigued me that it can be transmogrified using Lisp, as well.

There are, evidently, add-ons for Vim that fill the same niche as SLIME, so I could have taken that path, but Emacs seemed like the right choice. I can be stubborn in weird ways, such as by ignoring Evil mode, which provides a vi-like experience for Emacs. What little I remembered from my first toe-dip into the Emacs waters—using the GNU Emacs manual with the yellow cover in the mid-1980s—stuck with me (mostly just Control-x Control-c, or C-x C-c, to exit), so I was determined to forge ahead on that path in my 2007 crash course and, of course, since.

The book

So, once I started messing with SLIME, I came to realize that I was lacking the knowledge of some major underpinnings of Emacs. I had encountered Mastering Emacs before, considered buying it, and got distracted before doing so. The Mastering Emacs web site is an excellent resource in its own right and one that I had visited along the way. Right around the time I started my SLIME investigation, a Hacker News post about the book showed up; when I saw that the book was on sale in the comments on the post, I took the plunge and bought it.

Other than "lost" time, I have not even remotely regretted the purchase. Mickey Petersen has done an excellent job with the book and it is well worth its full price ($49.99) in my opinion. I am still working my way all the way through it—and will undoubtedly refer back to it regularly. It comes in both PDF and EPUB formats; there is no "dead tree" version. Petersen detailed the process of writing the book (in Emacs, naturally) on his site; he hired a professional editor to help him, which was a nice touch. The articles on his site are well-written, though, so perhaps his editor did not have all that much work to do.

It is an opinionated book, and rightly so. A dry recounting of the options and possibilities within Emacs would be far less useful. Which is not to say that such a manual does not have its place—and one of those places is inside Emacs itself, of course. But Petersen is providing a sort of guided tour of Emacs, describing pieces of the editor in a way that makes sense for a narrative, rather than a reference. Along the way, he clearly indicates what his opinions are, why he has them, and how that changes his Emacs environment. There is no expectation of rigid adherence to his suggestions, however; some of them I have adopted, others not, and I have made a few customizations of my own. None of that interferes with the rest of the book in any way.

The organization may seem a bit strange at first, but it has grown on me. Beyond the introductory material, he starts with the philosophy of Emacs, which is an important underpinning. Understanding that "Emacs is a tinkerer's editor" is fundamental to working with it. He notes that much of that tinkering will require learning some Emacs Lisp (elisp), which intimidates and chases away some potential users, sadly.

The book has a nice justification for "why Lisp?" that resonated strongly with me. It makes sense that a language where there is no distinction between code and data would be the basis for an editor (and so much more) that is Emacs and its ecosystem. Obviously my interest in delving further into Lisp made it easy to buy into the justification; others may not find it so compelling.

Terms

The "Important Conventions" section provides something of an impedance-matching layer between Emacs terms and those that are used elsewhere. Terms like "window", "frame", "mark", "point", "buffer", "kill", "yank", and so on have specific meanings in Emacs; some of them are confusing because they conflict with current usage. In many cases, Emacs started using these terms before the other uses came about, but it was not popular—or visible—enough to "enforce" its prior art.

That section also describes the buffer interface, its modeline, and mini-buffer. Buffers are the basic unit of interaction in Emacs, which can be displayed in windows, that are contained in frames. These days, of course, a window has a much different meaning, which generally corresponds to an Emacs frame. Buffers can have one major and multiple minor modes as well. Wrapping one's head around all of these terms is important, but the book has not even started talking about how to install or start up Emacs at this point.

It gets there, of course, and suggests that readers run the latest Emacs available, which is (or was) Emacs 28; since then, Emacs 29.1 was released. Petersen has a lengthy article that annotates the Emacs NEWS file for the release with the features that he found interesting. That is presumably partly in preparation for an updated version of Mastering Emacs before too long. Updates to the book are free for those who have purchased it.

One might guess that next up might be editing text, but that part does not actually come until a little ways beyond the midpoint of the 300+ page book. Before you get there, there are lengthy descriptions of the key sequences used by Emacs, which is certainly an area that those who are new to the editor will need to grasp. One of the most strongly worded opinions that Petersen offers is to remap the Caps Lock key as Control; in fact, he suggests disabling the left-hand-side Control key entirely. So far, I have been unable to take that last step, as there are decades of muscle memory to overcome. Eventually, remapping that key to Meta (the other major Emacs modifier key) has some appeal, but may prove to be insanity.

The extensive customize interface (M-x customize) is described in some detail; it allows changing an unbelievably vast array of options. Those changes can be just for the session or can be saved in the startup file (normally ~/.emacs.d/init.el these days, apparently, though ~/.emacs still works) so that they become permanent. The customize interface is one of the few that I have encountered in Emacs with buttons and other controls that make using the mouse, rather than the keyboard, seem like the right way to go.

Searching the internet for "how to do X in Emacs" will quickly lead to lots of different places with suggestions and ideas of varying quality. They will often require evaluating elisp code, which is easy enough to do, but it is a little unclear to me how well that all meshes with a running Emacs session. Petersen quickly describes M-x eval-region (and eval-buffer), but recommends putting code in the startup file and restarting Emacs if something goes wrong, which sounds a little worrisome. Petersen's article on elisp evaluation covers the options more extensively than the book does.

So far, I have not really encountered any major, inexplicable weirdness from evaluating code, so maybe I am overthinking things. I certainly don't want to be restarting Emacs with any frequency; once buffers/windows/frames are set up the way I like them, it is irritating to have to redo all of that. Maybe there is some package out there to save and restore the state; in truth, I would be shocked if there wasn't. But that's something for another day.

Help, movement, and editing

Petersen covers the extensive help system in Emacs, which is something that he says he uses quite frequently. I can only concur that knowing how to find out what a key does, what the keybinding for a command is (if any), what a command does, and so much more is incredibly empowering. The Emacs manual is available, as is the info documentation for many parts of the GNU system, man pages, and so on. I intend to figure out the EPUB reader so I can refer back to the book while wrestling with Emacs in the future.

But, wait, there's more. The "theory of movement" is the next major section, which covers things like loading and saving files (or finding files and saving buffers in Emacs-speak), switching between buffers, listing all of the buffers, killing (closing) buffers, and so on. He strongly suggests switching the C-x C-b buffer listing command to use M-x ibuffer instead. He also thinks that the C-x b command for switching buffers could have a much better auto-complete function, so he recommends using IDO mode (or FIDO mode in Emacs 27+) instead.

There are, of course, a myriad of ways to move the point (cursor) in Emacs: by character, line, word, s-expression, page, paragraph, forward, backward, and probably upside-down too. The mouse can be used (at least in a GUI-window Emacs), as can the arrow keys. The latter has been my downfall, to a certain extent, though Petersen is characteristically sanguine about "cheating" that way; it is something I would like to train myself out of, but not to the point of disabling (or rebinding) those keys.

And, finally, we reach the editing chapter (skipping over plenty to get there). One of the things it stresses is that "killing" text is the main way to get rid of it from a buffer, but that killing is quite distinct from the concept of deleting text as it is in most editors. Except for the single-character delete commands (forward and backward), pretty much all of the other ways to remove text do so by killing it and moving it to the kill ring. As might be guessed, the kill ring stores these kills so that they can be yanked (pasted) elsewhere into the buffer—or into some other buffer entirely.

As with the other parts of the book, Petersen describes the real-world use of the features and tries to point the reader in the right direction to get the most out of them. For example, one could use the region-marking commands to select a region (between the mark and point, in Emacs terms) containing three words, then kill it, but it is much faster to simply do three M-d commands to kill by words; consecutive kills append to the same kill-buffer entry, so the effect is exactly the same. It is tidbits like that, which are scattered throughout the book, that make it so useful.

Remembering said tidbits, commands, key sequences, and so on is an entirely separate problem, of course. I have had an emacs-cheat file, lovingly maintained with vi until recently, since the crash-course days. It has grown by leaps and bounds over the last couple of weeks. I find myself referring to it frequently (from inside Emacs, of course), but there seems to be hope that I am actually committing some of what I have learned to muscle memory. Time will tell.

So far, I have read through two-thirds of the book, while skimming and cherry-picking through the rest. Perhaps my newfound enthusiasm for Emacs (Lisp, ...) is coloring my judgment but I have found the book to be well-nigh amazing. I have some minor quibbles in spots; for example, the whole-line-or-region package that he recommends can be installed easily enough, at least if you add the MELPA repository as he also suggested, but he never mentions that you need to enable the package in init.el (or wherever).

Meanwhile, it would seem that MELPA is where the majority of Emacs packages live today, so why isn't it added by default? That is not Petersen's fault, of course; I should probably take it up with my distribution or with the Emacs project itself. (Ah, as it turns out, Richard Stallman objects to MELPA, so I guess I should bug my distribution—or just ignore the problem and move along.)

Even in the parts of the book that I have read, I have skipped over topics in this review: the undo ring, package manager, searching, tabs, and more. Beware that reading the book can be something of a time sink, as can attempting to apply what has been learned. I find myself pausing more often than I used to, but also generally moving around more efficiently in Emacs; so far, it is hard to see a net win there, but the signs are hopeful. My emacs-cheat file (always visible in another frame or window) is organized in a way that matches how my brain (allegedly) works, so something personalized along those lines may make sense for others.

I did not set out to write an article when I stumbled down this rabbit-hole-strewn path, but, as I got into the book, it seemed like something readers might want to learn about. For now, my goal is to see if I can wean myself away from vi, mostly as a test; I certainly have nothing against that editor and will use it again. For example, I have not even tried Emacs in a terminal, or over SSH, which may necessitate a return to my editing roots, at least temporarily. In the meantime, I am enjoying my exploration of Emacs (and Lisp)—learning new things is always fun.

Comments (53 posted)

Python is (mostly) made of syntactic sugar

"Sugar" is, to a certain extent, in the eye of the beholder—at least when it comes to syntax. Programming languages are often made up of a (mostly) irreducible core, with lots of sugary constructs sprinkled on top—the syntactic sugar. No one wants to be forced to do without the extra syntax—at least not for their favorite pieces—but it is worth looking at how a language's constructs can be built from the core. That is just what Brett Cannon has been doing for Python, on his blog and in talks, including a talk at PyCon back in April (YouTube video).

Cannon began with a definition of "syntactic sugar" as "syntax that is not necessary to get the semantic results you want in the end". But, eliminating the sugar will not necessarily result in code that is as fast as it would be with those "optional" constructs. In fact, eliminating some of them will "abuse the way the language works" in various ways. The idea is that if a construct had not been added to the language by the core developers, the same result can be obtained, "but probably in a much messier, long-winded way".

He set some rules for himself in his investigation, including that any syntactic-sugar replacement needed to work in Python 3.8, which was "the newest hotness" at the time he started. He stuck to things that were in the language specification and avoided things that were CPython-specific. He also restricted his techniques to those that could be applied to a single Python file, as opposed to whole-program analysis; he wanted to be able to write a tool that would apply the de-sugaring to a file such that the output would be a drop-in replacement for the original.

He was willing to live with some oddities in highly dynamic parts of the runtime, such as in locals() and globals(). In fact, he said, that already happens; inside of a list comprehension, there is an "extra" entry in locals() for a "magic little variable where we construct that list for you". Since CPython cheats, he decided that he could too. Beyond that, he did not take performance into consideration at all; it was purely a theoretical exercise.

Required syntax

As it turns out, "there are only ten pieces of syntax that you cannot live without"; he was able to eliminate more than 80 sugary pieces. He thinks, though, that once attendees see what has to be done to avoid some of them, they will agree that "a lot of this syntactic sugar is really handy to have".

The first element that you need is integers, which are the building blocks of lots of other things. A float can be represented as integers, bytes are arrays of integers, and Unicode strings are just arrays of codepoints, which are ... integers. "You can build all of the data representations that we have in the language using just integers." Everyone likes string literals, though, so even though you can reduce them to integers, that's not a pleasant path; that's true for other representations, as well, of course.

Function calls are another required piece; you need some way to call code. The while statement can provide both looping and conditional constructs, which makes it more useful than either for or if; those can only handle one or the other. So while is required and the other two can be defined in terms of it.

The try/except statement is another required piece. He could actually get rid of that requirement if he was willing to do the whole-program analysis and rewrite all of the functions in the style of Rust, with return values that are either "ok" or "an exception got raised". The raise statement is also required because of the no-CPython-internals restriction he placed on his experiment.

def is required in order to create functions that provide the API that other parts of the program can call. Both return and yield are needed, though there are ways around that; he could have brought back lambda, which would have allowed eliminating them from the list. Cannon said that he was "too lazy" to do so when he learned how; it is quite a bit of work and eliminating class had already worn him out.

The final two pieces of required syntax are assignment, which is perhaps an obvious one, and nonlocal, which is decidedly less so. The original namespace picture for Python was much simpler, but less powerful; it consisted of local, global, and built-in namespaces. The addition of closures complicated that picture; Cannon needed to keep nonlocal as one of his ten core syntax pieces in order to be able to control access to other namespaces.

So, just those ten pieces of syntax are enough to reimplement every other part of Python, including import, addition, class, and more. Technically, you can write your Python code just using those pieces, "you'd just hate yourself because you'd be having to write the weirdest code in the world".

Essentially, those constructs make up a Turing machine "with some Python flourishes", Cannon said. A Turing machine is more or less "the scientific definition of a computer" that can read, write, and process data; it can also make conditional decisions based on the data. while is available to make decisions, def and function calls for processing data, assignment for reading and writing, and so on. Things like try and except add the Python flourishes.

Examples

On his blog, Cannon has unraveled over 80 other pieces of syntax, but was not able to do all of that in a talk. Instead, he would give some examples that would hopefully give attendees an idea of what it entails. In addition, he hopes that it gives people a look under the hood in order to better understand how Python operates.

He started with a simple example of attribute access for the expression X.Y. That can be replaced with:

    getattr(X, "Y")

Fundamentally, every attribute access in Python is reduced to a call to getattr(). It quickly gets tiring to write attribute accesses that way, however, especially for long chains of them; in fact, he would continue to use the sugary version for the other examples in his talk, he said.

Sometimes, the language definition makes things a bit interesting, as with handling decorators. A Python decorator wraps a function definition, generally to transform the function in some fashion:

    # pre-decorators
    def Y():
        pass
    Y = X(Y)

    # using decorators
    @X
    def Y():
        pass

The decorator syntax was added to make it clearer what was going on right at the top of the function definition; if the function is long, it may be easy to miss that it is getting wrapped and re-assigned. But, as it turns out, the unraveled version cannot be the same as the pre-decorator version above; instead it must look like the following:

    def _Y():
        pass
    Y = X(_Y)

The language specification specifically requires that the new function name (Y) not be visible until after the wrapping has occurred, which surprised him when he was working on this. He believes that the decorator syntax is far more readable and is a great example of why syntactic sugar is added to the language. He knows that Python developers do not always agree with changes of that sort, pointing to the mess around the "walrus" operator (:=) as an obvious case in point, but "we have our reasons" for doing so.

He moved on to "special methods" (or "magic methods"), which are all of the "dunder" (double-underscore) method names on objects, __add__(), __index__(), __init__(), and so on. These are an area where syntactic sugar comes up the most and is the easiest to unravel, because those methods fundamentally implement the operations of interest. For example, ~X (using the bitwise inversion operator) unravels as: type(X).__invert__(X). That may be a little surprising, but the type() call is actually an optimization mechanism; it is much faster to retrieve __invert__() from the class, than from the instance dictionary. In the C code, it just requires following a few pointers to make the call:

    (*Py_TYPE(X)->tp_as_number->nb_invert)(X);

Even though dictionary lookup is fast, the above is "stupid fast", he said.

He moved on to the "most complex example we're going to have in this whole talk: binary operators". He said that attendees may think those operators should be fairly simple to unravel, but they will be surprised. His example was a simple subtraction: X ‑ Y. The operator does not really matter as the process is the same; he chose subtraction because he wanted to show an operator where the order of the operands matters. For subtraction, the first step is:

    type(X).__sub__(X, Y)

That seems pretty straightforward, but what if X does not define __sub__() or if it returns NotImplemented for the type of Y? In that case, the second operand gets a chance to provide an implementation:

    type(Y).__rsub__(Y, X)

The language defines the __rop__() versions of the operators to handle this case. The dunder operators return NotImplemented if they are unable to work with the two objects passed. Another optimization is that if both operands have the same type, only the first dunder operator (__sub__() in this case) will be tried, since there is no point in asking again as it is already known that the two types cannot be accommodated.

But there are class hierarchies where a subclass needs to be able to override the parent class's operator, no matter which side of the operator it falls on. For example, if Y is a subclass of the class of X, the class of Y may want to ensure that subtraction always results in that type. There is a set of rules that Python follows to ensure that such a situation is properly handled: the two types must be different, the right-hand side (RHS), Y in this case, must be a subclass of the left, both operands must have an __rsub__(), and the __rsub__() for Y must be different than that of X. If all of those are true, the RHS can boss its parent around. "Got it all?", he asked with a smile.

That is quite a bit of work to do every time that a binary operator is executed. That is why various efforts are made to avoid that overhead, such as Python extensions that move the operations into C code and the work being done on specialization by the Faster CPython team.

As his last example, he deconstructed if:

    if X:
        Y

    # can use while instead
    while X:
        Y
	break

However, that while version uses break, which is not one of the ten anointed syntax constructs. But break can be unraveled as well:

    _DONE = False
    while not _DONE and X:
        Y
	_DONE = True

But that contains two more pieces of syntax (not and and) that are not on the list. not is similar to the invert operation; it is just a special method call. In the talk, he did not discuss and at all, but for this particular case, he simply avoided both operators by using exceptions:

    try:
        while X:
	    Y
	    raise _DONE
    except _DONE:
        pass

"Once again, syntactic sugar is really frickin' handy, because I sure as heck would not want to write that every time I have an if." He almost forgot to point out that pass was also not on the list, but it could be replaced here with various things, such as None or a constant. His blog posts on not and and show other ways this could be handled.

He ended his talk by thanking his wife Andrea for proofreading most of the blog posts; he stopped asking after a while, so she is not responsible for typos in the later posts, he said with a chuckle. He also thanked Guido van Rossum for proofreading the class unraveling post, "which was huge". Beyond that, Van Rossum was always happy to answer the random questions Cannon posted to the python-dev mailing list about why things are the way they are in the language. Cannon suggested that those who are interested read the blog posts, perhaps starting with the last, "MVPy: Minimum Viable Python", which outlines the goals of the project and lists his ten syntactic elements.

In answer to an inaudible question, which must have referred to names like False and None, Cannon said that those were originally simply variable names and not syntax in Python, so he went back to that mode for his project. Those names were only turned into syntax as an aid to developers so that they did not overwrite them by mistake. The final question was about the != operator used in the subtraction example, which is, of course, also not on the list. Cannon acknowledged that, and pointed attendees at the rich comparison operators post for how that can be unraveled.

Comments (30 posted)

A more dynamic software I/O TLB

The kernel's software I/O translation lookaside buffer ("swiotlb") is an obscure corner of the DMA-support layer. The swiotlb was initially introduced to enable DMA for devices with special challenges, and one might have expected it to fade away as newer peripherals came along. Instead, though, the swiotlb has turned out to be useful in places outside of its original use cases. This patch set from Petr Tesarik now aims to update the swiotlb with an eye toward its continuing use indefinitely into the future.

One of the fundamental features of any reasonably capable I/O device is its ability to perform DMA — accessing data directly in main memory without the need to go through the CPU. Without DMA, I/O performance will be severely limited. But some devices are better at DMA than others. Older devices for PC-class hardware, reflecting the history of that architecture, were often limited to 24 bits of address space for DMA transfers, meaning that they could only access the lower 16MB of memory. That was plenty in the early days, but quickly became limiting as memory sizes grew. Another common problem is a 32-bit limitation, restricting access to the lower 4GB of memory.

The kernel has an extensive support layer that is intended to hide, as much as possible, the limitations of any given device. So, for example, when dealing with a device that has addressing limitations, the kernel will endeavor to allocate memory within the range that the device can reach. There are times, though, when that cannot be done. There may simply not be a sufficiently large contiguous chunk of memory available in a place where the device can reach it, for example. But, more often, the data to be transferred was placed without consideration for a device's limitations. User-space buffers and the kernel's page cache, for example, cannot be restricted to a low range of memory just in case they might be handed to an address-limited device.

The solution in this case is to employ a "bounce buffer". The kernel maintains some low-address memory to support DMA operations. When needed, a portion of the memory will be allocated to be used by the device for I/O. The kernel will copy data from the actual buffer into the bounce buffer (before the operation, for writes) or from the bounce buffer into the actual buffer (after the operation, for reads) as needed. Bounce buffers slow things down, but that is better than not being able to perform I/O at all.

Managing the memory for bounce buffers is the job of the swiotlb. This little subsystem might have once been expected to become unnecessary over time. As newer devices replace older ones, addressing limitations should eventually disappear. For the cases where they don't disappear, the increasing use of I/O memory-management units, which provide a separate address space for devices into which buffers can be mapped, would take care of the problem. Either way, there should be no need for the kernel to continue copying data for I/O through the CPU.

As Tesarik pointed out in the cover letter to the swiotlb patch series, though, that is not quite how things have turned out. Vendors continue to churn out systems, especially in the embedded space, that can handle more memory than their built-in devices can address. Other use cases have come along as well. One that is turning up now is confidential-computing applications, where systems may be running with encrypted memory that devices are unable to access. Confidential-computing systems can end up using bounce buffers for all of their I/O, regardless of whether the devices being used have addressing limitations.

So it seems like the swiotlb is going to stay around for a while, which suggests that fixing some of its own limitations might make sense. By default, the swiotlb allocates 64MB of memory at system boot; that behavior can be adjusted with the swiotlb= command-line parameter. That allocation is intentionally rather small; it was intended to not use much RAM while, hopefully, providing enough memory for the system's limited (hopefully) bounce-buffering needs. But 64MB can be too large a cost for some memory-constrained embedded systems, while being too small for systems where a lot of I/O must be bounce-buffered. The swiotlb, in other words, needs to become more dynamic.

Devices with addressing limitations are also, often, unable to perform scatter/gather I/O, meaning that they need to be presented with a single, physically contiguous buffer. So the swiotlb must be able to allocate physically contiguous memory, a task that has traditionally gotten harder as a system runs and its memory becomes fragmented; that is one of the reasons why swiotlb memory is allocated early in the boot process. A dynamic swiotlb will have to be able to allocate memory at any point, though.

The good news is that allocating physically contiguous chunks of memory has gotten easier over time. The work that has gone into segregating movable and reclaimable allocations has made memory compaction more successful; this helps with the allocation of huge pages — and other large, contiguous memory ranges, such as bounce buffers. The kernel's contiguous memory allocator (CMA) is also helpful in this regard. So it is not entirely unreasonable to expect that memory for bounce buffers can be allocated at any point in the system's operation.

A system with the dynamic swiotlb patches applied will start with the same, small allocation at the beginning — though users may want to make it smaller. Whenever a request for a bounce buffer cannot be satisfied, the swiotlb will allocate more memory. Here, things get a little complicated, though; bounce buffers can be allocated in contexts like interrupt handlers, which cannot block. So the swiotlb cannot allocate a new pool, sufficient for a number of future requests at that time. Instead, the requested space will be allocated directly from the memory-management subsystem, and a function call will be kicked off via a workqueue to do the larger allocation in a more forgiving context.

The "transient" buffers allocated when the swiotlb runs out of pool memory will be freed back to the system once they are unmapped. The regular pools, though, are held onto forever; once the pool grows, it will stay larger. This policy may make sense; a device that needs a certain amount of bounce buffering will likely continue to need those buffers in the future. Meanwhile, the swiotlb as a whole will probably stay small enough that retaining that memory should not be a problem. That said, it would not be entirely surprising to see some sort of shrinker mechanism added to the swiotlb in the future.

This patch set has been through seven revisions and developers seem to be mostly happy with it. Christoph Hellwig has applied the series, presumably to be pushed into the mainline during the 6.6 merge window. Of course, like every other kernel project, the work here is never really done; Tesarik has already let it be known that there are follow-up patches in the works.

Comments (1 posted)

Development statistics for the 6.5 kernel

The 6.5 kernel was released on August 27 after a nine-week development cycle. By that time, some 13,561 non-merge changesets had found their way into the mainline repository, the lowest number seen since the 5.15 release (12,377 changesets) in late 2021. Nonetheless, quite a bit of significant work was done in this cycle; read on for a look at where that work came from.

1,921 Developers contributed to 6.5, a slightly lower number than usual; 271 of those developers made their first kernel contribution for this release. The top contributors to the 6.5 kernel were:

Most active 6.5 developers

By changesets Uwe Kleine-König 272 2.0% Krzysztof Kozlowski 235 1.7% Christoph Hellwig 213 1.6% Ian Rogers 198 1.5% Arnd Bergmann 190 1.4% Hans de Goede 140 1.0% David Howells 120 0.9% Lijo Lazar 115 0.8% Johannes Berg 110 0.8% Dmitry Baryshkov 108 0.8% Konrad Dybcio 99 0.7% Mark Brown 96 0.7% Jani Nikula 96 0.7% Thomas Gleixner 91 0.7% Oswald Buddenhagen 91 0.7% Christophe JAILLET 88 0.6% Nishanth Menon 87 0.6% Andy Shevchenko 85 0.6% Jakub Kicinski 84 0.6% Thomas Zimmermann 74 0.5%

By changed lines Ian Rogers 26662 4.3% Zong-Zhe Yang 20395 3.3% Jakub Kicinski 17505 2.8% Hawking Zhang 16730 2.7% Mark Rutland 14829 2.4% Ping-Ke Shih 11090 1.8% Rohit Agarwal 8669 1.4% Benjamin Gaignard 8275 1.3% Takashi Iwai 7675 1.2% Hans de Goede 6112 1.0% Jonathan Kim 5820 0.9% Le Ma 5337 0.9% Jani Nikula 5334 0.9% Konrad Dybcio 5263 0.8% Bryan O'Donoghue 5119 0.8% Christoph Hellwig 4924 0.8% David Howells 4848 0.8% Pawel Laszczak 4827 0.8% Dmitry Baryshkov 4773 0.8% Johannes Berg 4748 0.8%

As was the case for 6.4, Uwe Kleine-König tops the list of individual contributors as measured by number of changesets, again mostly for work changing the internal platform-device driver API. Krzysztof Kozlowski is, again as with 6.4, in the second spot, having contributed a lot of devicetree changes. Christoph Hellwig continues refactoring code in the block and filesystem layers, with much of his work being in the Btrfs filesystem this time around. Ian Rogers (again, as with 6.4) contributed a lot of changes to the perf tool, and Arnd Bergmann made fixes all over the kernel tree.

Rogers touched the most lines of code this time around (in a cycle that featured relatively few large commits) by updating various vendor-specific perf events. Zong-Zhe Yang contributed 25 changes to the rtw89 WiFi driver, Jakub Kicinski added code to the ynl tool, Hawking Zhang added a relatively restrained set of amdgpu headers, and Mark Rutland reorganized and documented a number of low-level locking primitives.

The top testers and reviewers for this cycle were:

Test and review credits in 6.5
Tested-by Daniel Wheeler 165 11.5% Michael Kelley 63 4.4% Kan Liang 44 3.1% Guilherme G. Piccoli 40 2.8% Helge Deller 38 2.6% Oleksandr Natalenko 37 2.6% Heiko Stuebner 32 2.2% Randy Dunlap 25 1.7% Naama Meir 21 1.5% Rafal Romanowski 19 1.3% Pucha Himasekhar Reddy 19 1.3%	Reviewed-by Simon Horman 276 3.1% Konrad Dybcio 220 2.5% Christoph Hellwig 218 2.5% Hawking Zhang 161 1.8% Kees Cook 153 1.7% Andy Shevchenko 147 1.7% David Sterba 142 1.6% Krzysztof Kozlowski 142 1.6% Felix Kuehling 138 1.6% Conor Dooley 115 1.3% Dmitry Baryshkov 113 1.3%

There are two developers who are credited with having tested at least one patch for every day in this 63-day cycle, and 25 who reviewed at least one patch per day. Simon Horman, who reviews patches in the network subsystem, sustained a rate of over four per day. Overall, 1,177 (8.7%) of the commits in 6.5 had Tested-by tags, and 6,702 (49%) had Reviewed-by tags, continuing the longstanding, gradual increase in the use of those tags.

A total of 218 employers (that we were able to identify) supported work on the 6.5 kernel; the most active of those were:

Most active 6.5 employers

By changesets Intel 1503 11.1% AMD 1233 9.1% Linaro 1174 8.7% Google 922 6.8% (Unknown) 838 6.2% Red Hat 777 5.7% (None) 623 4.6% NVIDIA 381 2.8% SUSE 366 2.7% Huawei Technologies 347 2.6% Pengutronix 326 2.4% Qualcomm 303 2.2% Oracle 293 2.2% Meta 265 2.0% (Consultant) 261 1.9% IBM 236 1.7% Texas Instruments 193 1.4% Arm 180 1.3% Renesas Electronics 151 1.1% Collabora 147 1.1%

By lines changed Intel 66098 10.7% AMD 65508 10.6% Google 46208 7.4% Linaro 41969 6.8% Realtek 34719 5.6% (Unknown) 31602 5.1% Red Hat 26304 4.2% Qualcomm 23527 3.8% Meta 22721 3.7% Arm 18813 3.0% Collabora 16801 2.7% (None) 16168 2.6% SUSE 15021 2.4% NVIDIA 12800 2.1% Texas Instruments 11308 1.8% Oracle 8627 1.4% Huawei Technologies 7908 1.3% IBM 7311 1.2% (Consultant) 6897 1.1% MediaTek 6562 1.1%

The list of companies working on the kernel tends not to change much — at the upper end of the list, anyway. This list says little about what these companies are up to in the kernel, though. Below is a summary, for some of the top corporations, of the number of changesets touching some of the more active areas of the kernel (and the percentage that makes of the company's total contribution):

	AMD		Google	Intel	Linaro	NVIDIA	Red Hat	(None)	(Unknown)
Architecture	65	5%	213	23%	114	8%	347	30%	15	4%	40	5%	111	18%	135	17%
Core kernel	33	3%	182	20%	147	10%	74	6%	15	4%	51	7%	75	12%	53	7%
Block/filesystem	0	0%	62	7%	19	1%	15	1%	0	0%	154	21%	26	4%	42	5%
Drivers	1,106	91%	199	22%	1064	73%	597	52%	275	75%	286	39%	366	60%	413	51%
Networking	10	1%	74	8%	313	21%	60	5%	221	60%	133	18%	79	13%	104	13%
Tools	17	1%	235	26%	55	4%	17	1%	58	16%	112	15%	5	1%	41	5%
Documentation	27	2%	39	4%	29	2%	148	13%	15	4%	31	4%	53	9%	97	12%

This table gives some idea of where the top contributors put their effort in the kernel. The sharp-eyed may notice that the percentages do not add up to 100%; that is primarily because many changes touch more than one subsystem, and are thus counted more than once.

It will not be surprising to see that the hardware-manufacturing companies — AMD, Intel, and NVIDIA — focus their work in the driver subsystem. They are here first and foremost to ensure that their products work well with Linux, and that means working on drivers. Perhaps a bit more surprising is that these companies, all of which make CPUs, do not make many changes to the architecture-support code (or, at least, did not do that during the 6.5 development cycle). Google's commit count under arch/ exceeded that of those other three companies combined.

Linaro, which exists to support the Arm architecture, does put a fair amount of effort into the arch/ subtree. Lest one think that Linaro is a documentation champion, though, one should remember that the devicetree bindings live under Documentation/. That work is really more hardware support in the end. Of the groups listed above, developers with no known affiliation (the "None" and "Unknown" columns) are seemingly the most enthusiastic, percentage-wise, about contributing to the documentation.

A lot of work, from some companies at least, is done in the tools/ subtree. Activity there is focused on the perf tool, the growing self-test suite, the objtool utility that is an ever-more important part of the build process, and more. An increasing amount of the code in the kernel tree runs in user space.

Kernel development goes through periods of high and low activity; 6.5 was a slow cycle (though, clearly, in relative terms), but 6.6 might pick up the pace a bit. There are 11,642 non-merge changesets queued there as of this writing. At times, it seems, there is simply less work needing to be done. Experience says, though, that the patch rate will pick up again before too long; the kernel project, it seems, is never done.

Comments (8 posted)

The OpenSprinkler controller

The more one pays attention to the Internet of Things (IoT), the more one learns to appreciate simple, unconnected devices. Your editor long ago acquired an aversion to products that advertise themselves as "smart" or "WiFi-enabled". There can be advantages, though, to devices that contain microprocessors, are Internet connected, and are remotely accessible, if they are implemented well. The OpenSprinkler sprinkler timer would appear to be a case in point.

This article is being written in a part of the world with limited rainfall — a near-desert environment. That notwithstanding, the local humans have reached the conclusion that it would be a good idea to surround their homes with lush, green vegetation that evolved to thrive in a rather more humid environment, and which requires fairly intensive life support — and water pumped from the other side of the continental divide — to survive here. Western civilization, it seems, depends on us continuing to do this; otherwise we would surely not continue to put so many resources into it.

Providing life support to vegetation by dragging a hose around quickly loses any charm it may have once had, so the installation of automated sprinkler systems is common in these parts. The control system takes the form of a timer that, traditionally, has been programmed through a painful combination of dial turns and button pushes; anybody who has tried to figure out how to configure a bicycle computer (speedometer) will understand. More recently, of course, we have seen the advent of smart controllers that are said to make this process easier and to enable control of the system while vacationing in a distant location. The allure of being able to soak a Colorado front yard while lounging on a South-Pacific beach is, seemingly, irresistible.

These "smart" controllers have all of the afflictions of other IoT products. Their security is questionable at best, as is the security of the cloud-based systems they connect to — and store their data on. The cloud service itself exists at the whim of the vendor and can vanish at any time, turning a smart controller into a remarkably dumb brick. Any data collected by these devices is centrally stored and routinely exploited by unknown third parties. Occasionally, the device will cease functioning until an "upgrade" is applied (assuming the owner is asked about the upgrade at all) that adds new behaviors that the owner never asked for or wanted. An Internet failure can lead to a sprinkler failure, turning one's bit of American paradise into a forlorn dustscape.

All of this can make the old way of knobs and buttons look appealing.

OpenSprinkler appears to be an attempt to do things better. The device itself is based on an ESP8266 controller, and the hardware design is freely available (though with an unspecified license). The firmware that runs the device is available under GPLv3, and the mobile app carries the AGPLv3 license. There is a version of the software that can run on a Raspberry Pi for those who want to put together their own controller.

The hardware, as shipped, can manage up to eight zones; expansion modules (one is shown in the photo) can bring that number up to 72 — far more than your typical hardware-store sprinkler timer. The device has inputs for two sensors, either for rainfall or ground moisture. It has a built-in WiFi interface, and a version with a wired Ethernet port is also available.

The device lacks a display that is usable for more than the most basic of tasks, though; in an era when we all carry interaction devices in our pockets, that is seemingly unnecessary. Initially, the controller comes up in access-point mode; the owner can put their device onto that network and configure the controller. It is, indeed, possible to keep it in the access-point mode forever, at which point it can function with no supporting systems other than the device used to communicate with it.

More likely, though, one will want to put it onto the local WiFi network, which is easily accomplished; after that, the access point is shut down. The controller can display the IP address it was assigned on its tiny display; any web browser that can reach that address can be used to configure it. The Android app (which is not available via F-Droid despite being free software; it can be had via the Play Store) can scan the local net and find the controller automatically; after that it presents a screen that is essentially identical to the web interface.

The first configuration step, naturally, is to set a password on the device; a good password, after all, is the only thing preventing your jealous neighbors from learning your top-secret watering strategy or programming the device to turn the flower garden into a swamp. The dashboard screen shows the state of all zones (or as many as will fit on the screen) and allows individual zones to be run with a couple of clicks.

The programming interface is flexible, allowing the creation of multiple, named programs, each of which has its own zone list and watering schedule. Both day-of-week and every-Nth-day programs are supported — an important feature in locations where watering restrictions (imposed on those rare occasions where, say, the ability to put out fires is deemed to be even more important than green grass) require one or the other. There is an overall scaling feature for seasonal adjustments, and watering can be deferred based on input from either sensor (if any are connected).

The device will, by default, use NTP to keep its clock adjusted, sparing one from the shame of sprinklers that start five minutes too late. It is also able to connect to Weather Underground, download weather data, and adjust watering according to a number of different algorithms. There are options for connecting to remote logging servers to enable, for example, a sprinkler-status window in one's home automation dashboard. Assuming one has such a thing, of course.

It is, in other words, a capable and fully featured sprinkler timer. But that is not really the point here; this is not the Lawn Watering News. The real point is that OpenSprinkler is a device that one can buy or build independently, with open-source firmware that can be modified and updated at will, with an entirely optional, open-source app, that can be operated entirely without the use of any sort of cloud service whatsoever. That seems like a good demonstration of how the Internet of Things could work, at least for certain classes of devices.

One does not often need to control sprinkler systems from far away, in your editor's experience; it is more of an on-the-scene job. Should the need arise, though, there are a couple of options. One is to expose the built-in web server to the Internet, perhaps by poking a hole through the router firewall. Given the (presumably) low level of security auditing this device has received and the complete lack of any sort of security update mechanism, that may not be the wisest of ideas.

As an alternative, the OpenThings Cloud (OTC) service can be used as an intermediary. This service runs a simple proxy server, licensed under GPLv3. Unless one exposes the painfully long token assigned by this service, that will effectively block all unwanted access to the device (which will still require a password from anybody who gets through, of course). One can simply trust the OTC server to actually be running the posted software and not secretly monitoring whether your are watering your roses sufficiently; alternatively, setting up an independent server on a low-end cloud instance would be a straightforward task.

The end result is a device that uses open hardware and software to do a useful job in a way that respects its owner. OpenSprinkler is capable enough at its core task to stand out, even for users who do not care about the other aspects. But for those of us who prefer our computers (and our homes) to be under our control and serving our interests — rather than those of one or more random corporations — OpenSprinkler is truly a breath of fresh air. We do not actually need to embed computers in everything around us but, when that is a useful thing to do, this is an example of how it should be done.

Comments (31 posted)