Notes from Linaro Connect

Connect is an interesting event, in that it is a combination of an architecture-specific kernel developers' gathering and a members-only meeting session. Not being a member, your editor only participated in the former aspect. Sessions at Connect are usually short — 25 minutes — and focused on a specific topic; they also routinely run over their allotted time. There is an emphasis on discussion, especially in the relatively unstructured "hack sessions" that occupy much of the schedule. Many of the sessions are focused on training: how to upstream code, for example, or kernel debugging stories in Mandarin (video).

A lack of Mandarin language skills will prevent coverage of that last session, but there were others that were somewhat more comprehensible to the Mandarin-impaired.

Kernelci.org

Mark Brown, Tyler Baker, and Matt Hart ran a session about kernelci.org, arguably one of the community's most underappreciated testing resources. This operation, run by Linaro and BayLibre, performs automatic build-and-boot testing for the kernel; its infrastructure is hosted by Linaro and Collabora. Tests are run on the ARM, ARM64, MIPS, and x86 architectures. For every commit that is made, tests are run with every in-tree defconfig file — over 260 builds for every commit. The resulting kernels are booted on over 100 different boards. This operation, Brown said, greatly increases the likelihood that kernels will build and, as a result, the number of failed configurations is going down over time. That, in turn, makes merge windows less stressful.

As Baker explained, this testing structure is driven by the LAVA tool; it serves as a scheduler and job runner for board farms. LAVA is used and distributed by Debian, he said.

LAVA is completing a big transition to the v2 release, which makes a number of significant changes, Hart said. Job files are now created with Jinja2 templates, an improvement over the hand-written JSON used in v1. Jobs are run asynchronously, without polling, and ZeroMQ is used for communications. ReactOBus is used to run jobs from messages. LAVA v1 tried to apply a fair amount of magic to hide the differences between different test systems, but that proved hard to work with. So v2 requires more explicit configuration in this area.

The v2 system is settling in, but a permanent home for the ReactOBus daemon is yet to be identified. [Video]

Load tracking

Vincent Guittot ran a session on load tracking — keeping track of how much load each process puts on the CPU. Accurate load tracking can help the scheduler make better decisions about task placement in the system; it can also be helpful when trying to minimize the system's power consumption. The per-entity load tracking (PELT) mechanism in the kernel is better than what came before, but it is proving to not work as well as desired, especially when it comes to power management. The window-assisted load tracking (WALT) patches (described in this article) improve the situation, but that work has not made it into the mainline.

The complaints with PELT are that it is not responsive enough and that its metrics are not always stable. The tracking of small tasks can be inaccurate, causing a mostly idle CPU to appear to be busy. Load is not propagated between control groups when a task is migrated which, among other things, can cause erroneous CPU-frequency changes. The good news is that around 20 patches improving PELT have been merged since 4.7; they fix the small-task tracking and load-propagation issues. Upcoming work should improve the handling of blocked loads and address some of the frequency scaling issues.

A related problem is that the community has lacked realistic benchmarks to measure the results of load-tracking changes; that is being addressed with new tests. There are some interesting interactions with the processor's thermal management mechanisms, though. When asked, Guittot said that there has not been a lot of power-consumption testing so far; most of the work to this point has been focused on performance.

Future work includes improving utilization tracking for realtime tasks, which are currently not part of the load-tracking mechanism. There are also some practical problems on current hardware. Realtime tasks want to run at the maximum frequency, but a frequency change on a HiKey board takes 1.5ms. A realtime task needing 2ms of run time will not get maximum-frequency performance. A more responsive load-tracking mechanism could help the scheduler ensure that the CPU is running at the needed speed. There is also a focus on improving responsiveness, which comes down to ensuring that the CPU frequency is increased quickly when the need arises. A slow ramp-up will lead to observable behaviors like jumpy scrolling. Finally, there is a desire to improve the responsiveness of the PELT system, perhaps by introducing the windowing technique used in WALT. [Slides]

ARM Mali drivers

Many ARM systems come equipped with the ARM Mali GPU. In a session on the state of the Mali drivers, John Reitan and Ketil Johnsen described some of the work that is being done in this area. There is a software development kit for the Vulkan graphics API available under the BSD license. Not all GPUs support Vulkan, though; in particular, the HiKey board used by many ARM developers has no Vulkan support. Within a month or so ARM should be releasing its compute library that allows running code on the GPU. It may be useful for image-recognition tasks and more. It will show up on the ARM GitHub page once it's ready.

With the news items out of the way, the audience quickly moved the discussion to the topic its members were really interested in: prospects for open-sourcing the Mali driver code. The answer was that ARM has no intention of doing so, mostly out of fear of unspecified "patent issues". The risk of patent trolls is simply too great to allow that release to happen. This was not, of course, the answer that the audience wanted to hear, but nobody was particularly surprised.

Arnd Bergmann suggested that perhaps a free Vulkan driver could be released; Vulkan is simpler than the full OpenGL API and might thus pose a lower risk. The speakers are not lawyers and could not respond to that suggestion beyond agreeing that it is worth considering. Meanwhile, there is a possibility that free drivers for some subcomponents could be released in the relatively near future.

A related pain point around Mali is the lack of device-tree bindings in the kernel. The normal rule is that bindings are only accepted for drivers that are, themselves, in the kernel; there is no Mali driver there, thus no bindings. But that has led every SoC vendor to come up with its own customized bindings. There has been talk of loosening the rules a bit to allow the addition of bindings for some out-of-tree drivers to reduce this pain.

John Stultz pointed out that running the Mali drivers on mainline kernels is often difficult, and wondered if there were any improvements expected in that area. Development effort on the binary-only driver tends to be focused on kernels the customers are using, and those kernels are usually old. Internally, the Mali driver does usually work on the mainline, but it can take months for the patches to get out to the rest of the world.

It's also hard for distributors who would like to make the binary-only driver available to their users. One recent improvement, at least, is that the license on that driver has changed to allow it to be distributed. But it is still difficult to make a package that works on even a subset of boards. Meanwhile, every driver release tends to break systems, and the driver tends to break with kernel updates. As Grant Likely pointed out, having to keep the kernel and the user-space driver code in lockstep makes the creation of any sort of generic distribution difficult. It was agreed that a better job needs to be done here. [Video]

For those interested in other Connect sessions, the full set of videos and slides from the "BUD17" event can be found on this page. The next Connect will be held September 25 to 29 in San Francisco, California.

[Thanks to Linaro and the Linux Foundation for funding your editor's travel to Connect.]

Comments (4 posted)

2038: only 21 years away

That work, he said, is proceeding on three separate fronts, the first of which is the kernel itself. He has been working for the last five years to try to prepare the kernel for 2038. Much of that work involves converting 32-bit timestamps to 64-bit values, even on 32-bit systems. Some 32-bit timestamps also show up in the user-space API, which complicates the issue considerably. There is a plan for the enhancement of the user-space API with 2038-clean versions of the problematic system calls, but it has not yet gotten upstream. One recent exception is the statx() system call, which was merged for 4.11; statx() will serve as the year-2038-capable version of the stat() family of calls. There are quite a few other system calls still needing 2038-clean replacements, though.

There is one other person, Deepa Dinamani, working on the kernel side of things; she started as an Outreachy intern and has continued to work on the problem after the internship ended. Dinamani has a set of virtual filesystem layer patches in hand, which address one of the hardest problems, and she has plans for some other system calls as well. One of the trickier ones might be setsockopt(), which isn't easily fixed or emulated at the glibc level. There are device-mapper and input subsystem patches in an advanced state. Bergmann had a patch for the video4linux subsystem, but that was rejected and needs a new approach; a similar situation exists for the audio subsystem. Other areas needing work in the kernel are key management and realtime clocks.

For some system calls, there will be no replacement, since the best approach appears to be emulation in the C libraries — the second focus for the year-2038 effort. There has been a lot of work done in the glibc community in particular, he said; the plan is to be fully backward compatible at that level. That means that it will be possible to build a program with either 32-bit or 64-bit timestamps, and to use the larger timestamps even on older kernels. In other words, the glibc developers are trying to make things work everywhere, with a minimum of disruption. (See this draft design document for lots of details on the glibc plan.)

The third focus is on distribution builds, which can only really proceed once the first two pieces are in place. Most distributors, Bergmann said, are unlikely to even bother with 32-bit systems in 2038, so they won't have much to worry about. One big exception may be Debian, which seems interested in maintaining support, even though it looks like it will be a painful task. It may require a full rebuild at some point, which isn't much fun for anybody involved, but it is at least a process that is known to work. Compatibility is key in such a system; there is code being deployed now that may not be 2038-clean, but people want it to keep working if at all possible.

One big area of concern is automobiles. A lot of devices, such as handsets, will have long since failed for any of a number of other reasons by then, so there is little point in ensuring that they can survive 2038. But people keep cars going for a long time. There may still be cameras in use by then, and there is highly likely to be a lot of deeply embedded systems such as building infrastructure. Some of these systems are going to fail in 2038. That is why it is important to get the problem fixed as soon as possible.

There are some things that are going to be difficult to fix, though, even when the kernel, C libraries, and distributions are mostly taken care of; many of these are the result of the use of 32-bit time_t values in file formats. Thus, for example, cpio will fail, which is problematic because it is used by the RPM package format. The NFSv3, ext3, and XFS filesystems all have problems resulting from their use of 32-bit timestamps. The first two are likely to have gone out of use long before the problem hits, and plans for the repair of XFS are in the works. Then, of course, there is a whole list of applications that nobody has yet noticed are broken, and lots of in-house applications that cannot be fixed by the community.

When asked which tools he is using for this work, Bergmann replied that his core technique involves building kernels with the 32-bit time types removed completely. That will quickly point out the places that still need to be fixed. Beyond that, he said, it's mostly a manual process. It was suggested that sparse or GCC plugins could maybe help with this task.

As things wound down, John Stultz asked how much the work in the BSD camp, which has (in some variants) already solved its year-2038 problems, might help with Linux. The answer would appear to be "not much". BSD-based distributions have the advantage of being able to rebuild everything from scratch, so they do not need to maintain user-space ABI compatibility in the same way. There is some value in in the work that has been done to fix applications with year-2038 problems, but it's not clear how much that will help the Linux community.

[Thanks to Linaro and the Linux Foundation for funding your editor's travel to Connect.]

Comments (53 posted)

A kernel TEE party

A processor with TrustZone support can run in two modes, often termed the secure and non-secure worlds. The secure world can be given exclusive access to memory, devices, and more; those resources are inaccessible when running in the non-secure mode. A hardware trap mechanism is used to transition between the two worlds. The secure-world code, thus, looks a lot like an innermost kernel, protected from the Linux kernel, that has a limited set of security-relevant tasks to perform.

The secure world gains control first at boot time, allowing it to set up access to the rest of its resources and configure the environment for the non-secure kernel. This would also be the time to implement a secure-boot mechanism, if desired, ensuring that the non-secure kernel carries an acceptable signature. A number of functions can be implemented in the secure-world code. For example, cryptographic keys can be stored there, inaccessible to the rest of the system, but usable by the secure world to generate signatures. Naturally, there is interest in using the secure world to implement digital rights management mechanisms and other unpleasant things. Like most of these technologies, TrustZone can be used to ensure a user's control over their computer or to take it away.

The code running in the secure world is often called the "trusted execution environment", or TEE (as opposed to the non-secure "rich execution environment" or REE). As befits the ARM world, there are a lot of TEEs out there, each with its own interface to the kernel. The purpose of the generic TEE subsystem is to settle on one kernel-side interface for TEEs, with a standard communication mechanism for talking to them. To that end, it creates two sets of devices: /dev/teeN, and /dev/teeprivN. The former set allows user space to make requests of the TEE, while the latter is there for "supplicant" processes that provide services to the TEE.

A process needing a TEE service opens the appropriate device, then issues a series of ioctl() calls. The available commands include TEE_IOC_SHM_ALLOC to allocate a range of memory shared between the process and the TEE, TEE_IOC_INVOKE to call a function inside the TEE, and more. For each call, the TEE generates some sort of results which are passed back to the calling process. For the /dev/teepriv interfaces, the calls go the other way; when the TEE needs a user-space service (the contents of a file on disk, say), it will send a request to the waiting supplicant process via this interface.

The generic TEE code is, for the most part, a wrapper layer that turns user-space ioctl() calls into calls to the low-level driver. That driver will have registered itself, providing a tee_driver_ops structure with its operations. The biggest exception relates to the handling of shared-memory segments, which involves a certain amount of complexity. There is a fairly elaborate reference-counting mechanism that, for example, tracks the references created by every parameter passed into a TEE operation; that ensures that the memory is not freed while references still exist.

The low-level driver code is charged with communicating between the generic TEE layer and the actual TEE implementation. Much of this code is concerned with formatting requests as expected by the TEE and low-level communications. Shared memory is also an issue at this level, since mapping memory that the TEE can access requires cooperation with the TEE itself.

The generic TEE subsystem patches come with a driver for one TEE, an open-source implementation called OP-TEE. This system, developed by STMicroelectronics and Linaro, provides a simple operating-system kernel meant to run in the secure world. OP-TEE is meant to be the substrate on which TEEs are created; those wanting to learn more can have a look at this design document. There is also an OP-TEE "hello world" application giving an idea of what secure-world code looks like.

All of this structure is intended to build things like the OP-TEE secure data path (recently presented at Linaro Connect), the purpose of which is to enable Android devices to play protected content without letting the end users actually get their hands on it. But a free TEE environment should also make it easy to develop less user-hostile secure services as free software. Once the generic TEE patches are merged (something that seems like it should happen sometime this year), we'll have a standard interface for providing and using such services.

Comments (none posted)

Security quotes of the week

Comments (2 posted)

Critical vulnerability under “massive” attack imperils high-impact sites (Ars Technica)

Comments (24 posted)

Alert summary March 9, 2017 to March 15, 2017

Kernel release status

The March 14 4.11 regression report shows nine known problems.

Stable updates: 4.10.2, 4.9.14, and 4.4.53 were released on March 12, followed by 4.10.3, 4.9.15, and 4.4.54 on March 15.

Comments (none posted)

Quotes of the week

Comments (none posted)

Kernel podcast

Comments (none posted)

Five-level page tables

A page table, of course, maps a virtual memory address to the physical address where the data is actually stored. It is conceptually a linear array indexed by the virtual address (or, at least, by the page-frame-number portion of that address) and yielding the page-frame number of the associated physical page. Such an array would be large, though, and it would be hugely wasteful. Most processes don't use the full available virtual address space even on 32-bit systems, and they don't use even a tiny fraction of it on 64-bit systems, so the address space tends to be sparsely populated and, as a result, much of that array would go unused.

The solution to this problem, as implemented in the hardware for decades, is to turn the linear array into a sparse tree representing the address space. The result is something that looks like this:

The row of boxes across the top represents the bits of a 64-bit virtual address. To translate that address, the hardware splits the address into several bit fields. Note that, in the scheme shown here (corresponding to how the x86-64 architecture uses addresses), the uppermost 16 bits are discarded; only the lower 48 bits of the virtual address are used. Of the bits that are used, the nine most significant (bits 39-47) are used to index into the page global directory (PGD); a single page for each address space. The value read there is the address of the page upper directory (PUD); bits 30-38 of the virtual address are used to index into the indicated PUD page to get the address of the page middle directory (PMD). With bits 21-29, the PMD can be indexed to get the lowest level page table, just called the PTE. Finally, bits 12-20 of the virtual address will, when used to index into the PTE, yield the physical address of the actual page containing the data. The lowest twelve bits of the virtual address are the offset into the page itself.

At any level of the page table, the pointer to the next level can be null, indicating that there are no valid virtual addresses in that range. This scheme thus allows large subtrees to be missing, corresponding to ranges of the address space that have no mapping. The middle levels can also have special entries indicating that they point directly to a (large) physical page rather than to a lower-level page table; that is how huge pages are implemented. A 2MB huge page would be found directly at the PMD level, with no intervening PTE page, for example.

One can quickly see that the process of translating a virtual address is going to be expensive, requiring several fetches from main memory. That is why the translation lookaside buffer (TLB) is so important for the performance of the system as a whole, and why huge pages, which require fewer lookups, also help.

It is worth noting that not all systems run with four levels of page tables. 32-Bit systems use three or even two levels, for example. The memory-management code is written as if all four levels were always present; some careful coding ensures that, in kernels configured to use fewer levels, the code managing the unused levels is transparently left out.

Back when four-level page tables were merged, your editor wrote: "Now x86-64 users can have a virtual address space covering 128TB of memory, which really should last them for a little while." The value of "a little while" can now be quantified: it would appear to be about 12 years. Though, in truth, the real constraint appears to be the 64TB of physical memory that current x86-64 processors can address; as Kirill Shutemov noted in the x86 five-level page-table patches, there are already vendors shipping systems with that much memory installed.

As is so often the case in this field, the solution is to add another level of indirection in the form of a fifth level of page tables. The new level, called the "P4D", is inserted between the PGD and the PUD. The patches adding this level were merged for 4.11-rc2, even though there is, at this point, no support for five-level paging on any hardware. While the addition of four-level page tables caused a bit of nervousness, the five-level patches merged were described as "low risk". At this point, the memory-management developers have a pretty good handle on the changes that need to be made to add another level.

The patches adding five-level support for upcoming Intel processors is currently slated for 4.12. Systems running with five-level paging will support 52-bit physical addresses and 57-bit virtual addresses. Or, as Shutemov put it: "It bumps the limits to 128 PiB of virtual address space and 4 PiB of physical address space. This 'ought to be enough for anybody'." The new level also allows the creation of 512GB huge pages.

The current patches have a couple of loose ends to take care of. One of those is that Xen will not work on systems with five-level page tables enabled; it will continue to work on four-level systems. There is also a need for a boot-time flag to allow switching between four-level and five-level paging so that distributors don't have to ship two different kernel binaries.

Another interesting problem is described at the end of the patch series. It would appear that there are programs out there that "know" that only the bottom 48 bits of a virtual address are valid. They take advantage of that knowledge by encoding other information in the uppermost bits. Those programs will clearly break if those bits suddenly become part of the address itself. To avoid such problems, the x86 patches in their current form will not allocate memory in the new address space by default. An application that needs that much memory, and which does not play games with virtual addresses, can provide an address hint above the boundary in a call to mmap(), at which point the kernel will understand that mappings in the upper range are accessible.

Anybody wanting to play with the new mode can do so now with QEMU, which understands five-level page tables. Otherwise it will be a matter of waiting for the processors to come out — and the funds to buy a machine with that much memory in it. When the hardware is available, the kernel should be ready for it.

Comments (13 posted)

A deadline scheduler update

The deadline scheduler was merged in the 3.14 development cycle. It adds a realtime scheduling policy that, in many ways, is more powerful than traditional, priority-based scheduling. It allows for the specification of explicit latency constraints and avoids starvation of processes by design. The scheduler has better information about the constraints of the workload it is running and can thus make better decisions.

The kernel's scheduler is based on the earliest deadline first (EDF) algorithm, under which the process with the first-expiring deadline is the one that is chosen to run. EDF is enhanced with the constant bandwidth server (CBS) algorithm (described in detail in this article), which prevents a process that is unable to run for much of its period from interfering with others. Essentially, CBS says that a deadline process must use its CPU-time reservation over the course of its scheduling period, rather than procrastinating and expecting the full reservation to be available right before the deadline. The result is a scheduler that provides strong temporal isolation for tasks, where no process can prevent another from satisfying its deadlines.

At the moment, the mobile and embedded development community is putting a lot of effort into energy-aware scheduling. This work has a valuable goal — making scheduling decisions that minimize a system's energy use — but it has proved to be hard to get upstream, though it has been merged into the Android common kernel. For many workloads, it may be that deadline scheduling is a better fit for energy-conscious workloads in the end, Lelli said

A new feature under development is bandwidth reclaiming. One of the core features of deadline scheduling is that, when a process exceeds its CPU-time reservation (its CPU "bandwidth"), the scheduler will simply throttle it until its next scheduling period. This throttling is needed to ensure that the process cannot interfere with other processes on the system, but it can be a problem if a process occasionally has a legitimate need for more than its allotted time. Bandwidth reclaiming just does the obvious thing: it gives that process more time if it's not needed by other processes in the system.

What is perhaps less obvious is the determination of how much CPU time is not actually needed. This is done with the GRUB ("greedy utilization of unused bandwidth") algorithm described in this paper [PDF]. In short, GRUB tracks how much of the available CPU time is actually being used by the set of running deadline tasks and, from that, forms an estimate of how much CPU time will go unused. With that estimate in place, handing out some of the spare time to a deadline process that finds itself in need is a relatively straightforward business.

CPU-frequency scaling is an important tool in the power-management portfolio, but it has traditionally not played well with realtime scheduling algorithms, including deadline scheduling. In current kernels, it is assumed that realtime tasks need the full power of the CPU, so the presence of such tasks will cause the CPU to run at full speed. That may be wasteful, though, if the deadline processes running do not need that much CPU time.

Fixing that problem requires a number of changes, starting with the fact that the deadline scheduler itself assumes that the CPU will be running at full speed. The scheduler needs to be fixed so that it can scale reservation times to match the current speed of the processor. This awareness needs to be expanded to heterogeneous multiprocessing systems (such as big.LITTLE, where not all processors in the system are equally fast) as well.

Once that is done, it would be beneficial to be able to drive CPU-frequency selection from the deadline scheduler as well. The CFS scheduler used for normal tasks uses the per-entity load tracking mechanism to help with frequency selection, but the deadline scheduler currently just pushes the frequency to the maximum. Once the bandwidth reclaiming code is in, it will become possible to measure and use the actual load added by deadline tasks. At that point, a CPU frequency that efficiently gets all of the necessary work done can be chosen.

Of course, there are always a number of fiddly details to take care of. For example, on ARM systems, CPU-frequency changes are done in a separate worker thread. For CPU scaling and deadline scheduling to work together, a way for that thread to preempt deadline tasks (which are normally not preemptable) will need to be found.

The deadline scheduler currently works at the level of individual processes; it does not work with control groups. But there are times when it might make sense to give a deadline reservation to a group of processes. Lelli cited virtual-machine threads and rendering pipelines as a couple of candidate workloads for group deadline scheduling. The implementation envisioned here would be a sort of two-level hybrid hierarchy. At the top level, the EDF algorithm would be used to pick the next group to execute; within that group, though, normal realtime scheduling (FIFO or round-robin) would be used instead. Once this feature works, he said, it could supplant the longstanding realtime throttling mechanism.

Looking further ahead, Lelli said there is a scheme to extend the bandwidth reclaiming mechanism to allow priority demotion. Once a process exceeds its reservation, it will continue to run, but as a normal process without realtime priority. That priority will be restored once the next scheduling period starts. There is also a strong desire to have fully energy-aware scheduling in the deadline scheduler.

A more distant wishlist item is support for single-CPU affinity. The priority inheritance mechanism could also stand some improvements. Currently, a task that blocks a deadline task will inherit that task's deadline. Replacing that with an algorithm like the multiprocessor bandwidth inheritance protocol [PDF] would be desirable. There is also a wish for a dynamic feedback mechanism that could adjust a process's reservation based on its observed needs. But, for the time being, he said, nobody is actually working on these items.

The video of this session is available.

[Thanks to Linaro and the Linux Foundation for funding your editor's travel to Connect.]

Comments (2 posted)

Debian Project Leader election—2017 edition

After last year's anticlimactic Debian Project Leader (DPL) election, where the one and only candidate resoundingly beat "none of the above", this year's edition will be a contest, at least. Two candidates have stepped forward: current DPL Mehdi Dogguy and Chris Lamb. They both have outlined their platforms and the campaigning period has opened. Voting will take place in the first half of April, with results available on April 16.

While it does seem that DPL elections are seeing fewer candidates these days, it is rare to see a one-candidate election. It only ever happened once before, in 2011, when then-DPL Stefano Zacchiroli ran for re-election unopposed. Most other DPL elections have had three or more candidates. The decline in the number of candidates has led at least one Debian developer to wonder whether the project should consider eliminating the DPL position entirely.

Mehdi Dogguy

Dogguy is the incumbent this year and his platform reflects some of the experience he gained during his tenure as DPL. But some of the problems and areas to work on that he called out in 2016 still persist, so some parts of the platform have been recycled from last year's, which he links to. In particular, he points to the difficulty of recruiting more contributors. In both platforms, he notes a lack of documentation as a key contributing factor to the recruitment woes. It will require a multi-year effort to fix that, but he had some success in attracting students' attention to Debian at an event he participated in while DPL. Events like that, along with internship programs like Outreachy, will be helpful to the recruitment efforts and he intends to continue those kinds of activities if re-elected.

In his talk at DebConf16 (and in a related BoF), he has proposed the creation of a roadmap for the distribution to help guide its development:

He intends to spearhead the roadmap effort once the "Stretch" (Debian 9) release is out the door. The plan is to establish SMART criteria to help track the progress of the items on the roadmap.

Dogguy would also like to see the project's fundraising put on a more stable footing by adding a "partners program" that would make it easier to budget funds over longer time frames. These partners would commit to a certain level of support over a few years. In addition, he hopes that the funding provided by the partner companies and organizations will help reduce the fundraising burden for DebConf organizers so they can focus more on the event itself.

He would also like to push forward on the "Bikesheds" project, which has been percolating in Debian circles for some time. It is a way for developers to create their own repositories similar to the Ubuntu personal package archives (PPAs). The name evidently comes from a DebConf15 announcement—complaints have been heard about it—but the idea goes back to a 2013 posting to the debian-devel mailing list.

The project is not well understood within the Debian community, Dogguy said, so he would like to gather the right people for a sprint to fully describe what it is and what is needed to make it happen.

There is more to his platform, of course, but that brushes on the high points. His priorities for his next term if he is elected are: "ensuring the community remains safe and fun", the roadmap, clarifying Bikesheds, and the partners program. In many ways, he is planning to "stay the course" that he set in his current term, though there are some new initiatives he would like to help drive.

Chris Lamb

Lamb's platform begins with a bit of a lament that Debian is not more prominent among distributions: "Although the open source environment is not a zero-sum game, I cannot help feel a pang of regret whenever I hear initiatives — especially headline-grabbing ones — that are based on our derivatives rather than us." He worries that the distribution may be relegated to being a glue layer for containers or Internet of Things devices, which will make it harder to attract contributors.

That said, he believes that "Debian is working incredibly well", but that its image is not what it should be. Some amount of work on marketing the distribution is needed:

Like Dogguy, Lamb is concerned that the documentation for newcomers and potential contributors is lacking. In addition, some of the processes that contributors need to understand and use are seen, negatively, as hurdles and obstacles. It is something of a tragedy, he said, since Debian is at the forefront of various important initiatives such as the reproducible builds effort (which Lamb is active in). But without raising the visibility of Debian somewhat, he fears the distribution will be left behind to some extent. "We do not want to become — or even perceived to be — just a high-quality package repository."

Lamb sets out four concrete steps that he would take as DPL. He would like to organize more in-person gatherings and meetings, such as DebConfs, hackathons, summits, sprints, and bug-squashing parties. These meetings have effects that go well beyond whatever work gets done at the time because "they act as a social lubricant for subsequent online communications that can go awry". He would like to encourage and help organize more such events and to help grow the number and diversity of people who attend them.

The "onboarding" process for new users and developers needs improvement and he would like to make that one of his focuses. Usability testing could be one way to identify where the problems are in that process, as well as identifying areas where new users are stymied in their efforts to actually try Debian out. Creating a Debian-specific outreach initiative along the lines of Outreachy is also something he would like to do. Outreachy has been "incredibly successful", and a similar project within Debian would allow "more flexibility in the manner of contributions as well as underline Debian's dedication to 'universality' in all its forms with the public at large".

His final concrete goal would be to remove blockers to working efficiently on Debian. He would like to use some of the project's "surfeit of funds" to provide better hardware where it is needed, but also to change the culture somewhat. If projects could do their jobs better with some additional resource, he wants them to have "no hesitation in asking for it". He would like to find creative solutions for any social or personality blockers as well.

No DPL?

Before Lamb or Dogguy had nominated themselves, Guillem Jover posted an idea to the debian-project mailing list: not electing a DPL at all. The DPL role is largely symbolic in many ways, so perhaps the project could do without it:

Others in the short thread did not see things quite that way. Adam Borowski called the DPL position "a largely thankless, massive amount of work that's contrary to what Debian is about (code hacking and solving technical problems[1])"—though his footnote also added "flamewars". The title is a perk and removing it might make the job even less appealing. Others more or less agreed.

Referendum

The election is effectively a referendum on Dogguy's term as DPL. There are no huge differences in the approach or ideas that Dogguy and Lamb have laid out, so it may come down to a question of whether the incumbent has done a good (or even reasonable) job over the last year. There don't seem to be any external indications that Debian is unhappy with Dogguy's service, so it would not be a surprise to see him re-elected. In addition, Lamb has not run for the position before and first-time candidates have typically not been elected. But we'll have to wait and see who gets announced as the DPL in mid-April.

Comments (none posted)

Distribution quotes of the week

#                  Cat Supremacy License, version √-1
#
# Everyone who recognizes the superiority of the species Felis catus over
# mere Homo sapiens is hereby gladly granted the right to use, modify,
# distribute, sell, borrow, give, steal, pee on, print out and use as a
# kindle, deface, scribble on, give less, more or same respect as $PRESIDENT
# of $YOUR_COUNTRY gives to the Constitution, etc, and so on, this software;
# with or without modifications, in either a preferred or unpreferred form
# for modification, with or without a fee of any kind.
#
# Those who have yet to mend the error of their ways are also given all of
# the above rights, albeit grudgingly.

Comments (2 posted)

Chakra 2017.03 "Goedel" released

Comments (none posted)

NetBSD 7.1 released

Comments (none posted)

Bits from the DPL -- January-February 2017

Full Story (comments: none)

Ubuntu 12.04 (Precise Pangolin) reaches End of Life on April 28 2017

Full Story (comments: 1)

Ubuntu 12.04 ESM (Extended Security Maintenance)

Full Story (comments: none)

Distribution newsletters

• DistroWatch Weekly (March 13)
• Lunar Linux weekly news (March 10)
• Mageia Weekly roundup (March 10)
• openSUSE news (March 9)
• openSUSE news (March 13)
• Tumbleweed Review of the week (March 11)
• Ubuntu Weekly Newsletter (March 12)

Comments (none posted)

How to build an IoT project with Mongoose OS (Opensource.com)

Comments (none posted)

Red Hat Product Security Risk Report 2016

Comments (none posted)

Delayed execution for Python

A discussion about more efficient debug logging on the python-ideas mailing list quickly morphed into thinking about a more general construct for Python: a delayed evaluation mechanism. The basic problem is that Python will fully evaluate arguments to functions even when they will end up being unused. That is always inefficient but may not seriously affect performance except for expensive argument calculations, where that inefficiency can really add up.

The discussion started with a post from Barry Scott, who was looking for a way to avoid the costly evaluation of arguments to a debug logging routine. The value of the arguments would never be used because debugging was disabled. He had code something like the following:

    debug = False

    def debuglog(msg):
        if debug:
	    print('Debug: %s' % (msg,))

    debuglog('Info: %s' % (expensive(),))

A way to avoid the call to expensive(), since the value would never be used, is what Scott was seeking. Several people suggested using lambda to create a "callable" object and to pass that to debuglog():

    def debuglog(msg):
        if debug:
	    if callable(msg):
	        msg = msg()
            print('Debug: %s' % (msg,))

    debuglog(lambda: 'Info: %s' % (expensive(),))

Python's lambda expressions create a temporary function (i.e. a callable object), so the effect here is to pass the string as a function. It will only be evaluated in debuglog() if debugging messages are enabled, thus avoiding the expensive() call when it is not needed.

Victor Stinner suggested using a preprocessor to remove the debug code for production as PEP 511 (which he authored) would allow if it gets accepted. But Marc-Andre Lemburg said: "preprocessors are evil, let's please not have them in Python :-)". Instead, he recommended using the __debug__ flag and the -0 command-line option for production, which will cause Python to not generate any code for if __debug__: blocks. It will, however, also eliminate assert statements, which Stinner saw as a flaw to that plan.

Meanwhile, Steven D'Aprano noted that he has sometimes thought that Python should have some kind of delayed (or lazy) evaluation construct. For example purposes, he used a construct like "<#expensive()#>" as a way to create a "lightweight 'thunk' for delayed evaluation". Python does not have thunks, however; D'Aprano believes that adding them would help solve the delayed/lazy evaluation problem:

That thread led Joseph Hackman to propose a new delayed keyword to denote expressions that should be lazily evaluated. Here is an example of how the new keyword would be used:

    debuglog('Info: %s', (delayed: expensive(),))

Hackman went on to describe how he envisioned it working:

Joseph Jevnik pointed to the lazy module, which uses a decorator to identify functions that should be lazily evaluated. David Mertz also mentioned the Dask parallel computing library, which has a delayed() function to indicate deferred evaluation. In the earlier thread, comparisons were made to generators and Future objects, as well. Some of those might serve as inspiration for implementing lazy evaluation in the core language.

One problem with the proposal was pointed out by D'Aprano: adding new keywords to Python (or any language) is difficult. They are not backward compatible and will break working code. Something like "delayed" is quite likely to have been used by vast numbers of programs as a variable or function name, for example. He also had questions about the semantics of delayed, especially with regard to exactly what would actually trigger the evaluation; that is, when does the value become concrete?

D'Aprano and Chris Angelico asked some questions about how it would all work. Hackman would like to see it remain simple:

Angelico asked about putting delayed objects into collections (e.g. lists, dictionaries); would that cause the object to be evaluated or not? It is an important question, since variable-length argument lists and keyword arguments are both packaged into collections before being passed to the function. But Hackman thinks it should all work out (with "some TLC") because the actual, concrete values are not required to put objects into lists or as values in dictionaries.

There was some inevitable bikeshedding about names as well. Abe Dillon suggested "lazy" as a shorter keyword. Joshua Morton did some research on GitHub repositories and found that "deferred" seems to be the least used among the terms he tried ("delay, defer, lazy, delayed, deferred, etc."), but it still appears 350,000 times. There was also discussion of whether the colon was needed or desired, without any clear-cut consensus.

In a third thread, Michel Desmoulin described another use case for lazy (or delayed). He has some code that is similar to the following:

    val = conf.get('setting', load_from_db('setting'))

    val = conf.get('setting', lazy load_from_db('setting'))

That, of course, led to ideas for other ways to approach the problem, but they suffered from a few disadvantages. One could make conf a subclass of dict and define a __missing__() method to call load_from_db(), as Angelico suggested. Or create a different kind of dict subclass, as Markus Meskanen proposed. Both of those make some assumptions as Desmoulin pointed out:

He went on to outline a couple other features that are not "needed" in Python (e.g. list comprehensions), but he conceded that "you don't need lazy, it's just convenient". He further expanded on that in another post, showing the reasons he thinks that the other solutions, while potentially workable in some cases, are not particularly convenient or elegant:

Hackman plans to write up a PEP for lazy, though nothing has been posted yet. But Lemburg is unconvinced that the feature can be specified well enough to be useful:

In the end, any new keyword is going to be a hard sell. Some other syntax might make the feature more palatable, but BDFL Guido van Rossum is notoriously averse to operators that seem "Perl-esque", so that will need to be avoided as well. The lazy feature certainly garnered some support in the threads, but few core developers participated, so it is hard to know how the PEP might be received.

Comments (17 posted)

Development quotes of the week

Comments (none posted)

LLVM 4.0.0 released

Full Story (comments: 14)

MATE 1.18 released

Comments (5 posted)

SciPy 0.19.0

Full Story (comments: none)

U-Boot v2017.03 is released

Full Story (comments: none)

Development newsletters

• Emacs news (March 13)
• These Weeks in Firefox (March 14)
• What's cooking in git.git (March 8)
• What's cooking in git.git (March 10)
• What's cooking in git.git (March 13)
• What's cooking in git.git (March 14)
• Git Rev News (March 15)
• OCaml Weekly News (March 14)
• Perl Weekly (March 13)
• PostgreSQL Weekly News (March 12)
• Python Weekly (March 9)
• Ruby Weekly (March 9)
• This Week in Rust (March 14)
• Wikimedia Tech News (March 13)

Comments (none posted)

Haas: Parallel Query v2

Comments (2 posted)

GNU Toolchain now accepting donations with the support of the Free Software Foundation

Comments (none posted)

FSFE Newsletter - March 2017

Full Story (comments: none)

FSF: Do GitHub's updated terms of service conflict with copyleft?

Comments (none posted)

Three challenges for the web, according to its inventor

Comments (87 posted)

13th Netfilter Workshop

Full Story (comments: none)

CFP Deadlines: March 16, 2017 to May 15, 2017

If the CFP deadline for your event does not appear here, please tell us about it.

Schedule posted: Explore the roots of freedom at LibrePlanet 2017

Comments (none posted)

Netdev Conference: What you have been missing

Full Story (comments: none)

Events: March 16, 2017 to May 15, 2017

If your event does not appear here, please tell us about it.