Welcome to the LWN.net Weekly Edition for September 21, 2017

This edition contains the following feature content, most of which comes from the Open Source Summit North America and Linux Plumbers Conference events:

• Linking commits to reviews: A new tool to locate email discussions relevant to specific kernel commits.
• Building the kernel with clang: The ability to build the kernel with the LLVM clang compiler has been a wishlist item for years; it is getting closer to reality.
• Building an ARM64 laptop: Why would one want to build a laptop around an ARM64 processor, and how would one go about doing it?
• The rest of the 4.14 merge window: The 4.14 merge window closed on September 16; here's a summary of the last set of changes to be merged.
• Notes from the LPC scheduler microconference: several sessions on various aspects of CPU-scheduler development.
• Testing kernels: How can the testing of pre-release kernels be improved?

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Linking commits to reviews

In a talk in the refereed track of the 2017 Linux Plumbers Conference, Alexandre Courouble presented the email2git tool that links kernel commits to their review discussion on the mailing lists. Email2git is a plugin for cregit, which implements token-level history for a Git repository; we covered a talk on cregit just over one year ago. Email2git combines cregit with Patchwork to link the commit to a patch and its discussion threads from any of the mailing lists that are scanned by patchwork.kernel.org. The result is a way to easily find the discussion that led to a piece of code—or even just a token—changing in the kernel source tree.

Courouble began with a short demo of the tool. It can be accessed by typing (or pasting) in a commit ID on this web page, which brings up a list of postings of the patch to various mailing lists; following those links shows the thread where it was posted (and, often, discussed). Another way to get there is to use cregit; navigating to a particular file then clicking on a token will bring up a similar list that relates to the patch where the symbol was changed. Note that the Patchwork data only goes back to 2009, so commits before that time will not produce any results.

So, email2git allows those interested to get a look into the design decisions that went into a particular chunk of code. Without it, doing so manually is not particularly easy. There are several use cases that he presented, starting with security researchers, who want to understand the thinking when a patch was made. It can also be used in bug fixing and by newcomers to the kernel community. In addition, email2git is being used as part of a recently announced Linux Foundation project: Community Health Analytics Open Source Software (CHAOSS).

Email2git takes commits from the mainline Git repository and tries to match them up to patches that Patchwork has picked up. Patchwork scans around 70 mailing lists to extract patches and the discussion threads that follow. It provides a user-friendly online interface, though email2git accesses the Patchwork database directly. Cregit is used to find changes at the token level, which is more accurate than git blame, he said.

There is not any kind of direct mapping from patches posted on a mailing list to commits in the mainline Git tree, so email2git needs to find those matches. Initially, he used a method from some research papers that effectively did an exhaustive search comparing the diff output in a commit to that in all of the posted patches until a match is found. That did not scale once he started working with the entire data set, which is some 500K mainline commits and 1.4 million patches posted since 2009. Some kind of heuristics were needed to narrow down the search space.

Courouble ended using three pieces of the patch to match them to a Git commit. The first is the subject of the email, which is often carried over into the Git commit summary. That heuristic alone finds 55% of the commits directly. Step two is to look at the patch author; he has created a map of all patches from a given author, so those can be tried next. The third heuristic is to match up the files that are affected by the patches. Each of the last two steps does comparisons of the diff in the patch and commit to make a matching decision.

The results for different kernel directories vary fairly widely. Some, like mm, kernel, tools, and virt can match 60-90% of the commits. Those are likely to be subtrees where the email subject winds up in the commit, he said. On the other end of the scale, the net directory has less than 30% matches; it turns out that Patchwork does not track the relevant mailing list. Other subtrees fall somewhere in between.

There are a number of limitations of the current system. It is only tracking the mainline for one thing, there may be other trees of interest. It is dependent on Patchwork, which is a great resource but only goes back to 2009, so some data is missing. The mbox format of the data can be inconsistent; there are patches with dates from 1970 and 2040, for example.

For the future, the plan is to make the match data available through other means, such as via a REST interface. In addition, running an instance of Patchwork in-house would allow extracting more data from other lists and perhaps going further back in time. Adding tracking for linux-next, improving the algorithm to do incremental processing, and handling patch series that are discussed in multiple threads are all on the radar. Courouble is also interested in getting feedback and ideas for other features from kernel developers.

A few of those were offered up in the Q&A. Someone suggested that it would be nice to get the "0/X" patch associated with the thread. Courouble seemed surprised to hear that Patchwork did not track those, but thought it should be added. There were also suggestions that providing guidelines on how patches move from the mailing list into the Git repositories so that they can be more easily tracked or perhaps adding Git patch IDs into the mix might help.

[I would like to thank LWN's travel sponsor, The Linux Foundation, for assistance in traveling to Los Angeles for LPC.]

Comments (4 posted)

Building the kernel with Clang

Over the years, there has been a persistent effort to build the Linux kernel using the Clang C compiler that is part of the LLVM project. We last looked in on the effort in a report from the LLVM microconference at the 2015 Linux Plumbers Conference (LPC), but we have followed it before that as well. At this year's LPC, two Google kernel engineers, Greg Hackmann and Nick Desaulniers, came to the Android microconference to update the status; at this point, it is possible to build two long-term support kernels (4.4 and 4.9) with Clang.

Desaulniers began the presentation by answering the most commonly asked question: why build the kernel with Clang? To start with, the Android user space is all built with Clang these days, so Google would like to reduce the number of toolchains it needs to support. He acknowledged that it is really only a benefit to Google and is "not super useful" elsewhere. But there are other reasons that are beneficial to the wider community.

There are some common bugs that often pop up in kernel code, especially out-of-tree code like the third-party drivers that end up in Android devices. The developers are interested in using the static analysis available in Clang to spot those bugs, but the kernel needs to be built using Clang to do so. There are also a number of dynamic-analysis tools that can be used like the various sanitizers (e.g. AddressSanitizer or ASan) and their kernel equivalents (e.g. KernelAddressSanitizer or KASAN).

Clang provides a different set of warnings than GCC does; looking at those will result in higher quality code. It is clearly beneficial to all kernel users to have fewer bugs in it. There are some additional tools that are planned using Clang. One is a control-flow-analysis tool that could enumerate valid stack frames at compile time; those could be checked at run time to eliminate return-oriented programming (ROP) attacks. There is also work going on for link-time optimization (LTO) and profile-guided optimization (PGO) for Clang, which could provide better execution speed, especially for hot paths.

Building code with another compiler is a good way to shake out code that relies on undefined behaviors. Since the language specification does not define certain behaviors, compiler developers can choose whatever is convenient. That choice could change, so even a GCC upgrade might cause misbehavior if some kernel code is relying on undefined behavior. The hope, Desaulniers said, is that both the kernel and LLVM/Clang can improve their code bases from this effort. The kernel is a big project with a lot of code that can find bugs in the compiler; in fact, it already has.

Greg Kroah-Hartman said that "competition is good"; he was strongly in favor of the effort. Desaulniers was glad to hear that as he and others were worried that the tight coupling with GCC was being protected by the kernel developers. Kroah-Hartman said that there have been other compilers building the kernel along the way. Behan Webster also pointed to all of the new features that have come about in GCC over the past five years as a result of the competition with LLVM. Kroah-Hartman said that he wished there was a competitor to the Linux kernel.

Hackmann related the state of the upstream kernel: "we are very close to having a kernel that can be built with Clang". It does require using a recent Clang that has some fixes, but the x86_64 and ARM64 kernels can be built, though each architecture has one out-of-tree patch that needs to be applied to do so. There is also one Android-specific Kbuild change that is needed, but only if the Android open-source project (AOSP) pre-built toolchain is being used.

As announced on the kernel mailing list, there are patches available for the 4.4 and 4.9 kernels. There are also experimental branches of the Android kernels for 4.4 and 4.9 available from AOSP. More details can be found in the slides [PDF]. Those branches had just been pushed a few days earlier, Hackmann said, and the HiKey boards were able to build and boot that code shortly thereafter.

There have been LLVM bugs found in the process, though most of them have been fixed at this point, Desaulniers said. The initial work was done with LLVM 4.0, but they have since updated to 5.0 and are also building with the current LLVM development tree (which will become 6.0). You can probably build the kernel with 4.0, he said, but it will be much slower than building with 5.0 or later.

There are still some outstanding issues. Variable-length arrays as non-terminal fields in structures are not supported by Clang, there is a GNU C extension for inline functions that is not supported, and the LLVM assembler cannot be used to build the kernel. Hackmann noted that the GNU assembler is too liberal in what it accepts.

This work has shown that the FUD surrounding using a new toolchain for the kernel is unfounded, Desaulniers said. It is working now, but there are a few asterisks. Clang, the front end, can compile the kernel, but the assembler and the linker from GNU Binutils are needed to complete the build process.

Next up is figuring out how to do automated testing of LLVM and the kernel. Currently, the team is working with two specific LTS kernel branches and using specific LLVM versions. So he can't quite say that Clang will build any kernel, since there are so many different configuration options. A bot to check whether kernel patches will fail to build under Clang is in the works as well. An audience member noted that kernelci.org is looking at adding other compilers to its build-and-boot testing.

Hackmann and Desaulniers encouraged others to try building using Clang. All it takes is a simple "make CC=clang" on a properly equipped system. We are, it seems, quite close to having a two-compiler world for the Linux kernel.

[I would like to thank LWN's travel sponsor, The Linux Foundation, for assistance in traveling to Los Angeles for LPC.]

Comments (83 posted)

Building an ARM64 laptop

Processors based on the 64-bit ARM architecture have been finding their way into various types of systems, including mobile handsets and servers. There is a distinct gap in the middle of the range, though: there are no ARM64 laptops. Bernhard Rosenkränzer and a group of colleagues set out to change that situation by building such a laptop from available components. He showed up at the 2017 Open Source Summit North America to present the result.

He started by addressing the question of why one would want to build an ARM64 laptop in the first place. The ARM architecture is known for low power use — a useful feature in a laptop in its own right — but there is more to the ARM story than that; the ARM64 chips are fast and can beat single-core Intel Core-M processors on some benchmarks. An ARM64 laptop may not be good for fast kernel builds, but it can do what most people need, and it can do the kernel builds too in the end. ARM processors need no fans, meaning that the resulting laptop is lighter and will not burn the user's legs. There is little or no malware targeting ARM64 systems, for now at least.

Some of the motivations for this device may not resonate for people outside of the ARM ecosystem; Rosenkränzer talked about the ability to do native ARM64 builds rather than cross-compiling on an Intel system, for example. This system is also good for testing ARM64 hardware in the desktop setting — something that isn't really happening. An advantage that many users will appreciate, instead, is the potential to build a more open system. ARM64-based systems don't need any "secret BIOS" or mysterious management engines. That potential isn't fully realized yet, though, because a binary-only bootloader is still needed.

Given that there is value in an ARM64 laptop, he asked, why is nobody doing it? He didn't really answer the question, but did note that there are a couple of Chromebooks based on ARM64 chips. They are a good start, but Chromebooks are not a good substitute for a general-purpose desktop system. Their storage is limited, there is no SATA port, the keyboard is minimal, and the 12-inch screen is small. The pi-top is an interesting system, great for embedded use, but is also not well suited to desktop use.

It would be nice to avoid building an entire laptop from scratch, so the idea of replacing the insides of an x86-based system with an ARM64 board has some appeal. It can be done, he said, but this method doesn't scale. There is also the problem that no two laptops are the same inside, so this solution would be tied to a specific laptop model.

In the end, they went back to the pi-top, which does have some advantages. It's not bolted shut, so it's easy to modify. The display is connected via HDMI, and the touchpad is a USB device, meaning that they can be plugged into a different board with relatively little trouble. So could the Raspberry Pi processor in the pi-top be replaced with something more powerful? The answer is "yes", with the proviso that the power supply must be modified to be able to drive a more power-hungry processor. They used a Mediatek X20 board to build a proof-of-concept laptop. It works, but it still lacks adequate storage and has too little RAM.

To do this right, he said, requires a board with a fast ARM64 processor, at least an A72. Naturally, it should have a GPU that does not require binary-only drivers, so no Mali or PowerVR graphics. It needs a SATA or M.2 port so it can provide a decent amount of storage, and enough USB ports that some of them can be made available to the user. Lots of RAM is needed, HDMI output would be nice. And, of course, the board providing all of this needs to be affordable.

Such a board is not really available at the moment. The 96Boards systems are short on storage options. Only the DragonBoard 410c has open graphics, but lacks RAM and has an underpowered (for this task) CPU. The Tegra boards are more capable, but they are expensive and he does not trust NVIDIA to keep them open. The Banana Pi and ODROID boards lack performance. The i.MX8 might be interesting, he said; it has a Vivante GPU that can be driven by the free etnaviv driver, but it's not yet available.

An alternative might be to use a networking board; the MacchiatoBIN has some appeal, for example. It has a quad A72 processor, DDR4 DIMM memory, three SATA connectors, and a PCIe port that can be used for a graphics card. Unfortunately, that PCIe port lacks an aperture large enough to run a proper graphics adapter, so it won't work either.

Finally, though, they found the OpenQ 820. It has a four-core CPU, an Adreno 530 GPU that is "mostly supported" by the freedreno driver, 3GB of RAM, 32GB of UFS storage to hold the operating system, and two PCIe slots. Naturally, there is a snag: it only runs Android and doesn't support "real" Linux. But, Rosenkränzer said, they don't plan to use any of the Android binary-only drivers, so perhaps it could be made to work.

Getting there took a bit of effort. The bootloader for this system wants Android sparse filesystem images; fortunately, the Android Open Source Project has the tools needed to create those. In particular, its make_ext4fs tool can be made to do the job after a few patches are applied; the modified version can be found on GitHub. With that work done, they have an ARM64-based laptop that can boot and run. After fixing a few freedreno bugs, they have KDE running on it, and all of the operating-system packages have been ported. Battery support is still missing, so the laptop must stay plugged in, but that support is almost ready.

As always, there is more to be done. They would like to get away from using the pi-top HDMI interface; if they switched to MIPI the system would use less power, provide higher resolution, and have the HDMI port available for external connections. The problem is that MIPI displays only come in small sizes for now. They would also like to have a better case; the board doesn't quite fit in the pi-top case. Manufacturers are not cooperating, though; they want million-unit volume before they put any effort into it.

The cost of the system is about $600; there are hopes that it can be lowered in the future. Rosenkränzer also hopes that future boards will work better for this application, though he couldn't talk about any specific possibilities on the horizon. He concluded by saying that it's always likely to be some work to make a system. Nobody is trying to build commercial ARM64-based laptops, so it is unlikely that any boards oriented toward that use will appear in the market.

[Thanks to the Linux Foundation, LWN's travel sponsor, for supporting your editor's travel to the Open Source Summit.]

Comments (19 posted)

The rest of the 4.14 merge window

As is sometimes his way, Linus Torvalds released 4.14-rc1 and closed the merge window one day earlier than some might have expected. By the time, though, 11,556 non-merge changesets had found their way into the mainline repository, so there is no shortage of material for this release. Around 3,500 of those changes were pulled after the previous 4.14 merge-window summary; read on for an overview of what was in that last set.

User-visible changes include:

• The disk quota subsystem has received a fair amount of performance work, increasing file-creation performance in ext4 (with quotas enabled) by a factor of about two.

• The CIFS filesystem can now read and write extended attributes when using the SMB3 protocol.

• The heterogeneous memory management patch set has finally been merged. HMM exists to enable the use of devices with their own memory-management units that can access main memory. See this changelog for an overview of the current state of this work.

• The namespaced file capabilities patch has been merged. This allows file capabilities to be used within user namespaces. The scheme has been simplified somewhat, though, so that only a single security.capability extended attribute can exist for any given file.

• The kernel has gained support for the zstd compression algorithm; it claims better performance and a better compression ratio. See this changelog for an overview.

• The new IOCB_NOWAIT flag can be used to specify that asynchronous buffered block I/O operations should be as close to non-blocking as possible. Without this flag (or in current kernels) these operations can block on memory-management operations, for example.

• The entire firmware subtree has been removed from the kernel repository. The firmware that everybody actually uses has been maintained in its own repository for some years now, so these files were unused and unneeded.

• New hardware support includes: Rockchip RK805 power-management chips, STMicroelectronics STM32 low-power timers, ROHM BD9571MWV power-management chips, TI TPS68470 power-management chips, Altera/Intel mSGDMA engines, Mediatek MT2712 and MT7622 PCI host controllers, Spreadtrum SC9860 I2C controllers, NXP i.MX6/7 remote processors, Maxim MAX17211/MAX17215 fuel gauges, Intel PCH/PCU SPI controllers, Qualcomm "B" family I/O memory-management units, Synopsys HSDK reset and clock controllers, ZTE ZX pulse-width modulators, Socionext UniPhier thermal monitoring units, Realtek RTD129x realtime clocks, Synopsys AXS10X SDP Generic PLL clocks, Renesas R-Car USB 2.0 clocks, Allwinner R40, A10, and A20 clock controllers, STMicroelectronics STM32H7 reset and clock controllers, Altera Soft IP I2C interfaces, STMicroelectronics STM32F7 I2C controllers, Lantiq XWAY SoC RCU reset controllers, Lantiq XWAY SoC RCU USB 1.1/2.0 PHYs, and PWM-controlled vibrators.

Changes visible to kernel developers include:

• The structure-layout randomization plugin for GCC will now automatically rearrange the members of structures that consist entirely of function pointers.

• The build system has long used gperf to generate hashes. As of 4.14, though, that is no longer true. The gperf 3.1 broke that generation and, rather than fix it in a version-independent way, Linus simply ripped it out.

• The new memset16(), memset32(), and memset64() functions can be used to set a range of memory to an integer value.

• dma_alloc_noncoherent() has been removed; dma_alloc_attrs() should be used instead.

Now it is just a matter of finding and fixing the various bugs that were inevitably introduced with all of those changes. If that process goes well and follows the usual schedule, the final 4.14 release can be expected on November 5 or 12.

Comments (12 posted)

Notes from the LPC scheduler microconference

The scheduler workloads microconference at the 2017 Linux Plumbers Conference covered several aspects of the kernel's CPU scheduler. While workloads were on the agenda, so were a rework of the realtime scheduler's push/pull mechanism, a distinctly different approach to multi-core scheduling, and the use of tracing for workload simulation and analysis. As the following summary shows, CPU scheduling has not yet reached a point where all of the important questions have been answered.

Workload simulation

First up was Juri Lelli, who talked about the rt-app tool that tests the scheduler by simulating specific types of workloads. Work on rt-app has been progressing; recent additions include the ability to model mutexes, condition variables, memory-mapped I/O, periodic workloads, and more. There is a new JSON grammar that can be used to specify each task's behavior. Rt-app now generates a log file for each task, making it easy to extract statistics on what happened during a test run; for example, it can be used to watch how the CPU-frequency governor reacts when a "small" task suddenly starts requiring a lot more CPU time. Developers have been using it to see how scheduler patches change the behavior of the system; it is a sort of unit test for the scheduler.

In summary, he said, rt-app has become a flexible tool for the simulation of many types of workloads. The actual workloads are generated from traces taken when running the use cases of interest. It is possible (and done), he said, to compare the simulated runs with the original traces. The rt-app repository itself has a set of relatively simple workloads, modeling browser and audio playback workloads, for example. New workloads are usually created when somebody finds something that doesn't work well and wants a way to reproduce the problem.

Rafael Wysocki said that he often has trouble with kernel changes that affect CPU-frequency behavior indirectly. He would like to have a way to evaluate patches for this kind of unwanted side effect, preferably while patches are sitting in linux-next at the latest and are relatively easy to fix. Might it be possible to use rt-app for this kind of testing? Josef Bacik added that his group (at Facebook) is constantly fixing issues that come up with each new kernel; it gets tiresome, so he, too, would like to find and fix these problems earlier.

He went on to state that everybody seems to agree that rt-app is the tool for this job. There would be value in having all developers pool their workloads into a comprehensive battery of tests, but where is the right place to put them? The rt-app project itself isn't necessarily the right place for all these workloads, so it would seem that a different repository is indicated. The LISA project was suggested as one possible home. Steve Rostedt, however, suggested that these workloads could just go into the kernel tree directly. Lelli asked whether having rt-app as an external dependency would be acceptable; Rostedt said that it would.

Wysocki wondered about how the community could ensure that these workloads get run on a wide variety of hardware; it's not possible to emulate everything. Bacik replied that it's not possible to test everything either, and that attempts to do so would be a case of the perfect being the enemy of the good. If each interested developer runs the tests on their own system, the overall coverage will be good enough. It's an approach that works out well for the xfstests filesystem-testing suite, he said.

Peter Zijlstra complained that rt-app doesn't produce any direct feedback — there is no result saying whether the right thing happened or not. As a result, interpreting rt-app output "requires having a brain". Rostedt suggested adding a feature to compare runs against previous output and look for regressions. Patrick Bellasi noted, though, that to work well in this mode, the tests need to be fully reproducible; that requires care in setting up the tests. At this point, he said, rt-app is not a continuous-integration tool, but it could maybe be evolved in that direction.

Reworking push/pull

Rostedt gave a brief presentation on what he called a "first-world problem". When running realtime workloads on systems with a large number (over 60) of CPUs — something he said people actually want to do — realtime tasks do not always migrate between CPUs in an optimal manner. That is due to shortcomings in how that migration is handled.

When the last running realtime task on any given CPU goes idle, he said, the CPU will examine the system to see if any other CPUs are overloaded with realtime tasks. If it finds a CPU running more than one realtime process, it will pull one of those processes over and run it locally. Once upon a time, this examination was done by grabbing locks on the remote CPUs, but that does not scale well. Now, instead, the idle CPU will send an inter-processor interrupt (IPI) to tell other CPUs that they can push an extra realtime task in its direction.

That is an improvement, but imagine a situation with many CPUs, one of which is overloaded. All of the other CPUs go idle more-or-less simultaneously, which does happen at times. They will all send IPIs to the busy CPU. One of them will get the extra process from that CPU but, having pushed that process away, the CPU still has to process a pile of useless IPIs. That adds extra load to the one CPU in the system that was already known to be overloaded, leading to observed high latencies — the one thing a realtime system is supposed to avoid at all costs.

The proposed solution is to have only the first (for some definition of "first") idle CPU send the IPI. That IPI can then be forwarded on to other overloaded CPUs if needed. The result would be the elimination of the IPI storms. Nobody seemed to think that this was a bad solution. It was noted that it would be nice to have statistics indicating how well this mechanism is working, and the conversation devolved quickly into a discussion of tracepoints (or the lack thereof) in the scheduler code. Zijlstra said that he broke user space once as a result of tracepoints, so he will not allow the addition of any more. This is a topic that was to return toward the end of the session.

Multi-core scheduling

Things took a different turn when Jean-Pierre Lozi stood up to talk about multi-core scheduling issues. Lozi is the lead author of the paper called The Linux Scheduler: a Decade of Wasted Cores [PDF], which described a number of issues with the CPU scheduler. An attempt to go through a list of those bugs drew a strong response from Zijlstra, who claimed that most of them were fixed some time ago.

The biggest potential exception is "work conservation" — ensuring that no task languishes in a CPU run queue if there are idle CPUs available elsewhere in the system. On larger systems, CPUs will indeed sit idle while tasks wait, and Zijlstra said that things will stay that way. When there are a lot of cores, he said, ensuring perfect work conservation is unacceptably expensive; it requires the maintenance of a global state and simply does not scale. Any cure would be worse than the disease.

Lozi's proposed solution is partitioning the system, essentially turning it into a set of smaller systems that can be scheduled independently. Zijlstra expressed skepticism that such an idea would be accepted, leading to a suggestion from Lozi that this work may not be intended for the mainline kernel. There was a quick and contentious discussion on the wisdom of the idea and whether it could already be done using the existing CPU-isolation mechanisms. Mathieu Desnoyers eventually intervened to pull the discussion back to its original focus: a proposal to allow the creation of custom schedulers in the kernel. This work is based on Bossa, which was originally developed some ten years ago. It uses a domain-specific language to describe a scheduler and how its decisions will be made; different partitions in the system could then run different scheduling algorithms adapted to their specific tasks.

It was pointed out that the idea of enabling loadable schedulers was firmly shot down quite a few years ago. Even so, there was a brief discussion on how they could be enabled. The likely approach would be to create a new scheduler class that would sit below the realtime classes on the priority scale, but above the SCHED_OTHER class where most work runs. The discussion ran out of time, but it seems likely that this idea will eventually return; stocking up on popcorn for that event might be advisable. Zijlstra, meanwhile, insists that the incident with the throwable microphone was entirely accidental.

Workload analysis with tracing

Desnoyers started the final topic of the session by stating that there is a real need for better instrumentation of the scheduler so that its decisions can be understood and improved. He would like exact information on process switches, wakeups, priority changes, etc. It is important to find a way to add this information without creating ABI issues that could impede future scheduler development. That means that the events have to be created in a way that allows them to evolve without breaking user-space tools.

Zijlstra said that it would not be possible to add all of the desired information even to the existing tracepoints; that would bloat them too much. Desnoyers suggested adding a new version file to each tracepoint; an application could write "2" to it to get a new, more complete output format. Rostedt complained about the use of version numbers, though, and suggested writing a descriptive string to the existing format file instead. Zijlstra said that tracepoints should default to the newest format, but Rostedt said that would break existing tools. The only workable way is to default to the old format and allow aware tools to request a change.

That was about the point where Zijlstra (semi-jokingly) declared his intent to remove all of the tracepoints from the scheduler. "I didn't want this pony", he said.

There was a wandering discussion on how it might be possible to design a mechanism that resembles tracepoints, but which is not subject to the kernel's normal ABI guarantees. Bacik said that a lot of the existing scripts in this area use kprobes; they are simply updated whenever they break. Perhaps a new sort of "tracehook" could be defined that resembles a kprobe: it is a place to hook into a given spot in the code, but without any sort of predefined output. A function could be attached to that hook to create tracepoint-style output; it could be located in a separate kernel module or be loaded as an eBPF script. Either way, that glue code would be treated like an ordinary kernel module; it is using an internal API and, as a result, must adapt if that API changes.

The developers in the room seemed to like this idea, suggesting even that it might be a way to finally get some tracepoints into the virtual filesystem layer — a part of the kernel that does not allow tracepoints at all. Your editor was not convinced, though, that the ABI issues could be so easily dodged. As the session ended, it was resolved that the idea would be presented to Linus Torvalds for a definitive opinion, most likely during the Maintainers Summit in October.

[Your editor would like to thank the Linux Foundation, LWN's travel sponsor, for supporting his travel to LPC 2017].

Comments (10 posted)

Testing kernels

New kernels are released regularly, but it is not entirely clear how much in-depth testing they are actually getting. Even the mainline kernel may not be getting enough of the right kind of testing. That was the topic for a "birds of a feather" (BoF) meeting at this year's Linux Plumbers Conference (LPC) held in mid-September in Los Angeles, CA. Dhaval Giani and Sasha Levin organized the BoF as a prelude to the Testing and Fuzzing microconference they were leading the next day.

There were representatives from most of the major Linux distributors present in the room. Giani started things off by asking how much testing is being done on the stable kernels by distributors. Are they simply testing their own kernels and the backports of security and other fixes that come from the stable kernels? Beyond the semi-joking suggestion that testing is left to users, most present thought that there was little or no testing (beyond simple build-and-boot testing) of the stable kernels.

Part of the problem is that it is difficult to know what to choose in order to test "the upstream kernel". The linux-next tree is a moving target as are the stable trees (and the mainline itself). But there is value in finding bugs before they make it into a release. In order to try to find bugs before they actually get into releases, some distributors are starting to test the ‑rc1 kernels. That way, if bugs are found, they can be fixed before the release, but it takes a lot of hours and machines to do that well. There is also a question of which kernel configurations to test.

The upstream testing that is done for the mainline and stable kernels is fairly limited. There is a lot of it being done, but it doesn't go all that deep into kernel functionality. It takes Red Hat a year to stabilize the kernel chosen for a RHEL release; roughly 300 engineers work on that task, meaning it takes the company 300 person-years to test and harden a kernel.

Boot testing is well covered by various upstream testing efforts, so newly released kernels will boot. The majority of bugs that are found in those (or any other) kernel are in the drivers; the only people testing the drivers are those that have the hardware. The core itself is pretty safe, it is believed, and things like Ftrace, the scheduler, and memory management are used widely, so they get a fair amount of testing. Other, non-core or less popular functionality may not see much functional testing.

Red Hat has a large testing lab with something like 6000 machines of various sorts all over the world. It uses Beaker and tests lots of different kernel configurations. It currently runs tests on three RHEL kernels and two Fedora kernels, though there are plans to add the mainline releases. Different teams focus on drivers specific to their area of interest, so the storage teams test various storage devices, while the RDMA team tests that type of hardware.

The main problem is that it takes a lot of effort to analyze the bugs found with the tests. Any crashes that happen could simply be posted to the kernel mailing list as regressions, as was suggested, but even that takes some amount of triage and requires reducing the code to a reproducible test case.

Others who might want to test the drivers may be stymied by the lack of availability (or the cost) of the hardware. It also requires a good understanding of exactly what the device is supposed to do. Ideally, the driver writers would be testing the devices—generally they do—but even that is not a complete solution. Driver writers try to make sure the driver works for their use cases, but they have differing motivations depending on whether they are being paid to write it or simply doing so to support hardware they have, often with little or no documentation.

There are also performance regressions that need to be found in new kernels. That is a difficult problem to solve since "random performance testing" does not really help. There is a need to put together some guidelines for performance tests, so that an apples-to-apples comparison can be done.

The biggest problem for all of these testing efforts is resources. More people and more machines are needed in order to find and fix the bugs sooner.

Stable

The conversation then turned toward the stable kernels. There is a need to stop bugs from entering the kernel; if the mainline is perfect, there would be no need for the stable trees. Perfection is not possible, of course, but do the distributions even use the stable trees?

It turns out that Red Hat only cherry-picks fixes from the stable trees. Each minor release of RHEL has 8-10 thousand patches on top of its kernel, all of which have come from upstream. The RHEL kernel team looks at the stable trees and the latest mainline kernel to find fixes that should be applied. The amount of testing done varies based on which subsystem the patch applies to; some subsystems have a good track record on providing working patches, others less so. Generally, Red Hat only builds the stable kernels to test them against the RHEL kernel to see if a bug is from upstream or was introduced in RHEL.

SUSE does build the stable kernels, but also cherry-picks patches for inclusion. Stable kernel testing could be added to its testing grid, but it is not clear what value that would have for the company. Ubuntu is similar; other than building the stable kernels, there is no formal testing of them.

So the distributions generally care that the fixes that go into stable are correct, but they are testing them in their own kernels. It was suggested that perhaps a collaborative project could be put together by the Linux Foundation, in cooperation with Canonical, SUSE, Red Hat, and others, to put together a set of machines with a test suite to do testing for the stable series.

Linaro is currently working on a project for Google to test the stable kernels using the kernel self-tests (kselftest) and tests from the Linux Test Project (LTP). Those tests are run for every stable release. The self-tests do find bugs, but those who are writing self-tests are probably not the ones introducing the majority of the bugs. The self-tests are just the starting point, however, Linaro intends to add more tests.

One of the difficulties is the huge number of configurations. When a stable kernel is released, it might have 100 patches, but any test suite may not exercise many, or even any, of those fixes. There is a real question of what it means to test a stable release.

The 0-Day kernel test service is also doing more than just build-and-boot tests, including performance testing. The kernelci.org project is doing build-and-boot testing on lots of different hardware, which is quite valuable, but it doesn't do any real functional testing. Things are certainly getting better, overall, and what is there now is "surprisingly better than nothing", one attendee joked.

The self-tests are typically written by kernel developers, but it takes time and effort to turn personal tests into something that can be used more widely. Drivers generally do not have self-tests, because the driver writer didn't have any time to add one. In many cases, the code is of low quality because of that, as well. So the existing self-tests are likely to be in subsystems that are already well tested, but they have found bugs on architectures that are different from what the developer normally runs. Various ARM bugs have been found that way.

LTP tests many things, but there is also plenty that it does not test. It is used by some distributions and has definitely found bugs, but there is a need for more (and better) test suites.

Benchmarks and fuzzing

Beyond that, there is also a need for more benchmarking to detect performance regressions. Mel Gorman's MMTests were mentioned as something that could be used as the basis of a "kbench" benchmarking suite. Some in the room seemed unfamiliar with that test suite, which helped point out the need for better documentation. A test suites file for the kernel documentation directory might be a start, but any benchmark is going to need in-depth documentation.

There was some thought that it would be nice to have a benchmark that boiled down to a single number that could be compared between systems (like the idea behind BogoMips). There was also a fair amount of skepticism about how possible that might turn out to be, however.

Fuzzing for stable kernels was also discussed. Fuzzing the upstream kernels is the best option, since fixes must be made there, but it can find problems in backports for distribution kernels. It turns out that the syzkaller fuzzer generates small test cases to reproduce the problems that it finds. It was agreed that those should be added to the self-tests. Some of the bugs only manifest themselves under the Kernel Address Sanitizer (KASAN), but those tests can simply be skipped as "unsupported" for kernels that are not configured for KASAN.

More and more self-tests are being added to the mainline, but the stable kernels don't benefit from those. Some are running the latest self-tests with older kernels, but there was some thought that perhaps the self-tests themselves should be backported into the stable trees.

As the BoF wound down, Levin asked that distribution maintainers push the patches they are using to the stable trees. It is not uncommon to find a fix in a distribution kernel that should be in stable as well. He has been working on training a neural net to recognize stable-eligible patches, which elicited some laughter, but he said it is actually "going surprisingly well".

One person who was not at the BoF, but has a vested interest in what was being discussed, is stable maintainer Greg Kroah-Hartman. He got a chance to offer some of his opinions in the microconference, which opened with a short session where Levin replayed what was discussed in the BoF.

As Levin said, there were a number of points raised in the BoF without much, if any, resolution of those problems. Someone spoke up to suggest that more hardware be given to the kernelci.org project, but Kroah-Hartman would also like to see more functional testing. It may make sense for the Linaro and kernelci.org efforts to join forces, though, someone said.

Kroah-Hartman has no objection to the idea of backporting self-tests as long as they will run on the kernel in question. He agreed that it would be nice for distributions to be diligent about getting their fixes to the stable tree, but noted that Fedora and Debian are already doing a good job in that area. Distributions often try to get a fix to their users quickly, then do the work to get it fixed upstream, another participant said. Kroah-Hartman said that he often will leave a bug in stable if it is not fixed in the upstream kernel to both be "bug compatible" and to provide some pressure for it to get fixed.

It is clear that more kernel testing could be done, but it is less clear exactly what form it should take or who will actually do it. With luck, some progress on that will be made in the near future, which is likely to lead to more bugs found sooner. Perfection is impossible, of course, but an overall reduction in kernel bugs is something we can all hope for.

[I would like to thank LWN's travel sponsor, The Linux Foundation, for assistance in traveling to Los Angeles for LPC.]

Comments (6 posted)

Verified cryptography for Firefox 57

The Mozilla Security Blog announces that Firefox 57 will benefit from the addition of a formally verified crypto package. "The first result of this collaboration, an implementation of the Curve25519 key establishment algorithm (RFC7748), has just landed in Firefox Nightly. Curve25519 is widely used for key-exchange in TLS, and was recently standardized by the IETF. As an additional bonus, besides being formally verified, the HACL* Curve25519 implementation is also almost 20% faster on 64 bit platforms than the existing NSS implementation (19500 scalar multiplications per second instead of 15100) which represents an improvement in both security and performance to our users."

Comments (35 posted)

Moore: The 2017 Linux Security Summit

Paul Moore has posted his notes from the 2017 Linux Security Summit, held September 14 and 15 in Los Angeles. "LinuxKit was designed to make it easy for people to create their own Linux distribution, with a strong focus on minimal OS installs such as one would use in a container hosting environment. LinuxKit has several features that make it interesting from a security perspective, the most notable being the read-only rootfs which is managed using external tooling. Applications are installed via signed container images."

Comments (none posted)

Malicious software libraries found in PyPI

An advisory from the National Security Authority of Slovakia warns that they have found fake packages in PyPI, posing as well known libraries. "Copies of several well known Python packages were published under slightly modified names in the official Python package repository PyPI (prominent example includes urllib vs. urrlib3, bzip vs. bzip2, etc.). These packages contain the exact same code as their upstream package thus their functionality is the same, but the installation script, setup.py, is modified to include a malicious (but relatively benign) code." The administrators of PyPI were informed and the fake packages are gone now, however they were available from June 2017 to September 2017. (Thanks to Paul Wise)

Comments (29 posted)

Security quote of the week

— Matt Burns in TechCrunch

Comments (1 posted)

Kernel release status

The current development kernel is 4.14-rc1, released on September 16. Linus said: "Yes, I realize this is a day early, and yes, I realize that if I had waited until tomorrow, I would also have hit the 26th anniversary of the Linux-0.01 release, but neither of those undeniable facts made me want to wait with closing the merge window."

Stable updates: 4.13.2, 4.12.13, 4.9.50, 4.4.88, and 3.18.71 were all released on September 14. Among other things, these updates contain the fix for the recently disclosed Bluetooth vulnerability. 4.13.3, 4.12.14, and 4.9.51 followed on September 20. Note that 4.12.14 will be the final update in the 4.12.x series.

Comments (none posted)

2nd RDMA Miniconference Summary

Leon Romanovsky provides a summary of the RDMA miniconference at Linux Plumbers Conference. "Special thanks goes to Ram Amrani who did an excellent job to summarize the discussions and to Jason Gunthrope together with Christoph Lameter who helped me to organize and run this conference. The original etherpad is located at https://etherpad.openstack.org/p/LPC2017_RDMA and below you will find the copy of those notes"

Full Story (comments: 2)

Distribution quotes of the week

— Steve McIntyre

— Adam Williamson

Comments (4 posted)

Robinson: The state of open source accelerated graphics on ARM devices

Peter Robinson looks at the state of open source accelerated graphics on ARM devices. "Despite the two bad examples above there’s actually been a lot of good change in the last five years. We now have a number of options for fully accelerated 2D/3D graphics on ARM SoCs and I run GNOME Shell on Wayland, yes the full open source shiny, on a number of different devices regularly."

Comments (1 posted)

Schaller: Launching Pipewire

Christian Schaller announces Pipewire, a media system that is meant to eventually replace PulseAudio and handle video as well. "Anyway as work progressed Wim decided to also take a look at Jack, as supporting the pro-audio usecase was an area PulseAudio had never tried to do, yet we felt that if we could ensure Pipewire supported the pro-audio usecase in addition to consumer level audio and video it would improve our multimedia infrastructure significantly and ensure pro-audio became a first class citizen on the Linux desktop." A video-only version will be shipping in Fedora 27.

Comments (35 posted)

Purism and KDE to work together on free smartphone

Purism and KDE are working together to adapt Plasma Mobile to Purism's Librem 5 smartphone. "The shared vision of freedom, openness and personal control for end users has brought KDE and Purism together in a common venture. Both organisations agree that cooperating will help bring a truly free and open source smartphone to the market. KDE and Purism will work together to make this happen."

Comments (10 posted)

GNOME Foundation partners with Purism to support its efforts to build the Librem 5 smartphone

Last week KDE announced that they were working with Purism on the Librem 5 smartphone. The GNOME Foundation has also provided its endorsement and support of Purism’s efforts to build the Librem 5. "As part of the collaboration, if the campaign is successful the GNOME Foundation plans to enhance GNOME shell and general performance of the system with Purism to enable features on the Librem 5. Various GNOME technologies are used extensively in embedded devices today, and GNOME developers have experienced some of the challenges that face mobile computing specifically with the Nokia 770, N800 and N900, the One Laptop Per Child project’s XO laptop and FIC’s Neo1973 mobile phone."

Comments (20 posted)

Development quote of the week

— Matt Holt

Comments (4 posted)

EME is now a W3C recommendation

The World Wide Web Consortium has put out a press release trumpeting its publication of the "Encrypted Media Extensions" as an official recommendation and enshrining DRM into what was previously a standard for open communication. See the EFF's open letter for a less rosy view of this development. "Today, the W3C bequeaths an legally unauditable attack-surface to browsers used by billions of people. They give media companies the power to sue or intimidate away those who might re-purpose video for people with disabilities. They side against the archivists who are scrambling to preserve the public record of our era. The W3C process has been abused by companies that made their fortunes by upsetting the established order, and now, thanks to EME, they’ll be able to ensure no one ever subjects them to the same innovative pressures."

Comments (65 posted)

An intro to machine learning (Opensource.com)

Ulrich Drepper, once again an engineer at Red Hat, writes about machine learning on opensource.com. "Machine learning and artificial intelligence (ML/AI) mean different things to different people, but the newest approaches have one thing in common: They are based on the idea that a program's output should be created mostly automatically from a high-dimensional and possibly huge dataset, with minimal or no intervention or guidance from a human. Open source tools are used in a variety of machine learning and artificial intelligence projects. In this article, I'll provide an overview of the state of machine learning today."

Comments (10 posted)

• DistroWatch Weekly (September 18)

• Linux Mint Monthly News (September)

• Lunar Linux Weekly News (September 15)

• Ubuntu Kernel Team Weekly Summary (September 19)

• Ubuntu Server Development Summary (September 12)

• Ubuntu Weekly Newsletter (September 18)

• Free Software Foundation Europe Newsletter (September)

• GCC 5.5 Status Report (September 16)

• What's cooking in git.git (September 15)

• Git Rev News (September 20)

• LLVM Weekly (September 18)

• LXC/LXD/LXCFS Weekly Status (September 18)

• OCaml Weekly News (September 19)

• Perl Weekly (September 18)

• Weekly changes in and around Perl 6 (September 18)

• PostgreSQL Weekly News (September 17)

• Python Weekly Newsletter (September 14)

• Ruby Weekly News (September 14)

• This Week in Rust (September 19)

• Wikimedia Tech News (September 18)

• Fedora cwg meeting minutes (September 20)

• Ubuntu Server Team meeting minutes (September 19)

mini-DebConf 2017 in Toulouse, France

There will be a mini-DebConf in Toulouse, France, in conjunction with "Capitole du Libre", on November 18-19. The call for talks closes September 30.

Full Story (comments: none)

LibrePlanet celebrates 10th anniversary; CFP and registration open now

The Free Software Foundation (FSF) has announced that the tenth annual LibrePlanet free-software conference will be held March 24-25, 2018, in the Cambridge, MA. The theme for this year's conference is "Freedom. Embedded." The call for sessions closes November 2.

Comments (none posted)

CFP Deadlines: September 21, 2017 to November 20, 2017

The following listing of CFP deadlines is taken from the LWN.net CFP Calendar.

Deadline	Event Dates	Event	Location
September 30	November 18 November 19	Technical Dutch Open Source Event	Eindhoven, Netherlands
September 30	November 18 November 19	Toulouse mini-DebConf 2017	Toulouse, France
October 1	December 6 December 7	Linux-Lab	Firenze, Italy
October 7	November 11	Linux Developer Conference Brazil	São Paulo, Brazil
October 15	April 26 April 27	FLOSSUK – Spring Conference	Edinburgh, Scotland
October 24	October 26 October 27	Free Software and Open Source Symposium [CANCELLED]	Toronto, Canada
October 31	March 8 March 11	SCALE 16x	Pasadena, CA, USA
November 2	March 24 March 25	LibrePlanet	Cambridge, MA, USA
November 3	February 3 February 4	FOSDEM 2018	Brussels, Belgium
November 17	January 26 January 28	DevConf.cz	Brno, Czech Republic

If the CFP deadline for your event does not appear here, please tell us about it.

Announcing Google Code-in 2017

Google Code-in is for students ages 13-17 to learn about and contribute to open source software. This year's code-in runs from November 28-January 17, 2018. Open-source organizations can apply to participate in Google Code-in starting on October 9, 2017.

Comments (none posted)

Events: September 21, 2017 to November 20, 2017

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
September 20 September 22	X.org Developers Conference	Mountain View, CA, USA
September 21 September 22	International Workshop on OpenMP	Stony Brook, NY, USA
September 21 September 24	EuroBSDcon 2017	Paris, France
September 22 September 24	VideoLAN developers days	Paris, France
September 25 September 26	Open Source Backup Conference 2017	Köln, Germany
September 26	OpenStack Days UK	London, UK
September 26	Embedded Recipes 2017	Paris, France
September 27 September 29	Alpine Linux Persistence and Storage Summit	Lizumerhuette, Austria
September 27 September 29	Kernel Recipes 2017	Paris, France
September 29 September 30	Ohio LinuxFest	Columbus, OH, USA
September 30 October 1	RustFest Zürich	Zurich, Switzerland
October 2 October 4	O'Reilly Velocity Conference	New York, NY, USA
October 3	PostgresqlConf South Africa	Cape Town, South Africa
October 4 October 5	Lustre Administrator and Developer Workshop	Paris, France
October 5	Open Hardware Summit 2017	Denver, CO, USA
October 5 October 6	PyCon South Africa	Cape Town, South Africa
October 6 October 7	Seattle GNU/Linux Conference	Seattle, WA, USA
October 6 October 8	PyGotham	New York, NY, USA
October 6 October 8	Festival of the Commons	Athens, Greece
October 7 October 8	MiniDebConf Prishtina	Prishtina, Kosovo
October 7 October 8	Silicon Valley Code Camp	San Jose, CA, USA
October 10 October 12	Qt World Summit	Berlin, Germany
October 11 October 13	LibreOffice Conference 2017	Rome, Italy
October 14 October 16	GNOME.Asia Summit	Chongqing, China
October 16 October 20	Tcl/Tk Conference 2017	Houston, TX, USA
October 18 October 20	O'Reilly Velocity Conference	London, UK
October 18 October 20	Plone Conference 2017	Barcelona, Spain
October 18 October 20	14º Congresso Latino-americano de Software Livre e Tecnologias Abertas	Foz do Iguacu, Brazil
October 21 October 22	GStreamer Conference 2017	Prague, Czech Republic
October 21	7th Real-Time Summit	Prague, Czech Republic
October 21 October 22	Datenspuren	Dresden, Germany
October 21 October 22	openSUSE.Asia Summit 2017	Tokyo, Japan
October 21 October 22	All Systems Go!	Berlin, Germany
October 23 October 25	Open Source Summit Europe	Prague, Czech Republic
October 23 October 24	All Things Open	Raleigh, NC, USA
October 23 October 25	Embedded Linux Conference Europe	Prague, Czech Republic
October 23 October 28	FOSS4G + SoTM Argentina 2017	Buenos Aires, Argentina
October 23 October 29	Privacyweek	Vienna (Wien), Austria
October 24 October 27	PostgreSQL Conference Europe 2017	Warsaw, Poland
October 24 October 27	Linux Kernel Summit	Prague, Czech Republic
October 25 October 27	KVM Forum 2017	Prague, Czech Republic
October 25 October 27	PyCon DE	Karlsruhe, Germany
October 26 October 29	European Coreboot Conference 2017	Bochum, Germany
October 26 October 27	Free Software and Open Source Symposium [CANCELLED]	Toronto, Canada
October 26 October 27	OpenWrt Summit	Prague, Czech Republic
October 27	Tracing Summit 2017	Prague, Czech Republic
October 27 October 29	Mozilla Festival	London, UK
October 28 October 29	Wine Project Annual Conference 2017	Wrocław, Poland
October 28 October 29	freenode #live	Bristol, UK
October 30 November 1	O'Reilly Security Conference	New York, NY, USA
October 31 November 2	API Strategy & Practice Conference	Portland, OR, USA
November 3	SFLC’s Fall Conference	New York, NY, USA
November 3 November 4	Linux Piter #3	Saint-Petersburg, Russia
November 4 November 5	OpenFest 2017	Sofia, Bulgaria
November 4 November 5	Sonoj Convention	Cologne, Germany
November 4 November 5	Free Society Conference and Nordic Summit	Oslo, Norway
November 4 November 5	PyCon HK 2017	Hong Kong, Hong Kong
November 4 November 5	PyCon India 2017	Delhi, India
November 6 November 8	OpenStack Summit	Sydney, Australia
November 7 November 9	Cuban Free Technologies Conferences	Havana, Cuba
November 8 November 10	Netdev 2.2	Seoul, South Korea
November 9 November 10	International Hacking & Information Security Conference	Bucharest, Romania
November 11	Linux Developer Conference Brazil	São Paulo, Brazil
November 11 November 12	Intel® HPC Developer Conference	Denver, CO, USA
November 12 November 17	Supercomputing 17	Denver, CO, USA
November 18 November 19	Technical Dutch Open Source Event	Eindhoven, Netherlands
November 18 November 19	Toulouse mini-DebConf 2017	Toulouse, France
November 18 November 19	Capitole du Libre 2017	Toulouse, France

If your event does not appear here, please tell us about it.

Alert summary September 14, 2017 to September 20, 2017

Dist.	ID	Release	Package	Date
Arch Linux	ASA-201709-15		apache	2017-09-18
Arch Linux	ASA-201709-16		ettercap	2017-09-18
Arch Linux	ASA-201709-10		ffmpeg	2017-09-18
Arch Linux	ASA-201709-6		flashplugin	2017-09-14
Arch Linux	ASA-201709-9		kernel	2017-09-15
Arch Linux	ASA-201709-7		lib32-flashplugin	2017-09-14
Arch Linux	ASA-201709-14		lib32-libgcrypt	2017-09-18
Arch Linux	ASA-201709-13		libgcrypt	2017-09-18
Arch Linux	ASA-201709-8		linux-lts	2017-09-14
Arch Linux	ASA-201709-12		linux-zen	2017-09-18
Arch Linux	ASA-201709-11		newsbeuter	2017-09-18
Arch Linux	ASA-201709-5		tcpdump	2017-09-13
CentOS	CESA-2017:2685	C7	bluez	2017-09-13
CentOS	CESA-2017:2771	C7	emacs	2017-09-20
CentOS	CESA-2017:2679	C7	kernel	2017-09-13
CentOS	CESA-2017:2728	C7	postgresql	2017-09-14
Debian	DSA-3980-1	stable	apache2	2017-09-20
Debian	DSA-3975-1	stable	emacs25	2017-09-15
Debian	DLA-1098-1	LTS	freexl	2017-09-17
Debian	DSA-3976-1	stable	freexl	2017-09-17
Debian	DLA-1100-1	LTS	gdk-pixbuf	2017-09-20
Debian	DSA-3978-1	stable	gdk-pixbuf	2017-09-18
Debian	DSA-3977-1	stable	newsbeuter	2017-09-18
Debian	DSA-3979-1	stable	pyjwt	2017-09-19
Debian	DLA-1097-1	LTS	tcpdump	2017-09-15
Debian	DSA-3974-1	stable	tomcat8	2017-09-15
Debian	DLA-1096-1	LTS	wordpress-shibboleth	2017-09-13
Debian	DSA-3973-1	stable	wordpress-shibboleth	2017-09-14
Fedora	FEDORA-2017-292c77b3c1	F25	FlightGear	2017-09-18
Fedora	FEDORA-2017-caa564d86f	F26	FlightGear	2017-09-17
Fedora	FEDORA-2017-3a568adb31	F25	ImageMagick	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	ImageMagick	2017-09-19
Fedora	FEDORA-2017-c5d7fd07c5	F25	LibRaw	2017-09-16
Fedora	FEDORA-2017-80c4677540	F26	LibRaw	2017-09-15
Fedora	FEDORA-2017-3a568adb31	F25	WindowMaker	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	WindowMaker	2017-09-19
Fedora	FEDORA-2017-7dacb3c21c	F25	augeas	2017-09-14
Fedora	FEDORA-2017-3a568adb31	F25	autotrace	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	autotrace	2017-09-19
Fedora	FEDORA-2017-fe95a5b88b	F26	bluez	2017-09-13
Fedora	FEDORA-2017-3a568adb31	F25	converseen	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	converseen	2017-09-19
Fedora	FEDORA-2017-e4609f71f6	F26	cyrus-imapd	2017-09-15
Fedora	FEDORA-2017-8f27031c8f	F26	dmtx-utils	2017-09-19
Fedora	FEDORA-2017-3a568adb31	F25	drawtiming	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	drawtiming	2017-09-19
Fedora	FEDORA-2017-3a568adb31	F25	emacs	2017-09-19
Fedora	FEDORA-2017-a1dc0ef38c	F26	emacs	2017-09-13
Fedora	FEDORA-2017-8f27031c8f	F26	emacs	2017-09-19
Fedora	FEDORA-2017-b7e6e4cfc1	F25	freexl	2017-09-17
Fedora	FEDORA-2017-6679a0a2e1	F26	freexl	2017-09-17
Fedora	FEDORA-2017-e7ae1ed967	F26	gdm	2017-09-15
Fedora	FEDORA-2017-3a568adb31	F25	gtatool	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	gtatool	2017-09-19
Fedora	FEDORA-2017-3a568adb31	F25	imageinfo	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	imageinfo	2017-09-19
Fedora	FEDORA-2017-3a568adb31	F25	inkscape	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	inkscape	2017-09-19
Fedora	FEDORA-2017-769793738f	F26	jasper	2017-09-19
Fedora	FEDORA-2017-3a568adb31	F25	k3d	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	k3d	2017-09-19
Fedora	FEDORA-2017-7369ea045c	F26	kernel	2017-09-17
Fedora	FEDORA-2017-3a568adb31	F25	kxstitch	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	kxstitch	2017-09-19
Fedora	FEDORA-2017-c98c2e8e7a	F26	libwmf	2017-09-13
Fedora	FEDORA-2017-63ff51c0dc	F26	libwpd	2017-09-19
Fedora	FEDORA-2017-66adafeb3b	F25	lightdm	2017-09-15
Fedora	FEDORA-2017-d793fef58f	F26	lightdm	2017-09-14
Fedora	FEDORA-2017-bb5d87e9de	F25	mingw-libzip	2017-09-19
Fedora	FEDORA-2017-4d4914a260	F26	mingw-libzip	2017-09-19
Fedora	FEDORA-2017-3a568adb31	F25	perl-Image-SubImageFind	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	perl-Image-SubImageFind	2017-09-19
Fedora	FEDORA-2017-3a568adb31	F25	pfstools	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	pfstools	2017-09-19
Fedora	FEDORA-2017-3a568adb31	F25	php-pecl-imagick	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	php-pecl-imagick	2017-09-19
Fedora	FEDORA-2017-3a568adb31	F25	psiconv	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	psiconv	2017-09-19
Fedora	FEDORA-2017-8614a6e905	F26	python-django	2017-09-14
Fedora	FEDORA-2017-3a568adb31	F25	q	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	q	2017-09-19
Fedora	FEDORA-2017-b10e1a9166	F26	rawtherapee	2017-09-19
Fedora	FEDORA-2017-3a568adb31	F25	ripright	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	ripright	2017-09-19
Fedora	FEDORA-2017-3a568adb31	F25	rss-glx	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	rss-glx	2017-09-19
Fedora	FEDORA-2017-e136d63c99	F25	ruby	2017-09-16
Fedora	FEDORA-2017-3a568adb31	F25	rubygem-rmagick	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	rubygem-rmagick	2017-09-19
Fedora	FEDORA-2017-3a568adb31	F25	synfig	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	synfig	2017-09-19
Fedora	FEDORA-2017-3a568adb31	F25	synfigstudio	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	synfigstudio	2017-09-19
Fedora	FEDORA-2017-3a568adb31	F25	techne	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	techne	2017-09-19
Fedora	FEDORA-2017-a00a087fd4	F25	tomcat	2017-09-15
Fedora	FEDORA-2017-ab0def38cd	F26	tomcat	2017-09-14
Fedora	FEDORA-2017-3a568adb31	F25	vdr-scraper2vdr	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	vdr-scraper2vdr	2017-09-19
Fedora	FEDORA-2017-3a568adb31	F25	vips	2017-09-19
Fedora	FEDORA-2017-8f27031c8f	F26	vips	2017-09-19
Fedora	FEDORA-2017-ed735463e3	F25	xen	2017-09-16
Fedora	FEDORA-2017-e399a9008c	F26	xen	2017-09-15
Gentoo	201709-02		binutils	2017-09-17
Gentoo	201709-05		chkrootkit	2017-09-17
Gentoo	201709-14		curl	2017-09-17
Gentoo	201709-08		gdk-pixbuf	2017-09-17
Gentoo	201709-11		gimps	2017-09-17
Gentoo	201709-10		git	2017-09-17
Gentoo	201709-07		kpathsea	2017-09-17
Gentoo	201709-04		mod_gnutls	2017-09-17
Gentoo	201709-12		perl	2017-09-17
Gentoo	201709-13		squirrelmail	2017-09-17
Gentoo	201709-09		subversion	2017-09-17
Gentoo	201709-06		supervisor	2017-09-17
Gentoo	201709-03		webkit-gtk	2017-09-17
Mageia	MGASA-2017-0340	5, 6	389-ds-base	2017-09-16
Mageia	MGASA-2017-0339	5, 6	flash-player-plugin	2017-09-14
Mageia	MGASA-2017-0345	5	kernel	2017-09-16
Mageia	MGASA-2017-0342	6	kernel	2017-09-16
Mageia	MGASA-2017-0347	5	kernel-linus	2017-09-16
Mageia	MGASA-2017-0344	6	kernel-linus	2017-09-16
Mageia	MGASA-2017-0346	5	kernel-tmb	2017-09-16
Mageia	MGASA-2017-0343	6	kernel-tmb	2017-09-16
Mageia	MGASA-2017-0338	5, 6	libsndfile	2017-09-14
Mageia	MGASA-2017-0341	5, 6	mpg123	2017-09-16
openSUSE	openSUSE-SU-2017:2482-1		chromium	2017-09-15
openSUSE	openSUSE-SU-2017:2491-1	42.2 42.3	chromium	2017-09-15
openSUSE	openSUSE-SU-2017:2483-1	42.2 42.3	cvs	2017-09-15
openSUSE	openSUSE-SU-2017:2501-1	42.2	ffmpeg, ffmpeg2	2017-09-16
openSUSE	openSUSE-SU-2017:2502-1	42.3	ffmpeg, ffmpeg2	2017-09-16
openSUSE	openSUSE-SU-2017:2495-1	42.2	kernel	2017-09-15
openSUSE	openSUSE-SU-2017:2494-1	42.3	kernel	2017-09-15
openSUSE	openSUSE-SU-2017:2488-1	42.2 42.3	libreoffice	2017-09-15
openSUSE	openSUSE-SU-2017:2513-1	42.3	qemu	2017-09-18
openSUSE	openSUSE-SU-2017:2514-1	42.3	xen	2017-09-18
Oracle	ELSA-2017-2771	OL7	emacs	2017-09-19
Oracle	ELSA-2017-3620	OL6	kernel	2017-09-19
Oracle	ELSA-2017-2679-1	OL7	kernel	2017-09-13
Oracle	ELSA-2017-3620	OL7	kernel	2017-09-19
Oracle	ELSA-2017-2728	OL7	postgresql	2017-09-14
Red Hat	RHSA-2017:2771-01	EL7	emacs	2017-09-19
Red Hat	RHSA-2017:2726-01	OSP11.0	instack-undercloud	2017-09-13
Red Hat	RHSA-2017:2732-01	EL6.2	kernel	2017-09-14
Red Hat	RHSA-2017:2731-01	EL6.6	kernel	2017-09-14
Red Hat	RHSA-2017:2760-01	EL6.7	kernel	2017-09-19
Red Hat	RHSA-2017:2770-01	EL7.3	kernel	2017-09-19
Red Hat	RHSA-2017:2727-01	OSP11.0	openvswitch	2017-09-13
Red Hat	RHSA-2017:2728-01	EL7	postgresql	2017-09-14
Scientific Linux	SLSA-2017:2771-1	SL7	emacs	2017-09-19
Scientific Linux	SLSA-2017:2728-1	SL7	postgresql	2017-09-14
Slackware	SSA:2017-261-01		httpd	2017-09-18
Slackware	SSA:2017-258-02		kernel	2017-09-15
Slackware	SSA:2017-261-02		libgcrypt	2017-09-18
Slackware	SSA:2017-261-03		ruby	2017-09-18
SUSE	SUSE-SU-2017:2529-1	OS6 SLE12	emacs	2017-09-20
SUSE	SUSE-SU-2017:2459-1	OS6 SLE12	kernel	2017-09-14
SUSE	SUSE-SU-2017:2525-1	SLE11	kernel	2017-09-19
SUSE	SUSE-SU-2017:2521-1	SLE12	kernel	2017-09-18
SUSE	SUSE-SU-2017:2523-1	SLE12	kernel	2017-09-19
SUSE	SUSE-SU-2017:2450-1	SLE11	xen	2017-09-13
SUSE	SUSE-SU-2017:2519-1	SLE12	xen	2017-09-18
Ubuntu	USN-3425-1	14.04 16.04 17.04	apache2	2017-09-19
Ubuntu	USN-3346-2	14.04 16.04 17.04	bind9	2017-09-18
Ubuntu	USN-3418-1	14.04 16.04 17.04	gdk-pixbuf	2017-09-18
Ubuntu	USN-3423-1	12.04	kernel	2017-09-18
Ubuntu	USN-3422-1	14.04	kernel	2017-09-18
Ubuntu	USN-3417-1	17.04	libgcrypt20	2017-09-14
Ubuntu	USN-3421-1	17.04	libidn2-0	2017-09-18
Ubuntu	USN-3424-1	14.04 16.04 17.04	libxml2	2017-09-18
Ubuntu	USN-3420-1	16.04	linux, linux-aws, linux-gke, linux-kvm, linux-raspi2, linux-snapdragon	2017-09-18
Ubuntu	USN-3419-1	17.04	linux, linux-raspi2	2017-09-18
Ubuntu	USN-3419-2	16.04	linux-hwe	2017-09-18
Ubuntu	USN-3422-2	12.04	linux-lts-trusty	2017-09-18
Ubuntu	USN-3420-2	14.04	linux-lts-xenial	2017-09-18
Ubuntu	USN-3415-2	12.04	tcpdump	2017-09-13
Ubuntu	USN-3415-1	14.04 16.04 17.04	tcpdump	2017-09-13
Ubuntu	USN-3416-1	14.04 16.04 17.04	thunderbird	2017-09-14

Full Story (comments: none)