Welcome to the LWN.net Weekly Edition for August 8, 2019

• Escape sequences in Python strings: should the Python interpreter continue to magically fix up invalid string escape sequences?
• An end to implicit fall-throughs in the kernel: a two-year project to eliminate a source of kernel bugs reaches its conclusion.
• vDSO, 32-bit time, and seccomp: a 5.3 regression caused by a combination of factors, including the general difficulty of seccomp.
• Racket: Lisp for learning: a look at a language for creating other languages.
• The Compact C Type Format in the GNU toolchain: an old-but-new-again scheme for representing type information is added to the GNU tools.
• Mozilla's WebThings Gateway now available for Turris Omnia router: a distribution for web-of-things applications.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Escape sequences in Python strings

A change for Python 3.8—currently in beta—has produced some user-visible warnings, but the problem is often in code that a user cannot (or should not) change: third-party modules. The problem that the warning is trying to highlight is real, however. The upshot is that the handling of escape sequences (or non escape sequences, in truth) in Python string literals is in a rather messy state at this point.

Python string literals come in many different forms, but the main ones look like the following:

    a = 'foo'
    b = "bar"
    c = '''
    baz
    '''

But what to do about string literals with invalid escape sequences in them? A programmer who has put '\latex' as part of a string literal (to pick a not entirely random example) presumably actually wants '\\latex', which is what Python currently translates it to. Python does emit a DeprecationWarning in that case, but the warning was invisible by default until Python 3.7. However, that same programmer probably does not want '\tan(x)' to turn into a tab plus 'an(x)', but that is exactly what happens.

The change for Python 3.8 is to further elevate the warning to a SyntaxWarning, with plans to turn that into a SyntaxError in Python 3.9. A bug report filed in February 2018 shows the path of the change. But shortly after the Python 3.8 beta releases were made, Raymond Hettinger reported that he was seeing the warnings "pop up from time to time" from various third-party packages. Aaron Meurer concurred with Hettinger and pointed out a number of other problems he had encountered.

Serhiy Storchaka, who authored the patch for 3.8, replied that one of the examples given in the reports was in the PyParsing module, but that it showed a real problem that needed addressing, thus proving the usefulness of the change. A docstring in PyParsing referred to a regular expression, but did not escape the backslashes in the expression. The line in the docstring (delimited by ''') was:

    make_html = Regex(r"(\w+):(.*?):").sub(r"<\1>\2</\1>")

In mid-July, Hettinger reported a problem he encountered with the Bottle web framework, where, once again, the warning was showing up, but in code his students could not change. He said: "I think it is poor form to bombard end-users with warnings about things they can't fix." Storchaka pointed out that the problem in Bottle resulted from a bad backport of a change made five years ago.

But even though the warning became visible in Python 3.7, it was still difficult for projects to see. The problem is that the warning is emitted when the .py file is compiled into a .pyc file and never shown after that point. For many projects, as Meurer pointed out, that means it is quite difficult to actually test for the presence of the warning, even when the projects are diligently looking for it:

That led Nathaniel Smith to argue that the deprecation cycle was not really followed for these strings. He suggested perhaps storing these kinds of warnings in the .pyc file so that they could be emitted later, and not just at compilation time. Around the same time, Hettinger elevated the visibility of the problem with a post to the python-dev mailing list:

He also noted that ASCII art in docstrings was affected and that Python had been living with its current behavior for nearly 30 years without any huge downside. The main problem that warning for illegal escape sequences is trying to solve is for file names on Windows systems, where the backslash is used as a directory separator, he said. Many of those problems cannot even be caught by the warning; his example was '..\training\new_memo.doc', which would produce a corrupted file name, but no warning. One solution for those types of strings is to use Python's raw strings: r'..\training\new_memo.doc'.

But Chris Angelico was concerned that allowing illegal escape sequences to pass silently is problematic. He did a quick survey of some other languages and found that most gave a warning (or, in the case of Lua, an error), but that all of the others just ignored the spurious \ entirely, rather than turning it into \\ as Python does. "IMO Python's solution is better, but Lua's is best. A bad escape is an error."

In general, the consensus in the thread seems to be to slow down the process of turning the warning into something that users see, at least until the developers of modules in the Python Package Index (PyPI) can address the problem. There were lots of suggestions for ways to find these strings automatically in ways that will be seen by the developers, but those will need to wait for the Python 3.9 cycle.

The behavior with escape sequences has other wrinkles, as Steven D'Aprano pointed out. For example, escaped non-delimiting quotes are treated differently than other escape sequences:

    py> "abc \d \' xyz"
    "abc \\d ' xyz"

So D'Aprano believes a change needs to be made, but that the current pace is too fast: "This isn't a critical problem that needs to be fixed soonest." But Angelico is worried about simply kicking the can down the road. He would like to see some active steps be taken so that developers can see and fix these problems. The more-visible warning added for 3.8 serves that purpose but, as D'Aprano noted, there will likely be some undesirable effects of that, beyond just user unhappiness. Users will file bugs, for example, often without looking to see if there is an existing bug, and they may or may not file the bug in the right place. Furthermore: "The benefit of the desired change is relatively low."

While D'Aprano is generally in favor of giving warnings (and, presumably, eventually errors) for these illegal escape sequences, he was not convinced that it has been adequately justified. He mentioned that it would allow new escape sequences, but did not find that to be of huge benefit. Angelico was adamant, however, that the problem is larger than just blocking new escape sequences:

Storchaka has submitted a pull request to revert his change that elevated the DeprecationWarning to a SyntaxWarning. Based on the mood in the thread, one would guess that will be merged, but, if the problem is truly going to be addressed, additional work will be needed. No one has suggested dropping the DeprecationWarning, though some seem to think there is marginal utility to changing the long-established "escape illegal escape sequences" feature. But it is clear that doing so without adequate warning causes problems of a different—and more user-visible—sort.

Comments (22 posted)

An end to implicit fall-throughs in the kernel

The problem with C's fall-through behavior is that it is implicit, with no indication of whether the behavior is intended or not. Developers learn (the hard way, sometimes) to end each case with a break statement as a matter of habit, but it's still an easy thing to forget, and the resulting code is seen by the compiler as being entirely valid. A forgotten break almost certainly introduces a bug, even if it might not manifest itself for years. Many developers have had reason to wish that the C language required an explicit indication by the programmer that fall-through behavior is desired.

Making fall-through behavior explicit

The GCC compiler has, for some time, offered a warning option (-Wimplicit-fallthrough) intended to catch missing break statements. It will trigger for any case that falls through into a succeeding case unless an explicit marker has been placed to indicate that the behavior is correct. It turns out that that there are many ways of placing this marker, depending on the value provided with -Wimplicit-fallthrough; the full set can be found in the GCC documentation. At its most lax, the presence of any comment at all prior to a case statement will allow warning-free fall-through into that case. Using -Wimplicit-fallthrough=3 causes the compiler to look for comments matching a fairly large set of regular expressions.

In the kernel, one is most likely to find comments that look like:

Style	Count
/* fallthrough */	350
/* fall through */	3,091
/* falls through */	8
/* fall-through */	167

Note that case was ignored in generating the above numbers, and that there are probably other variants in the code as well. All of those comment styles will be recognized by GCC with -Wimplicit-fallthrough=3 and will thus cause warnings to be suppressed.

As can be seen, there are a lot of these comments in the kernel source now, some of which date back to before the beginning of the Git era. Their number has increased in recent times, though, due mostly to a focused effort by Gustavo A. R. Silva, who has been working to eliminate every implicit fall-through warning in the kernel. The point of this effort, beyond fixing any bugs caused by missing break statements, is to make it possible to enable -Wimplicit-fallthrough=3 in the kernel build by default. As long as that option generates a pile of warnings, developers will ignore them and nobody will notice when new ones are added. Once the kernel is warning-free, though, it should be possible to keep it that way.

This work, which was supported by the Linux Foundation, has resulted in over 400 patches since 2017. As Silva has worked through the kernel source eliminating fall-through warnings, he has encountered (and fixed) a number of bugs. A quick search turns up the following commits, for example:

d64062b57eeb	drm/amdgpu/gfx10: Fix missing break in switch statement
737298d18836	drm/amdkfd: Fix missing break in switch statement
60747828eac2	net: socket: Fix missing break in switch statement
84242b82d81c	rtlwifi: rtl8723ae: Fix missing break in switch statement
7850b51b6c21	scsi: mpt3sas: Add missing breaks in switch statements
5e420fe63581	scsi: aacraid: Fix missing break in switch statement
09186e503486	security: mark expected switch fall-throughs and add a missing break
b5be853181a8	crypto: ccree - fix missing break in switch statement
7264235ee74f	IB/hfi1: Add missing break in switch statement
cc5034a5d293	drm/radeon/evergreen_cs: fix missing break in switch statement
479826cc8611	staging: comedi: ni_660x: fix missing break in switch statement
5340f23df8fe	gpio: sprd: Add missing break in switch statement
df997abeebad	iscsi_ibft: Fix missing break in switch statement
2f10d8237396	drm/amd/powerplay: Fix missing break in switch
307b00c5e695	rtl8xxxu: Fix missing break in switch
5d25ff7a5448	scsi: ips: fix missing break in switch
a7ed5b3e7dca	staging: comedi: tio: fix multiple missing break in switch bugs
c24bfa8f21b5	spi: slave: Fix missing break in switch
ad0eaee6195d	ASoC: wm8994: Fix missing break in switch
9ba8376ce1e2	ptp: fix missing break in switch
4e57562b4846	iio: imu: inv_mpu6050: fix missing break in switch

The above list is clearly incomplete, though, since a number of the other fixes do not explicitly mention missing break statements in the subject line; see commit ed4ed4043a12, for example.

On July 27, a milestone of sorts was hit with this pull into the mainline from Silva's repository: the final implicit fall-through warning has been fixed, and -Wimplicit-fallthrough=3 is now the default for kernel builds. In other words, the kernel's coding style has just been tweaked to disallow the use of implicit fall-through in switch statements. It must be a satisfying conclusion to a long project.

Moving beyond comments

Naturally, though, some developers are not entirely happy with this change. Nobody who actually works in the kernel community seems to disagree with the idea of eliminating implicit fall-throughs, but the mechanism used to do so does not sit well with everybody; as Peter Zijlstra put it back in June: "I still consider it an abomination that the C parser looks at comments". Zijlstra, like others, would prefer that there were another way.

As it happens, there is another way. GCC starting with version 7 allows the use of a special attribute on a statement instead:

    __attribute__((fallthrough))

The use of this attribute can perhaps made more visually pleasing with a definition like:

    #define __fallthrough __attribute__((fallthrough))

The __fallthrough symbol could then be used instead of the magic comments to eliminate implicit fall-through warnings. A patch set implementing this option was posted by Miguel Ojeda in October 2018; it generated a fair amount of interest but was not adopted for a couple of reasons: it doesn't work with older versions of GCC, and the LLVM Clang compiler does not implement that attribute at all. Since Clang does successfully build the kernel in some settings (and is evidently used for the Android build), breaking it is not an appealing option.

The good news is that the Clang developers appear to be working on implementing this attribute. They also, it seems, have an implementation aimed at the proposed next standard for the C language, currently called C2X. The bad news is that the syntax there is different:

    [[fallthrough]]

If this version becomes an actual part of the C standard, it would probably be the preferable one to use going forward. But that may not be fully resolved for some time. In any case, the actual mechanism used to mark explicit fall-through behavior can be hidden behind a #define as shown above.

The one thing that can't be done is:

    #define __fallthrough /* fall through */

since the preprocessor will simply delete the comment rather than substituting it into the source. So the comments have to remain for now. But one can imagine that, at some point in the future when a better alternative is available, those thousands of comments are likely to be replaced with something like __fallthrough in a set of massive, tree-wide patches.

Meanwhile, though, the kernel has finally reached a point where there are no switch statements with implicit fall-through behavior, and the build process has been amended to prevent such behavior from being added in the future. One source of kernel bugs has been closed, a switch that can only be seen as a best-case scenario.

Comments (39 posted)

vDSO, 32-bit time, and seccomp

The virtual dynamic shared object (vDSO) mechanism is an optimization provided by the kernel to reduce the cost of certain frequently used system calls. The vDSO is a small region of kernel-provided memory that is normally mapped into the address space of every user-space process; it contains implementations of system calls that can, in some circumstances at least, do their work in a user-space context. That allows the caller to avoid making a real system call and, thus, to avoid the cost of a context switch into kernel mode. System calls related to timekeeping, such as gettimeofday() are implemented in the vDSO, since they can often run quickly in user space and they tend to be called frequently.

The vDSO has generally been implemented in an architecture-specific way, even though the functions it performs are mostly the same across architectures. In the 5.2 development cycle, Vincenzo Frascino added a generic vDSO implementation that factored out much of the architecture-specific code into a single implementation that could be used on all architectures. During the 5.3 merge window, the x86 architecture switched over to the generic version, and all was well — or so it seemed.

seccomp() sadness

In mid-July, Sean Christopherson (among others) reported that the generic vDSO change broke some seccomp() users on 32-bit x86 systems. seccomp(), remember, allows user space to provide a BPF program (still "classic BPF", not eBPF as is used almost everywhere else in a contemporary Linux system) to control which system calls may be made. It is used to reduce the attack surface of code that might be exposed to attackers in one way or another; using it correctly is hard, but the number of users has been on the rise.

While the vDSO can usually implement timekeeping system calls in user space, that is not always possible. If the calling program wants an esoteric clock that has not been implemented, or if the timekeeping hardware available on the system is not amenable to vDSO access, then the vDSO must fall back to calling into the kernel. Prior to 5.3, the architecture-specific vDSO used the native clock_gettime() call on the system it was running on; that meant calling the 32-bit clock_gettime() on 32-bit kernels.

The 32-bit time format is, of course, going to run out of range in January 2038. Quite a bit of work has gone into preparing systems for this particular apocalypse, though much work still remains. Given this problem, adding new users of 32-bit time interfaces is a way to become rather unpopular in kernel-development circles, so the generic vDSO implementation naturally used clock_gettime64() as the fallback timekeeping system call on all architectures. That is not the sort of thing that one would ordinarily even have to think about much; nobody wants to create a generic vDSO implementation that contains yet another year-2038 problem in need of fixing.

But there is a problem here. A surprising number of programs want to know what time it is at some point or another. Anybody putting together a seccomp() policy for a given program will almost certainly allow system calls like gettimeofday(); otherwise the target program will probably break. A program that fails to run is generally secure, but users, being generally unreasonable, tend to get disgruntled anyway.

Any rational seccomp() policy will, thus, allow for the fallback system call when the vDSO is unable to provide the time directly. But it turns out that, while these policies allowed clock_gettime() on 32-bit systems, they lacked the foresight to let clock_gettime64() through as well. The end result is that, when a program protected by one of these seccomp() policies runs on a 5.3 kernel, it is quickly and rudely killed when it tries to make a disallowed system call.

Kernel developers might protest that this change is required to avoid year-2038 problems. They might also be naturally inclined to disregard lame excuses about how clock_gettime64() was never needed before, or about how that system call didn't even exist until the 5.1 release. But, in the end, this is a regression, and the kernel community's policy on such things is fairly unambiguous. Somehow, programs running under existing seccomp() policies will need to continue to work when the final 5.3 kernel comes out.

Fixing the problem

Various ideas were raised for how that could be done, starting with a not-entirely-serious suggestion that the generic vDSO change could simply be reverted. Perhaps seccomp() rules could be bypassed for system calls that originate in the vDSO; this idea didn't get far given that, among other things, faking a vDSO return address is not a difficult thing to do. Bypassing seccomp() for clock_gettime64() specifically is an option, but that would defeat administrators who want to block all access to timekeeping information. The concept of "system-call aliases" was circulated, initially by Andy Lutomirski; it would create a short list of "equivalent" system calls that take the same arguments and do the same thing. If one call in the list was rejected by a seccomp() filter, the kernel would retry the policy with any aliases that might exist.

The alias idea got further than many, but it has problems of its own. For example, authors of seccomp() policies might genuinely want to discriminate between "equivalent" system calls. It seems like the sort of mechanism that could generate surprising results in general. Aliases might still be the long-term solution for this problem but, as Lutomirski pointed out, "it's getting quite late to start inventing new seccomp features to fix this". Something simpler is needed, at least for the 5.3 release.

That something is likely to be based on this patch series from Thomas Gleixner, which simply causes the vDSO to fall back to the 32-bit clock_gettime() system call on 32-bit systems. It is a solution that is pleasing to nobody, but it solves the regression issue for now.

Some other solution will be required eventually; it is not possible to support 32-bit time indefinitely. One possibility is that the authors of seccomp() policies change their code to allow clock_gettime64() as well. But, even if that could be done and widely deployed, there is no strong incentive for developers to do this work, since their existing policies will continue to function as intended. Some sort of multi-year deprecation process could be considered as a way to force policies to be fixed. But the eventual solution may just have to live in seccomp() instead, perhaps in the form of an alias list or other special exception. A long-term solution that is pleasing to everybody is difficult to envision.

This situation highlights a problem with seccomp() in general: it is difficult to write robust policies at that level of detail, and the resulting policies tend to be brittle in the best of times. Even if the kernel community avoids incompatible changes, a change in a library somewhere can invoke a new system call that a given seccomp() policy may frown upon. While the OpenBSD pledge() mechanism may not offer the degree of control provided by seccomp(), its use of relatively broad categories of functionality makes it easier to avoid problems like this. But Linux has seccomp(), with all its power and complexity. It seems highly likely that developers will unwittingly run into this sort of regression again in the future.

Comments (54 posted)

Racket: Lisp for learning

Lisp is one of the oldest programming languages still in use today—Fortran is older by a year, but the Lisp community (or communities) seems to be the more dynamic of the two. In any case, the Lisp landscape has a lot of nooks and crannies to explore; I recently ran into a dialect that I had not encountered before: Racket. That may simply reflect ignorance on my part, but, while I was introduced to Lisp (too) many moons ago, I had not really paid it much mind until I sat in on a talk about Lisp at linux.conf.au earlier this year. Something about Racket caught my eye, so I did some poking around to see what it is all about.

The dynamism in the Lisp world also means that there are lots of projects, subprojects, dialects, descendants, and so on to keep straight. Lisp itself has split into three main dialects: Common Lisp, Scheme, and Clojure. Common Lisp and Scheme each have multiple implementations. Racket is based on Scheme; it was known as "PLT Scheme" (after the PLT organization behind the language) until version 5.0 was released in 2010.

Descended from Scheme

So Racket is a Scheme descendant, but not a Scheme implementation. It is described on its home page as a cross-platform, general-purpose language, but one with a focus on language-oriented programming. The Racket Manifesto goes even further down that path:

And, in fact, there are already multiple languages defined for Racket, including a web-application language; a logic-based language similar to Prolog, datalog; Scribble for generating HTML and PDF documentation; and more. Racket, like Python, advertises itself as a "batteries included" language, with a standard library that contains many different useful modules, including for graphics and GUIs, networking, format parsing, left-pad, and so on; the Racket Documentation page has links to the ecosystem's extensive documentation.

Racket was created in part to further the educational aims of its developers, who are largely computer science professors and graduate students. The How to Design Programs textbook is based around Racket programming and its integrated development environment (IDE), but does not actually teach Racket itself. It starts with the Beginning Student Language (BSL) and moves on to various more advanced dialects; these languages are targeted at novice programmers and, in particular, ensuring that error diagnosis is usable by beginners. As a note in the preface puts it:

DrRacket

The IDE is DrRacket, which provides an interactive read-eval-print loop (REPL) in the "interactions window" and an editor for programs in the "definitions window", as can be seen in the screen shot below. The definitions window is at the top (with the rocket and LWN penguin), half obscured by the help information, which has been locked in place. The interactions window is at the bottom; it shows an arrow in the upper right that leads to help text (which can be locked in place as was done above). Graphical elements are handled like any other data type; the Quick introduction provides a kind of overview of the language using graphics, stepping through the major features of the language (which are, unsurprisingly, Lisp-like). It works its way up through binding, functions, macros, and so on, ending up with a brief look at the object system in the guise of the GUI library.

That introduction uses the "#lang slideshow" dialect of Racket; in general, "#lang" is used in the definitions window to say which language is being used. #lang racket is the standard dialect, though "typed/racket", "racket/gui", "web-server/insta", "datalog", and many others can be specified there. As one would guess, DrRacket has helpful parenthesis-handling features, syntax highlighting, debugging tools, and more. It is geared toward newbies, but does not look to be too "dumbed down" for those with more experience.

All of the different languages and dialects of Racket are a bit confusing, at least for this (relative) newbie, but it gets even further complicated. There is a new version of Racket under development that sits atop Chez Scheme instead of the existing C-based version; it is known as "Racket on Chez Scheme" or just "Racket CS". It is meant to eventually become the standard Racket, in large part because it will make for a more maintainable code base, according to the original project announcement in 2017 and the January 2019 progress report. Racket CS will be largely compatible with Racket, but there will be exceptions (particularly for extensions that use the C API), so the existing Racket implementation will still be maintained.

All of that means that users will have a choice when installing Racket, starting with the upcoming 7.4 release. Racket CS will be available as a beta download when the new version comes out. For Linux users who want to try Racket out, it is available for most distributions (e.g. Racket 7.0 for Fedora 29); if the most recent release is desired, it is available in binary form via a shell script installer. The source code for Racket is on GitHub; it is available under the LGPLv3. Racket is also a member project of the Software Freedom Conservancy

Parting thoughts

Racket is a daunting project to dip into. As with any overview, this one barely scratches the surface of what Racket has to offer. For those looking for a Lisp-like language to explore, Racket looks like a good choice; the documentation, textbook, and multiple domain-specific tutorials means that there is a wealth of information to digest and apply. The textbook, in particular, looks like something I will try to find some time to dig further into; I may not have truly grokked Lisp the first time around, but Racket may provide another opportunity to do so.

Comments (1 posted)

The Compact C Type Format in the GNU toolchain

Compiling a program into its binary form discards a lot of information found in the source code; that information can be needed when the time comes to track down a bug in the compiled program. To facilitate this work, compilers create debugging information that records the names and types of the variables used by a program, along with function names, the line numbers in the source program, and more; this information is then stored in one of many formats. DWARF is by far the most commonly used format on Unix-like systems, but it is not the only one.

Given the dominance of DWARF, one might wonder why anyone would want to work on alternatives. One problem is that DWARF is complex; rather than containing straightforward information about a program, a DWARF entry is essentially a program in its own right that can be run to generate the needed information. That makes DWARF flexible, but it's also complicated and verbose; the DWARF data associated with a program can be huge. That size means that, on most systems, the DWARF data for the installed programs is relegated to "debuginfo" packages that are not even present unless the owner has gone out of their way to install them.

CTF was created out of a desire to be able to perform most debugging tasks even in the absence of debuginfo packages, and to be able to do so in a simpler and faster way. DWARF can also expose a lot of information about the program source that some companies might wish to keep to themselves; CTF contains a lot less unneeded information. The CTF format was first created for the Solaris system, but has been used with the Linux DTrace port since 2012. There is, she said, nothing DTrace-specific about CTF, though. It is also available on FreeBSD and macOS.

The key difference between CTF and DWARF, perhaps, is that CTF limits itself to modeling the type system and managing the mapping from symbol-table entries to specific types. DWARF is much more ambitious, she said, modeling everything relating to the C language and how it maps to the hardware. CTF's simplicity means it can omit location lists, stack machines, and a lot of other machinery.

Bringing CTF to Linux requires contributing a lot of code upstream. One piece of the puzzle is adding support to GCC; the new -gt option will cause the compiler to generate CTF data. Getting that data into the final executable requires support in the binutils package as well; this includes enhancing the linker as well as adding support to tools like objdump and readelf. Work is also being done to get CTF support into the GDB debugger. Zannoni showed some sample output from the size utility; the CTF data (stored in an ELF section named .ctf) required about 213KB of space, as compared to over 4MB for the DWARF data for the same program. DWARF data for the kernel requires 1.6GB; the CTF data fits in just under 7MB.

CTF and DWARF data can coexist in the same ELF file, she said, since the CTF data has its own dedicated section. The CTF data is naturally smaller, but the format also includes compression to reduce the size requirements further. The result is that this data, unlike DWARF information, need not be stripped to get the executable file down to a reasonable size. Thus, while DWARF data is normally shipped in separate debuginfo packages, CTF data is easily included in the binary package and can be always available.

Internally, CTF data is stored in a structure called a "container" or a "dictionary". Each dictionary contains a header and a number of subsections dedicated to data like function information, variable information, types, and a string table for names not already present in the ELF string table. The header starts with a magic number (useful for determining the endianness of the rest of the data), a version number, and a set of flags. Version one is the original Solaris CTF, while version two was created during the porting of DTrace to Linux. It mostly increases a number of limits found in the first version. The third version is still being defined; it will include a number of header changes among other things. There is even a fourth version in an "initial planning stage". The intent is to keep this data ABI compatible, though, she said.

Returning to linker support, Zannoni noted that GCC will place a single .ctf section in each object file it creates. The linker then has to take these sections and merge them into a larger section, removing any duplicate information. There is a potential problem, though, in that different object files may define conflicting objects using the same names. When this happens, the linker will create a child dictionary associated with a specific translation unit for the conflicting data. Most of the time, though, a linked executable will contain one large shared CTF dictionary and perhaps a small number of tiny subdictionaries.

There is a libctf library being added to the binutils package that implements the ability to read and write CTF data; it is used by the compiler, the linker, and the debugger. This library, along with the readelf and objdump changes, were merged into the binutils trunk in May; the linker changes have been posted but need some more work before they can be merged. The hope is that all of the CTF support will land in binutils for the upcoming 2.33 release.

The GCC patches have been posted a few times; they too are being modified in response to review comments. One piece that has not yet been posted is link-time optimization support, but it is coming soon. With luck, she said, all of this support will be merged in time for the GCC 10 release due in 2020. GDB support is also under discussion on the project mailing list; the GDB 8.4 release is being targeted for this work.

Zannoni closed with a look at where things go from here. There is a set of discussions planned for the Toolchains microconference at the 2019 Linux Plumbers Conference. There are, evidently, still optimizations that can be made to further reduce the size of CTF data. There is also a fairly significant gap in that backtrace support is not yet present for CTF. An expansion to languages other than C is on the horizon. Then, there is that perennial lowest priority for development teams: documentation. The "specification" for the format lives in a C header file for now; that will clearly need to change in the future.

[Your editor thanks the Linux Foundation for supporting his travel to the event.]

Comments (86 posted)

Mozilla's WebThings Gateway now available for Turris Omnia router

The "Web of Things" (WoT) is meant as a way to enable Internet of Things (IoT) devices to appear on the web. Mozilla's entry into the WoT world is the WebThings project, which consists of both a Framework API and a Gateway software distribution to host applications. On July 25, the project announced the Gateway 0.9 release with support for the Turris Omnia wireless home router.

Back in April, Mozilla announced that its "Things" project for IoT had graduated from an experiment into WebThings. That project is billed as "an open platform for monitoring and controlling devices over the web". At that time, the Gateway software only targeted Raspberry Pi devices, but the new release moves into the consumer-hardware world. In addition, support for the new Raspberry Pi 4 has been added.

The Gateway is based on the OpenWrt router distribution, which has been modified to add the IoT-specific functionality. It is currently somewhat limited in its functionality, however:

There are guides for installing Gateway for the Turris Omnia and for Raspberry Pi. Beyond media to use for the installation image (or to boot from for the Raspberry Pi), the main hardware requirement is for USB adapters to allow communication using the Zigbee and Z-Wave "smart home protocols". Which will be needed depends on what IoT-enabled devices are installed.

Once it has booted for the first time, the Gateway software will bring up a wireless network that users will connect to. After that, the first-time setup is performed from a web browser on a connected device. Mozilla runs a tunneling service that the user can sign up for, which will give them a host name in the mozilla-iot.org domain that can be used to reach the Gateway. The final piece of the initial configuration is to add a user account; after that, the WebThings Gateway User Guide provides information on configuring the router to talk to the IoT devices of interest.

The first step for adding devices is to put them in a mode that is receptive to a new pairing, one at a time, then to tell the Gateway web application to scan for them. Once they are recognized (and renamed to something that makes more sense to the user), there are a number of different options. The device state can be queried (e.g. is a door open or a light on) or changed, for example; some devices may require an add-on in order to access them. Users can also create a floor plan of their house to place icons of the devices in the right locations.

Beyond that, there is a rules engine where automated changes can be programmed. So if the user wants a certain light to go on or off at a specific time, for example, that can be done. The interface is icon oriented, which should make it easier for less technical users. There is also an experimental Smart Assistant feature that allows voice or typed commands like "turn on the kitchen light" to be handled. The voice data is sent to Google's voice assistant API; the text commands are handled locally on the Gateway device. It is not clear why the assistant is not using Mozilla's speech-processing engine.

New for version 0.9 is a Notifier add-on that will send an email or SMS text message based on rules that the user specifies, so motion sensor activity could trigger a text message, for example. Accompanying the Gateway release is the 0.12 release of the WebThings Framework. It has made some changes to the Web Thing API to more closely align it with the recent W3C WoT Thing Description draft.

Centralizing IoT handling on a system controlled by the user is an admirable goal. The IoT world has so far proven to be an insecure morass of competing lock-in plays, or so it seems to this cynical observer. Wresting control of the devices from the manufacturers and placing it in the hands of their owners seems like an excellent step forward. Hopefully Mozilla sticks with this project for the long haul and that it gets the community support that it surely deserves—and needs.

Comments (6 posted)

Security quote of the week

Comments (1 posted)

Kernel release status

Stable updates: 5.2.6, 4.19.64, 4.14.136, 4.9.187, and 4.4.187 were released on August 4, followed by 5.2.7, 4.19.65, 4.14.137, 4.9.188, and 4.4.188 on August 6.

Comments (none posted)

Freedombone 4.0 released

Comments (5 posted)

Red Hat Enterprise Linux 7.7 released

Comments (1 posted)

Distribution quote of the week

Comments (none posted)

FFmpeg 4.2 released

Comments (1 posted)

The GNU C Library version 2.30 is now available

Version 2.30 of the GNU C Library (glibc) has been released. New features include Unicode 12.1.0 support; wrappers for the getdents64(), gettid(), and tgkill() system calls on Linux; addition of a bunch of POSIX-proposed pthreads calls; protections for memory allocation functions so that they cannot cause ptrdiff_t overflows; and more, such as fixes for two security problems:

Full Story (comments: 15)

GNOME and KDE to co-host the Linux App Summit in November

Full Story (comments: none)

Knoll: Technical vision for Qt 6

Comments (16 posted)

FSFE releases the REUSE 3.0 copyright/licensing specification

Comments (none posted)

• DistroWatch Weekly (August 5)

• Lunar Linux Weekly News (August 2)

• SparkyLinux News (July 31)

• Ubuntu Server Development Summary (August 6)

• Ubuntu Weekly Newsletter (August 3)

• Emacs News (August 5)

• These Weeks in Firefox (August 2)

• GCC 9.1.1 Status Report (July 31)

• GCC 9.2 Status Report (August 5)

• What's cooking in git.git (August 2)

• What's cooking in git.git (August 6)

• Koha Community Newsletter (July)

• LLVM Weekly (August 5)

• LXC/LXD/LXCFS Weekly Status (August 5)

• OCaml Weekly News (August 6)

• Perl Weekly (August 5)

• PostgreSQL Weekly News (August 4)

• Python Weekly Newsletter (August 1)

• Racket News (August 5)

• Ruby Weekly News (August 1)

• This Week in Rust (August 6)

• Wikimedia Tech News (August 5)

• Fedora FESCO meeting minutes (August 5)

• Linux Foundation Technical Advisory Board minutes (March 7)

• Linux Foundation Technical Advisory Board minutes (April 4)

• Linux Foundation Technical Advisory Board minutes (June 6)

CFP Deadlines: August 8, 2019 to October 7, 2019

If the CFP deadline for your event does not appear here, please tell us about it.

Events: August 8, 2019 to October 7, 2019

If your event does not appear here, please tell us about it.

Alert summary August 1, 2019 to August 7, 2019