Leading items

Welcome to the LWN.net Weekly Edition for October 26, 2023

This edition contains the following feature content:

• Hyphens, minus, and dashes in Debian man pages: just because something looks like a hyphen doesn't mean it is actually a hyphen.
• Toward safer GNU C Library tunable handling: some proposed changes to the handling of glibc tunable parameters within privileged programs.
• mseal() and what comes after: a failed solution for a valid use case.
• Weighted interleaving for memory tiering: a proposed feature for maximizing overall system memory bandwidth utilization.
• Home Assistant: ten years of privacy-focused home automation: a look at this popular home-automation system a decade into its existence.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Hyphens, minus, and dashes in Debian man pages

It is probably fair to say that most Linux users spend little time thinking about the troff typesetting program, despite that application's groundbreaking role in computing history. Troff (along with nroff) is still with us, though, even if they are called groff these days, and every now and then they make their presence known. A recent groff change created a bit of a tempest within the Debian community, and has effectively been reverted there. It all comes down to the question of what, exactly, is the character used to mark command-line options on Unix systems?

Last July, Sven Joachim filed a bug report regarding a change in groff, and in how it renders man pages for terminals in particular. A change to the handling of the character often referred to as "hyphen", "minus", or "dash" ("-") made many man pages rather harder to work with. To understand the problem, it's worth noting that Unicode provides a plethora of similar characters, some of which are:

Name	Codepoint
Hyphen-Minus	002D	-
Hyphen	2010	‐
En Dash	2013	–
Em Dash	2014	—
Minus Sign	2212	−

There are many more — Unicode is nothing if not generous in this regard. The term "dashes" will be used to refer to this class of glyphs here.

The specified behavior of groff is that an ASCII "-" (Hyphen-Minus) in the input becomes a Hyphen in the output. If the desire is to use Hyphen-Minus in the output, then the input should use the sequence "\-" instead. If the author of a man page types "--frobnicate" as an option name, the output will read "‐‐frobnicate" (with Hyphen) rather than "--frobnicate" (with Hyphen-Minus). The two look the same, but there is a crucial difference. A user who searches for "--frobnicate" in a man page will not find it if the wrong type of dash is used and, if that user cuts-and-pastes an example with the wrong dash, it will not work.

As an example, one can try pasting these two lines into a shell:

    /usr/bin/echo --help
    /usr/bin/echo ‐‐help

The results from one will be rather more helpful than from the other. Use of the wrong type of dash can also break URLs and corrupt file names.

Developers of free software are, of course, diligent about writing man pages; they do the job promptly, take their time to get every detail right, and can be expected to use the right kind of dash in every situation, even though the output from using the wrong kind looks exactly the same. They will surely not be bothered by the fact that a format designed to document command-line options contains a trap whereby the failure to add backslashes silently introduces problems for users who are distant in time and space.

Shockingly, this turns out not to be the case, and Linux man pages are overflowing with unescaped dashes. Years ago, the Debian project tried to address this problem by adding a check to its Lintian tool that would issue a warning when unescaped dashes were used. That check was dropped in 2015, though, after Niels Thykier concluded that it was simply being ignored: "The tag has existed since 2004 (commit fb2e7de). To date there are still 2000 packages with the issue." Since then, there has been no warning shown to Debian developers when man pages contain unescaped dashes.

Given the prevalence of this problem, it would arguably make sense to apply a fix at the processing level. And, indeed, groff has, for many years, duly remapped the Hyphen-Minus character (and a few others) in the man-page macros, making dash characters simply work as many would expect. That helpful behavior ended with the groff 1.23.0 release in July:

The an (man) and doc (mdoc) macro packages no longer remap the -, ', and ` input characters to Basic Latin code points on UTF-8 devices, but treat them as groff normally does (and AT&T troff before it did) for typesetting devices, where they become the hyphen, apostrophe or right single quotation mark, and left single quotation mark, respectively. This change is expected to expose glyph usage errors in man pages. See the "PROBLEMS" file for a recipe that will conceal these errors. A better long-term approach is for man pages to adopt correct input practices

Problems were indeed exposed, and users began to complain; bugs were filed and the topic showed up on the debian-devel mailing list as well. G. Branden Robinson, the upstream maintainer of groff and author of this change, defended the new behavior:

Mapping all hyphens and minus signs to a single character, as people whose blood pressure spikes over this issue tend to promote as a first resort, is an ineluctably information-discarding operation. In my opinion, man page source documents are not the correct place to discard that information.

Among other things, the information being discarded by this change includes whether line-breaking is allowed; Hyphen-minus does not allow it, while Hyphen does.

Others disagreed with Robinson's position; Russ Allbery said:

My opinion is that the world of documents that are handled by man do not encode meaningful distinctions between - and \-, and man should therefore unify those characters.

Colin Watson, who maintains Debian's groff package, admitted that he had overlooked this problem when he updated Debian to the 1.23.0 release:

I was aware of the change, but it somehow fell off my list of things to make a positive decision about when packaging 1.23.0. I'm rather inclined to revert this by adding the rest of the recipe above to debian/mandoc.local (while I agree with the idealized typographical point being made, I have approximately negative appetite for the Sisyphean task of fixing an entire distribution's manual pages in practice).

A few weeks later, he said that his plan was to leave the change in place during the current Debian 13 ("Trixie") development cycle, but then to revert it prior to the pre-release freeze to avoid inflicting problems on Debian's users. That would, in theory, give developers time to fix as many of the problems as possible. After the discussion went on for a while, though, he changed his mind, stating that he was unwilling to have his inbox filled with this discussion for the next year. So the remapping of "-" has been reinstated into Debian's version of groff.

This little episode may well be repeated in other distributions as they catch up with the groff 1.23.0 release. It also is probably not finished within Debian. This situation brings together the problems of documentation writing, typographic correctness, and Unicode look-alike code points, all of which are fertile ground for disagreement. The hopes that removing the remapping in groff would lead to the fixing of all those man pages may have been dashed, but that does not bar another attempt in the future.

Comments (155 posted)

Toward safer GNU C Library tunable handling

When considering the interface provided by the GNU C Library (glibc), thoughts naturally turn to the programming interface as specified by POSIX, along with numerous extensions added over the years. But glibc also provides a "tunables" interface to control how the library operates; rather than being managed by a C API, tunables are set with the GLIBC_TUNABLES environment variable. Glibc tunables have been a part of a few security problems involving setuid binaries, most recently the "Looney Tunables" bug disclosed at the beginning of October. The glibc developers are now considering significant changes to tunable handling in the hope of avoiding such problems in the future.

The tunables mechanism allows various aspects of the library's behavior to be adjusted at run time. Many of the provided knobs relate to memory-management algorithms hidden behind malloc(), but there are others as well. The list of available tunables on any given system can be seen by running:

    /lib64/ld-linux-x86-64.so.2 --list-tunables

The library provides no public functions for setting or querying tunable values; instead, users can set them, before running a program, by setting GLIBC_TUNABLES. This variable is processed early in the startup routine, well before a program's main() function is called. So a program has little control over which tunables are applied to it; in a sense, glibc tunables allow the library's behavior to be tweaked behind a program's back.

The ability to do such tweaking raises some obvious red flags when the program involved has the setuid or setgid permissions set. Those programs constitute a security boundary within the system, so allowing users on the unprivileged side of the boundary to adjust behavior on the privileged side is going to be fraught. Glibc has long recognized that some tunables should cease to be tunable when a program is setuid or setgid; to enforce that restriction, each tunable is marked, within glibc, with a "security level". If a tunable is marked SXID_ERASE, it will be ignored in privileged programs and erased from GLIBC_TUNABLES entirely, with the result that it will not be present in any child processes created by that program. SXID_IGNORE, instead, causes the tunable to be ignored but left in place. Without one of those settings, the tunable will be honored in privileged programs.

Even with this mechanism in place, the intersection of tunables and privileged programs has proved to be a reliable source of security bugs. So the glibc developers are looking at ways of eliminating that intersection entirely. Siddhesh Poyarekar posted a patch set setting all tunables to SXID_ERASE, essentially disabling the tunables mechanism for privileged programs entirely, shortly after the latest vulnerability disclosure. Some discussion followed and, one week later, Adhemerval Zanella proposed a deeper set of changes aimed at eliminating this class of problems.

Step one in Zanella's series is to eliminate support for the special /etc/suid-debug file. If that file exists (it can be empty), then the glibc.malloc.check tunable can turn on malloc() checking, even in privileged programs. Normally this feature is disabled in such programs because it can generate output to the standard error stream, which could be abused by an attacker. This debugging can be useful, so the check for this file was added as a way of giving the system administrator a way to make it available for privileged programs. It turns out, though, that the /etc/suid-debug check hasn't worked since the 2.34 release, when memory allocation was pushed out into a separate library; rather than fix this feature, this series just removes that support.

The next step is to ignore all tunables in privileged programs, essentially setting them all to SXID_IGNORE. A later patch then adds GLIBC_TUNABLES to the list of environment variables that are not passed through to child processes, effectively changing that setting to SXID_ERASE via a different means. There is another set of environment variables that can change many of the malloc() tunables as well; for example, MALLOC_MMAP_THRESHOLD can be used to change glibc.malloc.mmap_threshold. All of those variables are added to the "insecure variables" list as well.

That eliminates the handling of tunables from privileged programs, but the patch set makes another set of changes as well. Current glibc versions are relatively forgiving about invalid GLIBC_TUNABLES settings; a number of patches in this series put an end to that, aborting tunable parsing and emitting a warning when badly formatted tunable settings are found. Finally, there is a patch that causes GLIBC_TUNABLES to be parsed in place rather than copied to a separate string. That change is intended to avoid any problems that might come about as the result of memory-allocation failures; this processing happens before glibc's memory management is initialized, so allocation is trickier at this stage.

The end result should be a distinct reduction in tunable-related bugs and vulnerabilities in future glibc releases — if the series is accepted, of course. This change could come at a cost for users who have a reason to set these tunables for privileged programs, should any such users exist. But the glibc documentation is explicit in its statement that tunables are not considered part of the ABI and can change at any time. In this case, most users may well conclude that these changes are overdue and the cost is well worth paying if it results in fewer vulnerabilities in the future.

Comments (17 posted)

mseal() and what comes after

Jeff Xu recently proposed the addition of a new system call, named mseal(), that would allow applications to prevent modifications to selected memory mappings. It would enable the hardening of user-space applications against certain types of attacks; some other operating systems have this type of feature already. There is support for adding this type of mechanism to the Linux kernel as well, but it has become clear that mseal() will not land in the mainline in anything resembling its current form. Instead, it has become an example of how not to do kernel development at a number of levels.

Xu described the new system call's purpose as:

Memory sealing additionally protects the mapping itself against modifications. This is useful to mitigate memory corruption issues where a corrupted pointer is passed to a memory management syscall. For example, such an attacker primitive can break control-flow integrity guarantees since read-only memory that is supposed to be trusted can become writable or .text pages can get remapped.

The target user for this functionality is the Chrome browser which, among other things, includes a just-in-time (JIT) compilation engine for JavaScript code. Since it generates executable code on the fly, JIT compilation must be done with care, lest it create (and run) the wrong kind of code. As described in this blog post by Stephen Röttger, a lot of effort has gone into control-flow integrity and preventing the JIT system from becoming a tool for an attacker. If, however, an attacker is able to somehow force a memory-management system call that changes memory permissions, all bets are off. Thus, the Chrome developers would like to have a mechanism that puts those system calls off-limits for specific regions of memory, hardening the browser against that sort of attack.

The cover letter notes that mseal() is similar to mimmutable(), which was added recently to OpenBSD. The prototype for the proposed system call is quite different from mimmutable(), though:

    int mseal(void *addr, size_t len, unsigned int types, unsigned int flags);

The range of memory to be affected is indicated by addr and len. The flags must be passed as zero, and types controls which system calls will be blocked on that address range:

• MM_SEAL_MPROTECT: mprotect() and pkey_mprotect()
• MM_SEAL_MMAP: mmap()
• MM_SEAL_MUNMAP: munmap()
• MM_SEAL_MREMAP: mremap()
• MM_SEAL_MSEAL: future mseal() calls

Linus Torvalds was quick to object to the patch series, saying: "I have no objections to adding some kind of 'lock down memory mappings' model, but this isn't it". He had a number of complaints about the details of the implementation, but he later made it clear the design of the system call was wrong. Blocking munmap(), for example, makes little sense if other operations that can result in the unmapping of addresses (mmap() and mremap(), for example), are still allowed. The effort that was put into only blocking operations from specific system calls, he said, was overtly wrong; if unmapping a range of memory (for example) is blocked, it must be blocked from all directions or the protection provided will be illusory.

Matthew Wilcox questioned the complexity of the interface, suggesting instead that a couple of flags added to mprotect() would suffice. A memory region, he said, should either be immutable (with the possible option of further reducing access) or not, without regard to which system call was used. He later added:

This is the seccomp disease, only worse because you're trying to deny individual syscalls instead of building up a list of permitted syscalls. If we introduce a new syscall tomorrow which can affect VMAs, we'll then make it the application's fault for not disabling that new syscall. That's terrible design!

The conversation even brought about a rare appearance on linux-kernel by OpenBSD maintainer Theo de Raadt, who agreed with Torvalds and suggested that Linux should simply add mimmutable() rather than reinventing that functionality in a more complex form. Torvalds was amenable to this idea, though he suggested adding a flags argument for future changes — an idea that de Raadt disliked. That reflects the fact that OpenBSD controls its user space, so it can add a flags parameter later if the need arises; Linux has no such luxury, so that parameter must be present from the beginning if it is to exist at all.

Xu, instead, resisted the idea, prompting a typical (if relatively mild) de Raadt response. Indeed, Xu continued to cling to his proposed design despite the comments he had received, leading to a somewhat exasperated post from Wilcox, who tried to direct the conversation toward what the patch series is actually trying to accomplish:

Let's start with the purpose. The point of mimmutable/mseal/whatever is to fix the mapping of an address range to its underlying object, be it a particular file mapping or anonymous memory. After the call succeeds, it must not be possible to make any address in that virtual range point into any other object.

The secondary purpose is to lock down permissions on that range. Possibly to fix them where they are, possibly to allow RW->RO transitions.

With those purposes in mind, you should be able to deduce for any syscall or any madvise(), ... whether it should be allowed.

Xu, Wilcox concluded, needed do a better job of listening to the developers who were trying to help him.

At this point, it is clear that mseal() will not enter the kernel in anything like its current form. That leads to the question of what should be done instead. Röttger jumped into the conversation to point out that a pure mimmutable() solution does not do everything that the Chrome developers would like to see; they have cases where they want to prevent unmapping, but still need to be able to change memory protections with mprotect(). De Raadt described that case as "partial sealing" that means that the memory in question is not actually protected.

There will presumably be some sort of follow-up proposal that maintains that capability while removing the more complex options provided by mseal(). But whether that proposal will be mimmutable() or some variant thereof remains to be seen.

One can point to a number of things that went wrong here. The original proposal was seen by many as an implementation of what the Chrome developers said they wanted without looking deeply at what the real requirements (for Chrome and any other potential users) are. Google has no shortage of experienced developers who could have reviewed this submission before it was posted publicly, but that does not appear to have happened, with the result that a relatively inexperienced developer was put into a difficult position. Feedback on the proposal was resisted rather than listened to. The result was an interaction that pleased nobody.

Despite all of that, there is a use case here that everybody involved appears to see as legitimate. So it is just a matter of finding the right solution to the problem, and hopefully that problem is better understood now. If the next attempt looks a lot more like mimmutable() and reflects the feedback that has been given, the kernel may yet get the sealing capability that addresses the Chrome use case and provides for wider user-space hardening as well.

Comments (21 posted)

Weighted interleaving for memory tiering

The kernel has, for many years, had the ability to control how memory allocation is performed in systems with multiple NUMA nodes. More recently, NUMA nodes have also been pressed into service to represent different classes of memory; those nodes are now organized into tiers according to their performance characteristics. While memory-allocation policies can control the placement of pages at the NUMA-node level, the kernel provides no way to connect those policies with memory tiers. This patch series from Gregory Price aims to change this situation by allowing allocations to be placed across tiers in a weighted manner.

The days when computers simply had "memory" have passed. Systems now can contain numerous types of memory with wildly different performance profiles. Persistent memory can be available in large quantities, but it is relatively slow. CXL memory may be faster, though still not as fast as normal DRAM, and it can come and go during the life of the system. High-bandwidth memory can be faster than normal DRAM. Devices, too, can make their own memory available on the system's memory bus. Each memory provider is represented in the system as one or more CPU-less NUMA nodes.

Tiering is the process of grouping these nodes into levels (tiers) with similar performance, then working to place memory optimally in each tier; it is a work in progress. The kernel only gained a formal concept of memory tiers in the 6.1 release, when this series from Aneesh Kumar K.V. was merged. The kernel now assigns each node an "abstract distance" reflecting its relative performance; by default, all nodes have an abstract distance of 512. Memory with a lower abstract distance than that is expected to be faster than ordinary DRAM, while a higher abstract distance indicates slower memory. Notably, the distance is set by the driver that makes the memory available to the system; it is not subject to direct administrator control.

As memory is initialized, the kernel will organize it into distinct tiers, with the tier number for any given node obtained by dividing its abstract distance by 128. Normal memory, with its default abstract distance of 512, ends up in tier four as a result. There is, thus, room for four tiers of memory that is faster than DRAM, and a vast number of slower tiers.

The kernel can use tiering to make decisions about page placement; quite a bit of work has gone into trying to figure out how to move pages between the tiers. When pages are unused, they can be demoted to slower tiers rather than being reclaimed entirely; that is a relatively easy decision that fits into existing memory-management practices. Deciding when pages should be promoted back to faster memory, though, is a bit trickier. Importantly, this work has been focused on moving memory to the correct tier after it has been in use for a while and its relative busyness can be assessed.

Price's patch set builds on the tiering mechanism, but is aimed at a different problem: what is the optimal original placement for pages when they are first allocated? In particular, it addresses the practice of NUMA interleaving, wherein allocations are spread out across a set of NUMA nodes with the purpose of getting consistent performance from any of a set of the CPUs on the system. NUMA interleaving has worked for years, but it is relatively simple; it spreads pages equally across the set of nodes provided in the memory policy set by the application. Interleaving has no concept of tiers and no ability to bias allocations in any specific direction.

The patch series adds the concept of a "weight" to each NUMA tier. Weights are relative to each CPU, so the weight of tier 2 as seen from CPU 0 may be different than the weight of that tier seen from CPU 10. These weights determine how strongly allocation decisions should be biased toward each tier; a higher weight for a given tier results in more pages being allocated on nodes contained within that tier. Unlike the abstract distance, the tier weights are controlled by the administrator; a command like:

    echo 0:20 > /sys/devices/virtual/memory_tiering/memory_tier2/interleave_weight

would set the weight of tier 2, as seen by CPU 0, to 20. By default, all tiers have a weight of one.

These weights are used by the NUMA interleaving code when pages are allocated. If a system has two tiers, tier 4 (where normal DRAM ends up) and tier 8 (with slower memory), and the weights of those tiers for the current CPU are set to 20 and 10 respectively, then interleaved allocations will place twice as many pages on tier 4 as on tier 8. It's worth noting that, while the application specifies its NUMA allocation policy (which nodes it wants to allocate memory on), the weights are solely under the control of the administrator.

While promotion and demotion between tiers is aimed at putting the most-used pages on the fastest memory, weighted allocation has a different objective — necessarily, since the kernel cannot know at allocation time which pages will be the most heavily used. The intent, instead, is to try to obtain an optimal split of memory usage across memory controllers, maximizing the bandwidth utilization of each. Moving some pages from faster memory to a slower tier can result in an overall performance increase, even if the moved pages are actively used, if the end result is to keep both tiers busily moving data.

According to Price, this approach works:

Observed a significant increase (165%) in bandwidth utilization with the newly proposed multi-tier interleaving compared to the traditional 1:1 interleaving approach between DDR and CXL tier nodes, where 85% of the bandwidth is allocated to DDR tier and 15% to CXL tier with MLC -w2 option.

(MLC refers to Intel's proprietary memory latency checker tool).

This is the second version of this patch set (the first was posted by Ravi Jonnalagadda in late September). There have been a number of comments about the details of the implementation, but there does not appear to be any opposition to the concept in general. Since all tiers start with an identical weight, there will be no changes to NUMA interleaving unless the administrator has explicitly changed the system's configuration. There would thus appear to be little reason for this work not to advance once the details have been taken care of.

Comments (2 posted)

Home Assistant: ten years of privacy-focused home automation

Many home-automation devices come with their own mobile app or cloud service. However, using multiple apps or services is inconvenient, so it's (purposely) tempting to only buy devices from the same vendor, but this can lead to lock-in. One project that lets users manage home-automation devices from various vendors without lock-in is Home Assistant. Over its ten-year existence, it has developed into a user-friendly home-automation platform that caters to both technically inclined and less tech-savvy people.

In 2012, while finishing his master's thesis, Paulus Schoutsen purchased a Philips Hue starter kit consisting of a hub and three light bulbs. When he discovered that the hub had a local API accessible over the network, he wrote a Python script to control the light bulbs. Then he wrote a script that would turn on the lights at sunset, but soon he realized that the bulbs would turn on when no one was home. To address this, he developed a script that logged into his WiFi access point to detect connected phones and used this presence-detection as a condition in his light-automation script.

Recognizing its potential usefulness for others, Schoutsen refactored the code and made it configurable so that users would not have to write Python code themselves. On September 17, 2013, he released the first version of his Home Assistant project to GitHub. He started promoting it on the r/homeautomation subreddit whenever people sought advice on automating their homes; the project gained momentum as the community expanded and more developers contributed.

Today, Home Assistant is the second most actively developed project on GitHub. According to "The state of open source on GitHub" from 2022, Home Assistant ranked second in terms of the number of contributors (13,500), had the highest contributor growth among GitHub projects (3,300 new contributors in a year), and was the second most popular project for first-time contributors (2,600 developers making their first contribution on GitHub).

The core idea behind Home Assistant remains the same as its initial release: an open-source (Apache-licensed) hub to control various home-automation devices. It supports over 2,500 integrations that allow it to connect with devices and external services. Home Assistant can be installed on a Raspberry Pi or any other computer, preferably a dedicated system to provide a stable and reliable platform for home automation.

From tinkerers to non-technical users

Although Home Assistant initially targeted technically inclined users who enjoyed tinkering, the project's focus has shifted toward making home automation accessible to the general public. In 2017, Pascal Vizeli recognized the need for a simplified installation process after family members approached him for guidance. This led to what is now known as Home Assistant Operating System (formerly known as "Hass.io"). It is based on Buildroot, a tool for creating embedded Linux systems, and it runs Docker as a container engine. Vizeli also developed Home Assistant Supervisor, which manages Home Assistant Core (the actual Home Assistant application) and Home Assistant add-ons in separate containers.

Home Assistant Operating System became the recommended installation method for Home Assistant so that users no longer needed to run a series of command-line instructions to install the software. Instead, they could simply download an image, start it on a supported computer or a virtual machine, and use the web-based configuration wizard. Users no longer need to manage a Linux system: Home Assistant Operating System can be easily updated from the web-based interface. This broadened the appeal of Home Assistant beyond the small community of technically inclined home-automation enthusiasts.

Similarly, the introduction of add-ons made life easier for many users. A Home Assistant add-on is essentially a Docker container with another application running in it. In 2017, Franck Nijhof began developing add-ons to integrate a wide range of applications into Home Assistant. Users can now easily install the Mosquitto MQTT broker, an ACME client for obtaining TLS certificates from Let's Encrypt, a DynDNS client for updating public hostnames for dynamic IP addresses, a terminal or editor integrated into the web interface, and much more. These add-ons can be installed and configured directly from Home Assistant's web interface.

Home Assistant has also made strides in simplifying configuration. In its initial years, configuration was mainly done in YAML files. When LWN.net reviewed Home Assistant three years ago, the article showed a couple of YAML code examples. However, in recent years, more and more parts of the configuration are possible using the graphical interface. Although YAML is still required for some advanced features, Home Assistant has transitioned away from YAML-based configurations for integrations that communicate with devices and/or services.

This shift initially proved controversial among technically inclined users who favored code-based configurations. However, the Home Assistant developers aimed to attract a broader range of users by prioritizing ease of use. The ultimate goal is to make privacy and independence from cloud-based home-automation systems accessible to everyone.

But focusing on this user-friendliness makes the system less flexible for tinkerers who want to make their own decisions. For example, for the last few years Home Assistant has supported the two most recent upstream Python versions. This policy was changed earlier this year to support only the latest upstream version. Because 96% of users run one of the project's pre-built containers, the Home Assistant developers only need to develop and test their code against a single version that gets shipped in the container image. However, that forces the 4% of users installing Home Assistant Core in their own Python environment to upgrade their Python version at least once a year.

Corporate backing

For many years, Home Assistant had a biweekly release schedule, but the developers started to burn out from this frantic pace. So, to ensure the sustainability of Home Assistant's development, Schoutsen, Vizeli, and Ben Bangert established a company called Nabu Casa. Initially, Vizeli became the only one working full-time on Home Assistant at Nabu Casa. Five years later, the team consists of almost 30 people who are publishing a new Home Assistant release every month.

Nabu Casa is financially stable without external investors; it derives its revenue from subscriptions to Home Assistant Cloud, which is a service to provide secure remote access to the user's Home Assistant installation. Home Assistant Cloud operates as a proxy server using end-to-end encryption.

When users enable Home Assistant Cloud on their instance, the latter connects to one of the proxy servers. When the user later visits the instance's URL in a web browser, the proxy server identifies the hostname from the incoming request using the Server Name Indication (SNI) extension of the TLS handshake. It then forwards the request to the corresponding Home Assistant instance. The source code for the SniTun proxy and the hass-nabucasa cloud integration in Home Assistant is available under the GPLv3.

In 2021, Nabu Casa took over development of ESPHome, a build and deployment system that allows users to configure ESP32 and other microcontroller boards without having to program. Users simply write YAML files that describe the components connected to the board's pins. ESPHome then builds the firmware based on this configuration, enabling LEDs, buttons, sensors, and more to communicate with Home Assistant over a WiFi connection. Since Nabu Casa took over ESPHome's development, the project has adopted the same monthly release cycle as Home Assistant. And last year, Nabu Casa employed Michael Hansen to keep developing his open-source voice assistant Rhasspy.

The Open Home

The Home Assistant project always has been about more than just software. It is driven by Schoutsen's vision of The Open Home, which is built upon three pillars: privacy, choice, and sustainability.

Privacy means that a home should be a safe space where individuals can be themselves. In the vision of the Open Home, devices function locally without requiring a connection to an external server. No information about local actions, such as turning on a light bulb or adjusting a thermostat, should be transmitted beyond the home environment. If a product offers cloud connectivity, it should not be necessary for the basic operation of the device and should not be enabled by default. Many of Home Assistant's integrations use local APIs for device interaction.

Choice involves allowing users to select the devices that best meet their needs without restrictions. Users should not face lock-in, and manufacturers should not limit access to device data. Home Assistant enables users to mix devices from various manufacturers and use different home-automation protocols at the same time, including Zigbee, Z-Wave, Matter, Bluetooth, and more. All device data is collected locally by Home Assistant and doesn't leave the user's home. But, if users choose to incorporate devices that rely on cloud-based APIs, they have the freedom to do so using Home Assistant. Nabu Casa also offers an easy way to control Home Assistant with the cloud-based voice assistants from Google and Amazon for those who want this.

Sustainability revolves around creating a smart home that stands the test of time. Devices installed in a home should continue to operate seamlessly for many years. The sustainability goal was improved with the introduction of the energy dashboard in 2021. This feature provides graphs displaying energy use, solar panel production forecasts, and more. With this information, users can optimize the charging of electric cars or bicycles to those times when enough solar power is being generated.

The Home Assistant developers actively advocate for the principles of the Open Home when engaging with home-automation vendors. For example, when the Philips Hue hub started blocking third-party light bulbs, Schoutsen discouraged people from buying into the Philips Hue ecosystem. A few days later, Philips reversed its decision and announced a software update to allow third-party light bulbs again. When Insteon abruptly shut down its cloud servers, Schoutsen provided guidance on how users could still talk to their devices locally and announced an improved Insteon integration in Home Assistant. Most recently, when Signify (the new name for Phillips Lighting) announced that it would soon start forcing users to create an account and upload their data to the cloud, Home Assistant's founder contacted Signify to voice his concerns.

Conclusion

Home Assistant is more than just a home-automation-software project. It represents a movement that advocates for privacy, freedom of choice, and sustainability in every connected home. The project has an active user community, offering a forum, a Discord chat server, and the Home Assistant subreddit to get in touch with other users. There's also extensive documentation. While it may not yet be as user-friendly yet as some closed ecosystems with tightly integrated devices, Home Assistant is steadily progressing with each release.

Comments (30 posted)