LWN.net Weekly Edition for February 13, 2020

Welcome to the LWN.net Weekly Edition for February 13, 2020

This edition contains the following feature content:

• Enabling the persistent journal in Debian: a discussion on journaling makes it clear that Debian still hasn't fully come to agreement over systemd features.
• The rest of the 5.6 merge window: what the latter part of the 5.6 merge window brought into the mainline.
• Better tools for kernel developers: the posting of a simple but useful tool highlights a gap in support for the kernel community.
• Kernel operations structures in BPF: what looks like a new networking feature turns out to be far more generic and powerful.
• Lua and Python: the Lua community considers ways to become a bit more like Python.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Enabling the persistent journal in Debian

It seems unlikely that anyone on any "side" of the systemd war that has raged in Debian over the last few years thought that the results of the recent general resolution (GR) vote ended the matter. The vote showed a clear preference for moving ahead with systemd as the preferred init system, though it was far from any kind of landslide—there were definitely plenty of voters who would have preferred a different outcome. It was a complicated GR, with a wide spectrum of options, but at this point, the project as a whole has spoken. Actually implementing some of the changes that the GR enabled may not have the smooth path that some might have hoped for, however.

On February 1, Michael Biebl posted a message to the debian-devel mailing list noting that he had fixed a wishlist bug (from 2013) by enabling the systemd persistent journal. Prior to that, journald would log to the non-persistent /run/log/journal directory by default and rsyslog would create the persistent text log files in /var/log. The change to the Debian systemd package would create the /var/log/journal directory where journald will store its persistent binary log files. That way, the journals will still be available after a reboot.

The message said that new installs and upgrades of the systemd package would create the directory, but it also included instructions on how to revert to the existing behavior; further upgrades to the systemd package will respect that choice. Beyond that, though, running both the persistent journal and rsyslog means that the log files are effectively stored twice on disk, so Biebl may ask the Debian ftp-masters to lower the priority of rsyslog so that it is not installed by default for the upcoming Debian 11 ("bullseye") release. Those who want to have a different system logger can add it after the initial install, of course.

Steve McIntyre replied that the change to the systemd package was fine for new installs, but that he would rather not see upgrades of the package add the persistent journal. "Those people with existing logging setups will be surprised by this." Didier "OdyX" Raboud disagreed but suggested that the change be prominently announced. Biebl thought that it could be surprising not to add the functionality on an upgrade:

Moreover, there is some relevant history from a Debian downstream that has already made the switch to the persistent journal. Biebl asked Dimitri John Ledkov to comment on how the transition went for Ubuntu, which enabled the persistent journal on installation or upgrade in late 2017. Ledkov said that the process went quite well overall, with few real problems. A large chunk of the Ubuntu user base now has the change since "systems installed with / upgraded to 18.04 LTS overwhelmingly have persistent journal enabled by default across the board". Biebl agreed with Raboud that documenting the change is important, so he filed a bug to that effect.

rsyslog default

The possible switch away from rsyslog by default concerned Dmitry Smirnov, however. "I think that replacing rsyslog with journald is two steps back in regards to functionality and flexibility." He explained that rsyslog has more features than journald and is more mature, thus more stable. He felt that changing the default system logger "yields no benefits to say the least".

Russ Allbery agreed with Smirnov's thoughts about rsyslog, but pointed out that the change would just be for the system default.

There are some minor benefits to the switch as well, Allbery said:

Smirnov was unconvinced that "those benefits are enough to justify replacing a very solid and reliable logging system", however. But, as Allbery said, journald is already upstream of rsyslog today, so the default is to run two logging daemons, which seems somewhat suboptimal; he is not sure that it makes sense to keep doing so, especially with the addition of the persistent journal. He did acknowledge that the switch might require some relearning:

There was some additional grumbling about switching to the journalctl command to look at log entries, which some see as a regression in functionality; it is certainly different than using familiar tools on text log files for those who do not change the default, but only if rsyslog is dropped from the default install. The rsyslog option is not going away, as Allbery pointed out. In addition:

GR consequences

Inevitably, some complaints about the GR outcome and what it means going forward were aired. The result of the vote may not have necessarily dictated the change in default for Debian as a whole, at least directly, but dropping the priority of rsyslog was only mentioned as a possibility. Meanwhile, Debian project leader Sam Hartman said that the outcome of the vote has already had a positive impact on the community:

But a suggestion that systemd opponents leave Debian en masse over the results of the GR was met with a number of responses, some perhaps less well-considered than others. Hartman's reply was typically even-keeled; he had previously acknowledged the unhappiness and frustration the GR results might engender in some, but the call to leave seemingly went too far. "It sounds like you were suggesting that people should leave as a way of pressuring or punishing the project." That is not the kind of community that Hartman is hoping and aspiring for; there are options for those who are unhappy with the outcome:

That message is worth reading in its entirety. Hartman empathizes with those who are on the "outside", even though he has determined that systemd is the best choice—for now. He simply asks that those opposed respect the other project members and the Debian project itself. "If you don't try to burn the house down on the way out, we'll be happy to see you again later if your needs or Debian change."

The problems with Debian and systemd have been going on for quite some time now; it's been almost exactly six years since the Debian technical committee voted to make systemd the default for the "jessie" release. That did not really end the matter, obviously, and the recent GR outcome will not either—at least completely. The trend, however, looks like things are moving in a direction where the rift is starting to heal—or opponents are moving on. Both are good for the Debian community as a whole. Enabling the persistent journal is a pretty minor change in the grand scheme of things; overall, it was met with constructive criticism or agreement. One hopes that whatever other systemd-derived changes are coming over the next months or years can garner a similar response—perhaps even without the need to re-litigate the "systemd question" yet another time.

Comments (51 posted)

The rest of the 5.6 merge window

Linus Torvalds released the 5.6-rc1 prepatch and closed the merge window on February 9; at that point, 10,780 non-merge changesets had been pulled into the mainline repository for 5.6. That is substantially less than recent development cycles (14,350 for 5.5, 14,619 for 5.4), but is similar to what was going on at this time last year (10,843 for 5.0-rc1 in January 2019). About 6,000 of those changes were pulled since the first 5.6 merge-window article was written; read on for what was included in those changes.

Architecture-specific

• The Intel memory protection extension feature, merged in 3.19, has been removed. This feature failed to take the world by storm, and support for it has been removed from the GCC compiler.
• The RISC-V architecture has gained support for the KASAN address sanitizer.

Core kernel

• The last users of the 32-bit time_t type have been fixed, and the associated types have been removed from the kernel. That means that, modulo a few loose ends, the kernel is now year-2038 ready. User-space work will still be required, and applications on 32-bit systems will need to use an updated C library. See this merge commit for more details.
• The io_uring subsystem has gained many more supported operations; see this article for details.
• The pidfd_getfd() system call, which allows a process to extract an open file descriptor from another process, has been merged.
• The "bootconfig" mechanism allows the kernel to receive, at boot time, a file containing command-line options; its intended use appears to be for tasks like setting up boot-time kprobes. The bootconfig tool, found in the tools directory, can be used to add one of these files to an initramfs image. This commit contains documentation for this feature. There is also a new set of options to control boot-time tracing; see this commit for documentation.
• A change to how pipe waiting works promises to speed up parallel kernel builds — always a kernel developer's favorite workload — but can also trigger a bug with old versions of GNU Make.

Filesystems and block I/O

• The F2FS filesystem has gained compression support; see this commit for some more information.
• The new softreval NFS mount option allows attribute revalidation to succeed from cached values should the server go down. From the changelog: "The use case for this option is for ensuring that we can still (slowly) traverse paths and use cached information even when the server is down."
• NFS mounts over UDP are now disabled by default.
• NFS v4.2 server-to-server file copies are now supported.
• The ZoneFS filesystem, which facilitates low-level access to zoned block devices, has been merged. See this commit for more information.

Hardware support

• Clock: Qualcomm MSM8998 multimedia clock controllers, Qualcomm SC7180 clock controllers, Qualcomm IPQ6018 global clock controllers, and NXP IMX8MP CCM clock controllers.
• Graphics: Boe Himax8279d panels, R-Car DU color-management modules, Xinpeng XPP055C272 panels, Leadtek LTK500HD1829 panels, and Sony ACX424AKP DSI command-mode panels.
• Miscellaneous: Rockchip image signal processing units, TI TPS61050/TPS61052 LED controllers, ROHM BD71828 Power Management ICs, Qualcomm WCD9340/WCD9341 codecs, SGI IOC3 PS/2 controllers, and Mediatek system companion processors.
• PCI: Broadcom Brcmstb PCIe host controllers and Intel Gateway PCIe host controllers.

Memory management

• There is a new prctl() operation called PR_SET_IO_FLUSHER. Its purpose is to flag a process that is part of the memory-reclaim I/O path and which should thus not be throttled when memory constraints get tight. Within the kernel, it works by setting the PF_MEMALLOC_NOIO and PF_LESS_THROTTLE flags in the task structure.
• The dma-buf heaps subsystem, derived from the Android ION allocator, has been merged at last.

Security-related

• The /dev/random blocking pool has been removed. This change makes /dev/random behave more like /dev/urandom, in particular preventing it from blocking on available entropy once the pool has been initialized.

Virtualization and containers

• Linux guests running under VirtualBox can now mount folders exported by the host.

Internal kernel changes

• The pin_user_pages() API has been merged, allowing the kernel to better track user-space pages that have been pinned into RAM. The full infrastructure for making use of this information is still to come, though.
• The build system now tests that all UAPI header files include #ifdef guards to prevent them from being included twice.
• There is a new API that can be used to generate synthetic trace events in the kernel; see this commit for documentation.

The kernel development community now has until late March or early April, when the final 5.6 release is expected, to find and fix the bugs introduced by all of that work.

Comments (1 posted)

Better tools for kernel developers

By many accounts, the kernel project uses outdated tooling, far behind the state of the art that Kids Today tend to favor. The kernel's workflow has worked well (enough) for years, but there are signs that it may not be sustainable indefinitely. As a result, there has been an ongoing conversation about improving the kernel's workflow, but little has changed so far. The posting of a simple tool called get-lore-mbox is a sign that the rate of change may be about to increase.

The kernel project's reliance on email strikes many as quaint and antiquated. It may indeed partly be a natural outcome of the aging nature of the kernel community; many of the developers there, especially in the important maintainer positions, got started well before tools like web-based Git forges existed. Indeed, some of them got started using punch cards and may still be unconvinced of the virtues of, say, text editors. But the truth of the matter is that there are a number of good reasons for the kernel community's continued reliance on email; there is little else that can handle a community of that size and diversity.

So, while it seems that the future of email (as opposed to, say, proprietary services like Gmail) is uncertain at best, the path toward a replacement in the kernel community is unclear. Developers will have to be convinced that any new tools will make their lives better, not worse; busy maintainers have little patience for "improvements" that slow things down.

The get-lore-mbox tool, posted by Konstantin Ryabitsev is not the product of a vast development effort; it adds up to about 500 lines of Python code. Its core function is simple: given the message ID for any email in a thread of interest, it will download the entire thread from the lore.kernel.org archive into a local mbox file. That alone can be useful for anybody who wants to catch up on an email discussion that may have flown by, but there are some other useful features:

• With the -a option, only the actual patch series at the heart of the thread will be downloaded; it will be sorted and stored in a form suitable for passing directly to git am. Any useful tags posted as replies (Acked-by, for example) will be automatically added to the relevant patches.
• The -t option will cause tags posted in replies to the series cover letter to be applied to all patches in the series.
• If the thread contains multiple versions of a patch series, the latest one will be stored; the -v option can be used to request a specific version if need be.
• If no message ID is provided, it will read a message from the standard input and extract the ID from there, making single-keystroke operation possible in a number of mail readers.

Kernel maintainers tend to spend a lot of time reading through email threads, organizing the patches found therein, applying tags, etc. The initial response to this tool (including on the private kernel.org users list) has made it clear that they have been wanting something like this for a long time. The developer time that will be saved far outweighs the time it takes to write a gadget like this, but nobody has ever created such a thing.

One might well wonder why this tool didn't appear until now. It would seem that there were two things that needed to happen before this work could be done. One of those was the creation of a reliable archive for kernel mailing lists that could be easily queried by a program. Reliable archives have been scarce in general, and a single archive for all kernel-related lists didn't exist at all; without that, there is nothing for a tool like get-lore-mbox to obtain messages from. The lore.kernel.org archive has fixed that problem; it now seems like an indispensable part of the kernel workflow, but it's worth remembering that it is less than two years old.

Back in 1975, Fredrick Brooks wrote in The Mythical Man-Month that an effective programming team should be organized like a surgical team, with a set of people filling specific roles. The kernel certainly has its surgeon at the top, complete with many of the traits that such professionals are known to display. Brooks said that every team should have two secretaries; the kernel community seems to be able to get by without those. But he also said that each team needs a toolsmith, focused on creating the tools that the team needs.

The kernel project has long lacked a toolsmith. It doesn't even have somebody who can collect the tools created by maintainers for their own needs and make them suitable for more general use. As a result, maintainers often do without, or they use tools they threw together themselves when they really needed to be doing something else; that causes the community to lose a lot of opportunities for better productivity overall.

One of the open secrets in the free-software world is that, while some areas of development are well supported by companies, others tend to languish. Both within the kernel and beyond it, there are many pressing needs that, it seems, are nobody's problem, so nobody is funded to work on them. The costs of this neglect are many: burned-out developers trying to keep projects going in their spare time, security vulnerabilities, missing tests and documentation, and a lack of tools needed by the community as a whole, among others.

If only there were a successful industry consortium that could focus some attention (and resources) on problems of this nature. For the kernel there is one, in the form of the Linux Foundation, which has indeed supported the kernel community in a number of ways, including employing a few key developers and taking over maintenance of the kernel.org system after the 2011 compromise shone a spotlight on the consequences of neglect there. More recently, the Linux Foundation has supported the creation and operation of lore.kernel.org — and the development of get-lore-mbox as well.

The kernel workflow discussions have made it clear, though, that there is a lot more work to be done in this area. The kernel's processes and tools have suffered from a lack of attention for many years; that is partly a result of the fact that, on the surface, things are working smoothly. The kernel community has managed to organize itself well and functions at a level that few other projects can match. When the kernel community has found the time to focus on tools — Git being the primary example — the effects have rippled across the development community as a whole. But reaching that point has required going through a number of crises when the kernel project's processes proved unable to support the community.

It would be a great thing if the next big transition in the kernel's development process could happen without the "crisis" part. If sufficient effort and resources go into the tools that the kernel community will need in the coming years, that might just be possible. That suggests that the work being done at the Linux Foundation should not just continue, but grow into something bigger. Supporting this work (which requires getting its member companies to support it) is one of the best things that the Linux Foundation can do to support the kernel project as a whole.

Comments (130 posted)

Kernel operations structures in BPF

One of the more eyebrow-raising features to go into the 5.6 kernel is the ability to load TCP congestion-control algorithms as BPF programs; networking developer Toke Høiland-Jørgensen described it as a continuation of the kernel's "march towards becoming BPF runtime-powered microkernel". On its face, congestion control is a significant new functionality to hand over to BPF, taking it far beyond its existing capabilities. When one looks closer, though, one's eyebrow altitude may well increase further; the implementation of this feature breaks new ground in a couple of areas.

The use case for this feature seems clear enough. There are a number of such algorithms in use, each of which is suited for a different networking environment. There may be good reasons to distribute an updated or improved version of an algorithm and for recipients to be able to make use of it without building a new kernel or even rebooting. Networking developers can certainly benefit from being able to play with congestion-control code on the fly. One could argue that congestion control is not conceptually different from other tasks, such as flow dissection or IR protocol decoding, that can be done with BPF now — but congestion control does involve a rather higher level of complexity.

A look at the patch set posted by Martin KaFai Lau reveals that what has been merged for 5.6 is not just a mechanism for hooking in TCP congestion-control algorithms; it is far more general than that. To be specific, this new infrastructure can be used to allow a BPF program to replace any "operations structure" — a structure full of function pointers — in the kernel. It is, at this point, only capable of replacing the tcp_congestion_ops structure used for congestion control; experience suggests, though, that other uses will show up sooner rather than later.

The user-space API

On the user-space side, loading a new operations structure requires a few steps, the first of which is to use the bpf() system call to load an implementation of each function as a separate BPF program. The new BPF_PROG_TYPE_STRUCT_OPS type has been defined for these programs. In the attributes passed with each program, user space must provide the BPF type format (BTF) ID corresponding to the structure being replaced (specifying the actual function being implemented comes later). BTF is a relatively recent addition that describes the functions and data structures in the running kernel; it is currently used for type-checking of tracing functions among other purposes.

User space must also specify an integer offset identifying the function this program will replace. For example, the ssthresh() member of struct tcp_congestion_ops is the sixth field defined there, so this offset will be passed as five (since offsets start at zero). How this API might interact with structure layout randomization is not entirely clear.

As the programs for each structure member are loaded, the kernel will return a file descriptor corresponding to each. Then, user space must populate a structure that looks like this:

    struct bpf_tcp_congestion_ops {
	refcount_t refcnt;
	enum bpf_struct_ops_state state;
	struct tcp_congestion_ops data;
    };

The data field has the type of the structure to be replaced — struct tcp_congestion_ops in this case. Rather than containing function pointers, though, this structure should contain the file descriptors for the programs that have been loaded to implement those functions. The non-function fields of that structure should be set as needed, though the kernel can override things as described below.

The last step is to load this structure into the kernel. One might imagine a number of ways of doing this; the actual implementation is almost certainly something else. User space must create a special BPF map with the new BPF_MAP_TYPE_STRUCT_OPS type. Associated with this map is the BTF type ID of a special structure in the kernel (described below); that is how the map is connected with the structure that is to be replaced. Actually replacing the structure is accomplished by storing the bpf_tcp_congestion_ops structure filled in above into element zero of the map. It is also possible to query the map (to see the reference-count and state fields) or to remove the structure by deleting element zero.

BPF maps have grown in features and capability over the years. Even so, this seems likely to be the first place where map operations have this kind of side effect elsewhere in the kernel. It is arguably not the most elegant of interfaces; most user-space developers will never see most of it, though, since it is, like most of the BPF API, hidden behind a set of macros and magic object-file sections in the libbpf library.

The kernel side

Replacing an operations structure requires support in the kernel; there is no ability for user space to replace arbitrary structures at will. To make it possible to replace a specific type of structure, kernel code must create a structure like this:

    #define BPF_STRUCT_OPS_MAX_NR_MEMBERS 64
    struct bpf_struct_ops {
	const struct bpf_verifier_ops *verifier_ops;
	int (*init)(struct btf *btf);
	int (*check_member)(const struct btf_type *t,
			    const struct btf_member *member);
	int (*init_member)(const struct btf_type *t,
			   const struct btf_member *member,
			   void *kdata, const void *udata);
	int (*reg)(void *kdata);
	void (*unreg)(void *kdata);
	const struct btf_type *type;
	const struct btf_type *value_type;
	const char *name;
	struct btf_func_model func_models[BPF_STRUCT_OPS_MAX_NR_MEMBERS];
	u32 type_id;
	u32 value_id;
    };

There are more details here than can be easily covered in this article, and some of the fields of this structure are automatically filled in by macros. The verifier_ops structure has a number of functions used to verify that the individual replacement functions are safe to execute. There is a new field added to that structure in this patch set, struct_access(), which regulates which parts, if any, of the operations structure itself can be changed by BPF functions.

The init() function will be called first to do any needed global setup. check_member() determines whether a specific member of the target structure is allowed to be implemented in BPF, while init_member() verifies the exact value of any fields in that structure. In particular, init_member() can validate non-function fields (flags fields, for example). The reg() function actually registers the replacement structure after the checks have passed; in the congestion-control case, it will install the tcp_congestion_ops structure (with the appropriate BPF trampolines used for the function pointers) where the network stack will use it. unreg() undoes that action.

One structure of this type should be created with a specific name: the type of the structure to be replaced with bpf_ prepended. So the operations structure for the replacement of a tcp_congestion_ops structure is named bpf_tcp_congestion_ops. This is the "special structure" that user space must reference (via its BTF ID) when loading a new operations structure. Finally, a line is added to kernel/bpf/bpf_struct_ops_types.h:

    BPF_STRUCT_OPS_TYPE(tcp_congestion_ops)

The lack of a trailing semicolon is necessary. By virtue of some macro magic and including this file four times into bpf_struct_ops.c, everything is set up without the need of a special function to register this structure type.

In closing

For the curious, the kernel-side implementation of tcp_congestion_ops replacement can be found in net/ipv4/bpf_tcp_ca.c. There are two actual algorithm implementations (DCTCP and CUBIC) in the tree as well.

The ability to replace an arbitrary operations structure in the kernel potentially holds a lot of power; a huge portion of kernel code is invoked through at least one such structure. If one could replace all or part of the security_hook_heads structure, one could modify security policies in arbitrary ways, similar to what is proposed with KRSI, for example. Replacing a file_operations structure could rewire just about any part of the kernel's I/O subsystem. And so on.

Nobody is proposing to do any of these things — yet — but this sort of capability is sure to attract interested users. There could come a time when just about any kernel functionality is amenable to being hooked or replaced with BPF code from user space. In such a world, users will have a lot of power to change how their systems operate, but what we think of as a "Linux kernel" will become rather more amorphous, dependent on which code has been loaded from user space. The result is likely to be interesting.

Comments (6 posted)

Lua and Python

From a high-level perspective, Lua and Python are similar languages; both are "scripting" languages that are compiled into bytecode instructions that run on a virtual machine. But the focus of Lua has generally been toward embedding the language into some larger application or system, rather than as an alternative for, say, Python, Perl, or Ruby as a general-purpose language. That is not to say that Lua is not capable of handling any of the tasks those other languages do, but that it has not really been the target, seemingly. Some recent discussions in the Lua community have explored possible changes in that regard, particularly around the idea of providing a larger, richer standard library.

In mid-December, Gavin Holt posted a message to the lua-l mailing list noting his overall happiness with the language, but observing that support for the "batteries" was lagging. The "batteries" term presumably comes from Python, which has long had a "batteries included" philosophy—it ships with a large standard library that is meant to cover many of the tasks a user might encounter, without needing to install additional packages. As Lua has progressed through 5.1, 5.2, and 5.3 (with 5.4 on the horizon), however, some of the packages have not been updated, so Holt cannot use them with recent releases of the language. Many of the Lua libraries are written in C, which need to be compiled and distributed as binaries for some users.

As might be guessed from the last point, Holt is running on Windows—Linux users may be somewhat more accustomed to having and using a compiler on their systems. "I know the 'non-compiling' user may not be the main target audience for Lua, but as a beautiful interpreted language I feel the lack of 'Live Batteries' prevents Lua becoming as popular as it should." He wondered if there was any effort to put together a distribution of pre-compiled libraries.

There were several suggestions of places to look for the pieces that Holt needs in the thread, but the overall picture is that Lua lacks a standardized set of libraries—which has both good and bad points. Dael Vnaja thought it would be useful to have "an alternative Lua distribution/fork that includes these 'batteries' from the language itself and not from 3rd party libraries", but that the user base for Lua might be too small to sustain one. Coda Highland said that the actual user base was quite large, but that each of the big players has its own requirements: "Everyone has different needs so the places where serious money is getting poured into using Lua don't really have much of an incentive to standardize -- it wouldn't give them any benefit."

Choosing libraries

But there are some problems trying to choose reasonable libraries for Lua, Steve Litt said. First, options need to be found, then they must be installed and tried, which may require multiple iterations. If the library is not packaged by the distribution being used, it needs to be grabbed from elsewhere (possibly using the LuaRocks package manager) and any users of the code need to also get instructions on how to install the library. It is quite different in the Python world:

He suggested that "an official list of approved Lua libraries", which could be searched and that would provide information on the strengths and weaknesses of the library, would be a good place to start. It would be far better than "having to ask online and getting lots of opinions".

Some agreed with that idea, but others were not so sure. Python exists, so perhaps those who need a full-featured standard library should turn to that, Dibyendu Majumdar said. "Lua's strength is lack of any baggage, which makes it better for embedded use." But no one is suggesting that a set of batteries become a requirement for Lua—even the existing Lua standard library can be compiled out for use cases where it is not needed, Lorenzo Donati said. The main reason that Python is a better general-purpose language than Lua is "because of its libraries", he continued.

By all accounts, Lua works quite well in its niche, but some see the potential for it to grow beyond that niche—and the main thing holding it back is the lack of a rich, standard library. As Litt put it:

The conversation picked back up again in mid-January when Donati posted some links about the popularity of Python. Stefan Lederer replied that it is not surprising that Python is as popular as it is "because of powerful libraries and easy scripting", but that does not mean the Lua community should be trying to create a "full-fledged Better Version Of Python That Is Actually Lua". He thinks that Python suffers from problems on multicore systems that are only going to become more acute and believes that Lua is better positioned for multicore. That needs more work too, however.

Litt suggested that the existence of its batteries made Python a better choice for a larger project: "If you start a project in Python, you're more likely to finish it in Python as opposed to if you'd started it in Lua and expect to finish it in Lua." He posited two possible directions for Lua, either recognizing that its lack of batteries is holding it back for the general-purpose use case or simply deciding that Python already fills the other role, so Lua should stick with the embedded use case. He was not in favor of the latter:

But a set of standard libraries has its own set of downsides, Peter Hickman said. Languages all have their sets of tradeoffs; Lua is simple and fast, even though it sometimes lacks for a particular library he might need or want.

Collaboration platform

Oliver Schmidt thought that the missing piece is not really a set of batteries, but is instead a way to collaborate on gathering, rating, and discussing the existing (or new) libraries for Lua. In essence, there is a lack of a Lua community working together to create a code repository that is "useful for newcomers and common use cases". In order for Lua to grow, there is another path should perhaps be considered:

Currently, the LuaRocks site has a bunch of libraries that can be installed, but some important ones are missing and the site lacks many of the features Schmidt thinks are needed. One way forward might be to extend the LuaRocks site, but perhaps setting up a new site on an existing platform (e.g. GitHub) would be an alternative, he said. As became clear in the rest of that thread, though, the community effort required for any kind of solution may not really be available.

Blessing

There is also the question of which libraries would go into some putative richer standard library. Some in the threads seem to think that the Lua team should "bless" a set of libraries, though that has problems of its own. As one of Lua's founders and a member of said team, Roberto Ierusalimschy, put it: "I would break the steps even further: first step is a stable library. Nobody can bless anything unstable."

Another sub-thread took up the idea of a separate repository where the community parks its library code to be improved upon collaboratively; Rodrigo Azevedo proposed the LuaDEAL (for "Lua DEad Alive Libraries") project for creating a kind of standard library. A first step is to identify which libraries fit: "It means, when someone on this list asks about a library, everybody must agree on a first default answer. Public libraries are blessed by the community, not a specific restricted set of people." The choices are not meant to be the "best", just something that is useful to get a particular job done. In addition: "They must have a well defined and stable 'Lua-ish' API. Documentation is mandatory."

There were some discussions about what that API should look like, and even a fair amount of agreement, but the fact remains that the Lua community is small. Dennis Fischer split up Lua users into a handful of categories and noted that Lua enthusiasts would seem to be the smallest of those groups. Growing the numbers of enthusiasts is needed, but there are some problems in the way of doing so:

That seems to be the crux of the issue: there is no collection of Lua libraries that "everyone" could coalesce around, even if enough everyones could be found. There are tons of Lua libraries available, but finding the "right" one is difficult, such that many users simply go off and implement their own. Even if a library appears to be suitable, it can be hard to find out if will work on the operating system of interest or the Lua version (or alternate implementation such as LuaJIT) needed.

That stands in stark contrast to the Python world, where its large standard library is immediately available to import. Beyond that, add-on libraries are plentiful and generally well-specified on the Python Package Index (PyPI), which can be installed via pip. To a large extent, the idea of what is "Pythonic" is fairly well-understood in the community as well; "Lua-ish" seems to be a bit more amorphous, or at least not well socialized within the community.

Certainly Python has struggled at times with its batteries. Some modules were perhaps mistakenly adopted into the standard library, others have not aged well, and some have even bit-rotted to an extent, which has led the project to discuss removing some "dead" batteries. There are also a fair number of API warts among those modules. But overall, the choice made early on to include a standard library with the language itself has paid big dividends for Python—and its popularity.

Lua is not much younger than Python—Lua came about in 1993, while Python was started in 1989—but Lua's focus has always been different. Trying to change the inertia of the project and its community nearly 30 years down the road is likely to be a difficult task. It is not insurmountable, perhaps, but it will take a great deal of focused effort—one library at a time. Whether the effort is worth the price is hard to judge in the early going as well.

Nothing here is meant to disparage Lua or its community in any way. By all accounts, Lua is an excellent choice for a lot of tasks. Its community seems friendly and welcoming as well—in many ways lua-l is reminiscent of the Python discussion forums. The lack of batteries is not really a new topic in the community, and a variety of solutions have been tried, so it will be interesting to see where things go from here.

Comments (29 posted)