Leading items

Welcome to the LWN.net Weekly Edition for April 28, 2022

This edition contains the following feature content:

• The risks of embedded bare repositories in Git: embedding one Git repository inside another may seem strange, but there are reasons to do it — and potential risks that can result.
• Super Python (part 2): more discussion of how the Python super() function works and whether it should change.
• Handling messy pull-request diffstats: a solution to a common kernel-maintainer problem.
• Extending in-kernel TLS support: giving the kernel the ability to initiate TLS connections.
• An introduction to Linux audio plugin APIs: a comprehensive overview of the audio plugin landscape for Linux.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

The risks of embedded bare repositories in Git

Running code from inside a cloned Git repository is potentially risky, but normally just inspecting such a repository is considered to be safe. As a recent posting to the Git mailing list shows, however, there are still risks lurking inside these repositories; code that lives in them can be triggered in unexpected ways. In particular, malicious "bare" repositories can be added as a subdirectory of a repository; they can be configured to run code whenever Git commands are executed there, which is something that can happen in surprising ways. There is now an effort underway to try to address the problem in Git, without breaking the legitimate need for including bare repositories into a Git tree.

In early April, Glen Choo posted to the list about the security risk of bare repositories in Git working tree. He linked to an admirably detailed advisory from Justin Steven that documents the problem and how it can be triggered by a wide variety of tools, including shells, integrated development environments (IDEs), editors, and more. The advisory has proof-of-concept (PoC) code for a whole slew of different scenarios, including ones that can be used to reproduce the problem locally, if desired.

The risks from automatically running code that comes from a remote (possibly untrusted) repository are well-known, so git clone does not copy the configuration file (normally .git/config) to the local system. Git can be configured via the file (or by using the git config command) in a wide variety of ways, including such things as changing the meaning of certain git subcommands. That is clearly dangerous, which is why the configuration file is excluded from the clone operation.

Bare repositories inside regular ones

But bare repositories are different than regular repositories; instead of storing all of the housekeeping information (including config) in the .git subdirectory of the repository, a bare repository stores all of those files directly in the directory where the repository is created. The difference can be easily seen by comparing the contents of the two directories created by the following:

    $ mkdir tmp1 tmp2
    $ cd tmp1; git init
    Initialized empty Git repository in .../tmp1/.git/
    $ cd ../tmp2; git init --bare
    Initialized empty Git repository in .../tmp2/

The tmp1/.git and tmp2 directories will have much the same content, including a config file.

But a bare repository does not have a "work tree" so many Git commands run in tmp2 will fail, but that is easily rectified. In tmp2:

    $ git status
    fatal: this operation must be run in a work tree
    $ mkdir worktree
    $ echo $'\tworktree = "worktree"' >> config
    $ git status
    warning: core.bare and core.worktree do not make sense
    fatal: unable to set up work tree using invalid config

That error message points to another difference between a regular repository and a bare one; the config file has a different setting for core.bare. For a bare repository, as one might guess, it is set to "true", but that is easily fixed with an editor or other tool:

    $ $EDITOR config  # change bare = true to bare = false
    $ git status
    On branch master

    No commits yet

    nothing to commit (create/copy files and use "git add" to track)

Now any Git command that is run in tmp2 will refer to the bare repository there; it will consult the config file there and use whatever options have been set for it. Now if we move that directory (and rename it to better describe its nature), we might have the following:

    $ cd ../tmp1
    $ mv ../tmp2 mal
    $ git status
    # shows untracked file mal/
    $ cd mal
    $ git status
    # shows the same empty repository as above

We can, of course, add and commit mal/ and then we have a repository with a bare repository in it. Anyone who clones the tmp1 repository, will get mal/ and any malicious configuration that comes along for the ride. Triggering it is only a matter of somehow causing a Git command to be executed in mal/, which might happen as easily as simply trying to set the shell prompt. For example, Steven cites the git-prompt.sh file, which is included with Git; users of the script who cd into a malicious bare repository will (perhaps unknowingly) run git and fall into this hole.

The perils of fsmonitor

So far, though, mal/ is lacking in the malicious department. As mentioned, there are a number of Git configuration directives and hooks that can be used to potentially do malicious things, but for the most part those require that the victim execute specific Git commands in the bare repository. The core.fsmonitor directive is used more widely by Git, though, making it a useful primitive for code execution.

The idea behind fsmonitor is to reduce the search space for commands like git status by returning a list of files that may have changed since a given date and time. The directive can be set to a command to run that should return the list; if it returns a failure exit code, Git assumes all files could have changed and acts accordingly. Steven listed five fairly common Git commands that invoke the fsmonitor program (e.g. git status, git add).

So, using Steven's PoC as a guide, we can do the following (in mal/):

    $ echo $'\tfsmonitor = "echo \\"Pwned as $(id)\\">&2; false"' >> config
    $ git status
    Pwned as uid=1000(jake) gid=1000(jake) groups=...
    ...
    $ cd ..   # to tmp1/
    $ git add mal/
    $ git commit -m "adding mal"
    ...

Note that the commit does not output the "Pwned..." line, since it is done in the top-level repository. But that config now lurks in mal/ waiting for any Git command that uses fsmonitor, when executed from mal/.

This seems clearly to be a security hole, though whether it can or will be addressed in Git is not entirely clear. Embedded bare repositories are apparently used in benign ways, especially for testing purposes, and the Git project does not want to prohibit them. Choo's message was seeking a way to reduce the danger, which he described this way:

Many `git` commands can be affected by malicious config files, and many users have tools that will run `git` in the current directory or the subdirectories of a repo. Once the malicious repo has been cloned, very little social engineering is needed; the user might only need to open the repo in an editor or `cd` into the correct subdirectory.

He lists several possible fixes ranging from preventing bare repositories from being added to work trees (or ignoring them in favor of their parent repository), through checking for them with git fsck, to educating users but not changing Git. The fsck check seems like it will be pursued; Choo posted a patch that will test a tree to see if it contains a bare repository and warn if it does. "This will help hosting sites detect and prevent transmission of such malicious repos."

There is some interest in putting further guardrails on Git's behavior with respect to these bare repositories, but it is important to ensure that projects can still use embedded bare repositories, especially given that some have them in their Git commit history, which will never go away even if a different solution is found. Johannes Schindelin pointed to the libgit2 repository as one example of a project that has embedded bare repositories.

There was some discussion of various possibilities, which Choo summarized, noting that he believed: "We all agree that something needs to be done about embedded bare repos." He listed some options (beyond the fsck change, which he will be working on), but Taylor Blau was not entirely sure that something needed be done since "there is significant social engineering required in order to meaningfully exploit this". However, not much more than cloning a malicious repository and poking around in it a bit while using Git-aware tools is all that is really needed to trigger the problem, so it is not clear why Blau thinks that is a significant hurdle. In any case, Blau did think it was worth exploring options to "prevent this type of attack or make it substantially less likely to have a user run git commands that execute parts of the config opportunistically".

Blau thought that the most promising option was one that Choo described as: "Detect if the bare repo is embedded and do not read its config/hooks, but everything else still 'works'." Blau extended that idea to allow users to explicitly opt into reading the configuration from the embedded bare repositories with a configuration option that would need to be set by the user, since it would live in the main repository. As noted, git clone will not copy the configuration from the remote repository since it has long been identified as a security hole.

To opt-out (i.e., to allow legitimate use-cases to start reading embedded bare repository config again), the embedding repository would have to set a multi-valued `safe.embeddedRepo` configuration. This would specify a list of paths relative to the embedding repository's root of known-safe bare repositories.

The advantage of that approach is that it would likely disrupt few projects or workflows, since the number of (legitimate) embedded bare repositories with useful or necessary configuration is probably low. Those projects could provide instructions to their users on how to set up the configuration option, which might be a little annoying, but perhaps not all that disruptive. It looks like work is underway down that path, though there is no huge rush since the problem has been known for quite some time.

There are plenty of other pitfalls when using untrusted Git repositories, but those are already well-known; simply using make or the build script for an untrusted project is a leap of faith unless the repository is carefully scrutinized, for example. Grabbing a tar file of a repository can also bring with it unwanted baggage in the form of .git/config, hooks, or embedded bare repositories.

While Choo and Steven pointed to earlier occurrences of similar or related problems, from as early as 2017, the existence of those types of problems is not really too surprising. Git is a powerful tool, with a lot of configuration knobs that may interact in surprising or unexpected ways. Meanwhile, a bunch of tooling has grown up around it, which also may be doing somewhat unexpected, seemingly harmless, things—liking setting a shell prompt—that can lead to unpleasant outcomes. Any tool, such as an editor or IDE that tries to helpfully display repository information in its interface, may fall prey to attacks of this nature. Users should be alert to the presence of these bare repositories in any projects that they clone.

Comments (26 posted)

Super Python (part 2)

Python's super() built-in function can be somewhat confusing, as highlighted by a huge python-ideas thread that we started looking at last week. It is used by methods in class hierarchies to access methods and attributes in a parent class, but exactly which class that super() resolves to is perhaps a bit unclear in multiple-inheritance hierarchies. The discussion in the second "half" of the thread further highlighted some lesser-known parts of the language.

super() with arguments

Normally, the super() function is used without arguments, as seen in part 1, to refer to a parent class. Python uses a method resolution order (MRO) to determine which parent that is, in the case of multiple inheritance, and subsequent calls to super() in those parent classes will follow the MRO for the original class, which can be a source of confusion. super() with arguments behaves somewhat differently.

super() takes two optional arguments as described in the documentation:

    super([type[, object-or-type]])

The (somewhat confusingly named) type argument specifies the class for which a parent is being sought, while object-or-type tells super() where to find the MRO to be used. "For example, if __mro__ of object-or-type is D -> B -> C -> A -> object and the value of type is B, then super() searches C -> A -> object. " Once it finds the parent class, super() acts as a proxy to that parent class, allowing methods and attributes to be accessed from it.

A simple example:

    class Example(SomeClass):
        def method(self);	
            super(Example, self).method()
	    
    class Example2(SomeClass):
        def method(self);	
            super().method()

In the code above, Example and Example2 behave exactly the same way. The no-argument super() effectively fills in the class and self. If we take a slightly more complex hierarchy from part 1, we can modify it slightly to show where the confusion comes in:

    class Top:
	def method(self):
	    print('Top')
    class Left(Top):
	def method(self):
	    print('Left')
	    super().method()
    class Right(Top):
	def method(self):
	    print('Right')
	    super().method()
    class Bottom(Left, Right):
	def method(self):
	    print('Bottom')
	    super(Left, self).method()
    class Bottom2(Left, Right):
	def method(self):
	    print('Bottom2')
	    super(Right, self).method()

    Bottom().method()    # prints Bottom, Right, Top
    Bottom2().method()   # prints Bottom2, Top

The MRO in both cases is Bottom (or Bottom2) -> Left -> Right -> Top -> object, but Bottom2.method() explicitly asks super() to fast-forward the MRO to the entry after Right, while Bottom.method() requests the entry after Left. The result may seem counterintuitive, but is consistent and has been part of the language for a long time (since Python 2.3 in 2003).

The two-argument super() is far less common than the no-argument variety; the one-argument form is likely used even less. The super() documentation describes it this way: "If the second argument is omitted, the super object returned is unbound." That particular form is beyond the scope of the discussion (and quite possibly beyond my understanding); it is seen by some as a wart in the language. The two-argument version will be a central part of a "real world" example in the discussion.

Picking up again

After digesting the early discussion we looked at last week, Martin Milon ("malmiteria") came back to the mailing list with a lengthy message to kick off round two. His argument boils down to the fact that super() can be used in multiple ways, but that it is always tied to the MRO, when there might be reason to separate the two. super() is the tool that developers reach for because it is the only one available for proxying to parent classes. He included a "TLDR" early on that lists five types of super()-related functionality that he believes need to be addressed; he presented his proposal for each.

As with the previous round, though, there are often other ways to accomplish what Milon wants to do. In addition, his objections to how things currently work, and his complaints about the confusing nature of them, might be reasonable if Python were being designed from the ground up today. But there is an enormous body of code that relies on the current behavior of super()—confusing or no—which must continue to work. His style of frequent, and frequently voluminous, posting also made it hard for participants to keep up.

Paul Moore went through the TLDR list point by point, noting that Milon had not really justified precisely what was wrong with the existing model. Moore suggested some parts of the proposal could be turned into modules for the Python Package Index (PyPI), though he cautioned that "you don't seem to have the same mental model of inheritance as Python's". As far as changes to the language go, there needs to be a lot more justification and a definition of the semantics of the changes before any progress can be made. Furthermore:

Overall, you stand very little chance of getting anywhere with this as you seem to be unwilling to make the attempt to understand people's objections. And you just keep posting huge walls of text that no-one is going to read, because they make no sense to anyone who is familiar with, and comfortable with, Python's existing model. So you're never going to persuade anyone who doesn't already agree with you...

Chris Angelico also complained about the "wall of text" problem, but he focused on what he sees as the fundamental misconception in Milon's proposal, which says, early on, that super() allows calling a method "from *the* (most people don't think of multiple inheritance) parent". Angelico said:

You start out with the assumption that MOST PEOPLE think of super as a way to call THE, singular, parent. If this is indeed a problem, then it's not a problem with super, it's a problem with expectations. And you're talking about *working around* the MRO, as if it's a problem.

I got a little bit further into your post and found you fighting hard against the existing feature, and that's when I gave up on reading it. [...]

You've already been given a solution to your problem: if you don't want super, don't use super. The purpose of super is not what you're doing with it. Learn how super is meant to be used, then decide whether it's right for your class hierarchy.

Greg Ewing wondered how Milon could claim that most developers think that super() only targets a single parent, but Steven D'Aprano thought there was likely some truth to that: "I think that malmiteria is probably correct that most developers misunderstand or misuse super, or find it surprising. I know I do." The problem with the proposal lies elsewhere:

I think that where malmiteria gets it wrong is that he thinks that super is broken. [It's] not. But we might want to avoid MI [multiple inheritance], and use something like traits instead. Or composition.

Changing super would be tantamount to banning MI, and that would be a massively backwards incompatible breaking change. It isn't going to happen (and nor should it happen, MI is fine for those who need it).

Goblins

One of the examples that Milon used to demonstrate the well-known diamond problem for multiple inheritance was discussed quite a bit in the thread. That same problem, which is an ambiguity when a class inherits from two classes that both inherit from the same grandparent (in its simplest form), also exists in the example at the beginning of the article. The "gobelin" hierarchy gives it a bit of a twist, however:

class HighGobelin:
    def scream(self):
        print("raAaaaAar")

class CorruptedGobelin(HighGobelin):
    def scream(self):
        print("my corrupted soul makes me wanna scream")
        super().scream()

class ProudGobelin(HighGobelin):
    def scream(self):
        print("I ... can't ... contain my scream!")
        super().scream()

class HalfBreed(ProudGobelin, CorruptedGobelin):
    def scream(self):
        # 50% chance to call ProudGobelin scream, 50% chance to call CorruptedGobelin scream

Some minor code typos have been fixed in Milon's examples, which was another source of frustration for some participants in the thread. Milon suggested that conceptually the HalfBreed.scream() method should look like the following, which does not do what he intends:

def scream(self):
    if random.choice([True, False]):
	super(HalfBreed, self).scream()
    else:
	super(ProudGobelin, self).scream()

However, super(HalfBreed, self) does not deliver [ProudGobelin] behavior, but a mix of ProudGobelin, CorruptedGobelin, and HighGobelin [behavior].

We would expect the output to be :

"I ... can't ... contain my scream!"
"raAaaaAar"

But it is :

"I ... can't ... contain my scream!"
"my corrupted soul makes me wanna scream"
"raAaaaAar"

The "problem", of course, is that HalfBreed inherits from both proud and corrupted goblins, thus calling the two-argument super() to skip past HalfBreed still delivers the behavior of the other two. As D'Aprano noted, though, this is not really an inheritance relationship, but should be modeled by delegation.

By definition, inheritance requires HalfBreed to call the methods of all three superclasses. (That is at least the definition used in Python, other languages may do differently.) If HalfBreed is not using the methods of all its superclasses, it is not really an inheritance relationship.

He said that Milon himself provided the right solution further on in his message, with a HalfBreed.scream() that does exactly what he has specified:

def scream(self):
    if random.choice([True, False]):
	ProudGobelin.scream(self)
    else:
	CorruptedGobelin.scream(self)

In another part of the thread, Angelico said that there is a misconception about the identity of an object that stems from multiple inheritance:

Don't forget that, in Python, the object *is what it is*, regardless of which class's method you're calling. You're not calling the "proud part of this object" or anything like that. You're simply calling a method on an object. If you don't want to follow the standard way of locating a method, use an explicit lookup instead.

Beyond that, Ewing said that multiple inheritance imposes some expectations in Python:

[...] If you're using super at all with MI, your methods should all be designed so that it *doesn't matter* exactly what order they get called in, other than the fact that methods further up the hierarchy are called after methods further down.

If that's not the case, then super is the wrong tool for the job.

In Ewing's opinion, passing a type to super() other than the class where it is being used is a mistake: "you're trying to be too clever by half, and will get bitten". Milon said that sometimes the calling order of parent classes does matter; "if for any reasons today, MRO is not consistently the good order for you, all you can do is forget about it". But Ewing disagreed, saying that targeting other classes with super() "will only lead to difficulties". But he did wonder if the name of the built-in contributed to that: "I actually think super() is misnamed and should really be called next_class() or something like that. There might be less confusion about its intended use then."

Screwhammers

Moore, like D'Aprano and others, said that the goblin example should use delegation and, by way of an analogy, that Milon was looking at the problem the wrong way:

Maybe you do have use cases where super isn't the right tool for the job. That's entirely possible. But that doesn't mean we should modify super to handle those cases as well as its current function - *especially* not if there are other solutions available in Python today, which don't use super. If you're trying to hit a nail into a piece of wood, and your screwdriver isn't doing a good job at it, that means that you should learn about hammers, not that you should propose that screwdrivers get modified to get better at hitting nails...

But Milon believes that super() is trying to do too much by handling the proxying to parent classes while restricting it to only use the MRO for deciding which classes to proxy to. "Cases where the super reliance on MRO is not fitting the need still would benefit from super proxying feature." Continuing the analogy, he said that super() is actually a "screwhammer": "I'm telling you maybe we should break the screwhammer apart into a screwdriver and a hammer. Stop answering to me 'but this is not how to use a screwhammer'. I know."

D'Aprano saw things differently, noting that Python has both inheritance, where the developer does not "want to manually specify which superclass method to call" and instead wants "the interpreter to do it using the defined linearisation", and composition or delegation where the programmer does want to control the MRO. Combining those two tools into the same class hierarchy by using super() for both things, is where Milon is going astray.

It should come as no surprise that Milon disagreed; he wants the proxy behavior of super() without using the MRO, but that can be accomplished by way of existing language features, so the participants in the thread do not seem particularly inclined toward any sort of change. Brendan Barnwell summed up that lengthy branch of the discussion by (further) explaining the difference between what Milon wants and what Python offers:

You seem to be envisioning a system in which multiple inheritance gives a subclass a "menu" of behaviors from which it may explicitly choose, but does not actually combine the superclasses' behaviors into a single, definite behavior of the subclass. In that scheme, order of multiple inheritance would not matter, because the superclasses don't need to be "integrated" into the subclass; they just sit there waiting for the subclass to explicitly decide which superclass it wants to delegate to.

Maybe that's how MI works in some other languages, but not Python. Everything about a class --- super, MRO, everything --- is impacted by its ENTIRE superclass hierarchy, including the order in which classes are inherited. When a subclass inherits from multiple superclasses, the superclass behaviors are *combined* into a *single* set of behaviors for the subclass. They're not "held separate" somehow as alternative inheritance hierarchies; they are joined into one.

Adoption and breaking changes

Milon also raised the idea of "new" syntax to support Mixins, like those from the Django web framework, which he calls "adoption". It is:

Essentially a way to declare more than your parent, but their parents too, recursively. would look like that:

class A(B(C)): ...

the adoption syntax is non breaking, as it's simply an addition to the language, and has values on its own.

But, as Angelico pointed out, that kind of construct is already valid Python syntax (with a different meaning), so it would be a breaking change. Antoine Rozo pointed to a simple example: "`class A(B(C))` is already a valid Python syntax (for example you could use `class A(namedtuple(...))`), your change to add a specific meaning to this syntax would break existing code." The following is valid Python:

    import collections

    class A(collections.namedtuple('B', ['a', 'b'])):
        pass

    A(a = 3, b = 4)

How useful (or common) that feature is might be in question, but that does not really matter. Milon said that "breaking changes aren't a strict veto, there's been some, and there's gonna be more", but Eric V. Smith disagreed, noting that he had "code in a non-public repository that your proposed change will break" and was sure he was not alone. He cautioned about continuing down this path:

As a long-time core developer, I can assure you that this is one of those cases where a breaking change will not be allowed. I'm trying to save you some time here, but if you're committed to continuing this, then you can't say you weren't warned when it's ultimately rejected. My suggestion is to rework your proposal to not break any existing code. It's a constraint we've all had to live with at one time or another, as much as we don't like it.

There were plenty of other sub-threads along the way, including a tangent on what "full" multiple inheritance is—or is not—complete with "no true Scotsman" references. There was also one that looked at multiple inheritance in light of real world genetics, which is an imperfect match at best. All of the discussion has tailed off at this point.

To a certain extent, this discussion is reminiscent of the tuple replace() method discussion we looked at back in March. In both cases, a newcomer to python-ideas proposed changes that they thought were compelling features for the language, only to find that most participants were skeptical that they could ever be adopted. That is likely frustrating for the proposer, but the core developers need to cast a critical eye on new features. It seems well-nigh impossible, at this point, to add a feature that breaks existing code for one thing.

In this case, though, despite expending vats of virtual ink, Milon did not really ever show a problem that needed to be solved. He was unhappy with the way super() and the MRO currently work, but despite multiple examples never described a real-world situation that cannot be addressed with Python's enormous number of existing features.

super() may in fact do too much, but modifications to its behavior are an extremely hard sell. One could imagine using keyword arguments to super() in a way that preserved backward compatibility, which Milon toyed with some along the way, but without an example of existing code that needs it, even that probably would not go far. Compelling use cases are essential for new Python features. The discussion was illuminating, however, and will hopefully serve to increase the understanding of super() and multiple inheritance in Python—while also showing what is really needed to make progress on new features for the language.

Comments (20 posted)

Handling messy pull-request diffstats

[Editor's note: the following was written in response to frequent questions on the linux-kernel mailing list; it was pulled into the mainline during the 5.18 merge window.]

Subsystem maintainers routinely use git request-pull as part of the process of sending work upstream. Normally, the result includes a list of commits included in the request and a nice diffstat that shows which files will be touched and how much of each will be changed; examples abound on the kernel mailing lists. Occasionally, though, a repository with a relatively complicated development history will yield a massive diffstat containing a great deal of unrelated work. The result looks ugly and obscures what the pull request is actually doing. This document describes what is happening and how to fix things up; it is derived from The Wisdom of Linus Torvalds, which has been posted numerous times over the years (example 1, example 2).

A Git development history proceeds as a series of commits. In a simplified manner, mainline kernel development looks like this:

If one wants to see what has changed between two points, a command like this will do the job:

  $ git diff --stat --summary vN-rc2..vN-rc3

Here, there are two clear points in the history (vN-rc2 and vN-rc3); Git will, in a manner of speaking, subtract the commits present at the beginning point from those at the end point and display the resulting differences. The requested operation is unambiguous and easy enough to understand.

When a subsystem maintainer creates a branch and commits changes to it, the result in the simplest case is a history that looks like:

If that maintainer now uses git diff to see what has changed between the mainline branch (let's call it linus) and cN, there are still two clear end points (vN-rc2 and cN), and the result is as expected. So a pull request generated with git request-pull will also be as expected. But now consider a slightly more complex development history:

Our maintainer has created one branch at vN-rc1 and another at vN-rc2; the two were then subsequently merged into c2. Now a pull request generated for cN may end up being messy indeed, and developers often end up wondering why.

What is happening here is that there are no longer two clear end points for the git diff operation to use. The development culminating in cN started in two different places; to generate the diffstat, git diff ends up having to pick one of them and hoping for the best. If the diffstat starts at vN-rc1, it may end up including all of the changes between there and the second beginning point (vN-rc2), which is certainly not what our maintainer had in mind. With all of that extra junk in the diffstat, it may be impossible to tell what actually happened in the changes leading up to cN.

Maintainers often try to resolve this problem by, for example, rebasing the branch or performing another merge with the linus branch, then recreating the pull request. This approach tends not to lead to joy at the receiving end of that pull request; rebasing and/or merging just before pushing upstream is a well-known way to get a grumpy response.

So what is to be done? The best response when confronted with this situation is to indeed to do a merge with the branch you intend your work to be pulled into, but to do it privately, as if it were the source of shame. Create a new, throwaway branch and do the merge there:

The merge operation resolves all of the complications resulting from the multiple beginning points, yielding a coherent result that contains only the differences from the mainline branch. Once again, there are just two end points to use in generating the listing, so it will now be possible to generate a diffstat with the desired information:

  $ git diff -C --stat --summary linus..TEMP

Save the output from this command, then simply delete the TEMP branch; definitely do not expose it to the outside world. Take the saved diffstat output and edit it into the messy pull request, yielding a result that shows what is really going on. That request can then be sent upstream.

Comments (13 posted)

Extending in-kernel TLS support

The kernel gained support for the TLS protocol in the 4.13 release, which came out in September 2017. That support is incomplete, though, in that it does not provide the kernel with a way to initiate a TLS connection on its own. Instead, user space creates a socket and performs the TLS handshake before handing the socket to the kernel, which can then transfer data using TLS. The situation may be about to change as a result of this patch series from Chuck Lever — though user space will still need to remain in the picture.

TLS, of course, allows for the transfer of encrypted data over the network; it is the protocol that lurks behind HTTPS links, among other things. At this point, a significant fraction of the data transferred over the net is encrypted in this fashion. Once a connection has been established, encrypting data to send to the other end is relatively straightforward, as is decrypting received data. Establishing the connection, though, is a more complex affair, involving, among other things, algorithm negotiation and the provision and verification of public keys for one or both ends.

There are a few advantages to supporting TLS in the kernel, including a small performance boost and the ability to apply socket filters. TLS session establishment, though, is less performance-critical and, due to its complexity, potentially a bigger source of bugs and security problems. So, when TLS support was added to the kernel, it focused on the data-transmission problem, leaving the difficulties of session setup to user space. That is how kernel TLS support has worked in the intervening years.

This solution works, but there are times when it could be useful for the kernel to have the ability to initiate TLS sessions on it own; thus Lever's patch. That said, this patch set still does not bring the TLS handshake into the kernel, even though that is the desired goal eventually:

In the long run, our preference is to have a TLS handshake implementation in the kernel. However, it appears that would take a long time and there is some desire to avoid adding to the Linux kernel's "attack surface" without good reasons. So in the meantime we've created a prototype handshake implementation that calls out to user space where the actual handshake can be done by an existing library implementation of TLS.

This design requires that a special user-space process be running in any context (specifically, any network namespace) where there may be a need for the kernel to initiate TLS connections. That process will create a socket using the new AF_TLSH ("TLS helper") address-family type, then listen on that socket. When the kernel needs to have a TLS session established, the listen() call will return with a connected TCP socket; the process can then talk with the remote peer to get the session established. If that negotiation is successful, a setsockopt() call with the new SOL_TLS option can be used to describe the newly established session. Closing the socket will then return it to the kernel.

On the kernel side, instead, there is a new function to be called after the initial TCP connection has been made:

    int tls_client_hello_x509(struct socket *sock, void (*done)(void *data, int status),
			      void *data, const char *priorities, key_serial_t peerid,
			      key_serial_t cert);

This call will attempt to pass sock to the helper process; if that works, it will return zero; the negotiation will still be ongoing at that time. Once session setup succeeds (or fails), the done() callback will be called with the result of the operation; if a successful status is reported there, the kernel should be able to communicate over the socket using TLS. There is also tls_client_hello_psk(), which can be shared in situations where a pre-shared key exists.

Why, one might ask, is this capability needed? One answer comes in the form of a followup patch set implementing the remote procedure call (RPC) protocol over TLS. That, in turn, can be used to implement the NFS filesystem protocol over encrypted connections. In the future, Lever said, there may also be interest in using this feature to support the SMB filesystem protocol over QUIC connections, assuming, of course, that the kernel actually gets QUIC support one of these years.

The reaction to the TLS patches has been relatively muted, consisting solely of a set of Reviewed-by tags from Hannes Reinecke, who was also the author of one of the patches. The RPC-over-TLS patches, instead, have run into some disagreement from Trond Myklebust, the maintainer of the kernel's NFS client. He argued that the setup work could be done entirely in user space by the mount.nfs utility. Lever responded that there are situations where, it is felt, the kernel needs to make the decision on whether TLS should be used. The conversation wound down without arriving at a conclusion, so chances are good that this is a topic that will come up at the Linux Storage, Filesystem, and Memory-Management Summit in early May.

Comments (22 posted)

An introduction to Linux audio plugin APIs

The world of music and audio production is largely dominated by proprietary software vendors. Among them, Steinberg stands out as a company that created some of the most-used software, including the Cubase and Nuendo digital audio workstations. Steinberg is also known as the creator of the VST plugin API that, largely due to its licensing policy, has irritated developers enough to inspire multiple attempts at creating an open-source alternative. Even now, when the VST3 SDK is available under the GPLv3 license, the way the company exercises its control over the SDK keeps pushing developers away toward other open-source solutions.

This is an introduction to open-source plugin APIs for musicians and sound engineers alike. It focuses on the options in the larger ecosystem and how their shortcomings led to the creation of new alternatives with liberal licensing.

What audio and MIDI plugins do

Most audio editors, MIDI sequencers, and digital audio workstations (DAWs) are extensible. Sometimes that means writing a script that accesses the public API of a program (e.g. Lua in Ardour, ReaScript in Reaper). But the most common way is to write an extension, or plugin, using an API that is usually not specific to one single program. The audio editor or the DAW then would act as a "host" for that plugin, running it either in its own process or as a separate process (an increasingly popular option).

Two of the most common use cases for plugins in DAWs are processing audio (equalizers, reverbs, etc.), which means there's audio input and audio output, and turning notes into music, which means there's MIDI input and audio output. Most plugin APIs allow both audio and MIDI input and output. Additionally, some plugins exist for the purpose of controlling external gear, such as an audio interface with on-board digital signal processing like a built-in reverberation engine. These plugins are typically made by hardware vendors themselves, though there are some rare exceptions.

Unlike plugins in many image editors, audio and MIDI plugins are usually designed to work non-destructively and run in realtime mode. That means that you don't need to render a processed audio track to listen to it. Instead, you can hook up a MIDI keyboard and play a virtual instrument without ever recording a note, and you can change the value of plugin parameters over time.

A quick overview of current plugin APIs

These plugin APIs are currently in wide use:

• Virtual Studio Technology, or VST2/VST3, by Steinberg. It is supported on Windows, macOS, and Linux, and has broad host/DAW support.
• Real Time AudioSuite, or RTAS, by Digidesign/Avid (Windows, macOS, only supported in Avid Pro Tools).
• Avid Audio eXtension, or AAX, by Avid (Windows, macOS, only supported in Avid Pro Tools).
• Audio Unit, or AU, by Apple (macOS, iOS/iPadOS).
• LV2, which is an open-source project (Linux, Windows, macOS).

Industry-wide, plugins are mostly available as VST2/VST3 for both Windows and macOS and as AU for macOS. In some cases, plugins are also available as AAX and RTAS to target Avid Pro Tools users, because Avid infamously refuses to support any APIs other than its own.

On Linux specifically, LADSPA was the first widely used API for audio plugins; DSSI (Disposable Soft Synth Interface) was developed on top of it by the same team to make software synthesizers and samplers available as plugins too. There are several hundred LADSPA plugins available, but fewer than 50 DSSI plugins. Most audio editors and DAWs on Linux support LADSPA; few support DSSI. Both LADSPA and DSSI have now been largely replaced by LV2, which allows both audio and MIDI processing. There are well over a thousand LV2 plugins out there today.

There is also a growing number of plugins — both proprietary and free — written for the VST API and built natively on Linux, largely thanks to the VST3 SDK being available under the terms of both GPLv3 and a proprietary license. In fact, some spectacular free plugins, such as the Dexed and OB-Xd synthesizers, are only available as VST3 on Linux.

One final API worth mentioning here is the Generalized Music Plugin Interface, or GMPI. The original specification was designed between 2003 and 2005 by a diverse group of developers of both proprietary (SONAR, SynthEdit, FL Studio, etc.) and free (Ardour, etc.) software. Despite their varying backgrounds, these developers all shared the same sentiment: everyone felt fed up with Steinberg's dominance in the industry, which we will talk about in more detail below.

The GMPI specification was turned into an unofficial SDK by Tim Hockin (Google, but acting on his own behalf) and Jeff McClintock (SynthEdit), with Koen Tanghe (engineer at Ghent University at the time) working on his own version. The code by Hockin and McClintock did not receive much peer review, despite many requests. Eventually, the working group dissipated and McClintock used a cut-down version of the SDK for the third iteration of his SynthEdit plugin API. The fourth iteration will have a refactored version of the GMPI SDK.

I've heard a few explanations for what happened. The story from McClintock, shared in a private conversation, is the closest match to the discussions in the public mailing list. His impression is that many stakeholders only wanted a kind of VST2 with liberal licensing and weren't interested in major changes and improvements, so there was no incentive for SDK review. Another view, expressed by Paul Davis (Ardour), is that the MIDI Manufacturers Association, which announced the initiative in the first place, wanted the implementation phase to be closed. The Association reportedly also could not guarantee licensing that everyone would be happy with. So by the time an actual official SDK would have become available, many stakeholders would likely have lost their interest, and stakeholders likely weren't interested in an unofficial SDK.

I'm mentioning GMPI for three reasons. First of all, it was the first time in history that developers of proprietary and free software teamed up to develop an open-source audio/MIDI plugin API. Secondly, while largely unsuccessful, GMPI affected the design of both VST3 and LV2 (here is more info on the latter). And finally, McClintock resumed working on the full GMPI SDK last December. The SDK is available under the terms of the ISC license.

Where existing plugin APIs fall short

First off, if you want to develop an audio/MIDI plugin for use on Linux, your options are limited as compared to Windows and macOS.

AU (both v2 and v3) is not available anywhere but macOS and iOS/iPadOS. AAX and RTAS are cross-platform, but only work in Avid Pro Tools, which is unavailable on Linux. So far, SynthEdit appears to be pretty much the only real software that makes some use of GMPI. That leaves us VST2, VST3, LADSPA, DSSI, and LV2. These APIs are all supported by host applications on Linux, free and proprietary alike, although only two hosts, Qtractor and Zrythm, support all five.

Right off the bat, VST2 is at the end of its life. If you have a license for the VST2 SDK for an existing plugin, you can make a new version of that plugin as long as you want to. But Steinberg won't license the SDK for new plugins, and the need to license the use of this SDK was a major reason why VST2 never took off on Linux anyway. The limited number of native VST2 plugins that did appear for Linux were only possible thanks to a clean-room reverse-engineered VeSTige header file made by some LMMS developers.

LADSPA and DSSI are simple APIs, and the majority of audio hosts on Linux support LADSPA. However, these APIs have a number of technical limitations, they are not well maintained, and pretty much everyone stopped making new LADSPA and DSSI plugins years ago. Thus, we are down to VST3 and LV2.

VST3 has a specification that matches contemporary expectations and allows the creation of capable plugins. It's also extensible, though VST3 extensions are scattered across third-party vendors. However, over the years, Steinberg has managed to irritate developers of both proprietary and FOSS plugins in a major way. The way the company is sunsetting VST2 has upset plugin developers because it's impossible to use the old technology to create new work for hosts that still support VST2. Beyond that, Steinberg's own hosts and plugins will cease supporting VST2 by January 2024, so any Cubase and Nuendo projects that users have will fail to load entirely. Steinberg took some steps to make a VST2-to-VST3 transition possible but, reportedly, neither designed it well nor made developers aware of this feature in a timely manner, resulting in mistakes that cannot be easily undone now.

And then there are the inevitable technical controversies. The VST3 SDK is huge and has a number of hard-to-swallow idiosyncrasies like the enforced use of UTF-16. In particular, VST3 has an entirely new way of dealing with incoming MIDI data through a convenient abstraction layer that makes a lot of sense but which demands rearchitecting existing plugins, with no fallback to the old way. This complicates the task of maintaining both VST2 and VST3 versions of a plugin. To get a sense of how much work it takes to develop and maintain a fully functional VST3 version of a fairly complex plugin, you can read this comment on KVR by Urs Heckmann.

In summary, the industry is tired of living at the mercy of Steinberg. GMPI was the first major attempt to change the balance of forces, but, as described above, it wasn't successful.

LV2 was built in the mid-2000s using many ideas from the GMPI specification. It's modern, extensible, and allows all sorts of wonderful things. However, although well over 1,000 LV2 plugins are now available (here is a fairly recent, yet not exhaustive, list), there is a general sentiment that LV2 is difficult to get started with. Since its inception, I have observed a number of irritations that developers have with this API. Some of the issues are purely technical in nature. For example, plugin metadata is stored separately in text files using the Turtle syntax. For complex plugins, those files tend to become large. To give you an idea, the compiled metadata file for Mix Maxtrix has nearly 32K lines, although that's an extreme example. Other developers cite too much flexibility being the major issue they have with the API.

Then there are organizational issues like the lack of governance body and "too many cooks in the kitchen", which results in plugin developers feeling like they aren't being heard. The feeling that the LV2 maintainers are too defensive about their design decisions and don't like revisiting them does not help either. On an even larger scale, end users want an ecosystem, and LV2 has never established itself as such. That is, however, a very broad claim that, alone, is worth a separate conversation starting with how one defines an ecosystem for a plugin API.

There are more reasons for this API's slow adoption. One is that LV2 is not very popular with host developers outside of the free-software realm. The lack of LV2 support in hosts substantially limits the exposure of available plugins to end-users. Reaper is pretty much the only major DAW (on a global scale) that supports it, but then again Reaper supports almost every plugin API out there.

The other problem is that LV2 is currently unsupported by the major plugin-development frameworks; these allow writing code once and then building it for multiple plugin APIs. There's a ten-year-old fork of JUCE, one of those frameworks, by falkTX that adds native LV2 support; the upstream project is reimplementing this just now, with no planned completion date announced. Also, a third-party effort to add LV2 support to the iPlug2 framework was never merged into the mainstream project.

All of these factors affect the adoption of LV2. Thus we see, for example, OB-Xd developer closing the request for an LV2 version because JUCE6 has VST3 support and doesn't have LV2 support. It's difficult enough to get plugin developers to generate a native VST3 plugin even when they use JUCE, it's almost impossible to get them to release an LV2 version.

I have definitely seen most of the points mentioned here refuted by developers heavily invested with LV2. However even a naive observation suggests that LV2 is neither popular nor known outside of the FOSS ecosystem.

In conclusion

There are two open-source audio/MIDI plugin APIs in active use today: VST3 and LV2. Both APIs provide the means to create sophisticated plugins but, from the standpoint of a number of developers, both are somewhat problematic for technical reasons, as well as for how these APIs are managed by their respective maintainers.

For VST3, developers have to deal with Steinberg's aggressive licensing policy. This essentially means never building new VST2 plugins. For developers of virtual instruments, it means either dropping support for existing VST2 plugins entirely or accepting the burden of maintaining two architecturally different code bases. On top of that, various other design decision in the API have met resistance.

In case of LV2, developers have a steep learning curve and little choice in case they want to develop a plugin for multiple APIs from one code base. They would also be limited to just open-source host applications plus one proprietary digital audio workstation (Reaper), which significantly limits their exposure to users unless they support multiple APIs.

This situation has persisted for years now. In 2015, it resulted in the birth of another open-source API, CLAP. This project was resurrected late last year and is now finding its way to both free and proprietary software. We will talk about it in more detail in a followup article.

[Many thanks to Paul Davis, Robin Gareus, Filipe "falktx" Coelho, William Light, Alexandre Bique, Jeff McClintock for consulting on various technical aspects of audio/MIDI plugin APIs.]

Comments (6 posted)