Super Python (part 2)

Python's super() built-in function can be somewhat confusing, as highlighted by a huge python-ideas thread that we started looking at last week. It is used by methods in class hierarchies to access methods and attributes in a parent class, but exactly which class that super() resolves to is perhaps a bit unclear in multiple-inheritance hierarchies. The discussion in the second "half" of the thread further highlighted some lesser-known parts of the language.

super() with arguments

Normally, the super() function is used without arguments, as seen in part 1, to refer to a parent class. Python uses a method resolution order (MRO) to determine which parent that is, in the case of multiple inheritance, and subsequent calls to super() in those parent classes will follow the MRO for the original class, which can be a source of confusion. super() with arguments behaves somewhat differently.

super() takes two optional arguments as described in the documentation:

    super([type[, object-or-type]])

The (somewhat confusingly named) type argument specifies the class for which a parent is being sought, while object-or-type tells super() where to find the MRO to be used. "For example, if __mro__ of object-or-type is D -> B -> C -> A -> object and the value of type is B, then super() searches C -> A -> object. " Once it finds the parent class, super() acts as a proxy to that parent class, allowing methods and attributes to be accessed from it.

A simple example:

    class Example(SomeClass):
        def method(self);	
            super(Example, self).method()
	    
    class Example2(SomeClass):
        def method(self);	
            super().method()

In the code above, Example and Example2 behave exactly the same way. The no-argument super() effectively fills in the class and self. If we take a slightly more complex hierarchy from part 1, we can modify it slightly to show where the confusion comes in:

    class Top:
	def method(self):
	    print('Top')
    class Left(Top):
	def method(self):
	    print('Left')
	    super().method()
    class Right(Top):
	def method(self):
	    print('Right')
	    super().method()
    class Bottom(Left, Right):
	def method(self):
	    print('Bottom')
	    super(Left, self).method()
    class Bottom2(Left, Right):
	def method(self):
	    print('Bottom2')
	    super(Right, self).method()

    Bottom().method()    # prints Bottom, Right, Top
    Bottom2().method()   # prints Bottom2, Top

The MRO in both cases is Bottom (or Bottom2) -> Left -> Right -> Top -> object, but Bottom2.method() explicitly asks super() to fast-forward the MRO to the entry after Right, while Bottom.method() requests the entry after Left. The result may seem counterintuitive, but is consistent and has been part of the language for a long time (since Python 2.3 in 2003).

The two-argument super() is far less common than the no-argument variety; the one-argument form is likely used even less. The super() documentation describes it this way: "If the second argument is omitted, the super object returned is unbound." That particular form is beyond the scope of the discussion (and quite possibly beyond my understanding); it is seen by some as a wart in the language. The two-argument version will be a central part of a "real world" example in the discussion.

Picking up again

After digesting the early discussion we looked at last week, Martin Milon ("malmiteria") came back to the mailing list with a lengthy message to kick off round two. His argument boils down to the fact that super() can be used in multiple ways, but that it is always tied to the MRO, when there might be reason to separate the two. super() is the tool that developers reach for because it is the only one available for proxying to parent classes. He included a "TLDR" early on that lists five types of super()-related functionality that he believes need to be addressed; he presented his proposal for each.

As with the previous round, though, there are often other ways to accomplish what Milon wants to do. In addition, his objections to how things currently work, and his complaints about the confusing nature of them, might be reasonable if Python were being designed from the ground up today. But there is an enormous body of code that relies on the current behavior of super()—confusing or no—which must continue to work. His style of frequent, and frequently voluminous, posting also made it hard for participants to keep up.

Paul Moore went through the TLDR list point by point, noting that Milon had not really justified precisely what was wrong with the existing model. Moore suggested some parts of the proposal could be turned into modules for the Python Package Index (PyPI), though he cautioned that "you don't seem to have the same mental model of inheritance as Python's". As far as changes to the language go, there needs to be a lot more justification and a definition of the semantics of the changes before any progress can be made. Furthermore:

Overall, you stand very little chance of getting anywhere with this as you seem to be unwilling to make the attempt to understand people's objections. And you just keep posting huge walls of text that no-one is going to read, because they make no sense to anyone who is familiar with, and comfortable with, Python's existing model. So you're never going to persuade anyone who doesn't already agree with you...

Chris Angelico also complained about the "wall of text" problem, but he focused on what he sees as the fundamental misconception in Milon's proposal, which says, early on, that super() allows calling a method "from *the* (most people don't think of multiple inheritance) parent". Angelico said:

You start out with the assumption that MOST PEOPLE think of super as a way to call THE, singular, parent. If this is indeed a problem, then it's not a problem with super, it's a problem with expectations. And you're talking about *working around* the MRO, as if it's a problem.

I got a little bit further into your post and found you fighting hard against the existing feature, and that's when I gave up on reading it. [...]

You've already been given a solution to your problem: if you don't want super, don't use super. The purpose of super is not what you're doing with it. Learn how super is meant to be used, then decide whether it's right for your class hierarchy.

Greg Ewing wondered how Milon could claim that most developers think that super() only targets a single parent, but Steven D'Aprano thought there was likely some truth to that: "I think that malmiteria is probably correct that most developers misunderstand or misuse super, or find it surprising. I know I do." The problem with the proposal lies elsewhere:

I think that where malmiteria gets it wrong is that he thinks that super is broken. [It's] not. But we might want to avoid MI [multiple inheritance], and use something like traits instead. Or composition.

Changing super would be tantamount to banning MI, and that would be a massively backwards incompatible breaking change. It isn't going to happen (and nor should it happen, MI is fine for those who need it).

Goblins

One of the examples that Milon used to demonstrate the well-known diamond problem for multiple inheritance was discussed quite a bit in the thread. That same problem, which is an ambiguity when a class inherits from two classes that both inherit from the same grandparent (in its simplest form), also exists in the example at the beginning of the article. The "gobelin" hierarchy gives it a bit of a twist, however:

class HighGobelin:
    def scream(self):
        print("raAaaaAar")

class CorruptedGobelin(HighGobelin):
    def scream(self):
        print("my corrupted soul makes me wanna scream")
        super().scream()

class ProudGobelin(HighGobelin):
    def scream(self):
        print("I ... can't ... contain my scream!")
        super().scream()

class HalfBreed(ProudGobelin, CorruptedGobelin):
    def scream(self):
        # 50% chance to call ProudGobelin scream, 50% chance to call CorruptedGobelin scream

Some minor code typos have been fixed in Milon's examples, which was another source of frustration for some participants in the thread. Milon suggested that conceptually the HalfBreed.scream() method should look like the following, which does not do what he intends:

def scream(self):
    if random.choice([True, False]):
	super(HalfBreed, self).scream()
    else:
	super(ProudGobelin, self).scream()

However, super(HalfBreed, self) does not deliver [ProudGobelin] behavior, but a mix of ProudGobelin, CorruptedGobelin, and HighGobelin [behavior].

We would expect the output to be :

"I ... can't ... contain my scream!"
"raAaaaAar"

But it is :

"I ... can't ... contain my scream!"
"my corrupted soul makes me wanna scream"
"raAaaaAar"

The "problem", of course, is that HalfBreed inherits from both proud and corrupted goblins, thus calling the two-argument super() to skip past HalfBreed still delivers the behavior of the other two. As D'Aprano noted, though, this is not really an inheritance relationship, but should be modeled by delegation.

By definition, inheritance requires HalfBreed to call the methods of all three superclasses. (That is at least the definition used in Python, other languages may do differently.) If HalfBreed is not using the methods of all its superclasses, it is not really an inheritance relationship.

He said that Milon himself provided the right solution further on in his message, with a HalfBreed.scream() that does exactly what he has specified:

def scream(self):
    if random.choice([True, False]):
	ProudGobelin.scream(self)
    else:
	CorruptedGobelin.scream(self)

In another part of the thread, Angelico said that there is a misconception about the identity of an object that stems from multiple inheritance:

Don't forget that, in Python, the object *is what it is*, regardless of which class's method you're calling. You're not calling the "proud part of this object" or anything like that. You're simply calling a method on an object. If you don't want to follow the standard way of locating a method, use an explicit lookup instead.

Beyond that, Ewing said that multiple inheritance imposes some expectations in Python:

[...] If you're using super at all with MI, your methods should all be designed so that it *doesn't matter* exactly what order they get called in, other than the fact that methods further up the hierarchy are called after methods further down.

If that's not the case, then super is the wrong tool for the job.

In Ewing's opinion, passing a type to super() other than the class where it is being used is a mistake: "you're trying to be too clever by half, and will get bitten". Milon said that sometimes the calling order of parent classes does matter; "if for any reasons today, MRO is not consistently the good order for you, all you can do is forget about it". But Ewing disagreed, saying that targeting other classes with super() "will only lead to difficulties". But he did wonder if the name of the built-in contributed to that: "I actually think super() is misnamed and should really be called next_class() or something like that. There might be less confusion about its intended use then."

Screwhammers

Moore, like D'Aprano and others, said that the goblin example should use delegation and, by way of an analogy, that Milon was looking at the problem the wrong way:

Maybe you do have use cases where super isn't the right tool for the job. That's entirely possible. But that doesn't mean we should modify super to handle those cases as well as its current function - *especially* not if there are other solutions available in Python today, which don't use super. If you're trying to hit a nail into a piece of wood, and your screwdriver isn't doing a good job at it, that means that you should learn about hammers, not that you should propose that screwdrivers get modified to get better at hitting nails...

But Milon believes that super() is trying to do too much by handling the proxying to parent classes while restricting it to only use the MRO for deciding which classes to proxy to. "Cases where the super reliance on MRO is not fitting the need still would benefit from super proxying feature." Continuing the analogy, he said that super() is actually a "screwhammer": "I'm telling you maybe we should break the screwhammer apart into a screwdriver and a hammer. Stop answering to me 'but this is not how to use a screwhammer'. I know."

D'Aprano saw things differently, noting that Python has both inheritance, where the developer does not "want to manually specify which superclass method to call" and instead wants "the interpreter to do it using the defined linearisation", and composition or delegation where the programmer does want to control the MRO. Combining those two tools into the same class hierarchy by using super() for both things, is where Milon is going astray.

It should come as no surprise that Milon disagreed; he wants the proxy behavior of super() without using the MRO, but that can be accomplished by way of existing language features, so the participants in the thread do not seem particularly inclined toward any sort of change. Brendan Barnwell summed up that lengthy branch of the discussion by (further) explaining the difference between what Milon wants and what Python offers:

You seem to be envisioning a system in which multiple inheritance gives a subclass a "menu" of behaviors from which it may explicitly choose, but does not actually combine the superclasses' behaviors into a single, definite behavior of the subclass. In that scheme, order of multiple inheritance would not matter, because the superclasses don't need to be "integrated" into the subclass; they just sit there waiting for the subclass to explicitly decide which superclass it wants to delegate to.

Maybe that's how MI works in some other languages, but not Python. Everything about a class --- super, MRO, everything --- is impacted by its ENTIRE superclass hierarchy, including the order in which classes are inherited. When a subclass inherits from multiple superclasses, the superclass behaviors are *combined* into a *single* set of behaviors for the subclass. They're not "held separate" somehow as alternative inheritance hierarchies; they are joined into one.

Adoption and breaking changes

Milon also raised the idea of "new" syntax to support Mixins, like those from the Django web framework, which he calls "adoption". It is:

Essentially a way to declare more than your parent, but their parents too, recursively. would look like that:

class A(B(C)): ...

the adoption syntax is non breaking, as it's simply an addition to the language, and has values on its own.

But, as Angelico pointed out, that kind of construct is already valid Python syntax (with a different meaning), so it would be a breaking change. Antoine Rozo pointed to a simple example: "`class A(B(C))` is already a valid Python syntax (for example you could use `class A(namedtuple(...))`), your change to add a specific meaning to this syntax would break existing code." The following is valid Python:

    import collections

    class A(collections.namedtuple('B', ['a', 'b'])):
        pass

    A(a = 3, b = 4)

How useful (or common) that feature is might be in question, but that does not really matter. Milon said that "breaking changes aren't a strict veto, there's been some, and there's gonna be more", but Eric V. Smith disagreed, noting that he had "code in a non-public repository that your proposed change will break" and was sure he was not alone. He cautioned about continuing down this path:

As a long-time core developer, I can assure you that this is one of those cases where a breaking change will not be allowed. I'm trying to save you some time here, but if you're committed to continuing this, then you can't say you weren't warned when it's ultimately rejected. My suggestion is to rework your proposal to not break any existing code. It's a constraint we've all had to live with at one time or another, as much as we don't like it.

There were plenty of other sub-threads along the way, including a tangent on what "full" multiple inheritance is—or is not—complete with "no true Scotsman" references. There was also one that looked at multiple inheritance in light of real world genetics, which is an imperfect match at best. All of the discussion has tailed off at this point.

To a certain extent, this discussion is reminiscent of the tuple replace() method discussion we looked at back in March. In both cases, a newcomer to python-ideas proposed changes that they thought were compelling features for the language, only to find that most participants were skeptical that they could ever be adopted. That is likely frustrating for the proposer, but the core developers need to cast a critical eye on new features. It seems well-nigh impossible, at this point, to add a feature that breaks existing code for one thing.

In this case, though, despite expending vats of virtual ink, Milon did not really ever show a problem that needed to be solved. He was unhappy with the way super() and the MRO currently work, but despite multiple examples never described a real-world situation that cannot be addressed with Python's enormous number of existing features.

super() may in fact do too much, but modifications to its behavior are an extremely hard sell. One could imagine using keyword arguments to super() in a way that preserved backward compatibility, which Milon toyed with some along the way, but without an example of existing code that needs it, even that probably would not go far. Compelling use cases are essential for new Python features. The discussion was illuminating, however, and will hopefully serve to increase the understanding of super() and multiple inheritance in Python—while also showing what is really needed to make progress on new features for the language.

Index entries for this article
Python	Inheritance
Python	super()

---

to post comments