Super Python (part 1)

A mega-thread in the python-ideas mailing list is hardly surprising, of course; we have covered quite a few of them over the years. A recent example helps shine a light into a dark—or at least dim—corner of the Python language: the super() built-in function for use by methods in class hierarchies. There are some, perhaps surprising, aspects to super() along with wrinkles in how to properly use it. But it has been part of the language for a long time, so changes to its behavior, as was suggested in the thread, are pretty unlikely.

Inheritance and super()

Perhaps a bit of an introduction to Python inheritance is in order, at least for some readers. Python is generally written in an object-oriented fashion, with classes that can incorporate behavior from other classes via various mechanisms. One of those ways is via inheritance: one class directly references a parent class and gets some of its behavior from the parent. That can look like the following:

    class Shape: pass
    
    class Ellipse(Shape): pass

    class Circle(Ellipse): pass

The idea is that inheritance is a way to express an "is a" relationship; in the above, Circle is an Ellipse, which is a Shape, thus, transitively, Circle is also a Shape. Implicitly, Shape is an object, which is the base for all classes in Python. Behavior and attributes that are common to all shapes would be added to a parent class and those would be inherited by all of the descendant classes.

Perhaps this class hierarchy is being used in some sort of graphical display program, so all shapes might have a position and color as attributes. For behavior, shapes might need some basic operations, such as move() and draw(), but those are somewhat beyond the scope of what is needed to explain inheritance. If we modify Shape a bit, we can see this inheritance in action:

    class Shape:
	def __init__(self, x, y, color):
	    self.x = x
	    self.y = y
	    self.color = color

    c = Circle(x = 0, y = 0, color = 'blue')
    print(c.color)  # prints "blue"

The __init__() function initializes a shape with the given attributes and Circle inherits that initialization routine from Shape (by way of Ellipse); Python automatically follows the hierarchy when it cannot find a function and uses the first that it finds. The call to Circle() creates an instance of that type and initializes it using the __init__() from Shape; the result is a Circle object with three instance variables (x, y, and color).

But what if Circle needs to be initialized with additional information specific to that kind of shape? Maybe it needs a radius; a somewhat simplistic way to add that might be the following:

    class Circle(Ellipse):
	def __init__(self, x, y, color, r):
	    self.r = r
	    Ellipse.__init__(self, x, y, color)

Now, a Circle gets initialized with four parameters; it peels off one and calls its parent class's __init__() explicitly. Given the example so far, it could simply go straight to the __init__() method of Shape, but that is generally a bad practice; Ellipse may well have (or get) an __init__() of its own. But now, if the class hierarchy is refactored in some way and a class is inserted between Ellipse and Circle, or Circle gets some other parent, it needs to be changed in the class definition and every member function that specifically refers to Ellipse. The super() function is meant to help with that:

    class Circle(Ellipse):
	def __init__(self, x, y, color, r):
	    self.r = r
	    super().__init__(x, y, color)

super() is effectively a proxy for the parent class, without needing to name it directly. Should the hierarchy change, the new parent class just needs to be named once in the class definition. In a straightforward single-inheritance hierarchy like this, it is clear the path that Python will take to resolve super(); it simply walks the hierarchy upward until it finds a parent that implements the member function in question.

super() and multiple inheritance

But with multiple inheritance, things are a bit different. If we have a new class, that inherits from Circle and another class, there might be some question how methods get resolved—and in what order. If we torture our example a bit further, we might have:

    class Circle(Ellipse):
	def __init__(self, r, **kwargs):
	    self.r = r
	    super().__init__(**kwargs)

    class Shape3D(Shape):
	def __init__(self, z, **kwargs):
	    self.z = z
	    super().__init__(**kwargs)

    class Sphere(Circle, Shape3D):
	def __init__(self, **kwargs):
	    super().__init__(**kwargs)

So Sphere inherits from both Circle and the new Shape3D class, which in turn inherits from Shape. Since we are using keyword arguments to initialize the objects, using **kwargs allows the __init__() functions to just peel off the arguments of interest to them and pass the rest up the chain via super(). A Sphere could be created as follows:

    s = Sphere(x = 0, y = 0, z = -2, r = 2, color = 'red')

In order for that all to work, both parents of Sphere need to have their initialization routines called, and that is exactly what happens. Python has a method resolution order (MRO) that was adopted for Python 2.3 back in 2003. It uses C3 linearization to determine how to order the class hierarchy in a deterministic way that preserves the relationships between the classes; if no such ordering can be found, Python will raise an error when the classes are defined. The class.mro() method can be used to examine the MRO; for Sphere it looks like this:

    [<class '__main__.Sphere'>, <class '__main__.Circle'>,
    <class '__main__.Ellipse'>, <class '__main__.Shape3D'>,
    <class '__main__.Shape'>, <class 'object'>] 

That list shows the order in which the classes will be consulted when trying to resolve a method (or class attribute) for Sphere. It also indicates the path that super() will walk as it works its way through the hierarchy.

super()-thread

It is against that backdrop that Martin Milon ("malmiteria") posted to python-ideas. He said that the MRO is effectively masking problems that the programmer should be required to resolve manually:

in case of multiple [inheritance], resolving a child method from it's parent isn't an obvious task, and mro comes as a solution to that. However, i don't understand why we don't let the programmer solve it. I think this is similar to a merge conflict, and not letting the programmer resolve the conflict feels like silencing an error. This is especially infuriating when you realise that mro doesn't solve all possible scenarios, and then, simply refuses the opportunity to solve it to the programmer. Then, super relying on mro gives off some weird behaviors, mainly, it's possible for a child definition to affect what a call to super means in it's parent. This feels like a side effect (which is the 'too implicit' thing i refer to).

He had a proposed solution to that problem, which is further described in a GitHub project repository, that he calls "explicit method resolution" (EMR). It would make no changes for single inheritance, but would force developers to use a new __as_parent__() method to explicitly choose a particular implementation of a method from among multiple possibilities in the parents, grandparents, and so on of a class. If that is not done, an exception would be raised.

But explicitly choosing the method resolution is already available in Python, David Mertz said: "You don't need `super()` to call `SomeSpecificClass.method(self, other, args)`". Milon agreed, but said that "super acts as a proxy to a parent, and is by default the feature to refer a parent from within a class method". Christopher Barker pointed out that super() is actually a proxy for the parents, not just a parent; "even if the current class inherits from only one class, that superclass may inherit from more than one class".

Milon gave a simple example, which did not use super() at all, that shows how he thinks the MRO is causing problems: two classes, A and B both implement method() and class C inherits from both (class C(A, B)):

Today, a code such as ```C().method()``` works without any problems except of course when the method you wanted to refer to was the method from B. If class A and B both come from libraries you don't own, and for some reason have each a method named the same (named run, for example) the run method of C is silently ignoring the run method of B.

The C3 mechanism uses the order in which two or more parent classes are listed in a class definition to determine the MRO. But Milon is saying that having two methods in the parent classes constitutes a "conflict" that should be resolved by the programmer. As he said in another message: "My point is that : it is not correct to assume it is meaningful to give an order to multiple [inheritance]". Stephen J. Turnbull disagreed:

[...] of course it's *meaningful* to give an order to multiple inheritance. Sometimes that order is *not useful*. Other times it may be *confusing*. Pragmatically, C3 seems to give useful results frequently, unuseful or confusing results infrequently [...]

Steven D'Aprano had a more fundamental objection to the example: "Why are you inheriting from A if you don't want to inherit from A?" He noted that A and B were not designed for multiple inheritance, which is the basis of the problem.

Multiple [inheritance] in Python is **cooperative** -- all of the classes in question have to work together. If they don't, as A and B don't, bad things happen.

You can't just inherit from arbitrary classes that don't work together. "Uncooperative multiple inheritance" is an unsolvable problem, and is best refactored using composition instead.

In that earlier message, Milon also showed an example to better explain where he thinks super() is confusing:

super is not a proxy to parents, even plural, it's a proxy to the next in mro order.

in this case :

class Top:
    def method(self):
        print('Top')
class Left(Top):
    def method(self):
        print('Left')
        super().method()
class Right(Top):
    def method(self):
        print('Right')
        super().method()
class Bottom(Left, Right):
    def method(self):
        print('Bottom')
        super().method()

Bottom().super() would print "Bottom", "Left", "Right", "Top". super, in Left, [refers] to Right, which is not a parent of Left (and is completely absent of it's definition)

He presumably means Bottom().method() there toward the end. That result might be surprising to some—possibly even confusing—but the super() call in Left uses the MRO context of Bottom, which is where the original super() call was made. The MRO for Bottom is: Bottom, Left, Right, Top, object. In that context, Left.method() will call Right.method() via super(). So the result is what many, but probably not all, Python programmers would expect. super() allows for an automatic way to pass a method call up through a set of cooperating classes; as Barker put it: "If you didn't use super in the whole chain— it would stop when it found the method. If you do, then they are all [called], and each one only once." In order to use super(), programmers have to follow some rules; "And then it works predictably. Whether that's helpful or not depends on your use case."

Some of those "rules" are embodied in two classic blog posts, as D'Aprano pointed out. One is "a provocatively titled blog post called 'super considered harmful'" by James Knight, though the title has changed. D'Aprano said that after some discussion, Knight backed down a bit, but still said: "super is great, 'but you can't use it.'". D'Aprano disagreed: "Except of course thousands of people do use it. And it works." The other blog post is Raymond Hettinger's "Python's super() considered super!", which describes how to properly use the feature.

D'Aprano followed up with another group of links to multiple blog posts by Michele Simionato, who wrote the MRO documentation for Python 2.3 linked above. There are two series, one on super() and one on Mixins, and some other topics, including multiple inheritance in various forms. D'Aprano said:

Multiple inheritance in Python works the way it does because it is modelling cooperative MI and the MRO and super are the Right Way to handle cooperative MI.

That doesn't mean that cooperative MI doesn't have problems. Other languages forbid MI altogether because of those problems, or only allow it in a severely restricted version, or use a more primitive form of super. MI as defined by Python is perhaps the most powerful, but also the most potentially complicated, complex, convoluted and confusing.

The discussion, links, and such had given Milon some food for thought, which he appreciated. He planned to regroup and come back with a clearer proposal that incorporates elements from the discussion and tries to address a persistent call for "real life" examples, rather than the somewhat abstract examples he had been using. He posted the results on April 3, which led to the second "half" of the mega-thread. That will provide more opportunities to dig into super() and other Python inheritance topics, which is coming in part 2 of our tale.