LWN: Comments on "Super Python (part 2)"

Bottom2().method() didn't print Top

thomastthai — Sun, 16 Oct 2022 01:01:03 +0000

I realized 'super(Right, self).method' was missing '()'. Added them and the output is now correct.

class Bottom2(Left, Right):
def method(self):
print('Bottom2')
super(Right, self).method # <==========

>>> class Bottom2(Left, Right):
... def method(self):
... print('Bottom2')
... super(Right, self).method()
...
>>> Bottom2().method()
Bottom2
Top
>>>

Bottom2().method() didn't print Top

thomastthai — Sat, 15 Oct 2022 23:09:28 +0000

With Python 3.10.7, the following command didn't print "Top" for the last line as expected and per the article. Who else got a similar result?

>>> Bottom2().method()
Bottom2

-----------------------------------------------
>>> import platform
>>> platform.python_version()
'3.10.7'
>>> class Top:
... def method(self):
... print('Top')
...
>>>
>>> class Left(Top):
... def method(self):
... print('Left')
... super().method()
...
>>> class Right(Top):
... def method(self):
... print('Right')
... super().method()
...
>>> class Bottom(Left, Right):
... def method(self):
... print('Bottom')
... super(Left, self).method()
...
>>> class Bottom2(Left, Right):
... def method(self):
... print('Bottom2')
... super(Right, self).method
...
>>> Bottom().method()
Bottom
Right
Top
>>>
>>> Bottom.mro()
[<class '__main__.Bottom'>, <class '__main__.Left'>, <class '__main__.Right'>, <class '__main__.Top'>, <class 'object'>]
>>> Bottom2().method()
Bottom2
>>>
>>> Bottom2.mro()
[<class '__main__.Bottom2'>, <class '__main__.Left'>, <class '__main__.Right'>, <class '__main__.Top'>, <class 'object'>]
>>>

Super Python (part 2)

anselm — Mon, 13 Jun 2022 10:11:52 +0000

Anyways, hasn't Guido resigned from the BDFL? And I think people are beginning to realise that pushing him into doing that was a BAD move ...

AFAIR, Guido van Rossum stepped down from the BDFL position for Python voluntarily – he wasn't “pushed”, but felt increasingly frustrated about his role and probably also thought that, at age 60+ and having been in charge of Python for almost 30 years, it was time for some succession planning. His move came as a surprise for many people at the time.

Super Python (part 2)

Wol — Sat, 11 Jun 2022 15:19:02 +0000

Bear in mind that "Benevolent Dictator" is a very successful project survival strategy. And after all, it was Guido's project to start with. Plus people don't normally like forking projects.

I personally refer to Linus as a tyrant. After all, "Tyrant" is actually an elected position! Look at your Roman history, and tyrants were elected to absolute power - but they only have it as long as they keep your trust.

Anyways, hasn't Guido resigned from the BDFL? And I think people are beginning to realise that pushing him into doing that was a BAD move ...

Cheers,
Wol

Super Python (part 2)

madscientist — Sat, 11 Jun 2022 15:01:51 +0000

Check it out: https://en.wikipedia.org/wiki/Benevolent_dictator_for_life

Super Python (part 2)

mobirre — Sat, 11 Jun 2022 09:06:16 +0000

Well, you know guys, Python being the most popular and almost most easy to get started with Computer-Language these days, besides Java maybe, what makes me cautious about is is the inventor's van Rossum's title of benevolent dictator for life. Is this a joke anyway, or does he actually mean it?
I used to do some programming with Amiga-BASIC in thee days of my youth, the downside of that being, that the Language and IDE was copyrighted by Microsoft. So is van Rossum the Gates of the ever expanding open-source world anyway??

Super Python (part 2)

glasserc — Sat, 07 May 2022 18:38:19 +0000

OK, so just to circle back, does this mean that the comparison with C++ is not actually instructive?

> The example is, IMHO, meaningless because it wrongly expects that super is syntactic sugar for HighGoblin-but-with-self-bound

But like I said, the documentation literally says that one of the main use cases is to call the parent class without naming it explicitly. It seems like you're arguing that there are really two separate use cases here and they should not both be provided by `super`. Does that mean you agree with the "screwhammer" point of view (which I also agree with, per my first message)?

P.S. Sorry for the delayed response, work has been busy.

Super Python (part 2)

NYKevin — Fri, 29 Apr 2022 23:20:18 +0000

Just to make this a bit more explicit: When I said "you generally won't have a problem," my assumption was that you either blindly call super() to do "whatever the parent class wants to do," or you completely fulfill the documented contract on your own, without worrying about which superclass actually gets involved. The whole point of the Liskov Substitution Principle is that this should be a safe thing to do; if your code violates LSP, then you are not really doing OOP in the first place.

Super Python (part 2)

NYKevin — Fri, 29 Apr 2022 23:07:06 +0000

The by-design purpose of super() is "Please find and invoke the next class in the MRO, whatever that class may be." If you don't want to do that, you write the parent class's name explicitly (i.e. the example should say HighGoblin.scream(self) instead of super().scream()), or don't use inheritance in the first place. The example is, IMHO, meaningless because it wrongly expects that super is syntactic sugar for HighGoblin-but-with-self-bound,* but nobody ever promised that behavior.

* It would be possible, with minor difficulty, to make a HighGoblin-with-self-bound object in pure Python; it does not need to be a language feature. If anyone really wants that functionality, it could be a PyPI package.

Super Python (part 2)

glasserc — Fri, 29 Apr 2022 18:48:01 +0000

OK, I understand that if two different classes manipulate the same state then bad things can happen, but again, what does this have to do with the subject of the article, which is about super() and method resolution order? The "goblin" example is surprising to me but does not satisfy your point #2 (because all classes involved are stateless).

Super Python (part 2)

esemwy — Fri, 29 Apr 2022 14:35:54 +0000

Huh. If you don’t understand how your programming language works, it does things you don’t expect.

Honestly, I think getting bitten by this is a feature, not a bug. Multiple inheritance is tricky. A programmer would be well advised to read up on it before using it. If you don’t, expect the unexpected. Furthermore, if you don’t understand it, for Pete’s sake, don’t try to use it, or your coworkers will hate you.

Super Python (part 2)

martin.langhoff — Thu, 28 Apr 2022 19:01:05 +0000

Any software design/arch that relies on complex class inheritance as depicted here is ... not in a good place.

Simple subclassing has its place under the sun – base classes defining the API, etc. Once you get significantly past that, "has_a" relationships are far superior to "is_a".

Super Python (part 2)

NYKevin — Thu, 28 Apr 2022 15:55:16 +0000

For the most part, in C++, you generally won't have a problem unless all of the following occur:

1. Classes A and B are both subclasses of class C.
2. A and B both try to manipulate the state of the C instance.
3. A and B both maintain invariants which involve the state of the C instance.
4. A and B are not aware of each others' C-related invariants, and those invariants are not identical.
5. You multiply-inherit from A and B.

The reason for this is simple: The C instance is the only bit of state that is accessible to both A and B. If A and B are going to interfere with one another, the interference simply must pass through the C instance in some way. This is possible if C is a stateful class, and you can run into problems that way, but the instance-per-class encapsulation makes it much harder for this to happen "by accident."

You *can* also run into problems if A and B override non-disjoint subsets of C's methods, and one or both of them tacitly assumes that their overrides will always be called in preference to any other class, but this generally only happens if C has a lot of very specific requirements for how you're "allowed" to call its methods (e.g. "You must call foo() first, and then pass its return value to bar(), and then..."). In general OOP, we try to avoid this sort of thing, because it's probably a violation of the Single Responsibility Principle, and certainly a case of bad interface design. If you assume that methods are mostly or entirely independent of each other, and any (public) method may be called at any time, then it's hard for this problem to manifest in practice.

OTOH, here's that list in Python (assuming, for the sake of argument, that we *don't* have a naming clash):

1. Classes A and B are both subclasses of class C.
2. A and B both try to modify the behavior of C, and both modifications involve C's private member variables or other state in some way.
3. A and B are both maintain certain invariants with respect to C's behavior.
4. A and B are not aware of each others' C-related invariants, and those invariants are not identical.
5. You multiply-inherit from A and B.

In Python, it's all one big instance. If A decides to reach inside of C's internals and fiddle with a private member variable, that's perfectly fine and allowed (whereas in C++, it's generally not a thing you're allowed to do, unless you use the protected access modifier, which I personally don't see very often in code). As a result, it's entirely possible for A and B to both fiddle with C's internals, but in different ways, and break each other.

Super Python (part 2)

glasserc — Thu, 28 Apr 2022 13:55:08 +0000

That's very interesting, but none of the examples in the article have anything to do with accidentally using the parent class's attributes, which anyhow isn't affected by method resolution order or the use of super(). If we are focusing on "dependencies and side effects" (as described by cavok in another comment), then it seems like multiple inheritance would make those challenging both in C++ and Python, no?

Super Python (part 2)

gedeon — Thu, 28 Apr 2022 07:42:51 +0000

Thank you sir! This is the kind of comment that makes me want to keep reading comments on LWN (I usually don't bother on other sites).

Super Python (part 2)

NYKevin — Wed, 27 Apr 2022 23:32:44 +0000

I think it would help to compare and contrast Python's inheritance model with C++, because they really are two entirely different data models that just happen to be useful in similar (but not identical) use cases and happen to provide similar (but not identical) semantics.

In C++, inheritance is basically a formalized, type-safe version of the old C pattern where you add extra fields to the end of a struct to specialize it (see for example the old Berkeley sockets API). If A is the parent and B is the child, then every instance of B contains a complete instance of A. To construct a B, you construct an A first and then "tack on" the extra members to turn it into a B. The type of the object changes during the construction process. As a result, A remains totally encapsulated and has no knowledge that B even exists. A::foo and B::foo will not conflict with each other, for example, and so B cannot accidentally break A's invariants, unless A has wildly unreasonable expectations or B is doing something really nasty. The multiple-inheritance support, then, is simply a few extra RTTI/vtable tricks wherein you avoid creating multiple instances of a shared parent class, by adding an indirection layer that allows different child instances to unknowingly share the same parent instance. Since everything is properly encapsulated, this (usually) Just Works and all is right in the world.

In Python, it is a completely different story. First of all, a class is a first-class object. When you write "class Foo: ...", Python allocates a real object on the heap to represent the Foo class. You could think of this object as the world's heaviest vtable, I suppose, but it supports all of the same operations as any other Python object, and so we say that Python has first-class classes. Secondly, Python objects do not have this sort of inter-class encapsulation in their namespaces; a Python object has (at most) one __dict__ which contains all of its attributes, and these attributes don't "belong to" the classes in any meaningful sense (instead, they belong to the instance). If an attribute is not present in the __dict__, then it will be looked up by going through the classes according to the MRO, but this is mostly only relevant for method resolution because regular attributes are usually in __dict__ (hence the name "method resolution order" as opposed to "attribute resolution order"). As a result, instance-level attributes can and do conflict, unless you use the double-underscore name-mangling trick (any attribute whose name begins with a double underscore, and which is not a __special__ method name, is automatically name-mangled to include the name of the class, which at least rules out accidental collisions). But that trick was never very popular, and its usage is not standardized in PEP 8 or (to my knowledge) any other style guide.

There are advantages and disadvantages to both approaches; in particular, Python's metaclass support enables a much more natural and intuitive form of metaprogramming compared to the morass that is C++ template metaprogramming (C++ cannot use a metaclass-based approach, because it requires that classes exist at runtime). But the other side of the coin is that Python requires subclasses to be aware of their parents' invariants and private names, and to avoid stepping all over them. This is not too hard in the single-inheritance case, but it becomes more of a challenge in MI simply because there are more classes involved and more inter-class conflicts are therefore possible.

Super Python (part 2)

cavok — Wed, 27 Apr 2022 21:27:23 +0000

Althought I always considered super more lateral than vertical, a few times I found myself entangled in MI, mro and super. I suspect it will happen again..

It's easy to slip on the dependencies of side effects among the base classes. After some battling I end up wondering what I'm trying to do and if super (or, actually, MI) is the right tool for that.

Super Python (part 2)

glasserc — Wed, 27 Apr 2022 15:09:56 +0000

The linked code using ProudGobelin.scream(self) and CorruptedGobelin.scream(self) doesn't work by itself because the super call in ProudGobelin.scream will call CorruptedGobelin.scream, so you get a 50% chance of CorruptedGobelin and a 50% chance of ProudGobelin followed by CorruptedGobelin. You'd have to change both of them to refer to HighGobelin explicitly I guess.

Reading this article I found it harder to conceptualize a real use case for super as defended by the Python maintainers. You don't know what you will be called from and you don't know who you will be calling to, which seems like it would make everything especially side effects and argument handling tricky. I went to the Python documentation to see what example they offered. The documentation for super does not offer an example, but does include this text:

"There are two typical use cases for super. In a class hierarchy with single inheritance, super can be used to refer to parent classes without naming them explicitly ... The second use case is to support cooperative multiple inheritance in a dynamic execution environment."

The fact that these two use cases are totally disjoint does seem like an argument in favor of the "screwhammer" point of view!

The documentation also links to Raymond Hettinger's Python's super() considered super, which uses as an example a LoggingDict class which logs calls to __setitem__, and composes this class with OrderedDict. This is cool but I wonder how generalizable it is. There's also an example of shapes where multiple inheritance is used to implement color as well as position.

Super Python (part 2)

smurf — Wed, 27 Apr 2022 14:15:57 +0000

So if you want to follow the MRO you use super().

If not, you call the superclass of the current class, which is how things worked before super() was invented in the first place. (That code still can be found in quite a few old Python libraries.)

You can even write generic code to do this, by way of "super().__thisclass__.methodname(self, …)". Yes that's somewhat ugly, but it works.

Super Python (part 2)

gedeon — Wed, 27 Apr 2022 08:34:58 +0000

Thanks for the super article ;-). Even as a seasoned Python developer, a reminder on how this works is very much appreciated.