Growing pains for typing in Python [LWN.net]

Growing pains for typing in Python

Posted Jan 17, 2024 23:17 UTC (Wed) by Paf (subscriber, #91811)
Parent article: Growing pains for typing in Python

I appreciate the article.

As someone who comes to Python exclusively from typed languages, it is fascinating to me watching what I think is - frankly - a process of the language "growing up" and becoming more suitable for larger projects. It is also fascinating to have people argue against something which is happening by itself - Python is growing type checkers because *people see them as beneficial* (which they obviously are, for all sort of frankly obvious reasons). It's OK if the core Python developers want Python to be something else, but this steady growth of typing related tools? That's the actual usage telling you something.

Duck typing is a really fun way to program small stuff quickly. It's a laughably bad idea for large projects. I think Python is doing a pretty good job of striking a balance, but they're going to have to keep working at it, because typing is incredibly important for large programming projects. The path forward is almost certainly going to be towards more typing, because that's what the language doesn't have. Duck typing works great.

So it seems to me the question is can Python find a way to continue having both. It will likely require a more fulsome embrace of typing by the core developers so they can give it more careful attention rather than continuing to let the type system be defined by external tools. (It's amazing to read the bits where there are now multiple tools with *conflicting views of typing* but *it's important not to break them*. This way lies madness and the only way out is greater integration and - optional, sure - enforcement by some part of the core tools.

If the type system is basically enforced by a bunch of weird side projects, that's... it's just not going to go well over time.

to post comments

Growing pains for typing in Python

Posted Jan 18, 2024 2:55 UTC (Thu) by warrax (subscriber, #103205) [Link] (1 responses)

> Duck typing is a really fun way to program small stuff quickly.

I'm going to be nit-picky here. O'Caml is statically typed and supports Duck typing ("Structural types"). The term you're looking for is Run-time Type Checking.

Growing pains for typing in Python

Posted Jan 18, 2024 6:22 UTC (Thu) by benhoyt (subscriber, #138463) [Link]

Perhaps the term the original commentor was looking for was "run-time type checking", but to nitpick your nitpick :-), Python (without type annotations) definitely uses duck typing extensively, which is a form of run-time type checking. I'm not sure the term "duck typing" was coined for Python, but pretty close -- Pythonista Alex Martelli used the term on comp.lang.python back in 2000 [1]. See also the term in the Python glossary. [2]

In statically typed languages like O'Caml and Go, the related concept is called "structural typing" and not "duck typing". In Python with type annotations, structural typing (statically checked) is supported with PEP 544 "protocols". [3]

See also the sections on typing in the Wikipedia articles about Python [4] and O'Caml [5].

[1] https://groups.google.com/g/comp.lang.python/c/CCs2oJdyuz...
[2] https://docs.python.org/3/glossary.html#term-duck-typing
[3] https://docs.python.org/3.8/library/typing.html#nominal-v...
[4] https://en.wikipedia.org/wiki/Python_(programming_language)#Typing
[5] https://en.wikipedia.org/wiki/OCaml#Features

Growing pains for typing in Python

Posted Jan 18, 2024 16:29 UTC (Thu) by aigarius (guest, #7329) [Link] (9 responses)

Avoiding the typing has been the best thing ever, *especially* for large and highly distributed projects. It becomes indispensable when objects from one project needs to be processed by functions from another project and it is possible, even if the developers of either project have never even heard about each other. As long as the objects have the methods that the other projects function is calling on them and those methods represent same kind of logical sense then it will just work. There is no realistic way to have that described by a type system in any way that makes any sense to the computer without a significant knowledge investment from the developer that is doing the bridge/integration between the two projects. And at that point the typing system does not actually add any value.

I would argue the other way - typing is only useful in a *small* project where you have overview of the whole codebase and are able to specify definitions that are used across the whole codebase, at least at interface definition level.

Growing pains for typing in Python

Posted Jan 18, 2024 17:08 UTC (Thu) by pizza (subscriber, #46) [Link] (1 responses)

> As long as the objects have the methods that the other projects function is calling on them and those methods represent same kind of logical sense then it will just work

...except, of course, when it doesn't. Which is the entire point of this discussion.

BTW, the term for what you're referring to is "interfaces", and that is typically enforced by the same language mechanism that enforces typing. (It's routinely used in large/distributed Java environments in particular))

Growing pains for typing in Python

Posted Jan 19, 2024 16:41 UTC (Fri) by aigarius (guest, #7329) [Link]

Funny thing that - external interfaces can not enforce anything and the behavior of functions of the same name can easily be different without breaking any interfaces. If we have a class A and a function B, that come from completely different projects that have never even heard about each other, then there will never be compatible type or interface definitions between them. So you, as the person bringing those two pieces together are the first one to try to map types or interfaces from one project to the other. All you are doing is writing down your opinion of whether those things are compatible or not. You can not enforce any of that - the way the function uses the methods of the interface or how the class implements the methods can change in ways that do not violate *their* type and interface definitions, but will violate *your* assumptions.

A person needs to determine if there is a *logical* sense in *how* function B is using methods on class A. No type system can tell you that.

And those can be really simple things, like sorting objects in ascending order or serialization them to JSON for RPC calls.

I just hit one such case today, that is broken right now inside the Python standard library. If you use pathlib Path objects and construct a complex, multilevel dict with the Path objects somewhere in the structure and then try to pass this structure to another server via jsonrpc call, well the whole thing comes crashing down because the Path objects are not JSON-serializable. Even if they are just used as a fancier proxy for simple text strings. If you use fancy types you get punished with fancy failures.

Growing pains for typing in Python

Posted Jan 18, 2024 18:13 UTC (Thu) by Paf (subscriber, #91811) [Link]

Sure, if you're creating your own baroque classes and trying to do all kinds of things to them, this will be a problem for you. The answer is presumably not to do that, and also that it's pretty rare.

Also, presumably there are ways to work around the issues you describe - they certainly don't prevent C# and Java from having absurd numbers of libraries. Don't slow them down in the slightest as far as I can see.

I would also add that the world at large disagrees about large vs small projects - Larger Python projects are the ones driving use of type checking.

Growing pains for typing in Python

Posted Jan 18, 2024 19:47 UTC (Thu) by roc (subscriber, #30627) [Link]

> It becomes indispensable when objects from one project needs to be processed by functions from another project and it is possible, even if the developers of either project have never even heard about each other. As long as the objects have the methods that the other projects function is calling on them and those methods represent same kind of logical sense then it will just work.

Anyone got an example of this happening in a non-toy case? Because I've never seen it and I struggle to imagine how it's possible. You would have to be ridiculously lucky for the developers in the two projects to independently choose the exact same name, parameter ordering, etc for every method.

Growing pains for typing in Python

Posted Jan 19, 2024 5:36 UTC (Fri) by intelfx (subscriber, #130118) [Link]

> There is no realistic way to have that described by a type system in any way that makes any sense to the computer without a significant knowledge investment from the developer that is doing the bridge/integration between the two projects

Well, if the two projects themselves are not using typing, then of course not.

But if they do, they should be using structural typing that's been mentioned just one thread above.

Growing pains for typing in Python

Posted Jan 19, 2024 9:10 UTC (Fri) by ringerc (subscriber, #3071) [Link] (2 responses)

Where I find that this falls down is when you go to change things.

Test coverage is great and all. But it's amazing how many bugs can be present in a code base with 100 line based coverage. Even if you check coverage of branch permutations in functions (hardly anybody does), it's basically impossible to do it deeper than function scope, and only in small functions.

Type systems make it much easier to say "doing this shouldn't work" or "the meaning of this has changed and you need to check everywhere that consumes the old thing to make sure it will understand the new thing".

Yes, it's tedious, harder to design well, and can be lots of pointless-seeming labour. They are IMO overused unnecessarily. But for interfaces between large complex systems they're fantastic, and in languages without type systems people end up inventing their own inconsistent homebrew for this sort of job.

Growing pains for typing in Python

Posted Jan 20, 2024 10:06 UTC (Sat) by anton (subscriber, #25547) [Link] (1 responses)

Where I find that this falls down is when you go to change things.

My experience is that run-time type checking works surprisingly well in the face of changes. The case I have in mind is a game with a Lua API that has seen hundreds of add-ons written over the last 15 years, probably far beyond what the designers of the API imagined. The API has seen occasional updates (including downgrades, to prevent botting) of the API, and the add-ons interact with one another, and receive new versions separately, and are updated by the users separately. Yes, occasionally there are things that do not work because of some incompatibility of an add-on with the upgrade of a different one, but then that is usually solved by an upgrade of one of the involved add-ons.

Type systems make it much easier to say "doing this shouldn't work" or "the meaning of this has changed and you need to check everywhere that consumes the old thing to make sure it will understand the new thing".

Exactly. So what would have happened if the game designers had used Java with its static type checking rather than Lua for the API? The add-on authors would have gotten "doing this shouldn't work" for many of the things they did that were outside what the API designers had in mind. And when the API authors wanted to downgrade the API, if they wanted to maintain "doing this shouldn't work", they would have produced a new API, and the earlier add-ons would have stopped working, because they would have been incomatible with the new API. Likewise if one add-on B extended another one A, and A was upgraded. Yes, the benefit you describe above would be there, but the cost is that there would not have been such a thriving add-on ecosystem; that cost far exceeds the benefit in this case; and I think there are many others.

Growing pains for typing in Python

Posted Jan 20, 2024 12:20 UTC (Sat) by pizza (subscriber, #46) [Link]

> So what would have happened if the game designers had used Java with its static type checking rather than Lua for the API?

Nothing, if the API was designed properly to use Java's Interfaces.

Growing pains for typing in Python

Posted Jan 20, 2024 2:26 UTC (Sat) by gps (subscriber, #45638) [Link]

PEP 544 Protocols which could also be thought of as partial ABCs and other languages may call Interfaces or Structural Types exist and let us do what you claim is impossible.

Some of the largest proponents for and deployments of Python static type checking are those maintaining tens to hundreds of millions of lines of Python code.