Growing pains for typing in Python [LWN.net]

Growing pains for typing in Python

Posted Jan 18, 2024 16:29 UTC (Thu) by aigarius (guest, #7329)
In reply to: Growing pains for typing in Python by Paf
Parent article: Growing pains for typing in Python

Avoiding the typing has been the best thing ever, *especially* for large and highly distributed projects. It becomes indispensable when objects from one project needs to be processed by functions from another project and it is possible, even if the developers of either project have never even heard about each other. As long as the objects have the methods that the other projects function is calling on them and those methods represent same kind of logical sense then it will just work. There is no realistic way to have that described by a type system in any way that makes any sense to the computer without a significant knowledge investment from the developer that is doing the bridge/integration between the two projects. And at that point the typing system does not actually add any value.

I would argue the other way - typing is only useful in a *small* project where you have overview of the whole codebase and are able to specify definitions that are used across the whole codebase, at least at interface definition level.

to post comments

Growing pains for typing in Python

Posted Jan 18, 2024 17:08 UTC (Thu) by pizza (subscriber, #46) [Link] (1 responses)

> As long as the objects have the methods that the other projects function is calling on them and those methods represent same kind of logical sense then it will just work

...except, of course, when it doesn't. Which is the entire point of this discussion.

BTW, the term for what you're referring to is "interfaces", and that is typically enforced by the same language mechanism that enforces typing. (It's routinely used in large/distributed Java environments in particular))

Growing pains for typing in Python

Posted Jan 19, 2024 16:41 UTC (Fri) by aigarius (guest, #7329) [Link]

Funny thing that - external interfaces can not enforce anything and the behavior of functions of the same name can easily be different without breaking any interfaces. If we have a class A and a function B, that come from completely different projects that have never even heard about each other, then there will never be compatible type or interface definitions between them. So you, as the person bringing those two pieces together are the first one to try to map types or interfaces from one project to the other. All you are doing is writing down your opinion of whether those things are compatible or not. You can not enforce any of that - the way the function uses the methods of the interface or how the class implements the methods can change in ways that do not violate *their* type and interface definitions, but will violate *your* assumptions.

A person needs to determine if there is a *logical* sense in *how* function B is using methods on class A. No type system can tell you that.

And those can be really simple things, like sorting objects in ascending order or serialization them to JSON for RPC calls.

I just hit one such case today, that is broken right now inside the Python standard library. If you use pathlib Path objects and construct a complex, multilevel dict with the Path objects somewhere in the structure and then try to pass this structure to another server via jsonrpc call, well the whole thing comes crashing down because the Path objects are not JSON-serializable. Even if they are just used as a fancier proxy for simple text strings. If you use fancy types you get punished with fancy failures.

Growing pains for typing in Python

Posted Jan 18, 2024 18:13 UTC (Thu) by Paf (subscriber, #91811) [Link]

Sure, if you're creating your own baroque classes and trying to do all kinds of things to them, this will be a problem for you. The answer is presumably not to do that, and also that it's pretty rare.

Also, presumably there are ways to work around the issues you describe - they certainly don't prevent C# and Java from having absurd numbers of libraries. Don't slow them down in the slightest as far as I can see.

I would also add that the world at large disagrees about large vs small projects - Larger Python projects are the ones driving use of type checking.

Growing pains for typing in Python

Posted Jan 18, 2024 19:47 UTC (Thu) by roc (subscriber, #30627) [Link]

> It becomes indispensable when objects from one project needs to be processed by functions from another project and it is possible, even if the developers of either project have never even heard about each other. As long as the objects have the methods that the other projects function is calling on them and those methods represent same kind of logical sense then it will just work.

Anyone got an example of this happening in a non-toy case? Because I've never seen it and I struggle to imagine how it's possible. You would have to be ridiculously lucky for the developers in the two projects to independently choose the exact same name, parameter ordering, etc for every method.

Growing pains for typing in Python

Posted Jan 19, 2024 5:36 UTC (Fri) by intelfx (subscriber, #130118) [Link]

> There is no realistic way to have that described by a type system in any way that makes any sense to the computer without a significant knowledge investment from the developer that is doing the bridge/integration between the two projects

Well, if the two projects themselves are not using typing, then of course not.

But if they do, they should be using structural typing that's been mentioned just one thread above.

Growing pains for typing in Python

Posted Jan 19, 2024 9:10 UTC (Fri) by ringerc (subscriber, #3071) [Link] (2 responses)

Where I find that this falls down is when you go to change things.

Test coverage is great and all. But it's amazing how many bugs can be present in a code base with 100 line based coverage. Even if you check coverage of branch permutations in functions (hardly anybody does), it's basically impossible to do it deeper than function scope, and only in small functions.

Type systems make it much easier to say "doing this shouldn't work" or "the meaning of this has changed and you need to check everywhere that consumes the old thing to make sure it will understand the new thing".

Yes, it's tedious, harder to design well, and can be lots of pointless-seeming labour. They are IMO overused unnecessarily. But for interfaces between large complex systems they're fantastic, and in languages without type systems people end up inventing their own inconsistent homebrew for this sort of job.

Growing pains for typing in Python

Posted Jan 20, 2024 10:06 UTC (Sat) by anton (subscriber, #25547) [Link] (1 responses)

Where I find that this falls down is when you go to change things.

My experience is that run-time type checking works surprisingly well in the face of changes. The case I have in mind is a game with a Lua API that has seen hundreds of add-ons written over the last 15 years, probably far beyond what the designers of the API imagined. The API has seen occasional updates (including downgrades, to prevent botting) of the API, and the add-ons interact with one another, and receive new versions separately, and are updated by the users separately. Yes, occasionally there are things that do not work because of some incompatibility of an add-on with the upgrade of a different one, but then that is usually solved by an upgrade of one of the involved add-ons.

Type systems make it much easier to say "doing this shouldn't work" or "the meaning of this has changed and you need to check everywhere that consumes the old thing to make sure it will understand the new thing".

Exactly. So what would have happened if the game designers had used Java with its static type checking rather than Lua for the API? The add-on authors would have gotten "doing this shouldn't work" for many of the things they did that were outside what the API designers had in mind. And when the API authors wanted to downgrade the API, if they wanted to maintain "doing this shouldn't work", they would have produced a new API, and the earlier add-ons would have stopped working, because they would have been incomatible with the new API. Likewise if one add-on B extended another one A, and A was upgraded. Yes, the benefit you describe above would be there, but the cost is that there would not have been such a thriving add-on ecosystem; that cost far exceeds the benefit in this case; and I think there are many others.

Growing pains for typing in Python

Posted Jan 20, 2024 12:20 UTC (Sat) by pizza (subscriber, #46) [Link]

> So what would have happened if the game designers had used Java with its static type checking rather than Lua for the API?

Nothing, if the API was designed properly to use Java's Interfaces.

Growing pains for typing in Python

Posted Jan 20, 2024 2:26 UTC (Sat) by gps (subscriber, #45638) [Link]

PEP 544 Protocols which could also be thought of as partial ABCs and other languages may call Interfaces or Structural Types exist and let us do what you claim is impossible.

Some of the largest proponents for and deployments of Python static type checking are those maintaining tens to hundreds of millions of lines of Python code.