LWN: Comments on "Wrangling the typing PEPs"

Missing the point of loose languages

smurf — Mon, 27 Dec 2021 09:59:31 +0000

IMHO the best thing about explicit typing is documentation. I can see what a variable/parameter is supposed to be without checking external and possibly-out-of-date docs (including broken comments right above the function in question …).

A decent set of test cases is great for checking that the expected cases work as expected. The problem is that some [ft]ools insist on tests for each and every unexpected case, even if logically impossible to reach, while the real problems (i.e. those conditions you didn't even think of being remotely possible when you wrote the code) slip under the radar.

Missing the point of loose languages

NAR — Mon, 27 Dec 2021 08:26:01 +0000

Erlang is one language where the type system was added later. In my experience it's most useful for documentation - now there's a standard way to specify the type of the arguments in an API function. There's a tool (dialyzer) that checks for type errors: I worked on a project where the type annotations were added on later, about 95% of the errors it found were in dead code (so not covered by unit tests - one such example was a branch that only executed if the current date was before 2000, and we were adding the type specs in 2010). If there is a decent test coverage, there's little (but not zero!) value the type system can add.

Missing the point of loose languages

tbelaire — Tue, 21 Dec 2021 18:55:14 +0000

I hear you, but then I had to deal with upgrading some email lib from 2 to 3 which was mixing bytes and strings and it was so much easier to do in stages by first annotating the functions with types, and fixing mistakes, then doing the py3 upgrade.

Until I started annotating with types, I was dealing with "Oh, that's a bytes regex, doesn't work on str", on line xxx. And other errors, where as being careful and making sure the encoding/decoding happened exactly once, and using the types to check that across all the functions and not playing wack-a-mole with errors is so much nicer.

And when I'm working with pandas, I do want to know if I have a DF with only one column vs a Series, as the operations are different, but it's just `df['col'] ` vs `df[['col']]` to get each one. (I think). So I already need to check and keep the types separate, why not ask the compiler for help?

Missing the point of loose languages

smurf — Sun, 19 Dec 2021 10:15:56 +0000

Haskell is one of these "you need to wrap your head around these new concepts, otherwise you'll feel like bashing your noggin against a brick wall is the saner option when you actually start coding in it" languages.

That takes more up-front time and effort than most people want to spend. Plus everybody else in your team needs to do the same thing.

I was looking into using Elm (a Haskell-ish language for client-side web programming; transpiles to Javascript of course) recently. Great idea, but where do I get the month that's not in the calendar from, so that I can be up to speed on it?

Wrangling the typing PEPs

smitty_one_each — Sat, 18 Dec 2021 00:46:04 +0000

> Those are valuable resources, obviously, but they do tend to reinforce the message that the future of Python is typed.

I don't see a two-tier ecosystem consisting of:
1. Traditional python for scripting/exploratory work
2. Typed libraries for medium-large projects
...as a bad thing.

As with the 2-to-3 transition, there is just a turbulent crossover period. Welcome to reality.

Maybe it's a big enough deal to call it python 4 when codified and formalized.

Missing the point of loose languages

Wol — Fri, 17 Dec 2021 19:11:54 +0000

> I highly welcome a generic type annotation system that codes like Cython make use of. But imposing a mandatory rigid typing system is the wrong approach. If strong static typing is needed, there is a big choice of other languages out there (go, rust, C++, Java, Haskell, ....ML).

The problem with my favourite language is everything is of type string (or variant). The advantage of this same language is that everything is of type variant. It would be lovely if I could just restrict things by having the default as variant, but options like "integer", "number", and stuff like that available ...

(Said favourite language being DataBASIC, btw)

Cheers,
Wol

Missing the point of loose languages

mpr22 — Fri, 17 Dec 2021 16:57:20 +0000

> Why not use Haskell right away?

My understanding is that a lot of people who love static typing etc. still bounce off of Haskell.

(Partly because a lot of the literature around Haskell has a reputation for using terminology that even many CS graduates are unfamiliar with.)

Missing the point of loose languages

tnoo — Fri, 17 Dec 2021 15:56:56 +0000

Exactly my feeling!

Python was (still mostly is) so clean and at the same time powerful enough for all my use cases (data analysis, glue language for numerical codes). And just perfect for teaching introductory classes for scientists.

I highly welcome a generic type annotation system that codes like Cython make use of. But imposing a mandatory rigid typing system is the wrong approach. If strong static typing is needed, there is a big choice of other languages out there (go, rust, C++, Java, Haskell, ....ML).

But of course, this also comes with a high cost. Besides the need for compilation, for C++ we have the Design Patterns (Gang of Four) with Facade, Adapter, Decorator patterns etc that are not necessary in Python because of Duck-Typing.

For me Python is getting less and less attractive if more line noise is added, and simple code is becoming unreadable. From a comment above

def foo(a: object, b: type[T]) -> TypeGuard[T]:

does not look and feel like Python anymore. Why not use Haskell right away?

Wrangling the typing PEPs

mathstuf — Fri, 17 Dec 2021 12:26:20 +0000

> Perhaps this is one of the few times PHP has something to teach the wider community.

There's always been things to learn. This is just one of the times where it's not following a "eh…maybe let's not do it *that* way" pattern. ;)

Wrangling the typing PEPs

atnot — Fri, 17 Dec 2021 10:28:17 +0000

Yes, that is my point.

Wrangling the typing PEPs

LtWorf — Fri, 17 Dec 2021 07:05:33 +0000

It's not like mypy ever had any regard for consistency.

For example running the same mypy version with the same identical flags, but using different python versions will give different errors in a way that it might be impossible to use both on the same codebase.

Also in general every new version of mypy will require code changes to pass on a codebase. Especially if you enable the error on the # type: ignore comments, so as it gets better and requires less and less of them they need to be removed.

Wrangling the typing PEPs

NYKevin — Fri, 17 Dec 2021 06:39:08 +0000

Runtime type checkers will probably be perfectly happy with that, but I am not aware of a single static type checker that actually evals the annotations in a full Python environment for you.

Wrangling the typing PEPs

NYKevin — Fri, 17 Dec 2021 06:34:30 +0000

You can't just make up new dunder methods. Standard dunder methods are recognized by the Python compiler and (ultimately) converted into struct fields at the C level. Strictly speaking, if a dunder method is not recognized, it will be left alone, but a future version of Python might decide to use that name, and if it does, you get no backcompat guarantees (see https://docs.python.org/3/reference/lexical_analysis.html...). You don't even get a DeprecationWarning, either, it will just silently break one day.

However, classes are first class in Python, so you can absolutely do this already without using dunder names. Just write a.into(MyType), and it will pass the class object. Now the only hard part is convincing everyone that .into() should be spelled like that (more common is .as_foo() where the type is hard-coded into the method name).

Missing the point of loose languages

rdeforest — Fri, 17 Dec 2021 01:38:36 +0000

I've only read the intro paragraph of the article and the first comment, so this may be redundant or already refuted elsewhere, BUT!

Languages like Python, PHP and JavaScript gained enormous popularity BECAUSE the users didn't have to deal with types. The barriers to entry were low and the things type information help with didn't matter enough. Adding type information to these languages after they've become popular misses the point of how they got popular in the first place. I would even posit that it objectively worsens them. The eventual of triumph of TypeScript over JavaScript would refute my claim, if it happens.

I would suggest as an alternative to extending existing dynamic languages, that those seeking these features instead choose languages built around them. But those languages (Haskel, I'm looking at you) don't have the enormous communities and the network effects they bring...

In other words, Richard P Gabriel was right all along: "Worse Is Better".

Wrangling the typing PEPs

jkingweb — Fri, 17 Dec 2021 01:12:34 +0000

> It seems to me that dynamically and/or weakly typed languages like Python are struggling a bit in adding optional typing in general.

The introduction of a type system in PHP has been a great success. It has been gradual, optional, and backwards compatible across heterogeneous programs—and essentially universally adopted.

Perhaps this is one of the few times PHP has something to teach the wider community.

Wrangling the typing PEPs

atnot — Thu, 16 Dec 2021 23:23:56 +0000

It's my fault for not explaining this better, but Rusts's Into/From system is quite a bit more capable than that. In a python constructors, the set of things that can be converted is defined only in the constructor. With Into/From, it can be implemented on any type, even foreign ones.

So for example, you could have Into<String> implemented for Paths, or From<Path> implemented for Strings, it works exactly the same either way. There is no common way to do this in python right now. Although it could probably be built from two existing features:

- The constructor/dunder pair I mentioned previously. Functions like float() will attempt to convert the object into bytes using predefined conversions. If that fails it will ask the object to convert itself using __float__.
- Operators, where a + b will attempt to call a.__add__(b) and if that fails b.__radd__(a). Similarly, it could call a.__into__(MyType) and MyType.__from__(a) or even MyType(a) as you mentioned.

Wrangling the typing PEPs

NYKevin — Thu, 16 Dec 2021 22:48:32 +0000

List absolutely does not mean "object with list protocol." That's spelled collections.abc.MutableSequence (see this table: https://docs.python.org/3/library/collections.abc.html#co...). As of Python 3.9, typing.List is just a deprecated alias for list. Previously, it was "a list that can have type parameters," but now that functionality has been moved into the builtin.

(Strictly speaking, it is possible to create an object with the mutable sequence protocol which is not a subclass of MutableSequence, but this is significantly more work and there's no obvious benefit to doing so, so I would like to believe that nobody would do it.)

Wrangling the typing PEPs

NYKevin — Thu, 16 Dec 2021 22:35:56 +0000

In general, type[T] is only supposed to match reified types (types that you can pass as the second argument to isinstance()), which neither of those examples is. The intention is that it represents a runtime type object - but at runtime, list[int] is mostly useless because it does not actually check its constructor arguments (i.e. list[int](['x', 'y', 'z']) is perfectly happy to give you a list of strings), so you might as well just use list instead, or so the thinking went when they were designing this feature.

However, some people want to use parameterized types as a convenient way of representing the schema of some structured data which you e.g. want to unserialize. This is not what type[T] was designed to do, but that does not make it an invalid use case for type[T]. Unfortunately, allowing such parameterized types to inhabit type[T] now would create backcompat issues, because mypy would need to flag this function as possibly unsafe:

T = TypeVar('T', bound=SomeGenericClass)

def foo(a: object, b: type[T]) -> TypeGuard[T]:
    if not isinstance(a, b):
        return False
    # Do additional checks here...

(Specifically, if b is a parameterized type object, this throws a TypeError at runtime - but mypy considers it OK because type[T] is not allowed to match such types.)

Wrangling the typing PEPs

NYKevin — Thu, 16 Dec 2021 21:44:10 +0000

This is because Python doesn't have a standard "into" method. Instead, the canonical way to convert type X into type Y is to call Y's constructor on an instance of X (just like in C++, except that Python never implicitly calls a constructor when the types don't match - so it's like C++ if all single-argument constructors were explicit). But in Python, the values which are acceptable to Y's constructor are annotated on Y's constructor (or more commonly, on its __init__ method, which is technically not a constructor, but whatever), not on X, so there's simply no obvious way to statically annotate "anything that Y can accept as a constructor argument." Perhaps there should be - but ideally, we would generalize this to support all functions, not just __init__, so that you could instead write "anything that is acceptable to the foo function as an argument," with Y.__init__ being one possible value of "the foo function."

If you are writing Y yourself, you can fix this by using whatever annotation you used on Y's constructor, and you can even set up a convenient TypeAlias for it if desired.

Wrangling the typing PEPs

pj — Thu, 16 Dec 2021 21:11:18 +0000

> I've seen libraries that enforce filenames to be "str" with a check, just to then call the python open function which would have gladly accepted pathlib.Path and bytes as well.

I feel like the correct way to do this would be to explicitly infer the type. Which meand we need some way to refer to 'the type that open()' takes as its first argument, but that's somewhat complicated to obtain:

```
import inspect

def mywrapper(filename: list(inspect.signature(open).parameters.values())[0].annotation):
...
```

...doesn't exactly flow from the fingertips. And I'm not sure if the if the value to the annotation is valid as an annotation itself. *sigh* IWBNI there was like an `infer(open, 1)`.

Wrangling the typing PEPs

atnot — Thu, 16 Dec 2021 18:20:16 +0000

It would be really nice if python could figure out something comparable to e.g. Rust's Into/From there.

Basically the way it works is that you can declare a parameter to be say, Into<MyType>. This means you may pass any value on which .into::<MyType>() can be called to convert it. From<MyType> being the reverse.

Python kind of already has this by convention for a few types like __bytes__, __bool__, __iter__, __str__ (which is also overloaded for other uses), __int__, __float__, etc. but it's not really widely used or extensibile.

Wrangling the typing PEPs

anselm — Thu, 16 Dec 2021 17:50:53 +0000

Does Python have the possibility to specify a Procotol as "type" instead of nailing down the type to some specific ones?

Yes, this is explained in PEP-544, which deals with “duck typing”.

There's List, which means object with list procotol, as far as I understand it. But is there something similar for paths?

The os module implements PathLike as an “abstract base class” for classes that implement path-like behaviour (such as pathlib.Path). Basically this means that such classes support the __fspath__() method, which is supposed to return a representation of the object that is a str suitable to use as a file system path. If you wanted to say that a function accepts a parameter that is either a str to begin with, or a PathLike object, you could define it as

from os import fspath, PathLike

def do_something_with_path(path: str | PathLike) -> …:
    …
    f = open(fspath(path))
    …

(where os.fspath(s) will return s outright if s is of type str, or else s.__fspath__()). Of course you could also define a type variable like

PathOrStr = TypeVar('PathOrStr', str, PathLike)

and then simply use this as

def do_something_with_path(path: PathOrStr) -> ...:
    …

Can the developer create new such types for his own protocols?

Absolutely. See here.

Wrangling the typing PEPs

smurf — Thu, 16 Dec 2021 16:20:36 +0000

> This is a misuse of the type system.

It is just plain stupid. Why do some people write useless code like that?

Wrangling the typing PEPs

gdiscry — Thu, 16 Dec 2021 15:48:02 +0000

A correct annotation would use something like Union[str, bytes, os.PathLike[str], os.PathLike[bytes]] or Union[AnyStr, os.PathLike[AnyStr]].

Like mentioned in the article, the typeshed project is a good resource if one needs some inspiration, in this case os.fspath() or similar and the type aliases defined by typeshed could be useful.

Wrangling the typing PEPs

mb — Thu, 16 Dec 2021 15:13:42 +0000

How do I spell out a correct type annotation for a filesystem path?

Does Python have the possibility to specify a Procotol as "type" instead of nailing down the type to some specific ones? Similar to Traits in Rust. There's List, which means object with list procotol, as far as I understand it. But is there something similar for paths? Can the developer create new such types for his own protocols?

Wrangling the typing PEPs

ballombe — Thu, 16 Dec 2021 14:12:45 +0000

> I've seen libraries that enforce filenames to be "str" with a check, just to then call the python open function which would have gladly accepted pathlib.Path and bytes as well

This is a misuse of the type system.

Wrangling the typing PEPs

LtWorf — Thu, 16 Dec 2021 09:58:26 +0000

Maybe interesting to the readers but a function like

def f(i: Type[T]) -> T: ...

that basically is a factory, returning an instance of the passed type, will not work in all cases with mypy.

A List[int] will work fine, but a Tuple[int] won't work. In general from that definition mypy will understand "Any" or error out.

https://github.com/python/mypy/issues/9003

Wrangling the typing PEPs

LtWorf — Thu, 16 Dec 2021 09:50:16 +0000

Well people knew that PEP 563 was going to break my library (I had complained), but didn't seem to care until they got complaints from someone making a very similar library (obviously a worse one, but with a million more users) complained.

So I'm a bit surprised now about the interest for those not using annotation in the approved way.

Anyway for library writers… yes I prefer if libraries have (meaningful) annotations… but I miss a bit the duck typing days.

I've seen libraries that enforce filenames to be "str" with a check, just to then call the python open function which would have gladly accepted pathlib.Path and bytes as well.

So hints are nice, but unnecessarily restricting to one type just means the users have to do unneeded type conversions just to make the linters (or the runtime checks) happy.

Wrangling the typing PEPs

taladar — Thu, 16 Dec 2021 09:16:23 +0000

It seems to me that dynamically and/or weakly typed languages like Python are struggling a bit in adding optional typing in general.

There are probably a few reasons for this, some technical and some social.

On the one hand it is just hard to add a type system later to an API designed without one since a lot of the APIs in dynamic languages are a bit 'sloppy' from a typed perspective, often accepting different types for the same parameter and/or returning different types (and null/nil like values). This makes it hard to make a high quality typed API the way it would look like in a language that had static types from the start.

Obviously having types on just some functions and not others does not exactly help either.

The fact that optional typing annotations are not enforced in all situations in some of the optional typing languages also means defensive programming code and unit tests meant to check the same thing as the type system would in a statically typed language can not be removed and add a dual maintenance burden.

The social issue I see is that a lot of the people who really know how static typing works and who care about a good static type system have moved on to languages that suit their tastes better, leaving a high percentage of people who do not like static type systems to please for the dynamically typed language community to deal with.

At the same time the languages lose relevance as it becomes more and more clear that the whole 'sufficiently disciplined programmer' does not exist, especially not whole teams of them or even a whole language ecosystem or at least all the people working on your project and all of your project's dependencies.

Replacing type checks with unit tests also means a lot more work for inferior results (e.g. how do you test that no combination of parameters ever yields a different return type).

This means the existing code bases have a higher maintenance burden but most likely fewer maintainers highly motivated to adjust the API to be a great typed API.

Greenspun's tenth rule

ejr — Thu, 16 Dec 2021 01:29:46 +0000

I would be interested in a layout of benefits and drawbacks when compared to other optional type annotation systems like Common Lisp. Also with heavily-inferenced typing systems.