|
|
Subscribe / Log in / New account

Opposition to Python type hints

By Jake Edge
May 6, 2015

Type hints for Python seem to be a feature that is on the fast track. These optional hints provide information that would allow static type-checking. But a recent discussion on the python-dev mailing list shows that not everyone is on-board with the idea of adding the annotations inline—they would rather see the hints be "hidden" in separate files. Others seem opposed to the idea of type hints at all, fearing that the feature will change the character of the language entirely.

We first looked at the type hints back in December, though the discussion started in August 2014. The feature has been championed by Python's benevolent dictator for life (BDFL), Guido van Rossum, who also described the feature in a Python Language Summit session (and PyCon keynote) in April. It is targeted for the Python 3.5 release that is slated for September. The idea of static typing for Python goes back much further, however, at least as far as a December 2004 blog post from Van Rossum.

Self-described "mediocre programmer" Harry Percival posted his reaction to the type hints. He argued that type hints are "ugly" and will make the language harder to understand, especially for beginners. He didn't buy the argument that the type annotations for functions will truly be optional, suggesting that over time there will be a strong uptake of the feature. He didn't suggest doing away with type hints entirely, just that there be a recommendation to put those annotations into stub files.

The idea of stub files is already part of the type hints proposal, largely to support C extensions where there may be no Python code to annotate. There are other situations where they will be needed, including for modules that must support both Python 2 and 3. The syntax used by the annotations is only available in Python 3. Stub files have the same syntax as regular Python files—just containing function signatures with "pass" for the function body—but use a .pyi extension

There are some problems with always using stub files, however. For one thing, the way things are currently implemented, the stub files for a module would only be consulted for code that uses that module. The module itself would not have the benefit of the annotations, so its code could not be type-checked. Also, the annotations from stubs are not accessible for introspection from within Python code as inline annotations are.

Percival's post garnered a fair amount of support. To many, type hints are not particularly "Pythonic" and they will make Python code harder to read. By far the strongest condemnation came from Jack Diederich (who is evidently still smarting from a suggestion Van Rossum made in 2003). Some of his complaints are based on misunderstandings of the plans moving forward, but those misconceptions are shared by a fair number of others in what turned into a monster thread. But the core of his complaint can be found at the end of the message:

If this wasn't in a PEP and it wasn't going to ship in the stdlib very few people would use it. If you told everyone they had to install a different python implementation they wouldn't. This is much worse than that - it is Python4 hidden away inside a PEP.

There are many fine languages that have sophisticated type systems. And many bondage & discipline languages that make you type things three times to make really really sure you meant to type that. If you find those other languages appealing I invite you to go use them instead.

Adding type declarations to a dynamically typed language seems to be the crux of the problem for Diederich and others. It fundamentally changes what Python is. But Gregory P. Smith noted that the PEP is really targeted at creating a common way for projects using types (and type-checking) for Python to annotate code, not at all for a wholesale change in existing code:

One thing I feel is often overlooked in the discussion on this PEP: It is about creating a unified type expression syntax for everyone working on Python typing to centralize around. Regardless of if the PEPs version falls short for some purposes. It allows for sharing work. There are multiple ongoing projects that are trying to make use of type information with Python, this allows them to all speak the same language. (MyPy, MicroPython, Cython, the static analyzer we are trying to create at Google, several others not on the top of my head I'm sure, etc.)

We will not be putting type annotations anywhere in the stdlib or expecting anyone else to maintain them there. That would never happen until tools that are convincing enough in their utility for developers to _want_ to use are available and accepted. That'll be a developer workflow thing we could address with a later PEP. IF it happens at all.

A more practical complaint about the annotations came from Requests developer Cory Benfield, who tried to annotate some parts of that library. For certain parameters, he got incredibly complicated annotations that still didn't fully restrict the parameter types. He summarized the problems, as follows:

I suppose the TLDR of this email is that I think that libraries with 'pythonic' APIs are the least likely to take up this typing system because it will provide the least value to them. Maintaining signatures will be a pain (stub files are necessary), writing clear signatures will be a pain (see above), and writing signatures that are both sufficiently open to be back-compatible and sufficiently restrictive to be able to catch bugs seems like it might be very nearly impossible.

After Chris Angelico suggested that Benfield might "want to just stop caring about the exact type", Benfield pointed to exactly that as part of the problem:

This change doesn't catch any of the bugs anyone was going to hit (no-one is passing a callable to this parameter), and it doesn't catch any of the bugs someone might actually hit (getting the tuple elements in the wrong order, for example). At this point the type signature is worse than useless.

Another complaint, voiced by Arnaud Delobelle, is that type hints are a threat to duck typing. The concern is that if the tools start complaining about interfaces that use duck typing, it will cause programmers to shy away from that longstanding Python type convention. That "could encourage a very significant cultural change in the way Python code is written towards a less flexible mindset". But Van Rossum has not forgotten about duck typing:

For me, PEP 484 is a stepping stone. Among the authors of PEP 484 there was much discussion about duck typing, and mypy even has some limited support for duck typing (I think you can still find it by searching the mypy code for "protocol"). But we ran out of time getting all the details written up and agreed upon, so we decided to punt -- for now. But duck typing still needs to have a way to talk about things like "seek method with this type signature" (something like `def seek(self, offset: int, whence: int=SEEK_SET) -> int`) so the current proposal gets us part of the way there.

The hope is that once 3.5 is out (with PEP 484's typing.py included *provisional* mode) we can start working on the duck typing specification. The alternative would have been to wait until 3.6, but we didn't think that there would be much of an advantage to postponing the more basic type hinting syntax (it would be like refusing to include "import" until you've sorted out packages). During the run of 3.5 we'll hopefully get feedback on where duck typing is most needed and how to specify it -- valuable input that would be much harder to obtain of *no* part of the type hints notation were standardized.

The roughly one and a half to two years that fall between each major Python release were clearly part of Van Rossum's thinking. He chose to implement something that could be done before 3.5 hit its feature freeze, but to keep the feature in a provisional state so that it could still be changed during the 3.6 development cycle. Several things may get addressed during that cycle, including support for duck typing and, possibly, adding new syntax for variable type declarations (instead of the comment-based annotations that are part of PEP 484). That all assumes that the PEP is accepted for 3.5, which seems likely unless Van Rossum has a change of heart.

The concerns about the standard library getting annotations are largely based on misunderstandings—there are no plans afoot to add them en masse either inline or as stubs. In fact, changes of that sort to "modernize" the standard library are frowned upon, Van Rossum said, because they often cause subtle regressions "in dark corners". Adding stub files for the standard library in 3.6 is a possibility, PEP co-author Łukasz Langa said. Additions to the standard library might include inline annotations, if the author is a fan of type hints, but that is going to be a ways out, according to Van Rossum.

The standard library is often seen as a place to look at "good" Python code, but that isn't really the case. It is a shared community code base, as Nick Coghlan described it but, as Van Rossum pointed out, it is not the epitome of Python coding style:

The stdlib contains a lot of Python code, and you can learn a lot from it, but good coding habits aren't generally something you learn there -- the code is crusty (some of the oldest Python code in existence lives in the stdlib!), often has to bend over backwards to support backward compatibility, and is riddled with performance hacks.

The debate got a bit testy at times but, as is generally the case in the Python world, it never got really out of hand. That community has found a way to moderate its communication, generally by way of fairly gentle reminders about respecting others' views. In the end, any change of this magnitude is going to cause some to be worried about the impacts—as they should. Contrary to some of the less temperate reactions (evidently Twitter was especially bad), this hardly seems like a change that "breaks" the language. It would not be hugely surprising to see it as quite a popular feature in, say, five years, once all the kinks have been worked out.



to post comments

Opposition to Python type hints

Posted May 7, 2015 3:19 UTC (Thu) by josh (subscriber, #17465) [Link] (1 responses)

Personally, I really like Python as an expressive scripting language, and the single biggest thing that makes me want to use it less and other languages more is the lack of static typing and static error handling.

"dynamic" is not a feature; in every possible way it's a bug. That doesn't mean you can't have a type like "undefined", saying "I don't know what this is yet, let me compile my code with stubs and just raise an exception if I call them". But I'm tired of finding errors at runtime that many other languages would call out at compile time.

Real-world example from the other day: refactoring common code into a method, attempting to change all occurrences of a particular variable name into a more general name, and missing one in the middle of a large expression. Most languages would catch that at compile time; Python didn't catch it until the method was called. When your project's overall compile-debug cycle is a few minutes, problems like that add up.

Opposition to Python type hints

Posted May 7, 2015 17:08 UTC (Thu) by HelloWorld (guest, #56129) [Link]

> "dynamic" is not a feature; in every possible way it's a bug.
Amen to that.

Opposition to Python type hints

Posted May 7, 2015 10:12 UTC (Thu) by jezuch (subscriber, #52988) [Link]

> writing signatures that are both sufficiently open to be back-compatible and sufficiently restrictive to be able to catch bugs seems like it might be very nearly impossible

This is a general problem with adding things to an existing language that might not be exactly in line with its existing "ethos". When generics were introduced to Java, the standard library was retro-fitted to include type parameters on such essential types as the java.util.Map interface. But while they made the put() method to be put(K key, V value), the remove() method remained remove(Object key). The thinking was, AFAICT, that you should be able to remove anything from a collection - if it's a wrong type, it just won't be there. And also, backwards compatibility with some (IMO) dumb code that mixes types of keys. So now we're stuck with this... thing that in the best case is really annoying and in the worst case is a major source of bugs. Yes, tools like findbugs will point out where incompatible types are passed to remove(), but I shouldn't need a special static analyzer tool to avoid stupid bugs like that...

/rant :)

Opposition to Python type hints

Posted May 7, 2015 10:58 UTC (Thu) by rsidd (subscriber, #2582) [Link]

I already use python with optional typing. It's called cython and it works very well right now. Put the typed stuff in a module and import it into python -- it "just works", it catches type errors, and the speedup can be significant. A few years ago Guido seemed unconvinced but said it seems perfect for the scientific world, and success there would perhaps persuade him. The proposal was to replace selected stdlib functions with cython versions, starting perhaps with the enthought version or other scientific distros. Success there, he said, may help persuade him, but I guess that didn't happen. I wish, at least, there had been some convergence in the proposed typing syntax.

Opposition to Python type hints

Posted May 7, 2015 12:17 UTC (Thu) by ballombe (subscriber, #9523) [Link] (3 responses)

The paradox is that statically typed languages like Haskell does not require you to add type annotation because type inference will be good enough in most case.

Opposition to Python type hints

Posted May 7, 2015 15:08 UTC (Thu) by kjp (guest, #39639) [Link] (2 responses)

As a middle ground, Rust only forces you to declare types of function parameters.

I do agree with some of the push back on the python side, it does change the feel of the language, and perhaps people should be served moving elsewhere and giving up on using python for large codebases.

Opposition to Python type hints

Posted May 8, 2015 20:36 UTC (Fri) by peter-b (guest, #66996) [Link] (1 responses)

> As a middle ground, Rust only forces you to declare types of function parameters.

That's not strictly true; Rust forces you to declare types wherever it can't infer them. Of course, *in practice* this usually means that you only need to declare the types of function parameters. ;-)

Opposition to Python type hints

Posted May 14, 2015 10:26 UTC (Thu) by ssokolow (guest, #94568) [Link]

Or when it *won't* infer them by design.

Closures infer type signatures but functions don't because an explicit decision was made that functions should be fundamental units of API definition with no implicit dependencies to complicate reasoning about or parsing the code.

Opposition to Python type hints

Posted May 7, 2015 13:28 UTC (Thu) by ibukanov (subscriber, #3942) [Link] (8 responses)

My experience with scripting languages is that the most problematic feature is not lack of types but rather lack of declarations for names to check them before running the code. I made much less type errors than simple typos in names. It is so annoying to wait for results only to discover that I mistyped the name of variable to put the results of a long-running code.

Opposition to Python type hints

Posted May 7, 2015 14:15 UTC (Thu) by niner (subscriber, #26151) [Link] (5 responses)

That's not a problem of scripting languages in general. Perl has had use strict; for ages. This has been copied to Javascript as "use strict;" (yes, with the quotes, to provide backwards compatability). Even PHP has optional warnings about undeclared variables.

Python is the user unfriendly hold out here.

Opposition to Python type hints

Posted May 7, 2015 16:08 UTC (Thu) by ibukanov (subscriber, #3942) [Link] (4 responses)

Both Perl and JavaScript even in the strict modes check names for existence only at runtime when the name is accessed or assigned. What I want is to check variable names and ideally property/methods for existence before the execution. At least modern Perl with -w warns about name used only once, but that is not helpful when after refactoring a wrong name is left in several places on rarely accessed branches.

Note that to check for property/method existence one does not need a full-fledged static typesystem. With JavaScript tools like jslint or similar that checks property/methods against a predefined list are enough to capture most of my typos.

Opposition to Python type hints

Posted May 7, 2015 16:51 UTC (Thu) by niner (subscriber, #26151) [Link] (2 responses)

Perl does check variable names at compile time in strict mode. The only exception is fully qualified package globals like $Foo::Bar::global_variable. But those are very rare.

Perl 6 throws errors for invalid method names if possible at compile time. I.e. when it knows about the types involved and no fallbacks are present. It's also very helpfully complaining about invalid arguments to methods and functions. And its type system allows very fine grained specification of types and signatures. E.g. it allows one to declare a function that takes a hash (dict in Python) as argument and requires that this hash contains certain keys. So in essence, it makes a combination of duck typing and static types possible and useful.

Opposition to Python type hints

Posted May 7, 2015 17:42 UTC (Thu) by ibukanov (subscriber, #3942) [Link] (1 responses)

Hm, with perl 5.18.4:

~/s> cat x.pl
use strict;

my $a = 10;

if ($a) {
print "OK\n";
} else {
print $b . $b . "\n";
}
~/s> perl -w x.pl
OK

No warnings or errors about undeclared $b. What did I miss?

Opposition to Python type hints

Posted May 7, 2015 17:49 UTC (Thu) by niner (subscriber, #26151) [Link]

That $a and $b are predefined package globals in Perl 5. They should be used for sort blocks:

my @sorted = sort { $a <=> $b } @unsorted;

Try your example with $x and $y and you'll get:

~> perl x.pl
Global symbol "$y" requires explicit package name at x.pl line 6.
Global symbol "$y" requires explicit package name at x.pl line 6.
Execution of x.pl aborted due to compilation errors.

Yes, that's surprising behavior and it's one of the reasons for Perl 6 being a breaking change. Of course, $a and $b are not exactly good variable names anyway :)

Opposition to Python type hints

Posted May 12, 2015 13:36 UTC (Tue) by ehiggs (subscriber, #90713) [Link]

Python also has linters available. If you use vim with the syntastic extension then it calls out to any of a number of linters:

https://github.com/scrooloose/syntastic/tree/master/synta...

Then your editor will check the file every time you ask it to or on na hook (e.g. when saving the file). Then it puts red and/or yellow `>>` symbols in the gutter pointing to issues like in this image:

http://dancallahan.info/journal/python-flake8/syntastic-f...

Opposition to Python type hints

Posted May 7, 2015 15:11 UTC (Thu) by kjp (guest, #39639) [Link]

I never type an existing variable or function name directly in python source; I always use ctrl-p/ctrl-n in vim to autocomplete it. I load the other file containing the original name in a new tab if I have to, just so autocomplete can match on it. That and pyflakes help a lot. It also helps to use _lengthy_ function and variable names that are unique. However, refactoring modules and functions in a large codebase is still agony.

Opposition to Python type hints

Posted May 10, 2015 12:38 UTC (Sun) by mgedmin (subscriber, #34497) [Link]

flake8 helps a lot. Especially when integrated with your text editor (vim + Syntastic for me).


Copyright © 2015, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds