Adding Requests to the standard library

The Python Requests module is a popular alternative to the standard library's HTTP handling (e.g. urllib2). Kenneth Reitz, who is the "benevolent dictator for life" (BDFL) for Requests, came to the Python Language Summit to discuss the possibility of adding the module to the standard library. It is oft-requested, but is a bit controversial, he said. In addition, he wanted to use his slot as a forum to discuss the criteria for adding things to the standard library.

Requests has security as its top priority. It focuses on using industry best practices for things like SSL/TLS, connection pooling, encodings, headers, and so on. It removes a significant amount of the complexity for writing programs that interact with the web.

It is also the most popular package on the Python Package Index (PyPI, though Reitz used its other name: The CheeseShop), having been downloaded some 42 million times. That is more downloads than either setuptools or virtualenv. That popularity leads to frequent suggestions that Requests be added to the standard library.

The development of Requests is mostly done by Reitz and two core contributors. It has been feature-frozen for the last two years. It has a stable API and a stable development community, he said.

There are two major dependencies for Requests that would also have to be added if Requests moved into the standard library. One is chardet, which is an encoding detector that is based on Mozilla's character-recognition algorithm. It is quite useful when dealing with Unicode because servers can't always be trusted to specify the encoding correctly. It would be a good addition to the standard library regardless of what happens with Requests, he said. The other dependency is urllib3, which provides thread-safe connection pooling, file posting, and more. It is under active development and receives updates frequently.

The Requests project sees the module as "critical infrastructure for the Python community", Reitz said. For example, it is used by the pip Python package installation tool.

There are a number of arguments for inclusion into the standard library. If the Python community wants to provide libraries that embody best practices, then adding Requests would be "the right thing to do". From a sustainability standpoint, having Requests in the standard library would make it easier to get funding for the core contributors. Also, chardet would be good from a "batteries included" standpoint.

Reitz also relayed some observations that might make it difficult for Requests to be included. To start with, it comes with its own "carefully curated" (and frequently updated) bundle of certificates from certificate authorities (CAs) for SSL/TLS verification, while Python relies on the system CA bundle. Relying on the system certificate bundle would likely reduce the security of the library, he said.

Beyond that, the HTTP specifications and recommended usage change significantly over time; the Requests module keeps up with those changes. There are also situations where the project has turned around a release for a security fix in 12 hours, which would be difficult or impossible to do if it were included in the standard library.

The "biggest pitch" for Requests is that it is better than the parts of the standard library that cover the same ground. Pulling Requests into the standard library properly would require a bunch of work to integrate it and replace pieces of the standard library with parts of Requests. The project would also lose the ability for fast turnaround changes based on security problems or specification/usage changes.

He also wondered about the goals of the standard library. Now that Python ships with pip, is inclusion in the standard library really needed for modules like Requests? The "official stance" of the Requests project is that it is critical infrastructure, as he said earlier, but also that it is too critical to be included in the standard library.

That conclusion seemed largely agreeable to those present. One attendee said they would not be happy if they had to update Python to get a new Requests. Nick Coghlan noted that network security needs make the update cycle for Requests quite different from that of Python as a whole. It is also impossible to maintain network security for a release that has a "no new features" policy (e.g. Python 2.7).

There needs to be a way for new people to get the recommendations to use "external" modules like Requests, Łukasz Langa said. But there is a problem for some users (and companies), though, because they don't want to install additional, third-party dependencies, Thomas Wouters said. Requests is in a somewhat different category, though, since it is already installed for pip, Coghlan said.

Someone suggested that there be a set of modules that are vetted and endorsed by the core developers or the Python Software Foundation. Langa wondered what would happen if the maintenance stopped for an endorsed package, however. He suggested that the documentation get changed so that there is a section on deprecated modules and their suggested replacements.

There were some other concerns expressed. Glyph Lefkowitz argued that one of the big hurdles for users adopting Requests and other PyPI modules is the command line. Adding some sort of user interface to pip would help with that, he said.

Alex Gaynor noted that Python 3.4 added the asyncio module, but that Requests does not support it. He wondered how Requests could even be considered for inclusion in the standard library without asynchronous support using asyncio. Brett Cannon pointed out that there is a need for an informational PEP that describes the goals of the standard library.

Larry Hastings kind of summed up the session when he said that "batteries included may not make so much sense for everything anymore". Certainly the feeling in the room seemed to indicate that there is a mismatch between the frequency of Python releases and the needs of some modules, which means their users are better served by remaining separate.