Development [LWN.net]

Python looks at paths

By Jake Edge
April 13, 2016

The Python core developers cannot be accused of insufficient discussion when considering changes. A recent topic, which was something of a spin-off from the to_file() discussion we covered last week, spanned multiple threads in two different mailing lists—with several hundred posts in total. The question at hand concerned filesystem paths and the pathlib module that was added to the standard library for Python 3.4 on a provisional basis. Pathlib provides many useful features, but is not integrated with the rest of the standard library, making it difficult to use—changing that in one way or another is currently under consideration.

Pathlib is an object-oriented interface to filesystem paths that provides semantics that are appropriate to the underlying operating system (OS) where the program is running. PEP 428 described the motivations and goals for including the third-party pathlib module into the standard library; it was approved in November 2013.

The idea is to provide OS-independent ways to manipulate paths. For example, the / operator can be used to build up paths:

    from pathlib import Path

    p = Path('foo')
    q = p / 'bar' / 'baz'

On Unix systems, that will create a path object that resolves to foo/bar/baz, while on Windows it will be foo\bar\baz.

But there is a problem. Path objects are not strings and cannot be used where strings are expected. Furthermore, since pathlib was adopted provisionally, the rest of the language has not been changed to support it. Thus, path objects cannot be passed to built-ins like open() and friends, nor to standard library routines that might be expected to support them.

p-strings?

The discussion of adding a to_file() method for a quick and easy way to write a file from a string noted the presence of the write_text() method for path objects, but the need to import pathlib made it less "quick and easy" than some had hoped. Initially, that led Koos Zevenhoven to propose the addition of "p-strings", which would be similar in some ways to the f-strings for formatted output that are described in PEP 498 (and will appear in Python 3.6, which is scheduled for the end of 2016). P-strings would create path objects from strings:

    p"targetfile.txt".write_text(string_to_write)

That code would create a path object and write a string to the file it represents, much as the proposed to_file() interface would have done.

But, in that post to the python-ideas mailing list, Zevenhoven acknowledged that making pathlib.Path a subclass of the built-in str (string) class would probably solve most of the compatibility issues between strings and paths that were being discussed. If functions like open() could just treat a passed-in path object as a string, there would be no need for programmers using pathlib to manually call str(path) before passing a path to those routines. Several were in favor of subclassing str for paths, noting that other third-party path libraries had done so over the years—seemingly with no ill effects. Michel Desmoulin asked: "Can somebody defend, with practical examples, the imperative to refuse to inherit from str ?"

Brett Cannon took up that request by describing the reasons he thinks it does not make sense for path objects to be a subclass of strings (also as a blog post). He agreed that it would solve some short to medium term compatibility problems, which might indicate that its "practicality beats purity" (as The Zen of Python spells out). But there is another entry in that document that people may be forgetting, he said, "explicit is better than implicit".

Part of the problem that he sees is that while all paths are strings, not all strings are paths. That goes beyond strings that result in "file not found" errors; there are strings that are illegal for paths at all. He likened the choice to the compatibility break made for Unicode in Python 3:

While I'm sure some will say I'm overblowing the comparison, but Python 3 came about because the implicit compatibility of binary and textual data in Python 2 caused major headaches to the point that we made a backwards-incompatible change that has caused widespread ramifications (but which we seem to be coming out on the other side of). By not inheriting from str, pathlib.Path has avoided a similar potential issue from the get-go. Or another way of looking at it is to ask why doesn't dict inherit from str so that it's easier to make it easier to stick in the body of an HTTP response so it can be implicitly treated as JSON? If you [are] going, "eww, no because they are different types" then you at least understand the argument I'm trying to make (if you're going "I like that idea", then I think Perl might be more to your liking than Python and that's fine since the two languages just have different designs).

For Cannon, the "explicit" principle overrides the "practicality" argument, though others see it differently. He did suggest that getting support for pathlib objects into various places in Python would help alleviate the current push for "path strings"; he has opened a bug for importlib with the aim of doing just that.

The subclassing and p-string ideas were discussed at length, but there was a fairly clear divide among the participants. Guido van Rossum noticed that and wondered if it came from different kinds of use cases for the language:

These seems to be a growing difference between people who use Python for "serious" programming and those who consider themselves testers, QA integrators and the like (more power to them!). Maybe both groups can occasionally try to walk in the others' shoes.

The future of pathlib

Several other ideas were batted around in python-ideas before most of the conversation moved over to the python-dev mailing list. Cannon moved the discussion there with a question: "When should pathlib stop being provisional?" Van Rossum replied that it should either be removed or made non-provisional by the time 3.6 is released. Given the discussion, though, keeping it means enhancing pathlib in some fashion—either by making Python and the standard library more pathlib-friendly or by making pathlib objects inherit from the string class. Van Rossum made it clear that he is not unwilling to determine that pathlib is a failed experiment and remove it from the standard library for 3.6 and beyond.

But there is a mechanism that can be used by libraries and built-in functions to ease the pain of supporting both string-based and path-based arguments. In versions 3.4.5 and 3.5.2 (both upcoming maintenance releases), pathlib objects will have a new path attribute that provides a string representation of the path. That allows code like the following to transparently handle both path and string arguments:

    arg = getattr(arg, 'path', arg)

That code will extract a path attribute from the argument or simply return the argument if it is missing that attribute (e.g. a string). While it may be a little ugly looking, code like that sprinkled around in the right places would go a long way toward making pathlib objects "just work".

Nick Coghlan proposed broadening that idea some:

Option 4: define a rich-object-to-text path serialisation convention, as paths are not conceptually the same as arbitrary strings, and we can define a new protocol accepted by builtins and standard library modules, while third parties can't

His wording was perhaps a bit over the top—Van Rossum jokingly called it "a classic 'serious programming' solution (objects, abstractions, serialization, all big important words)"—but the idea behind it was sound. Coghlan was suggesting adding a new protocol so that objects could define their own path-as-string representation using a "private" attribute (which, for Python, means it begins and ends with double underscores). He suggested __path__, which would also be called "dunder path" in Python circles. In another message, he acknowledged that bikeshedding was possible (or, perhaps, likely) and that __fspath__ might make a better name.

Coghlan pointed out that there weren't that many places that needed to support this new protocol and that those changes didn't necessarily need to come all at once. It turned out to be a fairly popular solution to the quandary—modulo some amount of wrangling over details. Ethan Furman stepped up to help implement the changes, as did Cannon.

Those details include bikeshedding on the name, as predicted, though __fspath__ or __fspathname__ seem to be the clear leaders at this point. There was a question of whether the new member should be a method or an attribute (i.e. a function or a value), which seems to have resolved in favor of a method. The idea of a helper function to replace the getattr() mechanism described above also came up. It would return the value produced by anything that implements the __fspath__() protocol or simply the argument itself if it doesn't, which would allow libraries and other packages to support those kinds of objects relatively easily. The idea was popular; the only real question was where it should live (as a built-in, in pathlib, or elsewhere). The os module seems the most popular choice at this point.

Consensus

Throughout the discussion (also spanning multiple threads), summaries of the rough consensus were posted by Furman and Cannon. The most recent at the time of this writing were posted within minutes of each other on April 11 and show remarkable progress with only a few open issues. Van Rossum has put Cannon and Chris Angelico in charge of drafting a proposal, Cannon reported. It will probably be done as an addendum to PEP 428. That addendum will also clearly spell out the reasons that pathlib objects do not inherit from the string class, which was a complaint about the PEP as it stands.

Both Cannon and Furman agree that the remaining sticking points have to do with whether __fspath__() and os.fspath() can return byte arrays (i.e. the Python bytes type) as well as strings. For working with filesystems that support more than just names in the default encoding, it may be difficult to avoid handling byte arrays, sometimes without the language knowing the encoding (which is part of why bytes exists). For now, it appears that allowing them to return bytes is favored.

While it was a massive and scattered discussion, it proceeded rather smoothly. Van Rossum noted that there was some hostility toward pathlib that concerned him, but none of that leaked into the tone of the discussion. Opponents and advocates alike participated without rancor and tried to find a solution that would best solve the problems. Being able to peer into this kind of "design in the open" is one of the benefits of open-source projects. Meanwhile, a semi-unloved piece of the language has gotten the attention it needs—pathlib seems likely to stay for Python 3.6 and beyond.

Comments (10 posted)

Brief items

Quote of the week

As someone who's worked at multiple proprietary software companies, having a roadmap doesn't magically make code happen.

— Josh Berkus

Comments (none posted)

WordPress 4.5 released

Version 4.5 of the WordPress blogging platform has been released. Notable changes include a new, less obtrusive pop-up tool for inserting hyperlinks while editing a post, live previews for mobile and tablet layouts, and a faster image resizer.

Comments (none posted)

LXD 2.0 is available

Version 2.0 of the LXD container hypervisor has been released. Based on LXC 2.0, this is "the first production-ready release of LXD", serving as "essentially a system wide daemon offering a REST API to interact with containers." The announcement notes that, among other features, LXD supports a variety of storage backends and live migration via Checkpoint/Restore in Userspace (CRIU).

Full Story (comments: none)

Newsletters and articles

Development newsletters from the past week

Comments (none posted)

Hutterer: Why libinput doesn't have a lot of config options

Peter Hutterer writes about the cost of configuration options. "You see, whenever you write 'it's just 5 lines of code to make this an option', what I think is 'once the patch is reviewed and applied, I'll spend two days to write test cases and documentation. I'll need to handle any bug reports related to this, and I'm expected to make sure this option works indefinitely. Any addition of another feature may conflict with this option, so I need to make sure the right combination is possible and test cases are written.' So your work ends after writing a 5 line patch, my work as maintainer merely starts."

Comments (15 posted)

Page editor: Nathan Willis
Next page: Announcements>>