Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries [LWN.net]

Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries

Posted Jul 20, 2022 17:16 UTC (Wed) by domdfcoding (guest, #159754) [Link] (1 responses)

Slightly strange to have the headings be the modules you're *supposed* to use. At a glance it looks like you shouldn't use pathlib or secrets, because *they're* obsolete.

Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries

Posted Jul 20, 2022 23:25 UTC (Wed) by tialaramex (subscriber, #21167) [Link]

Yes, I think this article format works better with the things you should stop doing as titles. The primary audience is people who are doing X and should be doing Y. They know what X is, because that's what they're doing, so highlighting that in a heading is worth doing, they've no idea they should be doing Y so that doesn't mean anything to them.

The format would however work fine for announcing new libraries. A brief blurb about why could be useful in addition to the name of the library, but Python mostly doesn't have Rust's habit (which is divisive but I like it) of naming libraries based on puns, pop culture references etc. So a Python library for uploading Misc XML to the Cloud might be called Misc-XML-Cloud-Upload and it's redundant to say "For Uploading Misc XML to the Cloud" whereas the equivalent Rust library might be named stratus-various and a blurb helps us figure out whether we care.

Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries

Posted Jul 20, 2022 18:03 UTC (Wed) by iabervon (subscriber, #722) [Link] (6 responses)

The thing that's really amazing about using pathlib is that, since the objects have functions that proxy os (and open()), you can use that API to provide a whole VFS for whatever you want. Ever wonder what would happen if your code was running on a system with a different directory separator than the system it's actually running on, but you could still open files and do various filesystem operations? What if your disk contents were something different from what's actually on disk? If you provide a different Path as the root, all of the subsequent code, including suitably pathlib-using libraries that take a Path object, will use code you control.

Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries

Posted Jul 20, 2022 18:45 UTC (Wed) by josh (subscriber, #17465) [Link] (3 responses)

Yeah, that's a *great* feature of Python. This is something we need in Rust: a way to represent and manipulate Path values from another OS.

Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries

Posted Jul 20, 2022 19:32 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

Issue lives here: https://github.com/rust-lang/rust/issues/60845 :)

Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries

Posted Jul 22, 2022 23:44 UTC (Fri) by jschrod (subscriber, #1646) [Link] (1 responses)

You mean, like in Common Lisp?

Hooray for an integration programming language that combines the abstractions of CL with the system integration of Perl. It would be my dream language.

Pathlib looks good to me

Posted Jul 24, 2022 5:58 UTC (Sun) by CChittleborough (subscriber, #60775) [Link]

Having just read the doco for pathlib (which I had never heard of until I read these comments), I am very impressed. ISTM that pathlib is just as good for today’s world (everything runs a POSIX-style OS or Windows) as Common Lisp’s Pathnames were for the 80s and 90s ... only, y’know, in OO Python.

Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries

Posted Jul 20, 2022 19:37 UTC (Wed) by developer122 (guest, #152928) [Link]

It's not hard to see how one could make unix paths (for example) the language standard and translate them to windows or what-have-you (even very esoteric stuff) before it actually hits the OS. A really powerful feature.

Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries

Posted Jul 21, 2022 1:13 UTC (Thu) by cjwatson (subscriber, #7322) [Link]

This isn't hypothetical either: for example, https://devopshq.github.io/artifactory/ does exactly that, and we were able to take advantage of it to write code that could deal with either local or remote paths relatively painlessly. (Well, aside from the Artifactory-specific pain, anyway!)

Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries

Posted Jul 20, 2022 21:29 UTC (Wed) by k8to (guest, #15413) [Link] (7 responses)

It's hilarious to me that this article declares that dirname(dirname(abspath(file)) is unreadable, when these are extremely well documented unix library behaviors, that many people will already know, but declares Path(file).resolve().parent.parent is readable, when resolve() is entirely novel to this library, and .parent.parent leaves one entirely in the dark when and if anything interrogates the filesystem. dirname is well known to be a string operation. ..parent, who knows? Maybe pathlib is documented enough that these questions can be answered, but the old patterns seems far more readable to me.

Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries

Posted Jul 21, 2022 0:15 UTC (Thu) by neggles (subscriber, #153254) [Link] (4 responses)

.resolve() is neither novel nor unique, it’s just syntactic sugar for realpath(3). You can accomplish the same thing with Path.cwd().joinpath(your_relative_path_object).abspath() - find the current working directory, resolve the target path relative to cwd(), then run realpath on that absolute path to resolve symlinks.

Pathlib is very well documented - and parent == dirname, same operation. It’s effectively a string replace, taking the full path and removing everything after and including the last forward slash. You can cast a Path to a string (or a string to a Path) whenever you like, too, so it’s not like you can’t do horrible string replacement shenanigans if you really must.

But Path objects (and chaining them with methods, rather than nesting a kshjillion functions) are a massive improvement over dealing with raw path strings, especially once symlinks and OS differences are involved - with Pathlib you just use forward slashes and forget about it. And it’s in the standard library! no dependencies! There’s no downside here!

it hurts me every time I see someone import sys and os just to deal with file paths, and all the gymnastics they go through to make things work properly… I’ve refactored many scripts and small modules to use Pathlib and in doing so usually replaced 2-6 lines of confusing code with a single line that’s easy to follow

Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries

Posted Jul 21, 2022 7:04 UTC (Thu) by egorovv (subscriber, #102560) [Link] (3 responses)

"The fact that paths are treated as objects rather than strings" - I just wonder if there is anyone out there thinking this way using shell. I - can't even start thinking of '/usr/local/bin' as some sort of object.
I can imagine though that for someone learning about paths first time from pathlib - this may look like a neat idea.

Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries

Posted Jul 21, 2022 9:34 UTC (Thu) by mbunkus (subscriber, #87248) [Link] (2 responses)

Judging by incredible amount of issues we've had with shells & spaces in names, file names starting with dashes being interpreted as options, having to escape special characters over and over again etc. it is very, very hard to find anything good about thinking of paths as strings. "Don't worry, everything's just a string" just doesn't work well with the need for _some_ kind of structure & semantics (this here is the first option to the program; that there refers to a file even it might look like the name of an option; yes, that overly long thing consisting of multiple spaces, ` and ( ) is just a single file name).

On top of that the vast majority of humans never uses shells. If they ever learn about file names & paths, they will most likely think of different directories & different files as separate entities. When does a regular (non IT person, non admin) Windows user ever come into contact with a full path such as C:\users\mbunkus\Documents\My Funny Valentine.txt? They use file dialogs, they use Windows explorer, and in both cases each element is presented as a distinct entity. They click on the "My Documents" icon. They click on the "My Funny Valentine.txt" document. I'd bet a lot of money that our notion of thinking of paths as strings is very, very foreign to them all.

Us shell users are the tiny minority.

Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries

Posted Jul 21, 2022 21:25 UTC (Thu) by martin.langhoff (guest, #61417) [Link] (1 responses)

The higher the abstraction, the bigger the TOCTOU fall...

Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries

Posted Jul 25, 2022 1:42 UTC (Mon) by NYKevin (subscriber, #129325) [Link]

pathlib neither increases nor decreases the TOCTTOU risk in most contexts. Specifically:

* The attributes and division operator are all computed by pure string manipulation, and count as neither a "check" nor a "use" within the context of TOCTTOU. This is equivalent to calling functions such as os.path.join().
* Nearly every part of the pathlib API is either a pure string manipulation, has a non-OO os.path equivalent, or both.
* pathlib does expose methods such as stat(), exists(), is_dir(), etc., but all of those methods have direct and straightforward os or os.path equivalents. pathlib does make them marginally easier to use, but that's just because the status quo ante was terrible (functions were randomly distributed between os and os.path with seemingly no rhyme or reason, so you had to memorize them or look it up).
* pathlib paths can be used in place of strings nearly everywhere. In particular, they are suitable arguments to open(), which can be called with mode='x' to get the equivalent of O_CREAT | O_EXCL if needed. This was already possible with strings, of course, pathlib just hasn't lost the ability to do it.

Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries

Posted Jul 24, 2022 14:42 UTC (Sun) by randomguy3 (subscriber, #71063) [Link] (1 responses)

I think the main thing that makes the latter more readable is chaining functions rather than nesting them.

The specific names are secondary - but bear in mind many (maybe even most) python users are not well-versed in POSIX shells or libraries. I'm not sure there's much to choose between "resolve()" and the various other names that functionality has gone under in libraries throughout history, but I personally welcome "parent" and "filename".

Even after 20 years of using Linux, I still use basename when I want dirname, because it does the opposite of what I intuitively expect from its name. Dirname is less bad, but is still ambiguous when you know you're dealing with a path to a directory. They're just bad names, and I'm glad python took the opportunity to revisit them.

(FWIW, pathlib helpfully uses methods for filesystem operations and attributes for pure string manipulation, which makes the distinction easy to spot by looking for the parentheses. It really is a lovely, well-thought-out and well-documented library.)

Heinz: It's Time to Say Goodbye to These Obsolete Python Libraries

Posted Jul 25, 2022 1:29 UTC (Mon) by NYKevin (subscriber, #129325) [Link]

> (FWIW, pathlib helpfully uses methods for filesystem operations and attributes for pure string manipulation, which makes the distinction easy to spot by looking for the parentheses. It really is a lovely, well-thought-out and well-documented library.)

This also comports nicely with standard Python idioms: Attributes are supposed to be cheap and free of side effects, whereas method calls can be costly or impure (for example, some systems may update atime on some filesystem operations, but string manipulation is definitely not going to do that).