Lazy imports for Python

Posted Sep 7, 2022 21:59 UTC (Wed) by NYKevin (subscriber, #129325)
In reply to: Lazy imports for Python by mathstuf
Parent article: Lazy imports for Python

IMHO this is a result of poor API design. It should really look like this:

import mount # Filesystem support
import mount_xfs # XFS support

mount.mount(type=mount_xfs)

Or maybe you use type=mount_xfs.mount_type or something like that, but you can just pass the whole module if you want to. Python doesn't care. They're all PyObject* under the hood.

Lazy imports for Python

Posted Sep 8, 2022 13:03 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (5 responses)

While true, I think it'd be hard to argue that such APIs don't exist today. Is Python effectively forcing[1] these APIs to be redesigned with this feature?

For the project I know that will be affected by this, it is all in the library ctor stuff that stuff gets set up; there's no API *to* call. Again, not the best design, but doing it explicitly would make "everyone" unhappy because of how much work it ends up doing for users.

[1] Via feature request pressure to "support lazy loading" over time.

Lazy imports for Python

Posted Sep 8, 2022 16:51 UTC (Thu) by NYKevin (subscriber, #129325) [Link] (4 responses)

No. You can just tell end users "sorry, we don't have plans to support that" and then dupe all subsequent requests against that one. Just because people send you feature requests, it doesn't mean you have to act on them.

Having said that, you might consider moving that logic into an init() function that the application can call into, if possible. Then, when someone inevitably asks for lazy loading support, tell them "Sure, we support enable lazy loading, just make sure you call this init() function." Since they're already making a code change anyway (to enable lazy loading), they should not have a backcompat objection to that.

Lazy imports for Python

Posted Sep 8, 2022 17:51 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (3 responses)

> Having said that, you might consider moving that logic into an init() function that the application can call into

There's no function for `init` to bind; the relevant code is all stored in global initializers (again, not the design I'd prefer, but it's what I have).

Lazy imports for Python

Posted Sep 8, 2022 20:24 UTC (Thu) by NYKevin (subscriber, #129325) [Link] (2 responses)

You could have init() find the relevant code and eagerly import it (your imports would not be at the top level, so they would be eager regardless of the application's preferences). But that might make a bad design worse, depending on how difficult it is to "find" the relevant code.

Lazy imports for Python

Posted Sep 9, 2022 8:15 UTC (Fri) by farnz (subscriber, #17727) [Link] (1 responses)

Or, on the assumption that you're willing to do work to support lazy loading, you move all of the global code bar imports into the init function, and then have the last line of your module be init(). This is a bigger refactor, but it means that old users get the behaviour they expect (import runs the code), and new users can do a lazy load followed by a call to init to get the same behaviour.

Lazy imports for Python

Posted Sep 10, 2022 16:57 UTC (Sat) by NYKevin (subscriber, #129325) [Link]

The concern is that, possibly, the imports are done by the application and not by the library itself. In other words, we might have something like this:

main.py:

import primary_library
import some_plugin
import another_plugin

Each of the plugin files imports primary_library and then calls some magic init function from there, but the actual API is primary_library (i.e. you just import some_plugin for the magic init side-effect and not to actually use it directly). The plugins are third-party code. You can't just fix it in primary_library, because primary_library doesn't know about the plugins and can't find and init them by itself, even if the application does call primary_library.init().

The reasonable solution is to require the application to call primary_library.init(some_plugin) and explicitly say which plugin(s) to init. But that might be a more significant refactoring job. OTOH, explicit is better than implicit, and this is IMHO a superior coding style to just "write import x and magic happens."

Lazy imports for Python

Posted Sep 9, 2022 19:00 UTC (Fri) by smurf (subscriber, #17840) [Link] (2 responses)

Your workaround is fine when using an object-oriented design. However you can't use it e.g. with command-line plugins. When I call "mount -t xfs" I expect the mount command to find the XFS plugin somehow, and if the registration happens solely by import that won't work.

The fix for this is to park each imported part in a separate module, then use the filename to load it.

Lazy imports for Python

Posted Sep 10, 2022 20:03 UTC (Sat) by NYKevin (subscriber, #129325) [Link] (1 responses)

Well, yeah. If you force everything to use a stringly-typed interface, life sucks. But this article is about Python, which is not stringly-typed. I object to calling this a "workaround" - it's the correct way to design the API when you have real first-class modules available in your type system.

Lazy imports for Python

Posted Sep 11, 2022 3:18 UTC (Sun) by smurf (subscriber, #17840) [Link]

Python itself is not stringly-typed, but the command line sure is.