LWN: Comments on "Lazy imports for Python"

Lazy imports for Python

smurf — Thu, 15 Sep 2022 08:54:19 +0000

The idea behind lazy imports is to not access the file system at all until the module is actually used. As soon as you go off and search + read a .pyc or .pyo file header, the advantage of not going through the whole file system dance is gone (did you ever look at how many file system accesses Python needs to do to locate a module?) and you might as well import the thing.

Lazy imports for Python

arvidma — Wed, 14 Sep 2022 06:39:25 +0000

All of that recursive loading and parsing happens already today, since loading is eager. Putting a tag in each pyc file to indicate if it can be lazy loaded shouldn't need to add much cost (each pyc is touched twice instead of once, but they would all be in the page cache and that pyc-update ought to be comparatively cheaper than the parsing and compiling of the py-files).

This way, you only pay the cost once and any subsequent runs can take the fast path.

Lazy imports for Python

xnox — Wed, 14 Sep 2022 00:02:17 +0000

In python, import foo not only adds foo module to global namespace and allows accessing variables, functions, and classes that foo declares but it also typically executes any arbitrary code inside __init__ of said module which can have global side effects. For example establish network connections, migrating settings, writing out files or even simply setting process global environment variable.

import setup
import daemon
daemon.start()

Can fail, if setup has a side effect of exporting environment variables which start of Daemon expected to be set.

Same issues as shell source command, which also is hard to make "lazy".

Lazy imports for Python

NYKevin — Sun, 11 Sep 2022 19:24:05 +0000

Also, if you're trying to *catch* the exception, then it will automatically be eager because it's not a top-level import anymore (at an absolute minimum, it's inside of a try block).

Lazy imports for Python

smurf — Sun, 11 Sep 2022 03:18:36 +0000

Python itself is not stringly-typed, but the command line sure is.

Lazy imports for Python

NYKevin — Sat, 10 Sep 2022 20:03:51 +0000

Well, yeah. If you force everything to use a stringly-typed interface, life sucks. But this article is about Python, which is not stringly-typed. I object to calling this a "workaround" - it's the correct way to design the API when you have real first-class modules available in your type system.

Lazy imports for Python

NYKevin — Sat, 10 Sep 2022 16:57:27 +0000

The concern is that, possibly, the imports are done by the application and not by the library itself. In other words, we might have something like this:

main.py:

import primary_library
import some_plugin
import another_plugin

Each of the plugin files imports primary_library and then calls some magic init function from there, but the actual API is primary_library (i.e. you just import some_plugin for the magic init side-effect and not to actually use it directly). The plugins are third-party code. You can't just fix it in primary_library, because primary_library doesn't know about the plugins and can't find and init them by itself, even if the application does call primary_library.init().

The reasonable solution is to require the application to call primary_library.init(some_plugin) and explicitly say which plugin(s) to init. But that might be a more significant refactoring job. OTOH, explicit is better than implicit, and this is IMHO a superior coding style to just "write import x and magic happens."

Lazy imports for Python

smurf — Fri, 09 Sep 2022 19:14:19 +0000

Forgot to add: I just traced two dbus command line tools I'm using around here. They have comparable startup times (that is, way too long), but the number of modules they load without actually needing them is negligible.

Lazy imports for Python

smurf — Fri, 09 Sep 2022 19:12:03 +0000

Mmh. I'd assume that half that time is spent waiting for the introspection XML, and much of the other half might well be loading etree and decoding the thing.

DBus is nice, but its metadata system could have used a somewhat simpler data model.

Lazy imports for Python

smurf — Fri, 09 Sep 2022 19:05:47 +0000

This is why you have tests.

The same kind of argument has been used 20 years ago when libc started to use lazy loading. The solution was a loader flag that loaded eagerly and reported when something didn't resolve. On Python, the equivalent solution might well be a dedicated import checker.

Fortunately, tools like that do exist already. :-P

Lazy imports for Python

smurf — Fri, 09 Sep 2022 19:00:16 +0000

Your workaround is fine when using an object-oriented design. However you can't use it e.g. with command-line plugins. When I call "mount -t xfs" I expect the mount command to find the XFS plugin somehow, and if the registration happens solely by import that won't work.

The fix for this is to park each imported part in a separate module, then use the filename to load it.

Lazy imports for Python

azumanga — Fri, 09 Sep 2022 13:36:18 +0000

Some libraries I work on check for required external binaries and libraries at import, and throw an exception if anything is broken. That can be caught, or ignored if you just want the program to stop.

Now that exception will be thrown in whatever function the library is first used in. That may well be surprising, and could cause issues I imagine.

Lazy imports for Python

georgm — Fri, 09 Sep 2022 10:28:09 +0000

# time firewall-cmd --state
running

real	0m1.675s
user	0m0.861s
sys	0m0.105s

for a simple DBUS call...

Hopefully this will benefit from such a change

Lazy imports for Python

farnz — Fri, 09 Sep 2022 08:15:05 +0000

Or, on the assumption that you're willing to do work to support lazy loading, you move all of the global code bar imports into the init function, and then have the last line of your module be init(). This is a bigger refactor, but it means that old users get the behaviour they expect (import runs the code), and new users can do a lazy load followed by a call to init to get the same behaviour.

Lazy imports for Python

NYKevin — Thu, 08 Sep 2022 20:30:33 +0000

This either doesn't work, or it's basically useless.

foo.py:

import bar

bar.py:

print("Side effect!")

main.py:

import foo

Now, if you make main.py lazily import foo.py, then the side effects of bar.py will stop happening, which main might have accidentally relied on. So you would need to treat import statements as "statements other than declarations" - which effectively means that almost any nontrivial module will be eagerly imported. Or, alternatively, you have to recursively trace through all of the modules in the entire dependency tree and check each one for this lazy import flag - which is still pretty expensive and doesn't really save you all that much (consider the disk seeks!).

Lazy imports for Python

NYKevin — Thu, 08 Sep 2022 20:24:21 +0000

You could have init() find the relevant code and eagerly import it (your imports would not be at the top level, so they would be eager regardless of the application's preferences). But that might make a bad design worse, depending on how difficult it is to "find" the relevant code.

Lazy imports for Python

mathstuf — Thu, 08 Sep 2022 17:51:53 +0000

> Having said that, you might consider moving that logic into an init() function that the application can call into

There's no function for `init` to bind; the relevant code is all stored in global initializers (again, not the design I'd prefer, but it's what I have).

Lazy imports for Python

NYKevin — Thu, 08 Sep 2022 16:51:34 +0000

No. You can just tell end users "sorry, we don't have plans to support that" and then dupe all subsequent requests against that one. Just because people send you feature requests, it doesn't mean you have to act on them.

Having said that, you might consider moving that logic into an init() function that the application can call into, if possible. Then, when someone inevitably asks for lazy loading support, tell them "Sure, we support enable lazy loading, just make sure you call this init() function." Since they're already making a code change anyway (to enable lazy loading), they should not have a backcompat objection to that.

Lazy imports for Python

mathstuf — Thu, 08 Sep 2022 13:12:15 +0000

> You can't possibly know what "every application" is going to depend on (you might be a dependency of a dependency of a dependency that the application barely calls into at all), so I'm skeptical that this is a real thing.

The library is VTK. What happens is that there are "object factory" types that can give you back a subclass if they are "registered". So you ask for a `vtkOpenGLRenderWindow` and you get back one that actually does X or Cocoa without having to know which you want at every call site. There are a variety of classes that do this.

In C++, there's a mechanism to see "oh, you included the base class" and the buildsystem injects "and this TU knows about overloads in library X and Y, so call X and Y's registration mechanisms" through that include through some injected global static ctors that ensure registration happens. Due to the way various platforms actually work, this needs to be done in every TU to be reliable[1]. Python accesses this by importing the module that has overloads and then it works "by magic" due to the module initializer hooking this mechanism (since the wrappers need to include the relevant headers).

> because import-time side effects are evil.

I'm not saying they're not, but alas C++ gives such little control over global initializers, that for some solutions, there's no reasonable alternative. As much as *I'd* like to make these things scoped and explicit, getting everyone to update is beyond the software process capital I'm willing to spend (as if I had that much anyways).

[1] For example, static builds on macOS (at one point?) only call object-local global initializers when something from the object is actually used. There's no place to put a library ctor at the static-library level, so you just need to inject "everywhere" to be reliable.

Lazy imports for Python

mathstuf — Thu, 08 Sep 2022 13:03:33 +0000

While true, I think it'd be hard to argue that such APIs don't exist today. Is Python effectively forcing[1] these APIs to be redesigned with this feature?

For the project I know that will be affected by this, it is all in the library ctor stuff that stuff gets set up; there's no API *to* call. Again, not the best design, but doing it explicitly would make "everyone" unhappy because of how much work it ends up doing for users.

[1] Via feature request pressure to "support lazy loading" over time.

Lazy imports for Python

mathstuf — Thu, 08 Sep 2022 13:00:07 +0000

I suspect that avoiding even finding that bytecode is part of the expected savings here. However, maybe modules could have this mark to at least avoid recursively loading all of the needed modules immediately. That might be a nicer compromise point.

Lazy imports for Python

LtWorf — Thu, 08 Sep 2022 10:17:51 +0000

As a (mostly) python developer, I'd be happy to see this go in.

At work we have an internal python tool that to do --version loads 604 files.

This is mostly due to type annotations, that force the imports just to declare a function that uses a certain type.

Importing modules before using them is annoying and requires a lot of discipline, but is only a partial solution

As developer of typedload (like pydantic but better) I am not sure this would help me much. The code imports a bunch of modules to support those types, but the types are referenced so the import happens anyway. I'd have to do a lot of trickery to only reference them when they are actually needed, but that would probably mean that the library would no longer work with cython.

Lazy imports for Python

taladar — Thu, 08 Sep 2022 08:30:31 +0000

As if Python didn't already have enough ways to fail in interesting and exciting ways at runtime when the installed libraries don't match what it expected.

Lazy imports for Python

josh — Thu, 08 Sep 2022 08:20:24 +0000

I wonder if this could autodetect correctness? When byte-compiling a Python module, look for either a lack of any statements at top level other than undecorated declarations (or perhaps also decorators that have an annotation opting in), or an explicit opt in, and emit a flag in the bytecode (or a flag file if that's easier to check for). If that flag exists, do lazy loading for that module.

Lazy imports for Python

yaap — Thu, 08 Sep 2022 07:40:01 +0000

Interesting, it looks like a distant cousin of Emacs autoload package support: different in the details, but same goal to reduce start-up time. In Emacs an elisp package can declare some public commands as autoloaded, they will be implemented as stubs that load the full package on first use (and the stubs are replaced by the full commands).

Lazy imports for Python

NYKevin — Wed, 07 Sep 2022 22:06:11 +0000

> The PEP says that laziness will be per-module.

This is incorrect, I misread https://peps.python.org/pep-0690/#deep-eager-imports-over.... My bad.

Lazy imports for Python

NYKevin — Wed, 07 Sep 2022 21:59:55 +0000

IMHO this is a result of poor API design. It should really look like this:

import mount # Filesystem support
import mount_xfs # XFS support

mount.mount(type=mount_xfs)

Or maybe you use type=mount_xfs.mount_type or something like that, but you can just pass the whole module if you want to. Python doesn't care. They're all PyObject* under the hood.

Lazy imports for Python

NYKevin — Wed, 07 Sep 2022 21:57:51 +0000

> - setting up lazy imports only works from a `__name__ == '__main__'` context; and

The PEP says that laziness will be per-module. If you call set_lazy_imports() in foo.py, and bar.py does not call set_lazy_imports(), then all of the imports in bar.py will be eagerly evaluated (even if foo.py lazily imports those same modules first - what happens is that bar.py's import gets eagerly evaluated, and then when foo.py's lazy import eventually needs to be evaluated, the import machinery resolves it back to the same module that bar.py already imported, just like it would if both imports were eager). IMHO if some library wants to use lazy imports as part of an internal implementation detail that application code will never see, then we might as well allow it.

> - modules/packages have a way to say "I will never support lazy loading" to avoid having every single consumer have to add it to their exclusion list.

This basically translates to "I'm deliberately causing nontrivial import-time side effects that every application will necessarily depend on." You can't possibly know what "every application" is going to depend on (you might be a dependency of a dependency of a dependency that the application barely calls into at all), so I'm skeptical that this is a real thing. But regardless, I don't think Python should encourage this sort of chicanery, or go out of its way to support it, because import-time side effects are evil.

Lazy imports for Python

mathstuf — Wed, 07 Sep 2022 21:57:30 +0000

Any module which uses top-level code to talk to other modules and do any global state setup. For a kernel-flavored example:

```python
import mount # Filesystem support
import mount_xfs # Enable XFS support

mount.mount(type='xfs') # Oops, XFS isn't actually loaded because the `mount_xfs` name wasn't tickled.
```

Lazy imports for Python

willy — Wed, 07 Sep 2022 21:40:06 +0000

I'm not particularly python-savvy (but I enjoy the insights into the evolution of the language). What are the kinds of things that can break with lazy loading?

Lazy imports for Python

mathstuf — Wed, 07 Sep 2022 19:46:54 +0000

Seems mostly reasonable. However, there are two things I'd like to see:

- setting up lazy imports only works from a `__name__ == '__main__'` context; and
- modules/packages have a way to say "I will never support lazy loading" to avoid having every single consumer have to add it to their exclusion list.