LWN: Comments on "The return of lazy imports for Python"

The return of lazy imports for Python

empiko — Sun, 25 Dec 2022 21:35:07 +0000

Alternatively, you can do this:

foo = None

def func():
    if foo is None:
        import foo
    ...

The return of lazy imports for Python

mgedmin — Fri, 16 Dec 2022 05:24:18 +0000

Python 3.11 creates frame objects lazily, only when needed. I wonder how that affects this consideration.

The return of lazy imports for Python

mathstuf — Thu, 15 Dec 2022 00:13:50 +0000

While the direct configuration of hooks changing might be detectable, knowing what each hook cares about is…hard. Environment? Some module global state? Modules already loaded?

The return of lazy imports for Python

mathstuf — Thu, 15 Dec 2022 00:12:20 +0000

Yeah, this was part of a reconfigure for buildbot where restarting everything for a config update stopped all builds and caused them to start from scratch. This was disruptive enough that doing some, frankly, horrible things to `sys.modules` to support this was worth it.

The return of lazy imports for Python

warrax — Wed, 14 Dec 2022 19:31:03 +0000

Would that not be detectable via a quick check? How frequent are changes to the hooks?

Scientific Python SPEC 1

NYKevin — Wed, 14 Dec 2022 18:52:25 +0000

That seems really nifty, and it looks like they put the meat of the work on PyPI as a separate library: https://pypi.org/project/lazy_loader/

IMHO this is objectively superior to the PEP, since it allows laziness to be an implementation detail of the library, rather than something the application developer has to worry about. If an application developer goes plumbing lazy_loader into an existing library, they know perfectly well that they're carrying a patch and can't reasonably expect upstream to support it. Conversely, if an application developer doesn't want anything to do with lazy loading, they don't even have to know that it is there, because it Just Works.

The return of lazy imports for Python

NYKevin — Wed, 14 Dec 2022 18:46:26 +0000

Rule of thumb: reload() is for testing a module at the REPL (after you have modified it and want to test the modified version). If you put reload() in a .py file, it is likely to end in tears unless you know exactly what you are doing.

(It's sometimes necessary for things like dynamic plugins that you want to load at runtime, but this is the exception rather than the rule. Most of the time, you're better off loading each module no more than once.)

Scientific Python SPEC 1

hodgestar — Wed, 14 Dec 2022 15:15:15 +0000

There is also the Scientific Python SPEC 1 lazy importer. It requires libraries to opt-in (good) and uses the PEP 562 ability to override module `__getattr__`, etc that was implemented in Python 3.7. Python 3.7 is currently the oldest Python that is not end of life.

The return of lazy imports for Python

osma — Wed, 14 Dec 2022 14:09:39 +0000

I for one, as an application developer, would welcome the syntactic and performance sugar of optional, shallow lazy imports. I've done inlined imports in the past to avoid some extremely slow imports like NumPy, but that is cumbersome and adds a lot of unnecessary lines of code.

The return of lazy imports for Python

0x3333 — Wed, 14 Dec 2022 13:41:10 +0000

Guys, just focus on making the interpreter faster 🤣🤣🤣

The return of lazy imports for Python

mathstuf — Wed, 14 Dec 2022 13:35:03 +0000

Note that another difference in "manual inlining" is that when unloading/reloading a module, all of its global imports become `None`. Function-local and class-local imports are fine. This causes lots of…fun when a module is registered as callbacks and you suddenly start getting `logging is None` exceptions. Why this seems so "rigorously undocumented" is beyond me; granted, it might be *implied* by `importlib.reload`'s docs, but they do not seem to be written for mere mortals.

The return of lazy imports for Python

NYKevin — Wed, 14 Dec 2022 06:03:09 +0000

That has to call an entire function, compared to looking up a global. Even if the LOAD_GLOBAL code path grows a little extra complexity for lazy objects, a function call is still much more expensive, because it sets up and tears down an entire stack frame object. If you do that pervasively throughout the entire application, you'll end up churning the heap allocator unnecessarily (Python puts frame objects on the heap so that they can be referenced and kept alive by traceback objects when an exception is thrown).

The return of lazy imports for Python

coderanger — Wed, 14 Dec 2022 03:29:09 +0000

It does bail out long before loading the file a second time, but there's still some processing before the point where it can tell it's definitely a duplicate (relative->absolute resolution, etc).

The return of lazy imports for Python

nybble41 — Wed, 14 Dec 2022 03:28:03 +0000

The repeated import issue doesn't seem that difficult to work around:

def lazy_foo(): if not hasattr(lazy_foo, "cached"): import foo lazy_foo.cached = foo return lazy_foo.cached def func1(): lazy_foo().bar() def func2(): lazy_foo().baz()

This will import foo exactly once, the first time either func1() or func2() calls lazy_foo(). After that lazy_foo() just returns the module which was already imported.

The return of lazy imports for Python

xi0n — Wed, 14 Dec 2022 03:13:10 +0000

Because of import hooks. The meaning of the same ‘import foo’ can change in between invocations if a hook was added or removed from sys.meta_path.

The return of lazy imports for Python

mb — Tue, 13 Dec 2022 22:42:08 +0000

> Manual inlining invokes the import system every time the function is called,
> which has a noticeable cost.

Yes, why is that so expensive anyway?
This cost hit me often in the past already.

Can't the import system bail out early, if the import has already been done?