Lazy imports for Python

Posted Sep 8, 2022 8:20 UTC (Thu) by josh (subscriber, #17465)
Parent article: Lazy imports for Python

I wonder if this could autodetect correctness? When byte-compiling a Python module, look for either a lack of any statements at top level other than undecorated declarations (or perhaps also decorators that have an annotation opting in), or an explicit opt in, and emit a flag in the bytecode (or a flag file if that's easier to check for). If that flag exists, do lazy loading for that module.

Lazy imports for Python

Posted Sep 8, 2022 13:00 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

I suspect that avoiding even finding that bytecode is part of the expected savings here. However, maybe modules could have this mark to at least avoid recursively loading all of the needed modules immediately. That might be a nicer compromise point.

Lazy imports for Python

Posted Sep 8, 2022 20:30 UTC (Thu) by NYKevin (subscriber, #129325) [Link] (2 responses)

This either doesn't work, or it's basically useless.

foo.py:

import bar

bar.py:

print("Side effect!")

main.py:

import foo

Now, if you make main.py lazily import foo.py, then the side effects of bar.py will stop happening, which main might have accidentally relied on. So you would need to treat import statements as "statements other than declarations" - which effectively means that almost any nontrivial module will be eagerly imported. Or, alternatively, you have to recursively trace through all of the modules in the entire dependency tree and check each one for this lazy import flag - which is still pretty expensive and doesn't really save you all that much (consider the disk seeks!).

Lazy imports for Python

Posted Sep 14, 2022 6:39 UTC (Wed) by arvidma (guest, #6353) [Link] (1 responses)

All of that recursive loading and parsing happens already today, since loading is eager. Putting a tag in each pyc file to indicate if it can be lazy loaded shouldn't need to add much cost (each pyc is touched twice instead of once, but they would all be in the page cache and that pyc-update ought to be comparatively cheaper than the parsing and compiling of the py-files).

This way, you only pay the cost once and any subsequent runs can take the fast path.

Lazy imports for Python

Posted Sep 15, 2022 8:54 UTC (Thu) by smurf (subscriber, #17840) [Link]

The idea behind lazy imports is to not access the file system at all until the module is actually used. As soon as you go off and search + read a .pyc or .pyo file header, the advantage of not going through the whole file system dance is gone (did you ever look at how many file system accesses Python needs to do to locate a module?) and you might as well import the thing.