Disabling Python's lazy imports from the command line

The advent of lazy imports in the Python language is upon us, now that PEP 810 ("Explicit lazy imports") was accepted by the steering council and the feature will appear in the upcoming Python 3.15 release in October. There are a number of good reasons, performance foremost, for wanting to defer spending—perhaps wasting—the time to do an import before a needed symbol is used. However, there are also good reasons not to want that behavior, at least in some cases. The tension between those two positions is what led to an earlier PEP rejection, but it is also playing into a recent discussion of the API used to control lazy imports.

We looked at the PEP shortly before its acceptance and there is quite a bit of history of the idea going much further back than the 2022 rejection of a different PEP that would have made all imports lazy by default. PEP 810 adds a new "lazy" soft keyword that can be used to indicate a module (or symbol) that should not be imported immediately. Instead, proxy objects are created for the symbols that are resolved (instantiated or "reified") when they are needed. An example from our earlier article helps illustrate:

    lazy import abc  # abc is now bound to a lazy proxy object
    lazy from foo import bar, baz  # foo, bar, baz all proxies

    abc.def()  # loads module abc

    bar()  # resolves bar, which loads foo, baz still proxy
    baz()  # resolves baz, does not reload foo

There are various restrictions on lazy imports, such as that wildcard imports (e.g. from foo import *) cannot be lazy and that they can only appear at the module level, so not inside functions or classes. In addition, a lazy import is really only potentially lazy as there are a few ways that users or programs can alter the processing of imports. It is one of those settings that started the recent discussion.

In mid-February, Peter Bierma posted his concerns about the "-X lazy_imports=none" command-line setting, which turns off all lazy imports, resulting in what is often called "eager" imports, for a program and any of its dependencies. That flag can also be set via an environment variable or the sys.set_lazy_imports() call. The effect of lazy_imports=none is to make all imports be eager, meaning they work the way they always have, but using it overrides any explicit uses of lazy that modules may have made. Some of those lazy imports may have been done to avoid circular-dependency loops, thus lazy_imports=none potentially breaks them.

Bierma referred to a pull request from Pablo Galindo Salgado, one of the PEP's authors, to convert multiple uses of "old-style" lazy imports (generally placed in the functions where the symbols would be needed) in the standard library to the explicit version. Galindo Salgado eventually closed the request after David Ellis pointed out that multiple modules would fail with an ImportError if they were run in eager mode. Ellis generally preferred the new form, but thought that it needed "to be done with care (are there already tests to make sure the stdlib is still importable under lazy_imports=none?)".

One of the reasons given for having a way for libraries to turn off lazy imports is that the pip package installer needs to be able to prevent imports to avoid executing code from the package being installed. Currently, the pip developers ensure that any old-style lazy imports are resolved before code from the wheel gets installed. While pip installs files into the environment it is running in, the tool has always promised not to run any of the code in a wheel at install time, so it does any importing before that step in the installation process. If lazy imports become more widespread, particularly in the standard library, some mechanism to control those imports will be needed by pip and lazy_imports=none looked like it would do what was needed. Bierma said that a pip issue describes the problem, but he did not think lazy_imports=none actually solves it, and if using the flag causes problems it could lead to one of two outcomes:

1. Most libraries ignore the existence of -X lazy_imports=none, so disabling lazy imports results in circular import errors, making the flag effectively unusable anywhere outside the standard library. This seems most likely to me.
2. Or, people see this as too frustrating to reason about, so they continue to use the old system of using an eager import to mimic lazy imports, preventing widespread adoption of the new syntax.

Either way, I think it would be very unfortunate if the standard library couldn't use the new syntax because of this flag.

He suggested perhaps using the Python audit facility to prevent pip from importing anything after files from a wheel have been installed. Pip developer Damian Shaw agreed that using lazy_imports=none may not be the right approach. While it may be unfortunate that pip alters the environment it is running in, doing so is a longstanding pip "feature" that is deeply wired into its design—it cannot really be changed at this point.

Several in the thread agreed with Bierma that the lazy_imports=none option should simply be removed. Another pip developer, Paul Moore, also thought that pip should probably not use lazy_imports=none to combat the problem with arbitrary code execution from the installation of a wheel, which is the potential problem with doing imports that could pick up newly installed files. He had no opinion on whether that option should be removed, "but I don't think pip should be used as a reason to claim that it needs to stay".

There was some discussion of alternatives to removing lazy_imports=none in the thread, but most seemed to agree on removal. As Moore noted, the lazy imports filter provides a way for those who really want to disable the feature. It allows a program to pass a function that is run on each potential lazy import; if it returns False, the import is processed eagerly:

Anyone who really wants to force all imports to be eager can still do sys.set_lazy_imports_filter(lambda *args: False) - and that form makes it very clear what you'd change if you needed to add an exclusion list of modules that could be imported lazily.

Part of the problem with lazy_imports=none is that it is a big hammer, while filters provide a more fine-grained approach. Meanwhile, though, neither addresses the old-style lazy imports, which pip needs to disable as well. The current thinking seems to be to use a call to something like the resolve_all_lazy_imports() function described by Ellis to explicitly reify any pending imports and then disable further imports after that, which should make pip safe.

So one of the main intended users of the lazy_imports=none mode probably should not use it, but it turns out that another possible user would also be better served with other techniques. Back in the discussion of PEP 810, there was concern expressed about latency-sensitive programs being disrupted by a "surprise" lazy import. Michael Hall was one of those concerned, but said that he no longer thought the lazy_imports=none flag was the right approach for handling that either. An approach using resolve_all_lazy_imports() or similar should be sufficient, he thought.

After Donald Stufft wondered about the real security concern with pip importing modules during installation, Shaw explained that users might install a wheel to inspect it and would not expect that to execute code. Since the thread had gotten long, Moore summarized the key points for pip and lazy loading, including why pip must avoid running the new code:

Users have a reasonable expectation that running pip install <some_wheel> will not execute arbitrary code. That's a deliberate design feature of the wheel format.

While removing the flag (at least at run time with -X lazy_imports=none) seemed to be popular, it is notable that none of the PEP authors were part of the discussion. But Cornelius Krupp was concerned that the PEP authors would need to be involved in making this change and it would perhaps need to be run past the steering council again:

The only thing removing the flag would do is to signal that it's ok and expected for libraries to have a strict reliance on lazy imports for e.g. optional dependencies or circular dependencies, which is a significant change from the conditions under which the PEP was originally accepted, directly contradicting it's text and the expressed intentions of the authors.

Oscar Benjamin noted that the text of the PEP has changed over time, especially with regard to circular imports and the guards often placed around typing imports, which are only needed when a type checker is being used. The current text is a little contradictory in that respect. The final message (as of this writing) is from Brénainn Woodsend who reiterates a real problem with the existence of the flag; he and other library developers may be loath to remove their existing old-style lazy imports:

As long as -X lazy_imports=none exists, I'm reluctant to port existing forms of deferred imports to lazy import knowing that I would be slowing down or breaking anyone who uses -X lazy_imports=none.

It is not clear where things go from here. As noted, the PEP authors have not weighed in, but Galindo Salgado is obviously aware of the issue due to his closed pull request. It would seem that the use cases envisioned for the flag do not actually need it and there are other ways to accomplish the same thing, though not directly at run time from the command line. Once Python 3.15 ships later this year, it may be harder to retract the command-line flag, so it would seem that some kind of decision is needed here before too long.