From late-bound arguments to deferred computation, part 2

Discussion on PEP 671 ("Syntax for late-bound function argument defaults") has been going on—in fits and starts—since it was introduced last October. The idea is to provide a way to specify the default for a function argument that is evaluated in the scope of the function call, which will allow more concise, and visible, defaults. But there has been a persistent complaint that what the language needs is a more-general deferred computation feature; late-bound defaults would simply fall out as one specific user of the feature. The arrival of a proposal for deferred computation did not really accomplish that goal, however.

Last week, we looked at the discussion of late-bound argument defaults after it got rekindled in June. There are some developers who are interested in having a way to specify argument defaults that are evaluated at run time in the context of the call. For example:

    def fn(a, max=len(a)):
        ...

That function definition looks sensible at first glance, but len(a) is evaluated at function-definition time, so it does not refer to the argument a at all. PEP 671, authored by Chris Angelico, would add a new way to specify defaults that are evaluated using the context at the beginning of the function call (using "=>"), so the following would do what the example above looks like it is doing:

    def fn(a, max=>len(a)):
        ...

But throughout all of the discussions, deferred computation was repeatedly raised as a better way forward; it was mostly championed by Steven D'Aprano and David Mertz. The details of what that feature might look like were not ever described until fairly late in the game, which was annoying to several of the participants in the threads. Things got more heated than is usual in discussions on the python-ideas mailing list, but that heat finally dissipated after a deferred-computation proposal was made.

A proto-PEP

As Brendan Barnwell and Paul Moore noted, a counter-proposal, such as one for deferred computation, is not required in order to oppose PEP 671. But it is difficult—frustrating—to try to defend a proposal when the "opposition" is so amorphous; that frustration can clearly be seen in the discussions. On June 21, Mertz posted the first version of his proto-PEP for "Generalized deferred computation". It would use the soft keyword later to identify expressions that should only be evaluated when a value is actually required for a non-later expression:

    a = 5
    b = later a + 2        # b is a deferred object
    c = later a * b + b    # c is a deferred object
    print(c)               # c and b are evaluated

The result is 42, as expected. The PEP abstract notes that a deferred object can be thought of as a "thunk" or likened to an existing Python construct: "To a first approximation, a deferred object is a zero-argument lambda function that calls itself when referenced."

The proto-PEP, which has never progressed to the point of getting a non-placeholder number, mentions the delayed() interface from the Dask Python parallel-computation library as something of a model for the feature. However, a Dask delayed object is only evaluated when its compute() method is called, while Mertz's version would automatically evaluate deferred objects when their value is used in a non-deferred context.

The proposal also talked about having a way to determine whether an object was deferred without evaluating it. He suggested that type() would not evaluate a deferred object or, perhaps, an isdeferred() function would be added for that purpose. That puzzled Barnwell who noted that it violates a basic Python principle: arguments to functions are evaluated "before the function gets any say in the matter". Making type() or isdeferred() do something different would be strange.

Mertz agreed that was a problematic piece, but he would like to find a way to query an object without evaluating it. Carl Meyer suggested that PEP 690 ("Lazy Imports"), which he co-authored, might provide some help on that; in general, even though Mertz's proposal was borne out of the PEP 671 discussion, Meyer thought it is more closely aligned with the lazy imports feature. But Angelico pointed out that there is a fundamental question that was not answered in the deferred computation proposal:

What are the scoping rules for deferred objects? Do they access names where they are evaluated, or where they are defined? Consider:

def f(x):
    spam = 1
    print(x)

def g():
    spam = 2
    f(later spam)

g()

Does this print 1 or 2?

This is a fundamental and crucial point, and must be settled early. You cannot defer this. :)

Angelico is demonstrating the two different scopes here. If later spam is evaluated in the scope where it was defined, spam will be 2, but if it is evaluated in the scope where it is evaluated, it will be 1.

Meyer agreed that specifying the scoping was crucial and thought it obvious that deferred objects would use the names from the scope where they were defined. Mertz initially disagreed, but eventually came around to that view as well. What that means, though, is that the feature cannot support late-bound arguments as Angelico had been saying for some time. The example given in the proto-PEP is:

    def func(items=[], n=later len(items)):
	n  # Evaluate the Deferred and re-bind the name n
	items.append("Hello")
	print(n)

    func([1, 2, 3])  # prints: 3

Angelico pointed out that the lookup of items in that example would be done in the surrounding scope:

That's why PEP 671 has the distinction that the late-bound default is scoped within the function body, not its definition. Otherwise, this doesn't work.

No one has replied to Angelico, so it would seem that using the generalized mechanism for late-bound arguments is a non-starter. That does not necessarily mean that the deferred computation feature should be abandoned, but it would seem that whatever barrier it imposed for PEP 671 has disappeared.

More problems

Meanwhile, there are other problem spots for deferred computation, including further difficulties with the late-bound-arguments use case. Martin Di Paola noted that Dask, PySpark, the Django object-relational mapping (ORM) layer for databases, and other similar tools do much more under the covers than simply evaluate the deferred object. In each case, the framework can manipulate the abstract syntax tree (AST) to optimize the evaluation in some fashion. Moore was concerned that only having the automatic-evaluation mechanism would preclude these other use cases (and those that may arise) from using the proposed mechanism:

My concern is that we're unlikely to be able to justify *two* forms of "deferred expression" construct in Python, and your proposal, by requiring transparent evaluation on reference, would preclude any processing (such as optimisation, name injection, or other forms of AST manipulation) of the expression before evaluation.

I suspect that you consider evaluation-on-reference as an important feature of your proposal, but could you consider explicit evaluation as an alternative? Or at the very least address in the PEP the fact that this would close the door on future explicit evaluation models?

Another problem arose from a familiar source; Angelico pointed out another late-bound-argument use case that fails. Stephen J. Turnbull reframed the problem and suggested a possible way forward, though things got ever more tangled.

    def foo(cookiejar=defer []):

foo() produces a late bound empty list that will be used again the next time foo() is invoked.

Now, we could modify the defer syntax in function parameter default values to produce a deferred deferred object [...]

He guessed that might lead Angelico to reply with something akin to the "puke emoji", but instead Angelico poked holes in that idea as well:

def foo(cookiejar=defer []):
    cookiejar.append(1) # new list here
    cookiejar.append(2) # another new list???

So the only way around it would be to make the defer keyword somehow magical when used in a function signature, which kinda defeats the whole point about being able to reuse another mechanic to achieve this.

It was around that point where Mertz seemed to perhaps throw in the towel on the idea, at least in terms of supporting late-bound defaults using it. Overall, it would seem that a lot more thought and work will need to go into a deferred-computation proposal before it can really even get off the ground. It is a popular feature from other languages, especially functional languages, but there may not be a real fit for it in the Python language. As mentioned, the technique is already used in a variety of ways in the ecosystem—without any change to the language syntax.

In the two months since that flurry of discussion, things have been quiet. The barriers to pushing PEP 671 forward would seem to have lessened some, at minimum, though that may still not be enough to get it across the finish line. As noted, there are no huge wins for adding this syntax; there are already only slightly more verbose ways to manually create late-bound defaults.

But back in October, Guido van Rossum was pleased to see work happening on fixing that particular language wart; he is no longer the final arbiter, of course, but he still has plenty of influence within the core Python development community. In order to proceed, if Angelico is still wanting to do so, he will need a core developer to sponsor the PEP, since, surprisingly, he is not one. Even if the PEP were to fail, it might be worth pushing it a bit further if a sponsor can be found; that way, the next time the idea gets raised, much of the discussion can be short-circuited by referring to the PEP—in theory anyway.

Index entries for this article
Python	Arguments
Python	Deferred computation
Python	Python Enhancement Proposals (PEP)/PEP 671

---

to post comments