LWN: Comments on "Late-bound argument defaults for Python"

Late-bound argument defaults for Python

mathstuf — Wed, 01 Dec 2021 21:57:54 +0000

Sure, you *can*. But they don't act like you think:

def f(p=[]):
    p.append(1)
    return p
print(f()) # [1]
print(f()) # [1, 1]

This is because the default is only created upon function definition, not upon function entry. And given the mutability of Python, you can fiddle with that default inadvertently.

Late-bound argument defaults for Python

hellcat_coder — Wed, 01 Dec 2021 18:12:59 +0000

Can't we default arguments to empty lists, dictionaries or even objects as of now in functions? Or am I missing something?

Late-bound argument defaults for Python should focus on the source scope

ccurtis — Thu, 18 Nov 2021 16:12:56 +0000

I like the "@" syntax ("&" would also work for me), but the function signature proposed seems fundamentally flawed. In this statement:

The x=x parameter uses global x as the default. The y=x parameter uses the local x as the default. We can live with that difference. We *need* that difference in behaviour, otherwise these examples won't work:
    def method(self, x=>self.attr)  # @x=self.attr

    def bisect(a, x, lo=0, hi=>len(a))  # @hi=len(a)

This comment implies that a "global x" exists and so means a finer-grained specification is warranted. If true, this seems a much better approach:

    def method(self, x=self.attr+@self.attr)  # 'self' is global self, '@self' is the local self

That said, I know nothing more about Python than what I just read and I don't expect Van Rossum to be reading this, but if it makes sense someone may want to mention it on the list. It is the RHS scope that is of interest...

Late-bound argument defaults for Python

smurf — Wed, 17 Nov 2021 13:55:38 +0000

Not really. Binding to function args should proceed strictly left-to-right, evaluating a =: clause iff the function call didn't pass a value for it.

Late-bound argument defaults for Python

foom — Wed, 17 Nov 2021 12:33:32 +0000

With that definition, you could have funny functions like:
def f(a=:b+1, b=:a+1): ...

And it'd work if you call it like f(a=5) or f(b=5), but f() would return an unknown variable error.

Late-bound argument defaults for Python

smurf — Sat, 13 Nov 2021 13:02:21 +0000

Why? This is about default values, which by definition don't appear in your function call in the first place.

Presumably these get evaluated left-to-right *after* your arguments get bound. Anything else would make no sense whatsoever.

Late-bound argument defaults for Python

lobachevsky — Sat, 13 Nov 2021 11:48:16 +0000

Maybe I'm missing that from the discussion, but the defining late bindings left to right in the function definition makes me somewhat unhappy when I think about keyword-only arguments.

For me the point of keyword-only arguments is that I don't need to know the order they appear in in the function definition, but if they were to use late-bound arguments now, the function definition order becomes important again. This seems like a potential footgun for API breakage, when the definition order were to change.

Late-bound argument defaults for Python

malmedal — Fri, 12 Nov 2021 20:41:27 +0000

Yes, that was what I meant by "or needs to be evaluated late". Perhaps that was a bit terse.

Late-bound argument defaults for Python

NYKevin — Fri, 12 Nov 2021 20:00:43 +0000

Even doing that is questionable, because the decorator does not "see" a code object that evaluates to an empty list. It just sees an empty list, and has to figure out how to manufacture a new empty list each time the function is called. So this would not work:

@late
def foo(x=bar()):
    ...

By the time @late receives control, bar() has already been called and has returned some value. There is no way that @late can plausibly figure out that it needs to call bar() again in the future. The best it can do is call something like copy.copy() on bar()'s return value, and hope that's close enough.

Late-bound argument defaults for Python

malmedal — Fri, 12 Nov 2021 18:53:15 +0000

No, not something that references another argument or needs to be evaluated late.

What you can do is to write something such that this:
@late
def foo(a, b=[]):
...

would give b a new empty list on each invocation instead of reusing the original.

Late-bound argument defaults for Python

nybble41 — Fri, 12 Nov 2021 16:40:37 +0000

Ah, so it depends on whether there is an assignment. Thank you for clearing that up. It still seems quite non-intuitive that adding an assignment later in the function (without an explicit "nonlocal" declaration) can completely change the scope of an earlier use of the variable, but at least the given example would work.

> However, if some_functor() had rebound dflt to some other value before returning (for example, if dflt were a loop variable), and we were using late-binding defaults, then the late-binding would very likely get the new value and not the value that was originally computed by something_expensive(blah). This is probably not what the programmer intended to happen …

Actually that is exactly how I would expect it to work, based on the behavior of free variables in other languages, e.g. Common Lisp or even Javascript. Closures capture the variable (a.k.a. the binding), not the specific value in the variable at the time the closure is created. Though my preference is for languages like Haskell (or Rust) where (shared) variables are always immutable so this situation doesn't come up. (Of course if the variable is something like an IORef or STRef then reading from it gives the most recent value, but in that case the effect is rather obvious and generally intentional.)

Of course, these other languages with closures and mutation also have support for explicit *local* (not just function) scope, so you can express whether you want to assign to an existing variable (CL: setf) or create a new binding (CL: let). Python has no block scope—apart from some special cases like generator & dict/set comprehension expressions—which makes it difficult to control the scope of free variables in a closure. Even within comprehensions like "[(lambda: i) for i in range(N)]" the variable is reused across the loop iterations rather than being freshly bound for each value, so this just gives a list of functions which all return N-1. To get the more intuitive result you would need something ugly like "[(lambda j: lambda: j)(i) for i in range(N)]" to emulate the missing local binding in the body of the loop.

Late-bound argument defaults for Python

smurf — Fri, 12 Nov 2021 10:51:17 +0000

Short answer: No. Decorators aren't designed to do that and adding that capability to the CPython core would be an unholy mess.

Late-bound argument defaults for Python

Visse — Fri, 12 Nov 2021 10:03:36 +0000

Could this be achieved using decorators instead of adding new syntax?
Something like:

      @late_bound_arguments
      def foo(a, b = None, c = len(a)):
           ...

This would have the benefit of being more searchable than new syntax.

Late-bound argument defaults for Python

NYKevin — Fri, 12 Nov 2021 00:16:46 +0000

> That was my first thought as well, but in Python unknown / unassigned variables default to being local, not free variables with lexical scope as in most other languages.

It's more complicated than that.

In Python, *most* interesting things happen at runtime, but name resolution is one of the few things that actually happens at compile time (I believe for performance reasons). The compiler looks at each variable, and follows a process like* this (first matching rule wins):

0. If the scope has an explicit global/nonlocal statement, then the variable is interpreted as such and bytecode is emitted accordingly.
1. If, anywhere in the scope, there's an assignment, then the variable is local to that scope, and we emit LOAD_FAST/STORE_FAST bytecode.
2. If a variable of the same name exists in an enclosing function (not class) scope, then it's a non-local or "closed over" variable. We emit complicated bytecode which sets up a closure. Closure variables are looked up by name at the time the closure is executed, so if the enclosing function rebinds the variable before returning, the closure will observe the new binding. This is different to how function parameters work, and so is a common source of confusion. If you're used to C++ closures, this is roughly equivalent to using [&] instead of [=], automatically, on every closure, without any option of doing it differently.
3. If none of the above rules applies, then it's a global or a builtin, and we emit LOAD_GLOBAL (we do not emit STORE_GLOBAL, because rule 1 or rule 0 would have applied in that case). LOAD_GLOBAL checks for globals and then for builtins at runtime.

Corollary: The set of variables in each non-global scope is fixed at compile time, because we have to emit the correct bytecode in order for a variable to be looked up in any non-global scope. You cannot add new variables to a non-global scope at runtime, and trying to evaluate a non-global variable before you assign to it raises UnboundLocalError instead of looking for a global variable of the same name.

In the specific case shown, since there is no assignment involved (i.e. the default value expression does not involve the walrus operator), I assume that the late-binding logic would generate a closure under rule 2 (and if it did not, then IMHO that would be a pretty egregious bug). The resulting bytecode would be ugly, but it should work correctly. However, if some_functor() had rebound dflt to some other value before returning (for example, if dflt were a loop variable), and we were using late-binding defaults, then the late-binding would very likely get the new value and not the value that was originally computed by something_expensive(blah). This is probably not what the programmer intended to happen, and in fact early-binding defaults are commonly used to work around this problem.

* I did not actually pull up the CPython source code, so it's possible that I have missed a case or oversimplified something.

Late-bound argument defaults for Python

nybble41 — Thu, 11 Nov 2021 15:58:00 +0000

That was my first thought as well, but in Python unknown / unassigned variables default to being local, not free variables with lexical scope as in most other languages. In the absence of a "nonlocal" declaration, which is not possible here due to the syntax, the "dflt" in the default argument expression is merely an uninitialized local variable and would not resolve to the variable of the same name in the enclosing function.

Late-bound argument defaults for Python

epa — Thu, 11 Nov 2021 08:03:57 +0000

Why would that ability be lost? Since dflt isn't in scope at call time, it can be bound at compile time instead. There would only be an ambiguity if you had an argument called dflt also.

Late-bound argument defaults for Python

NYKevin — Thu, 11 Nov 2021 01:10:00 +0000

Here is a more straightforward example of that use case:

funcs = []
for i in range(4):
    funcs.append(lambda x=i: x)
print([f() for f in funcs])  # [0, 1, 2, 3]

If you write the seemingly-obvious lambda: i, you get [3, 3, 3, 3] instead, because at the moment the function is actually called, i=3. In effect, we use one wart in the language to cancel out another.

Late-bound argument defaults for Python

mathstuf — Wed, 10 Nov 2021 22:40:31 +0000

That loses the ability to default arguments to some computation in the defining scope:

def some_functor(blah):
    dflt = something_expensive(blah)
    def f(d=dflt):
        pass
    return f

Late-bound argument defaults for Python

khim — Wed, 10 Nov 2021 21:54:02 +0000

C++ supported late-bound defaults exclusively for more than quarter-century.

Late-bound argument defaults for Python

benhoyt — Wed, 10 Nov 2021 20:42:07 +0000

Better late than never.

I see what you did there. :-)

My question: what about not using new syntax? I get that it wouldn't be backwards compatible, but it could be done on a file-by-file basis with a "from __future__ import late_bound_defaults", avoiding the need for new syntax entirely. Was that discussed at all?

Late-bound argument defaults for Python

keithp — Wed, 10 Nov 2021 20:30:16 +0000

Hah! Snek is ahead of the game — it only supports late-bound arguments as that required less code in the compiler and runtime.

    Welcome to Snek version 1.7
    > def foo(a, b = len(a)):
    +  return b
    + 
    > foo("hello")
    5

Now to decide if I care enough to go implement early-bound arguments.

Late-bound argument defaults for Python

smurf — Wed, 10 Nov 2021 18:26:46 +0000

This would *finally* allow us to default arguments to empty lists, directories, or in fact any other new-and-empty object.

Better late than never.