|
|
Subscribe / Log in / New account

The return of None-aware operators for Python

The return of None-aware operators for Python

Posted Jan 5, 2024 9:17 UTC (Fri) by marcH (subscriber, #57642)
Parent article: The return of None-aware operators for Python

> sometimes I've found the best solution is to stop the None's getting into the list (or whatever structure)", because it encourages better coding practices.

This is spot on; best and most accurate comment on the entire topic. Many times when writing code like "42 if x is None else x" I wondered "Wait, why can x be None in the first place?" and sometimes I did change the code so x could not be None any more.

Python's None is nowhere near as bad NULL in C/C++ (the so-called "billion dollar mistake") but it's still not great. So even if the new shortcuts were great, intuitive and very readable (which is debatable and debated), they would still only offer a very slightly lazier way to hide what is often another design issue.


to post comments

The return of None-aware operators for Python

Posted Jan 5, 2024 13:10 UTC (Fri) by khim (subscriber, #9252) [Link] (22 responses)

> Python's None is nowhere near as bad NULL in C/C++ (the so-called "billion dollar mistake") but it's still not great.

Huh? Python's None is worse than NULL. At least with NULL you know that you have to deal with pointers to hit that corner-case. None may be everywhere in Python.

> So even if the new shortcuts were great, intuitive and very readable (which is debatable and debated), they would still only offer a very slightly lazier way to hide what is often another design issue.

Sure, but the only bulletproof solution is to drop dynamic typing and add mandatory types. And then it wouldn't be a Python anymore.

The return of None-aware operators for Python

Posted Jan 5, 2024 16:11 UTC (Fri) by marcH (subscriber, #57642) [Link]

> Huh? Python's None is worse than NULL

It's better because it gives you a clear stack trace instantly.

> Sure, but the only bulletproof solution is to drop dynamic typing and add mandatory types. And then it wouldn't be a Python anymore.

OK I should really have avoided this bad, pointless and distracting comparison with C...

The return of None-aware operators for Python

Posted Jan 5, 2024 17:02 UTC (Fri) by NYKevin (subscriber, #129325) [Link] (3 responses)

> Sure, but the only bulletproof solution is to drop dynamic typing and add mandatory types. And then it wouldn't be a Python anymore.

This is a simplification. Python does provide support for static typing if you can be bothered to use it. If you run a FOSS project (or a closed-source project, for that matter), you can mandate that all code must pass a static type analysis (e.g. with mypy) before it may be merged. But that's up to you as a user of Python - the language won't force you to do it (just like it won't force you to write unit tests, run a linter, use a formatter, etc.).

The return of None-aware operators for Python

Posted Jan 8, 2024 7:02 UTC (Mon) by LtWorf (subscriber, #124958) [Link] (2 responses)

It's not like mypy and similar tools are perfect… they miss a lot of things. Even by enforcing them you still have dynamic typing.

And the typing cannot express some things, for example a function that takes T as a parameter and returns an instance of T as a result… there's been an issue open since years to support this, but in general it isn't supported.

The return of None-aware operators for Python

Posted Jan 8, 2024 8:39 UTC (Mon) by gdiscry (subscriber, #91125) [Link] (1 responses)

And the typing cannot express some things, for example a function that takes T as a parameter and returns an instance of T as a result… there's been an issue open since years to support this, but in general it isn't supported.

Do you mean something like this? (Python 3.12 syntax for brevity, but previous versions only require small tweaks)

def create_instance[T](cls: type[T]) -> T:
    ...

If not, I'm curious about what you meant.

Nevertheless, it's true that not all APIs can be correctly annotated in Python and it's frustrating when that happens. But when it can, it's really nice to avoid defensive programming at runtime.

The return of None-aware operators for Python

Posted Jan 10, 2024 15:44 UTC (Wed) by LtWorf (subscriber, #124958) [Link]

The return of None-aware operators for Python

Posted Jan 5, 2024 18:00 UTC (Fri) by marcH (subscriber, #57642) [Link] (16 responses)

BTW even Rust (the pinnacle of language design as we all know and revere) has Option so None can't be that bad.

But you have a point: Option must be explicit and cannot be everywhere.

OK, enough Apples to Oranges comparisons, I said I would stop...

The return of None-aware operators for Python

Posted Jan 5, 2024 20:44 UTC (Fri) by NYKevin (subscriber, #129325) [Link] (14 responses)

I would argue that Rust's None-aware operators are much better than what is proposed for Python, simply because they cause an implicit return rather than silently propagating a None. The latter is reasonable if you have the full machinery of monads at your disposal (so Haskell is correct to implement Maybe-bind that way), but Python does not (and will never) have monads.

Because of Python's laissez-faire multi-paradigm attitude, it's actually quite difficult to design a good implementation of None-aware operators. You can't really use the Rust solution, because Python is dynamically-typed and likes to signal errors with exceptions rather than sentinel values (i.e. you can't reasonably define it to propagate Result<V, E> or the like, since Result is not even a thing in Python). But the TypeError or ValueError that you tend to get from None is usually the wrong exception to throw, and propagating None as if it was NaN will make it difficult to get a traceback. The idiomatic behavior, in some cases, is to eagerly check for None and raise an exception.

Maybe this syntax could work?

x = y ?? raise FooError('y should not be None.')

But that is going to be problematic. Raise is a statement, not an expression, so you'd need to make a special case to allow it in this one context, or you'd need to convert it into an expression. And then people will also want to write x = y ?? z, so you need to allow for that as well.

I have no idea how this is supposed to be extended for ?. and ?[], because where are you supposed to put the raise?

The return of None-aware operators for Python

Posted Jan 7, 2024 14:21 UTC (Sun) by cpitrat (subscriber, #116459) [Link] (13 responses)

> x = y ?? raise FooError('y should not be None.')

May I interest you in assert?

The return of None-aware operators for Python

Posted Jan 8, 2024 22:06 UTC (Mon) by NYKevin (subscriber, #129325) [Link] (12 responses)

assert is not properly used for any purpose other than as a "live comment" (i.e. a comment that actually gets executed). This is because the -O flag disables asserts, so the implication is that your code is expected to be correct even when asserts are not run. To be more explicit about this, a proper assert should be describing an invariant which the surrounding code is intended to guarantee (and which it actually does guarantee, assuming it has been written correctly). Since the invariant is already upheld by the surrounding code, you don't have to check it for correctness, so the assert is just for documentation purposes (to inform the next person who reads the code that the invariant exists and is important).

(Also, you can't specify the exception type, but that's small potatoes in comparison.)

The return of None-aware operators for Python

Posted Jan 8, 2024 22:13 UTC (Mon) by khim (subscriber, #9252) [Link] (4 responses)

> assert is not properly used for any purpose other than as a "live comment" (i.e. a comment that actually gets executed). This is because the -O flag disables asserts, so the implication is that your code is expected to be correct even when asserts are not run.

I have just tested and that doesn't happen. What version of python are you using??? I have never observed that effect in Python, but there are many implementations, maybe one of them does that, but for me it's reason not to use it rather then change use of assert.

The return of None-aware operators for Python

Posted Jan 8, 2024 22:22 UTC (Mon) by mb (subscriber, #50428) [Link]

You don't pass -O

$ cat t.py
assert 2+2==5
$ python3.11 t.py
[...]
AssertionError
$ python3.11 -O t.py

The return of None-aware operators for Python

Posted Jan 8, 2024 23:19 UTC (Mon) by NYKevin (subscriber, #129325) [Link] (1 responses)

This has been documented as standard behavior at https://docs.python.org/3/using/cmdline.html#cmdoption-O basically forever (I believe it was probably added at some point in Python 2.x and then carried over into 3). I do not know why the link you provide appears to show Python ignoring this flag, but it is likely a problem with either Godbolt or your use of it, because this is not new or exotic behavior.

The return of None-aware operators for Python

Posted Jan 8, 2024 23:26 UTC (Mon) by NYKevin (subscriber, #129325) [Link]

I'm mistaken. This has been standard behavior since version 1.5, released in 1998, according to this old documentation: https://docs.python.org/release/1.5/ref/ref-8.html#HEADIN...

As far as I can tell, 1.4 did not have an assert statement, so the assert statement has never been unconditional in any released version of Python.

The return of None-aware operators for Python

Posted Jan 8, 2024 23:40 UTC (Mon) by ABCD (subscriber, #53650) [Link]

It seems that godbolt.org doesn't pass the "compiler arguments" to the interpreter the way you expected. If you use the PYTHONOPTIMIZE=1 environment variable instead, you get the expected behavior: https://godbolt.org/z/z8Wo5qYT4

The return of None-aware operators for Python

Posted Jan 8, 2024 23:36 UTC (Mon) by marcH (subscriber, #57642) [Link] (1 responses)

> a "live comment" (i.e. a comment that actually gets executed)

> so the assert is just for documentation purposes (to inform the next person who reads the code that the invariant exists and is important).

Small contradiction here.

I didn't know about the -O flag and I've always "run" asserts and every time one is hit it is massively more useful than a comment!

> so the implication is that your code is expected to be correct even when asserts are not run.

Agreed.

The return of None-aware operators for Python

Posted Jan 8, 2024 23:37 UTC (Mon) by NYKevin (subscriber, #129325) [Link]

> Small contradiction here.

Sorry, my brain is smaller than yours, can you please elaborate?

The return of None-aware operators for Python

Posted Jan 9, 2024 11:28 UTC (Tue) by cpitrat (subscriber, #116459) [Link]

I'm not sure why I was sure it was possible to choose the exception type, not only the message. And I had forgotten about -O deactivating it.

But you can easily define your own raiseIfNone.

The return of None-aware operators for Python

Posted Jan 9, 2024 15:11 UTC (Tue) by atnot (subscriber, #124910) [Link] (3 responses)

There's another super annoying issue with python's assert and it's that because assert is just a single keyword that checks a bool, you get zero information from it beyond "AssertionError". Unlike the asserts in many other languages or ecosystem, you don't get any info about what assertion failed, what the left or right hand side of the comparison was or any sort of message describing the errot. If you're lucky, you can maybe get the source location from the stack trace but that's about it. It's just never worth using them.

The return of None-aware operators for Python

Posted Jan 9, 2024 15:28 UTC (Tue) by kleptog (subscriber, #1183) [Link]

assert has a second argument where you can give a more descriptive message, and include the arguments if you like. But you have to remember to do it.

The return of None-aware operators for Python

Posted Jan 9, 2024 19:13 UTC (Tue) by NYKevin (subscriber, #129325) [Link]

Those things are all provided in unittest, and in contexts other than unit testing, I never find myself reaching for such functionality in the first place.

The return of None-aware operators for Python

Posted Jan 10, 2024 16:28 UTC (Wed) by laarmen (subscriber, #63948) [Link]

pytest actually does some voodoo that supercharges Python standard assert, including displaying subexpressions. Granted, it won't help if your assertion fails during normal runtime as opposed to test runs, but there might be a way to reuse that particular bit outside of the test framework?

The return of None-aware operators for Python

Posted Jan 6, 2024 0:40 UTC (Sat) by tialaramex (subscriber, #21167) [Link]

Although _technically_ Option's None is a langitem† for the most part Option is just a normal sum type. The word "None" in Rust isn't actually magic, the fact that in practice an Option<&T> with value None is going to be represented as an all-zeroes bit pattern on real computers, the same bit pattern as a NULL pointer in C, is in some sense a coincidence, albeit one that was inevitable.

Let's make our own, custom user-defined generic sum type we'll call it Perhaps<T> and it'll have two values, Huh, and Well(T). Ours works exactly the same way and sure enough the Huh value of Perhaps<&T> will also be represented by an all-zeroes bit pattern.

This behaviour is guaranteed by the language (it is named the Guaranteed Niche Optimisation and is crucial to the language's design) but similar behaviours are delivered in practice for more exotic arrangements. For example Rust's char type needs 4 bytes, but it only handles Unicode "scalar values" (basically think codepoints unless you really care about the minutia of Unicode) so there are a *lot* of unused values. Accordingly, a sum type with Tafkap, SimpsonsMeme and Other(char) will fit in the same 4 bytes as the char alone, because Rust will just squeeze Tafkap and SimpsonsMeme in as bit patterns which aren't valid for char. You aren't promised this will work, but the Rust compiler is going to do it anyway because it's faster and smaller and easy.

† Rust only "really" has one loop, the one introduced by the keyword loop. But for and while both work fine, because the compiler just transforms them into loops, this "de-sugaring" is actually spelled out in the documentation, and the de-sugaring of for (since it's a modern iterator for-each not a C-style for) needs all of IntoIterator, Iterator, Some and None - and so they have to be langitems, annotated core library features which must exist or the language won't work.

In Python things are very different, None is a completely different type than whatever you expected, just as the null pointer is a distinct type in C++.


Copyright © 2025, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds