Szorc: Mercurial's Journey to and Reflections on Python 3

Posted Jan 20, 2020 10:33 UTC (Mon) by smurf (subscriber, #17840)
In reply to: Szorc: Mercurial's Journey to and Reflections on Python 3 by HelloWorld
Parent article: Szorc: Mercurial's Journey to and Reflections on Python 3

> It very much is about language features.

Well, sure, if you have a nice functional language where everything is lazily evaluated then of course you can write generic code that doesn't care whether the evaluation involves a loop or a context switch or whatever.

But while Python is not such a language, neither is Go, so in effect you're shifting the playing ground here.

> Not having error handling built into the language doesn't mean you have to check for errors on every call.

No? then what else do you do? Pass along a Haskell-style "Maybe" or "Either"? that's just error checking by a different name.

Szorc: Mercurial's Journey to and Reflections on Python 3

Posted Jan 20, 2020 12:51 UTC (Mon) by HelloWorld (guest, #56129) [Link] (1 responses)

Well, sure, if you have a nice functional language where everything is lazily evaluated then of course you can write generic code that doesn't care whether the evaluation involves a loop or a context switch or whatever.

You don't need lazy evaluation for this to work. Scala is not lazily evaluated and it works great there.

No? then what else do you do? Pass along a Haskell-style "Maybe" or "Either"? that's just error checking by a different name.

You can factor out the error checking code into a function, so you don't need to write it more than once. After all, this is what we do as programmers: we detect common patterns, like “call a function, fail if it failed and proceed if it didn't” and factor them out into functions. This function is called flatMap in Scala, and it can be used like so:

getX().flatMap { x =>
  getY(x).map { y =>
    x + y
  }
}

But this is arguably hard to read, which is why we have for comprehensions. The following is equivalent to the above code:

for {
  x <- getX
  y <- getY x
} yield x + y

I would argue that if you write it like this, it is no harder to read than what Python gives you:

x = getX
y = getY(x)
return x + y

But the Scala version is much more informative. Every function now tells you in its type how it might fail (if at all), which is a huge boon to maintainability. You can also easily see which function calls might return an error, because you use <- instead of = to obtain their result. And it is much more flexible, because it's not limited to error handling but can be used for things like concurrency and other things as well. It's also compositional, meaning that if your function is concurrent and can fail, that works too.

Szorc: Mercurial's Journey to and Reflections on Python 3

Posted Jan 20, 2020 18:30 UTC (Mon) by darwi (subscriber, #131202) [Link]

> But the Scala version is much more informative. Every function now tells you in its type how it might fail (if at all), which is a huge boon to maintainability

Long time ago (~2013), I worked as a backend SW engineer. We transformed our code from Java (~50K lines) to Scala (~7K lines, same functionality).

After the transition was complete, not a single NullPointerException was seen anywhere in the system, thanks to the Option[T] generics and pattern matching on Some()/None. It really made a huge difference.

NULL is a mistake in computing that no modern language should imitate :-( After my Scala experience, I dread using any language that openly accepts NULLs (python3, when used in a large 20k+ code-base, included!).

Szorc: Mercurial's Journey to and Reflections on Python 3

Posted Jan 20, 2020 15:46 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (9 responses)

> Pass along a Haskell-style "Maybe" or "Either"? that's just error checking by a different name.

Yes, but with these types, *ignoring* (or passing on in Python) the error takes explicit steps rather than being implicit. IMO, that's a *far* better default. I would think the Zen of Python agrees…

Szorc: Mercurial's Journey to and Reflections on Python 3

Posted Jan 20, 2020 17:21 UTC (Mon) by HelloWorld (guest, #56129) [Link] (8 responses)

> Yes, but with these types, *ignoring* (or passing on in Python) the error takes explicit steps rather than being implicit. IMO, that's a *far* better default.

No, passing the error on does not take explicit steps, because the monadic bind operator (>>=) takes care of that for us. And that's a Good Thing, because in the vast majority of cases that is what you want to do. The problem with exceptions isn't that error propagation is implicit, that is actually a feature, but that it interacts poorly with the type system, resources that need to be closed, concurrency etc..

Szorc: Mercurial's Journey to and Reflections on Python 3

Posted Jan 20, 2020 18:28 UTC (Mon) by smurf (subscriber, #17840) [Link] (6 responses)

Python doesn't have a problem with resources to be closed (that's what "with foo() as bar"-style context managers are for), nor concurrency (assuming that you use Trio or anyio).

Typing exceptions is an unsolved problem; conceivably it could be handled by a type checker like mypy. However, in actual practice most code is documented as possibly-raising a couple of well-known "special" exceptions derived from some base type ("HTTPError"), but might actually raise a couple of others (network error, cancellation, character encoding …). Neither listing them all separately (much too tedious) nor using a catch-all BaseException (defeats the purpose) is a reasonable solution.

Szorc: Mercurial's Journey to and Reflections on Python 3

Posted Jan 20, 2020 22:44 UTC (Mon) by HelloWorld (guest, #56129) [Link] (2 responses)

Python doesn't have a problem with resources to be closed (that's what "with foo() as bar"-style context managers are for), nor concurrency (assuming that you use Trio or anyio).

Sure, you can solve every problem that arises from adding magic features to the language by adding yet more magic. First, they added exceptions. But that interacted poorly with resource cleanup, so they added with to fix that. Then they realized that this fix interacts poorly with asynchronous code, and they added async with to cope with that. So yes, you can do it that way, because given enough thrust, pigs fly just fine. But you have yet to demonstrate a single advantage that comes from doing so.

On the other hand, there are trivial things that can't be done with with. For instance, if you want to acquire two resources, do stuff and then release them, you can just nest two with statements. But what if you want to acquire one resource for each element in a list? You can't, because that would require you to nest with statements as many times as there are elements in the list. In Scala with a decent library (ZIO or cats-effect), resources are a Monad, and lists have a traverse method that works with ALL monads, including the one for resources and the one for asynchronous tasks. But while asyncio.gather (which is basically the traverse equivalent for asynchronous code) does exist, there's no such thing in contextlib, which proves my point exactly: you end up with code that is constrained to specific use cases when it could be generic and thus much easier to learn because it works the same for _all_ monads.

Szorc: Mercurial's Journey to and Reflections on Python 3

Posted Jan 21, 2020 6:51 UTC (Tue) by smurf (subscriber, #17840) [Link] (1 responses)

> But what if you want to acquire one resource for each element in a list?

You use an [Async]ExitStack. It's even in contextlib.

Yes, functional languages with Monads and all that stuff in them are super cool. No question. They're also super hard to learn compared to, say, Python.

Szorc: Mercurial's Journey to and Reflections on Python 3

Posted Jan 21, 2020 14:32 UTC (Tue) by HelloWorld (guest, #56129) [Link]

> You use an [Async]ExitStack. It's even in contextlib.
You can *always* write more code to fix any problem. That isn't the issue here, it's about code reuse. ExitStack shouldn't be needed, and neither should AsyncExitStack. These aren't solutions but symptoms.

> They're also super hard to learn compared to, say, Python.
For the first time, you're actually making an argument for putting the things in the language. But I'm not buying it, because I see how much more stuff I need to learn about in Python that just isn't necessary in fp. There's no ExitStack or AsyncExitStack in ZIO. There's no `with` statement. There's no try/except/finally, there's no ternary operator, no async/await, no assignment expressions, none of that nonsense. It's all just functions and equational reasoning. And equational reasoning is awesome _because_ it is so simple that we can teach it to high school students.

Szorc: Mercurial's Journey to and Reflections on Python 3

Posted Jan 21, 2020 2:32 UTC (Tue) by HelloWorld (guest, #56129) [Link] (2 responses)

I also think you're conflating two separate issues when it comes to error handling: language and library design. On the language side, this is mostly a solved problem. All you need is sum types, because they allow you to express that a computation either succeeded with a certain type or failed with another. The rest can be done in libraries.

If listing the errors that an operation can throw is too tedious, I would argue that that is not a language problem but a library design problem, because if you can't even list the errors that might happen in your function, you can't reasonably expect people to handle them either. You need to constrain the number of ways that a function can fail in, normally by categorising them in some way (e. g. technical errors vs. business domain errors). I think this is actually yet another way in which strongly typed functional programming pushes you towards better API design.

Unfortunately Scala hasn't proceeded along this path as far as I would like, because much of the ecosystem is based on cats-effect where type-safe error handling isn't the default. ZIO does much better, which is actually a good example of how innovation can happen when you implement these things in libraries as opposed to the language. Java has checked exceptions, and they're utterly useless now that everything is async...

Szorc: Mercurial's Journey to and Reflections on Python 3

Posted Jan 21, 2020 7:13 UTC (Tue) by smurf (subscriber, #17840) [Link] (1 responses)

> Java has checked exceptions, and they're utterly useless now that everything is async

… and unstructured.

The Java people have indicated that they're going to migrate their async concepts towards Structured Concurrency, at which point they'll again be (somewhat) useful.

> If listing the errors that an operation can throw is too tedious, I would argue that that is not a language problem but a library design problem

That's one side of the medal. The other is that IMHO a library which insists on re-packaging every kind of error under the sun in its own exception type is intensely annoying because that loses or hides information.

There's not much commonality between a Cancellation, a JSON syntax error, a character encoding problem, or a HTTP 50x error, yet an HTTP client library might conceivably raise any one of those. And personally I have no problem with that – I teach my code to retry any 40x errors with exponential back-off and leave the rest to "retry *much* later and alert a human", thus the next-higher error handler is the Exception superclass anyway.

Szorc: Mercurial's Journey to and Reflections on Python 3

Posted Mar 19, 2020 16:52 UTC (Thu) by bjartur (guest, #67801) [Link]

Nesting result types explicitly is helpful because it makes you wonder when an exponential backoff is appropriate.

How about getWeather:: String→ DateTime→ IO (DnsResponse (TcpSession (HttpResponse (Json Weather)))) where each layer can fail? Of course, there's some leeway in choosing how to layer the types (although handling e.g. out-of memory errors this way would be unreasonable IMO).

Szorc: Mercurial's Journey to and Reflections on Python 3

Posted Jan 21, 2020 22:42 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

Even `>>=` is explicit error handling here (as it is with all the syntactic sugar that boils down to it). Using the convenience operators like >>= or Rust's Result::and_then or other similar methods are explicitly handling error conditions. Because the compiler knows about them it can clean up all the resources and such in a known way versus the unwinder figuring out what to do.

As a code reviewer, implicit codepaths are harder to reason about and don't make me as confident when reviewing such code (though the bar may also be lower in these cases because error reporting of escaping exceptions may be louder ignoring the `except BaseException: pass` anti-pattern instances).