LWN: Comments on "A proposal for "rebooted" Python asynchronous I/O support"

A proposal for "rebooted" Python asynchronous I/O support

tack — Sun, 06 Jan 2013 02:35:32 +0000

In which framework were you tasked with debugging a maze of yield froms?

This is very different from the callback world of Twisted. The code reads fairly linearly like it would with gevent, except that your exit/entry points are explicit rather than implicit.

Having coded with both approaches (that is, explicit yields vs. greenlet-style implicitness) I prefer to be explicit. I rather like the visual cue that signals where my function will suspend and resume and I find this improves debuggability rather than hinders it. IMO, yielding gives you a nice bit of async magic while not violating the principle of least astonishment.

A proposal for "rebooted" Python asynchronous I/O support

cmccabe — Sat, 05 Jan 2013 19:53:53 +0000

From the libabc readme:

> Avoid thinking about main loops/event dispatchers.
>  - Get your stuff right in the kernel: fds are awesome, expose them
>    in userspace and in the library, because people can easily integrate
>    them with their own poll() loops of whatever kind they have.
>  - Don't hide file descriptors away in your headers.
>  - Never add blocking kernel syscalls, and never add blocking library
>    calls either (with very few exceptions). Userspace code is primarily
>    asynchronous around event loops, and blocking calls are generally
>    incompatible with that.
>  - Corollary of that: always O_NONBLOCK!

I can't think of any library I have used in the past few years that has used a glib-based main loop. So basically, if your library depends on some event loop framework... you're doing it wrong.

Self-fulfilling prophecy

man_ls — Fri, 04 Jan 2013 11:53:46 +0000

I would have very much appreciated an explicit Python mention, e.g.:

Let the discussion about the color of Arthur "Two Sheds" Jackson's second shed begin!

But bikeshedding? Heresy!

A proposal for "rebooted" Python asynchronous I/O support

smurf — Fri, 04 Jan 2013 11:23:15 +0000

Given that a model like gevent's is (a) a whole damn lot more natural to code to than all that "yield from" stuff, and (b) explicitly out of scope, I'd have to agree to the "rigid and obsolete" comment.

Debugging a maze of "yield from"s and/or explicitly registered callbacks is a *major* PITA. Been there, done that, given up, and re-implemented a couple of ten thousand lines with gevent instead of Twisted.

Fortunately, implementing a Twisted reactor on top of gevent has already been done, so the migration can be done in steps and is rather painless.

I am fairly sure that the same thing will be done for this PEP, once it's moderately final.

Self-fulfilling prophecy

JanC_ — Thu, 03 Jan 2013 22:23:20 +0000

And of course this was meant to read like this:

You forget that this particular BDFL named his programming language after a comedy group

(Can we get a post editor, please? ;) )

Self-fulfilling prophecy

JanC_ — Thu, 03 Jan 2013 22:18:14 +0000

<p>You forget that this particular BDFL named his programming language after <a href="">a comedy group</a></p>

A proposal for "rebooted" Python asynchronous I/O support

intgr — Thu, 03 Jan 2013 18:12:38 +0000

In "advance" of what? This isn't a specification, this is a proposal, and it's not ready yet.

You are kindly invited to collaborate and improve it.

A proposal for "rebooted" Python asynchronous I/O support

wahern — Tue, 01 Jan 2013 00:10:42 +0000

FWIW, I believe Python coroutines will work much like Javascript generator trampolines: http://www.neilmix.com/2007/02/07/threading-in-javascript...

A proposal for "rebooted" Python asynchronous I/O support

wahern — Mon, 31 Dec 2012 23:47:58 +0000

I think you misunderstood what I was saying. To understand the fundamental problem in the CPython (and Jython) VM, just think about the difference between CPython and Stackless. You can't `yield from' inside a new function invocation because of the way call frames are implemented in CPython. Only the callee of a `yield from' statement can yield. But you can explicitly stack yield froms in a call chain. Kinda messy, but it's a fundamental limitation of CPython.

Coroutines in languages like Lua are implemented more like threads, so coroutine semantics aren't limited to a single callee invocation. In Lua, invocation C in A->B->C can yield, even though coroutine resumption began in A. In CPython, when resuming A only A can yield. A can, however, _explicitly_ invoke B as a coroutine. But B (perhaps a library call) can't hide the details. A needs to adorn it's call to B in order to get coroutine semantics. Again, not so in Lua or some other languages.

A proposal for "rebooted" Python asynchronous I/O support

Arach — Mon, 31 Dec 2012 22:53:10 +0000

Rigid and obsolete in advance.

A proposal for "rebooted" Python asynchronous I/O support

tack — Sun, 30 Dec 2012 04:36:54 +0000

You've misunderstood the PEP. Coroutines are implemented using generators, yes, but they can be arbitrarily nested (within runtime stack depth limits of course). In other words, coroutine A can "yield from" coroutine B which can "yield from" coroutine C and so on. If C yields a Future object, the stack unravels -- through B and A -- and the coroutine scheduler will resume C's execution when the Future it yielded is done.

A proposal for "rebooted" Python asynchronous I/O support

wahern — Sun, 30 Dec 2012 03:35:32 +0000

From the PEP Guido says they're basically just generators underneath, which means these aren't like more modern coroutine implementations which can yield from sub-subroutine invocations. Being restricted to the first-level invocation will hardly remove the necessity for callbacks in high-level application code, unfortunately.

This is why I much prefer Lua. It might not have the huge library, but at least the VM is done right.

A proposal for "rebooted" Python asynchronous I/O support

ppisa — Sun, 30 Dec 2012 00:33:29 +0000

I like proposal and please, invest time to to ensure good compatibility with libevent-2 main loop and consider even its buffer events. I think that that library is good candidate for generic mainloop for most of applications.

But there is need to live with GLIB based mainloop as well. Its interface is so rigid and unfortunate that only option for other GUI systems (like Qt) was to switch to GLIB mainloop when GLIB based loop libraries are considered for them. I hope that GLIB community would invest time to that problem some day but it cannot be done without API level compatibility break.

You can look at my ul_evpoll effort as a source for some ideas too. It started in libevent-1 days as an solution to write libraries requiring system level events registration and compatibility with GLIB and yet to be able to switch to libevent. Then the project little extended to provide own implementation of select, poll, epoll and winevent mechanism and ability to plug-in into GLIB, libevent-1 and libevent-2 based applications.

The ul_evpoll library is part of uLAN and OrtCAN SF.net repositories
* http://ulan.sourceforge.net/
* http://ulan.git.sourceforge.net/git/gitweb-index.cgi
I have prepared snapshot for simple standalone testing
* http://cmp.felk.cvut.cz/~pisa/ulan/ul_evpoll/ul_evpoll-12...
Documentation - qute short, written in a hurry and with many sins against English language
* http://cmp.felk.cvut.cz/~pisa/ulan/ul_evpoll/doc/ulevpoll...

Self-fulfilling prophecy

man_ls — Sat, 29 Dec 2012 19:02:34 +0000

Probably van Rossum said the quoted sentence as a joke, and my humor detector is on holidays too. It just sounded a bit harsh coming from a "benevolent dictator" to me.

Self-fulfilling prophecy

corbet — Sat, 29 Dec 2012 18:52:39 +0000

Bear in mind that this PEP comes after a few months of quite intense conversation. Have a look at the python-ideas list starting in October. It's not like the conversation is starting now, in the middle of the holiday lull.

Self-fulfilling prophecy

man_ls — Sat, 29 Dec 2012 18:38:41 +0000

Yes, you are right on that respect. Not too much discussion though, especially for such an interesting topic. Python lost momentum in the async server area when it could have led the pack.

Self-fulfilling prophecy

augustz — Sat, 29 Dec 2012 17:48:56 +0000

The comments look constructive and on-point to me.

A proposal for "rebooted" Python asynchronous I/O support

tack — Sat, 29 Dec 2012 16:26:02 +0000

With a model like Twisted's, you have a twisty little maze of callbacks.

So one of the primary drivers of this PEP is coroutines, which let us solve the callback hell problem seen in languages like Javascript (especially with node.js). To standardize on coroutines used for async IO, you also need to standardize a bunch of other things like a Future-like object and a minimal event loop.

Once these things are defined and codified, libraries like Twisted and Kaa can conform to them, and this just might, if we play our cards right, pave the way for async library interoperability in Python land. Today each library takes a different and incompatible approach.

Also Twisted's API is not PEP8 compliant. :)

A proposal for "rebooted" Python asynchronous I/O support

njs — Sat, 29 Dec 2012 14:32:58 +0000

Twisted is seriously awesome. But I would be very surprised to see it end up in the standard library. Just to start with, twisted as a whole is a huge sprawling mess of variable quality code, so you'd have to extract some small pieces (just the reactor interface?), and even those interfaces probably have stuff in them that you'd want to clean up given how many years its been since they were first designed. And twisted relies heavily on zope-style interfaces, which have been rejected from the stdlib before. And twisted is still a long way from running on Python 3.

A proposal for "rebooted" Python asynchronous I/O support

marduk — Sat, 29 Dec 2012 13:30:53 +0000

I didn't read the entire PEP but it seems that Twisted, ZeroMQ and Tornado were thought of and can easily implement the interface for the PEP. It seems that one of the goals is to have a a common interface (think wsgi or the db-api) while a second goal is to have a reference implementation included in the standard library.

Self-fulfilling prophecy

robert_s — Sat, 29 Dec 2012 13:28:03 +0000

"Let the bikeshedding begin!"

You could say the same thing about any PEP. But miraculously a lot of them get accepted.

Self-fulfilling prophecy

man_ls — Sat, 29 Dec 2012 12:19:51 +0000

Let the bikeshedding begin!

Not a very good start for a healthy discussion. As could be expected there are to date only a handful of messages in the thread.

A proposal for "rebooted" Python asynchronous I/O support

cglass — Sat, 29 Dec 2012 11:38:35 +0000

I'll go ahead and ask the obvious question here, apologies if this seems obvious to people more familiar with the topic at hand:
Why not twisted? Why reimplement the same functionality with a slightly different interface?
It certainly isn't perfect, but I would be interested in understanding the reasons.