2to3 was a bad idea

Posted Feb 21, 2018 11:32 UTC (Wed) by niner (subscriber, #26151)
In reply to: 2to3 was a bad idea by vstinner
Parent article: Hovmöller: Moving a large and old codebase to Python3

Inline::Perl5 works with a single process using Perl 5's embedding interface. So yes, there are two interpreters/virtual machines, but they are in the same process. Data has to be marshalled/converted between the two and one needs shims to make e.g. classes of the one language available in the other. Inline::Perl5 makes this rather seamless though with the effect that for most cases it just magically works(tm). Of course Perl's flexibility as a language helps in this. Note that there are also Inline::Python modules both for Perl 5 and Perl 6. They allow for the same kind of interoperability between Python and Perl.

For compatibility between Python 2 and Python 3 code, embedding is not an option since ironically their internals are too similar, i.e. they share symbol names. But combining the two is still possible using two processes and IPC, like you imagined Inline::Perl5 would work. That would have been an option since day one of Python3K. Of course it's not perfect and performance would be an issue. But there are countless cases where something like this could have spared users a lot of pain and made it easier for people to start using Python 3, even though some essential 3rd party libraries had not migrated yet. And it'd have made it easier for library authors to upgrade without losing their user base. It could even still help today.

As the author of Inline::Perl5 I can tell you, that getting it to a point where I could use Perl 5's database support in Perl 6 was an effort of two fun afternoons of hacking. Once you've got a basic communications channel open, it's really just a matter of making it more comfortable to use and blur the borders between languages.

Regarding the success of Python 3 vs. Perl 6 I can share a user's perspective: my company is stuck on Python 2 with no sane way to upgrade while we are already using some Perl 6 code in production. And the sole reason for the latter is the ability to combine both Perls in a system.

2to3 was a bad idea

Posted Feb 22, 2018 6:59 UTC (Thu) by mb (subscriber, #50428) [Link] (8 responses)

>For compatibility between Python 2 and Python 3 code, embedding is not an option

I don't think this is the case.

It is easily possible to have a program that runs on 3.x and 2.7, as long as new 3.x-only features and old deprecated pre-2.7 features are not used.
So in the transition period it's just needed to first port to 2.7 and then to 3.x.
Code embedding is not needed at all, because the syntax is fully compatible. The rest is handled by the highly dynamic nature of the language. You can just do if 3: foo; else bar for API differences in the libraries.

2to3 was a bad idea

Posted Feb 22, 2018 7:41 UTC (Thu) by niner (subscriber, #26151) [Link] (6 responses)

The point is that you have to port all your code including all your dependencies and their dependencies and ... to 2.7 and then again all your code and all those dependencies to 3.x. If that were easy or quick, it wouldn't have taken Python 3 10 years to gain acceptance.

2to3 was a bad idea

Posted Feb 22, 2018 8:35 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (5 responses)

>If that were easy or quick, it wouldn't have taken Python 3 10 years to gain acceptance.

2.7 didn't exist 10 years back nor did 3.x releases that bridged the gap between 2.x and 3.x releases. If 3.0 had all of that relevant functionality of the latest 3.x release and 2.7 was released before 3.0 and the appropriate libraries like six or future were released along with 3.0 release, the time taken would be less but the Python community didn't have all the insight into this that they have now.

2to3 was a bad idea

Posted Feb 22, 2018 18:36 UTC (Thu) by raven667 (subscriber, #5198) [Link] (4 responses)

> If 3.0 had all of that relevant functionality of the latest 3.x release and 2.7 was released before 3.0 and the appropriate libraries like six or future were released along with 3.0 release, the time taken would be less but the Python community didn't have all the insight into this that they have now.

I'm not sure of that, I imagine that there were python developers who were pointing out the transition difficulties at the time, who saw the engineering and social challenges, but were ignored by the leadership. It might be good for someone more familiar with the details to go back and figure out who was right and who was wrong, to have some accountability for those decisions, to identify who has a good intuition for likely future outcomes and who doesn't. One of humanities super-powers over other living things is the ability to accurately predict the future, to envision consequences, but at the time decisions get made you don't know for certain what is likely to be outcome, without looking back you can't assign weight to conflicting opinions and worldviews, so every decision is like its being made for the first time.

2to3 was a bad idea

Posted Feb 23, 2018 20:20 UTC (Fri) by rahvin (guest, #16953) [Link] (3 responses)

> It might be good for someone more familiar with the details to go back and figure out who was right and who was wrong, to have some accountability for those decisions, to identify who has a good intuition for likely future outcomes and who doesn't.

I don't think pointing fingers will get you anywhere, regardless of who was responsible they appear to have mostly figured it out at this point. People hopefully learned from the mistake. Besides, it's probably more than one or even two people that made the initial decision. IME pointing fingers is literally the worst thing you can do. It's a destructive process to attempt to assign blame, and it often destroys team cohesion.

2to3 was a bad idea

Posted Feb 26, 2018 21:46 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (2 responses)

No finger pointing was mentioned. The idea is to analyze how the decision-making process works (or doesn't) and how to improve it. Those who care about blame will find some and assign it anyways.

2to3 was a bad idea

Posted Feb 26, 2018 22:16 UTC (Mon) by raven667 (subscriber, #5198) [Link]

Yes, that is what I meant, playing some blame and shame game is not effective to generate learning, only the avoidance of pain, whereas accountability and analysis of the decision making process, after the outcomes are known, can generate better decisions in the future about the future.

2to3 was a bad idea

Posted Mar 6, 2018 11:02 UTC (Tue) by dgm (subscriber, #49227) [Link]

I mostly agree. The question is not about pointing fingers, but about avoiding such bad decissions in the future. For that, looking at the root cause is the most important thing. Were there any good arguments that were being ignored? If so, why? Of course, you have to balance that with analysis-paralisis. Decissions need to be made in reasonable time, otherwise they are not any good.

Who got it right is important, but only so much. You have to take into account the broken clock effect (you know, even a broken clock gives the right time twice a day), so better start listening to people with future-viewing super powers once they have proved themselves, at least, a couple of times.

2to3 was a bad idea

Posted Feb 26, 2018 21:49 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

Yes, Python and C code could be Python2 and Python3 compatible, but that requires code changes. True embedding means I can link to libpython2.7.so *and* libpython3.5m.so at the same time and mediate between them (see Lunatic[1] for how it could work). The use of the same symbols and global variable names prevents this approach.

[1]https://labix.org/lunatic-python