Python 3 adoption
Python 3 adoption
Posted Apr 16, 2015 6:54 UTC (Thu) by Cyberax (✭ supporter ✭, #52523)In reply to: Python 3 adoption by fandingo
Parent article: Python 3 adoption
WTF? It's such a braindead statement, that I'm out of hands for a facepalm.
Linux has a stable userspace ABI. I can take a static binary compiled 15 years ago and it will _still_ _work_ on the newest kernels. That's why kernel release version doesn't matter.
> You're so hung up on version numbers, but this problem would've happened eventually even if it meant going to 2.8. You had plenty of warning, the library authors had plenty of warning; yet, you both did nothing. A version number isn't going to change that unpreparedness.
It's NOT A PROBLEM OF VERSION NUMBERS.
It's a problem of a minor Python update _breaking_ _user_ _code_. On purpose: https://bugs.python.org/issue21308
Want to try something similar with Linux? Simply submit a change that removes readdir() syscall and replaces it with a better designed interface. In a -rc6 release. And then say something like: "But it's to fix a terrible mistake from years ago!"
But please, give me a heads up a couple of hours before your submit your patches. I want to have enough time to grab popcorn and enjoy the show.
> So you say this is an ongoing problem, even though that bug report has more people saying that their issue was fixed than are still complaining. The question becomes where's your published fix? Surely you'd still be working 20-hour days if you can't point to a fix.
Actually, you're right. Here's a raw diff for docker-py: http://pastebin.com/3JpveFxu We used a dirty workaround for Boto which I'm not going to submit. Gevent patches were sent to its author. I have a patch for websocket-client, will submit it later.
I'm definitely going to find some time to submit all the changes upstream.
> The Gevent project is still dead; boto and your product aren't going to work with this Python 2.8 either, and the next round of forward leaning distros will pick up the newest stable version of Python anyways. What's the difference if nobody is maintaining Gevent regardless of Python version number?
Python 2.7 is going to be supported for the next 6 years. That was acceptable for us earlier. Now we're looking for an alternative with saner maintainers.
> It was not. It was a terrible mistake all those years ago to forgo default certificate validation by `ssl`.
Again, let me repeat. Python changes broke the code that was explicitly doing the right thing.
This change could have been incremental - a couple of optional parameters to pass the CA store parameters and perhaps a global function to override the default behavior. Such changes would have preserved the existing software that does its own validation _and_ they would have provided secure defaults for everybody else.
However, Python developers instead backported SSL layer from Python3, knowing perfectly well that it's going to break stuff. Let me quote them:
> > I doubt adding a ton of new APIs and code can be uneventful, but good
> luck :)
> They don't call it "Rawhide" for nothing! :)"
Or maybe:
> I spent hours looking at this patch, which certainly doesn't constitute a real review, but is probably about as good as your going to get on this behemouth.
So yes, I maintain that the developers are complete idiots.
> Honestly, just leave and stop whining. Oh no, the code that hasn't been updated in ages stops working when the rest of the environment is updated -- that's not in any way surprising!
I have code written for Java 1.3 that was released (wow!) 17 years ago. It still works perfectly on the most recent JDKs. Why should I rewrite the code that does what it should?
Posted Apr 16, 2015 7:54 UTC (Thu)
by daniels (subscriber, #16193)
[Link] (4 responses)
Sigh. Just stop here.
Even if your argument wasn't ridiculous (you're smarter than Guido and the rest of the Python team? ok, sure), quite what gives you the right to wander around abusing people because you disagree with their decisions is absolutely beyond me.
Grow up and accept that reasonable people can arrive at different conclusions without one of them being idiots. Your abuse contributes nothing whatsoever to the conversation, except a growing wish for comment moderation.
Posted Apr 16, 2015 8:04 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link] (3 responses)
Python 2.7 maintainers are STILL complete idiots since they go forward with obviously breaking changes for a minor release in a stable branch. This is not a case of a bug that crawled through the cracks in a patch, it was a deliberate breakage.
Posted Apr 16, 2015 8:14 UTC (Thu)
by corbet (editor, #1)
[Link] (2 responses)
Posted Apr 16, 2015 8:23 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
It was perfectly possible to write a small patch to add validation by default and allow to pass a custom CA-store, without breaking any existing code. It would have required a couple of dirty hacks inside the library and possibly some code duplication to add support for SNI, but it was entirely doable.
Python maintainers instead backported the whole SSL infrastructure from Python 3 which has a lot of changes inside of it. Here's the patch: https://bugs.python.org/file36423/ssl-backport.diff - it's almost 13000 lines long.
Posted Apr 16, 2015 8:31 UTC (Thu)
by daniels (subscriber, #16193)
[Link]
Posted Apr 16, 2015 15:06 UTC (Thu)
by fandingo (guest, #67019)
[Link] (7 responses)
> Actually, you're right. Here's a raw diff for docker-py: http://pastebin.com/3JpveFxu We used a dirty workaround for Boto which I'm not going to submit. Gevent patches were sent to its author. I have a patch for websocket-client, will submit it later.
So, umm, thanks for linking to that monstrosity of a diff. I read through it twice, and honestly, I don't see anything that would be relevant to incompatibilities introduced in 2.7.9. This more looks like a diff from a series of changes that included stuff super-related to the SSL problem -- like updating from Docker 1.12 to 1.18. Perhaps you can identify some areas that handle the SSL change.
If there were any SSL changes, they must be in <<unamed_class>>._post_json() or <<unamed_class>>._post() or in a different file.
I don't see any merge requests for gevent that could be possibly from you. Of course, that doesn't change the underlying problem that gevent is unmaintained.
I don't see that websocket-client ever had any incompatibilities. No open or closed issues, no mention in the changelog.
It all seems like subterfuge.
Getting back to the minor/major change. I originally thought your catastrophe originated from updating Python on your company's infrastructure. However, it seems that the problem was your customers running your code on their infrastructure. Since you weren't willing to temporarily say "we don't support 2.7.9" and barf out an error message until you have a proper workaround, putting this change (along with the whole boatload of stuff that will come) in a 2.8 doesn't solve *any* problem. 1) Gevent is still unmaintained and will break. 2) Ubuntu, Fedora, Arch, Debian Sid, etc. -- the ones who actually have 2.7.9 -- would upgrade to 2.8 in their next distro release, and it's going to affect all the same users.
Walk me through the situation where this change in a hypothetical 2.8 doesn't blow up in your face the same way.
> I have code written for Java 1.3 that was released (wow!) 17 years ago. It still works perfectly on the most recent JDKs. Why should I rewrite the code that does what it should?
The Lady of the Lake,
I dunno. Languages are made by different people for different purposes. Some of those include rigid compatibility. Python was obviously made to ruin your life, but hey, at least its successful at what it set out to do.
Posted Apr 16, 2015 18:10 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link] (6 responses)
Working on it...
> Getting back to the minor/major change. I originally thought your catastrophe originated from updating Python on your company's infrastructure. However, it seems that the problem was your customers running your code on their infrastructure. Since you weren't willing to temporarily say "we don't support 2.7.9" and barf out an error message until you have a proper workaround
> Walk me through the situation where this change in a hypothetical 2.8 doesn't blow up in your face the same way.
You know, the way major Python upgrades have always worked.
Posted Apr 16, 2015 19:07 UTC (Thu)
by fandingo (guest, #67019)
[Link] (5 responses)
> Working on it...
So, what was the point of posting that diff? You assumed that I wouldn't read it? It's not sounding like "several 20-hour days" worth of work anymore.
I am also still curious about the timeline when this blew up because it doesn't make sense. I don't see Debian or Ubuntu backporting these SSL changes. Outside of Debian Sid and Ubuntu 14.10, I'm struggling to even find an apt-based distro that actually has 2.7.9. Even Sid and 14.10 only got 2.7.9 1 month ago.
> Easy. A new major version is released that has its own Python symlink and can live alongside Python 2.7. Old code can still use Python 2.7 explicitly, especially if it has some custom SSL processing.
Sure that's possible. Which apt-based distro are you using that actually does this? Ubuntu and Debian only package one Python 2 version at a time, so you're talking about custom packaging. Then, of course, there's nothing stopping you from creating a 2.7.8 package and symlinking to your heart's content. Plus, your users are going to be using whatever the default is, and I can't see any situation where a hypothetical 2.8 doesn't become the default immediately in the next distro release.
> we were not willing to hold all updates for this.
Just the hypothetical ones that do things exactly like you want to avoid.
It seems more like you have an ideological axe to grind, and you're making up or exaggerating a minor situation to fit your ideology. You're not providing the technical info to substantiate how devastating this change was and are more interested in talking abstractly about major/minor version numbers and holy compatibility.
Posted Apr 16, 2015 22:56 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link] (4 responses)
> I am also still curious about the timeline when this blew up because it doesn't make sense. I don't see Debian or Ubuntu backporting these SSL changes. Outside of Debian Sid and Ubuntu 14.10, I'm struggling to even find an apt-based distro that actually has 2.7.9. Even Sid and 14.10 only got 2.7.9 1 month ago.
Please note, that it's NOT Python 2.7.9, it's a backport to Python 2.7.8.
We started seeing it in the wild right after the New Year. I can try to get precise details from their version control repo.
> Sure that's possible. Which apt-based distro are you using that actually does this? Ubuntu and Debian only package one Python 2 version at a time, so you're talking about custom packaging.
And then there's CentOS 6 which is still packaged with Python 2.6 by default.
Incidentally, I don't see hordes torch-wielding of CentOS users demanding SSL fixes to be backported into Python 2.6
> Plus, your users are going to be using whatever the default is, and I can't see any situation where a hypothetical 2.8 doesn't become the default immediately in the next distro release.
> Just the hypothetical ones that do things exactly like you want to avoid.
Posted Apr 18, 2015 1:39 UTC (Sat)
by fandingo (guest, #67019)
[Link] (3 responses)
In that light, I can definitely understand why you'd be angry about breaking changes pulled into an earlier version within a distro version. Yet, I don't see what it has to do with the Python developers. The Python developers made a new point release, and spent a considerable amount of time discussing and warning about the change.
> That's fine. A transition period of 2-3 years (one release cycle) would have given everyone time to adapt or at least to explicitly set the Python version in scripts.
Is there any indication that the state of Gevent is actually going to change? There have been a couple of pull requests before the actual release of 2.7.9 to fix this behavior, and the author(s) haven't done anything with them. It seems like the only hope would be that Gevent is forked, and people start getting the fork through their distro or pypi.
All of that being said, the fix in Gevent was simple, and if that library was maintained, the library would've been updated 3 months before 2.7.9 was released, which was when the first patch and merge request were posted.
Posted Apr 18, 2015 6:39 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
> The Python developers made a new point release, and spent a considerable amount of time discussing and warning about the change.
> Is there any indication that the state of Gevent is actually going to change? There have been a couple of pull requests before the actual release of 2.7.9 to fix this behavior, and the author(s) haven't done anything with them.
> All of that being said, the fix in Gevent was simple, and if that library was maintained, the library would've been updated 3 months before 2.7.9 was released, which was when the first patch and merge request were posted.
Posted Apr 18, 2015 7:28 UTC (Sat)
by fandingo (guest, #67019)
[Link]
If by "safe" you mean without compatibility issues, then you're terribly, terribly wrong. You obviously didn't read the release notes or PEP 466, which are linked in the notes. I'm in disbelief that you could possibly make this comment after all that we've discussed.
> The same thing happened with SuSE and Amazon Linux.
SUSE is on Python 2.6, so no, it didn't. I still don't understand why more entities doing the wrong thing absolves them of fault. The ssl changes were 2.7.9; what possible situation exists for taking all of the changes and putting them in something called the previous version. That's madness.
> I think only RedHat/CentOS refrained from doing it.
And Fedora.
You're really not going to budge an inch that those distros messed up by putting this serious compatibility break that you bemoan in a package that they call 2.7.8 are you? Wow, that really makes it difficult to take you seriously.
> There are now fixes for the SSL behavior.
They've been there since September 21, 3 months before the release of 2.7.9. But there weren't any maintainers to merge the simple change, so no update was available. That's the part that you seem so unwilling to admit: The necessary packages were simple, and identified long before users were impacted, but because there isn't any active development of Gevent, the work never got published.
> Other than that, gevent simply works. Why should it change every month?
Not with the current 2.7 release, so yeah, it does need to change. Considering that there's 46 open merge requests for a litany of issues, I'd say that users would like to see quite a bit of change. I don't see why you're so happy and content with using unmaintained code. Perhaps it will take some remote execution vulnerability to change your mind.
> And that's the problem. Python developers shouldn't expect people to scramble to do substantial code rewrites for patchlevel releases.
In what world is a half-dozen lines of code 3 months in advance scrambling? This is such complete and utter nonsense.
Posted Apr 19, 2015 21:12 UTC (Sun)
by pboddie (guest, #50784)
[Link]
Most people won't read this far into this thread, but that's the real issue here: there's a policy decision not to release another minor version of Python 2.x, so that not only will the core developers not support a Python 2.8 release themselves, they will also deny others the opportunity to make such a version with that label; this is why Stackless Python was renamed to just Stackless. Personally, I find this aversion to even a single subsequent minor version (2.8 in this case), in a series that isn't favoured by its principal developers, to be counterproductive. Such a version is an ideal place to put all the backwards-incompatible fixes that "finish the job", instead of pretending that some other "final" minor version (2.7 in this case) wasn't broken all along. It also won't confuse distribution packagers who risk breaking things because they didn't follow or can't second-guess every last policy decision of the core developers, see a "patch release" and dish it out believing that the usual conservatism has been upheld. In short, changing the rules and making late updates to 2.7.x into something they shouldn't have been is what has caused this problem. Conceding that 2.8 should have existed for such purposes, perhaps explicitly indicating a change to the rules for that version, would have been the correct decision.
Python 3 adoption
Python 3 adoption
...and without it somebody else would have called them complete idiots for leaving an apparently insecure situation in place. They were in a difficult spot and made the best decision they could. You may not agree with it — I'm not sure I agree with it — but calling them idiots does seem to be over the top. We should be able to treat each other better than that. I have no problem with expressions of disagreement here, but I'd really rather not see that kind of personal attack, please?
Python 3 adoption
Python 3 adoption
It was NOT a binary decision.
Python 3 adoption
Python 3 adoption
[angels sing]
her arm clad in the purest shimmering samite, held aloft ssl3
from the bosom of the water signifying by Divine Providence that you,
Cyberax, Python developer, must rewrite your code.
[singing stops]
Python 3 adoption
Mostly a 'send' method that I had to pull up into ssladapter from the default HTTP handler. A couple of other changes as well.
It was backported into 2.7.8. And yes, we were not willing to hold all updates for this.
Easy. A new major version is released that has its own Python symlink and can live alongside Python 2.7. Old code can still use Python 2.7 explicitly, especially if it has some custom SSL processing.
Python 3 adoption
Python 3 adoption
This diff alone isn't.
The Ubuntu 14.04 backport was committed sometime in December. You can see the offending code in the Ubuntu patch: http://archive.ubuntu.com/ubuntu/pool/main/p/python2.7/py... - grep for "_sslwrap".
Ubuntu did this during 2.6 -> 2.7 migration. You could install both versions alongside and either use an explicit '/usr/bin/python26' or use DPKG alternatives mechanism to select an interpreter.
That's fine. A transition period of 2-3 years (one release cycle) would have given everyone time to adapt or at least to explicitly set the Python version in scripts.
The problem was very real - people downloading default Ubuntu machine images on Amazon EC2 suddenly stopped being able to use our software.
Python 3 adoption
Python 3 adoption
The same thing happened with SuSE and Amazon Linux. They all have backported a patch that Python developers insisted was safe and essential. I think only RedHat/CentOS refrained from doing it.
A 'point release' would be Python 2.8, they issued a minor update instead.
There are now fixes for the SSL behavior. Other than that, gevent simply works. Why should it change every month?
And that's the problem. Python developers shouldn't expect people to scramble to do substantial code rewrites for patchlevel releases.
Python 3 adoption
To cut a long thread short
A 'point release' would be Python 2.8, they issued a minor update instead.