Python 3 adoption
Python 3 adoption
Posted Apr 14, 2015 23:44 UTC (Tue) by Cyberax (✭ supporter ✭, #52523)Parent article: Python 3 adoption
Unfortunately, that's not true. The most recent update (2.7.10) broke a lot of SSL infrastructure by porting all the SSL shit from Python 3.3 instead of doing a couple of small changes.
Posted Apr 15, 2015 15:40 UTC (Wed)
by fandingo (guest, #67019)
[Link] (30 responses)
The security situation was dire, and they had to do something. Furthermore, the three SSL/HTTPS changes were all breaking changes. As a developer, I'd rather my code breaks once for three reasons than have to deal with three separate interruptions.
Posted Apr 15, 2015 20:40 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link] (29 responses)
The next best option would have been to add SSL validation as a backwards-compatible smallish fix. But they didn't do that - instead Python developers CHANGED THE WHOLE FREAKING SSL STACK. So it broke lots of applications that actually _do_ _the_ _cetificate_ _validation_.
I had to spend several 20-hour workdays to track down and fix all the incompatibilities in our internal apps after a routine 'apt-get upgrade' broke everything. And I still have no idea what's wrong with Boto+gevent combination - I simply switched them to unsecured HTTP.
I wonder with what part of "stable branch" and "we don't break userspace" the developers of Python have difficulties?
Fuck them. I'm not starting any new projects in Python. I'm now back to Java for server-side and Golang for small client utilities.
Posted Apr 15, 2015 22:30 UTC (Wed)
by fandingo (guest, #67019)
[Link] (28 responses)
Really? When the issue was discussed, it was more like many people thought cert validation was occurring because that's a really important part of SSL. It's also worth pointing that you do seem to care about cert validation -- enough to even implement it in your applications.
> I had to spend several 20-hour workdays to track down and fix all the incompatibilities in our internal apps after a routine 'apt-get upgrade' broke everything.
You (and/or your coworkers) have no one to blame but yourself. 1) Didn't pay attention to upstream; yet, you use their updated code. 2) Didn't pay attention to your distro update policy or read the update notes. 3) Apparently, apply upgrades to production that cause major problems without even basic testing.
PEP 466 was published on March 23, 2014 and approved on April 19, 2014. The implications were extremely clear. Python 2.7.9 was released on December 10, 2014 (235 days later). There was plenty of time to make the minor changes necessary, and they really are minor.
The best part is that I went to see if LWN had written an article about the ssl issue. Not only had they on September 10 (https://lwn.net/Articles/611243/), but also, you were the first comment complaining about breaking compatibility! Yet, poor Cyberax wants to pretend he was caught unawares by those mean old Python devs.
The other interesting part is that no Ubuntu LTS release has 2.7.9, and in Debian, Wheezy doesn't either. I know there's other apt-based distros, but it certainly doesn't sound like you're running anything resembling "enterprise."
Lastly, "several 20-hour workdays?!" You can't possibly be serious. If the problem is so severe, that it not only takes that long to solve, but it's that important to solve ASAP, I think downgrading to 2.7.8 would've been my solution after maybe 3 hours. Or perhaps, just replacing ssl.py with the previous version.
> And I still have no idea what's wrong with Boto+gevent combination - I simply switched them to unsecured HTTP.
Everything I could find about boto and/or gevent problems with ssl on Python 2.7.9 indicates that there's been a fix available since September 21, which was months before 2.7.9 was actually released.
> I wonder with what part of "stable branch" and "we don't break userspace" the developers of Python have difficulties?
Perhaps the fact that they don't absolutely subscribe to either belief. There's plenty of discussion in PEP 466 and 476 about why this unusual change was made.
Posted Apr 15, 2015 23:02 UTC (Wed)
by dlang (guest, #313)
[Link] (6 responses)
The third item is the only one that is reasonably their fault. The first two are the reason for using distros in the first place. Nobody is able to follow every upstream to the level of detail needed to catch this ahead of time.
as to the third, you make the assertion that they did the upgrades "without even basic testing". You probably have a different definition of "basic testing" that I do, but I don't find it possible to test every single function of every single tool for every single upgrade.
Trying to do that leads to paralysis.
Posted Apr 16, 2015 0:12 UTC (Thu)
by fandingo (guest, #67019)
[Link] (5 responses)
Cyberax specifically knew about this 3 entire months before it was published! This isn't even remotely a tale about someone not knowing this change was coming. He obviously read and commented on an article that said the version number and release schedule. It's kind of a different situation when there is documented proof that he did know about this change.
Furthermore, he says they were using apt-get, and that almost certainly means Ubuntu or Debian. None of the Ubuntu LTS releases have anything newer than 2.7.5. Debian testing and unstable do, but Testing only got it on March 16th. The point being that there aren't too many places one could've picked up python 2.7.9 with apt-get.
I've worked at places that use Gentoo, Fedora, contemporary EL, or the most ancient EL distros imaginable. Different strokes for different folks, and I think that they all have their places. But, if you want to use something that changes quicker, you *have* to pay closer attention.
> You probably have a different definition of "basic testing" that I do, but I don't find it possible to test every single function of every single tool for every single upgrade.
Automated testing, TDD? I thought that was common place a decade ago. I can't imagine deploying code where every method and class aren't testable, especially for something like interfacing with AWS. This problem would've been caught by any Boto interaction with AWS. Any!
Furthermore, the mention of Boto and the extensive amount of work to fix the problem indicate that this functionality was critical to the application, which means it should've been covered.
The other point worth mentioning is that there is an issue on the Gevent Github page that is easily searchable, and there was a very easy workaround on September 21. It also works for Boto.
Basically, Cyberax's story is complete nonsense. He just wanted to bitch about Python, yet again, so he concocted a story. The dates don't make any sense. There was already a workaround nearly 3 months before the Python release for the specific libraries he mentions. Then, the distro both doesn't make sense (Debian Testing/Unstable or Ubuntu 14.10 in an enterprise that does blind updates?), and the timings are crazy (Python 2.7.5 released in Debian Testing and Ubuntu 14.10 on March 16, 2015).
Maybe there was a problem with their application, maybe they are running a newer Apt-based distro, and maybe the update slipped past them. Okay. The parts that seem totally unreasonable are 1) that he knew about this change in September by commenting on an LWN article about the compatibility break (and complained about it) and 2) that he would spend "several 20-hour days" trying to fix this problem and not type in "boto python 2.7.9 ssl" in Google and find the same solution that I did, which was posted in September. Hell, that solution is damn simple anyways, and any programmer should be able to recreate a missing method. Or, you know, just downgrade the Python package.
Getting back to testing, don't you have multiple environments? I don't know what his application does, but it seems difficult to imagine that if any sort of staggered upgrades across environments were being used that, at minimum, one of the developers would notice that every call to AWS was failing spectacularly.
Posted Apr 16, 2015 3:41 UTC (Thu)
by dlang (guest, #313)
[Link] (4 responses)
what you are missing is that they didn't deploy their code. they applied system patches.
Every month when windows patches are applied in your network, does every function of every application (including ones that you didn't write) get tested before being deployed to anything important?
If so, you are a 1%er (if not more rare)
Posted Apr 16, 2015 6:03 UTC (Thu)
by fandingo (guest, #67019)
[Link]
First, that's not my job, and second, I don't use Windows at work, so I don't have the first clue. Additionally, I wouldn't really consider workstation updates anywhere near critical, but perhaps I simply don't work at a company with enough employees to fill a stadium. (I don't see how this relates to the issue at hand anyways.)
I feel like I'm in crazy town talking to you. Do you not promote system updates to your dev, test, and integration environments before pushing them to production. This isn't some sort of sneaky problem that only triggers itself under rare circumstances. If boto is unpatched and talks to AWS in any manner, it's error city. Cursory test coverage would hit it, and developers just doing their normal work would uncover it in no time.
You seem to want to turn this into some sort of hypothetical or abstract discussion. That's not the situation. It's a very specific set of circumstances that affect a defined set of libraries (boto and gevent -- really just gevent). I don't really care about talking about abstract stuff when there's already a concrete discussion to have.
The facts are that this error would be apparent to any developer or system administrator who has segregated environments and staggers updates through them because the errors would occur so often and obviously that they could not be overlooked. It was either sloppy engineering that allowed this patch to be deployed to production without testing or lazy management who didn't realize they were dependent on dead code (gevent) and didn't bother to follow changes that might affect that dead library.
Posted Apr 17, 2015 9:52 UTC (Fri)
by asaz989 (guest, #67798)
[Link] (2 responses)
Posted Apr 17, 2015 23:58 UTC (Fri)
by zlynx (guest, #2285)
[Link] (1 responses)
I especially hate mock objects, because now the code is being tested against a pretend version of the real thing and the mock probably has bugs.
So without integration tests on real libraries, services and hardware, the code has not really been tested at all.
Posted Apr 18, 2015 7:25 UTC (Sat)
by cortana (subscriber, #24596)
[Link]
Posted Apr 16, 2015 2:10 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link] (19 responses)
> You (and/or your coworkers) have no one to blame but yourself. 1) Didn't pay attention to upstream; yet, you use their updated code.
I've actually read an article on LWN about Python turning on validation by default. And this is not a problem in itself, boto has always used certificate storage and validated the Amazon AWS certs. We even used it in a couple of important pieces.
> 2) Didn't pay attention to your distro update policy or read the update notes.
Somehow GCC or Golang developers manage not to break the world and when they DO need to do it, they release them in a major update.
> 3) Apparently, apply upgrades to production that cause major problems without even basic testing.
Guess what happens when they break?
> Everything I could find about boto and/or gevent problems with ssl on Python 2.7.9 indicates that there's been a fix available since September 21, which was months before 2.7.9 was actually released.
> The other interesting part is that no Ubuntu LTS release has 2.7.9, and in Debian, Wheezy doesn't either. I know there's other apt-based distros, but it certainly doesn't sound like you're running anything resembling "enterprise."
Lots of other people have been burned by it. For example: https://bugs.launchpad.net/ubuntu/+source/python-eventlet...
> Lastly, "several 20-hour workdays?!" You can't possibly be serious.
Let me repeat - the problem is that Python developers went and _broke_ _existing_ _code_ _that_ _was_ _doing_ _the_ _right_ _thing_. That was totally avoidable.
In short, the whole Python ecosystem is screwed up. They can't be trusted with producing a reliable base, so I'm not going to use them anymore for new projects.
Posted Apr 16, 2015 5:49 UTC (Thu)
by fandingo (guest, #67019)
[Link] (18 responses)
Haven't you learned enough about the Linux kernel over the past few years (you ever quoted some Linux-isms in your first reply) to realize that version numbers don't mean *anything*? Perhaps you can explain 2.6.39->3.x->3.19->4.0; based on your thought process, that's two major compatibility breaks, and at least one brain aneurysm when dropping a minor number.
The comparison to GCC is specious because they only implement a base language. There's no stdlib. Golang is also a bad example because it's a whopping 8 years old -- unsurprisingly they haven't gone through these long-term issues and have greatly benefitted from witnessing the past.
You're so hung up on version numbers, but this problem would've happened eventually even if it meant going to 2.8. You had plenty of warning, the library authors had plenty of warning; yet, you both did nothing. A version number isn't going to change that unpreparedness.
> I'm serious. I had to do emergency bugfixing and testing because our customers' machines suddenly stopped working. Even finding the issue itself took some time - it's only indirectly seen in some libraries (like boto) as failing SSL requests without any stacktraces pointing out the problem.
So you say this is an ongoing problem, even though that bug report has more people saying that their issue was fixed than are still complaining. The question becomes where's your published fix? Surely you'd still be working 20-hour days if you can't point to a fix.
This seems more like a tale about lazy, uncaring, or incompetent library authors (boto, gevent, and you) that can't seem to resolve issues that were clear a year ago. In the case of gevent, that seems to make sense: Gevent appears to be a dead project and hasn't had any activity in over a year. Perhaps you and boto should either stop using a dead project, freeze every dependency (includign cpython), start maintaining gevent yourselves, or move to something else.
Actually, when you look back on it, this is entirely the fault of people using unmaintained code that they're somehow expecting to continue to work perpetually. There has been exactly one commit to Gevent since 2.7.9 was released; it was 12/13, three days after and unrelated. No activity on the entire project in 2015. Let's say that the Python developers put this in a hypothetical 2.8, what difference does it make? The Gevent project is still dead; boto and your product aren't going to work with this Python 2.8 either, and the next round of forward leaning distros will pick up the newest stable version of Python anyways. What's the difference if nobody is maintaining Gevent regardless of Python version number?
> Let me repeat - the problem is that Python developers went and _broke_ _existing_ _code_ _that_ _was_ _doing_ _the_ _right_ _thing_. That was totally avoidable.
It was not. It was a terrible mistake all those years ago to forgo default certificate validation by `ssl`. There's nothing that can change that. The situation last year was to either continue to put our thumbs up our asses, even though we knew it was leading to bad behavior and confused users, or bite the bullet and fix things. I'll heap blame on them for their original decision, but dramatically increasing security in a sound, responsible way must be applauded. You and your dependent library authors could've started testing over a year ago, but didn't for whatever reason.
> In short, the whole Python ecosystem is screwed up. They can't be trusted with producing a reliable base, so I'm not going to use them anymore for new projects.
Honestly, just leave and stop whining. Oh no, the code that hasn't been updated in ages stops working when the rest of the environment is updated -- that's not in any way surprising! However, it doesn't sound like you have much choice. You're making libraries and products for customers that want to use Python, so, yeah...
Posted Apr 16, 2015 6:54 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link] (13 responses)
Linux has a stable userspace ABI. I can take a static binary compiled 15 years ago and it will _still_ _work_ on the newest kernels. That's why kernel release version doesn't matter.
> You're so hung up on version numbers, but this problem would've happened eventually even if it meant going to 2.8. You had plenty of warning, the library authors had plenty of warning; yet, you both did nothing. A version number isn't going to change that unpreparedness.
It's a problem of a minor Python update _breaking_ _user_ _code_. On purpose: https://bugs.python.org/issue21308
Want to try something similar with Linux? Simply submit a change that removes readdir() syscall and replaces it with a better designed interface. In a -rc6 release. And then say something like: "But it's to fix a terrible mistake from years ago!"
But please, give me a heads up a couple of hours before your submit your patches. I want to have enough time to grab popcorn and enjoy the show.
> So you say this is an ongoing problem, even though that bug report has more people saying that their issue was fixed than are still complaining. The question becomes where's your published fix? Surely you'd still be working 20-hour days if you can't point to a fix.
I'm definitely going to find some time to submit all the changes upstream.
> The Gevent project is still dead; boto and your product aren't going to work with this Python 2.8 either, and the next round of forward leaning distros will pick up the newest stable version of Python anyways. What's the difference if nobody is maintaining Gevent regardless of Python version number?
> It was not. It was a terrible mistake all those years ago to forgo default certificate validation by `ssl`.
This change could have been incremental - a couple of optional parameters to pass the CA store parameters and perhaps a global function to override the default behavior. Such changes would have preserved the existing software that does its own validation _and_ they would have provided secure defaults for everybody else.
However, Python developers instead backported SSL layer from Python3, knowing perfectly well that it's going to break stuff. Let me quote them:
> > I doubt adding a ton of new APIs and code can be uneventful, but good
Or maybe:
> I spent hours looking at this patch, which certainly doesn't constitute a real review, but is probably about as good as your going to get on this behemouth.
So yes, I maintain that the developers are complete idiots.
> Honestly, just leave and stop whining. Oh no, the code that hasn't been updated in ages stops working when the rest of the environment is updated -- that's not in any way surprising!
Posted Apr 16, 2015 7:54 UTC (Thu)
by daniels (subscriber, #16193)
[Link] (4 responses)
Sigh. Just stop here.
Even if your argument wasn't ridiculous (you're smarter than Guido and the rest of the Python team? ok, sure), quite what gives you the right to wander around abusing people because you disagree with their decisions is absolutely beyond me.
Grow up and accept that reasonable people can arrive at different conclusions without one of them being idiots. Your abuse contributes nothing whatsoever to the conversation, except a growing wish for comment moderation.
Posted Apr 16, 2015 8:04 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link] (3 responses)
Python 2.7 maintainers are STILL complete idiots since they go forward with obviously breaking changes for a minor release in a stable branch. This is not a case of a bug that crawled through the cracks in a patch, it was a deliberate breakage.
Posted Apr 16, 2015 8:14 UTC (Thu)
by corbet (editor, #1)
[Link] (2 responses)
Posted Apr 16, 2015 8:23 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
It was perfectly possible to write a small patch to add validation by default and allow to pass a custom CA-store, without breaking any existing code. It would have required a couple of dirty hacks inside the library and possibly some code duplication to add support for SNI, but it was entirely doable.
Python maintainers instead backported the whole SSL infrastructure from Python 3 which has a lot of changes inside of it. Here's the patch: https://bugs.python.org/file36423/ssl-backport.diff - it's almost 13000 lines long.
Posted Apr 16, 2015 8:31 UTC (Thu)
by daniels (subscriber, #16193)
[Link]
Posted Apr 16, 2015 15:06 UTC (Thu)
by fandingo (guest, #67019)
[Link] (7 responses)
> Actually, you're right. Here's a raw diff for docker-py: http://pastebin.com/3JpveFxu We used a dirty workaround for Boto which I'm not going to submit. Gevent patches were sent to its author. I have a patch for websocket-client, will submit it later.
So, umm, thanks for linking to that monstrosity of a diff. I read through it twice, and honestly, I don't see anything that would be relevant to incompatibilities introduced in 2.7.9. This more looks like a diff from a series of changes that included stuff super-related to the SSL problem -- like updating from Docker 1.12 to 1.18. Perhaps you can identify some areas that handle the SSL change.
If there were any SSL changes, they must be in <<unamed_class>>._post_json() or <<unamed_class>>._post() or in a different file.
I don't see any merge requests for gevent that could be possibly from you. Of course, that doesn't change the underlying problem that gevent is unmaintained.
I don't see that websocket-client ever had any incompatibilities. No open or closed issues, no mention in the changelog.
It all seems like subterfuge.
Getting back to the minor/major change. I originally thought your catastrophe originated from updating Python on your company's infrastructure. However, it seems that the problem was your customers running your code on their infrastructure. Since you weren't willing to temporarily say "we don't support 2.7.9" and barf out an error message until you have a proper workaround, putting this change (along with the whole boatload of stuff that will come) in a 2.8 doesn't solve *any* problem. 1) Gevent is still unmaintained and will break. 2) Ubuntu, Fedora, Arch, Debian Sid, etc. -- the ones who actually have 2.7.9 -- would upgrade to 2.8 in their next distro release, and it's going to affect all the same users.
Walk me through the situation where this change in a hypothetical 2.8 doesn't blow up in your face the same way.
> I have code written for Java 1.3 that was released (wow!) 17 years ago. It still works perfectly on the most recent JDKs. Why should I rewrite the code that does what it should?
The Lady of the Lake,
I dunno. Languages are made by different people for different purposes. Some of those include rigid compatibility. Python was obviously made to ruin your life, but hey, at least its successful at what it set out to do.
Posted Apr 16, 2015 18:10 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link] (6 responses)
Working on it...
> Getting back to the minor/major change. I originally thought your catastrophe originated from updating Python on your company's infrastructure. However, it seems that the problem was your customers running your code on their infrastructure. Since you weren't willing to temporarily say "we don't support 2.7.9" and barf out an error message until you have a proper workaround
> Walk me through the situation where this change in a hypothetical 2.8 doesn't blow up in your face the same way.
You know, the way major Python upgrades have always worked.
Posted Apr 16, 2015 19:07 UTC (Thu)
by fandingo (guest, #67019)
[Link] (5 responses)
> Working on it...
So, what was the point of posting that diff? You assumed that I wouldn't read it? It's not sounding like "several 20-hour days" worth of work anymore.
I am also still curious about the timeline when this blew up because it doesn't make sense. I don't see Debian or Ubuntu backporting these SSL changes. Outside of Debian Sid and Ubuntu 14.10, I'm struggling to even find an apt-based distro that actually has 2.7.9. Even Sid and 14.10 only got 2.7.9 1 month ago.
> Easy. A new major version is released that has its own Python symlink and can live alongside Python 2.7. Old code can still use Python 2.7 explicitly, especially if it has some custom SSL processing.
Sure that's possible. Which apt-based distro are you using that actually does this? Ubuntu and Debian only package one Python 2 version at a time, so you're talking about custom packaging. Then, of course, there's nothing stopping you from creating a 2.7.8 package and symlinking to your heart's content. Plus, your users are going to be using whatever the default is, and I can't see any situation where a hypothetical 2.8 doesn't become the default immediately in the next distro release.
> we were not willing to hold all updates for this.
Just the hypothetical ones that do things exactly like you want to avoid.
It seems more like you have an ideological axe to grind, and you're making up or exaggerating a minor situation to fit your ideology. You're not providing the technical info to substantiate how devastating this change was and are more interested in talking abstractly about major/minor version numbers and holy compatibility.
Posted Apr 16, 2015 22:56 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link] (4 responses)
> I am also still curious about the timeline when this blew up because it doesn't make sense. I don't see Debian or Ubuntu backporting these SSL changes. Outside of Debian Sid and Ubuntu 14.10, I'm struggling to even find an apt-based distro that actually has 2.7.9. Even Sid and 14.10 only got 2.7.9 1 month ago.
Please note, that it's NOT Python 2.7.9, it's a backport to Python 2.7.8.
We started seeing it in the wild right after the New Year. I can try to get precise details from their version control repo.
> Sure that's possible. Which apt-based distro are you using that actually does this? Ubuntu and Debian only package one Python 2 version at a time, so you're talking about custom packaging.
And then there's CentOS 6 which is still packaged with Python 2.6 by default.
Incidentally, I don't see hordes torch-wielding of CentOS users demanding SSL fixes to be backported into Python 2.6
> Plus, your users are going to be using whatever the default is, and I can't see any situation where a hypothetical 2.8 doesn't become the default immediately in the next distro release.
> Just the hypothetical ones that do things exactly like you want to avoid.
Posted Apr 18, 2015 1:39 UTC (Sat)
by fandingo (guest, #67019)
[Link] (3 responses)
In that light, I can definitely understand why you'd be angry about breaking changes pulled into an earlier version within a distro version. Yet, I don't see what it has to do with the Python developers. The Python developers made a new point release, and spent a considerable amount of time discussing and warning about the change.
> That's fine. A transition period of 2-3 years (one release cycle) would have given everyone time to adapt or at least to explicitly set the Python version in scripts.
Is there any indication that the state of Gevent is actually going to change? There have been a couple of pull requests before the actual release of 2.7.9 to fix this behavior, and the author(s) haven't done anything with them. It seems like the only hope would be that Gevent is forked, and people start getting the fork through their distro or pypi.
All of that being said, the fix in Gevent was simple, and if that library was maintained, the library would've been updated 3 months before 2.7.9 was released, which was when the first patch and merge request were posted.
Posted Apr 18, 2015 6:39 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
> The Python developers made a new point release, and spent a considerable amount of time discussing and warning about the change.
> Is there any indication that the state of Gevent is actually going to change? There have been a couple of pull requests before the actual release of 2.7.9 to fix this behavior, and the author(s) haven't done anything with them.
> All of that being said, the fix in Gevent was simple, and if that library was maintained, the library would've been updated 3 months before 2.7.9 was released, which was when the first patch and merge request were posted.
Posted Apr 18, 2015 7:28 UTC (Sat)
by fandingo (guest, #67019)
[Link]
If by "safe" you mean without compatibility issues, then you're terribly, terribly wrong. You obviously didn't read the release notes or PEP 466, which are linked in the notes. I'm in disbelief that you could possibly make this comment after all that we've discussed.
> The same thing happened with SuSE and Amazon Linux.
SUSE is on Python 2.6, so no, it didn't. I still don't understand why more entities doing the wrong thing absolves them of fault. The ssl changes were 2.7.9; what possible situation exists for taking all of the changes and putting them in something called the previous version. That's madness.
> I think only RedHat/CentOS refrained from doing it.
And Fedora.
You're really not going to budge an inch that those distros messed up by putting this serious compatibility break that you bemoan in a package that they call 2.7.8 are you? Wow, that really makes it difficult to take you seriously.
> There are now fixes for the SSL behavior.
They've been there since September 21, 3 months before the release of 2.7.9. But there weren't any maintainers to merge the simple change, so no update was available. That's the part that you seem so unwilling to admit: The necessary packages were simple, and identified long before users were impacted, but because there isn't any active development of Gevent, the work never got published.
> Other than that, gevent simply works. Why should it change every month?
Not with the current 2.7 release, so yeah, it does need to change. Considering that there's 46 open merge requests for a litany of issues, I'd say that users would like to see quite a bit of change. I don't see why you're so happy and content with using unmaintained code. Perhaps it will take some remote execution vulnerability to change your mind.
> And that's the problem. Python developers shouldn't expect people to scramble to do substantial code rewrites for patchlevel releases.
In what world is a half-dozen lines of code 3 months in advance scrambling? This is such complete and utter nonsense.
Posted Apr 19, 2015 21:12 UTC (Sun)
by pboddie (guest, #50784)
[Link]
Most people won't read this far into this thread, but that's the real issue here: there's a policy decision not to release another minor version of Python 2.x, so that not only will the core developers not support a Python 2.8 release themselves, they will also deny others the opportunity to make such a version with that label; this is why Stackless Python was renamed to just Stackless. Personally, I find this aversion to even a single subsequent minor version (2.8 in this case), in a series that isn't favoured by its principal developers, to be counterproductive. Such a version is an ideal place to put all the backwards-incompatible fixes that "finish the job", instead of pretending that some other "final" minor version (2.7 in this case) wasn't broken all along. It also won't confuse distribution packagers who risk breaking things because they didn't follow or can't second-guess every last policy decision of the core developers, see a "patch release" and dish it out believing that the usual conservatism has been upheld. In short, changing the rules and making late updates to 2.7.x into something they shouldn't have been is what has caused this problem. Conceding that 2.8 should have existed for such purposes, perhaps explicitly indicating a change to the rules for that version, would have been the correct decision.
Posted Apr 16, 2015 9:52 UTC (Thu)
by NAR (subscriber, #1313)
[Link] (2 responses)
That's the problem with Linux on the desktop. On Windows I can expect a 10 years old binary (effectively unmaintained code) working on a current version, on Linux I can't expect the same code working during the 6-12 months lifecycle of a distribution! And many people think it's fine.
The ideologically correct solution would be to package all software to all distributions and let distributions maintain it, but it's obviously ridiculous (the distributions are already underpowered, have different release cycles, can't handle proprietary code, getting software into distribution could be complicated, etc). The technical solution is to bundle everything that can't be trusted to a distribution (practically everything up from glibc) and distribute that. With half dozen dependencies we've seriously looked at releasing a vmware image instead of "source tarball and instructions on getting the right versions of the right software". That's where containers and Lennart's latest proposal and other stuff come into play. Where distributions failed.
Posted Apr 16, 2015 12:40 UTC (Thu)
by lgeorget (guest, #99972)
[Link] (1 responses)
Actually, you run into some problems and have to use the "compatibility parameters" (and cross fingers) if you want to run a 10-years-old software under Windows. And if it does not run into dependencies problems, that's because every software will usually come with all the DLLs it needs when you download it from Internet. You can easily have ten times the same "shared library" scattered around deep inside C:\Programs under Windows. So, I don't think you can actually compare Windows and Linux on that point.
As an Computer Science PhD student, I sometimes need to make old unmaintained code to run (such as prototypes, or obscure libraries) and I usually succeed. I admit it can be a bit cumbersome to fetch and compile by hand a lot of libraries to meet the dependency of a software but you can do it.
Posted Apr 16, 2015 14:47 UTC (Thu)
by NAR (subscriber, #1313)
[Link]
So what? C:\Program Files* uses 13GB out of the 500 GB disk on the computer I'm typing this. That's less than 3%. Should I care? I don't think so. Again, the difference is that on Windows you can expect that 10 years old binary will work, on Linux you shouldn't expect that unmaintained code will work in a year.
Computer Science PhD student [...] I usually succeed
I think it would be a shame if you don't :-) But it is one thing to fetch and compile dependencies for an academic project and a completely other thing is to figure out why some software broke after an apt-get upgrade on a live system...
Posted Apr 20, 2015 14:21 UTC (Mon)
by jwakely (subscriber, #60262)
[Link]
GCC implements several languages and also the stdlib for some of them.
Posted Apr 20, 2015 11:45 UTC (Mon)
by niner (subscriber, #26151)
[Link]
This is exactly the kind of attitude that people find off-putting. People who have this attitude have no one to blame but themselves, when their user base moves somewhere else. Others would probably have started to look for faults in their own action when 7 years after the initial release, their new major version is still almost invisible in their user base.
Python 3 adoption
Python 3 adoption
Python 3 adoption
Python 3 adoption
Python 3 adoption
Python 3 adoption
Python 3 adoption
Python 3 adoption
Python 3 adoption
Python 3 adoption
Python 3 adoption
Really. Our apps are actually just OK with plain HTTP, since they are used over trusted networks or over VPNs. Important pieces even use custom security protocols.
I don't follow all the minutae of mailing lists. And this particular bug was not even known until people started encountering it.
See above. It was a MINOR UPDATE. Not a Python 2.8, but a routine minor update.
Our customers did. We provide tools that our clients install on their own systems.
No. There's still no fix: https://github.com/gevent/gevent/issues/477
Python 2.7 in Ubuntu 14.04 with the recent updates has it. And _of_ _course_, Python 2.7 in Ubuntu 12.04 does not have it (even with the most recent patches).
I'm serious. I had to do emergency bugfixing and testing because our customers' machines suddenly stopped working. Even finding the issue itself took some time - it's only indirectly seen in some libraries (like boto) as failing SSL requests without any stacktraces pointing out the problem.
Python 3 adoption
Python 3 adoption
WTF? It's such a braindead statement, that I'm out of hands for a facepalm.
It's NOT A PROBLEM OF VERSION NUMBERS.
Actually, you're right. Here's a raw diff for docker-py: http://pastebin.com/3JpveFxu We used a dirty workaround for Boto which I'm not going to submit. Gevent patches were sent to its author. I have a patch for websocket-client, will submit it later.
Python 2.7 is going to be supported for the next 6 years. That was acceptable for us earlier. Now we're looking for an alternative with saner maintainers.
Again, let me repeat. Python changes broke the code that was explicitly doing the right thing.
> luck :)
> They don't call it "Rawhide" for nothing! :)"
I have code written for Java 1.3 that was released (wow!) 17 years ago. It still works perfectly on the most recent JDKs. Why should I rewrite the code that does what it should?
Python 3 adoption
Python 3 adoption
...and without it somebody else would have called them complete idiots for leaving an apparently insecure situation in place. They were in a difficult spot and made the best decision they could. You may not agree with it — I'm not sure I agree with it — but calling them idiots does seem to be over the top. We should be able to treat each other better than that. I have no problem with expressions of disagreement here, but I'd really rather not see that kind of personal attack, please?
Python 3 adoption
Python 3 adoption
It was NOT a binary decision.
Python 3 adoption
Python 3 adoption
[angels sing]
her arm clad in the purest shimmering samite, held aloft ssl3
from the bosom of the water signifying by Divine Providence that you,
Cyberax, Python developer, must rewrite your code.
[singing stops]
Python 3 adoption
Mostly a 'send' method that I had to pull up into ssladapter from the default HTTP handler. A couple of other changes as well.
It was backported into 2.7.8. And yes, we were not willing to hold all updates for this.
Easy. A new major version is released that has its own Python symlink and can live alongside Python 2.7. Old code can still use Python 2.7 explicitly, especially if it has some custom SSL processing.
Python 3 adoption
Python 3 adoption
This diff alone isn't.
The Ubuntu 14.04 backport was committed sometime in December. You can see the offending code in the Ubuntu patch: http://archive.ubuntu.com/ubuntu/pool/main/p/python2.7/py... - grep for "_sslwrap".
Ubuntu did this during 2.6 -> 2.7 migration. You could install both versions alongside and either use an explicit '/usr/bin/python26' or use DPKG alternatives mechanism to select an interpreter.
That's fine. A transition period of 2-3 years (one release cycle) would have given everyone time to adapt or at least to explicitly set the Python version in scripts.
The problem was very real - people downloading default Ubuntu machine images on Amazon EC2 suddenly stopped being able to use our software.
Python 3 adoption
Python 3 adoption
The same thing happened with SuSE and Amazon Linux. They all have backported a patch that Python developers insisted was safe and essential. I think only RedHat/CentOS refrained from doing it.
A 'point release' would be Python 2.8, they issued a minor update instead.
There are now fixes for the SSL behavior. Other than that, gevent simply works. Why should it change every month?
And that's the problem. Python developers shouldn't expect people to scramble to do substantial code rewrites for patchlevel releases.
Python 3 adoption
To cut a long thread short
A 'point release' would be Python 2.8, they issued a minor update instead.
Actually, when you look back on it, this is entirely the fault of people using unmaintained code that they're somehow expecting to continue to work perpetually.
Python 3 adoption
Python 3 adoption
You can easily have ten times the same "shared library" scattered around deep inside C:\Programs under Windows
Python 3 adoption
Python 3 adoption
Python 3 adoption