LWN: Comments on "Python 3 adoption"

pdb vs. VisualVM

richardfearn — Fri, 24 Apr 2015 16:42:33 +0000

> The Python debugger (pdb) looks "pretty sad compared to VisualVM".

That's an odd comparison. Those two things are fairly different. VisualVM is a monitoring/profiling tool, not a debugger.

It's a nice tool, though, and it comes included with the JDK.

Python 3 Is A Big Improvement

ldo — Tue, 21 Apr 2015 00:14:48 +0000

Frankly, I don’t care what the Mercurial developers or anybody else thinks about Python 3. It just makes me glad I don’t need to use Mercurial (git-remote-hg FTW!).

As far as I’m concerned, Python 3 is a wide-open field right now, with plenty of room for pioneers to stake out a claim. So the PyMTP folks can’t be bothered porting their library to Python 3? So I created my own, and made it more Pythonic while I was at it. Pycairo was ported, but then abandoned? That’s why I created Qahirah, which offers a more complete coverage of Cairo functionality than Pycairo could manage. And so on. And so on.

Python 3 adoption

jwakely — Mon, 20 Apr 2015 14:21:22 +0000

> The comparison to GCC is specious because they only implement a base language. There's no stdlib.

GCC implements several languages and also the stdlib for some of them.

Python 3 adoption

m4rtink — Mon, 20 Apr 2015 12:48:19 +0000

> python wedged itself into key system components in most major distros, so there was always going to be part of the community that was risk averse and willing to hold on to mature code. who wants to take a risk migrating code for an OS installer that works? etc etc

Well, we are currently doing just that[0] - porting the Anaconda installer to Python 3. :)

Overall I think the switch to Python 3 is definitely worth it, as Python 3 just behaves in a more cleaner and correct way. We found lots of of places where the code was actually depending on undefined or undocumented behavior that Python 2 didn't care about, but which is an error in Python 3.

Thanks to this (and also the changes needed for bytes/strings separation) a lot of the Anaconda code will be much cleaner and more robust once the Python 3 port is merged back to master.

[0] https://github.com/M4rtinK/anaconda/tree/python3

Python 3 adoption

niner — Mon, 20 Apr 2015 11:45:28 +0000

"You (and/or your coworkers) have no one to blame but yourself. 1) Didn't pay attention to upstream; yet, you use their updated code. 2) Didn't pay attention to your distro update policy or read the update notes. 3) Apparently, apply upgrades to production that cause major problems without even basic testing."

This is exactly the kind of attitude that people find off-putting. People who have this attitude have no one to blame but themselves, when their user base moves somewhere else. Others would probably have started to look for faults in their own action when 7 years after the initial release, their new major version is still almost invisible in their user base.

To cut a long thread short

pboddie — Sun, 19 Apr 2015 21:12:56 +0000

A 'point release' would be Python 2.8, they issued a minor update instead.

Most people won't read this far into this thread, but that's the real issue here: there's a policy decision not to release another minor version of Python 2.x, so that not only will the core developers not support a Python 2.8 release themselves, they will also deny others the opportunity to make such a version with that label; this is why Stackless Python was renamed to just Stackless.

Personally, I find this aversion to even a single subsequent minor version (2.8 in this case), in a series that isn't favoured by its principal developers, to be counterproductive. Such a version is an ideal place to put all the backwards-incompatible fixes that "finish the job", instead of pretending that some other "final" minor version (2.7 in this case) wasn't broken all along. It also won't confuse distribution packagers who risk breaking things because they didn't follow or can't second-guess every last policy decision of the core developers, see a "patch release" and dish it out believing that the usual conservatism has been upheld.

In short, changing the rules and making late updates to 2.7.x into something they shouldn't have been is what has caused this problem. Conceding that 2.8 should have existed for such purposes, perhaps explicitly indicating a change to the rules for that version, would have been the correct decision.

Python 3 adoption

fandingo — Sat, 18 Apr 2015 07:28:05 +0000

> They all have backported a patch that Python developers insisted was safe and essential.

If by "safe" you mean without compatibility issues, then you're terribly, terribly wrong. You obviously didn't read the release notes or PEP 466, which are linked in the notes. I'm in disbelief that you could possibly make this comment after all that we've discussed.

> The same thing happened with SuSE and Amazon Linux.

SUSE is on Python 2.6, so no, it didn't. I still don't understand why more entities doing the wrong thing absolves them of fault. The ssl changes were 2.7.9; what possible situation exists for taking all of the changes and putting them in something called the previous version. That's madness.

> I think only RedHat/CentOS refrained from doing it.

And Fedora.

You're really not going to budge an inch that those distros messed up by putting this serious compatibility break that you bemoan in a package that they call 2.7.8 are you? Wow, that really makes it difficult to take you seriously.

> There are now fixes for the SSL behavior.

They've been there since September 21, 3 months before the release of 2.7.9. But there weren't any maintainers to merge the simple change, so no update was available. That's the part that you seem so unwilling to admit: The necessary packages were simple, and identified long before users were impacted, but because there isn't any active development of Gevent, the work never got published.

> Other than that, gevent simply works. Why should it change every month?

Not with the current 2.7 release, so yeah, it does need to change. Considering that there's 46 open merge requests for a litany of issues, I'd say that users would like to see quite a bit of change. I don't see why you're so happy and content with using unmaintained code. Perhaps it will take some remote execution vulnerability to change your mind.

> And that's the problem. Python developers shouldn't expect people to scramble to do substantial code rewrites for patchlevel releases.

In what world is a half-dozen lines of code 3 months in advance scrambling? This is such complete and utter nonsense.

Python 3 adoption

cortana — Sat, 18 Apr 2015 07:25:05 +0000

Why not both? Unit tests are still useful for testing how your code will react when a collaborator behaves in a way that is documented, but is difficult to reproduce in a test environment.

Python 3 adoption

Cyberax — Sat, 18 Apr 2015 06:39:17 +0000

> Why aren't you pointing the finger at whomever at Ubuntu/Debian that decided to do this?
The same thing happened with SuSE and Amazon Linux. They all have backported a patch that Python developers insisted was safe and essential. I think only RedHat/CentOS refrained from doing it.

> The Python developers made a new point release, and spent a considerable amount of time discussing and warning about the change.
A 'point release' would be Python 2.8, they issued a minor update instead.

> Is there any indication that the state of Gevent is actually going to change? There have been a couple of pull requests before the actual release of 2.7.9 to fix this behavior, and the author(s) haven't done anything with them.
There are now fixes for the SSL behavior. Other than that, gevent simply works. Why should it change every month?

> All of that being said, the fix in Gevent was simple, and if that library was maintained, the library would've been updated 3 months before 2.7.9 was released, which was when the first patch and merge request were posted.
And that's the problem. Python developers shouldn't expect people to scramble to do substantial code rewrites for patchlevel releases.

Python 3 adoption

fandingo — Sat, 18 Apr 2015 01:39:57 +0000

That sounds like a packaging error by the Ubuntu packagers or perhaps Debian -- I'm not sure whether Ubuntu totally forks from Debian Testing or largely pulls updates throughout an Ubuntu release's life -- rather than something that can be blamed on the Python developers. Why aren't you pointing the finger at whomever at Ubuntu/Debian that decided to do this?

In that light, I can definitely understand why you'd be angry about breaking changes pulled into an earlier version within a distro version. Yet, I don't see what it has to do with the Python developers. The Python developers made a new point release, and spent a considerable amount of time discussing and warning about the change.

> That's fine. A transition period of 2-3 years (one release cycle) would have given everyone time to adapt or at least to explicitly set the Python version in scripts.

Is there any indication that the state of Gevent is actually going to change? There have been a couple of pull requests before the actual release of 2.7.9 to fix this behavior, and the author(s) haven't done anything with them. It seems like the only hope would be that Gevent is forked, and people start getting the fork through their distro or pypi.

All of that being said, the fix in Gevent was simple, and if that library was maintained, the library would've been updated 3 months before 2.7.9 was released, which was when the first patch and merge request were posted.

Python 3 adoption

zlynx — Fri, 17 Apr 2015 23:58:47 +0000

This is why unit tests are good but not sufficient. They only show that code works the way that the testers think it does.

I especially hate mock objects, because now the code is being tested against a pretend version of the real thing and the mock probably has bugs.

So without integration tests on real libraries, services and hardware, the code has not really been tested at all.

Python 3 adoption

asaz989 — Fri, 17 Apr 2015 09:52:17 +0000

At my previous work, apt-get dist-upgrade was not something you just ran routinely - in fact, it is an Even Scarier Thing To Do than a code deploy, since often unit test environments don't test the base system. Hence, very very slow staged deployments.

Python 3 adoption

Cyberax — Thu, 16 Apr 2015 22:56:25 +0000

> So, what was the point of posting that diff? You assumed that I wouldn't read it? It's not sounding like "several 20-hour days" worth of work anymore.
This diff alone isn't.

> I am also still curious about the timeline when this blew up because it doesn't make sense. I don't see Debian or Ubuntu backporting these SSL changes. Outside of Debian Sid and Ubuntu 14.10, I'm struggling to even find an apt-based distro that actually has 2.7.9. Even Sid and 14.10 only got 2.7.9 1 month ago.
The Ubuntu 14.04 backport was committed sometime in December. You can see the offending code in the Ubuntu patch: http://archive.ubuntu.com/ubuntu/pool/main/p/python2.7/py... - grep for "_sslwrap".

Please note, that it's NOT Python 2.7.9, it's a backport to Python 2.7.8.

We started seeing it in the wild right after the New Year. I can try to get precise details from their version control repo.

> Sure that's possible. Which apt-based distro are you using that actually does this? Ubuntu and Debian only package one Python 2 version at a time, so you're talking about custom packaging.
Ubuntu did this during 2.6 -> 2.7 migration. You could install both versions alongside and either use an explicit '/usr/bin/python26' or use DPKG alternatives mechanism to select an interpreter.

And then there's CentOS 6 which is still packaged with Python 2.6 by default.

Incidentally, I don't see hordes torch-wielding of CentOS users demanding SSL fixes to be backported into Python 2.6

> Plus, your users are going to be using whatever the default is, and I can't see any situation where a hypothetical 2.8 doesn't become the default immediately in the next distro release.
That's fine. A transition period of 2-3 years (one release cycle) would have given everyone time to adapt or at least to explicitly set the Python version in scripts.

> Just the hypothetical ones that do things exactly like you want to avoid.
The problem was very real - people downloading default Ubuntu machine images on Amazon EC2 suddenly stopped being able to use our software.

Python 3 adoption

fandingo — Thu, 16 Apr 2015 19:07:24 +0000

> Mostly a 'send' method that I had to pull up into ssladapter from the default HTTP handler. A couple of other changes as well.

> Working on it...

So, what was the point of posting that diff? You assumed that I wouldn't read it? It's not sounding like "several 20-hour days" worth of work anymore.

I am also still curious about the timeline when this blew up because it doesn't make sense. I don't see Debian or Ubuntu backporting these SSL changes. Outside of Debian Sid and Ubuntu 14.10, I'm struggling to even find an apt-based distro that actually has 2.7.9. Even Sid and 14.10 only got 2.7.9 1 month ago.

> Easy. A new major version is released that has its own Python symlink and can live alongside Python 2.7. Old code can still use Python 2.7 explicitly, especially if it has some custom SSL processing.

Sure that's possible. Which apt-based distro are you using that actually does this? Ubuntu and Debian only package one Python 2 version at a time, so you're talking about custom packaging. Then, of course, there's nothing stopping you from creating a 2.7.8 package and symlinking to your heart's content. Plus, your users are going to be using whatever the default is, and I can't see any situation where a hypothetical 2.8 doesn't become the default immediately in the next distro release.

> we were not willing to hold all updates for this.

Just the hypothetical ones that do things exactly like you want to avoid.

It seems more like you have an ideological axe to grind, and you're making up or exaggerating a minor situation to fit your ideology. You're not providing the technical info to substantiate how devastating this change was and are more interested in talking abstractly about major/minor version numbers and holy compatibility.

Python 3 adoption

Cyberax — Thu, 16 Apr 2015 18:10:08 +0000

> Perhaps you can identify some areas that handle the SSL change.
Mostly a 'send' method that I had to pull up into ssladapter from the default HTTP handler. A couple of other changes as well.

Working on it...

> Getting back to the minor/major change. I originally thought your catastrophe originated from updating Python on your company's infrastructure. However, it seems that the problem was your customers running your code on their infrastructure. Since you weren't willing to temporarily say "we don't support 2.7.9" and barf out an error message until you have a proper workaround
It was backported into 2.7.8. And yes, we were not willing to hold all updates for this.

> Walk me through the situation where this change in a hypothetical 2.8 doesn't blow up in your face the same way.
Easy. A new major version is released that has its own Python symlink and can live alongside Python 2.7. Old code can still use Python 2.7 explicitly, especially if it has some custom SSL processing.

You know, the way major Python upgrades have always worked.

Python 3 adoption

fandingo — Thu, 16 Apr 2015 15:06:34 +0000

>> So you say this is an ongoing problem, even though that bug report has more people saying that their issue was fixed than are still complaining. The question becomes where's your published fix? Surely you'd still be working 20-hour days if you can't point to a fix.

> Actually, you're right. Here's a raw diff for docker-py: http://pastebin.com/3JpveFxu We used a dirty workaround for Boto which I'm not going to submit. Gevent patches were sent to its author. I have a patch for websocket-client, will submit it later.

So, umm, thanks for linking to that monstrosity of a diff. I read through it twice, and honestly, I don't see anything that would be relevant to incompatibilities introduced in 2.7.9. This more looks like a diff from a series of changes that included stuff super-related to the SSL problem -- like updating from Docker 1.12 to 1.18. Perhaps you can identify some areas that handle the SSL change.

If there were any SSL changes, they must be in <<unamed_class>>._post_json() or <<unamed_class>>._post() or in a different file.

I don't see any merge requests for gevent that could be possibly from you. Of course, that doesn't change the underlying problem that gevent is unmaintained.

I don't see that websocket-client ever had any incompatibilities. No open or closed issues, no mention in the changelog.

It all seems like subterfuge.

Getting back to the minor/major change. I originally thought your catastrophe originated from updating Python on your company's infrastructure. However, it seems that the problem was your customers running your code on their infrastructure. Since you weren't willing to temporarily say "we don't support 2.7.9" and barf out an error message until you have a proper workaround, putting this change (along with the whole boatload of stuff that will come) in a 2.8 doesn't solve *any* problem. 1) Gevent is still unmaintained and will break. 2) Ubuntu, Fedora, Arch, Debian Sid, etc. -- the ones who actually have 2.7.9 -- would upgrade to 2.8 in their next distro release, and it's going to affect all the same users.

Walk me through the situation where this change in a hypothetical 2.8 doesn't blow up in your face the same way.

> I have code written for Java 1.3 that was released (wow!) 17 years ago. It still works perfectly on the most recent JDKs. Why should I rewrite the code that does what it should?

The Lady of the Lake,
[angels sing]
her arm clad in the purest shimmering samite, held aloft ssl3
from the bosom of the water signifying by Divine Providence that you,
Cyberax, Python developer, must rewrite your code.
[singing stops]

I dunno. Languages are made by different people for different purposes. Some of those include rigid compatibility. Python was obviously made to ruin your life, but hey, at least its successful at what it set out to do.

Python 3 adoption

NAR — Thu, 16 Apr 2015 14:47:26 +0000

You can easily have ten times the same "shared library" scattered around deep inside C:\Programs under Windows

So what? C:\Program Files* uses 13GB out of the 500 GB disk on the computer I'm typing this. That's less than 3%. Should I care? I don't think so. Again, the difference is that on Windows you can expect that 10 years old binary will work, on Linux you shouldn't expect that unmaintained code will work in a year.

Computer Science PhD student [...] I usually succeed

I think it would be a shame if you don't :-) But it is one thing to fetch and compile dependencies for an academic project and a completely other thing is to figure out why some software broke after an apt-get upgrade on a live system...

Python 3 adoption

lgeorget — Thu, 16 Apr 2015 12:40:38 +0000

>On Windows I can expect a 10 years old binary (effectively unmaintained code) working on a current version, on Linux I can't expect the same code working during the 6-12 months lifecycle of a distribution! And many people think it's fine.

Actually, you run into some problems and have to use the "compatibility parameters" (and cross fingers) if you want to run a 10-years-old software under Windows. And if it does not run into dependencies problems, that's because every software will usually come with all the DLLs it needs when you download it from Internet. You can easily have ten times the same "shared library" scattered around deep inside C:\Programs under Windows. So, I don't think you can actually compare Windows and Linux on that point.

As an Computer Science PhD student, I sometimes need to make old unmaintained code to run (such as prototypes, or obscure libraries) and I usually succeed. I admit it can be a bit cumbersome to fetch and compile by hand a lot of libraries to meet the dependency of a software but you can do it.

Python 3 adoption

NAR — Thu, 16 Apr 2015 09:52:38 +0000

Actually, when you look back on it, this is entirely the fault of people using unmaintained code that they're somehow expecting to continue to work perpetually.

That's the problem with Linux on the desktop. On Windows I can expect a 10 years old binary (effectively unmaintained code) working on a current version, on Linux I can't expect the same code working during the 6-12 months lifecycle of a distribution! And many people think it's fine.

The ideologically correct solution would be to package all software to all distributions and let distributions maintain it, but it's obviously ridiculous (the distributions are already underpowered, have different release cycles, can't handle proprietary code, getting software into distribution could be complicated, etc). The technical solution is to bundle everything that can't be trusted to a distribution (practically everything up from glibc) and distribute that. With half dozen dependencies we've seriously looked at releasing a vmware image instead of "source tarball and instructions on getting the right versions of the right software". That's where containers and Lennart's latest proposal and other stuff come into play. Where distributions failed.

Python 3 adoption

daniels — Thu, 16 Apr 2015 08:31:54 +0000

While we're laying down the grounds for when personal abuse of free software developers is acceptable, here's mine: never.

Python 3 adoption

Cyberax — Thu, 16 Apr 2015 08:23:08 +0000

> ...and without it somebody else would have called them complete idiots for leaving an apparently insecure situation in place.
It was NOT a binary decision.

It was perfectly possible to write a small patch to add validation by default and allow to pass a custom CA-store, without breaking any existing code. It would have required a couple of dirty hacks inside the library and possibly some code duplication to add support for SNI, but it was entirely doable.

Python maintainers instead backported the whole SSL infrastructure from Python 3 which has a lot of changes inside of it. Here's the patch: https://bugs.python.org/file36423/ssl-backport.diff - it's almost 13000 lines long.

Python 3 adoption

corbet — Thu, 16 Apr 2015 08:14:42 +0000

...and without it somebody else would have called them complete idiots for leaving an apparently insecure situation in place. They were in a difficult spot and made the best decision they could. You may not agree with it — I'm not sure I agree with it — but calling them idiots does seem to be over the top. We should be able to treat each other better than that. I have no problem with expressions of disagreement here, but I'd really rather not see that kind of personal attack, please?

Python 3 adoption

Cyberax — Thu, 16 Apr 2015 08:04:52 +0000

I don't care if I'm smarter than Guido. Probably not.

Python 2.7 maintainers are STILL complete idiots since they go forward with obviously breaking changes for a minor release in a stable branch. This is not a case of a bug that crawled through the cracks in a patch, it was a deliberate breakage.

Python 3 adoption

daniels — Thu, 16 Apr 2015 07:54:59 +0000

> So yes, I maintain that the developers are complete idiots.

Sigh. Just stop here.

Even if your argument wasn't ridiculous (you're smarter than Guido and the rest of the Python team? ok, sure), quite what gives you the right to wander around abusing people because you disagree with their decisions is absolutely beyond me.

Grow up and accept that reasonable people can arrive at different conclusions without one of them being idiots. Your abuse contributes nothing whatsoever to the conversation, except a growing wish for comment moderation.

Python 3 adoption

Cyberax — Thu, 16 Apr 2015 06:54:30 +0000

> Haven't you learned enough about the Linux kernel over the past few years (you ever quoted some Linux-isms in your first reply) to realize that version numbers don't mean *anything*? Perhaps you can explain 2.6.39->3.x->3.19->4.0; based on your thought process, that's two major compatibility breaks, and at least one brain aneurysm when dropping a minor number.
WTF? It's such a braindead statement, that I'm out of hands for a facepalm.

Linux has a stable userspace ABI. I can take a static binary compiled 15 years ago and it will _still_ _work_ on the newest kernels. That's why kernel release version doesn't matter.

> You're so hung up on version numbers, but this problem would've happened eventually even if it meant going to 2.8. You had plenty of warning, the library authors had plenty of warning; yet, you both did nothing. A version number isn't going to change that unpreparedness.
It's NOT A PROBLEM OF VERSION NUMBERS.

It's a problem of a minor Python update _breaking_ _user_ _code_. On purpose: https://bugs.python.org/issue21308

Want to try something similar with Linux? Simply submit a change that removes readdir() syscall and replaces it with a better designed interface. In a -rc6 release. And then say something like: "But it's to fix a terrible mistake from years ago!"

But please, give me a heads up a couple of hours before your submit your patches. I want to have enough time to grab popcorn and enjoy the show.

> So you say this is an ongoing problem, even though that bug report has more people saying that their issue was fixed than are still complaining. The question becomes where's your published fix? Surely you'd still be working 20-hour days if you can't point to a fix.
Actually, you're right. Here's a raw diff for docker-py: http://pastebin.com/3JpveFxu We used a dirty workaround for Boto which I'm not going to submit. Gevent patches were sent to its author. I have a patch for websocket-client, will submit it later.

I'm definitely going to find some time to submit all the changes upstream.

> The Gevent project is still dead; boto and your product aren't going to work with this Python 2.8 either, and the next round of forward leaning distros will pick up the newest stable version of Python anyways. What's the difference if nobody is maintaining Gevent regardless of Python version number?
Python 2.7 is going to be supported for the next 6 years. That was acceptable for us earlier. Now we're looking for an alternative with saner maintainers.

> It was not. It was a terrible mistake all those years ago to forgo default certificate validation by `ssl`.
Again, let me repeat. Python changes broke the code that was explicitly doing the right thing.

This change could have been incremental - a couple of optional parameters to pass the CA store parameters and perhaps a global function to override the default behavior. Such changes would have preserved the existing software that does its own validation _and_ they would have provided secure defaults for everybody else.

However, Python developers instead backported SSL layer from Python3, knowing perfectly well that it's going to break stuff. Let me quote them:

> > I doubt adding a ton of new APIs and code can be uneventful, but good
> luck :)
> They don't call it "Rawhide" for nothing! :)"

Or maybe:

> I spent hours looking at this patch, which certainly doesn't constitute a real review, but is probably about as good as your going to get on this behemouth.

So yes, I maintain that the developers are complete idiots.

> Honestly, just leave and stop whining. Oh no, the code that hasn't been updated in ages stops working when the rest of the environment is updated -- that's not in any way surprising!
I have code written for Java 1.3 that was released (wow!) 17 years ago. It still works perfectly on the most recent JDKs. Why should I rewrite the code that does what it should?

Python 3 adoption

fandingo — Thu, 16 Apr 2015 06:03:26 +0000

> Every month when windows patches are applied in your network, does every function of every application (including ones that you didn't write) get tested before being deployed to anything important?

First, that's not my job, and second, I don't use Windows at work, so I don't have the first clue. Additionally, I wouldn't really consider workstation updates anywhere near critical, but perhaps I simply don't work at a company with enough employees to fill a stadium. (I don't see how this relates to the issue at hand anyways.)

I feel like I'm in crazy town talking to you. Do you not promote system updates to your dev, test, and integration environments before pushing them to production. This isn't some sort of sneaky problem that only triggers itself under rare circumstances. If boto is unpatched and talks to AWS in any manner, it's error city. Cursory test coverage would hit it, and developers just doing their normal work would uncover it in no time.

You seem to want to turn this into some sort of hypothetical or abstract discussion. That's not the situation. It's a very specific set of circumstances that affect a defined set of libraries (boto and gevent -- really just gevent). I don't really care about talking about abstract stuff when there's already a concrete discussion to have.

The facts are that this error would be apparent to any developer or system administrator who has segregated environments and staggers updates through them because the errors would occur so often and obviously that they could not be overlooked. It was either sloppy engineering that allowed this patch to be deployed to production without testing or lazy management who didn't realize they were dependent on dead code (gevent) and didn't bother to follow changes that might affect that dead library.

Python 3 adoption

fandingo — Thu, 16 Apr 2015 05:49:28 +0000

If you want to hate Python, go for it. I'm not about to get into some kind of holy war over languages. What I will argue is how flawed your argument is.

Haven't you learned enough about the Linux kernel over the past few years (you ever quoted some Linux-isms in your first reply) to realize that version numbers don't mean *anything*? Perhaps you can explain 2.6.39->3.x->3.19->4.0; based on your thought process, that's two major compatibility breaks, and at least one brain aneurysm when dropping a minor number.

The comparison to GCC is specious because they only implement a base language. There's no stdlib. Golang is also a bad example because it's a whopping 8 years old -- unsurprisingly they haven't gone through these long-term issues and have greatly benefitted from witnessing the past.

You're so hung up on version numbers, but this problem would've happened eventually even if it meant going to 2.8. You had plenty of warning, the library authors had plenty of warning; yet, you both did nothing. A version number isn't going to change that unpreparedness.

> I'm serious. I had to do emergency bugfixing and testing because our customers' machines suddenly stopped working. Even finding the issue itself took some time - it's only indirectly seen in some libraries (like boto) as failing SSL requests without any stacktraces pointing out the problem.

So you say this is an ongoing problem, even though that bug report has more people saying that their issue was fixed than are still complaining. The question becomes where's your published fix? Surely you'd still be working 20-hour days if you can't point to a fix.

This seems more like a tale about lazy, uncaring, or incompetent library authors (boto, gevent, and you) that can't seem to resolve issues that were clear a year ago. In the case of gevent, that seems to make sense: Gevent appears to be a dead project and hasn't had any activity in over a year. Perhaps you and boto should either stop using a dead project, freeze every dependency (includign cpython), start maintaining gevent yourselves, or move to something else.

Actually, when you look back on it, this is entirely the fault of people using unmaintained code that they're somehow expecting to continue to work perpetually. There has been exactly one commit to Gevent since 2.7.9 was released; it was 12/13, three days after and unrelated. No activity on the entire project in 2015. Let's say that the Python developers put this in a hypothetical 2.8, what difference does it make? The Gevent project is still dead; boto and your product aren't going to work with this Python 2.8 either, and the next round of forward leaning distros will pick up the newest stable version of Python anyways. What's the difference if nobody is maintaining Gevent regardless of Python version number?

> Let me repeat - the problem is that Python developers went and _broke_ _existing_ _code_ _that_ _was_ _doing_ _the_ _right_ _thing_. That was totally avoidable.

It was not. It was a terrible mistake all those years ago to forgo default certificate validation by `ssl`. There's nothing that can change that. The situation last year was to either continue to put our thumbs up our asses, even though we knew it was leading to bad behavior and confused users, or bite the bullet and fix things. I'll heap blame on them for their original decision, but dramatically increasing security in a sound, responsible way must be applauded. You and your dependent library authors could've started testing over a year ago, but didn't for whatever reason.

> In short, the whole Python ecosystem is screwed up. They can't be trusted with producing a reliable base, so I'm not going to use them anymore for new projects.

Honestly, just leave and stop whining. Oh no, the code that hasn't been updated in ages stops working when the rest of the environment is updated -- that's not in any way surprising! However, it doesn't sound like you have much choice. You're making libraries and products for customers that want to use Python, so, yeah...

Python 3 adoption

dlang — Thu, 16 Apr 2015 03:41:38 +0000

> Automated testing, TDD? I thought that was common place a decade ago. I can't imagine deploying code where every method and class aren't testable, especially for something like interfacing with AWS. This problem would've been caught by any Boto interaction with AWS. Any!

what you are missing is that they didn't deploy their code. they applied system patches.

Every month when windows patches are applied in your network, does every function of every application (including ones that you didn't write) get tested before being deployed to anything important?

If so, you are a 1%er (if not more rare)

Python 3 adoption

Cyberax — Thu, 16 Apr 2015 02:10:05 +0000

> Really? When the issue was discussed, it was more like many people thought cert validation was occurring because that's a really important part of SSL.
Really. Our apps are actually just OK with plain HTTP, since they are used over trusted networks or over VPNs. Important pieces even use custom security protocols.

> You (and/or your coworkers) have no one to blame but yourself. 1) Didn't pay attention to upstream; yet, you use their updated code.
I don't follow all the minutae of mailing lists. And this particular bug was not even known until people started encountering it.

I've actually read an article on LWN about Python turning on validation by default. And this is not a problem in itself, boto has always used certificate storage and validated the Amazon AWS certs. We even used it in a couple of important pieces.

> 2) Didn't pay attention to your distro update policy or read the update notes.
See above. It was a MINOR UPDATE. Not a Python 2.8, but a routine minor update.

Somehow GCC or Golang developers manage not to break the world and when they DO need to do it, they release them in a major update.

> 3) Apparently, apply upgrades to production that cause major problems without even basic testing.
Our customers did. We provide tools that our clients install on their own systems.

Guess what happens when they break?

> Everything I could find about boto and/or gevent problems with ssl on Python 2.7.9 indicates that there's been a fix available since September 21, which was months before 2.7.9 was actually released.
No. There's still no fix: https://github.com/gevent/gevent/issues/477

> The other interesting part is that no Ubuntu LTS release has 2.7.9, and in Debian, Wheezy doesn't either. I know there's other apt-based distros, but it certainly doesn't sound like you're running anything resembling "enterprise."
Python 2.7 in Ubuntu 14.04 with the recent updates has it. And _of_ _course_, Python 2.7 in Ubuntu 12.04 does not have it (even with the most recent patches).

Lots of other people have been burned by it. For example: https://bugs.launchpad.net/ubuntu/+source/python-eventlet...

> Lastly, "several 20-hour workdays?!" You can't possibly be serious.
I'm serious. I had to do emergency bugfixing and testing because our customers' machines suddenly stopped working. Even finding the issue itself took some time - it's only indirectly seen in some libraries (like boto) as failing SSL requests without any stacktraces pointing out the problem.

Let me repeat - the problem is that Python developers went and _broke_ _existing_ _code_ _that_ _was_ _doing_ _the_ _right_ _thing_. That was totally avoidable.

In short, the whole Python ecosystem is screwed up. They can't be trusted with producing a reliable base, so I'm not going to use them anymore for new projects.

Python 3 adoption

fandingo — Thu, 16 Apr 2015 00:12:03 +0000

> The third item is the only one that is reasonably their fault. The first two are the reason for using distros in the first place. Nobody is able to follow every upstream to the level of detail needed to catch this ahead of time.

Cyberax specifically knew about this 3 entire months before it was published! This isn't even remotely a tale about someone not knowing this change was coming. He obviously read and commented on an article that said the version number and release schedule. It's kind of a different situation when there is documented proof that he did know about this change.

Furthermore, he says they were using apt-get, and that almost certainly means Ubuntu or Debian. None of the Ubuntu LTS releases have anything newer than 2.7.5. Debian testing and unstable do, but Testing only got it on March 16th. The point being that there aren't too many places one could've picked up python 2.7.9 with apt-get.

I've worked at places that use Gentoo, Fedora, contemporary EL, or the most ancient EL distros imaginable. Different strokes for different folks, and I think that they all have their places. But, if you want to use something that changes quicker, you *have* to pay closer attention.

> You probably have a different definition of "basic testing" that I do, but I don't find it possible to test every single function of every single tool for every single upgrade.

Automated testing, TDD? I thought that was common place a decade ago. I can't imagine deploying code where every method and class aren't testable, especially for something like interfacing with AWS. This problem would've been caught by any Boto interaction with AWS. Any!

Furthermore, the mention of Boto and the extensive amount of work to fix the problem indicate that this functionality was critical to the application, which means it should've been covered.

The other point worth mentioning is that there is an issue on the Gevent Github page that is easily searchable, and there was a very easy workaround on September 21. It also works for Boto.

Basically, Cyberax's story is complete nonsense. He just wanted to bitch about Python, yet again, so he concocted a story. The dates don't make any sense. There was already a workaround nearly 3 months before the Python release for the specific libraries he mentions. Then, the distro both doesn't make sense (Debian Testing/Unstable or Ubuntu 14.10 in an enterprise that does blind updates?), and the timings are crazy (Python 2.7.5 released in Debian Testing and Ubuntu 14.10 on March 16, 2015).

Maybe there was a problem with their application, maybe they are running a newer Apt-based distro, and maybe the update slipped past them. Okay. The parts that seem totally unreasonable are 1) that he knew about this change in September by commenting on an LWN article about the compatibility break (and complained about it) and 2) that he would spend "several 20-hour days" trying to fix this problem and not type in "boto python 2.7.9 ssl" in Google and find the same solution that I did, which was posted in September. Hell, that solution is damn simple anyways, and any programmer should be able to recreate a missing method. Or, you know, just downgrade the Python package.

Getting back to testing, don't you have multiple environments? I don't know what his application does, but it seems difficult to imagine that if any sort of staggered upgrades across environments were being used that, at minimum, one of the developers would notice that every call to AWS was failing spectacularly.

Python 3 adoption

dlang — Wed, 15 Apr 2015 23:02:08 +0000

> You (and/or your coworkers) have no one to blame but yourself. 1) Didn't pay attention to upstream; yet, you use their updated code. 2) Didn't pay attention to your distro update policy or read the update notes. 3) Apparently, apply upgrades to production that cause major problems without even basic testing.

The third item is the only one that is reasonably their fault. The first two are the reason for using distros in the first place. Nobody is able to follow every upstream to the level of detail needed to catch this ahead of time.

as to the third, you make the assertion that they did the upgrades "without even basic testing". You probably have a different definition of "basic testing" that I do, but I don't find it possible to test every single function of every single tool for every single upgrade.

Trying to do that leads to paralysis.

Python 3 adoption

fandingo — Wed, 15 Apr 2015 22:30:35 +0000

> It works and not many people care about certs.

Really? When the issue was discussed, it was more like many people thought cert validation was occurring because that's a really important part of SSL. It's also worth pointing that you do seem to care about cert validation -- enough to even implement it in your applications.

> I had to spend several 20-hour workdays to track down and fix all the incompatibilities in our internal apps after a routine 'apt-get upgrade' broke everything.

You (and/or your coworkers) have no one to blame but yourself. 1) Didn't pay attention to upstream; yet, you use their updated code. 2) Didn't pay attention to your distro update policy or read the update notes. 3) Apparently, apply upgrades to production that cause major problems without even basic testing.

PEP 466 was published on March 23, 2014 and approved on April 19, 2014. The implications were extremely clear. Python 2.7.9 was released on December 10, 2014 (235 days later). There was plenty of time to make the minor changes necessary, and they really are minor.

The best part is that I went to see if LWN had written an article about the ssl issue. Not only had they on September 10 (https://lwn.net/Articles/611243/), but also, you were the first comment complaining about breaking compatibility! Yet, poor Cyberax wants to pretend he was caught unawares by those mean old Python devs.

The other interesting part is that no Ubuntu LTS release has 2.7.9, and in Debian, Wheezy doesn't either. I know there's other apt-based distros, but it certainly doesn't sound like you're running anything resembling "enterprise."

Lastly, "several 20-hour workdays?!" You can't possibly be serious. If the problem is so severe, that it not only takes that long to solve, but it's that important to solve ASAP, I think downgrading to 2.7.8 would've been my solution after maybe 3 hours. Or perhaps, just replacing ssl.py with the previous version.

> And I still have no idea what's wrong with Boto+gevent combination - I simply switched them to unsecured HTTP.

Everything I could find about boto and/or gevent problems with ssl on Python 2.7.9 indicates that there's been a fix available since September 21, which was months before 2.7.9 was actually released.

> I wonder with what part of "stable branch" and "we don't break userspace" the developers of Python have difficulties?

Perhaps the fact that they don't absolutely subscribe to either belief. There's plenty of discussion in PEP 466 and 476 about why this unusual change was made.

Python 3 adoption

Cyberax — Wed, 15 Apr 2015 20:40:32 +0000

They should have just left Python 2.7 as-is. It works and not many people care about certs.

The next best option would have been to add SSL validation as a backwards-compatible smallish fix. But they didn't do that - instead Python developers CHANGED THE WHOLE FREAKING SSL STACK. So it broke lots of applications that actually _do_ _the_ _cetificate_ _validation_.

I had to spend several 20-hour workdays to track down and fix all the incompatibilities in our internal apps after a routine 'apt-get upgrade' broke everything. And I still have no idea what's wrong with Boto+gevent combination - I simply switched them to unsecured HTTP.

I wonder with what part of "stable branch" and "we don't break userspace" the developers of Python have difficulties?

Fuck them. I'm not starting any new projects in Python. I'm now back to Java for server-side and Golang for small client utilities.

Python 3 adoption

smoogen — Wed, 15 Apr 2015 20:30:24 +0000

I have to disagree on the change that broke the ecosystem. I spent quite a bit of time in the nineties hacking code that would not compile because it was K&R and other code was ANSI and the compilers did not like to mix the two.

Officially it wasn't a break in the ecosystem, in practicality it was a major pain in the ass. Trying to get patches upstream was met by "K&R was good enough when I started and I will damn well retire with it." and other phrases. And on some platforms you ended up with weird #ifdef because the K&R code compiled under the new compiler but had a different side effect and vice versa (similar ANSI C would compile on a K&R but did produce the effect the very smart programmer had intended.)

On the rest of your response I can't answer as I don't know enough about Van Rossum's employment or Go. I do agree that I expect that Python2's death will have little to do with Python 3/4/5/etc. [Any more than I find working perl4 scripts in places you would think "Its been 20+ years people..]

Python 3 adoption

southey — Wed, 15 Apr 2015 15:46:40 +0000

the project is not communicating well about deprecated features or packages

This sums up a major reason why I gave up trying to move to Python 3. Really Python 3 was taking too long to stabilize because various backwards incompatibilities were introduced between versions (not just 3.0). If you were lucky then you could fix the problem yourself. Of course, fixing not only required you to fixing and rebuilding your code but all dependencies with the new Python version without introducing incompatibilities with other Python versions. That in turn meant you could not rely on any distro's automated package management and needed developer versions that may contain other bugs or unexpected features. Otherwise you would have find some workaround until it got fixed upstream and then remember to remove your workaround.

That other reason was that is too much of a pain to remember the new small syntax changes like those with print and dict's iteritems. These widespread syntax usages should have been allowed when valid (that is, essentially non unicode changes). Then it would have been easy to transition to the new syntax.

As others have commented, there has not been a sufficient reason to change when you do not have to address unicode. If the developers and others want the community to change then they need to step up and do more than just pump out new versions. There has to be stricter testing (such as testing well used Python packages that have their own tests) to avoid introducing incompatibilities and far slower removal of depreciated syntax.

Python 3 adoption

fandingo — Wed, 15 Apr 2015 15:40:42 +0000

The truth of the matter was that they put off putting in that breaking change (actually verifying certificates) for far too many years. It was always going to break a bunch of programs that connect to suspect web services. In the end, I think that they had stuck their heads so far into the sand that they finally popped up in China, and the sudden onrush of daylight made them aware that now is better never.

The security situation was dire, and they had to do something. Furthermore, the three SSL/HTTPS changes were all breaking changes. As a developer, I'd rather my code breaks once for three reasons than have to deal with three separate interruptions.

Python 3 adoption

dashesy — Wed, 15 Apr 2015 06:33:12 +0000

type hints are the only new feature I deem useful enough to worth the pain of the upgrade, now have to wait till PyCharm supports it.

Python 3 adoption

b7j0c — Wed, 15 Apr 2015 05:33:35 +0000

> Barry Warsaw described a Twitter tool and format called PEX that takes a "well-written setup.py" and turns that into a kind of executable. It contains the Python interpreter, shared libraries, and imported modules needed to run the program.

ugh. containers are being oversold as a panacea, but this just seems to scream for something like Docker. any tool that requires source code to be deployed suffers from complex deployment configuration, and often ops staffers are loathe to include third-party modules (typically not due to trust or safety, just because it requires more configuration tooling which is a pain in the ass). python isn't alone here...perl and php are equally painful to configure for deployment once you go beyond the base install.

for most people who build enterprise code, Go's model is vastly superior. "go build program.go; scp program host:/path/.".

Python 3 adoption

ibukanov — Wed, 15 Apr 2015 05:26:48 +0000

Was interpreter startup time mentioned as a reason for slow Python 3 adoption? Mercurial developers mention that as one of the reasons they stick with Python 2. It is sad that that it is often faster to compile and run a Go program than to invoke Python code doing similar things.

Python 3 adoption

b7j0c — Wed, 15 Apr 2015 05:23:24 +0000

but C has never had a change that broke the ecosystem....

python wedged itself into key system components in most major distros, so there was always going to be part of the community that was risk averse and willing to hold on to mature code. who wants to take a risk migrating code for an OS installer that works? etc etc

even then, there was little to compel anyone else to get that excited about python3. it just doesn't seem that motivating. sure, there are some worthwhile fixes...but nothing to make people stop and take notice, especially with so many other neat distractions happening elsewhere in the platforms world (Rust, Go, node.js etc)

it doesn't help that GVR's employer released python2 tooling and then crowed about using Go. if you can't get BDFL to get his employer on board, what hope is there?

most python2 programmers at this point are probably looking at Go instead of python3. i know its sporting to pick on Go, but it is closer to python syntactically than any other major language and it offers many more meaningful advances than python3. Go also has hype and a hypeful tool in its brag bag (Docker)...python3's hype meter is limited to minor syntax tweaks. yawn.

python2 will go on for another decade at least. if and when it does die, python3 will have had nothing to do with it