|
|
Subscribe / Log in / New account

Python 3 at Facebook

By Jake Edge
June 27, 2018

PyCon

Python 3 adoption has clearly picked up over the last few years, though there is still a long way to go. Big Python-using companies tend to have a whole lot of Python 2.7 code running on their infrastructure and Facebook is no exception. But Jason Fried came to PyCon 2018 to describe what has happened at the company over the last four years or so—it has gone from using almost no Python 3 to it becoming the dominant version of Python in the company. He was instrumental in helping to make that happen and his talk [YouTube video] may provide other organizations with some ideas on how to tackle their migration.

Fried started working at Facebook in 2011 and he quickly found that he needed to teach himself Python because it was much easier to get code reviewed if it was in Python. At some point later, he found that he was the driving force behind Python 3 adoption at Facebook. He never had a plan to do that, it just came about as he worked with Python more and more.

[Jason Fried]

He started out by being active in the internal Python group. He was often the first to answer questions that came up. He eventually became famous ("or maybe infamous") with the Pythonistas at Facebook because, when he saw a problem with how the language was being used, he didn't ask permission, he simply fixed it. That works at Facebook because there is no real top-down hierarchy of control; everyone has as much power to back out a change you make as you have to make the change to begin with. Over time, his changes built up credibility within the Facebook Python community that would serve him well in the migration process.

Changing something like the Python language version at "Facebook scale" was going to take some time and a lot of diplomacy, he said. He wanted to tell the "story about how I and couple of engineers used our free time, with no authority whatsoever, and made Python 3 the dominant version at Facebook."

In 2013, there was rudimentary support for Python 3.3 at Facebook. It was there as part of a task for adding Python 3 support to the build system. But that task was blocked on Python 3 support in the Facebook libraries, which was in turn blocked by no Python 3 support in the build system. It was something of a catch-22; Python 3 was "available" but nothing in the Facebook environment supported it.

In addition, there was lots of negative sentiment about Python 3 at Facebook in 2013. The overall thinking was that the company would simply stay on Python 2.7 forever. There was talk of jumping ship to another language entirely. Even he said (in an internal group) that Python 3 would never happen at Facebook. Only one person challenged him on that statement and suggested that he do something about it; at the time, he ignored the suggestion, but it did stick in his head.

Some hope

There was, actually, some hope, he said. In January 2013, the four imports from __future__ (print_function, division, absolute_imports, and unicode_literals) were required by a "linter" that was being used. They were added in an attempt to extend the life of the Python 2 code base. They were added everywhere in order to quiet the linter, which ended up making it easier to convert modules to Python 3.

The Apache Thrift framework for serialization and remote procedure calls is "used everywhere" at Facebook. Since it was Python 2-only, it was a core blocker. But adding Python 3 support was popular in a poll for new Thrift features that the Facebook Thrift group had run. He voted for it, but not because he was on the Python 3 bandwagon at that point; he thought the Python 2 interface needed a refactor as it looked like it had come from Java.

His thinking started to switch when he saw Guido van Rossum give a talk at Yelp in San Francisco on something called "Tulip", which is what eventually became the asyncio module. He had always been a fan of asynchronous programming in Python, but found that it was fragmented because of the differences between the frameworks (e.g. Twisted, gevent) that provided it. Tulip looked like it would make asynchronous I/O interoperable rather than fragmented. Before that talk was even over, he was communicating with the Facebook Thrift team, suggesting that Thrift should simply support Tulip for Python 3, rather than wait for Twisted, gevent, and others to port to Python 3. A few days later, the Thrift team published a roadmap that showed Python 3 and Tulip support coming.

Both of those arrived in early 2014, but then nothing happened for six months; users did not show up, they had no plans to show up, and they, in fact, did not know about the changes at all.

A new project

In August 2014, he started a project to rewrite a service that he had inherited. He started planning to do it using gevent and Python 2, but then realized it would be obsolete at the time it was written if he did so. In order for something to change, someone needs to be the first one; for Facebook and Python 3, that was him. "For Python 3 in your organization, I think that person should be you."

So he started his project using Python 3 and "everything was broken"; it was no wonder that no one was using Python 3. The build system would not even build his code and all of the third-party wheel packages were only available for Python 2. When he finally fixed enough things to allow his service to be built, it would immediately fail when it was run—someplace deep in the guts of the code that sets up service entry points in the Facebook system.

So in order to get his code running, he had to fix everything else; he rebuilt hundreds of third-party wheels so that they would work with both Python versions and he had to make any internal libraries be 2/3 compatible. Every day, though, someone would commit a Python 2-only change into one of his dependencies. Not surprisingly, he got tired of fixing regressions. One solution would be to force Python 3 compliance within the organization, but Facebook is not a place where that is possible. But, if you act like you have some authority, people will start to believe that you do, indeed, have that authority.

He used up a lot of his social capital to add Pyflakes linting into the build process. He was able to justify adding it because there already was a PEP 8 linter, but Pyflakes would address other code quality issues; in addition, Pyflakes had few false positives so it did not overly irritate the developers. He set things up so that Pyflakes would run on all code that was put up for review, first for Python 2 and then for Python 3. That helped spread the job of keeping Python 3 compatibility out to all of the developers and not just him, which allowed him to make progress with his project.

Early on, he had to be responsive to help people understand that "no, the linter is not wrong" and that there was value to making the code work with Python 3. If the developers had started believing that moving to Python 3 was difficult, they would fall back on the "let's stay with Python 2 forever" mindset. He made it easy for developers to do the right thing with respect to keeping the code running on Python 3. It was easier to just "shut the linter up" and, by extension, him, than it was to complain about it, so most developers just did so.

Education

With all that in place, he had stopped the bleeding, but little or no progress toward running more Python 3 at Facebook was being made. He joined the team that did training on Python programming for new employees at Facebook. The linters already complained if the code was not compatible with 2 and 3, but he wanted to get to a point where 2/3 compatible code was only written for legacy projects and that new code should be written in Python 3. Once again he took matters into his own hands: in 2015, he changed the slides for the new employee Python class to make that statement. The idea was that at some unknown point in the future, Facebook will want to switch to Python 3, so writing Python 2-only code at this point makes no sense since it will have to be rewritten someday. He taught new hires that all of this should just work with the Facebook infrastructure and build systems and that if it didn't, they should file a bug or try to fix it themselves. "Strangely enough, that's what happened."

In January 2015, he "finally shipped" his project. He spent most of the rest of the year telling people how much better it was and why they should switch to using Python 3 where they could. Over the year, various allies in the effort to switch to Python 3 at Facebook made themselves known.

One of those allies was Łukasz Langa, who had "somehow convinced Instagram to move to Python 3". In 2016, he and Langa formed a brand new team in Facebook to shepherd Python within the company, which they dubbed "The Ministry of Silly Walks". Because they were "the Python team", the "perceived authority" he mentioned earlier worked; people assumed they could make decisions about Python at Facebook.

In 2016, he was seeing slow but steady growth in the amount of Python 3 that was being run at the company. There was mention of it in meetings and he regularly heard of new projects that were using it. The tide of opinion had changed at Facebook even though Python 3 was not the default and projects needed to actively choose to use it. By May 2016, he signaled his intention to switch the build system default to Python 3, which was overwhelmingly supported so he made that switch a few days later—with no ill effects.

Toward the end of 2016, there was a post from a project team that reported its results in switching to Python 3. The developers simply ran 2to3 on the code and fixed a few things that it complained about. When they ran the resulting code, they found it was 40% faster and used half the memory. This points to a persistent myth that Fried has heard: Python 3 is slower than Python 2. That may have been true for earlier releases of Python 3, but it is definitely not true now, he said.

Nice things

In early 2017, Instagram finished its migration to Python 3 and Facebook was reaping the benefits of this "glorious future where you can have nice things". Upgrades of Python versions were not particularly scary and brought new features that could be used. Facebook developers now focus on problems like using the new static typing features or migrating services to use asyncio. "Python at Facebook is fun again."

The problem now is that everyone is asking when Python 2 support can be retired. When there are regressions in the Python 2 support for a library or module, it is common to hear developers ask if those users can simply move to Python 3. It is the reverse of the problem he had a few years prior. "Oh what a wonderful world in which I live."

He showed a graph of the Facebook service entry points in Python over time, starting in Q3 of 2015 where there were four total Python 3 entry points. At the time of the switch defaulting to Python 3 in mid-2016, Facebook already had 4% of its entry points as Python 3. In March 2018, it crossed over the 50% line; in mid-May, when he gave the talk, it was 55% of the "tens of thousands of Facebook entry points" that were running Python 3. At Facebook it is now embarrassing to have code that only runs on Python 2, Fried said.

He then reviewed the process. He noted that you have to do more than just build something new; you have to lead developers to it by "being the change you want to see". You should get other people to help, even if they don't know they are helping, which is where linters and unit tests come into play. It is important to educate new hires for where you are heading. Once you get there, or some of the way there, celebrate by enjoying the "nice things": write some "awesome stuff in Python 3". Seeing how the new features can be used will make others want to convert.

He fielded some questions from the audience. One asked how they might make this happen in a more traditional, hierarchical organization. Fried thought that might actually be easier since, instead of thousands of developers that need to be convinced, it should be possible to work up the management chain starting with a manager who recognizes the benefits. It could also be harder if the culture is conservative, but focusing on code quality improvements may help there. Another question focused on code that is monolithic, rather than broken up into multiple entry points; for that Fried suggested looking at the Instagram keynote [YouTube video] from PyCon 2017.

There was lots in the talk that other organizations can use, but it is clear that having an advocate and shepherd with a lot of perseverance will be important. Companies that are planning a conversion of this sort will likely want to have someone like Fried on board.

[I would like to thank LWN's travel sponsor, the Linux Foundation, for assistance in traveling to Cleveland for PyCon.]

Index entries for this article
ConferencePyCon/2018
PythonPython 3/Adoption


to post comments

Python 3 at Facebook

Posted Jun 28, 2018 9:18 UTC (Thu) by dottedmag (subscriber, #18590) [Link] (2 responses)

So is Facebook a unique organization when one can spend 5 months rewriting a service they have inherited in a new language (and actually just fixing things so that would work), and then just decide to continue pushing for this language?

Would that ever work in a smaller company with way more limited resources?

Python 3 at Facebook

Posted Jun 28, 2018 9:59 UTC (Thu) by epa (subscriber, #39769) [Link]

I'm sure he didn't do only that for five months.

Python 3 at Facebook

Posted Jul 1, 2018 8:00 UTC (Sun) by k8to (guest, #15413) [Link]

I think this is what "engineering driven organizations" in the bay area look like frequently.

I noped on out of a place like that. You couldn't co-ordinate with anyone reliably, and teams had essentially no idea what the users of their systems actually needed. So I think there are a lot of places where this strategy would work fine. But I am going to try not work there.

Python 3 at Facebook

Posted Jul 2, 2018 22:55 UTC (Mon) by starchip (guest, #125445) [Link] (9 responses)

OK. I understand the issues that Jason had to confront and I sincerely congratulate him for conquering each one such that the goal was reached.

However, if the goal was incorrect, would we be congratulating his strategy?
No - we would not. The fact that a strategy works does NOT show that the goal was correct, which is what this article is implying.

I mean that it is implying that the goal was correct. It provides no evidence that it was/is correct it just assumes that readers will accept that it was/is.
This is an argument that has been around for thousands of years. It is called; "The end justifies the means". mmm. Does it? What justifies the end?
Ahhh - maybe that was the first infinite loop humans encountered.

What Jason has is an extremely bad overall Company strategy regarding IT infrastructure that he had to fight against. He did and he won. Gratz Jason.
Yet you have not won.
Has that strategy changed? Will there be another Jason fighting for Python4 at Facebook? Or maybe Java 3524466? Will that person be called Jason1?

If the overall strategy is wrong and the ability to change that strategy is unavailable or difficult, then that is what needs to be fixed. Not sneaking in a migration from languagewhateverversion into languagewhateverversionplus1. Jason has fixed the symptoms - not the disease. The disease is still there. Or tell me that it isn't.

I am uninterested in "how to fool your company into doing the right thing". I am very interested in "how your company can do the right thing".

Again I would like to record my admiration for the struggle that Jason went through and succeeded in. Myself, Jason and far too many others have done this before.
Let's not have to do it again.

Python 3 at Facebook

Posted Jul 3, 2018 0:29 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (7 responses)

> What Jason has is an extremely bad overall Company strategy regarding IT infrastructure that he had to fight against.
Bad strategy? Why?

Py2 works. It's better than Py3. It's generally faster than Py3. It's stable. It's going to be supported until at least 2027.

Now convince me why people need to convert to Py3 religion. Is it going to lead to significant savings? (nope) Perhaps it's going to make people more productive? (nope) Does it solve long-standing architectural issues? (nope, see GIL)

So what are the business reasons for the migration?

Python 3 at Facebook

Posted Jul 3, 2018 11:18 UTC (Tue) by anselm (subscriber, #2796) [Link] (3 responses)

At least the speed argument seems to no longer apply. From the article:

The developers simply ran 2to3 on the code and fixed a few things that it complained about. When they ran the resulting code, they found it was 40% faster and used half the memory. This points to a persistent myth that Fried has heard: Python 3 is slower than Python 2. That may have been true for earlier releases of Python 3, but it is definitely not true now, he said.

It's reasonably easy to imagine scenarios where that would be an important factor.

Python 3 at Facebook

Posted Jul 3, 2018 17:55 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

It can go either way. Py3 in general used to be slower than Py2: https://stackoverflow.com/questions/37052139/string-perfo... or https://www.raspberrypi.org/forums/viewtopic.php?t=183829 It's getting better, but it's still not that great.

I did several migrations and sometimes we actually got significant speed improvements. Not because of 2->3 switch itself, but because people went through the codebase and fixed stuff like the use of .items() instead of .iteritems().

Python 3 at Facebook

Posted Jul 4, 2018 0:32 UTC (Wed) by anselm (subscriber, #2796) [Link]

It probably depends on the use case but there has been considerable optimisation work done in recent Python versions (the web pages you cited mostly talk about Python 3.4, which from an optimisataion POV is ancient history). There will be even more improvement in the future but we're now at a point where a fear of performance regressions compared to Python 2.7 shouldn't keep one from moving to Python 3.6 or 3.7.

Python 3 at Facebook

Posted Jul 4, 2018 5:23 UTC (Wed) by daniel (guest, #3181) [Link]

Python2 and Python3 are both pathetically slow, because efficiency is not ingrained in the project culture. But they are better than Bash, that is the best I can say about that. Python stewards consciously choose to pigeonhole Python as a scripting language for no discernible reason. My advice for language learners these days: start with Python to get the idea, then move on quickly to Go.

Python 3 at Facebook

Posted Jul 8, 2018 11:03 UTC (Sun) by morksigens (guest, #92681) [Link] (1 responses)

I find your anti-Python-3 posts quite silly and bitter. I develop in Python 3 day to day and can't wait to drop Python 2 support everywhere. Python 3 has 10 years of language evolution in it, 2.x has not. It's many small nice things that add up. Also, regarding speed: https://hackernoon.com/which-is-the-fastest-version-of-py....

Python 3 at Facebook

Posted Jul 8, 2018 23:02 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]

> I find your anti-Python-3 posts quite silly and bitter.
I suppose it's the best argument Py3 advocates have. I proposed a simple exercise - try to convince me with objective facts why Py3 switch is warranted.

> I develop in Python 3 day to day and can't wait to drop Python 2 support everywhere.
I have it on good authority that KoolAid also tastes fine.

> Python 3 has 10 years of language evolution in it, 2.x has not.
In other words: "Python 2.7, being a great improvement on its successor, has managed to skip 10 years of language devolution".

> Also, regarding speed

Let me quote:
"Python 3.7 is 1.19x faster than Python 2.7, but the only Python 3.x release to beat the Python 2.7 benchmark I ran. The speed.python.org benchmark shows similar results."

So Python _only_ _now_ has caught up with Py2.7. And that only barely.

Py3.7 had been released quite literally a couple of weeks ago and it is not supported by RHEL or Debian Stable, so I wouldn't be able to use it for at least couple more years.

Python 3 at Facebook

Posted Jul 11, 2018 12:16 UTC (Wed) by mathewcohle (guest, #118622) [Link]

One strict business reason might be the hiring of new developers: I've encountered that people shrug when hear that the code base is in Py2.7

Python 3 at Facebook

Posted Jul 3, 2018 8:50 UTC (Tue) by farnz (subscriber, #17727) [Link]

Facebook's strategy is that migrations like Python 2 to Python 3 have to be engineer-led. Somebody has to step up, do the work, and convince the rest of the company that that work is of value, otherwise it does not happen. The idea is that if it's not of value, nobody will do it, and if it is of value, someone like Jason will step up and do the work.

Jason didn't have to fight the company strategy - it's just that somebody had to be first, and lead the rest of the company in a better direction. Changing the company direction is easy for an individual engineer at Facebook; just do the work and get your peers to agree that it's of value.


Copyright © 2018, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds