LWN: Comments on "Making Python faster"

Making Python faster

HelloWorld — Thu, 22 Jun 2017 17:33:32 +0000

There is a difference, but the real question remains: why not just write the software in a language with decent performance in the first place? It's not like there's a lack of choice…

What is the implication of this trend for security?

fujimotos — Sun, 18 Jun 2017 17:18:05 +0000

My casual impression is that this trend ("Rewriting built-in modules in C")
can introduce a number of nasty memory-related bugs, especially if these
modules involve tricky data structures.

Yes, C is definitely faster and will improve benchmark results. But a whole
class of issues (overflow, memory corruption etc.) can be avoided if we stick
to Python.

So: will the trend make CPython more insecure? Or is this not the case?

Making Python faster

robert_s — Sat, 17 Jun 2017 23:18:23 +0000

No - you need to rewrite *the critical bits* in C. There's a difference.

Making Python faster

mb — Fri, 16 Jun 2017 14:36:27 +0000

Interesting results. There must be something strange with the Debian Python 3.6 build.

> It's unclear to me how to run correctly your benchmark, since the speed changes depending on the time.

Yes, it's a bit jerky and I'll have to work on this. It doesn't do CPU pinning and other things to get rock stable results.

Thanks a lot for digging into this.
I very much appreciate your work on Python and it enables me to eventually drop Python 2 support. That would be great. :)

Making Python faster

vstinner — Fri, 16 Jun 2017 11:14:38 +0000

Warning: It's a quick & dirty benchmark, I didn't tune my system nor used PGO compilation.

It's unclear to me how to run correctly your benchmark, since the speed changes depending on the time. So I chose to run the program and interrupt it (CTRL+c) when it displays "t: 10.2s" at the top.

I compiled manually the 3.5 and 3.6 Git branches using "./configure --with-lto && make" and I get a different trend than you.

* Python 3.5 @ t: 10.2s: 190.83 k stmt/s
* Python 3.6 @ t: 10.2s: 196.36 k stmt/s

In my case, Python 3.6 is 1.03x faster (3%) than Python 3.5. Well, basically it runs at the same speed.

My system Python (I'm running Fedora 25):

* python2 (2.7) @ t: 10.2s: 197.74 k stmt/s
* python3 (3.5) @ t: 10.2s: 198.65 k stmt/s

Again, no significant different between python2 and python3 here.

Note: Fedora doesn't use PGO to build python2 and python3 packages. (But they are working on enabling it!)

Making Python faster

mb — Fri, 16 Jun 2017 11:00:28 +0000

>Oh, if PGO is not enabled on Debian

I did not check this.
I only concluded it from my benchmark results.
So I might be completely wrong.

Making Python faster

vstinner — Fri, 16 Jun 2017 10:52:55 +0000

Oh, if PGO is not enabled on Debian, the difference that you see between Python 3.5 and 3.6 might be explained by the C compilation which is not deterministic. It's really hard to run a stable benchmark, see my FOSDEM talk:
https://fosdem.org/2017/schedule/event/python_stable_benc...

And perf documentation:
http://perf.readthedocs.io/en/latest/run_benchmark.html#h...
and
http://perf.readthedocs.io/en/latest/system.html

Making Python faster

mb — Fri, 16 Jun 2017 09:22:40 +0000

>I would be interested by a benchmark on Python 3.7 since we finally implemented a CALL_METHOD bytecode to optimize method calls ;-)

I just ran a quick test using the latest git versions of 3.6 and 3.7 with and without PGO:

with pgo
$ python3.7 ./awlsim-test examples/EXAMPLE.awlpro
Speed: 215.80 k stmt/s (= 4.634 us/stmt) 55.0 stmt/cycle

without pgo
$ python3.7 ./awlsim-test examples/EXAMPLE.awlpro
Speed: 191.10 k stmt/s (= 5.233 us/stmt) 55.0 stmt/cycle

with pgo
$ python3.6 ./awlsim-test examples/EXAMPLE.awlpro
Speed: 201.55 k stmt/s (= 4.961 us/stmt) 55.0 stmt/cycle

without pgo
$ python3.6 ./awlsim-test examples/EXAMPLE.awlpro
Speed: 183.01 k stmt/s (= 5.464 us/stmt) 55.0 stmt/cycle

So I guess that Debian does not enable PGO.
This might also make sense w.r.t. stable builds.

The improvement in 3.7 looks quite impressive. Although the non-PGO variant still is a bit slower than 2.7, but it is pretty damn close.

I think the goal must be to beat 2.7 performance without PGO, because compiling two versions of a program with different compilers and then claiming one version being faster is cheating. :)

>I am interested. I just wanted to say that I'm sorry, I have no clue since I don't know your application at all. Please contact me directly to explain me how to run your benchmark.

Just clone the git repository and run the command as shown above. That's it. :)
Disclaimer: awlsim-test is not a proper benchmark. It's just a rough estimation.

Thanks a lot for your support.

Making Python faster

vstinner — Thu, 15 Jun 2017 21:35:10 +0000

> Well, there are some of articles and talks that claim Python 3.6 would be faster than Python 2.7. That's not true per se.

And the last slide says that in a few cases, Python 3.6 is still 10-20% slower in a few specific benchmarks, and up to 3x slower for the startup time.

> Most of the time in my application is spent by doing method calls and attribute lookup.

I would be interested by a benchmark on Python 3.7 since we finally implemented a CALL_METHOD bytecode to optimize method calls ;-)

> It could possibly only give a clue about the regression between 3.5 and 3.6.

Yes, according to your benchmark, there is a net regression, 3.6 is slower than 3.5. It would help to know if your Python binary was built using PGO. Sadly, I don't know how to check in Python :-( You have to look how Python was built in your Linux distro.

> So in the end Python developers should be interested in explaining performance regressions on "random applications"

I am interested. I just wanted to say that I'm sorry, I have no clue since I don't know your application at all. Please contact me directly to explain me how to run your benchmark.

Making Python faster

zlynx — Thu, 15 Jun 2017 19:04:52 +0000

Mostly true, but there's a large class of algorithms where doing more work gets you done faster because of pipeline bubbles or GPU compute. It can be a lot faster to just do 25% more work and discard it than to carefully check up-front which 25% actually needs to be done.

Making Python faster

mb — Thu, 15 Jun 2017 17:41:43 +0000

>Sorry, I cannot explain a performance regression on a random application.

Well, there are some of articles and talks that claim Python 3.6 would be faster than Python 2.7.
That's not true per se.

There might be a lot of cases where 3.6 is faster than 2.7, but as long as there is a single thing where 3.6 is slower than 2.7 and my application hits it, that means 3.6 is slower.

I don't use fancy Python 3 features. So this is 2.7 stuff that got slower in 3.x. Not new 3.x features that got faster during the 3.x development.

Most of the time in my application is spent by doing method calls and attribute lookup.
The performance critical part basically does not call into libraries.

>You should try to measure the difference, profile the code using cProfile

My code has cProfile support built in and it is a matter of setting an environment variable for the performance critical parts to be measured. So I think I have a pretty good idea of where most of the time is spent.
However for me it's hard to relate that to Python itself.
And profiling does not help at all to get an idea of why I don't see a 3.5/3.6 being faster than 2.7.
It could possibly only give a clue about the regression between 3.5 and 3.6.

In the end I think this shows one thing:
The Python test suite does not cover "my" use case very good. And I would be very surprised if I was the only one with that problem.

So in the end Python developers should be interested in explaining performance regressions on "random applications", because "random applications" are the real world. Test cases that show an impressive performance improvement are not.

Making Python faster

dgm — Thu, 15 Jun 2017 13:22:52 +0000

Sometimes you have, but often it is not the case. You could also use a better algorithm, for instance. But nothing defeats the speed of code that does not run. If you can avoid doing something, then don't do it.

Making Python faster

vstinner — Thu, 15 Jun 2017 13:04:21 +0000

Sorry, I cannot explain a performance regression on a random application. You should try to measure the difference, profile the code using cProfile or Linux perf, etc. I don't know neither if Debian compiles CPython using PGO, but I know that Ubuntu does it. PGO provides 10-20% speedup, so it's worth it ;-)

Making Python faster

mb — Thu, 15 Jun 2017 11:30:21 +0000

The performance of Python 3 is becoming pretty good by now.
In my application the speed of 3.5 is about the same as that of 2.7.
However I can't confirm that 3.6 got even faster. In fact it is quite a bit slower for me.

$ python2.7 ./awlsim-test examples/EXAMPLE.awlpro
Speed: 195.28 k stmt/s (= 5.121 us/stmt) 55.0 stmt/cycle

$ python3.5 ./awlsim-test examples/EXAMPLE.awlpro
Speed: 196.27 k stmt/s (= 5.095 us/stmt) 55.0 stmt/cycle

$ python3.6 ./awlsim-test examples/EXAMPLE.awlpro
Speed: 181.10 k stmt/s (= 5.522 us/stmt) 55.0 stmt/cycle

https://awlsim.de/

I'm using the Debian/sid versions of Python.
Is there anything wrong with these? Perhaps not using LTO or profile guided optimization?

Making Python faster

meyert — Thu, 15 Jun 2017 06:19:28 +0000

Reads like you need to rewrite code in C to make it fast.