An introduction to SciPy

Posted Jan 19, 2021 23:59 UTC (Tue) by Cyberax (✭ supporter ✭, #52523)
In reply to: An introduction to SciPy by xxiao
Parent article: An introduction to SciPy

> really?
Yes.

> the heavy duty is still executed in c/c++ modules, the python/scipy here are just glue
The problem happens once you can't express an algorithm using existing primitives in scipy/numpy and forced to do a tight loop in actual Python.

My most recent example: a dynamic programming-based optimization task. It can't be readily vectorized and pure Python version is just too slow.

An introduction to SciPy

Posted Jan 20, 2021 14:36 UTC (Wed) by mikapfl (subscriber, #84646) [Link] (4 responses)

In that case, use https://numba.pydata.org/ . Just-in-time compilation happens under the hood, and the result is usually faster than any C++ code scientists write. Integration with the rest of python is great and you can easily drop the big and bad GIL. For me, it really is the missing piece in the high performance python ecosystem.

An introduction to SciPy

Posted Jan 20, 2021 18:36 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

These are all workarounds for inherent Python brokenness and they don't work well unless you tailor you code to do that. Cython is similar.

An introduction to SciPy

Posted Jan 21, 2021 15:55 UTC (Thu) by mikapfl (subscriber, #84646) [Link] (1 responses)

Well, all software are workarounds for hardware's inherent brokenness. I'm an engineer, I don't care if something is "just a workaround" as long as it works and is fast.

An introduction to SciPy

Posted Jan 21, 2021 21:23 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

The problem with workarounds is that the problem doesn't actually go away. And likely to bite you eventually, especially in large long-lived codebases. We've had this issue with Cython before when a tightly optimized code suddenly started to work incorrectly.

Sometimes you have no choice but to use workarounds. But with a nice alternative (Julia) existing, why bother?

An introduction to SciPy

Posted Jan 21, 2021 16:03 UTC (Thu) by marcel.oliver (subscriber, #5441) [Link]

Just a data point: I have some, admittedly simple, Cython scripts that survived the Python 2 to 3 transition without change, which is more than can be said for a lot of other Python code, even though the transition of scientific software was generally relatively painless.

One of these is a simple tridiagonal solver, which is more than 3 times faster than Scipy's solve_banded (which shows that built-in functions are not always best as, in this case, the built-in solver is more general) and about 120 times faster than native Python.

So there are certainly speed and GIL issues in Python in serious need of fixing (maybe Numba is the way forward, but I did not have the need to try it yet). And, of course, Julia is an interesting and clean approach (that would probably have eaten Scipy's lunch if it was around 10 years earlier) but the tradeoffs for switching Ecosystems are still not so clear, especially since heavy-duty numerical code is still dominated, for good reason, by Fortran and C/C++.

The big advantage of Python is that the language and software landscape is not domain specific, yet it works very well and very naturally for array-based computation and data processing. So it's pretty seemless from symbolic computation (Sympy) via numerics to dedicated plotting, output, GUI, and networking components.