Reducing Python's startup time

Posted Aug 18, 2017 15:19 UTC (Fri) by ehiggs (subscriber, #90713)
Parent article: Reducing Python's startup time

Startup time in Python on a busy NFS device or parallel filesystem (GPFS, Lustre, etc) is particularly painful. If you strace the startup you can watch it stat hundreds of files. If you start using pkg_resources, where it needs to crawl the filesystem to see what's available, then it goes to the thousands or tens of thousands depending on the size of the project and this results in a startup time in minutes on HPC systems.

$ strace -c -e open,stat  python3 -c "import numpy; import pkg_resources; print('hello')"
hello
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.000006           0      1193       129 stat
  0.00    0.000000           0       451         2 open
------ ----------- ----------- --------- --------- ----------------
100.00    0.000006                  1644       131 total

You would think that zipapp[1] fixes this by putting all the files into a single zipped executable, but last time I checked it only zips the application being distributed and not the dependencies.

[1] https://docs.python.org/3/library/zipapp.html

Reducing Python's startup time

Posted Aug 19, 2017 9:01 UTC (Sat) by gebi (guest, #59940) [Link]

maybe pex (python executable files)[0] can help here?

it creates a single executable including your python code and all dependencies.

[0]: https://github.com/pantsbuild/pex

Reducing Python's startup time

Posted Aug 21, 2017 16:35 UTC (Mon) by epa (subscriber, #39769) [Link] (1 responses)

I wonder if statting hundreds of files could be partly replaced by making a directory listing of the relatively small number of directories on the include path? That at least cuts out a stat() call to test if a file exists, though you do still need an NFS round trip to do anything further with it like find out its timestamp or open it.

Reducing Python's startup time

Posted Aug 21, 2017 19:50 UTC (Mon) by njs (subscriber, #40338) [Link]

Python 3's import system already does this optimization. pkg_resources might not.