Modifying the Python object model [LWN.net]

Modifying the Python object model

Posted May 20, 2018 6:33 UTC (Sun) by njs (subscriber, #40338)
In reply to: Modifying the Python object model by roc
Parent article: Modifying the Python object model

It may have been a garbled reference to the way refcounting causes memory write traffic all over the place. Workloads like Instagram's are often memory-bound, and they rely heavily on starting up a parent process, loading in their main code and data, and then forking it to get lots of workers that all share most of their data through CoW. But writes from refcounts and marking can break the CoW and create unnecessarily high memory usage. (They used to disable the GC entirely, and then later contributed a feature to Python to mark their shared objects as immortal so the GC would skip marking them.)

[1] https://instagram-engineering.com/dismissing-python-garba...
[2] https://bugs.python.org/issue31558

to post comments

Modifying the Python object model

Posted May 20, 2018 18:21 UTC (Sun) by rweikusat2 (subscriber, #117920) [Link] (1 responses)

The extremely amusing thing here is obviously that they disabled the additional, cycle-breaking tracing collector because it was causing their problems :->.

Modifying the Python object model

Posted May 24, 2018 8:36 UTC (Thu) by njs (subscriber, #40338) [Link]

Indeed! That's part of why I'm uncertain and just guessing :-). But it is perhaps easier to fix the scattered memory writes problem with a tracing collector than with refcounting.

Or heck, maybe he meant he wanted a compacting collector... that would certainly improve memory locality in some cases.