How common are allocation failures [LWN.net]

How common are allocation failures

Posted Apr 25, 2026 14:49 UTC (Sat) by devdanzin (subscriber, #183390)
In reply to: How common are allocation failures by DemiMarie
Parent article: Using LLMs to find Python C-extension bugs

You can say that with a well behaved program + system combo, they should be rare.

But in Python it's simple to run into them if your program isn't prepared to, e.g., handle problematic input. A too large string multiplication or an attempt to create a gigantic NumPy array (or even a plain list) will result in a MemoryError that is recoverable and can just result in a log entry or a message to the user.

With very resource starved VMs, you can hit MemoryErrors even in well behaved programs, but they'll probably be gracefully handled by them (including aborting if it makes sense). And as I said elsewhere, it might just make a single request fail and let the program continue running.

I'm gathering a few examples of MemoryErrors in production for another answer here, should be able to post it tonight.

to post comments

How common are allocation failures

Posted Apr 26, 2026 14:14 UTC (Sun) by devdanzin (subscriber, #183390) [Link]

I've tried to collect a few links showing real world OOM situations, but couldn't find many. I found a few more about processing large images, pandas, etc. But here are the main ones, many of them only showing triggerable and recoverable `MemoryError`.

- https://discuss.python.org/t/memoryerror-despite-having-e..., where it's shown a case that raises `MemoryError` despite it looking like the array should fit in RAM.
- https://github.com/msgpack/msgpack-python/issues/239, a real world issue on VMs with low memory amounts resulting in wrong tracebacks.
- https://github.com/gmpy2/gmpy2/issues/280, where GMP aborts on memory error and the user would like to be able to catch an exception instead.
- https://blog.stackademic.com/python-in-production-the-15-..., where a recoverable pandas `MemoryError` brought everythong down.
- https://medium.com/@ryan_forrester_/understanding-and-fix..., showing examples of handling `MemoryError` and a plausible situation.
- https://medium.com/brexeng/debugging-and-preventing-memor..., an interesting article giving an enhanced way of handling `MemoryError`.
- https://pythonspeed.com/articles/python-out-of-memory/ shows how easy it is to get a segfault on an OOM that should be easily recoverable.
- https://discuss.python.org/t/how-must-we-handle-integer-o..., where it's pointed that `string = "a" * 9223372036854775807` raises a `MemoryError`
- https://discuss.python.org/t/trying-to-understand-roundin..., where it's pointed that `import decimal; decimal.getcontext().prec = decimal.MAX_PREC; decimal.Decimal(1) / 3` raises a `MemoryError`
- https://discuss.python.org/t/faster-large-integer-multipl..., where Tim Peters show that `x = 1 << 1000000000000` raises a recoverable `MemoryError`, unlike GMP.
- https://discuss.python.org/t/a-product-function-which-sup..., where it's pointed that `from itertools import product; next(product(range(1 << 30), repeat=2))` causes a real `MemoryError` (exhausts memory in the system), but it's recoverable if not OOM terminated.