Separate process

Posted Mar 18, 2024 19:23 UTC (Mon) by jhumphries (guest, #129504)
Parent article: Insecurity and Python pickles

What is the overhead for unpickling an ML model? If the overhead is sufficiently high, wouldn't it make sense to just isolate the unpickle code in a separate process since the context switch overhead is negligible?

Separate process

Posted Mar 26, 2024 16:27 UTC (Tue) by sammythesnake (guest, #17693) [Link]

Getting the data from the "unpickling process" to the process you want it in would involve serialising/deserialising all over again, though!

You could potentially use this to do various sanity checks/sanitisation before re-pickling for interprocess transfer, but it would probably make more sense to do that as a preprocessing step before the data gets to your code at all.

I wonder if a safe-unpickle library could be written that does some magic on the code execution part of the unpickling process to disable access to any variables outside of the unpickled objects and ensures the methods of the created objects match the definition of the loaded modules. Come to think of it, why wouldn't this be part of the built in pickle functionality already :-/