Python's os.urandom() in the absence of entropy

Blocking os.urandom()

Traditionally, os.urandom() has been implemented on Linux by opening /dev/urandom and reading the requested amount of data. This interface is non-blocking; it will not wait if the amount of entropy in the system's entropy pool is low. The implementation of /dev/urandom is such that the quality of the random data it returns will be high even if the entropy pool is depleted — with one possible exception. Immediately after the system boots, when the entropy pool will contain little or no entropy, /dev/urandom may return relatively predictable data. In most systems, this window of poor randomness is only open for a few seconds at most, but exceptions do exist.

In the Python 3.5.0 release, os.urandom() was changed to use the relatively new getrandom() system call on Linux. Unless it has been called with the GRND_NONBLOCK flag, getrandom() will wait, if need be, for the system entropy pool to be initialized. os.urandom() does not supply that flag, meaning that it can block if the entropy pool has not yet accumulated enough randomness. It seems like a relatively small change, with the prospect of being sure of living up to the "suitable for cryptographic use" promise in compensation. But one need only look at Python issue 26839 to see that the implications are not quite as simple as one would expect.

It turns out that, in some distribution configurations, Python scripts are run at the very beginning of the user-space bootstrap process. If the entropy pool is not yet ready, those scripts will block until entropy-pool initialization is complete. If there is nothing else going on, and especially if the system is booting as a virtualized guest, it may take a long time to accumulate enough entropy to proceed. In the incident that led to the bug report, the boot process simply hung for 90 seconds until systemd lost patience and killed the blocking process. That kind of behavior, created in the search for unpredictable random numbers, has quite predictable effects in the form of unhappy users.

From this sprung the bug-tracker entry referenced above, which turned into a fierce discussion on the wisdom of the API change and whether os.urandom() should return crypto-quality randomness at any cost. The discussion spilled over onto the python-dev list when 3.5 release manager Larry Hastings despaired of reaching any sort of consensus and asked Van Rossum to simply rule on the matter. The resulting thread led many participants to question whether they wanted to continue following the list at all but, in the end, it did come to some useful conclusions.

If one is doing cryptographically sensitive work early in the bootstrap process — generating an SSH host key, for example — then blocking the boot almost certainly makes sense. The consequences of the alternative — generating weak keys — can be severe. In this case, though, it turns out that such high-quality randomness was not needed. Nobody was generating keys; instead, Python was initializing its own internal random-number generator and setting up dictionary randomization to defend against hash-collision attacks. These internal calls were (inadvertently) changed when os.urandom() was changed, but there seems to be a rough consensus that they do not need blocking behavior.

So the proper fix for the observed boot hang is to do these internal initializations without blocking on the entropy pool. For Python 3.5, the os.urandom() change will also be partially reverted, in that the function will, once again, be non-blocking. It will call getrandom() with the GRND_NONBLOCK flag and, if that call fails, fall back to reading /dev/urandom as before. With these fixes in place, the blocking part of the change is effectively reverted and the immediate problem has been solved.

Blocking or exceptions?

That still leaves open the issue of how os.urandom() should behave; developers who are concerned about security are adamant that it should not return data when the entropy pool is not yet ready. So there is still pressure on Van Rossum (and the community as a whole) to specify blocking behavior starting with the upcoming 3.6 release. Python's benevolent dictator seems inclined to downplay the issue:

It became clear in the discussion, though, that opposition to returning questionable randomness from os.urandom() is strong. It seems likely that, in 3.6 and later releases, os.urandom() will no longer return data drawn from an uninitialized entropy pool. The question of how it will behave is, as yet, unresolved, though. In the end, Van Rossum asked the proponents of two different approaches to write up their ideas as Python enhancement proposals (PEPs); he will then choose between the two.

The first approach, favored by Victor Stinner, is to simply make os.urandom() blocking and be done with it — after ensuring that Python itself uses non-blocking behavior during its initialization. Changing to blocking behavior is arguably an incompatible change in a longstanding Python API but, as Stinner points out: "First of all, no user complained yet that 'os.urandom()' blocks. This point is currently theoretical." As long as the problems with starting Python itself are resolved, the thinking goes, there should not be problems for other users.

The alternative comes from Nick Coghlan. With this proposal, os.urandom() will raise a BlockingIOError exception if random data cannot be had without blocking. Adding a new exception to an established API has its own hazards; no existing code will be expecting that exception, so surprising explosions might result. But, for a problem that should only be possible during the bootstrap process, Coghlan believes that this is the best approach:

This proposal also envisions adding a function to the in-development "secrets" module — secrets.wait_for_system_rng() — that would simply block until the system's entropy pool is fully initialized and ready. The small (possibly nonexistent) body of code that breaks with unhandled BlockingIOError exceptions could call this function to ensure the availability of strong random data from os.urandom().

It is not clear when a decision between these two proposals will be made. It is worth noting, though, that Coghlan has indicated that he is happy enough with Stinner's proposal that he can support it should that be the one that is accepted in the end. So the discussion may have been long and painful, but the end result should be strong random data in Python in a way that the community as a whole is able to agree upon. Hopefully that means everybody can rest and prepare for the inevitable debate over whether this change should be backported to Python 2.