Python and crypto-strength random numbers by default

Please consider subscribing to LWN

Subscriptions are the lifeblood of LWN.net. If you appreciate this content and would like to see more of it, your subscription will help to ensure that LWN continues to thrive. Please visit this page to join up and keep LWN on the net.

There are various types of random number generators (RNGs) that target different use cases, but a programming language can only have one default. For high-security random numbers (e.g. cryptographic keys and the like), it is a grievous error to use the wrong kind of RNG, while other use cases are typically more forgiving. The Python community is in the middle of a debate about how it should be handling random numbers within the language's standard library.

The random module

On the python-ideas mailing list, Guido van Rossum kicked off the discussion by noting that he had been contacted by OpenBSD founder Theo de Raadt about the random.random() RNG that ships with Python. That RNG is based on the Mersenne Twister (MT) algorithm, which is a pseudo-RNG (PRNG) with an enormously long period. The PRNG can either be seeded by the user (with a call to random.seed()) or, by default, it is seeded with 2500 bytes of entropy from the os.urandom() pool (the strong, non-blocking random pool supplied by the operating system). From a given seed, the same sequence of random numbers will be generated, which makes it deterministic.

Using MT-derived random numbers for cryptographic or other security purposes is definitely not recommended by security experts. Given enough output from the RNG, the seed can be recovered, which means that all further random numbers can be perfectly predicted—generally with disastrous results. But for many uses, such as games, testing, or simulations, the random module provides a simple-to-use way to get perfectly adequate random numbers. The Python documentation for the module clearly indicates that it should not be used for security purposes and suggests either os.urandom() or the random.SystemRandom class.

But Van Rossum noted that "as the meme goes -- nobody reads the docs", so De Raadt's concern is that people will simply reach for the default provided by the language—no matter what they are using the random numbers for. According to Van Rossum, De Raadt is advocating the arc4random() RNG that OpenBSD uses (and, in fact, has switched to by default). Some of that advocacy can be seen in a forwarded email message that Van Rossum posted. Though De Raadt was invited to participate in the thread, "he's too busy" to do so, which left Van Rossum in a bit of a quandary:

That set off quite a discussion, which now spans four threads in python-ideas.

Donald Stufft responded that he sees a problem with the current approach, but that he doesn't have any "great solution" for it:

But, as Paul Moore and others pointed out, those with non-security needs for random numbers should be able to expect to access them easily from the standard library. He often uses the random module for games of various sorts, as well as for things like Monte Carlo simulations where millions of results are used. While the reproducibility feature is not particularly important to him, he would expect that a simple API would provide it. People doing crypto or other security programming ought to be looking for an RNG beyond the default:

Steven D'Aprano concurred:

Part of the problem, of course, is that there is a large body of existing code that expects the random module to behave in a certain way. Some of it requires the reproducibility feature and isn't being used for security purposes. Changing the default would break those programs a few years down the road (if the change follows the normal Python deprecation schedule). As Stephen J. Turnbull put it:

Stufft's proposal

That first thread got rather long, so Stufft attempted something of a reboot in a message that was rather long itself. He had talked to De Raadt to try to better understand his concerns, which Stufft said were not being reflected in the earlier thread. He broke down the users of random numbers into three groups: those who need deterministic output (and reproducibility), those who need cryptographic-strength random numbers (and don't want determinism), and those who aren't sure if their use case is security-sensitive or not. The last group generally doesn't need deterministic output, but currently that's what they get by default. It is Stufft's belief (that is shared with De Raadt, he said) that the members of that group would be better served with a crypto-strength RNG.

Those in group #3 might be told to use os.urandom() (or the random.SystemRandom class that is based on it), but they may end up using the standard random functions due to either the performance of the stronger alternative or bad advice (if, indeed, their need is security-sensitive). In fact, Stufft said, the performance of os.urandom() is one of the main reasons not to make it the default:

Stufft looked at StackOverflow and other sites with Python code to see what kinds of advice there was about RNG choices. His post had several snippets of code that used the module-level random functions in dubious ways.

That led him to propose that Python change its random-number handling in several ways. It would provide a means to access a fast user-space RNG like arc4random() and add a new class (random.SomeKindOfRandom) that uses it. The proposal would also move the MT-based RNG to a random.DeterministicRandom class and deprecate the module-level functions in the random module. That would mean that users would (eventually) have to choose the kind of randomness they wanted before they would be able to get random numbers. They would call functions like:

 random.SomeKindOfRandom().random()
 random.DeterministicRandom().random()

As might be guessed, Stufft's suggestions were not met with universal acclaim. There were concerns about backward compatibility and for how today's code could be switched, with minimal changes, to run (correctly) in this new world.

But Nick Coghlan strongly objected to the idea of deprecating all of the module-level functions. As he pointed out, nearly all of the module-level functions could provide any kind of random numbers, though they need to have good performance. Making someone choose what type they want, before they can even get a random number is "a *huge* regression in Python's usability for educational use cases". He said that a common "hello world" kind of program for using random numbers in Python is to roll a six-side die:

He continued:

He noted that there are calls at the module-level that imply the need for a deterministic RNG (like seed() and others that are stateful) and which can be deprecated to help those who really need them to move to an alternative API. Essentially, he is asking that those who don't care about the RNG wars (and those new to Python in particular) be left out—they can still call the functions they call today.

Breaking compatibility "in the name of the public good" is the wrong approach, Antoine Pitrou said. He and others are tired of seeing security trump all other considerations whenever a topic like this is raised. Since a change like Stufft has suggested can't fix the problem of bad advice on the internet, compromising on a change that maintains compatibility is preferable.

Coghlan's proposals

Coghlan started another thread as something of a "pre-PEP" on enhancing the random module. It would create three types of RNGs in Python: seedable, seedless, and system. The seedless version would be the default and guidance would be provided that those who need deterministic random numbers should use the seeded version, while those with strong security needs should use the system RNG.

Coghlan's proposal lays out plans for Python 3.6, which is likely destined for late 2016 (or early 2017), and for 3.7, which is probably due sometime in 2018—a change of this nature for Python can take a while, for sure. Essentially, his idea is to deprecate the stateful methods and module-level functions (seed(), getstate(), and setstate()), except for the SeedableRandom class. Those functions would warn the user, but not actually cause an error—except if the default type has been changed.

The most controversial part of Coghlan's proposal would provide a random.set_default_instance() function to specify which of the three types would be used by the module-level functions. Both Stufft and D'Aprano are concerned that would allow any module to affect all of the random numbers generated from that point on, potentially nullifying the choice another module has made.

Beyond that, though, Moore would like to see some kind of cost/benefit analysis. He is trying to remain open-minded, but is concerned about the disruption for users, particularly those who don't have any security requirements for their use of random numbers. While he recognizes that it is somewhat emotionally stated, the benefits to him seem to be: "Users of code written based on bad advice will be protected from the consequences (as long as the code runs on a sufficiently new version of Python)".

Based on some of that feedback, Coghlan went ahead and drafted PEP 504 that simplifies earlier proposals. It dispenses with the user-space CSPRNG (e.g. arc4random() or something similar) and changes the default to the system RNG (i.e. random.urandom()). If a program uses random.seed() or the other stateful calls (i.e. getstate() or setstate()), the random module will switch to the existing MT-based deterministic PRNG.

He proposes that this be done for 3.6, with a silent-by-default deprecation warning for the fallback to MT. In 3.7 it would instead be a visible-by-default runtime warning. In his announcement of the PEP, he said: "That approach would provide a definite security improvement over the status quo, while restricting the compatibility break to a performance regression in applications that use the module level API without calling seed(), getstate() or setstate()."

Van Rossum's reaction

But it turns out that Van Rossum is not particularly happy with that PEP as the end result of the discussion. He stopped following the "mega-threads", but doesn't see much value in changing things:

Though Stufft reiterated his belief that the existing module "guides you towards using an insecure source of random numbers rather than a secure one", Van Rossum was not impressed with that argument:

The discussion is continuing at the time of this writing, but there is a sense that adding access to a fast user-space PRNG, perhaps based on ChaCha20 (like arc4random()), may be in the offing. A change in the default would leave some users and programs behind eventually, but it would seem that most believe users who require deterministic random numbers make up a small minority. Van Rossum, though, seems unconvinced that there is much urgency for a solution. He suggested adding the new method for 3.6, but waiting "a few releases" to decide if any change to the default was warranted. "Security isn't served well by panicky over-reaction", he said.

There is plenty of time left in the 3.6 development schedule, so perhaps other proposals and ideas will eventually win the day. But with Van Rossum firmly opposed to the changes that have been proposed so far, a big change seems unlikely for 3.6. Only time will tell.

Note: you can avoid this step in the future by logging into your LWN account.