homepage

Hi,
Lib/random.py has a fallback if os.urandom() returns NotImplementedError
---
from os import urandom as _urandom
...
 def seed(self, a=None):
 if a is None:
 try:
 a = long(_hexlify(_urandom(16)), 16)
 except NotImplementedError:
 import time
 a = long(time.time() * 256) # use fractional seconds
---
In 2.6, this is indeed what happens in Lib/os.py where "import urandom from os" gets [2]:
---
if not _exists("urandom"):
 def urandom(n):
...
 try:
 _urandomfd = open("/dev/urandom", O_RDONLY)
 except (OSError, IOError):
 raise NotImplementedError("/dev/urandom (or equivalent) not found")
---
however, in 2.7, things have shuffled around as a result of issue Issue #13703 and now _PyOS_URandom will return an OSError if it can't find /dev/urandom [3].
This means if you "import random" without "/dev/urandom" available it crashes trying to seed
I'm not sure if this is intentional? One easy solution would be to catch OSError in random.py and fall back then too
[1] http://hg.python.org/cpython/file/70274d53c1dd/Python/random.c#l227
[2] http://hg.python.org/cpython/file/9f8771e09052/Lib/os.py#l746
[3] http://hg.python.org/cpython/file/70274d53c1dd/Lib/random.py#l111 

Actually, this regression appeared after the Hash Randomization patches prior to 2.6.8, 2.7.3, 3.1.4 and 3.2.3.
Also, it not only breaks `from os import urandom`, but also prevents installation of many third-party packages that use setuptools or distribute, where the interpreter bails out with: "OSError: No such file or directory /dev/urandom" inside setup.py on all Tru64 machines, and HPUX 11.00 and 11.11 (at least).
As best I can tell it's failing either because dev_urandom_noraise aborts the interpreter if /dev/urandom is missing, or later an uncaught PyExc_OSError in dev_urandom_python triggers for the same reason. In either case there's no NotImplemented exception raised for the fallback code be used :(

The root of the problem preventing me from running some 3rd party setup.py scripts correctly is the mismatch between (recently) raising an OSError in Python/random.c, but catching only NotImplementedError in Lib/random.py.
For backwards compatibility (previous releases all raised and caught NotImplementedError), here is a patch that stops Python bailing out with "OSError: No such file or directory /dev/urandom" for me. Tested with Python-2.6.8, Python-2.7.3 and Python-3.2.3, on HPUX 11.00, HPUX 11.11 and Tru 64 5.1.
Arguably, as long as 3rd party code doesn't rely on the NotImplementedError exception, changing random.py to catch OSErrors would work equally well.
This patch does not stop dev_urandom_noraise() from halting the interpreter on machines with no /dev/urandom, but that seems intentional to me so I didn't try to fix it.

Looks like a critical regression to me.

As for the patch: looks good on the principle. However, PyExc_NotImplementedError has no "errno" or "filename", attribute, so you shouldn't use PyErr_SetFromErrnoWithFilename. Instead, simply call PyErr_SetString.

Hi Antoine,
Thanks for the heads up.
I've attached a revised patch that doesn't misuse PyErr_SetFromErrnoWithFilename.

I disagree that the regression is critical. IIUC, it fails on systems without urandom, such as Tru64 and HPUX. However, failure to support such systems is *not* critical, IMO; I think that OS-specific failures should be considered critical only if they occur on Linux, Windows, or OSX.
So I suggest that the priority of this issue is reduced.
More relevant than breaking HPUX is, IMO, that urandom is actually documented to raise NotImplementedError, so the patch looks good. For best compatibility, the actual spelling of the error message from 2.6 should be restored.
I'm puzzled by 
https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=KRNG11I
which claims that HPUX 11.11 (at least) *does* have /dev/urandom. Maybe your installation doesn't have KRNG11i in /etc/loadmods?
Also, the claim that it breaks Tru64 contradicts with
http://en.wikipedia.org/wiki//dev/random
which claims that Tru64 5.1B (at least) does have /dev/urandom.

An interesting question is whether the patch should be applied to 2.6 and 3.1. It is not a security fix in itself, which suggests that it shouldn't apply. OTOH, it's a follow-up to an earlier security fix.

I'm not sure what systems are defined as critical or not.
Although python is not really installable/configurable by end-users on ESXi, I noticed during development because we use python very early in the boot, before /dev/urandom appears for us (it comes from a kernel module loaded later).

We're running Tru64 UNIX 5.1A, not 5.1B which definitely doesn't have /dev/urandom.

We do not have KRNG11i installed. It did not ship with the original installation of HP-UX 11.11. It needs to be loaded after-the-fact and we cannot be ensured that our customers will have this module installed nor do we wish to make it a requirement.

New changeset 2e8b01583839 by Antoine Pitrou in branch '3.2':
Issue #15340: Fix importing the random module when /dev/urandom cannot be opened.
http://hg.python.org/cpython/rev/2e8b01583839
New changeset a53fc9982e2a by Antoine Pitrou in branch 'default':
Issue #15340: Fix importing the random module when /dev/urandom cannot be opened.
http://hg.python.org/cpython/rev/a53fc9982e2a 

New changeset edbf37ace03c by Antoine Pitrou in branch '2.7':
Issue #15340: Fix importing the random module when /dev/urandom cannot be opened.
http://hg.python.org/cpython/rev/edbf37ace03c 

Ok, this is now fixed in 3.2/3.3/2.7. I'll leave it to Martin and Benjamin whether this should be backported to 2.6 and 3.1.
(Georg, this changeset should probably be ported to 3.3.0 too)

Now picked into the 3.3.0 release clone as 6a782496f90a.

It's actually up to Barry (and Benjamin) to decide whether to apply this to 2.6 (and 3.1).

New changeset 6a782496f90a by Antoine Pitrou in branch 'default':
Issue #15340: Fix importing the random module when /dev/urandom cannot be opened.
http://hg.python.org/cpython/rev/6a782496f90a 

This bug also causes problems when you try to install Python in a Linux chroot environment or systemd-nspawn - before mounting devtmpfs.
For example, this Redhat bug "yum traceback with python-2.6.6-29.el6_2.2 and higher + missing /dev/urandom"
 * https://bugzilla.redhat.com/show_bug.cgi?id=893034 

2.6 and 3.1 don't receive bug fixes anymore, closing.