homepage

When a os.makedirs is used under an autofs directory, it crashes. For example, on my machine, `os.makedirs('/net/prodigy/tmp')` crashes with the following traceback:
Traceback (most recent call last):
 ...
 File "/usr/lib64/python2.7/os.py", line 157, in makedirs
 mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/net/prodigy/tmp'
In this case, '/net' is an autofs directory that automatically mounts the "prodigy" directory by connecting to a host called "prodigy" using NFS. The problem seems to be related to the fact that the "/net/prodigy" directory does not actually exist until it is first accessed.
I tried running `mkdir -p /net/prodigy/tmp`, and it succeeds even though the "/net/prodigy" directory did not exist before the "mkdir" command was run.
I'm not sure exactly how `mkdir -p` is implemented, but one potential workaround for Python's makedirs would be to add the following at the top of the function:
 os.stat(name)
This stat call really only needs to be run the first time makedirs is called (it does not need to be used for each recursive call).

As makedirs in 3.x doesn’t handle EPERM and is otherwise the same, I presume the error is there as well.
I also presume, that after the failed makedirs(), the directory is mounted? I'd just handle the error just we handle EEXIST in 3.x now.

Hynek: it doesn't seem like that would work, since legitimate EPERM errors are much more likely.

David: What do you mean? I'd just catch EPERM and look whether the directory "appeared" in the meantime. If it exists, continue. If not: re-raise. Am I missing something? I may have a look at GNU mkdir if it doesn't help.

To me, this doesn't look like a os.makedirs() bug, but rather an autofs one (didn't know it was still in use).
I don't like the idea of adding such a kludge (i.e. catching EPERM and checking whether the directory appeared in between).
Could you please provide an strace output of both "os.makedirs()" and "mkdir -p"?
Thanks.

Charles, I don't think you can blame autofs here. The problem at hand is that makedirs() never checks whether the directory exists (that would trigger the mount too I presume). Instead, it tries a mkdir and looks if it gets an EEXIST.
If you try that approach in this case where /net is non-writable and /net/prodigy appears only on demand, it fails with an EPERM instead.
Have a look at http://hg.python.org/cpython/file/8215aaccc9dd/Lib/os.py#l136 especially line 151.

> Charles, I don't think you can blame autofs here. The problem at hand is that makedirs() never checks whether the directory exists (that would trigger the mount too I presume).
Yes, it does. Have a look at line 148:
"""
 if head and tail and not path.exists(head):
"""
> Instead, it tries a mkdir and looks if it gets an EEXIST.
Actually, EEXIST is just caught to cope with race conditions (i.e. the
directory got created in between, TOCTTOU race).
> If you try that approach in this case where /net is non-writable and /net/prodigy appears only on demand, it fails with an EPERM instead.
Actually, no.
makedirs() does a recursive depth-first traversal:
makedirs('/net/prodigy/foo') will actually do something like:
"""
stat('/net/prodigy/foo') == ENOENT
stat('/net/prodigy') == ENONENT
mkdir('/net/prodigy') == EPERM
"""
The NFS mount should appear upon the first - or second - stat() call,
before mkdir().
But I'd like to be sure about that, that's why I think an strace()
output would be useful ;-)

Hynek: you said "just like EEXIST", which doesn't check to see if the directory exists before continuing, thus my confusion.
If mkdir -p does a stat first, then changing the makedirs algorithm to match is probably not a bad idea for OS compatibility reasons, with autofs just being an example of why.

>> Charles, I don't think you can blame autofs here. The problem at hand is that makedirs() never checks whether the directory exists (that would trigger the mount too I presume).
> 
> Yes, it does. Have a look at line 148:
> """
> if head and tail and not path.exists(head):
> """
Now that's embarrassing. I take everything back and claim the opposite.
 
>> If you try that approach in this case where /net is non-writable and /net/prodigy appears only on demand, it fails with an EPERM instead.
> 
> Actually, no.
> makedirs() does a recursive depth-first traversal:
> makedirs('/net/prodigy/foo') will actually do something like:
> """
> stat('/net/prodigy/foo') == ENOENT
> stat('/net/prodigy') == ENONENT
> mkdir('/net/prodigy') == EPERM
> """
> The NFS mount should appear upon the first - or second - stat() call,
> before mkdir().
> 
> But I'd like to be sure about that, that's why I think an strace()
> output would be useful ;-)
I glanced over coreutil's mkdir code and it seems that it changes the directory while creating the directories. I suspect it's related to that. At least it does no fstab parsing. ;) The code is wayyy too intricate to be grokable at glancing though. mkdir -p seems to be quite a can of worms.
Andrew, please send straces. :)

Andrew, are you still with us?

> Andrew, are you still with us?
I'm here, but it's been a busy few weeks. I'll see if I can spend some time on this today.

Some interesting information. If I do `os.mkdir('/net/prodigy/tmp')`, it gives "OSError: [Errno 13] Permission denied: '/net/prodigy/tmp'". However, if I instead do `os.mkdir('/net/prodigy/tmp/hi')`, it succeeds. (Note that I'm being careful to start from a fresh unmounted state in both cases).
I will post straces in a few minutes.

I guess this is the magic in mkdir -p:
mkdir("expert", 0755) = -1 EACCES (Permission denied)
chdir("expert") = 0
mkdir("tmp", 0755) = -1 EEXIST (File exists)
I'm not sure how I feel about that. I think we have more or less a policy in os & shutil not to change directories unless really necessary (like make_archive).

> I guess this is the magic in mkdir -p:
>
> mkdir("expert", 0755)          = -1 EACCES (Permission denied)
> chdir("expert")             = 0
> mkdir("tmp", 0755)           = -1 EEXIST (File exists)
>
> I'm not sure how I feel about that. I think we have more or less a policy in os & shutil not to change directories unless really necessary (like make_archive).
Calling chdir() should be avoided, as it is not thread safe.
Actually, the real problem is that stat() alone doesn't trigger
automount, to avoid mount storms e.g. when using find, see:
https://bugzilla.redhat.com/show_bug.cgi?id=497830
http://lkml.indiana.edu/hypermail/linux/kernel/1109.1/00100.html
Since calling chdir() is not an option, if we want to fix this, I
guess the way to go would be to open() the directory, and not use
path.exists().
But I find really uncomfortable with adding such a kludge, especially
since this problem is probably present in many different places
throughout the stdlib.
Furthermore, it seems that kernel from 2.6.38 do trigger automount
upon stat() (see
http://lkml.indiana.edu/hypermail/linux/kernel/1109.1/00210.html), so
I'd suggest to drop this as "won't fix".

I agree, closing.
Thank you for your valuable assistance Andrew!

This isn't fixed. All of the examples I've given were with a 3.3.0 kernel. Doing a stat would be a fix.

If I read it correctly, we _do_ a stat:
stat("/net/prodigy", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
mkdir("/net/prodigy/tmp", 0777) = -1 EACCES (Permission denied)
ISTM that this is a different problem altogether, or am I missing something?
If mkdir'ing of deeper directories works, it sounds like a race condition to me.

My suspicion re the race condition is that a chdir waits for autofs to mount but mkdir doesn't. You could check that yourself if a chdir/sleep would help.

Hmm. Maybe there's a difference between doing stat('/net/prodigy') vs. stat('/net/prodigy/tmp'). Just a guess, but maybe the former can succeed before the mount completes, but the latter has to wait for the mount to complete.

By the way, if my hunch about the difference in stat of '/net/prodigy' vs. '/net/prodigy/tmp' is correct, then this would explain why makedirs on deeper directories work. Specifically, one of the shallower stat calls would force the mount to complete before proceeding to the next deeper level.

Yes, creating the directories in a bottom-up way (i.e. '/net', '/net/prodigy', '/net/prodigy/foo') could maybe avoid this problem.
But this is definietely an autofs bug, and there are probably many other places where such code might break.

I sense a very easy fix: just do a stat() on the final path before we do anything (and quickly return if present). Moving current makedirs into a private _makedirs that gets called if the stat fails.
I'm still feeling uneasy about that though. It _is_ a kludge.

I agree that the proposed solutions are hacks, and still propose to close this as "won't fix".
I think this is a bug in Linux. mkdir("/net/prodigy/tmp") ought to trigger the mount, then fail with EEXIST.

Alright, closing for good then.
Andrew, if you want to get this fixed, you should report this to the autofs folks, because it's definitely not a Python bug.

I see no evidence that this is a bug in Linux, and I think it's ridiculous to close it when a trivial one-line fix is available. I won't reopen it because it's obvious no one wants to address this. :(

I posted a bug report with the kernel here:
https://bugzilla.kernel.org/show_bug.cgi?id=43262 

> I see no evidence that this is a bug in Linux,
"""
stat("/net/prodigy", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
mkdir("/net/prodigy/tmp", 0777) = -1 EACCES (Permission denied)
"""
As you can see, a stat() is already done on /net/prodigy.
Event if it wasn't done, calling mkdir() ought to trigger the mount.
So this is *definitely* as bug in autofs.
> and I think it's ridiculous to close it when a trivial one-line fix is available.
Which one-line fix do you propose?
> I won't reopen it because it's obvious no one wants to address this. :(
It's not that we don't want to address this, but rather that we want
to avoid introducing hacks to work around OS bugs.

> """
> stat("/net/prodigy", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> mkdir("/net/prodigy/tmp", 0777) = -1 EACCES (Permission denied)
> """
> 
> As you can see, a stat() is already done on /net/prodigy.
To be fair, that shouldn’t trigger a mount. Otherwise a `ls -l` on /net
would mount all volumes.
> Event if it wasn't done, calling mkdir() ought to trigger the mount.
I’m not sure if/where the behavior is defined – let’s what the Linux
people say.

>> """
>> stat("/net/prodigy", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
>> mkdir("/net/prodigy/tmp", 0777) = -1 EACCES (Permission denied)
>> """
>>
>> As you can see, a stat() is already done on /net/prodigy.
>
> To be fair, that shouldn’t trigger a mount. Otherwise a `ls -l` on /net
> would mount all volumes.
Not sure what "that" is: my view is that mkdir should trigger the mount,
/net/prodigy is already there and available. ls -l doesn't invoke mkdir(2),
so you wouldn't get a mount storm when it is mkdir that triggers the mount.

>>> """
>>> stat("/net/prodigy", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
>>> mkdir("/net/prodigy/tmp", 0777) = -1 EACCES (Permission denied)
>>> """
>>>
>>> As you can see, a stat() is already done on /net/prodigy.
>>
>> To be fair, that shouldn’t trigger a mount. Otherwise a `ls -l` on /net
>> would mount all volumes.
> 
> Not sure what "that" is: my view is that mkdir should trigger the mount,
> /net/prodigy is already there and available. ls -l doesn't invoke mkdir(2),
> so you wouldn't get a mount storm when it is mkdir that triggers the mount.
Sure, I was refering (as I hoped that my quoting would indicate) to the
stat on /net/prodigy, not the mkdir. I commented on the mkdir in the
next paragraph.

> Which one-line fix do you propose?
Doing a stat("/net/prodigy/tmp") before mkdir("/net/prodigy/tmp") is an extremely simple workaround.
Of course, I would love to see clear documentation of how the kernel is defined to behave in this situation. It would certainly be intuitive for mkdir("/net/prodigy/tmp") to force a mount of "/net/prodigy", but defined behavior isn't always intuitive. :)

> Of course, I would love to see clear documentation of how the kernel 
> is defined to behave in this situation. It would certainly be 
> intuitive for mkdir("/net/prodigy/tmp") to force a mount of 
> "/net/prodigy", but defined behavior isn't always intuitive. :)
Maybe I'm confused, but the presence of "/net/prodigy" is *not*
the issue here, and what gets mounted is *not* "/net/prodigy",
but "/net/prodigy/tmp" (do "mount" to confirm or dispute).

> Maybe I'm confused, but the presence of "/net/prodigy" is *not*
> the issue here, and what gets mounted is *not* "/net/prodigy",
> but "/net/prodigy/tmp" (do "mount" to confirm or dispute).
No, /net/prodigy is the mountpoint in this case:
amcnabb@sage:~ :) mount |grep prodigy
prodigy:/ on /net/prodigy type nfs4 ([redacted])

I still don't like the idea of adding such a kludge to work around OS
bugs, so I'd suggest closing as won't fix.

+1 for closing.