homepage

When running the following program I get frequent
errors like this one
Exception in thread Thread-4:
Traceback (most recent call last):
 File "C:\Python24\lib\threading.py", line 442, in
__bootstrap
 self.run()
 File "os.remove.py", line 25, in run
 os.remove(filename)
OSError: [Errno 13] Permission denied:
'c:\\docume~1\\joag\\locals~1\\temp\\tmpx91tkx'
When leaving out the touch statement(line 24) in the
loop of the class, I do not get any errors.
This is on Windows XP SP2 with python-2.4.2 (you should
have an exe touch somewhere in you path for this to
work) Can somebody shed any light on this please?
Thanks in advance
Joram Agten

Logged In: YES 
user_id=38376
If Python gives you a permission error, that's because
Windows cannot remove the file. Windows does, in general,
not allow you to remove files that are held open by some
process.
I suggest taking this issue to comp.lang.python. The bug
tracker is not the right place for code review and other
support issues.

Logged In: YES 
user_id=31435
The problem is that there's no reason to believe anything he
did here _does_ leave files open. I can confirm the
"permission denied" symptom, and even if I arrange for the
call to "touch" to run a touch.bat that doesn't even look at
the filename passed to it (let alone open or modify the file).
I also see a large number of errors of this sort:
Exception in thread Thread-8:
Traceback (most recent call last):
 File "C:\python24\lib\threading.py", line 442, in __bootstrap
 self.run()
 File "osremove.py", line 21, in run
 touch(filename)
 File "osremove.py", line 8, in touch
 stdin=None, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
 File "C:\python24\lib\subprocess.py", line 490, in __init__
 _cleanup()
 File "C:\python24\lib\subprocess.py", line 398, in _cleanup
 inst.poll()
 File "C:\python24\lib\subprocess.py", line 739, in poll
 _active.remove(self)
ValueError: list.remove(x): x not in list
Those are clearly due to subprocess.py internals on Windows,
where the poll() and wait() methods and the module internal
_cleanup() function aren't called in mutually threadsafe
ways. _Those_ errors can be stopped by commenting out the
_cleanup() call at the start of Popen.__init__() (think
about what happens when multiple threads call _cleanup() at
overlapping times on Windows: all those threads can end up
trying to remove the same items from _active, but only one
thread per item can succeed).
The "permission denied" errors persist, though.
So there's at least one class of subprocess.py Windows bugs
here, and another class of Windows mysteries. I believe
subprocess.py is a red herring wrt the latter, though. For
example, I see much the same if I use os.system() to run
`touch` instead.

Logged In: YES 
user_id=1276121
I did post on the python mailing list first, but got no 
responce there, after further looking into it, I seriously 
think there is at least one bug here.
here is the link to the post: 
http://mail.python.org/pipermail/python-list/2006-
February/323650.html

Logged In: YES 
user_id=38376
> The problem is that there's no reason to believe anything
> he did here _does_ leave files open.
Except that he's hitting the file system quite heavily, and
asyncronously. My guess is that Windows simply gets behind
(a quick filemon test indicates that this is indeed the
case; just before a crash, I see the events CREATE/SUCCESS,
QUERY/SUCCESS, QUERY/SUCCESS, WRITE/SUCCESS, and
OPEN/SHARING VIOLATION for the failing file, with lots of
requests for other files interleaved).
Unless someone wants to fix Windows, a simple workaround
would be to retry the os.remove a few times before giving up
(with a time.sleep(0.1) in between).

Logged In: YES 
user_id=31435
[/F]
> Except that he's hitting the file system quite heavily,
Except that _without_ the call to touch(), he's hitting it
even more heavily, creating and destroying little files just
as fast as the OS can do it in each of 10 threads -- but
there aren't any errors then.
> and asyncronously.
What's asynch here? The OP's touch() function waits for the
spawned process to terminate, and the test driver doesn't
try to delete the file until after that.
> My guess is that Windows simply gets behind
> (a quick filemon test indicates that this is indeed the
> case; just before a crash, I see the events
> CREATE/SUCCESS, QUERY/SUCCESS, QUERY/SUCCESS,
> WRITE/SUCCESS, and OPEN/SHARING VIOLATION for the
> failing file, with lots of requests for other files
> interleaved).
That's consistent with the symptom reported: an exception
raised upon trying to remove the file, but not during any
other file operation. Does it tell you more than _just_
that? It doesn't for me.
> Unless someone wants to fix Windows,
As above, because removing the call to the internal `touch`
function makes all problems go away it's not obvious that
this is a Windows problem.
> a simple workaround would be to retry the os.remove a
> few times before giving up (with a time.sleep(0.1) in
> between).
Because of the internal threading errors in subprocess.py
(see my first comment), the threads in the test program
still usually die, but with instances of list.remove(x)
ValueErrors internal to subprocess.py.
If I hack around that, then this change to the test
program's file-removal code appears adequate to eliminate
all errors on my box (which is a zippy 3.4 GHz):
 try:
 os.remove(filename)
 except OSError:
 time.sleep(0.1)
 os.remove(filename)
It's possible that some virus-scanning or file-indexing
gimmick on my box is opening these little files for its own
purposes -- although, if so, I'm at a loss to account for
why a single "os.remove(filename)" never raises an exception
when the `touch()` call is commented out.
OTOH, with the `touch()` call intact, the time.sleep(0.1)
above is not adequate to prevent os.remove() errors if I
change the file-writing code to:
 f.write("test" * 250000)
Even boosting the sleep() to 0.4 isn't enough then.
That does (mildly) suggest there's another process opening
the temp files, and doing something with them that takes
time proportional to the file size. However, the
os.remove() errors persist when I disable all such gimmicks
(that I know about ;-)) on my box.
It seems likely I'll never determine a cause for that. The
bad thread behavior in subprocess.py is independent, and
should be repaired regardless.

Logged In: YES 
user_id=38376
"Does it tell you more than _just_ that? It doesn't for me."
All requests against the file in question were issued by the
python process; there's no sign of virus checkers or other
external applications.
Also, whenever things failed, there were always multiple
requests for cmd.exe (caused by os.system) between the WRITE
request and the failing OPEN request.
 
My feel, after staring at filemon output, is that this is a
problem in the Windows file I/O layer. NTFS queues the
various operations, and calling an external process with
stuff still in the queue messes up the request scheduling.

Logged In: YES 
user_id=1276121
for the subprocess.py I did the following in a few places
try:
 _active.remove(self)
except:
 pass
see also bug 1199282
https://sourceforge.net/tracker/index.php?func=detail&aid=1199282&group_id=5470&atid=105470
in my current script I circumvent the "Permission denied"
error in the following way:
removed = False
while not removed:
try:
 os.remove(file)
 except OSError, error:
 logger.warning("could not remove file %s, %s"
%(file, error))
 time.sleep(1)
 else:
 removed = True
I also have a virus scanner (Mcafee, corporate stuff), and
still get the same behaviour when disabling the virus scanner.
>My feel, after staring at filemon output, is that this is a
>problem in the Windows file I/O layer. NTFS queues the
>various operations, and calling an external process with
>stuff still in the queue messes up the request scheduling.
this seems strange to me, since every thread works with its
own temp files, and all requests are send one after another
to the file I/O layer

Logged In: YES 
user_id=1276121
When running the same script, but now with the windows
remove function (cmd /c rm filename)
still problems occur, so maybe this is a windows problem
after all?
or does subprocess do things wrong?

Logged In: YES 
user_id=1276121
further looking into the problem:
when using the win32api calls to write to the file, this
seems to work.
see os.remove_win.py

Logged In: YES 
user_id=1276121
os.remove3.py
when using the os.write() function and os.close() function
to do the file writing, no problems are seen either.
so this seems to be a bug in the 'normal' file input output
functions under windows.
they don't seem to be thread safe in a way or another.
maybe the close() function returns too early?

Needs confirmation with recent versions, has nice tests.

Tested with Python 3.0.1 (r301:69561, Feb 13 2009, 20:04:18) [MSC v.1500
32 bit (Intel)] on win32 (windows xp sp2)
os.remove.py still gives the same error
Exception in thread Thread-4:
Traceback (most recent call last):
 File "c:\Python30\lib\threading.py", line 507, in _bootstrap_inner
 self.run()
 File "os.remove.py", line 25, in run
 os.remove(filename)
WindowsError: [Error 32] The process cannot access the file because it
is being used by another process:
'c:\\docume~1\\agtenjo\\locals~1\\temp\\tmpcwbddg'
os.remove2.py still gives the same error
c:\docume~1\agtenjo\locals~1\temp\tmpa3plim
The process cannot access the file because it is being used by another
process.

Tested with Python 3.0.1 (r301:69561, Feb 13 2009, 20:04:18) [MSC v.1500
32 bit (Intel)] on win32 (windows xp sp2) (pywin 213)
os.remove2_py30.py gives no error
os.remove_winpy30.py gives no error

os.remove2_py30.py gives no error
 should be
os.remove3_py30.py gives no error

touch.exe can be found here:
http://www.helge.mynetcologne.de/touch/index.htm#download 

Could this problem be associated with issue4749? It was found that 
something goes wrong when two cmd children processes are spawned from 
different threads, when the first exits, it is closing file handles 
shared with the first (or something like that) and it's causing a 
problem with logging in issue4749. That bug has been closed since it's 
not a problem with logging so I'm searching for other similar bugs to 
see if we can create a new bug that documents the cause and link to 
these other bugs that are all showing different symptoms of the bug. 
Thoughts?

@Brian, Tim, any views on this?

I think this c win32 issue describes a similar problem
http://stackoverflow.com/questions/3764072/c-win32-how-to-wait-for-a-pending-delete-to-complete 

Similar reference regarding the same basic behavior: http://blogs.msdn.com/b/oldnewthing/archive/2012/09/07/10347136.aspx
Short version: Indexing and anti-virus tools prevent deletion from occurring.
Longer version: 
DeleteFile (and all the stuff that ultimately devolves to DeleteFile) operate in a funny way on Windows. Internally, it opens a HANDLE to the file, marks it as pending deletion, and closes the HANDLE. If no one snuck in and grabbed another HANDLE to the file during that time, then the file is deleted when DeleteFile's hidden HANDLE is closed. Well designed anti-virus/indexing tools use oplocks ( http://blogs.msdn.com/b/oldnewthing/archive/2013/04/15/10410965.aspx ) so they can open a file, but seamlessly get out of the way if a normal process needs to take exclusive control of a file or delete it. Sadly "well-designed" is not a term usually associated with anti-virus tools, so errors like this are relatively commonplace.
Workarounds like using GetTempFileName() and MoveFile() to move the file out of the way will work, though I believe they introduce their own race conditions (the temp file itself is created but the HANDLE is closed immediately, which could mean a race to open the empty file by the bad anti-virus that would block MoveFile()).
Basically, if you're running on Windows, and you're using unfriendly anti-virus/indexing tools, there is no clean workaround that maintains the same behavior. You can't keep creating and deleting a file of the same name over and over without risking access denied errors.
That said, you could probably get the same results by opening and closing the file only once. Change from the original pseudocode:
while 1:
 with open(myfilename, ...) as myfile:
 myfile.write(...)
 do_stuff_with(myfilename)
 os.remove(myfilename)
to (assuming the default file sharing permissions are amenable):
with open(myfilename, ...) as myfile:
 while 1:
 myfile.write(...)
 myfile.flush()
 myfile.seek(0)
 do_stuff_with(myfilename)
 myfile.truncate()
Same basic pattern, except this time, you're rebuilding the same file over and over without ever leaving it unowned long enough for anti-virus/indexing to swoop in and steal it from you.

Please pretend I didn't leave off the actual os.remove call at the end of my second example that bypasses the issue.

Unless someone claims and can show that there is a bug in current cpython on current Windows (not xp), I think this should be closed. Josh, I interpret your post(s) as suggesting that there is not, or am I over-interpreting.

I think you're overinterpreting. The bug probably still exists on Windows if you're using a poorly designed anti-virus or indexing tool; nothing fundamental has changed in how files are deleted on Windows since this was opened.

I sometimes have problems deleting files in Windows Explorer. Sometimes I have to wait for reboot, sometimes I have to login as admin. None of this is a bug in Python. The original post, nearly 9 years ago, was not obviously a Python bug report but appears to be question about quirks in Windows. Two Python experts disagreed whether there is a Python bug.
In any case, an open issue needs a repeatable test case that demonstrates a problem with current Python running on a currently supported OS, preferably without 'poorly designed' 3rd party software involved.