homepage

I noticed regrtest claimed it cannot delete folder after test_tarfile
ran. It was like this.
Traceback (most recent call last):
 File "e:\python-dev\py3k\lib\runpy.py", line 160, in _run_module_as_main
 "__main__", fname, loader, pkg_name)
 File "e:\python-dev\py3k\lib\runpy.py", line 73, in _run_code
 exec(code, run_globals)
 File "e:\python-dev\py3k\lib\test\test_tarfile.py", line 1566, in <module>
 test_main()
 File "e:\python-dev\py3k\lib\test\test_tarfile.py", line 1563, in test_main
 shutil.rmtree(TEMPDIR)
 File "e:\python-dev\py3k\lib\shutil.py", line 283, in rmtree
 onerror(os.remove, fullname, sys.exc_info())
 File "e:\python-dev\py3k\lib\shutil.py", line 281, in rmtree
 os.remove(fullname)
WindowsError: [Error 32] プロセスはファイルにアクセスできません。別のプロセスが
使用中です。: 'E:\\PYTHON~1\\py3k\\@test_5276_tmp\\tmp.tar'
# It says "Process cannot access the file. Another process is using it."
I tried to reproduce this by running test_tarfile alone, but failed.
But I succeeded to do it with following command.
 py3k -m test.regrtest test_capi test_tarfile
(Probably there is timing problem like GC... The reason why I think
so is below)
I think _Stream object left opened after exception occured in (even
after returned from function Tarfile#open) because it lives in stack
frame. Because t._extfileobj is True by default, TarFile object won't close it explicitly.

Just to be sure: Have you checked this bug doesn’t apply to 2.7? Can you add a regression test?

I created separate small test case to reproduce, and I tried it on
Python2.7, I couldn't reproduce the issue. And tried it on Python3.1,
I couldn't there neither. (Sorry about that... I included 3.1 in
version box)
And exception might not be related. Because error still occurred
even if I removed "#" before exc_clear() in my attached script.
But if I removed "#" before gc.collect(), test case runs fine.
And I copied Python3.1's Lib/tarfile.py into Python3.2's Lib,
still error occurred. Maybe it's not problem of tarfile module
itself ....?

I think the tests should go into 2.7 and 3.1 even if they don’t have the bug. Adding Antoine to nosy, since he’s listed for gc in Misc/maintainers.rst.

I tried test_tar_pipe_open_read_error_v2.py on py3k,
I saw 3 uncollectable objects are reported. But they
are *collected* by gc.collect() without
gc.set_debug(gc.DEBUG_LEAK). (I'm not familiar to gc,
so maybe this is normal)
I didn't see uncollectable objects in release27-maint
and release31-maint.
And I replaced py3k's Lib/unittest with release31-maint's
one, the error was gone. Maybe problem is in Lib/unittest.
Actually, if I run the code in test_null_tarfile without
unittest, everything runs fine.
E:\python-dev>py3k test_tar_pipe_open_read_error_v2.py
.
----------------------------------------------------------------------
Ran 1 test in 0.050s
OK
===========================
gc: collectable <cell 00C4F968>
gc: collectable <tuple 00BF1968>
gc: collectable <function 00C53C98>
gc: collectable <tuple 00BF5C78>
gc: collectable <function 00C53CF8>
gc: collectable <tuple 00BF99A0>
gc: collectable <function 00C53D58>
gc: collectable <tuple 00C2D8C0>
gc: collectable <function 00C53DB8>
gc: collectable <tuple 00C37348>
gc: collectable <function 00C53E18>
gc: collectable <tuple 00C37460>
gc: collectable <function 00C53E78>
gc: collectable <tuple 00C37540>
gc: collectable <function 00C53ED8>
gc: collectable <function 00C53F38>
gc: collectable <type 00C73CE0>
gc: collectable <dict 00C4DD58>
gc: collectable <getset_descriptor 00C52AB8>
gc: collectable <getset_descriptor 00C52AF8>
gc: collectable <tuple 00AF7778>
gc: collectable <frame 00C28D70>
gc: collectable <method 00AE6278>
gc: collectable <frame 00C299A8>
gc: collectable <tuple 00BD4498>
gc: collectable <frame 00C29B50>
gc: collectable <frame 00C29CF8>
gc: collectable <frame 00C753A0>
gc: collectable <frame 00C75F40>
gc: collectable <tuple 00C375E8>
gc: collectable <frame 00C760E8>
gc: collectable <method 00C52B38>
gc: collectable <frame 00C766C8>
gc: collectable <_io.FileIO 00C52D38>
gc: collectable <_io.BufferedWriter 00BF0758>
gc: collectable <frame 00C78F80>
gc: collectable <tuple 00AEB9F8>
gc: collectable <_AssertRaisesContext 00C4FAF0>
gc: collectable <dict 00C554D0>
gc: collectable <frame 00C79328>
gc: collectable <TarFile 00C4FB98>
gc: collectable <frame 00C89390>
gc: collectable <dict 00C55770>
gc: collectable <list 00AEF978>
gc: collectable <frame 00C89780>
gc: collectable <frame 00C8A040>
gc: collectable <frame 00C89CC0>
gc: collectable <frame 00C89E70>
gc: collectable <traceback 00C56038>
gc: collectable <tuple 00C4F850>
gc: collectable <IOError 00C508F8>
gc: collectable <tuple 00C4FC08>
gc: collectable <ReadError 00C4CC50>
gc: collectable <method 00AE6338>
gc: collectable <tuple 00BF5150>
gc: collectable <tuple_iterator 00C4FA80>
gc: collectable <map 00C4FAB8>
gc: collectable <list 00AFAD38>
gc: uncollectable <_Stream 00C4FB28>
gc: uncollectable <dict 00C55620>
gc: uncollectable <_LowLevelFile 00C4FB60>
===========================
Traceback (most recent call last):
 File "test_tar_pipe_open_read_error_v2.py", line 25, in test_main
 unittest.main()
 File "e:\python-dev\py3k\lib\unittest\main.py", line 95, in __init__
 self.runTests()
 File "e:\python-dev\py3k\lib\unittest\main.py", line 231, in runTests
 sys.exit(not self.result.wasSuccessful())
SystemExit: False
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "test_tar_pipe_open_read_error_v2.py", line 33, in <module>
 test_main()
 File "test_tar_pipe_open_read_error_v2.py", line 30, in test_main
 shutil.rmtree(TEMPDIR)
 File "e:\python-dev\py3k\lib\shutil.py", line 283, in rmtree
 onerror(os.remove, fullname, sys.exc_info())
 File "e:\python-dev\py3k\lib\shutil.py", line 281, in rmtree
 os.remove(fullname)
WindowsError: [Error 32] プロセスはファイルにアクセスできません。別のプロセスが
使用中です。: 'c:\\docume~1\\ocean\\locals~1\\temp\\__foobarbaz__\\tmp.tar'
gc: 61 uncollectable objects at shutdown:
[54634 refs]

The error went away when I commented out following line.
Lib/unittest/case.py(133)
 self.exception = exc_value.with_traceback(None)
I found this by brute force.... I noticed that
test_tar_pipe_open_read_error_v2.py starts to fail from r75241.
It stores exc_value into _AssertRaisesContext#exception.
And when the exception was re-raised like this, (_Stream#_read)
 try:
 buf = self.cmp.decompress(buf)
 except IOError:
 raise ReadError("invalid compressed data")
IOError will be set to ReadError's *__cause__* attribute.
ReadError's traceback is cleared in unittest, but IOError's
traceback is not. It contains reference to frame which contains
_AssertRaisesContext object. And ReadError will be set to
_AssertRaisesContext's *exception* attribute, cyclic reference
is made. It cannot be freed without GC.
# gc.get_referents() was really helpful.
I'll attach the test script "test_assert_raises.py" to reproduce
the error. The result should be like this.
E:\python-dev>py3k test_assert_raises.py
--------------------------
[('foo 2',), None, RuntimeError('foo 1',)]
RuntimeError('foo 1',)
[('foo 1',), <traceback object at 0x00BD2738>]
<traceback object at 0x00BD2738>
--------------------------
.
----------------------------------------------------------------------
Ran 1 test in 0.090s
OK
===========================
gc: collectable <cell 00BD0B28>
gc: collectable <tuple 00BD0B60>
gc: collectable <function 00BCFE78>
gc: collectable <tuple 00BD0AF0>
gc: collectable <function 00BCFED8>
gc: collectable <tuple 00BD0BD0>
gc: collectable <function 00BCFF38>
gc: collectable <tuple 00BD0C08>
gc: collectable <function 00BCFF98>
gc: collectable <tuple 00BD0C40>
gc: collectable <function 00BD4038>
gc: collectable <tuple 00BD0C78>
gc: collectable <function 00BD4098>
gc: collectable <tuple 00BD0CB0>
gc: collectable <function 00BD40F8>
gc: collectable <function 00BD4158>
gc: collectable <type 00C07550>
gc: collectable <dict 00B1FAB8>
gc: collectable <getset_descriptor 00BD2678>
gc: collectable <getset_descriptor 00BD26B8>
gc: collectable <tuple 00AC53F8>
gc: collectable <frame 00C08858>
gc: collectable <method 00AC43F8>
gc: collectable <frame 00C09628>
gc: collectable <tuple 00BD0038>
gc: collectable <frame 00C097D0>
gc: collectable <frame 00C09978>
gc: collectable <frame 00C0C1B8>
gc: collectable <frame 00C0CEF0>
gc: collectable <tuple 00BD0E38>
gc: collectable <frame 00C0D098>
gc: collectable <method 00AC43B8>
gc: collectable <frame 00C0D810>
gc: collectable <_AssertRaisesContext 00BD0F18>
gc: collectable <frame 00C0DF18>
gc: collectable <tuple 00BD0EA8>
gc: collectable <RuntimeError 00B9FC50>
gc: collectable <traceback 00BD2738>
gc: collectable <tuple 00BD0F50>
gc: collectable <ValueError 00B9FC98>
gc: collectable <dict 00B22818>
gc: collectable <method 00AC4338>
gc: collectable <tuple 00BD0B98>
gc: collectable <tuple_iterator 00BD0E70>
gc: collectable <map 00BD0EE0>
gc: collectable <list 00B21938>
===========================
gc: 46 uncollectable objects at shutdown:
[45415 refs]

Here is the patch to fix this issue. (Please forget first patch)
E:\python-dev>py3k -m test.regrtest test_tarfile
[1/1] test_tarfile
1 test OK.
[85902 refs]
E:\python-dev>py3k test_assert_raises.py
--------------------------
[('foo 2',), None, RuntimeError('foo 1',)]
RuntimeError('foo 1',)
[('foo 1',), None]
None
--------------------------
.
----------------------------------------------------------------------
Ran 1 test in 0.050s
OK
===========================
gc: collectable <cell 00C824D0>
gc: collectable <tuple 00C82508>
gc: collectable <function 00C80E18>
gc: collectable <tuple 00C82498>
gc: collectable <function 00C80E78>
gc: collectable <tuple 00C82578>
gc: collectable <function 00C80ED8>
gc: collectable <tuple 00C825B0>
gc: collectable <function 00C80F38>
gc: collectable <tuple 00C825E8>
gc: collectable <function 00C80F98>
gc: collectable <tuple 00C82620>
gc: collectable <function 00C85038>
gc: collectable <tuple 00C82658>
gc: collectable <function 00C85098>
gc: collectable <function 00C850F8>
gc: collectable <type 00B78010>
gc: collectable <dict 00B31578>
gc: collectable <getset_descriptor 00C7CD78>
gc: collectable <getset_descriptor 00C7CDB8>
gc: collectable <tuple 00AC53F8>
===========================
gc: collectable <TextTestResult 00C827A8>
gc: collectable <_WritelnDecorator 00C82690>
gc: collectable <dict 00B316C8>
gc: collectable <dict 00B31818>
gc: collectable <list 00B1D3B8>
gc: collectable <list 00C7CC78>
gc: collectable <list 00B2B878>
gc: collectable <list 00C7CCF8>
gc: collectable <list 00B1D2B8>
gc: collectable <TestCase 00C82770>
gc: collectable <dict 00B314D0>
gc: collectable <dict 00B31770>
gc: collectable <list 00B1CF38>
gc: collectable <method 00AC4438>
gc: collectable <method 00AC45F8>
gc: collectable <method 00AC4638>
gc: collectable <method 00AC42F8>
gc: collectable <method 00AC45B8>
gc: collectable <method 00AC4538>
gc: 40 uncollectable objects at shutdown:
[45216 refs]

You shouldn't use DEBUG_LEAK except for debugging purposes :)
As the doc mentions: "To debug a leaking program call gc.set_debug(gc.DEBUG_LEAK). Notice that this includes gc.DEBUG_SAVEALL, causing garbage-collected objects to be saved in gc.garbage for inspection."
It's true that the message at shutdown should be eliminated in this case, though.
Anyway, I've committed a fix in r84798.

Here is the simple test case to demonstrate this issue.

Adding Michael for review of py3k_fix__AssertRaisesContext.patch.

Note that the original issue (test_tarfile failures on the Windows buildbots) now seems fixed thanks to the various tarfile and test_tarfile improvements.
I have no opinion on whether assertRaises should take care of cleaning the tracebacks or not. As the test_tarfile issue shows, the resource consumption issues were tied to actual bugs both in the library and in the test suite.

> Note that the original issue (test_tarfile failures on the Windows
> buildbots) now seems fixed thanks to the various tarfile and
> test_tarfile improvements.
Yes, thanks. :-)

The patch py3k_fix__AssertRaisesContext.patch looks good. A test would be nice. The code already attempts to sanitize the traceback, so sanitizing __cause__ and __context__ seems reasonable.

frame.clear() was committed in issue17934, it would allow a less brutal resolution.

See issue1565525 for the new helper function in the traceback module.

New changeset 6ab3193e890e by Antoine Pitrou in branch '3.4':
Issue #9815: assertRaises now tries to clear references to local variables in the exception's traceback.
http://hg.python.org/cpython/rev/6ab3193e890e
New changeset 553fe27521be by Antoine Pitrou in branch 'default':
Issue #9815: assertRaises now tries to clear references to local variables in the exception's traceback.
http://hg.python.org/cpython/rev/553fe27521be 

I've committed an alternate patch using traceback.clear_frames().