homepage

> cat x.py
class X(object):
 def __init__(self, x):
 self.x = x
 print 'creating X(%r)' % x
 def __del__(self):
 print 'deleting X(%r)' % self.x
class A(object):
 x = X('new')
class B:
 x = X('old')
> python x.py
creating X('new')
creating X('old')
deleting X('old')
Python 2.4.2 (#2, Jan 13 2006, 12:00:38)
Python 2.6a0 (trunk:51513M, Aug 23 2006, 14:17:11)

Logged In: YES 
user_id=835142
It looks like the class object is not deleted alltogether:
class X(object):
 def __init__(self, x):
 self.x = x
 print 'creating X(%r)' % x
 def __del__(self):
 print 'deleting X(%r)' % self.x
 
class A(object):
 x = X('new')
del A
Output:
creating X('new')
deleting X('new')

Logged In: YES 
user_id=849994
Note that new-style classes are always part of a reference
cycle (you can find this out via gc.get_referrers).
Therefore, they will not be deleted instantly, but only
after gc collects them (you can trigger that via gc.collect).

Logged In: YES 
user_id=835142
Yes, I've found that (using gc.get_referrers!), but this
does not explain why A is not cleaned up when the program exits.
Note that if I put class A definition inside a function, it
does get cleaned up. Must be some funny interation between
module and new-style class objects.

Logged In: YES 
user_id=849994
There's also this sentence in the __del__ docs:
"""
It is not guaranteed that __del__() methods are called for
objects that still exist when the interpreter exits.
"""

Logged In: YES 
user_id=835142
I used __del__ just to illustrate the problem. In real life
application, X was a type defined in a C module and I've
noticed that it's tp_dealloc is not called on instances
assigned to class variables.
BTW, what are the circumstances when __del__() methods are
not called for objects that still exist when the interpreter
exits?

Logged In: YES 
user_id=764593
The funny interaction with modules is probably that the 
module retains a reference to the class (and vice versa), 
so the class can't go away until the module does -- and a 
module in sys.modules can't go away.
The __del__ methods are not called if the interpreter can't 
decide which to call first. For example, if
 A.attr=B
 B.attr=A
then A and B form a cycle (like the class and its defining 
module). If only one has a __del__ method, it gets called, 
but if both do, then python doesn't know which to call 
first, so it never calls either.
You may have a cycle like
module <==> class <==>instanceA
 \ <==>instanceB
So that it can't decide whether to take care of instanceA 
or instanceB first.
Or it might be that the __del__ methods actually are being 
called, but not until module teardown has begun, so they 
don't work right.

Logged In: YES 
user_id=764593
Looking at your example code (as best I can guess about 
indentation), it looks like
module x <==> class X
module x <==> class A ==> A's instance x ==> class X
module x <==> class B ==> B's instance x ==> class X
So the x instances can't go away until A and B do, which 
means at module cleanup. But when the module cleans up, it 
may well clean up X before A, so that A.x no longer has an 
active class, so that it can't find its __del__ method.

Logged In: YES 
user_id=764593
I suggest changing status to Pending Close - not a bug.

Logged In: YES 
user_id=835142
Is it true that a class retains reference to the module? The '__module__' attribute 
is a string, not a reference to the module. Maybe you are talking about 
something else ...

Logged In: YES 
user_id=764593
More precisely, it retains a reference to the module's 
__dict__ as its globals. At the moment, I'm not finding 
proof that this happens directly in the class, but it does 
happen in class methods -- including __del__.

In the light of no further results, closing this bug.

The problem still exists in 2.5.1.
The explanations given so far are not correct. With x.py as before (see 
attached):
>>> import sys, gc, x
creating X('new')
creating X('old')
>>> del x,sys.modules['x']
deleting X('old')
>>> gc.collect()
deleting X('new')
6
which shows that the cycles in x module are resolvable by GC.
The problem is not that there are uncollectable objects but that GC is 
ran on exit before x becomes dead.
>>> import sys, gc, x
creating X('new')
creating X('old')
>>> gc.set_debug(1)
>>> sys.exit()
gc: collecting generation 2...
gc: objects in each generation: 463 2034 0
gc: done.
deleting X('old')
Looking at the comments in Py_Finalize, it looks like GvR intended to 
run GC after destroying the modules, but it led to problems:
(from svn blame Python/pythonrun.c)
 32278 gvanrossum 
 9025 guido /* Destroy all modules */
 8403 guido PyImport_Cleanup();
 9025 guido 
 32278 gvanrossum /* Collect final garbage. This disposes of 
cycles created by
 34776 tim_one * new-style class definitions, for example.
 34776 tim_one * XXX This is disabled because it caused too 
many problems. If
 34776 tim_one * XXX a __del__ or weakref callback triggers 
here, Python code has
 34776 tim_one * XXX a hard time running, because even the sys 
module has been
 34776 tim_one * XXX cleared out (sys.stdout is gone, 
sys.excepthook is gone, etc).
 34776 tim_one * XXX One symptom is a sequence of information-
free messages
 34776 tim_one * XXX coming from threads (if a __del__ or 
callback is invoked,
 34776 tim_one * XXX other threads can execute too, and any 
exception they encounter
 34776 tim_one * XXX triggers a comedy of errors as subsystem 
after subsystem
 34776 tim_one * XXX fails to find what it *expects* to find 
in sys to help report
 34776 tim_one * XXX the exception and consequent unexpected 
failures). I've also
 34776 tim_one * XXX seen segfaults then, after adding print 
statements to the
 34776 tim_one * XXX Python code getting called.
 34776 tim_one */
 34776 tim_one #if 0
 32278 gvanrossum PyGC_Collect();
 34776 tim_one #endif
Commenting out PyGC_Collect() seems like a too radical solution because 
no module referenced cycles get collected, not even those without 
__del__.
I have not tried it yet, but it looks like a possible solution is to 
call PyGC_Collect() at the end of _PyModule_Clear.

In PyImport_Cleanup(), sys and __builtin__ are the last ones deleted.
What if PyGC_Collect() is called just before?

amaury> What if PyGC_Collect() is called just before?
That would work. With the following patch:
===================================================================
--- Python/import.c	(revision 67183)
+++ Python/import.c	(working copy)
@@ -498,7 +498,10 @@
 			PyDict_SetItem(modules, key, Py_None);
 		}
 	}
-
+	/* Collect garbage remaining after deleting the
+	 modules. Mostly reference cycles created by new style
+	 classes. */
+ 	PyGC_Collect();
 	/* Next, delete sys and __builtin__ (in that order) */
 	value = PyDict_GetItemString(modules, "sys");
 	if (value != NULL && PyModule_Check(value)) {
$ ./python.exe x.py
creating X('new')
creating X('old')
deleting X('old')
deleting X('new')

Reopened because the history shows comments and patches months after it was set to closed and won't fix, see msg61886.

This is superseded by issue812369, but as a stop-gap measure, I don't see any downside of applying gc-import.patch.

Does anyone want to weigh in on this? I am merging in the issue812369 nosy list.
I would like to either apply gc-import.patch or close this as superseded by issue812369.

looks harmless to me. though i think issue812369 looks okay as well at first glance.

Here is a patch adding a call to gc.collect() after cleaning up most modules, with tests.

Here is an updated patch after the latest changes on default.

New changeset f0833e6ff2d2 by Antoine Pitrou in branch 'default':
Issue #1545463: Global variables caught in reference cycles are now garbage-collected at shutdown.
http://hg.python.org/cpython/rev/f0833e6ff2d2 

This issue is endly fixed in 3.4. Since changing the shutdown sequence is a delicate change, I won't backport to bugfix branches.

The test seems to be failing on Windows.

> The test seems to be failing on Windows.
Yes. I'll try to setup a new Windows dev environment and take a look :-/

I think the problem is that the __del__ method fails on Windows, maybe because sys.stdout and sys.__stderr__ have been replaced by None.
Consider the following program:
 import os
 class C:
 def __del__(self, write=os.write):
 write(1, b"BEFORE\n")
 print("__del__ called")
 write(1, b"AFTER\n")
 l = [C()]
 l.append(l)
On Unix I get
 BEFORE
 __del__ called
 AFTER
but on Windows I only get
 BEFORE
I would suggest using os.write() instead of print() in the tests.

I will try a fix.

On Windows my encoding for stdout, stderr is "cp1252" which is implemented in pure python.
By the time that _PyGC_DumpShutdownStats() runs the encoding.cp1252 module has been purged so stdout and stderr are broken.
I am afraid I will have to leave this to you Antoine...

Your diagnosis seems right about test_garbage_at_shudown
(I can reproduce under Linux using `PYTHONIOENCODING=iso8859-15 ./python -m test -v test_gc`).

Here is a patch, it seems to work on the custom buildbots. The problem was two-fold:
- PyErr_Warn() is too high-level, it will invoke linecache and others
- encodings and codecs shouldn't be cleared before the final shutdown

New changeset 8a5bebea9fec by Antoine Pitrou in branch 'default':
Issue #1545463: At shutdown, defer finalization of codec modules so that stderr remains usable.
http://hg.python.org/cpython/rev/8a5bebea9fec