Michael Droettboom wrote: > Interesting... > > When you get a chance, would you mind running the attached script? This > is how I was finding object leaks before. It takes a single commandline > argument that is the number of iterations. Can you send me the outputs > from 1 and 2 iterations? That way we should be able to see what type of > object is being leaked, which is a good first step. And here is the result of the script modified for gtk: efiring@manini:~/programs/py/mpl/tests$ python memleak_gui_gtk.py 1 55352 55417 *** <type 'gtk.gdk.Colormap'> *** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'> *** <class 'gtk._gtk.WidgetFlags'> *** <class 'gtk._gtk.WidgetFlags'> *** <type 'gtk.gdk.Window'> *** <class 'matplotlib.backends.backend_gtk.FigureCanvasGTK'> *** <class 'matplotlib.backends.backend_gtk.NavigationToolbar2GTK'> *** <type 'gtk.gdk.Pixmap'> *** <type 'gtk.Tooltips'> *** <type 'gtk.Label'> *** <type 'gtk.Window'> *** <type 'gtk.VBox'> uncollectable list: [] efiring@manini:~/programs/py/mpl/tests$ python memleak_gui_gtk.py 2 55352 55421 *** <type 'gtk.gdk.Colormap'> *** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'> *** <class 'gtk._gtk.WidgetFlags'> *** <class 'gtk._gtk.WidgetFlags'> *** <type 'gtk.gdk.Window'> *** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'> *** <class 'gtk._gtk.WidgetFlags'> *** <class 'gtk._gtk.WidgetFlags'> *** <type 'gtk.gdk.Window'> *** <class 'matplotlib.backends.backend_gtk.FigureCanvasGTK'> *** <class 'matplotlib.backends.backend_gtk.NavigationToolbar2GTK'> *** <type 'gtk.gdk.Pixmap'> *** <type 'gtk.Tooltips'> *** <type 'gtk.Label'> *** <type 'gtk.Window'> *** <type 'gtk.VBox'> uncollectable list: [] Eric
On Jul 5, 2007, at 2:13 PM, Michael Droettboom wrote: > > Interesting. I don't get that, but I do get some random segfaults (I > got lucky the first time I tested). It looks like wxPython doesn't retain a reference to the wxApp PyObj for you: kmcivor@400exp209:~/Projects/matplotlib-svn$ pythonw Python 2.4.4 (#1, Oct 18 2006, 10:34:39) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import wx >>> app = wx.PySimpleApp() >>> del app >>> wx.GetApp() Segmentation fault > I'm awfully surprised that wx.GetApp() would return an iterator, as > you > are getting, so maybe it's corruption of some sort? My guess is that Eric got lucky and ob_type was pointing to the listiterator's C type instance. > Since I didn't want to just put the wxapp global variable back in, > I assigned it to the figure that creates it, therefore stick around > as long as the figure does. (Is that the correct thing for its > lifetime?) I don't think this will work if you create two figures, destroy the first one, and then create another figure. Once created, the wxApp needs to exist for the life of the python process. I'll go ahead an put the global variable back in. > Also, I'm a little puzzled by this code in show() in backend_wx.py: > > wxapp = wx.GetApp() > if wxapp is not None: > # wxPython 2.4 has no wx.App.IsMainLoopRunning() method > imlr = getattr(wxapp, 'IsMainLoopRunning', lambda: False) > if imlr(): > wxapp.MainLoop() > > If I'm reading this correctly, shouldn't it be "if not imlr()"? Yes, it should be. I'll try to code with my eyes open from now on. :-/ Ken
Ken McIvor wrote: > On Jul 5, 2007, at 2:13 PM, Michael Droettboom wrote: >> >> Interesting. I don't get that, but I do get some random segfaults (I >> got lucky the first time I tested). > > It looks like wxPython doesn't retain a reference to the wxApp PyObj for > you: > > kmcivor@400exp209:~/Projects/matplotlib-svn$ pythonw > Python 2.4.4 (#1, Oct 18 2006, 10:34:39) > [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import wx > >>> app = wx.PySimpleApp() > >>> del app > >>> wx.GetApp() > Segmentation fault This qualifies as a wx bug, doesn't it? If wx doesn't retain the reference, then instead of a segfault shouldn't it raise an exception? Eric
On Jul 5, 2007, at 3:48 PM, Eric Firing wrote: > > This qualifies as a wx bug, doesn't it? I believe so. I'll file it. > If wx doesn't retain the reference, then instead of a segfault > shouldn't it raise an exception? I'd expect wx.GetApp() to work like the rest of wxPython and always return the wx.App instance. This has been fixed in revision 3463. Ken
Ken McIvor wrote: >> This qualifies as a wx bug, doesn't it? > I believe so. I'll file it. I agree - a segfault is ALWAYS a bug. >> If wx doesn't retain the reference, then instead of a segfault >> shouldn't it raise an exception? > > I'd expect wx.GetApp() to work like the rest of wxPython and always > return the wx.App instance. If a wx.App has not been created, it returns None: >>> import wx >>> wx.GetApp() >>> a = wx.GetApp() >>> print a None Which is probably what it should do if the wxApp() has been deleted. In any case, you can only create one wxApp per program instance, and it can not be destroyed and re-started, so keeping a global instance around is probably the way to go. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chr...@no...
More results: I've built and tested a more recent pygtk+ stack. (glib-2.12, gtk+-2.10.9, librsvg-2.16.1, libxml2-2.6.29, pygobject-2.13.1, pygtk-2.10.4...). The good news is that the C-level leaks I was seeing in pygtk 2.2 and 2.4 are resolved. In particular, using an SVG icon and Gdk rendering no longer seems problematic. I would suggest that anyone using old versions of pygtk should upgrade, rather than spending time on workarounds for matplotlib -- do you all agree? And my Gtk patch should probably be reverted to use an SVG icon for the window again (or to only do it on versions of pygtk > 2.xx). I don't know what percentage of users are still using pygtk-2.4 and earlier... There is, however, a new patch (attached) to fix a leak of FileChooserDialog objects that I didn't see in earlier pygtk versions. I have to admit that I'm a bit puzzled by the solution -- it seems that the FileChooserDialog object refuses to destruct whenever any custom Python attributes have been added to the object. It doesn't really need them in this case so it's an easy fix, but I'm not sure why that was broken -- other classes do this and don't have problems (e.g. NavigationToolbar2GTK). Maybe a pygtk expert out there knows what this is about. It would be great if this resolved the linear memory growth that Eric is seeing with the Gtk backend. GtkCairo seems to be free of leaks. QtAgg (qt-3.3) was leaking because of a cyclical reference in the signals between the toolbar and its buttons. (Patch attached). Qt4 is forthcoming (I'm still trying to compile something that runs the demos cleanly). I tried the FltkAgg backend, but it doesn't seem to close the window at all when the figure is closed -- instead I get dozens of windows open at once. Is that a known bug or correct behavior? Cheers, Mike Eric Firing wrote: > Michael Droettboom wrote: >> Eric Firing wrote: >>> So, this test is still showing problems, with similar memory >>> consumption in these three backends. >> Not necessarily. By default, Python allocates large pools from the >> operating system and then manages those pools itself (though its >> PyMalloc call). Prior to Python 2.5, those pools were never freed. >> With Python 2.5, empty pools, when they occur, are freed back to the >> OS. Due to fragmentation issues, even if there is enough free space >> in those pools for new objects, new pools may need to be created >> anyway, since Python objects can't be moved once they are created. >> So seeing modest increases in memory usage during a long-running >> Python application is typical, and not something that can be avoided >> wiinaccurate at finding memory leaksthout micro-optimizing for pool >> performance (something that may be very difficult). If memory usage >> is truly increasing in an unbounded way, then, yes, there may be >> problems, but it should eventually stabilize (though in a test such >> as memleak_gui that may take many iterations). It's more interesting >> to see the curve of memory usage over time than the average over a >> number of iterations. > > I agree. I just ran 2000 iterations with GtkAgg, plotted every 10th > point, and the increase is linear (apart from a little bumpiness) over > the entire range (not just the last 1000 iterations reported below): > > Backend GTKAgg, toolbar toolbar2 > Averaging over loops 1000 to 2000 > Memory went from 31248k to 35040k > Average memory consumed per loop: 3.7920k bytes > > Maybe this is just the behavior of pymalloc in 2.5? > > >> For further reading, see: >> http://evanjones.ca/python-memory.html >> README.valgrind in the Python source >> http://mail.python.org/pipermail/python-dev/2006-March/061991.html >> >> Because of this, using the total memory allocated by the Python >> process to track memory leaks is pretty blunt tool. More important >> metrics are the total number of GC objects (gc.get_objects()), GC >> garbage (gc.garbage), and using a tool like Valgrind or Purify to >> find mismatched malloc/frees. Another useful tool (but I didn't >> resort to yet with matplotlib testing) is to build Python with >> COUNT_ALLOCS, which then gives access to the total number of mallocs >> and frees in the Python interpreter at runtime. >> >> IMO, the only reasonable way to use the total memory usage of Python >> to debug memory leaks is if you build Python without pool allocation >> (--without-pymalloc). That was how I was debugging memory leaks last >> week (in conjunction with valgrind, and the gc module), and with that >> configuration, I was only seeing memory leakage with Pygtk 2.4, and a >> very small amount with Tk. Are your numbers from a default build? >> If so, I'll rebuild my Python and check my numbers against yours. If >> they match, I suspect there's little we can do. > > I used stock Python 2.5 from ubuntu Feisty. I should compile a > version as you suggest, but I haven't done it yet. > > Eric > >> >> Cheers, >> Mike >> > >
Eric Firing wrote: > > I just committed a change to the output formatting of memleak_gui so > that if you redirect it to a file, that file can be loaded with > pylab.load() in case you want to plot the columns. (At least this is > true if you don't use the -c option.) > Great. Sorry for stomping on that ;) > Yesterday, before your commits, I compared memleak_gui with stock > Python 2.4 versus stock 2.5 (both from ubuntu feisty) and found very > little difference in the OS memory numbers. Are they still increasing linearly? I'm still seeing some mystery leaks with Gtk, Qt4 and (much smaller) on Tk. Qt and Wx seem fine here. Unfortunately Qt4 crashes valgrind, so it's not of much use. I'm curious whether your results match that. I'm not terribly surprised that 2.4 isn't different from 2.5, since the case in which entire memory pools are freed in 2.5 is probably hard to trigger. Cheers, Mike
Eric Firing wrote: > Attached are runs with gtk, wx, qtagg, and tkagg. Quite a variety of > results: tkagg is best, with only slow memory growth and a constant > number of python objects; qtagg grows by 2.2k per loop, with no > increase in python object count; wx (which is built on gtk) consumes > 3.5k per loop, with an increasing object count; gtk consumes 1.8k per > loop with an increasing object count. > > All runs are on stock ubuntu feisty python 2.5. Thanks for these results. Unfortunately, I'm seeing different results here. [dagnabbit!] None of them have an increasing object count for me, which leads me to suspect there's some version difference between your environment and mine that isn't being accounted for. Gtk[Agg|Cairo] -- 1.3k per loop. Wx[Agg] -- 0.010k per loop QtAgg -- 2.3k per loop (which is in the same ballpark as your result) Qt4Agg -- 1.4k per loop (which seems to be in the same ballpark as Darren Dale's result) TkAgg -- 0.29k per loop I don't know if the size of memory per loop is directly comparable between your environment and mine, but certainly the shape of the curve, and whether the number of Python objects is growing is very relevant. I made some more commits to SVN on 07/03/07 necessary for recent versions of gtk+ and qt. Did you (by any chance) not get those patches? It would also be interesting to know which versions of the toolkits you have, as they are probably different from mine. Is it safe to assume that they are all the stock Ubuntu feisty packages? In any case, I have updated memleak_gui.py to display the relevant toolkit versions. I've also attached a script to display the toolkit versions. Its output on my machine is: # pygtk version: (2, 10, 4), gtk version: (2, 10, 9) # PyQt4 version: 4.2, Qt version 40300 # pyqt version: 3.17.2, qt version: 30303 # wxPython version: 2.8.4.0 # Tkinter version: $Revision: 50704 ,ドル Tk version: 8.4, Tcl version: 8.4 Cheers, Mike
Eric, It turns out that the key difference is in pygobject. I was using 2.13.2 (from source) here at work, and Ubuntu Feisty currently ships 2.12.3. 2.13.x is an unstable branch, of course, (gnome-related packages follow the even/odd versioning scheme), so I shouldn't have been using it to test with in the first place... ...however, there are only about 40 patches between 2.12.3 and 2.13.2. Two of them are very crucial memory leak patches: http://svn.gnome.org/viewcvs/pygobject?limit_changes=100&view=revision&revision=653 http://svn.gnome.org/viewcvs/pygobject?limit_changes=100&view=revision&revision=642 When I patched 2.12.3 against these patches, the Python object leaks go away. There may be workarounds matplotlib can make -- I'll look into that now. Hopefully the workarounds won't be too problematic -- I would hate to complicate matplotlib code for the sake of something that should resolve itself in the next Gnome release cycle... I updated the memleak_gui.py script to display the pygobject/glib versions as well when testing the gtk backend. Cheers, Mike Michael Droettboom wrote: > Eric Firing wrote: > >> Michael Droettboom wrote: >> >>> Interesting... >>> >>> When you get a chance, would you mind running the attached script? >>> This is how I was finding object leaks before. It takes a single >>> commandline argument that is the number of iterations. Can you send >>> me the outputs from 1 and 2 iterations? That way we should be able >>> to see what type of object is being leaked, which is a good first step. >>> >> And here is the result of the script modified for gtk: >> >> efiring@manini:~/programs/py/mpl/tests$ python memleak_gui_gtk.py 1 >> 55352 55417 >> *** <type 'gtk.gdk.Colormap'> >> *** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'> >> *** <class 'gtk._gtk.WidgetFlags'> >> *** <class 'gtk._gtk.WidgetFlags'> >> *** <type 'gtk.gdk.Window'> >> *** <class 'matplotlib.backends.backend_gtk.FigureCanvasGTK'> >> *** <class 'matplotlib.backends.backend_gtk.NavigationToolbar2GTK'> >> *** <type 'gtk.gdk.Pixmap'> >> *** <type 'gtk.Tooltips'> >> *** <type 'gtk.Label'> >> *** <type 'gtk.Window'> >> *** <type 'gtk.VBox'> >> >> uncollectable list: [] >> >> efiring@manini:~/programs/py/mpl/tests$ python memleak_gui_gtk.py 2 >> 55352 55421 >> *** <type 'gtk.gdk.Colormap'> >> *** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'> >> *** <class 'gtk._gtk.WidgetFlags'> >> *** <class 'gtk._gtk.WidgetFlags'> >> *** <type 'gtk.gdk.Window'> >> *** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'> >> *** <class 'gtk._gtk.WidgetFlags'> >> *** <class 'gtk._gtk.WidgetFlags'> >> *** <type 'gtk.gdk.Window'> >> *** <class 'matplotlib.backends.backend_gtk.FigureCanvasGTK'> >> *** <class 'matplotlib.backends.backend_gtk.NavigationToolbar2GTK'> >> *** <type 'gtk.gdk.Pixmap'> >> *** <type 'gtk.Tooltips'> >> *** <type 'gtk.Label'> >> *** <type 'gtk.Window'> >> *** <type 'gtk.VBox'> >> >> uncollectable list: [] >> >> Eric >> >> > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > Matplotlib-devel mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-devel > >
On 7/9/07, Michael Droettboom <md...@st...> wrote: > ...however, there are only about 40 patches between 2.12.3 and 2.13.2. > Two of them are very crucial memory leak patches: > When I patched 2.12.3 against these patches, the Python object leaks go > away. There may be workarounds matplotlib can make -- I'll look into > that now. Hopefully the workarounds won't be too problematic -- I would > hate to complicate matplotlib code for the sake of something that should > resolve itself in the next Gnome release cycle... I wouldn't spend any time on working around a leak that is already fixed in gobject. We just need to update the FAQ so GTK users will know that if they have a problem with leaks in GTK*, they need to upgrade. JDH
Eric Firing wrote: > I just updated from svn and tried to rerun the wx test, but ran into an > error: > > efiring@manini:~/programs/py/mpl/tests$ python > wxapp.Yield() > NameError: global name 'wxapp' is not defined I think I just saw a note that Ken had committed a patch that a user had provided that kept the wx back-end from re-starting an event loop if there was one already running -- maybe that has something to do with this bug? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chr...@no...
Yes -- the global wxapp variable was removed (a very good thing). I just committed a patch to fix this crash (r3460) Cheers, Mike Christopher Barker wrote: > Eric Firing wrote: > >> I just updated from svn and tried to rerun the wx test, but ran into an >> error: >> >> efiring@manini:~/programs/py/mpl/tests$ python >> wxapp.Yield() >> NameError: global name 'wxapp' is not defined >> > > I think I just saw a note that Ken had committed a patch that a user had > provided that kept the wx back-end from re-starting an event loop if > there was one already running -- maybe that has something to do with > this bug? > > -Chris > > > > >
Mike, New exception: efiring@manini:~/programs/py/mpl/tests$ python ../matplotlib_units/unit/memleak_gui.py -dwx -s1000 -e2000 > ~/temp/memleak_wx_0705.asc Traceback (most recent call last): File "../matplotlib_units/unit/memleak_gui.py", line 58, in <module> pylab.close(fig) File "/usr/local/lib/python2.5/site-packages/matplotlib/pylab.py", line 742, in close _pylab_helpers.Gcf.destroy(manager.num) File "/usr/local/lib/python2.5/site-packages/matplotlib/_pylab_helpers.py", line 28, in destroy figManager.destroy() File "/usr/local/lib/python2.5/site-packages/matplotlib/backends/backend_wx.py", line 1405, in destroy self.frame.Destroy() File "/usr/local/lib/python2.5/site-packages/matplotlib/backends/backend_wx.py", line 1364, in Destroy wxapp.Yield() AttributeError: 'listiterator' object has no attribute 'Yield' Eric
Interesting. I don't get that, but I do get some random segfaults (I got lucky the first time I tested). I'm awfully surprised that wx.GetApp() would return an iterator, as you are getting, so maybe it's corruption of some sort? Reverting to revision 3441 on backend_wx.py does resolve this issue for me, so it is related to removing the wxapp global variable. While I like the idea of removing global variables, that was problematic, since when the wxapp variable is dereferenced, the whole wx.App is destructed, (hence, I believe the segfaults). Since I didn't want to just put the wxapp global variable back in, I assigned it to the figure that creates it, therefore stick around as long as the figure does. (Is that the correct thing for its lifetime?) Anyway, it seems to fix memleak_gui.py for me. Ken and Tim will probably want to check that I didn't cause more mainloops to start than necessary in the process. Also, I'm a little puzzled by this code in show() in backend_wx.py: wxapp = wx.GetApp() if wxapp is not None: # wxPython 2.4 has no wx.App.IsMainLoopRunning() method imlr = getattr(wxapp, 'IsMainLoopRunning', lambda: False) if imlr(): wxapp.MainLoop() If I'm reading this correctly, shouldn't it be "if not imlr()"? If it is correct, maybe it needs a comment as to why mainloops should be started if a mainloop is already running. Cheers, Mike Eric Firing wrote: > Mike, > > New exception: > > efiring@manini:~/programs/py/mpl/tests$ python > ../matplotlib_units/unit/memleak_gui.py -dwx -s1000 -e2000 > > ~/temp/memleak_wx_0705.asc > Traceback (most recent call last): > File "../matplotlib_units/unit/memleak_gui.py", line 58, in <module> > pylab.close(fig) > File "/usr/local/lib/python2.5/site-packages/matplotlib/pylab.py", > line 742, in close > _pylab_helpers.Gcf.destroy(manager.num) > File > "/usr/local/lib/python2.5/site-packages/matplotlib/_pylab_helpers.py", > line 28, in destroy > figManager.destroy() > File > "/usr/local/lib/python2.5/site-packages/matplotlib/backends/backend_wx.py", > line 1405, in destroy > self.frame.Destroy() > File > "/usr/local/lib/python2.5/site-packages/matplotlib/backends/backend_wx.py", > line 1364, in Destroy > wxapp.Yield() > AttributeError: 'listiterator' object has no attribute 'Yield' > > > Eric >
I am about to update the memory leak question in the FAQ, but I thought I'd run it by the list first. I removed language that talked about much earlier releases of mpl, and the paragraph about leaks in older versions of Numeric and numarray. It seems like we should recommend numpy when the user experiences problems with Numeric or numarray, but I wanted to confirm that before adding a paragraph to that effect. The second question (below), can be ommitted -- it really belongs in a developer's FAQ. Any recommendations on a better place to post that information? <snip> matplotlib appears to be leaking memory, what should I do? First, determine if it is a true memory leak. Python allocates memory in pools, rather than one object at a time, which can make memory leaks difficult to diagnose. Memory usage may appear to increase rapidly with each iteration, but should eventually reach a steady state if no true memory leaks exist. (For more fine-grained memory usage reporting, you can build a custom Python with the --without-pymalloc flag, which disables pool allocation.) If after sufficient iterations, you still see that memory usage is increasing, it is a likely a bonafide memory leak that should be reported. memleak_gui.py <http://matplotlib.svn.sourceforge.net/viewvc/matplotlib/trunk/matplotlib/unit/memleak_gui.py?view=markup> (in the unit directory of the source tree), contains an example script useful for diagnosing and reporting memory leaks. Please use something like it when reporting leaks so we get an idea of the magnitude of the problem (i.e. bytes per figure). Also please provide your platform, matplotlib version, backend and as much information about the versions of the libraries you are using: freetype, png and zlib. It would also help if you posted code so I could look for any obvious coding errors vis-a-vis matplotlib. There are some known memory leaks in matplotlib-0.90.1 when used in conjunction with the Tk, Gtk, Wx, Qt and Qt4 GUI backends. Many of these leaks have resolutions in the current SVN version of matplotlib. However, the following library versions are known to have leaks that matplotlib triggers. If you have one of these versions and are experiencing memory leaks, you should upgrade your library. This is not an exhaustive list. * *Wx backend:*wxPython-2.8.2 or earlier in the 2.8 series * *Gtk backend:*pygobject-2.12.x, pygtk-2.4.0 I'd like to help diagnose a memory leak, rather than just report it. How do you recommend I do that? I thought you'd never ask! Python memory leaks tend to fall into one of the following categories: * *Uncollectable garbage:* The Python garbage collector is not infallible. It can not collect objects that the Python that have __del__ methods or weakrefs and contain cycles. If curious, you can read about this problem in horrific detail <http://svn.python.org/view/python/trunk/Modules/gc_weakref.txt?view=markup>. You can obtain a list of all uncollectable objects as follows: import gc print gc.garbage To see what cycles these objects participate in, there is a useful function in matplotlib.cbook: from matplotlib import cbook cbook.print_cycles(gc.garbage) This will print out all of the reference cycles that are preventing the uncollectable objects from being freed. The code should then be modified to prevent these cycles, or break the cycles during destruction. * *Real references:* Sometimes objects are legitimately being held onto by other Python objects. When this happens, you would see the total number of Python objects in the interpreter increase with each iteration of your test, even when you didn't intend any data to "stick around". You can print out the total number of objects in the interpreter: import gc print len(gc.get_objects()) By comparing the objects before and after your test, you can determine which of them remain between iterations: original_objects = [id(x) for x in gc.get_objects()] # ... do something that leaks objects new_objects = [x for x in gc.get_objects() if id(x) not in original_objects] You can then determine what is referencing those objects and causing them to stick around: print gc.get_referents(x) The code should be modified to prevent these unwanted references. * *C/C++ leaks in extension objects:* These is the classic problem of objects that are "malloc'd/new'd" and never "freed/deleted" in C or C++ parlance. There are many memory debuggers available such as Purify or Valgrind to help find these errors. I recommend reading the documentation about using these tools in conjunction with Python <http://svn.python.org/view/python/trunk/Misc/README.valgrind?view=markup> to avoid a lot of false positives.
On 7/9/07, Michael Droettboom <md...@st...> wrote: > > I am about to update the memory leak question in the FAQ, but I thought I'd > run it by the list first. I removed language that talked about much earlier > releases of mpl, and the paragraph about leaks in older versions of Numeric > and numarray. It seems like we should recommend numpy when the user > experiences problems with Numeric or numarray, but I wanted to confirm that > before adding a paragraph to that effect. vis-a-vis numpy, since we are dropping support entirely for Numeric and numarray, you can definitely recommend numoy :-) The looks very good, but I would also point them to memleak_hawaii or something to diagnose a pure image backend. Many times it is simply the user script that is failing (they fail to properly close a figure they have opened) so the FAQ should at least point them to an image generation script that is written correctly. The GUI leaks are are current problem, but they are 2nd order. We first need to make sure the pure MPL part is not leaking (it isn't currently, but it might one day if we screw something up in extension code) and once we've verified that, we can isolate the GUI leaks. Thanks, JDH
That is at least something to go by. ;) Thanks, Mike Eric Firing wrote: > Michael Droettboom wrote: >> Interesting... >> >> When you get a chance, would you mind running the attached script? >> This is how I was finding object leaks before. It takes a single >> commandline argument that is the number of iterations. Can you send >> me the outputs from 1 and 2 iterations? That way we should be able >> to see what type of object is being leaked, which is a good first step. > > And here is the result of the script modified for gtk: > > efiring@manini:~/programs/py/mpl/tests$ python memleak_gui_gtk.py 1 > 55352 55417 > *** <type 'gtk.gdk.Colormap'> > *** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'> > *** <class 'gtk._gtk.WidgetFlags'> > *** <class 'gtk._gtk.WidgetFlags'> > *** <type 'gtk.gdk.Window'> > *** <class 'matplotlib.backends.backend_gtk.FigureCanvasGTK'> > *** <class 'matplotlib.backends.backend_gtk.NavigationToolbar2GTK'> > *** <type 'gtk.gdk.Pixmap'> > *** <type 'gtk.Tooltips'> > *** <type 'gtk.Label'> > *** <type 'gtk.Window'> > *** <type 'gtk.VBox'> > > uncollectable list: [] > > efiring@manini:~/programs/py/mpl/tests$ python memleak_gui_gtk.py 2 > 55352 55421 > *** <type 'gtk.gdk.Colormap'> > *** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'> > *** <class 'gtk._gtk.WidgetFlags'> > *** <class 'gtk._gtk.WidgetFlags'> > *** <type 'gtk.gdk.Window'> > *** <class 'matplotlib.backends.backend_gtk.FileChooserDialog'> > *** <class 'gtk._gtk.WidgetFlags'> > *** <class 'gtk._gtk.WidgetFlags'> > *** <type 'gtk.gdk.Window'> > *** <class 'matplotlib.backends.backend_gtk.FigureCanvasGTK'> > *** <class 'matplotlib.backends.backend_gtk.NavigationToolbar2GTK'> > *** <type 'gtk.gdk.Pixmap'> > *** <type 'gtk.Tooltips'> > *** <type 'gtk.Label'> > *** <type 'gtk.Window'> > *** <type 'gtk.VBox'> > > uncollectable list: [] > > Eric >
I had no trouble reproducing this on my Ubuntu Feisty box. It turns out that wxPython leaks a dictionary for every object whose class subclasses a Wx class. There is a fix for this that made it into wxPython-2.8.3.0: http://cvs.wxwidgets.org/viewcvs.cgi/wxWidgets/wxPython/src/helpers.cpp.diff?r1=1.145&r2=1.145.4.1 I have verified this on my source-built wxPython-2.8.4.0. If I remove this line, I can reproduce the reference leak. ** I would recommend that anyone using a wxPython-2.8.x prior to wxPython-2.8.4 upgrade. There are binary packages available for a number of distributions on wxpython.org. ** As an aside, I filed a bug for this on Ubuntu launchpad. I don't know if this qualifies for the kind of fix they would normally make as a maintenance release, but I thought it was worth trying. https://bugs.launchpad.net/ubuntu/+source/wxwidgets2.8/+bug/124381 Cheers, Mike Eric Firing wrote: > Michael Droettboom wrote: >> Interesting... >> >> When you get a chance, would you mind running the attached script? >> This is how I was finding object leaks before. It takes a single >> commandline argument that is the number of iterations. Can you send >> me the outputs from 1 and 2 iterations? That way we should be able >> to see what type of object is being leaked, which is a good first step. > > efiring@manini:~/programs/py/mpl/tests$ python memleak_gui_wx.py 1 > 75891 76010 > *** <class 'wx._core.PySimpleApp'> > *** <class 'wx._core._wxPyDeadObject'> > > uncollectable list: [] > > efiring@manini:~/programs/py/mpl/tests$ python memleak_gui_wx.py 2 > GnomePrintCupsPlugin-Message: The ppd file for the CUPS printer > Dell424 could not be loaded. > GnomePrintCupsPlugin-Message: The ppd file for the CUPS printer pslj4m > could not be loaded. > 75891 76014 > *** <class 'wx._core.PySimpleApp'> > *** <class 'wx._core._wxPyDeadObject'> > > uncollectable list: [] > > > Eric >
My sincere apologies for the multiple copies of the e-mail sent this morning. I was getting "SMTP server down" messages, but clearly the messages were sent anyway. I'm not a spammer, really! ;) Mike