matplotlib-devel Mailing List for matplotlib (Page 21)

Eric Firing wrote:
> Attached are runs with gtk, wx, qtagg, and tkagg. Quite a variety of 
> results: tkagg is best, with only slow memory growth and a constant 
> number of python objects; qtagg grows by 2.2k per loop, with no 
> increase in python object count; wx (which is built on gtk) consumes 
> 3.5k per loop, with an increasing object count; gtk consumes 1.8k per 
> loop with an increasing object count.
>
> All runs are on stock ubuntu feisty python 2.5.
Thanks for these results. Unfortunately, I'm seeing different results 
here. [dagnabbit!] None of them have an increasing object count for 
me, which leads me to suspect there's some version difference between 
your environment and mine that isn't being accounted for.
Gtk[Agg|Cairo] -- 1.3k per loop.
Wx[Agg] -- 0.010k per loop
QtAgg -- 2.3k per loop (which is in the same ballpark as your result)
Qt4Agg -- 1.4k per loop (which seems to be in the same ballpark as 
Darren Dale's result)
TkAgg -- 0.29k per loop
I don't know if the size of memory per loop is directly comparable 
between your environment and mine, but certainly the shape of the curve, 
and whether the number of Python objects is growing is very relevant.
I made some more commits to SVN on 07/03/07 necessary for recent 
versions of gtk+ and qt. Did you (by any chance) not get those 
patches? It would also be interesting to know which versions of the 
toolkits you have, as they are probably different from mine. Is it safe 
to assume that they are all the stock Ubuntu feisty packages? In any 
case, I have updated memleak_gui.py to display the relevant toolkit 
versions. I've also attached a script to display the toolkit versions. 
Its output on my machine is:
# pygtk version: (2, 10, 4), gtk version: (2, 10, 9)
# PyQt4 version: 4.2, Qt version 40300
# pyqt version: 3.17.2, qt version: 30303
# wxPython version: 2.8.4.0
# Tkinter version: $Revision: 50704 ,ドル Tk version: 8.4, Tcl version: 8.4
Cheers,
Mike

I just tried to commit a rename of 'rcdefaults.py' to 'rcsetup.py', but
I got an error:
-------------
...$ svn commit -m"renamed rcdefaults.py to rccsetup.py to avoid conflict"
Sending matplotlib/__init__.py
Deleting matplotlib/rcdefaults.py
Adding matplotlib/rcsetup.py
svn: Commit failed (details follow):
svn: COPY of rcsetup.py: 403 Forbidden (https://svn.sourceforge.net)
-------------
If anybody knows what the reason for this might be, please let me know...
Greetings,
Norbert
Eric Firing wrote:
> Norbert Nemec wrote:
> 
>> Hmm - let me think.... We already have
>> rc
>> rcParams
>> rc_params
>> rcdefaults
>> rcParamDefaults
>> defaultParams
>> in the main module of maplotlib
>>
>> How about calling the new module 'rcdefaultparams.py', simply to make
>> the confusion complete and because I really feel that no other name
>> would fit the current "naming scheme" better... ;-)
>> 
>
> Yes, it is confusing, there are too many similar names. I suspect some 
> are used infrequently enough that we could change them without too much 
> pain.
>
> But the new module is really two things: 1) rc utilities (mainly 
> validation facilities) and 2) a set of default values. If these are 
> kept together the module could be called "rc_init.py" because everything 
> is mainly used for rc initialization, although there are things still in 
> mpl's __init__.py that are also part of the rc initialization. Or it 
> could be called "rc_utils.py" or "rcsetup.py". I would prefer any of 
> these to rcdefaultparams.py.
>
> Furthermore, even after factoring out the rc things as you have done the 
> mpl namespace is badly cluttered with things like checkdep_dvipng, 
> (which is actually part of the rc validation, so maybe it should be in 
> your new module) so still more refactoring and/or renaming might be in 
> order. I can imagine a class being used to good effect to organize the 
> whole business of rc handling.
>
> One more miscellaneous thought: shouldn't mpl.rc() be using the 
> validation functions instead of simply stuffing inputs into rcParams?
>
> I suppose this brings us back to the old "traits, properties, or 
> neither" question. But incremental improvements such as the one you 
> have made are still helpful.
>
> Eric
> 
>> Greetings,
>> Norbert
>>
>>
>>
>> John Hunter wrote:
>> 
>>> On 6/30/07, Norbert Nemec <Nor...@gm...> wrote:
>>> 
>>> 
>>>> Hi there,
>>>>
>>>> I just checked in some major reorganization work in __init__.py
>>>>
>>>> The main intention was to move the list of option defaults to a separate
>>>> file 'rcdefaults.py' that could be imported from setup.py to access the
>>>> settings with minimal dependencies on the remaining code.
>>>> 
>>>> 
>>> I haven't tested this but I did take a brief look at it and I think
>>> your cleaning and organizing is useful. I think we have a naming
>>> problem though -- this __init__ module defines an rcdefaults function,
>>> which is likely to cause confusion with the new rcdefaults module.
>>> Eg,
>>>
>>> >from matplotlib import rcdefaults
>>>
>>> will be ambiguous. You may want to consider a new name.
>>>
>>> DH
>>>
>>> -------------------------------------------------------------------------
>>> This SF.net email is sponsored by DB2 Express
>>> Download DB2 Express C - the FREE version of DB2 express and take
>>> control of your XML. No limits. Just data. Click to get it now.
>>> http://sourceforge.net/powerbar/db2/
>>> _______________________________________________
>>> Matplotlib-devel mailing list
>>> Mat...@li...
>>> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>>>
>>> 
>>> 
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by DB2 Express
>> Download DB2 Express C - the FREE version of DB2 express and take
>> control of your XML. No limits. Just data. Click to get it now.
>> http://sourceforge.net/powerbar/db2/
>> _______________________________________________
>> Matplotlib-devel mailing list
>> Mat...@li...
>> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>> 
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>
> 

When I was trying to use the pylab interface from a wxPython application, I
noticed that the application did not close completely after exiting. A
python process continued to run in the background even though no windows
were open.
The show command was starting a MainLoop() for the wx application even if
the MainLoop() was already running. When the application closed, this extra
MainLoop() kept running in the background. I was able to fix this problem in
my own install of matplotlib by only starting the wx MainLoop if the
MainLoop was not already running.
I've submitted a patch to correct this problem (Tracker #1747313). I hope
it's in the correct format.

On Tuesday 03 July 2007 04:33:46 pm Eric Firing wrote:
> Michael Droettboom wrote:
> > Eric Firing wrote:
> >> I just committed a change to the output formatting of memleak_gui so
> >> that if you redirect it to a file, that file can be loaded with
> >> pylab.load() in case you want to plot the columns. (At least this is
> >> true if you don't use the -c option.)
> >
> > Great. Sorry for stomping on that ;)
> >
> >> Yesterday, before your commits, I compared memleak_gui with stock
> >> Python 2.4 versus stock 2.5 (both from ubuntu feisty) and found very
> >> little difference in the OS memory numbers.
> >
> > Are they still increasing linearly? I'm still seeing some mystery leaks
> > with Gtk, Qt4 and (much smaller) on Tk. Qt and Wx seem fine here.
>
> Attached are runs with gtk, wx, qtagg, and tkagg. Quite a variety of
> results: tkagg is best, with only slow memory growth and a constant
> number of python objects; qtagg grows by 2.2k per loop, with no increase
> in python object count; wx (which is built on gtk) consumes 3.5k per
> loop, with an increasing object count; gtk consumes 1.8k per loop with
> an increasing object count.
>
> All runs are on stock ubuntu feisty python 2.5.
>
> Eric
>
> > Unfortunately Qt4 crashes valgrind, so it's not of much use.
> > I'm curious whether your results match that. I'm not terribly surprised
> > that 2.4 isn't different from 2.5, since the case in which entire memory
> > pools are freed in 2.5 is probably hard to trigger.
I am swamped at work, and have not been able to follow this thread closely. 
But I just updated from svn and ran memleak_gui.py with qt4:
# columns are: iteration, OS memory (k), number of python objects
#
 0 37364 53792
 10 37441 53792
 20 37441 53792
 30 37525 53792
 40 37483 53792
 50 37511 53792
 60 37539 53792
 70 37568 53792
 80 37596 53792
 90 37624 53792
 100 37653 53792
# columns above are: iteration, OS memory (k), number of python objects
#
# uncollectable list: []
#
# Backend Qt4Agg, toolbar toolbar2
# Averaging over loops 30 to 100
# Memory went from 37525k to 37653k
# Average memory consumed per loop: 1.8286k bytes
Darren

Michael Droettboom wrote:
> Eric Firing wrote:
>>
>> I just committed a change to the output formatting of memleak_gui so 
>> that if you redirect it to a file, that file can be loaded with 
>> pylab.load() in case you want to plot the columns. (At least this is 
>> true if you don't use the -c option.)
>>
> Great. Sorry for stomping on that ;)
>> Yesterday, before your commits, I compared memleak_gui with stock 
>> Python 2.4 versus stock 2.5 (both from ubuntu feisty) and found very 
>> little difference in the OS memory numbers.
> Are they still increasing linearly? I'm still seeing some mystery leaks 
> with Gtk, Qt4 and (much smaller) on Tk. Qt and Wx seem fine here. 
Attached are runs with gtk, wx, qtagg, and tkagg. Quite a variety of 
results: tkagg is best, with only slow memory growth and a constant 
number of python objects; qtagg grows by 2.2k per loop, with no increase 
in python object count; wx (which is built on gtk) consumes 3.5k per 
loop, with an increasing object count; gtk consumes 1.8k per loop with 
an increasing object count.
All runs are on stock ubuntu feisty python 2.5.
Eric
> Unfortunately Qt4 crashes valgrind, so it's not of much use.
> I'm curious whether your results match that. I'm not terribly surprised 
> that 2.4 isn't different from 2.5, since the case in which entire memory 
> pools are freed in 2.5 is probably hard to trigger.
> 
> Cheers,
> Mike

Eric Firing wrote:
>
> I just committed a change to the output formatting of memleak_gui so 
> that if you redirect it to a file, that file can be loaded with 
> pylab.load() in case you want to plot the columns. (At least this is 
> true if you don't use the -c option.)
>
Great. Sorry for stomping on that ;)
> Yesterday, before your commits, I compared memleak_gui with stock 
> Python 2.4 versus stock 2.5 (both from ubuntu feisty) and found very 
> little difference in the OS memory numbers.
Are they still increasing linearly? I'm still seeing some mystery leaks 
with Gtk, Qt4 and (much smaller) on Tk. Qt and Wx seem fine here. 
Unfortunately Qt4 crashes valgrind, so it's not of much use. 
I'm curious whether your results match that. I'm not terribly surprised 
that 2.4 isn't different from 2.5, since the case in which entire memory 
pools are freed in 2.5 is probably hard to trigger.
Cheers,
Mike

Michael Droettboom wrote:
> Eric Firing wrote:
>> I also made memleak_gui.py more flexible with arguments. For example, 
>> here are tests with three backends, a generous number of loops, and 
>> suppression of intermediate output:
> 
> Those changes are really helpful. I just added code to display the 
> total number of objects in the Python interpreter (len(gc.get_objects()) 
> with each iteration as well, as that can be useful. (It doesn't rule 
> out memory leaks, but if it is increasing, that is definitely a problem.)
> 
> I also added a commandline option to print out any cycles involving 
> uncollectable objects, and added the necessary function to do so to 
> cbook.py.
> 
> Cheers,
> Mike
Mike,
Good, thank you.
I just committed a change to the output formatting of memleak_gui so 
that if you redirect it to a file, that file can be loaded with 
pylab.load() in case you want to plot the columns. (At least this is 
true if you don't use the -c option.)
Yesterday, before your commits, I compared memleak_gui with stock Python 
2.4 versus stock 2.5 (both from ubuntu feisty) and found very little 
difference in the OS memory numbers.
Eric

Eric Firing wrote:
> I also made memleak_gui.py more flexible with arguments. For example, 
> here are tests with three backends, a generous number of loops, and 
> suppression of intermediate output:
Those changes are really helpful. I just added code to display the 
total number of objects in the Python interpreter (len(gc.get_objects()) 
with each iteration as well, as that can be useful. (It doesn't rule 
out memory leaks, but if it is increasing, that is definitely a problem.)
I also added a commandline option to print out any cycles involving 
uncollectable objects, and added the necessary function to do so to 
cbook.py.
Cheers,
Mike

John Hunter wrote:
> On 7/2/07, Michael Droettboom <md...@st...> wrote:
>> Forgot to attach the patches.
>
> Michael -- if you send me your sf ID I'll add you to the committers
> list and you can check these in directly.
mdboom
> Vis-a-vis the gtk question, I agree that we should encourage people to
> upgrade who are suffering from the leak rather than work around it. I
> would like to summarize the status of known leaks for the FAQ so
> perhaps you could summarize across the backends what kind of leaks
> remain in the --without-pymalloc with the known problems fixed (eg the
> gtk upgrade). If you could simply send me an update for the memory
> leak FAQ (don't worry about the formatting, I can take care of that)
> that would be great. Or if you are feeling doubly adventurous, you
> can simply update the FAQ in the htdocs/faq.html.template svn document
> and commit it along with your other changes.
Will do.
Cheers,
Mike

On 7/2/07, Michael Droettboom <md...@st...> wrote:
> Forgot to attach the patches.
Michael -- if you send me your sf ID I'll add you to the committers
list and you can check these in directly.
Vis-a-vis the gtk question, I agree that we should encourage people to
upgrade who are suffering from the leak rather than work around it. I
would like to summarize the status of known leaks for the FAQ so
perhaps you could summarize across the backends what kind of leaks
remain in the --without-pymalloc with the known problems fixed (eg the
gtk upgrade). If you could simply send me an update for the memory
leak FAQ (don't worry about the formatting, I can take care of that)
that would be great. Or if you are feeling doubly adventurous, you
can simply update the FAQ in the htdocs/faq.html.template svn document
and commit it along with your other changes.
Thanks for all the very useful and detailed work!
JDH

Forgot to attach the patches.
Oops,
Mike
Michael Droettboom wrote:
> More results:
>
> I've built and tested a more recent pygtk+ stack. (glib-2.12, 
> gtk+-2.10.9, librsvg-2.16.1, libxml2-2.6.29, pygobject-2.13.1, 
> pygtk-2.10.4...). The good news is that the C-level leaks I was seeing 
> in pygtk 2.2 and 2.4 are resolved. In particular, using an SVG icon and 
> Gdk rendering no longer seems problematic. I would suggest that anyone 
> using old versions of pygtk should upgrade, rather than spending time on 
> workarounds for matplotlib -- do you all agree? And my Gtk patch should 
> probably be reverted to use an SVG icon for the window again (or to only 
> do it on versions of pygtk > 2.xx). I don't know what percentage of 
> users are still using pygtk-2.4 and earlier...
>
> There is, however, a new patch (attached) to fix a leak of 
> FileChooserDialog objects that I didn't see in earlier pygtk versions. 
> I have to admit that I'm a bit puzzled by the solution -- it seems that 
> the FileChooserDialog object refuses to destruct whenever any custom 
> Python attributes have been added to the object. It doesn't really need 
> them in this case so it's an easy fix, but I'm not sure why that was 
> broken -- other classes do this and don't have problems (e.g. 
> NavigationToolbar2GTK). Maybe a pygtk expert out there knows what this 
> is about. It would be great if this resolved the linear memory growth 
> that Eric is seeing with the Gtk backend.
>
> GtkCairo seems to be free of leaks.
>
> QtAgg (qt-3.3) was leaking because of a cyclical reference in the 
> signals between the toolbar and its buttons. (Patch attached).
>
> Qt4 is forthcoming (I'm still trying to compile something that runs the 
> demos cleanly). 
>
> I tried the FltkAgg backend, but it doesn't seem to close the window at 
> all when the figure is closed -- instead I get dozens of windows open at 
> once. Is that a known bug or correct behavior?
>
> Cheers,
> Mike
>
> Eric Firing wrote:
> 
>> Michael Droettboom wrote:
>> 
>>> Eric Firing wrote:
>>> 
>>>> So, this test is still showing problems, with similar memory 
>>>> consumption in these three backends.
>>>> 
>>> Not necessarily. By default, Python allocates large pools from the 
>>> operating system and then manages those pools itself (though its 
>>> PyMalloc call). Prior to Python 2.5, those pools were never freed. 
>>> With Python 2.5, empty pools, when they occur, are freed back to the 
>>> OS. Due to fragmentation issues, even if there is enough free space 
>>> in those pools for new objects, new pools may need to be created 
>>> anyway, since Python objects can't be moved once they are created. 
>>> So seeing modest increases in memory usage during a long-running 
>>> Python application is typical, and not something that can be avoided 
>>> wiinaccurate at finding memory leaksthout micro-optimizing for pool 
>>> performance (something that may be very difficult). If memory usage 
>>> is truly increasing in an unbounded way, then, yes, there may be 
>>> problems, but it should eventually stabilize (though in a test such 
>>> as memleak_gui that may take many iterations). It's more interesting 
>>> to see the curve of memory usage over time than the average over a 
>>> number of iterations.
>>> 
>> I agree. I just ran 2000 iterations with GtkAgg, plotted every 10th 
>> point, and the increase is linear (apart from a little bumpiness) over 
>> the entire range (not just the last 1000 iterations reported below):
>>
>> Backend GTKAgg, toolbar toolbar2
>> Averaging over loops 1000 to 2000
>> Memory went from 31248k to 35040k
>> Average memory consumed per loop: 3.7920k bytes
>>
>> Maybe this is just the behavior of pymalloc in 2.5?
>>
>>
>> 
>>> For further reading, see:
>>> http://evanjones.ca/python-memory.html
>>> README.valgrind in the Python source
>>> http://mail.python.org/pipermail/python-dev/2006-March/061991.html
>>>
>>> Because of this, using the total memory allocated by the Python 
>>> process to track memory leaks is pretty blunt tool. More important 
>>> metrics are the total number of GC objects (gc.get_objects()), GC 
>>> garbage (gc.garbage), and using a tool like Valgrind or Purify to 
>>> find mismatched malloc/frees. Another useful tool (but I didn't 
>>> resort to yet with matplotlib testing) is to build Python with 
>>> COUNT_ALLOCS, which then gives access to the total number of mallocs 
>>> and frees in the Python interpreter at runtime.
>>>
>>> IMO, the only reasonable way to use the total memory usage of Python 
>>> to debug memory leaks is if you build Python without pool allocation 
>>> (--without-pymalloc). That was how I was debugging memory leaks last 
>>> week (in conjunction with valgrind, and the gc module), and with that 
>>> configuration, I was only seeing memory leakage with Pygtk 2.4, and a 
>>> very small amount with Tk. Are your numbers from a default build? 
>>> If so, I'll rebuild my Python and check my numbers against yours. If 
>>> they match, I suspect there's little we can do.
>>> 
>> I used stock Python 2.5 from ubuntu Feisty. I should compile a 
>> version as you suggest, but I haven't done it yet.
>>
>> Eric
>>
>> 
>>> Cheers,
>>> Mike
>>>
>>> 
>> 
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>
> 

More results:
I've built and tested a more recent pygtk+ stack. (glib-2.12, 
gtk+-2.10.9, librsvg-2.16.1, libxml2-2.6.29, pygobject-2.13.1, 
pygtk-2.10.4...). The good news is that the C-level leaks I was seeing 
in pygtk 2.2 and 2.4 are resolved. In particular, using an SVG icon and 
Gdk rendering no longer seems problematic. I would suggest that anyone 
using old versions of pygtk should upgrade, rather than spending time on 
workarounds for matplotlib -- do you all agree? And my Gtk patch should 
probably be reverted to use an SVG icon for the window again (or to only 
do it on versions of pygtk > 2.xx). I don't know what percentage of 
users are still using pygtk-2.4 and earlier...
There is, however, a new patch (attached) to fix a leak of 
FileChooserDialog objects that I didn't see in earlier pygtk versions. 
I have to admit that I'm a bit puzzled by the solution -- it seems that 
the FileChooserDialog object refuses to destruct whenever any custom 
Python attributes have been added to the object. It doesn't really need 
them in this case so it's an easy fix, but I'm not sure why that was 
broken -- other classes do this and don't have problems (e.g. 
NavigationToolbar2GTK). Maybe a pygtk expert out there knows what this 
is about. It would be great if this resolved the linear memory growth 
that Eric is seeing with the Gtk backend.
GtkCairo seems to be free of leaks.
QtAgg (qt-3.3) was leaking because of a cyclical reference in the 
signals between the toolbar and its buttons. (Patch attached).
Qt4 is forthcoming (I'm still trying to compile something that runs the 
demos cleanly). 
I tried the FltkAgg backend, but it doesn't seem to close the window at 
all when the figure is closed -- instead I get dozens of windows open at 
once. Is that a known bug or correct behavior?
Cheers,
Mike
Eric Firing wrote:
> Michael Droettboom wrote:
>> Eric Firing wrote:
>>> So, this test is still showing problems, with similar memory 
>>> consumption in these three backends.
>> Not necessarily. By default, Python allocates large pools from the 
>> operating system and then manages those pools itself (though its 
>> PyMalloc call). Prior to Python 2.5, those pools were never freed. 
>> With Python 2.5, empty pools, when they occur, are freed back to the 
>> OS. Due to fragmentation issues, even if there is enough free space 
>> in those pools for new objects, new pools may need to be created 
>> anyway, since Python objects can't be moved once they are created. 
>> So seeing modest increases in memory usage during a long-running 
>> Python application is typical, and not something that can be avoided 
>> wiinaccurate at finding memory leaksthout micro-optimizing for pool 
>> performance (something that may be very difficult). If memory usage 
>> is truly increasing in an unbounded way, then, yes, there may be 
>> problems, but it should eventually stabilize (though in a test such 
>> as memleak_gui that may take many iterations). It's more interesting 
>> to see the curve of memory usage over time than the average over a 
>> number of iterations.
>
> I agree. I just ran 2000 iterations with GtkAgg, plotted every 10th 
> point, and the increase is linear (apart from a little bumpiness) over 
> the entire range (not just the last 1000 iterations reported below):
>
> Backend GTKAgg, toolbar toolbar2
> Averaging over loops 1000 to 2000
> Memory went from 31248k to 35040k
> Average memory consumed per loop: 3.7920k bytes
>
> Maybe this is just the behavior of pymalloc in 2.5?
>
>
>> For further reading, see:
>> http://evanjones.ca/python-memory.html
>> README.valgrind in the Python source
>> http://mail.python.org/pipermail/python-dev/2006-March/061991.html
>>
>> Because of this, using the total memory allocated by the Python 
>> process to track memory leaks is pretty blunt tool. More important 
>> metrics are the total number of GC objects (gc.get_objects()), GC 
>> garbage (gc.garbage), and using a tool like Valgrind or Purify to 
>> find mismatched malloc/frees. Another useful tool (but I didn't 
>> resort to yet with matplotlib testing) is to build Python with 
>> COUNT_ALLOCS, which then gives access to the total number of mallocs 
>> and frees in the Python interpreter at runtime.
>>
>> IMO, the only reasonable way to use the total memory usage of Python 
>> to debug memory leaks is if you build Python without pool allocation 
>> (--without-pymalloc). That was how I was debugging memory leaks last 
>> week (in conjunction with valgrind, and the gc module), and with that 
>> configuration, I was only seeing memory leakage with Pygtk 2.4, and a 
>> very small amount with Tk. Are your numbers from a default build? 
>> If so, I'll rebuild my Python and check my numbers against yours. If 
>> they match, I suspect there's little we can do.
>
> I used stock Python 2.5 from ubuntu Feisty. I should compile a 
> version as you suggest, but I haven't done it yet.
>
> Eric
>
>>
>> Cheers,
>> Mike
>>
>
>

Norbert,
Revision 3445 has some very small changes to fix problems resulting from 
your reorganization. The questions of module and other naming are still 
open.
Eric

Michael Droettboom wrote:
> Eric Firing wrote:
>> So, this test is still showing problems, with similar memory 
>> consumption in these three backends.
> Not necessarily. By default, Python allocates large pools from the 
> operating system and then manages those pools itself (though its 
> PyMalloc call). Prior to Python 2.5, those pools were never freed. 
> With Python 2.5, empty pools, when they occur, are freed back to the 
> OS. Due to fragmentation issues, even if there is enough free space in 
> those pools for new objects, new pools may need to be created anyway, 
> since Python objects can't be moved once they are created. So seeing 
> modest increases in memory usage during a long-running Python 
> application is typical, and not something that can be avoided 
> wiinaccurate at finding memory leaksthout micro-optimizing for pool 
> performance (something that may be very difficult). If memory usage is 
> truly increasing in an unbounded way, then, yes, there may be problems, 
> but it should eventually stabilize (though in a test such as memleak_gui 
> that may take many iterations). It's more interesting to see the curve 
> of memory usage over time than the average over a number of iterations.
I agree. I just ran 2000 iterations with GtkAgg, plotted every 10th 
point, and the increase is linear (apart from a little bumpiness) over 
the entire range (not just the last 1000 iterations reported below):
Backend GTKAgg, toolbar toolbar2
Averaging over loops 1000 to 2000
Memory went from 31248k to 35040k
Average memory consumed per loop: 3.7920k bytes
Maybe this is just the behavior of pymalloc in 2.5?
> For further reading, see:
> http://evanjones.ca/python-memory.html
> README.valgrind in the Python source
> http://mail.python.org/pipermail/python-dev/2006-March/061991.html
> 
> Because of this, using the total memory allocated by the Python process 
> to track memory leaks is pretty blunt tool. More important metrics are 
> the total number of GC objects (gc.get_objects()), GC garbage 
> (gc.garbage), and using a tool like Valgrind or Purify to find 
> mismatched malloc/frees. Another useful tool (but I didn't resort to 
> yet with matplotlib testing) is to build Python with COUNT_ALLOCS, which 
> then gives access to the total number of mallocs and frees in the Python 
> interpreter at runtime.
> 
> IMO, the only reasonable way to use the total memory usage of Python to 
> debug memory leaks is if you build Python without pool allocation 
> (--without-pymalloc). That was how I was debugging memory leaks last 
> week (in conjunction with valgrind, and the gc module), and with that 
> configuration, I was only seeing memory leakage with Pygtk 2.4, and a 
> very small amount with Tk. Are your numbers from a default build? If 
> so, I'll rebuild my Python and check my numbers against yours. If they 
> match, I suspect there's little we can do.
I used stock Python 2.5 from ubuntu Feisty. I should compile a version 
as you suggest, but I haven't done it yet.
Eric
> 
> Cheers,
> Mike
> 

Eric Firing wrote:
> So, this test is still showing problems, with similar memory 
> consumption in these three backends.
Not necessarily. By default, Python allocates large pools from the 
operating system and then manages those pools itself (though its 
PyMalloc call). Prior to Python 2.5, those pools were never freed. 
With Python 2.5, empty pools, when they occur, are freed back to the 
OS. Due to fragmentation issues, even if there is enough free space in 
those pools for new objects, new pools may need to be created anyway, 
since Python objects can't be moved once they are created. So seeing 
modest increases in memory usage during a long-running Python 
application is typical, and not something that can be avoided 
wiinaccurate at finding memory leaksthout micro-optimizing for pool 
performance (something that may be very difficult). If memory usage is 
truly increasing in an unbounded way, then, yes, there may be problems, 
but it should eventually stabilize (though in a test such as memleak_gui 
that may take many iterations). It's more interesting to see the curve 
of memory usage over time than the average over a number of iterations.
For further reading, see:
http://evanjones.ca/python-memory.html
README.valgrind in the Python source
http://mail.python.org/pipermail/python-dev/2006-March/061991.html
Because of this, using the total memory allocated by the Python process 
to track memory leaks is pretty blunt tool. More important metrics are 
the total number of GC objects (gc.get_objects()), GC garbage 
(gc.garbage), and using a tool like Valgrind or Purify to find 
mismatched malloc/frees. Another useful tool (but I didn't resort to 
yet with matplotlib testing) is to build Python with COUNT_ALLOCS, which 
then gives access to the total number of mallocs and frees in the Python 
interpreter at runtime.
IMO, the only reasonable way to use the total memory usage of Python to 
debug memory leaks is if you build Python without pool allocation 
(--without-pymalloc). That was how I was debugging memory leaks last 
week (in conjunction with valgrind, and the gc module), and with that 
configuration, I was only seeing memory leakage with Pygtk 2.4, and a 
very small amount with Tk. Are your numbers from a default build? If 
so, I'll rebuild my Python and check my numbers against yours. If they 
match, I suspect there's little we can do.
Cheers,
Mike

Norbert Nemec wrote:
> Hmm - let me think.... We already have
> rc
> rcParams
> rc_params
> rcdefaults
> rcParamDefaults
> defaultParams
> in the main module of maplotlib
> 
> How about calling the new module 'rcdefaultparams.py', simply to make
> the confusion complete and because I really feel that no other name
> would fit the current "naming scheme" better... ;-)
Yes, it is confusing, there are too many similar names. I suspect some 
are used infrequently enough that we could change them without too much 
pain.
But the new module is really two things: 1) rc utilities (mainly 
validation facilities) and 2) a set of default values. If these are 
kept together the module could be called "rc_init.py" because everything 
is mainly used for rc initialization, although there are things still in 
mpl's __init__.py that are also part of the rc initialization. Or it 
could be called "rc_utils.py" or "rcsetup.py". I would prefer any of 
these to rcdefaultparams.py.
Furthermore, even after factoring out the rc things as you have done the 
mpl namespace is badly cluttered with things like checkdep_dvipng, 
(which is actually part of the rc validation, so maybe it should be in 
your new module) so still more refactoring and/or renaming might be in 
order. I can imagine a class being used to good effect to organize the 
whole business of rc handling.
One more miscellaneous thought: shouldn't mpl.rc() be using the 
validation functions instead of simply stuffing inputs into rcParams?
I suppose this brings us back to the old "traits, properties, or 
neither" question. But incremental improvements such as the one you 
have made are still helpful.
Eric
> 
> Greetings,
> Norbert
> 
> 
> 
> John Hunter wrote:
>> On 6/30/07, Norbert Nemec <Nor...@gm...> wrote:
>> 
>>> Hi there,
>>>
>>> I just checked in some major reorganization work in __init__.py
>>>
>>> The main intention was to move the list of option defaults to a separate
>>> file 'rcdefaults.py' that could be imported from setup.py to access the
>>> settings with minimal dependencies on the remaining code.
>>> 
>> I haven't tested this but I did take a brief look at it and I think
>> your cleaning and organizing is useful. I think we have a naming
>> problem though -- this __init__ module defines an rcdefaults function,
>> which is likely to cause confusion with the new rcdefaults module.
>> Eg,
>>
>> >from matplotlib import rcdefaults
>>
>> will be ambiguous. You may want to consider a new name.
>>
>> DH
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by DB2 Express
>> Download DB2 Express C - the FREE version of DB2 express and take
>> control of your XML. No limits. Just data. Click to get it now.
>> http://sourceforge.net/powerbar/db2/
>> _______________________________________________
>> Matplotlib-devel mailing list
>> Mat...@li...
>> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>>
>> 
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Hmm - let me think.... We already have
 rc
 rcParams
 rc_params
 rcdefaults
 rcParamDefaults
 defaultParams
in the main module of maplotlib
How about calling the new module 'rcdefaultparams.py', simply to make
the confusion complete and because I really feel that no other name
would fit the current "naming scheme" better... ;-)
Greetings,
Norbert
John Hunter wrote:
> On 6/30/07, Norbert Nemec <Nor...@gm...> wrote:
> 
>> Hi there,
>>
>> I just checked in some major reorganization work in __init__.py
>>
>> The main intention was to move the list of option defaults to a separate
>> file 'rcdefaults.py' that could be imported from setup.py to access the
>> settings with minimal dependencies on the remaining code.
>> 
>
> I haven't tested this but I did take a brief look at it and I think
> your cleaning and organizing is useful. I think we have a naming
> problem though -- this __init__ module defines an rcdefaults function,
> which is likely to cause confusion with the new rcdefaults module.
> Eg,
>
> >from matplotlib import rcdefaults
>
> will be ambiguous. You may want to consider a new name.
>
> DH
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>
> 

John Hunter wrote:
> On 6/30/07, Eric Firing <ef...@ha...> wrote:
>> Mike,
>>
>> All this sounds like great progress--thanks! I particularly appreciate
>> the descriptions of what problems you found and how you found them.
>>
>> John et al.: is there a maintainer for each of these backends? I think
> 
> gtk: Steve Chaplin or me
> wx: Ken McIvor
> qt: Darren?
> tk: Charlie?
> 
> After we get these patches in, we can just give Michael commit
> privileges :-) I can probably look at this Monday, but if you want to
> commit and test some of these before then, please do so.
Done. It looks like there is still plenty of memory leakage, but there 
are improvements, and the huge list of uncollectable garbage with tkAgg 
is gone.
I also made memleak_gui.py more flexible with arguments. For example, 
here are tests with three backends, a generous number of loops, and 
suppression of intermediate output:
python ../unit/memleak_gui.py -d wx -s 500 -e 1000 -q
uncollectable list: []
Backend WX, toolbar toolbar2
Averaging over loops 500 to 1000
Memory went from 29316k to 31211k
Average memory consumed per loop: 3.7900k bytes
python ../unit/memleak_gui.py -d tkagg -s 500 -e 1000 -q
uncollectable list: []
Backend TkAgg, toolbar toolbar2
Averaging over loops 500 to 1000
Memory went from 29202k to 31271k
Average memory consumed per loop: 4.1380k bytes
python ../unit/memleak_gui.py -d gtkagg -s 500 -e 1000 -q
uncollectable list: []
Backend GTKAgg, toolbar toolbar2
Averaging over loops 500 to 1000
Memory went from 29324k to 31131k
Average memory consumed per loop: 3.6140k bytes
So, this test is still showing problems, with similar memory consumption 
in these three backends.
Eric
> 
> JDH