matplotlib-devel Mailing List for matplotlib

On 6/30/07, Eric Firing <ef...@ha...> wrote:
> Mike,
>
> All this sounds like great progress--thanks! I particularly appreciate
> the descriptions of what problems you found and how you found them.
>
> John et al.: is there a maintainer for each of these backends? I think
gtk: Steve Chaplin or me
wx: Ken McIvor
qt: Darren?
tk: Charlie?
After we get these patches in, we can just give Michael commit
privileges :-) I can probably look at this Monday, but if you want to
commit and test some of these before then, please do so.
JDH

On 6/30/07, Norbert Nemec <Nor...@gm...> wrote:
> Hi there,
>
> I just checked in some major reorganization work in __init__.py
>
> The main intention was to move the list of option defaults to a separate
> file 'rcdefaults.py' that could be imported from setup.py to access the
> settings with minimal dependencies on the remaining code.
I haven't tested this but I did take a brief look at it and I think
your cleaning and organizing is useful. I think we have a naming
problem though -- this __init__ module defines an rcdefaults function,
which is likely to cause confusion with the new rcdefaults module.
Eg,
from matplotlib import rcdefaults
will be ambiguous. You may want to consider a new name.
DH

Hi there,
I just checked in some major reorganization work in __init__.py
The main intention was to move the list of option defaults to a separate
file 'rcdefaults.py' that could be imported from setup.py to access the
settings with minimal dependencies on the remaining code.
To do so, I had to untangle a number of unnessessary dependancies within
__init__.py. I hope, the behavior of the code
should not have changed in the course of doing so. In case problems
arise, please contact me directly, I don't know
how closely I can follow the mailing list.
I hope that from this starting point, future maintenance of the option
settings will be somewhat easier.
Greetings,
Norbert

I haven't tried the svn version yet, but it looks like it does basically the
same thing as my version. The only thing different with mine I think is that
if there are empty directories, I don't add them to the list.
On 6/30/07, Andrew Straw <str...@as...> wrote:
>
> Dear Tim,
>
> I checked in a similar patch from Tocer a couple days ago. Does your
> version do anything different? Does the version in svn work for you?
>
> http://matplotlib.svn.sourceforge.net/viewvc/matplotlib/trunk/matplotlib/lib/matplotlib/__init__.py?r1=3391&r2=3418
>
> (Sorry, I don't really use Windows or py2exe.)
>
> -Andrew
>
> Tim Swast wrote:
> > I was having trouble with py2exe not getting the datafiles that it
> > needed. I noticed someone else got it to work by copying all the files,
> > but losing the directory structure. I tried that, but I still had
> > missing datafiles at runtime.
> >
> > This function will return everything needed for py2exe to work
> > correctly. Instead of returning one tuple, it returns a list of tuples,
> > so the use changes a little bit, but at least it works.
> >
> > def get_py2exe_datafiles():
> > outdirname = 'matplotlibdata'
> > mplfiles = []
> >
> > for root, dirs, files in os.walk(get_data_path()):
> > py2exe_files = []
> >
> > # Append root to each file so py2exe can find them
> > for file in files:
> > py2exe_files.append(os.sep.join([root, file]))
> >
> > if len(py2exe_files) > 0:
> > py2exe_root = root[len(get_data_path()+os.sep):]
> >
> > if len(py2exe_root) > 0:
> > mplfiles.append((os.sep.join([outdirname, py2exe_root]),
> > py2exe_files))
> > else:
> > # Don't do a join for the root directory
> > mplfiles.append((outdirname, py2exe_files))
> >
> > return mplfiles
> >
> >
> > Sorry for not submitting as a patch: I haven't quite figured out how to
> > do that yet.
> >
> >
> > ------------------------------------------------------------------------
> >
> >
> -------------------------------------------------------------------------
> > This SF.net email is sponsored by DB2 Express
> > Download DB2 Express C - the FREE version of DB2 express and take
> > control of your XML. No limits. Just data. Click to get it now.
> > http://sourceforge.net/powerbar/db2/
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Matplotlib-devel mailing list
> > Mat...@li...
> > https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>
>

Mike,
All this sounds like great progress--thanks! I particularly appreciate 
the descriptions of what problems you found and how you found them.
John et al.: is there a maintainer for each of these backends? I think 
it is very important that Mike's patches be checked out and applied ASAP 
if they are OK; or if there is a problem, then that info needs to get 
back to Mike. This should be very high priority. I can do a quick 
check and commit if necessary, but it would make more sense for someone 
more familiar with backends and gui code to do it. Or at least for 
others to do some testing on different platforms if I make the commits.
Eric
Michael Droettboom wrote:
> I've been looking into the memory leaks exercised by the memleak_gui.py 
> unit test. I've searched through the mailing list for information, but 
> I'm new to the party here, so forgive me if I'm not fully current.
> 
> Eric Firing wrote:
> "I think we have a similar problem with all interactive backends (the 
> only one I didn't test is Qt4Agg) which also makes me suspect we are 
> violating some gui rule, rather than that gtk, qt3, wx, and tk all have 
> leaks."
> 
> Unfortunately, from what I've seen, there isn't a single rule being 
> broken, but instead I've been running into lots of different "surprises" 
> in the various toolkits. But I am starting to suspect that my old 
> versions of Gtk (or PyGtk) have some bonafide leaks.
> 
> I just finished submitting patches (to the tracker) for a number of 
> memory leaks in the Tk, Gtk, and Wx backends (other backends will 
> hopefully follow). I did all my testing on RHEL4 with Python 2.5 and 
> recent SVN matplotlib (rev. 3244), so it's quite possible that memory 
> leaks still remain on other platforms.
> 
> Tk:
> See the patch:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1745400&group_id=80706&atid=560722 
> 
> 
> Even after this patch, Tkinter still leaks 28 bytes every time it is 
> initialized (every time a toplevel window is created). There may be a 
> way to avoid the subsequent initializations, but it wasn't immediately 
> obvious to me, and given the small size of this leak, I've passed on it 
> for now.
> 
> Gtk:
> See the patch:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1745406&group_id=80706&atid=560722 
> 
> 
> The patch fixes a number of Python-level leaks.
> Unfortunately, the Gdk rendering backend still leaks gtk.gdk.Window 
> objects, and I have so far been unable to determine a fix. GtkAgg, 
> however, does not have this leak.
> Under pygtk-2.4.0, the toolbars leak gdk.Pixbuf's and file selector 
> dialogs.
> Since these issues appear to be bugs in gtk and/or pygtk itself, I will 
> probably first try a more recent version of them. (The gtk installed on 
> RHEL4 is ancient (2.4), and I haven't yet tried building my own. If 
> anyone has a more recent version of pygtk handy, I would appreciate a 
> report from memleak_gui.py after applying my patch.)
> 
> Wx:
> See the patch:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1745408&group_id=80706&atid=560722 
> 
> 
> This one was fairly simple, though surprising. Top-level windows do not 
> fully destroy themselves as long as there are pending events. Manually 
> flushing the event queue causes the windows to go away. See the 
> wxPython docs:
> http://wxwidgets.org/manuals/stable/wx_wxwindow.html#wxwindowdestroy
> 
> There were no further leaks in Wx/WxAgg that I was able to detect (in my 
> environment).
> 
> 
> As an aside, I thought I'd share the techniques I used to find these 
> leaks (hopefully not to be pedantic, but these things were hard to come 
> by online...), and it might be useful to some.
> 
> For C/C++ memory leaks, I really like valgrind (though it is 
> Linux-only), though be sure to follow the directions to get it to play 
> well with Python. I recommend rebuilding Python with 
> "--without-pymalloc" to make memory reporting in general much more 
> sensical (though slower). See:
> http://svn.python.org/view/python/trunk/Misc/README.valgrind
> For an example, you can see the rsvg memory leak here:
> 
> ==15979== 1,280 bytes in 20 blocks are definitely lost in loss record 
> 13,506 of 13,885
> ==15979== at 0x4004405: malloc (vg_replace_malloc.c:149)
> ==15979== by 0x314941: (within /usr/lib/libart_lgpl_2.so.2.3.16)
> ==15979== by 0x315E0C: (within /usr/lib/libart_lgpl_2.so.2.3.16)
> ==15979== by 0x31624A: art_svp_intersector (in 
> /usr/lib/libart_lgpl_2.so.2.3.16)
> ==15979== by 0x316660: art_svp_intersect (in 
> /usr/lib/libart_lgpl_2.so.2.3.16)
> ==15979== by 0x6BFA86C: (within /usr/lib/librsvg-2.so.2.8.1)
> ==15979== by 0x6BFD801: rsvg_render_path (in 
> /usr/lib/librsvg-2.so.2.8.1)
> ==15979== by 0x6BFD9E7: rsvg_handle_path (in 
> /usr/lib/librsvg-2.so.2.8.1)
> ==15979== by 0x6BFFDB5: rsvg_start_path (in /usr/lib/librsvg-2.so.2.8.1)
> ==15979== by 0x6C07F59: (within /usr/lib/librsvg-2.so.2.8.1)
> ==15979== by 0x9C24BB: xmlParseStartTag (in /usr/lib/libxml2.so.2.6.16)
> ==15979== by 0xA4DADC: xmlParseChunk (in /usr/lib/libxml2.so.2.6.16)
> ==15979== by 0x6C08ED5: rsvg_handle_write_impl (in 
> /usr/lib/librsvg-2.so.2.8.1)
> ==15979== by 0x6C09539: rsvg_handle_write (in 
> /usr/lib/librsvg-2.so.2.8.1)
> ==15979== by 0x5D51B51: (within 
> /usr/lib/gtk-2.0/2.4.0/loaders/svg_loader.so)
> ==15979== by 0x32871F: (within /usr/lib/libgdk_pixbuf-2.0.so.0.400.13)
> ==15979== by 0x32885E: gdk_pixbuf_new_from_file (in 
> /usr/lib/libgdk_pixbuf-2.0.so.0.400.13)
> ==15979== by 0x5CEE34: (within /usr/lib/libgtk-x11-2.0.so.0.400.13)
> ==15979== by 0x5CF095: gtk_window_set_default_icon_from_file (in 
> /usr/lib/libgtk-x11-2.0.so.0.400.13)
> ==15979== by 0x59FCD1B: _wrap_gtk_window_set_default_icon_from_file 
> (gtk.c:38156)
> ==15979== by 0x80C29B6: PyEval_EvalFrameEx (ceval.c:3564)
> 
> For finding cycles that result in uncollectable garbage, I wrote a 
> cycle-finding utility (see attached file). (I plan to submit this as a 
> Python Cookbook entry). Given a list of objects of interest, it will 
> print out all the reference cycles involving those objects (though it 
> can't traverse through extension objects that don't expose reference 
> information to the garbage collector).
> 
> Objects that leak because legitimate Python references to them are still 
> around are some of the trickiest to find. (This was an answer to a job 
> interview question I was once asked: "How can you leak memory in 
> Java/some other garbage collected language?") I find it useful to 
> compare snapshots of all the objects in the interpreter before and after 
> a suspected leak:
> 
> existing_objects = [id(x) for x in gc.get_objects()]
> # ... do something that leaks ...
> # ... delete everything you can ...
> remaining_objects = [x for x in gc.get_objects() if id(x) not in 
> existing_objects]
> 
> One can then scour through remaining_objects for anything that suspected 
> of causing the problem, and do cycle detection on those objects, if 
> necessary to find ways to forcibly remove them.
> 
> Cheers,
> Mike
> 
> 
> ------------------------------------------------------------------------
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Dear Tim,
I checked in a similar patch from Tocer a couple days ago. Does your 
version do anything different? Does the version in svn work for you?
http://matplotlib.svn.sourceforge.net/viewvc/matplotlib/trunk/matplotlib/lib/matplotlib/__init__.py?r1=3391&r2=3418
(Sorry, I don't really use Windows or py2exe.)
-Andrew
Tim Swast wrote:
> I was having trouble with py2exe not getting the datafiles that it 
> needed. I noticed someone else got it to work by copying all the files, 
> but losing the directory structure. I tried that, but I still had 
> missing datafiles at runtime.
> 
> This function will return everything needed for py2exe to work 
> correctly. Instead of returning one tuple, it returns a list of tuples, 
> so the use changes a little bit, but at least it works.
> 
> def get_py2exe_datafiles():
> outdirname = 'matplotlibdata'
> mplfiles = []
> 
> for root, dirs, files in os.walk(get_data_path()):
> py2exe_files = []
> 
> # Append root to each file so py2exe can find them
> for file in files:
> py2exe_files.append(os.sep.join([root, file]))
> 
> if len(py2exe_files) > 0:
> py2exe_root = root[len(get_data_path()+os.sep):]
> 
> if len(py2exe_root) > 0:
> mplfiles.append((os.sep.join([outdirname, py2exe_root]), 
> py2exe_files))
> else:
> # Don't do a join for the root directory
> mplfiles.append((outdirname, py2exe_files))
> 
> return mplfiles
> 
> 
> Sorry for not submitting as a patch: I haven't quite figured out how to 
> do that yet.
> 
> 
> ------------------------------------------------------------------------
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

I was having trouble with py2exe not getting the datafiles that it needed. I
noticed someone else got it to work by copying all the files, but losing the
directory structure. I tried that, but I still had missing datafiles at
runtime.
This function will return everything needed for py2exe to work correctly.
Instead of returning one tuple, it returns a list of tuples, so the use
changes a little bit, but at least it works.
def get_py2exe_datafiles():
 outdirname = 'matplotlibdata'
 mplfiles = []
 for root, dirs, files in os.walk(get_data_path()):
 py2exe_files = []
 # Append root to each file so py2exe can find them
 for file in files:
 py2exe_files.append(os.sep.join([root, file]))
 if len(py2exe_files) > 0:
 py2exe_root = root[len(get_data_path()+os.sep):]
 if len(py2exe_root) > 0:
 mplfiles.append((os.sep.join([outdirname, py2exe_root]),
py2exe_files))
 else:
 # Don't do a join for the root directory
 mplfiles.append((outdirname, py2exe_files))
 return mplfiles
Sorry for not submitting as a patch: I haven't quite figured out how to do
that yet.