Thanks for your advice with installing matplotlib on cygwin. I = downloaded and installed the windows binaries and it worked. Anyway, the reason that I didn't want to use binaries in the first place = was because I wanted to modify the matplotilb source code. But it seems = like even with the binaries, if I change the source code then it will = still affect the operation of the program when I run it, which is what I = want. In particular, I am looking to speed up the pcolor() function because it = runs exceedingly slow with large mesh sizes. I believe the reason it is = running slow is because of a memory leak. When I do the following: from pylab import * n=3D200 [x,y]=3Dmeshgrid(arange(n+1)*1./n,arange(n+1)*1./n) z=3Dsin(x**2 + y**2) and then do pcolor(x,y,z) repeatedly, the memory usage increases by about 15 MB each time, and it = runs progressively slower.each time. By using the profiler I can see = that almost all the time is spent in the pcolor function itself, rather = than any functions it calls. When I take out the line = "self.add_collection(collection)", there is no memory leak and it runs = much faster (of course, since I'm not actually adding the collection on, = when I do that i can no longer actually see the plot). It seems to me = like what is happening is that it is repeatedly appending the same = PolyCollection to the collections list, using up more and more memory. = What I want to know is what the "collections" are and how they are used = so I can figure out a way of getting rid of extraneous collections. -Alexander Mont
>>>>> "Alexander" == Alexander Mont <ale...@co...> writes: Alexander> Thanks for your advice with installing matplotlib on Alexander> cygwin. I downloaded and installed the windows binaries Alexander> and it worked. Anyway, the reason that I didn't want Alexander> to use binaries in the first place was because I wanted Alexander> to modify the matplotilb source code. But it seems like Alexander> even with the binaries, if I change the source code Alexander> then it will still affect the operation of the program Alexander> when I run it, which is what I want. Alexander> In particular, I am looking to speed up the pcolor() Alexander> function because it runs exceedingly slow with large Alexander> mesh sizes. I believe the reason it is running slow is Alexander> because of a memory leak. When I do the following: Alexander> from pylab import * n=200 Alexander> [x,y]=meshgrid(arange(n+1)*1./n,arange(n+1)*1./n) Alexander> z=sin(x**2 + y**2) Alexander> and then do Alexander> pcolor(x,y,z) Alexander> repeatedly, the memory usage increases by about 15 MB Alexander> each time, and it runs progressively slower.each At least with matplotlib CVS (and I don't think it's a CVS vs 0.84 issue) the memory consumption is rock solid with your example (see below for my test script). What is your default "hold" setting in rc? If True, you will be overlaying plots and will get the behavior you describe. In the example below, I make sure to "close" the figure each time -- a plain clear with clf should suffice though. My guess is that you are repeatedly calling pcolor with hold : True and are simply overlaying umpteen pcolors (to test for this, print the length of the collections list ax = gca() print len(ax.collections) if this length is growing, you've found your problem. A simple pcolor(x,y,z,hold=False) should suffice. You can also change the default hold setting in your config file http://matplotlib.sf.net/matplotlibrc JDH Example code: #!/usr/bin/env python import os, sys, time import matplotlib #matplotlib.interactive(True) #matplotlib.use('Cairo') matplotlib.use('Agg') from pylab import * def report_memory(i): pid = os.getpid() a2 = os.popen('ps -p %d -o rss,sz' % pid).readlines() print i, ' ', a2[1], return int(a2[1].split()[1]) # take a memory snapshot on indStart and compare it with indEnd indStart, indEnd = 30, 201 for i in range(indEnd): figure(1); clf() n=200 [x,y]=meshgrid(arange(n+1)*1./n,arange(n+1)*1./n) z=sin(x**2 + y**2) pcolor(x,y,z) savefig('tmp%d' % i, dpi = 75) close(1) val = report_memory(i) if i==indStart: start = val # wait a few cycles for memory usage to stabilize end = val print 'Average memory consumed per loop: %1.4fk bytes\n' % ((end-start)/float(indEnd-indStart)) """ Average memory consumed per loop: 0.0053k bytes """
On Tue, Nov 01, 2005 at 09:41:35PM -0600, John Hunter wrote: > >>>>> "Alexander" == Alexander Mont <ale...@co...> writes: > > Alexander> Thanks for your advice with installing matplotlib on > Alexander> cygwin. I downloaded and installed the windows binaries > Alexander> and it worked. Anyway, the reason that I didn't want > Alexander> to use binaries in the first place was because I wanted > Alexander> to modify the matplotilb source code. But it seems like > Alexander> even with the binaries, if I change the source code > Alexander> then it will still affect the operation of the program > Alexander> when I run it, which is what I want. > > Alexander> In particular, I am looking to speed up the pcolor() > Alexander> function because it runs exceedingly slow with large > Alexander> mesh sizes. I believe the reason it is running slow is > Alexander> because of a memory leak. When I do the following: > > Alexander> from pylab import * n=200 > Alexander> [x,y]=meshgrid(arange(n+1)*1./n,arange(n+1)*1./n) > Alexander> z=sin(x**2 + y**2) > > Alexander> and then do > > Alexander> pcolor(x,y,z) > > Alexander> repeatedly, the memory usage increases by about 15 MB > Alexander> each time, and it runs progressively slower.each > > At least with matplotlib CVS (and I don't think it's a CVS vs 0.84 > issue) the memory consumption is rock solid with your example (see > below for my test script). What is your default "hold" setting in rc? > If True, you will be overlaying plots and will get the behavior you > describe. In the example below, I make sure to "close" the figure > each time -- a plain clear with clf should suffice though. My guess > is that you are repeatedly calling pcolor with hold : True and are > simply overlaying umpteen pcolors (to test for this, print the length > of the collections list > > ax = gca() > print len(ax.collections) > > if this length is growing, you've found your problem. A simple > > pcolor(x,y,z,hold=False) > > should suffice. > > You can also change the default hold setting in your config file > http://matplotlib.sf.net/matplotlibrc > > JDH I can confirm that memory leaks indeed are not a problem with a CVS build on Debian. I can't seem to restore the pre-built Debian stable 0.82 so I haven't tested it. However, the problem is still that pcolor is too slow to use it interactively on a plot with a half-dozen 200x608 warped grids, even with shading='flat' and no antialiasing. Would you consider accepting a 'structured grid' as a primitive patch type? What consequence will this have for your various backends? Presumably someone will want triangular meshes as well if they are doing serious FEM work. I created a prototype app using OpenGL and quad strips. The performance with this is acceptable, but I need a lot more 2D graphing features. I would much rather make an existing product better than rewrite from scratch. In my particular case the grid warping function can be expressed analytically. Is it reasonable to consider warping a 2D image directly using an AGG filter function? Could this be embedded in an existing matplotlib graph, above some objects and below others? Thanks in advance, Paul Kienzle pki...@ni...
Dr. Kienzle, Did you get my message before about improving the pcolor function by changing the way the "verts" array is created? Although I couldn't get it to work on my machine (maybe because I'm not actually re-building it, I'm just editing the source file and then exiting and restarting) someone else on the mailing list said he didn't have problems. If you can, make those changes and report back to me on how wll they work. Also, is it just the "pcolor" function that is too slow or is the "draw" function too slow as well? -Alex Mont ----- Original Message ----- From: "Paul Kienzle" <pki...@ja...> To: "John Hunter" <jdh...@ni...> Cc: "Alexander Mont" <ale...@co...>; <mat...@li...> Sent: Thursday, November 03, 2005 11:10 AM Subject: Re: [Matplotlib-users] Memory leak with pcolor > On Tue, Nov 01, 2005 at 09:41:35PM -0600, John Hunter wrote: >> >>>>> "Alexander" == Alexander Mont <ale...@co...> writes: >> >> Alexander> Thanks for your advice with installing matplotlib on >> Alexander> cygwin. I downloaded and installed the windows binaries >> Alexander> and it worked. Anyway, the reason that I didn't want >> Alexander> to use binaries in the first place was because I wanted >> Alexander> to modify the matplotilb source code. But it seems like >> Alexander> even with the binaries, if I change the source code >> Alexander> then it will still affect the operation of the program >> Alexander> when I run it, which is what I want. >> >> Alexander> In particular, I am looking to speed up the pcolor() >> Alexander> function because it runs exceedingly slow with large >> Alexander> mesh sizes. I believe the reason it is running slow is >> Alexander> because of a memory leak. When I do the following: >> >> Alexander> from pylab import * n=200 >> Alexander> [x,y]=meshgrid(arange(n+1)*1./n,arange(n+1)*1./n) >> Alexander> z=sin(x**2 + y**2) >> >> Alexander> and then do >> >> Alexander> pcolor(x,y,z) >> >> Alexander> repeatedly, the memory usage increases by about 15 MB >> Alexander> each time, and it runs progressively slower.each >> >> At least with matplotlib CVS (and I don't think it's a CVS vs 0.84 >> issue) the memory consumption is rock solid with your example (see >> below for my test script). What is your default "hold" setting in rc? >> If True, you will be overlaying plots and will get the behavior you >> describe. In the example below, I make sure to "close" the figure >> each time -- a plain clear with clf should suffice though. My guess >> is that you are repeatedly calling pcolor with hold : True and are >> simply overlaying umpteen pcolors (to test for this, print the length >> of the collections list >> >> ax = gca() >> print len(ax.collections) >> >> if this length is growing, you've found your problem. A simple >> >> pcolor(x,y,z,hold=False) >> >> should suffice. >> >> You can also change the default hold setting in your config file >> http://matplotlib.sf.net/matplotlibrc >> >> JDH > > I can confirm that memory leaks indeed are not a problem with a CVS build > on Debian. I can't seem to restore the pre-built Debian stable 0.82 so > I haven't tested it. > > However, the problem is still that pcolor is too slow to use it > interactively on a plot with a half-dozen 200x608 warped grids, even > with shading='flat' and no antialiasing. > > Would you consider accepting a 'structured grid' as a primitive patch > type? What consequence will this have for your various backends? > Presumably someone will want triangular meshes as well if they are > doing serious FEM work. > > I created a prototype app using OpenGL and quad strips. The performance > with this is acceptable, but I need a lot more 2D graphing features. > I would much rather make an existing product better than rewrite from > scratch. > > In my particular case the grid warping function can be expressed > analytically. Is it reasonable to consider warping a 2D image directly > using an AGG filter function? Could this be embedded in an existing > matplotlib graph, above some objects and below others? > > Thanks in advance, > > Paul Kienzle > pki...@ni... > > > ------------------------------------------------------- > SF.Net email is sponsored by: > Tame your development challenges with Apache's Geronimo App Server. > Download > it for free - -and be entered to win a 42" plasma tv or your very own > Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php > _______________________________________________ > Matplotlib-users mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-users >
On Thu, 2005年11月03日 at 11:10 -0500, Paul Kienzle wrote: > However, the problem is still that pcolor is too slow to use it > interactively on a plot with a half-dozen 200x608 warped grids, even > with shading='flat' and no antialiasing. I posted a CVS patch to the devel list a while ago which implemented an alternate image class which could plot data on a stretched (rectangular) grid much faster than pcolor can. The patch wasn't entirely complete as I'm unsure what the user-interface should be but it is usable and does what you seem to want to do without additional dependencies and with fast zooming and panning once data is loaded. I've tested with data up to 2048x2048 in size and it's quite usable on my laptop. If you are interested I can give you a patch against the current CVS. Nich
>>>>> "Nicholas" == Nicholas Young <su...@su...> writes: Nicholas> I posted a CVS patch to the devel list a while ago which Nicholas> implemented an alternate image class which could plot Nicholas> data on a stretched (rectangular) grid much faster than Nicholas> pcolor can. The patch wasn't entirely complete as I'm Nicholas> unsure what the user-interface should be but it is Nicholas> usable and does what you seem to want to do without Nicholas> additional dependencies and with fast zooming and Nicholas> panning once data is loaded. I've tested with data up Nicholas> to 2048x2048 in size and it's quite usable on my laptop. Nicholas> If you are interested I can give you a patch against the Nicholas> current CVS. Hi Nicholas, Sorry for dropping the ball on this one. I just committed your patch to CVS. Of course, the interface issue remains to be determined, but the core logic is now in CVS Checking in lib/matplotlib/image.py; /cvsroot/matplotlib/matplotlib/lib/matplotlib/image.py,v <-- image.py new revision: 1.22; previous revision: 1.21 done Checking in src/_image.cpp; /cvsroot/matplotlib/matplotlib/src/_image.cpp,v <-- _image.cpp new revision: 1.36; previous revision: 1.35 done Checking in src/_image.h; /cvsroot/matplotlib/matplotlib/src/_image.h,v <-- _image.h new revision: 1.16; previous revision: 1.15 JDH