matplotlib-users Mailing List for matplotlib

On 10/10/2013 15:05, Martin MOKREJŠ wrote:
> Hi,
> rendering some of my charts takes almost 50GB of RAM. I believe below is a stracktrace
> of one such situation when it already took 15GB. Would somebody comments on what is
> matplotlib doing at the very moment? Why the recursion?
> 
> The charts had to have 262422 data points in a 2D scatter plot, each point has assigned
> its own color. They are in batches so that there are 153 distinct colors but nevertheless,
> I assigned to each data point a color value. There are 153 legend items also (one color
> won't be used).
Hello Martin,
can I ask what is the meaning of plotting a scatter plot with 200
thousands points in it? Either you visualize it on a screen much larger
than mine, or you are not going to be able to distinguish the single
data points. Maybe you should rethink the visualization tool you are using.
Nevertheless, I'm perfectly able to plot a scatter plot with 262422 data
points each with its own color just fine, and the python process
consumes a few hundred Mb of ram (having quite a few other datasets
loaded in memory)::
 import numpy as np
 import matplotlib.pyplot as plt
 n = 262422
 x = np.random.rand(n)
 y = np.random.rand(n)
 c = np.random.rand(n)
 f = plt.figure()
 a = f.add_subplot(111)
 a.scatter(x, y, c=c, s=50)
 plt.show()
and a possible solution using exactly 153 different colors, but again, I
don't see how you can distinguish between hundreds different shades of
colors::
 n = 262422 #22
 ncolors = 153
 x = np.random.rand(n)
 y = np.random.rand(n)
 c = np.random.rand(ncolors)
 f = plt.figure()
 a = f.add_subplot(111)
 for i in xrange(n // ncolors):
 a.scatter(x[i*ncolors:(i+1)*ncolors],
 y[i*ncolors:(i+1)*ncolors], c=c, s=50)
 plt.show()
Unfortunately the code you provide is too contrived to be useful to
understand the root cause of your problem.
Cheers,
Daniele

Michael Droettboom wrote:
> Sorry to repeat myself, but please reduce this to a short, self contained example, that is absolutely minimal to demonstrate the problem. http://sscce.org/ should help better explain what I'm after. I don't want to find the needle in the haystack here -- there is code in your example that doesn't even run, for example.
> 
> That said, are you really after creating a legend entry for each of the dots? (See below). That just isn't going to work, and I'm not surprised it eats up excessive amounts of memory. I think you want (and can) reduce this to a single scatter call.
> 
> _series = [_ax1.scatter(_x, _y, color=_c, s=objsize, label=_l, hatch='.') for _x, _y, _c, _l in izip(mydata_x, mydata_y, colors, legends)] # returns PathCollection object
Are you sure? I think it was concluded on this list that scatter cannot (or does not) take
nested lists of lists with series like histogram and piechart do. I cannot find the thread
but maybe you are more lucky. I even think that I already opened a bugreport/feature requested
in the past for this. But maybe not.
Martin
> 
> Mike
> 
> On 10/12/2013 12:57 PM, Martin MOKREJŠ wrote:
>> Hi,
>> so here is some quick but working example. I added there are 2-3 functions (unused)
>> as a bonus, you can easily call them from the main function using same API
>> (except the piechart). I hope this shows what I lack in matplotlib - a general API
>> so that I could easily switch form scatter plot to piechart or barchart without altering
>> much the function arguments. Messing with return objects line2D, PathCollection, Rectangle
>> is awkward and I would like to stay away from matplotlib's internals. ;) Some can be sliced,
>> so not, you will see in the code.
>>
>> This eatmem.py will take easily all your memory. Drawing 300000 dots is not feasible
>> with 16GB of RAM. While the example is for sure inefficient in many places generating the data
>> in python does not eat RAM. That happens afterwards.
>>
>> I would really like to hear whether matplotlib could be adjusted instead. ;) I already mentioned
>> in this thread that it is awkward to pre-create colors before passing all data to a drawing
>> function. I think we could all save a lot if matplotlib could dynamically fetch colors
>> on the fly from user-created generator, same for legends descriptions. I think my example
>> code shows the inefficient approach here. Would I have more time I would randomize a bit
>> more the sublist of each series so that the numbers in legends would be more variable
>> but that is a cosmetic issue.
>> Probably due to my ignorance you will see that figures with legends have different font
>> sizes, axes are rescaled and the figure. Of course I wanted to have the drawing same via both
>> approaches but failed badly. The files/figures with legends should be just accompanied by the
>> legend "table" underneath but the drawing itself should be same. Maybe an issue with DPI settings
>> but not only.
>>
>> I placed some comments in the code, please don't take them in person. ;) Of course
>> I am glad for the existing work and am happy to contribute my crap. I am fine if you rewamp
>> this ugly code into matplotlib testsuite, provide similar function (the API mentioned above)
>> so that I could use your code directly. That would be great. I just tried to show multiple
>> issues at once, notably that is why I included those unused functions. You will for sure find
>> a way to use them.
>>
>> Regarding the "unnecessary" del() calls etc., I think I have to use keep some, Ben, because
>> the function is not always left soon enough. I could drop some, you are right, but for some
>> I don't think so. Matplotlib cannot recycle the memory until me (upstream) deletes the reference
>> so ... go and test this lousy code. Now you have a testcase. ;) Same with the gc.collect() calls.
>> Actually, the main loop with 10 iteration is there just to show why I always want to clear
>> a figure when entering a function and while leaving it as well. It happened too many times that
>> I drawed over an old figure, and this was posted also few times on this list by others. That is
>> a weird behavior in my opinion. We, users, are just forced to use too low-level functions.
>>
>> So, have fun eating your memory! :))
>> Martin
> 
> 
> -- 
> _ 
> |\/|o _|_ _. _ | | \.__ __|__|_|_ _ _ ._ _ 
> | ||(_| |(_|(/_| |_/|(_)(/_|_ |_|_)(_)(_)| | | 
> 
> http://www.droettboom.com
> 
-- 
Martin Mokrejs, Ph.D.
Bioinformatics
Donovalska 1658
149 00 Prague
Czech Republic
http://www.iresite.org
http://www.iresite.org/~mmokrejs

2013年10月14日 Mark Lawrence <bre...@ya...>:
> On 14/10/2013 13:51, OCuanachain, Oisin (Oisin) wrote:
>> Hi,
>>
>> I am having problems with a script. It runs a number of iterations and
>> plots and saves a number of plots on each iteration. After the plots
>> have been saved I issue the pyplot.close(‘all’) command so despite many
>> plots being created only 4 should be open at any given time which should
>> not cause any memory problems. When I run the script however I see the
>> RAM usage gradually growing without bound and eventually causing the
>> script to crash. Interestingly I have found if I comment out the
>> pyplot.ion() and pyplot.ioff() the problem vanishes. So I do have a
>> workaround but it would still be good to have this fixed in case I
>> forget about it in future and loose another weekend’s work.
>>
>> My OS is Windows XP Service Pack 3
>> Python 2.6
>> Matplotlib 1.0.1
>>
>
> Is this actually a matplotlib problem or could it be a Windows problem
> as discussed here http://bugs.python.org/issue19246 ?
I think this is different. That bug report is not about RAM usage
growing without bound but memory allocation failing with plenty of RAM
available.
Goyo

On 2013年10月14日 8:26 AM, OCuanachain, Oisin (Oisin) wrote:
>
>
> Hi Eric,
>
> If .emf is no longer supported in current versions of matplotlib is
> there an alternative SVG-type format I can use ? I use .emf because I
> find that it tends to produce the clearest plots independent of how I
> re-size them when imported into documents. If memory serves I used to
> have a lot of problems with legibility when using raster-type formats
> like .png. If I get a chance I will try installing an up-to-date
> version of matplotlib and see if I can re-produce the behaviour.
>
> Oisin.
>
Oisin,
The ps, pdf, and svg vector formats are fully supported. (Postscript 
inherently lacks support for transparency, though, so pdf and svg are 
recommended.)
Eric

-----Bunteachtaireacht-----
From: Eric Firing [mailto:ef...@ha...] 
Sent: 14 October 2013 19:09
To: mat...@li...
Subject: Re: [Matplotlib-users] Maidir le: Memory leak when using pyplot.ion() ?
On 2013年10月14日 7:48 AM, OCuanachain, Oisin (Oisin) wrote:
> Hi Mike,
>
> ion(), ioff() are useful to get immediate feedback when developing a 
> script, when it is fully debugged I then increase the number of 
> iterations and leave it running over the weekend. At that point I 
> could obviously also have removed ion(), ioff() but given that I had 
> no idea that this was necessary my sim crashed and I lost a weekend's 
> worth of sim time. Anyway, whether or not ion(),ioff() are needed in 
> this particular script is really besides the point. If the script, 
> however unusual, is revealing a bug in matplotlib it should be logged 
> so that it can hopefully be fixed.
>
> Oisín
Oisín,
Certainly we want to find and fix bugs, with memory leaks being high priority. (I don't think we have seen a genuine mpl memory leak for quite a while; I am not aware of any at present.) We are not trying to maintain old mpl versions such as 1.0.1, however. You are using the emf backend, which has been removed. Therefore, unless you can reproduce the problem with mpl 1.3.x or 1.4.x in a SSCCE (http://sscce.org/), it is unlikely that your report will lead to a bug fix. Perhaps it will lead to some useful insight, however.
I see someone has suggested that the problem might be in Windows. 
Another possibility is that it is in TkAgg, which I suspect is your default interactive backend. I dimly recall that there was a time when TkAgg could leak memory, but I don't remember whether that was fixed by
1.0.1 or not.
Eric
Hi Eric, 
If .emf is no longer supported in current versions of matplotlib is there an alternative SVG-type format I can use ? I use .emf because I find that it tends to produce the clearest plots independent of how I re-size them when imported into documents. If memory serves I used to have a lot of problems with legibility when using raster-type formats like .png. If I get a chance I will try installing an up-to-date version of matplotlib and see if I can re-produce the behaviour. 
Oisin. 

On 2013年10月14日 7:48 AM, OCuanachain, Oisin (Oisin) wrote:
> Hi Mike,
>
> ion(), ioff() are useful to get immediate feedback when developing a
> script, when it is fully debugged I then increase the number of
> iterations and leave it running over the weekend. At that point I could
> obviously also have removed ion(), ioff() but given that I had no idea
> that this was necessary my sim crashed and I lost a weekend’s worth of
> sim time. Anyway, whether or not ion(),ioff() are needed in this
> particular script is really besides the point. If the script, however
> unusual, is revealing a bug in matplotlib it should be logged so that it
> can hopefully be fixed.
>
> Oisín
Oisín,
Certainly we want to find and fix bugs, with memory leaks being high 
priority. (I don't think we have seen a genuine mpl memory leak for 
quite a while; I am not aware of any at present.) We are not trying to 
maintain old mpl versions such as 1.0.1, however. You are using the emf 
backend, which has been removed. Therefore, unless you can reproduce 
the problem with mpl 1.3.x or 1.4.x in a SSCCE (http://sscce.org/), it 
is unlikely that your report will lead to a bug fix. Perhaps it will 
lead to some useful insight, however.
I see someone has suggested that the problem might be in Windows. 
Another possibility is that it is in TkAgg, which I suspect is your 
default interactive backend. I dimly recall that there was a time when 
TkAgg could leak memory, but I don't remember whether that was fixed by 
1.0.1 or not.
Eric

I see you are using matplotlib 1.0.1. There have been several memory leak
bugs fixed since then, so I would suggest upgrading. I also notice you are
using the "emf" backend for saving figures. If I remember correctly, that
backend has been deprecated (or maybe even removed) in the latest release
(v1.3.x). So, i would suggest trying v1.2.1. It does contain many bugfixes
and should still be compatible with your existing code.
I should also warn that the behavior with pyplot.ion() was "odd" back in
the 1.0.1 days and prior. If you are just simply running the script as-is
with a more recent matplotlib, or with a different backend in v1.0.1, the
script might behave a little differently than you'd expect. We would
welcome feedback on your usage of pyplot.ion().
Cheers!
Ben Root

On 14/10/2013 13:51, OCuanachain, Oisin (Oisin) wrote:
> Hi,
>
> I am having problems with a script. It runs a number of iterations and
> plots and saves a number of plots on each iteration. After the plots
> have been saved I issue the pyplot.close(‘all’) command so despite many
> plots being created only 4 should be open at any given time which should
> not cause any memory problems. When I run the script however I see the
> RAM usage gradually growing without bound and eventually causing the
> script to crash. Interestingly I have found if I comment out the
> pyplot.ion() and pyplot.ioff() the problem vanishes. So I do have a
> workaround but it would still be good to have this fixed in case I
> forget about it in future and loose another weekend’s work.
>
> My OS is Windows XP Service Pack 3
> Python 2.6
> Matplotlib 1.0.1
>
Is this actually a matplotlib problem or could it be a Windows problem 
as discussed here http://bugs.python.org/issue19246 ?
-- 
Roses are red,
Violets are blue,
Most poems rhyme,
But this one doesn't.
Mark Lawrence

Hi Mike,
ion(), ioff() are useful to get immediate feedback when developing a script, when it is fully debugged I then increase the number of iterations and leave it running over the weekend. At that point I could obviously also have removed ion(), ioff() but given that I had no idea that this was necessary my sim crashed and I lost a weekend's worth of sim time. Anyway, whether or not ion(),ioff() are needed in this particular script is really besides the point. If the script, however unusual, is revealing a bug in matplotlib it should be logged so that it can hopefully be fixed.
Oisín
From: Michael Droettboom [mailto:md...@st...]
Sent: 14 October 2013 18:13
To: mat...@li...
Subject: Re: [Matplotlib-users] Memory leak when using pyplot.ion() ?
I haven't had a chance to look into where the memory is actually leaking, ion/ioff are intended for interactive use, and here you are saving a large number of plots to files. Why do you need ion at all?
Mike
On 10/14/2013 08:51 AM, OCuanachain, Oisin (Oisin) wrote:
Hi,
I am having problems with a script. It runs a number of iterations and plots and saves a number of plots on each iteration. After the plots have been saved I issue the pyplot.close('all') command so despite many plots being created only 4 should be open at any given time which should not cause any memory problems. When I run the script however I see the RAM usage gradually growing without bound and eventually causing the script to crash. Interestingly I have found if I comment out the pyplot.ion() and pyplot.ioff() the problem vanishes. So I do have a workaround but it would still be good to have this fixed in case I forget about it in future and loose another weekend's work.
My OS is Windows XP Service Pack 3
Python 2.6
Matplotlib 1.0.1
The code below is a stripped down version of my script which still exhibits the problem.
Oisín.
# -*- coding: utf-8 -*-
import sys
import time
import numpy as np
from matplotlib import pyplot
import os
 # Main script body
try:
 for gain in range(1,20,2):
 for PortToTest in range(8):
 dirname = '.\crash'
 f = open(dirname + '\\results.m','w')
 runname = '\P' + str(PortToTest) + str(gain) + \
 '_' + time.strftime('d%dh%Hm%Ms%S')
 dirname = dirname + runname
 os.mkdir(dirname)
 os.system('copy ' + sys.argv[0] + ' ' + dirname )
 nIts = 50
 # Decimate data for plotting if many iterations are run
 if(nIts>10):
 echoPlotDec = 10
 else:
 echoPlotDec = 1
 ResidN = np.zeros((4,2*nIts))
 MaxSl = np.zeros((4,2*nIts))
 MaxOld = np.zeros((4,2*nIts))
 MaxNew = np.zeros((4,2*nIts))
 EchoA = np.zeros((2*nIts,160))
 for kk in range(2*nIts):
 ResidN[0,kk] = np.random.rand(1,1)
 ResidN[1,kk] = np.random.rand(1,1)
 ResidN[2,kk] = np.random.rand(1,1)
 ResidN[3,kk] = np.random.rand(1,1)
 MaxSl[0,kk] = np.random.rand(1,1)
 MaxSl[1,kk] = np.random.rand(1,1)
 MaxSl[2,kk] = np.random.rand(1,1)
 MaxSl[3,kk] = np.random.rand(1,1)
 MaxOld[0,kk] = np.random.rand(1,1)
 MaxOld[1,kk] = np.random.rand(1,1)
 MaxOld[2,kk] = np.random.rand(1,1)
 MaxOld[3,kk] = np.random.rand(1,1)
 MaxNew[0,kk] = np.random.rand(1,1)
 MaxNew[1,kk] = np.random.rand(1,1)
 MaxNew[2,kk] = np.random.rand(1,1)
 MaxNew[3,kk] = np.random.rand(1,1)
 EchoA[kk,:] = np.random.rand(1,160)
 f.close()
 pyplot.ion()
 pyplot.figure()
 pyplot.hold(True)
 LegendTexts = ("A","B","C","D")
 pyplot.title("R (" + runname +")")
 pyplot.xlabel("Index")
 pyplot.ylabel("Noise (dB)")
 pyplot.grid(True)
 pyplot.hold(True)
 pyplot.plot(np.transpose(ResidN),'.-')
 pyplot.legend(LegendTexts,loc=1)
 pyplot.axis([0, 2*nIts, -33, -25])
 pyplot.savefig(dirname + '\\results.emf',format='emf')
 pyplot.figure()
 pyplot.hold(True)
 pyplot.title("Coefs")
 pyplot.xlabel("Coef Index")
 pyplot.ylabel("Coef Value")
 pyplot.grid(True)
 pyplot.hold(True)
 pyplot.plot(np.transpose(EchoA[0:nIts-1:echoPlotDec,:]),'.-')
 pyplot.plot(np.transpose(EchoA[nIts:2*nIts-1:echoPlotDec,:]),'*-')
 pyplot.axis([0, 160, -0.5, 2])
 pyplot.savefig(dirname + '\\CoefsA.emf',format='emf')
 pyplot.figure()
 pyplot.hold(True)
 pyplot.title("MaxAbs, Old = '.', New = '*' ")
 pyplot.xlabel("Iteration")
 pyplot.ylabel("o/p (LSBs)")
 pyplot.grid(True)
 pyplot.hold(True)
 pyplot.plot(np.transpose(MaxOld),'.-')
 pyplot.plot(np.transpose(MaxNew),'*-')
 pyplot.axis([0, 2*nIts, 32, 128])
 pyplot.savefig(dirname + '\\MaxAbsA.emf',format='emf')
 pyplot.figure()
 pyplot.hold(True)
 pyplot.title("MaxAbs")
 pyplot.xlabel("Iteration")
 pyplot.ylabel("(LSBs)")
 pyplot.grid(True)
 pyplot.hold(True)
 pyplot.plot(np.transpose(MaxSl),'.-')
 pyplot.axis([0, 2*nIts, 0, 64])
 pyplot.savefig(dirname + '\\MaxAbsSl.emf',format='emf')
 pyplot.close('all')
except RuntimeError, msg:
 print 'Exception occurred in main script body'
 print >>sys.stderr, msg
 raise
finally:
 print "Test done"
 # Display plots
 pyplot.ioff()
------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
_______________________________________________
Matplotlib-users mailing list
Mat...@li...<mailto:Mat...@li...>
https://lists.sourceforge.net/lists/listinfo/matplotlib-users
--
 _
|\/|o _|_ _. _ | | \.__ __|__|_|_ _ _ ._ _
| ||(_| |(_|(/_| |_/|(_)(/_|_ |_|_)(_)(_)| | |
http://www.droettboom.com

I haven't had a chance to look into where the memory is actually 
leaking, ion/ioff are intended for interactive use, and here you are 
saving a large number of plots to files. Why do you need ion at all?
Mike
On 10/14/2013 08:51 AM, OCuanachain, Oisin (Oisin) wrote:
>
> Hi,
>
> I am having problems with a script. It runs a number of iterations and 
> plots and saves a number of plots on each iteration. After the plots 
> have been saved I issue the pyplot.close('all') command so despite 
> many plots being created only 4 should be open at any given time which 
> should not cause any memory problems. When I run the script however I 
> see the RAM usage gradually growing without bound and eventually 
> causing the script to crash. Interestingly I have found if I comment 
> out the pyplot.ion() and pyplot.ioff() the problem vanishes. So I do 
> have a workaround but it would still be good to have this fixed in 
> case I forget about it in future and loose another weekend's work.
>
> My OS is Windows XP Service Pack 3
>
> Python 2.6
>
> Matplotlib 1.0.1
>
> The code below is a stripped down version of my script which still 
> exhibits the problem.
>
> Oisín.
>
> # -*- coding: utf-8 -*-
>
> import sys
>
> import time
>
> import numpy as np
>
> from matplotlib import pyplot
>
> import os
>
> # Main script body
>
> try:
>
> for gain in range(1,20,2):
>
> for PortToTest in range(8):
>
> dirname = '.\crash'
>
> f = open(dirname + '\\results.m','w')
>
> runname = '\P' + str(PortToTest) + str(gain) + \
>
> '_' + time.strftime('d%dh%Hm%Ms%S')
>
> dirname = dirname + runname
>
> os.mkdir(dirname)
>
> os.system('copy ' + sys.argv[0] + ' ' + dirname )
>
> nIts = 50
>
> # Decimate data for plotting if many iterations are run
>
> if(nIts>10):
>
> echoPlotDec = 10
>
> else:
>
> echoPlotDec = 1
>
> ResidN = np.zeros((4,2*nIts))
>
> MaxSl = np.zeros((4,2*nIts))
>
> MaxOld = np.zeros((4,2*nIts))
>
> MaxNew = np.zeros((4,2*nIts))
>
> EchoA = np.zeros((2*nIts,160))
>
> for kk in range(2*nIts):
>
> ResidN[0,kk] = np.random.rand(1,1)
>
> ResidN[1,kk] = np.random.rand(1,1)
>
> ResidN[2,kk] = np.random.rand(1,1)
>
> ResidN[3,kk] = np.random.rand(1,1)
>
> MaxSl[0,kk] = np.random.rand(1,1)
>
> MaxSl[1,kk] = np.random.rand(1,1)
>
> MaxSl[2,kk] = np.random.rand(1,1)
>
> MaxSl[3,kk] = np.random.rand(1,1)
>
> MaxOld[0,kk] = np.random.rand(1,1)
>
> MaxOld[1,kk] = np.random.rand(1,1)
>
> MaxOld[2,kk] = np.random.rand(1,1)
>
> MaxOld[3,kk] = np.random.rand(1,1)
>
> MaxNew[0,kk] = np.random.rand(1,1)
>
> MaxNew[1,kk] = np.random.rand(1,1)
>
> MaxNew[2,kk] = np.random.rand(1,1)
>
> MaxNew[3,kk] = np.random.rand(1,1)
>
> EchoA[kk,:] = np.random.rand(1,160)
>
> f.close()
>
> pyplot.ion()
>
> pyplot.figure()
>
> pyplot.hold(True)
>
> LegendTexts = ("A","B","C","D")
>
> pyplot.title("R (" + runname +")")
>
> pyplot.xlabel("Index")
>
> pyplot.ylabel("Noise (dB)")
>
> pyplot.grid(True)
>
> pyplot.hold(True)
>
> pyplot.plot(np.transpose(ResidN),'.-')
>
> pyplot.legend(LegendTexts,loc=1)
>
> pyplot.axis([0, 2*nIts, -33, -25])
>
> pyplot.savefig(dirname + '\\results.emf',format='emf')
>
> pyplot.figure()
>
> pyplot.hold(True)
>
> pyplot.title("Coefs")
>
> pyplot.xlabel("Coef Index")
>
> pyplot.ylabel("Coef Value")
>
> pyplot.grid(True)
>
> pyplot.hold(True)
>
> pyplot.plot(np.transpose(EchoA[0:nIts-1:echoPlotDec,:]),'.-')
>
> pyplot.plot(np.transpose(EchoA[nIts:2*nIts-1:echoPlotDec,:]),'*-')
>
> pyplot.axis([0, 160, -0.5, 2])
>
> pyplot.savefig(dirname + '\\CoefsA.emf',format='emf')
>
> pyplot.figure()
>
> pyplot.hold(True)
>
> pyplot.title("MaxAbs, Old = '.', New = '*' ")
>
> pyplot.xlabel("Iteration")
>
> pyplot.ylabel("o/p (LSBs)")
>
> pyplot.grid(True)
>
> pyplot.hold(True)
>
> pyplot.plot(np.transpose(MaxOld),'.-')
>
> pyplot.plot(np.transpose(MaxNew),'*-')
>
> pyplot.axis([0, 2*nIts, 32, 128])
>
> pyplot.savefig(dirname + '\\MaxAbsA.emf',format='emf')
>
> pyplot.figure()
>
> pyplot.hold(True)
>
> pyplot.title("MaxAbs")
>
> pyplot.xlabel("Iteration")
>
> pyplot.ylabel("(LSBs)")
>
> pyplot.grid(True)
>
> pyplot.hold(True)
>
> pyplot.plot(np.transpose(MaxSl),'.-')
>
> pyplot.axis([0, 2*nIts, 0, 64])
>
> pyplot.savefig(dirname + '\\MaxAbsSl.emf',format='emf')
>
> pyplot.close('all')
>
> except RuntimeError, msg:
>
> print 'Exception occurred in main script body'
>
> print >>sys.stderr, msg
>
> raise
>
> finally:
>
> print "Test done"
>
> # Display plots
>
> pyplot.ioff()
>
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
>
>
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
-- 
 _
|\/|o _|_ _. _ | | \.__ __|__|_|_ _ _ ._ _
| ||(_| |(_|(/_| |_/|(_)(/_|_ |_|_)(_)(_)| | |
http://www.droettboom.com

Sorry to repeat myself, but please reduce this to a short, self 
contained example, that is absolutely minimal to demonstrate the 
problem. http://sscce.org/ should help better explain what I'm after. I 
don't want to find the needle in the haystack here -- there is code in 
your example that doesn't even run, for example.
That said, are you really after creating a legend entry for each of the 
dots? (See below). That just isn't going to work, and I'm not 
surprised it eats up excessive amounts of memory. I think you want (and 
can) reduce this to a single scatter call.
_series = [_ax1.scatter(_x, _y, color=_c, s=objsize, label=_l, 
hatch='.') for _x, _y, _c, _l in izip(mydata_x, mydata_y, colors, 
legends)] # returns PathCollection object
Mike
On 10/12/2013 12:57 PM, Martin MOKREJŠ wrote:
> Hi,
> so here is some quick but working example. I added there are 2-3 functions (unused)
> as a bonus, you can easily call them from the main function using same API
> (except the piechart). I hope this shows what I lack in matplotlib - a general API
> so that I could easily switch form scatter plot to piechart or barchart without altering
> much the function arguments. Messing with return objects line2D, PathCollection, Rectangle
> is awkward and I would like to stay away from matplotlib's internals. ;) Some can be sliced,
> so not, you will see in the code.
>
> This eatmem.py will take easily all your memory. Drawing 300000 dots is not feasible
> with 16GB of RAM. While the example is for sure inefficient in many places generating the data
> in python does not eat RAM. That happens afterwards.
>
> I would really like to hear whether matplotlib could be adjusted instead. ;) I already mentioned
> in this thread that it is awkward to pre-create colors before passing all data to a drawing
> function. I think we could all save a lot if matplotlib could dynamically fetch colors
> on the fly from user-created generator, same for legends descriptions. I think my example
> code shows the inefficient approach here. Would I have more time I would randomize a bit
> more the sublist of each series so that the numbers in legends would be more variable
> but that is a cosmetic issue.
> Probably due to my ignorance you will see that figures with legends have different font
> sizes, axes are rescaled and the figure. Of course I wanted to have the drawing same via both
> approaches but failed badly. The files/figures with legends should be just accompanied by the
> legend "table" underneath but the drawing itself should be same. Maybe an issue with DPI settings
> but not only.
>
> I placed some comments in the code, please don't take them in person. ;) Of course
> I am glad for the existing work and am happy to contribute my crap. I am fine if you rewamp
> this ugly code into matplotlib testsuite, provide similar function (the API mentioned above)
> so that I could use your code directly. That would be great. I just tried to show multiple
> issues at once, notably that is why I included those unused functions. You will for sure find
> a way to use them.
>
> Regarding the "unnecessary" del() calls etc., I think I have to use keep some, Ben, because
> the function is not always left soon enough. I could drop some, you are right, but for some
> I don't think so. Matplotlib cannot recycle the memory until me (upstream) deletes the reference
> so ... go and test this lousy code. Now you have a testcase. ;) Same with the gc.collect() calls.
> Actually, the main loop with 10 iteration is there just to show why I always want to clear
> a figure when entering a function and while leaving it as well. It happened too many times that
> I drawed over an old figure, and this was posted also few times on this list by others. That is
> a weird behavior in my opinion. We, users, are just forced to use too low-level functions.
>
> So, have fun eating your memory! :))
> Martin
-- 
 _
|\/|o _|_ _. _ | | \.__ __|__|_|_ _ _ ._ _
| ||(_| |(_|(/_| |_/|(_)(/_|_ |_|_)(_)(_)| | |
http://www.droettboom.com

Hi,
I am having problems with a script. It runs a number of iterations and plots and saves a number of plots on each iteration. After the plots have been saved I issue the pyplot.close('all') command so despite many plots being created only 4 should be open at any given time which should not cause any memory problems. When I run the script however I see the RAM usage gradually growing without bound and eventually causing the script to crash. Interestingly I have found if I comment out the pyplot.ion() and pyplot.ioff() the problem vanishes. So I do have a workaround but it would still be good to have this fixed in case I forget about it in future and loose another weekend's work.
My OS is Windows XP Service Pack 3
Python 2.6
Matplotlib 1.0.1
The code below is a stripped down version of my script which still exhibits the problem.
Oisín.
# -*- coding: utf-8 -*-
import sys
import time
import numpy as np
from matplotlib import pyplot
import os
 # Main script body
try:
 for gain in range(1,20,2):
 for PortToTest in range(8):
 dirname = '.\crash'
 f = open(dirname + '\\results.m','w')
 runname = '\P' + str(PortToTest) + str(gain) + \
 '_' + time.strftime('d%dh%Hm%Ms%S')
 dirname = dirname + runname
 os.mkdir(dirname)
 os.system('copy ' + sys.argv[0] + ' ' + dirname )
 nIts = 50
 # Decimate data for plotting if many iterations are run
 if(nIts>10):
 echoPlotDec = 10
 else:
 echoPlotDec = 1
 ResidN = np.zeros((4,2*nIts))
 MaxSl = np.zeros((4,2*nIts))
 MaxOld = np.zeros((4,2*nIts))
 MaxNew = np.zeros((4,2*nIts))
 EchoA = np.zeros((2*nIts,160))
 for kk in range(2*nIts):
 ResidN[0,kk] = np.random.rand(1,1)
 ResidN[1,kk] = np.random.rand(1,1)
 ResidN[2,kk] = np.random.rand(1,1)
 ResidN[3,kk] = np.random.rand(1,1)
 MaxSl[0,kk] = np.random.rand(1,1)
 MaxSl[1,kk] = np.random.rand(1,1)
 MaxSl[2,kk] = np.random.rand(1,1)
 MaxSl[3,kk] = np.random.rand(1,1)
 MaxOld[0,kk] = np.random.rand(1,1)
 MaxOld[1,kk] = np.random.rand(1,1)
 MaxOld[2,kk] = np.random.rand(1,1)
 MaxOld[3,kk] = np.random.rand(1,1)
 MaxNew[0,kk] = np.random.rand(1,1)
 MaxNew[1,kk] = np.random.rand(1,1)
 MaxNew[2,kk] = np.random.rand(1,1)
 MaxNew[3,kk] = np.random.rand(1,1)
 EchoA[kk,:] = np.random.rand(1,160)
 f.close()
 pyplot.ion()
 pyplot.figure()
 pyplot.hold(True)
 LegendTexts = ("A","B","C","D")
 pyplot.title("R (" + runname +")")
 pyplot.xlabel("Index")
 pyplot.ylabel("Noise (dB)")
 pyplot.grid(True)
 pyplot.hold(True)
 pyplot.plot(np.transpose(ResidN),'.-')
 pyplot.legend(LegendTexts,loc=1)
 pyplot.axis([0, 2*nIts, -33, -25])
 pyplot.savefig(dirname + '\\results.emf',format='emf')
 pyplot.figure()
 pyplot.hold(True)
 pyplot.title("Coefs")
 pyplot.xlabel("Coef Index")
 pyplot.ylabel("Coef Value")
 pyplot.grid(True)
 pyplot.hold(True)
 pyplot.plot(np.transpose(EchoA[0:nIts-1:echoPlotDec,:]),'.-')
 pyplot.plot(np.transpose(EchoA[nIts:2*nIts-1:echoPlotDec,:]),'*-')
 pyplot.axis([0, 160, -0.5, 2])
 pyplot.savefig(dirname + '\\CoefsA.emf',format='emf')
 pyplot.figure()
 pyplot.hold(True)
 pyplot.title("MaxAbs, Old = '.', New = '*' ")
 pyplot.xlabel("Iteration")
 pyplot.ylabel("o/p (LSBs)")
 pyplot.grid(True)
 pyplot.hold(True)
 pyplot.plot(np.transpose(MaxOld),'.-')
 pyplot.plot(np.transpose(MaxNew),'*-')
 pyplot.axis([0, 2*nIts, 32, 128])
 pyplot.savefig(dirname + '\\MaxAbsA.emf',format='emf')
 pyplot.figure()
 pyplot.hold(True)
 pyplot.title("MaxAbs")
 pyplot.xlabel("Iteration")
 pyplot.ylabel("(LSBs)")
 pyplot.grid(True)
 pyplot.hold(True)
 pyplot.plot(np.transpose(MaxSl),'.-')
 pyplot.axis([0, 2*nIts, 0, 64])
 pyplot.savefig(dirname + '\\MaxAbsSl.emf',format='emf')
 pyplot.close('all')
except RuntimeError, msg:
 print 'Exception occurred in main script body'
 print >>sys.stderr, msg
 raise
finally:
 print "Test done"
 # Display plots
 pyplot.ioff()

On 10/13/2013 7:49 PM, Aiyong WANG wrote:
> Today I setuped a ubuntu virtual machine, installed matplotlib 1.3.1
> (builded and installed from source), installed ImageMagick via
> 'apt-get', and tried again that sample.
> It worked.
>
> So Does the imagemagick writer not work on Windows, or am I missing
> something in my Win7 ?
>
> Can anybody help ?
> Thank you.
>
>
> 2013年10月13日 Aiyong WANG <gep...@gm... <mailto:gep...@gm...>>
>
> Hi.
> I'm using windows 7, with python(x,y) 2.7.5.0 installed. I then
> installed matplotlib 1.3.1 and imagemagick.
> Something goes wrong when try to save animation as a gif file.
> I used a base_animation.py within ipython notebook as followed:
>
> code:
> -------------------------------------------------------------------------------------
> import numpy as np
> from matplotlib import pyplot as plt
> from matplotlib import animation
> # First set up the figure, the axis, and the plot element we want to
> animate
> fig = plt.figure()
> ax = fig.add_subplot(111, xlim=(0, 2), ylim=(-2, 2))
> line, = ax.plot([], [], lw=2)
> # initialization function: plot the background of each frame
> def init():
> line.set_data([], [])
> return line,
> # animation function. This is called sequentially
> def animate(i):
> x = np.linspace(0, 2, 1000)
> y = np.sin(2 * np.pi * (x - 0.01 * i))
> line.set_data(x, y)
> return line,
> # call the animator. blit=True means only re-draw the parts that
> have changed.
> anim = animation.FuncAnimation(fig, animate, init_func=init,
> frames=100, interval=20, blit=True)
> # this is how you save your animation to file:
> #anim.save('animation.gif', writer='imagemagick_file', fps=30)
> anim.save('animation.gif', writer='imagemagick', fps=30)
> plt.show()
> ----------------------------------------------------------------------------------
>
>
>
> And I got an error message like this:
>
> ---------------------------------------------------------------------------
> RuntimeError Traceback (most recent call last)
> <ipython-input-2-7b2f7b9edcb4> in<module>()
> 26 # this is how you save your animation to file:
> 27 #anim.save('animation.gif', writer='imagemagick_file', fps=30)
> ---> 28 anim.save('animation.gif', writer='imagemagick', fps=30)
> 29
> 30 plt.show()
>
> D:\Python27\lib\site-packages\matplotlib\animation.pyc insave(self, filename, writer, fps, dpi, codec, bitrate, extra_args, metadata, extra_anim, savefig_kwargs)
> 716 #TODO: Need to see if turning off blit is really necessary
> 717 anim._draw_next_frame(d, blit=False)
> --> 718 writer.grab_frame(**savefig_kwargs)
> 719
> 720 # Reconnect signal for first draw if necessary
>
> D:\Python27\lib\site-packages\matplotlib\animation.pyc ingrab_frame(self, **savefig_kwargs)
> 202 # frame format and dpi.
> 203 self.fig.savefig(self._frame_sink(), format=self.frame_format,
> --> 204 dpi=self.dpi, **savefig_kwargs)
> 205 except RuntimeError:
> 206 out, err= self._proc.communicate()
>
> D:\Python27\lib\site-packages\matplotlib\figure.pyc insavefig(self, *args, **kwargs)
> 1419 self.set_frameon(frameon)
> 1420
> -> 1421 self.canvas.print_figure(*args, **kwargs)
> 1422
> 1423 if frameon:
>
> D:\Python27\lib\site-packages\matplotlib\backend_bases.pyc inprint_figure(self, filename, dpi, facecolor, edgecolor, orientation, format, **kwargs)
> 2218 orientation=orientation,
> 2219 bbox_inches_restore=_bbox_inches_restore,
> -> 2220 **kwargs)
> 2221 finally:
> 2222 if bbox_inchesand restore_bbox:
>
> D:\Python27\lib\site-packages\matplotlib\backends\backend_agg.pyc inprint_raw(self, filename_or_obj, *args, **kwargs)
> 495 close= False
> 496 try:
> --> 497 renderer._renderer.write_rgba(filename_or_obj)
> 498 finally:
> 499 if close:
>
> RuntimeError: Error writing to file
>
> --
>
> Can anybody help?
> ------------------------------------------------------------------
> WANG Aiyong
>
>
>
>
> --
> ------------------------------------------------------------------
> WANG Aiyong
`convert.exe` is a Windows system file. Set `animation.convert_path` in 
`matplotlib.rc` to the full path to ImageMagick's `convert.exe`. Works 
for me.
Christoph

Today I setuped a ubuntu virtual machine, installed matplotlib 1.3.1
(builded and installed from source), installed ImageMagick via 'apt-get',
and tried again that sample.
It worked.
So Does the imagemagick writer not work on Windows, or am I missing
something in my Win7 ?
Can anybody help ?
Thank you.
2013年10月13日 Aiyong WANG <gep...@gm...>
> Hi.
> I'm using windows 7, with python(x,y) 2.7.5.0 installed. I then installed
> matplotlib 1.3.1 and imagemagick.
> Something goes wrong when try to save animation as a gif file.
> I used a base_animation.py within ipython notebook as followed:
>
> code:
>
> -------------------------------------------------------------------------------------
> import numpy as np
> from matplotlib import pyplot as plt
> from matplotlib import animation
>
> # First set up the figure, the axis, and the plot element we want to
> animate
> fig = plt.figure()
> ax = fig.add_subplot(111, xlim=(0, 2), ylim=(-2, 2))
> line, = ax.plot([], [], lw=2)
>
> # initialization function: plot the background of each frame
> def init():
> line.set_data([], [])
> return line,
>
> # animation function. This is called sequentially
> def animate(i):
> x = np.linspace(0, 2, 1000)
> y = np.sin(2 * np.pi * (x - 0.01 * i))
> line.set_data(x, y)
> return line,
>
> # call the animator. blit=True means only re-draw the parts that have
> changed.
> anim = animation.FuncAnimation(fig, animate, init_func=init,
> frames=100, interval=20, blit=True)
>
> # this is how you save your animation to file:
> #anim.save('animation.gif', writer='imagemagick_file', fps=30)
> anim.save('animation.gif', writer='imagemagick', fps=30)
>
> plt.show()
>
> ----------------------------------------------------------------------------------
>
>
>
> And I got an error message like this:
>
> ---------------------------------------------------------------------------RuntimeError Traceback (most recent call last)<ipython-input-2-7b2f7b9edcb4> in <module>() 26 # this is how you save your animation to file: 27 #anim.save('animation.gif', writer='imagemagick_file', fps=30)---> 28 anim.save('animation.gif', writer='imagemagick', fps=30) 29 30 plt.show()
> D:\Python27\lib\site-packages\matplotlib\animation.pyc in save(self, filename, writer, fps, dpi, codec, bitrate, extra_args, metadata, extra_anim, savefig_kwargs) 716 #TODO: Need to see if turning off blit is really necessary 717 anim._draw_next_frame(d, blit=False)--> 718 writer.grab_frame(**savefig_kwargs) 719 720 # Reconnect signal for first draw if necessary
> D:\Python27\lib\site-packages\matplotlib\animation.pyc in grab_frame(self, **savefig_kwargs) 202 # frame format and dpi. 203 self.fig.savefig(self._frame_sink(), format=self.frame_format,--> 204 dpi=self.dpi, **savefig_kwargs) 205 except RuntimeError: 206 out, err = self._proc.communicate()
> D:\Python27\lib\site-packages\matplotlib\figure.pyc in savefig(self, *args, **kwargs) 1419 self.set_frameon(frameon) 1420 -> 1421 self.canvas.print_figure(*args, **kwargs) 1422 1423 if frameon:
> D:\Python27\lib\site-packages\matplotlib\backend_bases.pyc in print_figure(self, filename, dpi, facecolor, edgecolor, orientation, format, **kwargs) 2218 orientation=orientation, 2219 bbox_inches_restore=_bbox_inches_restore,-> 2220 **kwargs) 2221 finally: 2222 if bbox_inches and restore_bbox:
> D:\Python27\lib\site-packages\matplotlib\backends\backend_agg.pyc in print_raw(self, filename_or_obj, *args, **kwargs) 495 close = False 496 try:--> 497 renderer._renderer.write_rgba(filename_or_obj) 498 finally: 499 if close:
> RuntimeError: Error writing to file
>
> --
>
> Can anybody help?
> ------------------------------------------------------------------
> WANG Aiyong
>
-- 
------------------------------------------------------------------
WANG Aiyong