matplotlib-users Mailing List for matplotlib

Hi Ben,
 thank you for your comments. Looks I will have a bad sleep tonight. :( Some quick
answers below.
Benjamin Root wrote:
> 
> 
> 
> On Thu, Oct 10, 2013 at 10:21 AM, Michael Droettboom <md...@st... <mailto:md...@st...>> wrote:
> 
> Thanks. This is much more helpful.
> 
> What we need, however, is a "self contained, standalone example". The code below calls functions that are not present. See http://sscce.org/ for why this is so important. Again, I would have to guess what those functions do -- it may be relevant, it may not. If I have something that I can *just run* then I can use various introspection tools to see what is going wrong.
> 
> Mike
> 
> 
> That being said, I do see a number of anti-patterns here that could be significant. For example:
> 
> for _x, _y, _c in izip(mydata_x, mydata_y, colors):
> # _Line2D = _ax1.plot(_x, _y) # returns Line2D object
> _my_PathCollection = _ax1.scatter(_x, _y, color=_c, s=objsize) # , label=_l) # returns PathCollection object
> _series.append(_my_PathCollection)
> 
> Could be more concisely written as:
> 
> _series = [_ax1.scatter(_x, _y, color=_c, s=objsize) for _x, _y, _c in izip(mydata_x, mydata_y, colors)]
> 
> Python can then more intelligently handle memory management by intelligently allocating the memory for _series. You can then use _series.extend() for when you are doing the scatter plots for _ax2 with a similar list comprehension (or even a generator statement).
You are right the .append() is ugly, maybe is a the real source of troubles. I somehow
do not understand myself right now why under the "if legends:" use ax1 instead of ax2.
Weird. I actually stopped using legends with this function because that was my first guess
that they cause the memory issues. Seems the culprit is elsewhere so I should add them
back and likely fix the ax2 vs. ax1 copy/paste (most likely) error.
As you could have seen, I used in the past label=_l but for some reason I switched away
to the current ugly code. Will try to find out why I did that.
Hmm, I don't know what you mean with _series.extend() at the moment, will read some
python Intro on using lists. :(
> 
> I would also question the need to store _series in the first place. You use it for the call to legend, but you could have simply passed a label to each call of scatter as well.
As I said, I used that in the past but somehow that did not work. Maybe time to re-try that.
> 
> Some other things of note:
> 
> 1) The clear() call here is completely useless as the figure is already clear.
> _figure = pylab.figure()
> _figure.clear()
Right, I was just trying to ensure everything is cleared. I somewhat suspect python
garbage collector does not recycle too often, and therefore added more and more del()
and gc.collect() calls.
> 
> 2) When limits are set on an axis, autoscaling for that axis is automatically turned off anyway, so no need to turn if off yourself (also not sure why you are calling out to an external function here):
> _ax1.set_autoscale_on(False)
> set_limits(_ax1, xmin, xmax, ymin, ymax)
The set_limits() is called because I got unstable coordinates in every figure.
Sometimes, matplotlib used wider offset from the axes line while sometimes not.
So, I basically force same layout for expected layouts.
> 
> 3) Finally, some discussion on the end of your function here:
> if legends:
> _figure.savefig(filename, dpi=100) #, bbox_inches='tight')
> del(_my_PathCollection)
> del(_ax2)
> else:
> _figure.savefig(filename, dpi=100)
> 
> del(_series)
> del(_ax1)
> _figure.clear()
> del(_figure)
> pylab.clf()
> pylab.close()
> first, as discussed, you can easily eliminate the need for _my_PathCollection and possibly even _series. Second, when calling _figure.clear(), all of its axes objects are deleted for you, so you don't need to delete them yourself. Third, you delete the _figure object, but then call "pylab.clf()". I haven't double-checked exactly what would happen, but I think you might run the risk of accidentially clearing some other existing figure by doing that. Lastly, you then call pylab.close(), which I point out the same caveat as before. Really, all you needed was pylab.close() and you can eliminate the 5 preceding lines and the other two del()'s. All del() really does is remove the variable out of scope. Once that object is out of everybody's scope, then the gc can clean it up. Since the function was ending anyway, there is no point in deleting the variable.
Right, but I suspect that garbage collector does not recycle quickly enough unused objects
after the function is left. If I generate many figure sin a loop, one after another, it
appeared to me helpful to interleave the function calls with the gc.collect() calls.
> 
> I don't know if this would fix your problem, and there are a bunch of other style issues here (particularly, pylab really shouldn't be used this way), but hopefully this gives some food for thought.
I think I will start tomorrow finishing up the broken testcase so that we can be sure
where was the culprit. Then should improve the function as you proposed. I am not sure
some places what you really mean but will resolve it hopefully.
I was thinking about submitting several other functions like this one for discussion and
improvement, so that so that such wrapper functions could be included in matplotlib. I am
sure you would not like the many function argument and would prefer kwargs instead, but
something have same API would be helpful if I want to switch easily between scatter,
histplot, piechart. Actually, the hist2d substring in this function name is a remnant of
my attempts to do 2d charts but I did not take that route in the end. Just in case you were
puzzled by the function name. ;)
Thank you,
Martin
> 
> Cheers!
> Ben Root

I'm pleased to announce the release of matplotlib version 1.3.1. This is a bugfix release.
It may be downloaded from here, or installed through the package manager of your choice (when available):
http://matplotlib.org/downloads
The changelog is copied below:
New in 1.3.1
------------
1.3.1 is a bugfix release, primarily dealing with improved setup and
handling of dependencies, and correcting and enhancing the
documentation.
The following changes were made in 1.3.1 since 1.3.0.
Enhancements
````````````
- Added a context manager for creating multi-page pdfs (see
 `matplotlib.backends.backend_pdf.PdfPages`).
- The WebAgg backend should no have lower latency over heterogeneous
 Internet connections.
Bug fixes
`````````
- Histogram plots now contain the endline.
- Fixes to the Molleweide projection.
- Handling recent fonts from Microsoft and Macintosh-style fonts with
 non-ascii metadata is improved.
- Hatching of fill between plots now works correctly in the PDF
 backend.
- Tight bounding box support now works in the PGF backend.
- Transparent figures now display correctly in the Qt4Agg backend.
- Drawing lines from one subplot to another now works.
- Unit handling on masked arrays has been improved.
Setup and dependencies
``````````````````````
- Now works with any version of pyparsing 1.5.6 or later, without displaying
 hundreds of warnings.
- Now works with 64-bit versions of Ghostscript on MS-Windows.
- When installing from source into an environment without Numpy, Numpy
 will first be downloaded and built and then used to build
 matplotlib.
- Externally installed backends are now always imported using a
 fully-qualified path to the module.
- Works with newer version of wxPython.
- Can now build with a PyCXX installed globally on the system from source.
- Better detection of Gtk3 dependencies.
Testing
```````
- Tests should now work in non-English locales.
- PEP8 conformance tests now report on locations of issues.
Mike
-- 
 _
|\/|o _|_ _. _ | | \.__ __|__|_|_ _ _ ._ _
| ||(_| |(_|(/_| |_/|(_)(/_|_ |_|_)(_)(_)| | |
http://www.droettboom.com

On Thu, Oct 10, 2013 at 10:21 AM, Michael Droettboom <md...@st...>wrote:
> Thanks. This is much more helpful.
>
> What we need, however, is a "self contained, standalone example". The
> code below calls functions that are not present. See http://sscce.org/for why this is so important. Again, I would have to guess what those
> functions do -- it may be relevant, it may not. If I have something that I
> can *just run* then I can use various introspection tools to see what is
> going wrong.
>
> Mike
>
>
That being said, I do see a number of anti-patterns here that could be
significant. For example:
 for _x, _y, _c in izip(mydata_x, mydata_y, colors):
 # _Line2D = _ax1.plot(_x, _y) # returns Line2D object
 _my_PathCollection = _ax1.scatter(_x, _y, color=_c, s=objsize)
# , label=_l) # returns PathCollection object
 _series.append(_my_PathCollection)
Could be more concisely written as:
 _series = [_ax1.scatter(_x, _y, color=_c, s=objsize) for _x, _y, _c
in izip(mydata_x, mydata_y, colors)]
Python can then more intelligently handle memory management by
intelligently allocating the memory for _series. You can then use
_series.extend() for when you are doing the scatter plots for _ax2 with a
similar list comprehension (or even a generator statement).
I would also question the need to store _series in the first place. You use
it for the call to legend, but you could have simply passed a label to each
call of scatter as well.
Some other things of note:
1) The clear() call here is completely useless as the figure is already
clear.
 _figure = pylab.figure()
 _figure.clear()
2) When limits are set on an axis, autoscaling for that axis is
automatically turned off anyway, so no need to turn if off yourself (also
not sure why you are calling out to an external function here):
 _ax1.set_autoscale_on(False)
 set_limits(_ax1, xmin, xmax, ymin, ymax)
3) Finally, some discussion on the end of your function here:
 if legends:
 _figure.savefig(filename, dpi=100) #, bbox_inches='tight')
 del(_my_PathCollection)
 del(_ax2)
 else:
 _figure.savefig(filename, dpi=100)
 del(_series)
 del(_ax1)
 _figure.clear()
 del(_figure)
 pylab.clf()
 pylab.close()
first, as discussed, you can easily eliminate the need for
_my_PathCollection and possibly even _series. Second, when calling
_figure.clear(), all of its axes objects are deleted for you, so you don't
need to delete them yourself. Third, you delete the _figure object, but
then call "pylab.clf()". I haven't double-checked exactly what would
happen, but I think you might run the risk of accidentially clearing some
other existing figure by doing that. Lastly, you then call pylab.close(),
which I point out the same caveat as before. Really, all you needed was
pylab.close() and you can eliminate the 5 preceding lines and the other two
del()'s. All del() really does is remove the variable out of scope. Once
that object is out of everybody's scope, then the gc can clean it up. Since
the function was ending anyway, there is no point in deleting the variable.
I don't know if this would fix your problem, and there are a bunch of other
style issues here (particularly, pylab really shouldn't be used this way),
but hopefully this gives some food for thought.
Cheers!
Ben Root

Thanks. This is much more helpful.
What we need, however, is a "self contained, standalone example". The 
code below calls functions that are not present. See http://sscce.org/ 
for why this is so important. Again, I would have to guess what those 
functions do -- it may be relevant, it may not. If I have something 
that I can *just run* then I can use various introspection tools to see 
what is going wrong.
Mike
On 10/10/2013 10:12 AM, Martin MOKREJŠ wrote:
> Michael Droettboom wrote:
>> Can you provide a complete, standalone example that reproduces the
>> problem. Otherwise all I can do is guess.
>>
>> The usual culprit is forgetting to close figures after you're done with
>> them.
> Thanks, I learned that through matplotlib-1.3.0 give spit over me a warning message some weeks
> ago. Yes, i do call _figure.clear() and pylab.clf() but only after the savefig() returns, which
> is not the case here. Also use gc.collect() a lot through the code, especially before and after
> I draw every figure. That is not enough here.
>
>
>
>
>
> from itertools import izip, imap, ifilter
> import pylab
> import matplotlib
> # Force matplotlib not to use any X-windows backend.
> matplotlib.use('Agg')
> import pylab
>
> F = pylab.gcf()
>
> # convert the view of numpy array to tuple
> # http://matplotlib.1069221.n5.nabble.com/RendererAgg-int-width-int-height-dpi-debug-False-ValueError-width-and-height-must-each-be-below-32768-td27756.html
> DefaultSize = tuple(F.get_size_inches())
>
>
>
> def draw_hist2d_plot(filename, mydata_x, mydata_y, colors, title_data, xlabel_data, ylabel_data, legends, legend_loc='upper right', legend_bbox_to_anchor=(1.0, 1.0), legend_ncol=None, xmin=None, xmax=None, ymin=None, ymax=None, fontsize=10, legend_fontsize=8, dpi=100, tight_layout=False, legend_inside=False, objsize=0.1):
> # hist2d(x, y, bins = None, range=None, weights=None, cmin=None, cmax=None **kwargs)
>
> if len(mydata_x) != len(mydata_y):
> raise ValueError, "%s: len(mydata_x) != len(mydata_y): %s != %s" % (filename, len(mydata_x), len(mydata_y))
>
> if colors and len(mydata_x) != len(colors):
> sys.stderr.write("Warning: draw_hist2d_plot(): %s: len(mydata_x) != len(colors): %s != %s.\n" % (filename, len(mydata_x), len(colors)))
>
> if colors and legends and len(colors) != len(legends):
> sys.stderr.write("Warning: draw_hist2d_plot(): %s, len(colors) != len(legends): %s != %s.\n" % (filename, len(colors), len(legends)))
>
> if mydata_x and mydata_y and filename:
> if legends:
> if not legend_ncol:
> _subfigs, _ax1_num, _ax2_num, _legend_ncol = get_ncol(legends, fontsize=legend_fontsize)
> else:
> _subfigs, _ax1_num, _ax2_num, _legend_ncol = 3, 213, 313, legend_ncol
> else:
> _subfigs, _ax1_num, _legend_ncol = 3, 313, 0
>
> set_my_pylab_defaults()
> pylab.clf()
> _figure = pylab.figure()
> _figure.clear()
> _figure.set_tight_layout(True)
> gc.collect()
>
> if legends:
> # do not crash on too tall figures
> if 8.4 * _subfigs < 200:
> _figure.set_size_inches(11.2, 8.4 * (_subfigs + 1))
> else:
> # _figure.set_size_inches() silently accepts a large value but later on _figure.savefig() crashes with:
> # ValueError: width and height must each be below 32768
> _figure.set_size_inches(11.2, 200)
> sys.stderr.write("Warning: draw_hist2d_plot(): Wanted to set %s figure height to %s but is too high, forcing %s instead. You will likely get an incomplete image.\n" % (filename, 8.4 * _subfigs, 200))
> if myoptions.debug > 5: print "Debug: draw_hist2d_plot(): Changed %s figure size to: %s" % (filename, str(_figure.get_size_inches()))
> _ax1 = _figure.add_subplot(_ax1_num)
> _ax2 = _figure.add_subplot(_ax2_num)
> else:
> _figure.set_size_inches(11.2, 8.4 * 2)
> _ax1 = _figure.gca()
> if myoptions.debug > 5: print "Debug: draw_hist2d_plot(): Changed %s figure size to: %s" % (filename, str(_figure.get_size_inches()))
>
> _series = []
> #for _x, _y, _c, _l in izip(mydata_x, mydata_y, colors, legends):
> for _x, _y, _c in izip(mydata_x, mydata_y, colors):
> # _Line2D = _ax1.plot(_x, _y) # returns Line2D object
> _my_PathCollection = _ax1.scatter(_x, _y, color=_c, s=objsize) # , label=_l) # returns PathCollection object
> _series.append(_my_PathCollection)
>
> if legends:
> #for _x, _y, _c, _l in izip(mydata_x, mydata_y, colors, legends):
> for _x, _y, _c in izip(mydata_x, mydata_y, colors):
> _my_PathCollection = _ax1.scatter(_x, _y, color=_c, s=objsize) # , label=_l)
> _series.append(_my_PathCollection)
>
> _ax2.legend(_series, legends, loc='upper left', bbox_to_anchor=(0,0,1,1), borderaxespad=0., ncol=_legend_ncol, mode='expand', fontsize=legend_fontsize)
> _ax2.set_frame_on(False)
> _ax2.tick_params(bottom='off', left='off', right='off', top='off')
> pylab.setp(_ax2.get_yticklabels(), visible=False)
> pylab.setp(_ax2.get_xticklabels(), visible=False)
> else:
> for _x, _y, _c in izip(mydata_x, mydata_y, colors):
> _ax1.scatter(_x, _y, color=_c, s=objsize) #, marker='^') # keeps eating memory in:
> #
> # draw_hist2d_plot(filename, _data_xrow, _data_yrow, _my_colors, _title, _xlabel, _ylabel, [], xmin=None, xmax=None, ymin=None, ymax=None, fontsize=10, dpi=100)
> # File "/blah.py", line 14080, in draw_hist2d_plot
> # _ax1.scatter(_x, _y, color=_c, s=objsize) #, marker='^')
> # File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line 6247, in scatter
> # self._process_unit_info(xdata=x, ydata=y, kwargs=kwargs)
> # File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line 1685, in _process_unit_info
> # self.xaxis.update_units(xdata)
> # File "/usr/lib64/python2.7/site-packages/matplotlib/axis.py", line 1332, in update_units
> # converter = munits.registry.get_converter(data)
>
> # pylab.subplots_adjust(left = (5/25.4)/_figure.xsize, bottom = (4/25.4)/_figure.ysize, right = 1 - (1/25.4)/_figure.xsize, top = 1 - (3/25.4)/_figure.ysize)
>
> _ax1.set_xlabel(xlabel_data, fontsize=fontsize)
> _ax1.set_ylabel(ylabel_data, fontsize=fontsize)
> _ax1.set_xmargin(0.05)
> _ax1.set_ymargin(0.05)
> _ax1.set_autoscale_on(False)
>
>
> set_limits(_ax1, xmin, xmax, ymin, ymax)
>
> if fontsize == 10:
> _ax1.set_title('\n'.join(wrap(title_data, 100)), fontsize=fontsize+2)
> elif fontsize == 12:
> _ax1.set_title('\n'.join(wrap(title_data, 90)), fontsize=fontsize+2)
> else:
> _ax1.set_title('\n'.join(wrap(title_data, 100)), fontsize=fontsize+2)
>
> if legends:
> _figure.savefig(filename, dpi=100) #, bbox_inches='tight')
> del(_my_PathCollection)
> del(_ax2)
> else:
> _figure.savefig(filename, dpi=100)
>
> del(_series)
> del(_ax1)
> _figure.clear()
> del(_figure)
> pylab.clf()
> pylab.close()
> # pylab.rcdefaults()
>
> gc.collect()
>
>
>
> That's the whole function. I used to suspect _ax1.scatter() in the past but probably
> only because I hit the memory problems earlier. That is worked around now by using
> on disk bsddb3 file or gdbm somewhere upstream. This particular function is nevertheless
> fed with just a huge list numbers, and that is not the issue in itself.
>
> I would be glad if I could tell matplotlib: Here you have 100 colors, use them for all data
> as you wish, just spread them evenly over the whole dataset so that first 1/100th of the data
> gets the first color, second 1/100th of the data gets the second color, and so on. Optionally,
> if you would like to say: use the 100 colors in cycles for all data points, just loop through
> the colors as long as you need some. In both scenarios, I could have avoided the two for loops
> in the above code and necessity to generate those objects. Same for legend stuff.
>
> Martin
>
>> Mike
>>
>> On 10/10/2013 09:05 AM, Martin MOKREJŠ wrote:
>>> Hi,
>>> rendering some of my charts takes almost 50GB of RAM. I believe below is a stracktrace
>>> of one such situation when it already took 15GB. Would somebody comments on what is
>>> matplotlib doing at the very moment? Why the recursion?
>>>
>>> The charts had to have 262422 data points in a 2D scatter plot, each point has assigned
>>> its own color. They are in batches so that there are 153 distinct colors but nevertheless,
>>> I assigned to each data point a color value. There are 153 legend items also (one color
>>> won't be used).
>>>
>>> ^CTraceback (most recent call last):
>>> ...
>>> _figure.savefig(filename, dpi=100)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1421, in savefig
>>> self.canvas.print_figure(*args, **kwargs)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/backend_bases.py", line 2220, in print_figure
>>> **kwargs)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 505, in print_png
>>> FigureCanvasAgg.draw(self)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 451, in draw
>>> self.figure.draw(self.renderer)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>>> draw(artist, renderer, *args, **kwargs)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1034, in draw
>>> func(*args)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>>> draw(artist, renderer, *args, **kwargs)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line 2086, in draw
>>> a.draw(renderer)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>>> draw(artist, renderer, *args, **kwargs)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 718, in draw
>>> return Collection.draw(self, renderer)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>>> draw(artist, renderer, *args, **kwargs)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 276, in draw
>>> offsets, transOffset, self.get_facecolor(), self.get_edgecolor(),
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 551, in get_edgecolor
>>> return self._edgecolors
>>> KeyboardInterrupt
>>> ^CError in atexit._run_exitfuncs:
>>> Traceback (most recent call last):
>>> File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
>>> func(*targs, **kargs)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
>>> gc.collect()
>>> KeyboardInterrupt
>>> Error in sys.exitfunc:
>>> Traceback (most recent call last):
>>> File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
>>> func(*targs, **kargs)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
>>> gc.collect()
>>> KeyboardInterrupt
>>>
>>> ^C
>>>
>>>
>>> Clues what is the code doing? I use mpl-1.3.0.
>>> Thank you,
>>> Martin
>>>
>>> ------------------------------------------------------------------------------
>>> October Webinars: Code for Performance
>>> Free Intel webinars can help you accelerate application performance.
>>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
>>> the latest Intel processors and coprocessors. See abstracts and register >
>>> http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Matplotlib-users mailing list
>>> Mat...@li...
>>> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>>
-- 
 _
|\/|o _|_ _. _ | | \.__ __|__|_|_ _ _ ._ _
| ||(_| |(_|(/_| |_/|(_)(/_|_ |_|_)(_)(_)| | |
http://www.droettboom.com

Michael Droettboom wrote:
> On 10/10/2013 09:47 AM, Martin MOKREJŠ wrote:
>> Benjamin Root wrote:
>>>
>>>
>>> On Thu, Oct 10, 2013 at 9:05 AM, Martin MOKREJŠ <mmo...@gm... <mailto:mmo...@gm...>> wrote:
>>>
>>> Hi,
>>> rendering some of my charts takes almost 50GB of RAM. I believe below is a stracktrace
>>> of one such situation when it already took 15GB. Would somebody comments on what is
>>> matplotlib doing at the very moment? Why the recursion?
>>>
>>> The charts had to have 262422 data points in a 2D scatter plot, each point has assigned
>>> its own color. They are in batches so that there are 153 distinct colors but nevertheless,
>>> I assigned to each data point a color value. There are 153 legend items also (one color
>>> won't be used).
>>>
>>> ^CTraceback (most recent call last):
>>> ...
>>> _figure.savefig(filename, dpi=100)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1421, in savefig
>>> self.canvas.print_figure(*args, **kwargs)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/backend_bases.py", line 2220, in print_figure
>>> **kwargs)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 505, in print_png
>>> FigureCanvasAgg.draw(self)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 451, in draw
>>> self.figure.draw(self.renderer)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>>> draw(artist, renderer, *args, **kwargs)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1034, in draw
>>> func(*args)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>>> draw(artist, renderer, *args, **kwargs)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line 2086, in draw
>>> a.draw(renderer)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>>> draw(artist, renderer, *args, **kwargs)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 718, in draw
>>> return Collection.draw(self, renderer)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>>> draw(artist, renderer, *args, **kwargs)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 276, in draw
>>> offsets, transOffset, self.get_facecolor(), self.get_edgecolor(),
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 551, in get_edgecolor
>>> return self._edgecolors
>>> KeyboardInterrupt
>>> ^CError in atexit._run_exitfuncs:
>>> Traceback (most recent call last):
>>> File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
>>> func(*targs, **kargs)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
>>> gc.collect()
>>> KeyboardInterrupt
>>> Error in sys.exitfunc:
>>> Traceback (most recent call last):
>>> File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
>>> func(*targs, **kargs)
>>> File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
>>> gc.collect()
>>> KeyboardInterrupt
>>>
>>> ^C
>>>
>>>
>>> Clues what is the code doing? I use mpl-1.3.0.
>>> Thank you,
>>> Martin
>>>
>>>
>>> Unfortunately, that stacktrace isn't very useful. There is no recursion there, but rather the perfectly normal drawing of the figure object that has a child axes, which has child collections which have child artist objects.
>>>
>>> Without the accompanying code, it would be difficult to determine where the memory hog is.
>> Could there be places where gc.collect() could be introduced? Are there places where matplotlib
>> could del() unnecessary objects right away? I think the problem is with huge lists or pythonic
>> dicts. I could save 10GB of RAM when I converted one python dict to a bsddb3 file having just
>> 10MB on disk. I speculate matplotlib in that code keeps the data in some huge list or more likely
>> a dict and that is the same issue.
>>
>> Are you sure you cannot see where a problem is? It happens (is visible) only with huge number of
>> dots, of course.
> 
> Matplotlib generally keeps data in Numpy arrays, not lists or 
> dictionaries (though given that matplotlib predates Numpy, there are 
> some corner cases we've found recently where arrays are converted to 
> lists and back unintentionally).
Just a brief note. I don't use Numpy myself in my code, so consider that
while replicating my use case. ;) The code is merely what I think Tony Yu 
of Chao Yue proposed or somebody, sorry, don't remember now, proposed to
me on this list in the past. I am writing it now really from top of my head,
maybe I remember rubbish. ;)
Martin

Benjamin Root wrote:
> 
> On Thu, Oct 10, 2013 at 9:47 AM, Martin MOKREJŠ <mmo...@gm... <mailto:mmo...@gm...>> wrote:
> 
> Benjamin Root wrote:
> >
> >
> >
> > On Thu, Oct 10, 2013 at 9:05 AM, Martin MOKREJŠ <mmo...@gm... <mailto:mmo...@gm...> <mailto:mmo...@gm... <mailto:mmo...@gm...>>> wrote:
> >
> > Hi,
> > rendering some of my charts takes almost 50GB of RAM. I believe below is a stracktrace
> > of one such situation when it already took 15GB. Would somebody comments on what is
> > matplotlib doing at the very moment? Why the recursion?
> >
> > The charts had to have 262422 data points in a 2D scatter plot, each point has assigned
> > its own color. They are in batches so that there are 153 distinct colors but nevertheless,
> > I assigned to each data point a color value. There are 153 legend items also (one color
> > won't be used).
> >
> > ^CTraceback (most recent call last):
> > ...
> > _figure.savefig(filename, dpi=100)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1421, in savefig
> > self.canvas.print_figure(*args, **kwargs)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/backend_bases.py", line 2220, in print_figure
> > **kwargs)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 505, in print_png
> > FigureCanvasAgg.draw(self)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 451, in draw
> > self.figure.draw(self.renderer)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
> > draw(artist, renderer, *args, **kwargs)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1034, in draw
> > func(*args)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
> > draw(artist, renderer, *args, **kwargs)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line 2086, in draw
> > a.draw(renderer)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
> > draw(artist, renderer, *args, **kwargs)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 718, in draw
> > return Collection.draw(self, renderer)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
> > draw(artist, renderer, *args, **kwargs)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 276, in draw
> > offsets, transOffset, self.get_facecolor(), self.get_edgecolor(),
> > File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 551, in get_edgecolor
> > return self._edgecolors
> > KeyboardInterrupt
> > ^CError in atexit._run_exitfuncs:
> > Traceback (most recent call last):
> > File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
> > func(*targs, **kargs)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
> > gc.collect()
> > KeyboardInterrupt
> > Error in sys.exitfunc:
> > Traceback (most recent call last):
> > File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
> > func(*targs, **kargs)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
> > gc.collect()
> > KeyboardInterrupt
> >
> > ^C
> >
> >
> > Clues what is the code doing? I use mpl-1.3.0.
> > Thank you,
> > Martin
> >
> >
> > Unfortunately, that stacktrace isn't very useful. There is no recursion there, but rather the perfectly normal drawing of the figure object that has a child axes, which has child collections which have child artist objects.
> >
> > Without the accompanying code, it would be difficult to determine where the memory hog is.
> 
> Could there be places where gc.collect() could be introduced? Are there places where matplotlib
> could del() unnecessary objects right away? I think the problem is with huge lists or pythonic
> dicts. I could save 10GB of RAM when I converted one python dict to a bsddb3 file having just
> 10MB on disk. I speculate matplotlib in that code keeps the data in some huge list or more likely
> a dict and that is the same issue.
> 
> Are you sure you cannot see where a problem is? It happens (is visible) only with huge number of
> dots, of course.
> 
> 
> I am not going to claim that matplotlib is the most lean graphing library out there, and we already do know where we can make continued improvements, but the symptom you are describing (50 GB for a couple hundred thousand scatter points) is just unheard of for matplotlib. Without a simple, concise, complete code example to demonstrate your problem, we can only hazard guesses. For all I know, you might be "appending" to numpy arrays in a loop prior to plotting, which would eat up significant amount of memory without it being the fault of matplotlib.
> 
> As far as I am aware, we don't do very large dictionaries, so I am doubtful that is the issue either.
> 
> As a side note, I have typically found that situations where del() significantly improved memory usage were typically situations where I was "doing it wrong" in the first place and a simple refactor of the code improved memory and (sometimes) speed, with an added benefit of improved readability. I have even seen situations where calling del() in the wrong places (say, for a list created at the beginning of the loop) actually hurt performance because python couldn't recycle that chunk of memory.
> 
> Give us a code example that reproduces your problem, and then we can start doing some more serious debugging.
Should be in your Inboxes now. I have to rush for a meeting now, so there was no example call
to that function with sample data, but hope I wrote already enough as I knew number of dots and legends
to be drawn. Yeah, the number of columns is determined elsewhere, put 2 as a value into that variable.
Surely one can rewrite the code, but ideally I would also propose that matplotlib is improved so that
others with similarly bad coding style do not hit the issue. ;)
Thank you for your time,
Martin

On 10/10/2013 09:47 AM, Martin MOKREJŠ wrote:
> Benjamin Root wrote:
>>
>>
>> On Thu, Oct 10, 2013 at 9:05 AM, Martin MOKREJŠ <mmo...@gm... <mailto:mmo...@gm...>> wrote:
>>
>> Hi,
>> rendering some of my charts takes almost 50GB of RAM. I believe below is a stracktrace
>> of one such situation when it already took 15GB. Would somebody comments on what is
>> matplotlib doing at the very moment? Why the recursion?
>>
>> The charts had to have 262422 data points in a 2D scatter plot, each point has assigned
>> its own color. They are in batches so that there are 153 distinct colors but nevertheless,
>> I assigned to each data point a color value. There are 153 legend items also (one color
>> won't be used).
>>
>> ^CTraceback (most recent call last):
>> ...
>> _figure.savefig(filename, dpi=100)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1421, in savefig
>> self.canvas.print_figure(*args, **kwargs)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/backend_bases.py", line 2220, in print_figure
>> **kwargs)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 505, in print_png
>> FigureCanvasAgg.draw(self)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 451, in draw
>> self.figure.draw(self.renderer)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>> draw(artist, renderer, *args, **kwargs)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1034, in draw
>> func(*args)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>> draw(artist, renderer, *args, **kwargs)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line 2086, in draw
>> a.draw(renderer)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>> draw(artist, renderer, *args, **kwargs)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 718, in draw
>> return Collection.draw(self, renderer)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>> draw(artist, renderer, *args, **kwargs)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 276, in draw
>> offsets, transOffset, self.get_facecolor(), self.get_edgecolor(),
>> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 551, in get_edgecolor
>> return self._edgecolors
>> KeyboardInterrupt
>> ^CError in atexit._run_exitfuncs:
>> Traceback (most recent call last):
>> File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
>> func(*targs, **kargs)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
>> gc.collect()
>> KeyboardInterrupt
>> Error in sys.exitfunc:
>> Traceback (most recent call last):
>> File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
>> func(*targs, **kargs)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
>> gc.collect()
>> KeyboardInterrupt
>>
>> ^C
>>
>>
>> Clues what is the code doing? I use mpl-1.3.0.
>> Thank you,
>> Martin
>>
>>
>> Unfortunately, that stacktrace isn't very useful. There is no recursion there, but rather the perfectly normal drawing of the figure object that has a child axes, which has child collections which have child artist objects.
>>
>> Without the accompanying code, it would be difficult to determine where the memory hog is.
> Could there be places where gc.collect() could be introduced? Are there places where matplotlib
> could del() unnecessary objects right away? I think the problem is with huge lists or pythonic
> dicts. I could save 10GB of RAM when I converted one python dict to a bsddb3 file having just
> 10MB on disk. I speculate matplotlib in that code keeps the data in some huge list or more likely
> a dict and that is the same issue.
>
> Are you sure you cannot see where a problem is? It happens (is visible) only with huge number of
> dots, of course.
Matplotlib generally keeps data in Numpy arrays, not lists or 
dictionaries (though given that matplotlib predates Numpy, there are 
some corner cases we've found recently where arrays are converted to 
lists and back unintentionally).
As Ben said, the traceback looks quite normal -- and it doesn't show 
what any of the values are. If you can provide us with a script that 
reproduces this, that's the only way we can really plug in and see what 
might be going wrong. It doesn't have to have anything proprietary, 
such as your data. You can even start with one of the existing 
examples, if that helps.
Mike
> _
> |\/|o _|_ _. _ | | \.__ __|__|_|_ _ _ ._ _
> | ||(_| |(_|(/_| |_/|(_)(/_|_ |_|_)(_)(_)| | |
>
> http://www.droettboom.com

Michael Droettboom wrote:
> Can you provide a complete, standalone example that reproduces the 
> problem. Otherwise all I can do is guess.
> 
> The usual culprit is forgetting to close figures after you're done with 
> them.
Thanks, I learned that through matplotlib-1.3.0 give spit over me a warning message some weeks
ago. Yes, i do call _figure.clear() and pylab.clf() but only after the savefig() returns, which
is not the case here. Also use gc.collect() a lot through the code, especially before and after
I draw every figure. That is not enough here.
from itertools import izip, imap, ifilter
import pylab
import matplotlib
# Force matplotlib not to use any X-windows backend.
matplotlib.use('Agg')
import pylab
F = pylab.gcf()
# convert the view of numpy array to tuple
# http://matplotlib.1069221.n5.nabble.com/RendererAgg-int-width-int-height-dpi-debug-False-ValueError-width-and-height-must-each-be-below-32768-td27756.html
DefaultSize = tuple(F.get_size_inches())
def draw_hist2d_plot(filename, mydata_x, mydata_y, colors, title_data, xlabel_data, ylabel_data, legends, legend_loc='upper right', legend_bbox_to_anchor=(1.0, 1.0), legend_ncol=None, xmin=None, xmax=None, ymin=None, ymax=None, fontsize=10, legend_fontsize=8, dpi=100, tight_layout=False, legend_inside=False, objsize=0.1):
 # hist2d(x, y, bins = None, range=None, weights=None, cmin=None, cmax=None **kwargs)
 if len(mydata_x) != len(mydata_y):
 raise ValueError, "%s: len(mydata_x) != len(mydata_y): %s != %s" % (filename, len(mydata_x), len(mydata_y))
 if colors and len(mydata_x) != len(colors):
 sys.stderr.write("Warning: draw_hist2d_plot(): %s: len(mydata_x) != len(colors): %s != %s.\n" % (filename, len(mydata_x), len(colors)))
 if colors and legends and len(colors) != len(legends):
 sys.stderr.write("Warning: draw_hist2d_plot(): %s, len(colors) != len(legends): %s != %s.\n" % (filename, len(colors), len(legends)))
 if mydata_x and mydata_y and filename:
 if legends:
 if not legend_ncol:
 _subfigs, _ax1_num, _ax2_num, _legend_ncol = get_ncol(legends, fontsize=legend_fontsize)
 else:
 _subfigs, _ax1_num, _ax2_num, _legend_ncol = 3, 213, 313, legend_ncol
 else:
 _subfigs, _ax1_num, _legend_ncol = 3, 313, 0
 set_my_pylab_defaults()
 pylab.clf()
 _figure = pylab.figure()
 _figure.clear()
 _figure.set_tight_layout(True)
 gc.collect()
 if legends:
 # do not crash on too tall figures
 if 8.4 * _subfigs < 200:
 _figure.set_size_inches(11.2, 8.4 * (_subfigs + 1))
 else:
 # _figure.set_size_inches() silently accepts a large value but later on _figure.savefig() crashes with:
 # ValueError: width and height must each be below 32768
 _figure.set_size_inches(11.2, 200)
 sys.stderr.write("Warning: draw_hist2d_plot(): Wanted to set %s figure height to %s but is too high, forcing %s instead. You will likely get an incomplete image.\n" % (filename, 8.4 * _subfigs, 200))
 if myoptions.debug > 5: print "Debug: draw_hist2d_plot(): Changed %s figure size to: %s" % (filename, str(_figure.get_size_inches()))
 _ax1 = _figure.add_subplot(_ax1_num)
 _ax2 = _figure.add_subplot(_ax2_num)
 else:
 _figure.set_size_inches(11.2, 8.4 * 2)
 _ax1 = _figure.gca()
 if myoptions.debug > 5: print "Debug: draw_hist2d_plot(): Changed %s figure size to: %s" % (filename, str(_figure.get_size_inches()))
 _series = []
 #for _x, _y, _c, _l in izip(mydata_x, mydata_y, colors, legends):
 for _x, _y, _c in izip(mydata_x, mydata_y, colors):
 # _Line2D = _ax1.plot(_x, _y) # returns Line2D object
 _my_PathCollection = _ax1.scatter(_x, _y, color=_c, s=objsize) # , label=_l) # returns PathCollection object
 _series.append(_my_PathCollection)
 if legends:
 #for _x, _y, _c, _l in izip(mydata_x, mydata_y, colors, legends):
 for _x, _y, _c in izip(mydata_x, mydata_y, colors):
 _my_PathCollection = _ax1.scatter(_x, _y, color=_c, s=objsize) # , label=_l)
 _series.append(_my_PathCollection)
 _ax2.legend(_series, legends, loc='upper left', bbox_to_anchor=(0,0,1,1), borderaxespad=0., ncol=_legend_ncol, mode='expand', fontsize=legend_fontsize)
 _ax2.set_frame_on(False)
 _ax2.tick_params(bottom='off', left='off', right='off', top='off')
 pylab.setp(_ax2.get_yticklabels(), visible=False)
 pylab.setp(_ax2.get_xticklabels(), visible=False)
 else:
 for _x, _y, _c in izip(mydata_x, mydata_y, colors):
 _ax1.scatter(_x, _y, color=_c, s=objsize) #, marker='^') # keeps eating memory in:
 #
 # draw_hist2d_plot(filename, _data_xrow, _data_yrow, _my_colors, _title, _xlabel, _ylabel, [], xmin=None, xmax=None, ymin=None, ymax=None, fontsize=10, dpi=100)
 # File "/blah.py", line 14080, in draw_hist2d_plot
 # _ax1.scatter(_x, _y, color=_c, s=objsize) #, marker='^')
 # File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line 6247, in scatter
 # self._process_unit_info(xdata=x, ydata=y, kwargs=kwargs)
 # File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line 1685, in _process_unit_info
 # self.xaxis.update_units(xdata)
 # File "/usr/lib64/python2.7/site-packages/matplotlib/axis.py", line 1332, in update_units
 # converter = munits.registry.get_converter(data)
 # pylab.subplots_adjust(left = (5/25.4)/_figure.xsize, bottom = (4/25.4)/_figure.ysize, right = 1 - (1/25.4)/_figure.xsize, top = 1 - (3/25.4)/_figure.ysize)
 _ax1.set_xlabel(xlabel_data, fontsize=fontsize)
 _ax1.set_ylabel(ylabel_data, fontsize=fontsize)
 _ax1.set_xmargin(0.05)
 _ax1.set_ymargin(0.05)
 _ax1.set_autoscale_on(False)
 set_limits(_ax1, xmin, xmax, ymin, ymax)
 if fontsize == 10:
 _ax1.set_title('\n'.join(wrap(title_data, 100)), fontsize=fontsize+2)
 elif fontsize == 12:
 _ax1.set_title('\n'.join(wrap(title_data, 90)), fontsize=fontsize+2)
 else:
 _ax1.set_title('\n'.join(wrap(title_data, 100)), fontsize=fontsize+2)
 if legends:
 _figure.savefig(filename, dpi=100) #, bbox_inches='tight')
 del(_my_PathCollection)
 del(_ax2)
 else:
 _figure.savefig(filename, dpi=100)
 del(_series)
 del(_ax1)
 _figure.clear()
 del(_figure)
 pylab.clf()
 pylab.close()
 # pylab.rcdefaults()
 gc.collect()
That's the whole function. I used to suspect _ax1.scatter() in the past but probably
only because I hit the memory problems earlier. That is worked around now by using
on disk bsddb3 file or gdbm somewhere upstream. This particular function is nevertheless
fed with just a huge list numbers, and that is not the issue in itself.
I would be glad if I could tell matplotlib: Here you have 100 colors, use them for all data
as you wish, just spread them evenly over the whole dataset so that first 1/100th of the data
gets the first color, second 1/100th of the data gets the second color, and so on. Optionally,
if you would like to say: use the 100 colors in cycles for all data points, just loop through
the colors as long as you need some. In both scenarios, I could have avoided the two for loops
in the above code and necessity to generate those objects. Same for legend stuff.
Martin
> 
> Mike
> 
> On 10/10/2013 09:05 AM, Martin MOKREJŠ wrote:
>> Hi,
>> rendering some of my charts takes almost 50GB of RAM. I believe below is a stracktrace
>> of one such situation when it already took 15GB. Would somebody comments on what is
>> matplotlib doing at the very moment? Why the recursion?
>>
>> The charts had to have 262422 data points in a 2D scatter plot, each point has assigned
>> its own color. They are in batches so that there are 153 distinct colors but nevertheless,
>> I assigned to each data point a color value. There are 153 legend items also (one color
>> won't be used).
>>
>> ^CTraceback (most recent call last):
>> ...
>> _figure.savefig(filename, dpi=100)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1421, in savefig
>> self.canvas.print_figure(*args, **kwargs)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/backend_bases.py", line 2220, in print_figure
>> **kwargs)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 505, in print_png
>> FigureCanvasAgg.draw(self)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 451, in draw
>> self.figure.draw(self.renderer)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>> draw(artist, renderer, *args, **kwargs)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1034, in draw
>> func(*args)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>> draw(artist, renderer, *args, **kwargs)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line 2086, in draw
>> a.draw(renderer)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>> draw(artist, renderer, *args, **kwargs)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 718, in draw
>> return Collection.draw(self, renderer)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
>> draw(artist, renderer, *args, **kwargs)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 276, in draw
>> offsets, transOffset, self.get_facecolor(), self.get_edgecolor(),
>> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 551, in get_edgecolor
>> return self._edgecolors
>> KeyboardInterrupt
>> ^CError in atexit._run_exitfuncs:
>> Traceback (most recent call last):
>> File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
>> func(*targs, **kargs)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
>> gc.collect()
>> KeyboardInterrupt
>> Error in sys.exitfunc:
>> Traceback (most recent call last):
>> File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
>> func(*targs, **kargs)
>> File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
>> gc.collect()
>> KeyboardInterrupt
>>
>> ^C
>>
>>
>> Clues what is the code doing? I use mpl-1.3.0.
>> Thank you,
>> Martin
>>
>> ------------------------------------------------------------------------------
>> October Webinars: Code for Performance
>> Free Intel webinars can help you accelerate application performance.
>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
>> the latest Intel processors and coprocessors. See abstracts and register >
>> http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Matplotlib-users mailing list
>> Mat...@li...
>> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
> 
> 
-- 
Martin Mokrejs, Ph.D.
Bioinformatics
Donovalska 1658
149 00 Prague
Czech Republic
http://www.iresite.org
http://www.iresite.org/~mmokrejs

On Thu, Oct 10, 2013 at 9:47 AM, Martin MOKREJŠ <mmo...@gm...> wrote:
> Benjamin Root wrote:
> >
> >
> >
> > On Thu, Oct 10, 2013 at 9:05 AM, Martin MOKREJŠ <mmo...@gm...<mailto:
> mmo...@gm...>> wrote:
> >
> > Hi,
> > rendering some of my charts takes almost 50GB of RAM. I believe
> below is a stracktrace
> > of one such situation when it already took 15GB. Would somebody
> comments on what is
> > matplotlib doing at the very moment? Why the recursion?
> >
> > The charts had to have 262422 data points in a 2D scatter plot,
> each point has assigned
> > its own color. They are in batches so that there are 153 distinct
> colors but nevertheless,
> > I assigned to each data point a color value. There are 153 legend
> items also (one color
> > won't be used).
> >
> > ^CTraceback (most recent call last):
> > ...
> > _figure.savefig(filename, dpi=100)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py",
> line 1421, in savefig
> > self.canvas.print_figure(*args, **kwargs)
> > File
> "/usr/lib64/python2.7/site-packages/matplotlib/backend_bases.py", line
> 2220, in print_figure
> > **kwargs)
> > File
> "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py",
> line 505, in print_png
> > FigureCanvasAgg.draw(self)
> > File
> "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py",
> line 451, in draw
> > self.figure.draw(self.renderer)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py",
> line 54, in draw_wrapper
> > draw(artist, renderer, *args, **kwargs)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py",
> line 1034, in draw
> > func(*args)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py",
> line 54, in draw_wrapper
> > draw(artist, renderer, *args, **kwargs)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line
> 2086, in draw
> > a.draw(renderer)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py",
> line 54, in draw_wrapper
> > draw(artist, renderer, *args, **kwargs)
> > File
> "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 718,
> in draw
> > return Collection.draw(self, renderer)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py",
> line 54, in draw_wrapper
> > draw(artist, renderer, *args, **kwargs)
> > File
> "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 276,
> in draw
> > offsets, transOffset, self.get_facecolor(), self.get_edgecolor(),
> > File
> "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 551,
> in get_edgecolor
> > return self._edgecolors
> > KeyboardInterrupt
> > ^CError in atexit._run_exitfuncs:
> > Traceback (most recent call last):
> > File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
> > func(*targs, **kargs)
> > File
> "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90,
> in destroy_all
> > gc.collect()
> > KeyboardInterrupt
> > Error in sys.exitfunc:
> > Traceback (most recent call last):
> > File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
> > func(*targs, **kargs)
> > File
> "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90,
> in destroy_all
> > gc.collect()
> > KeyboardInterrupt
> >
> > ^C
> >
> >
> > Clues what is the code doing? I use mpl-1.3.0.
> > Thank you,
> > Martin
> >
> >
> > Unfortunately, that stacktrace isn't very useful. There is no recursion
> there, but rather the perfectly normal drawing of the figure object that
> has a child axes, which has child collections which have child artist
> objects.
> >
> > Without the accompanying code, it would be difficult to determine where
> the memory hog is.
>
> Could there be places where gc.collect() could be introduced? Are there
> places where matplotlib
> could del() unnecessary objects right away? I think the problem is with
> huge lists or pythonic
> dicts. I could save 10GB of RAM when I converted one python dict to a
> bsddb3 file having just
> 10MB on disk. I speculate matplotlib in that code keeps the data in some
> huge list or more likely
> a dict and that is the same issue.
>
> Are you sure you cannot see where a problem is? It happens (is visible)
> only with huge number of
> dots, of course.
>
>
I am not going to claim that matplotlib is the most lean graphing library
out there, and we already do know where we can make continued improvements,
but the symptom you are describing (50 GB for a couple hundred thousand
scatter points) is just unheard of for matplotlib. Without a simple,
concise, complete code example to demonstrate your problem, we can only
hazard guesses. For all I know, you might be "appending" to numpy arrays in
a loop prior to plotting, which would eat up significant amount of memory
without it being the fault of matplotlib.
As far as I am aware, we don't do very large dictionaries, so I am doubtful
that is the issue either.
As a side note, I have typically found that situations where del()
significantly improved memory usage were typically situations where I was
"doing it wrong" in the first place and a simple refactor of the code
improved memory and (sometimes) speed, with an added benefit of improved
readability. I have even seen situations where calling del() in the wrong
places (say, for a list created at the beginning of the loop) actually hurt
performance because python couldn't recycle that chunk of memory.
Give us a code example that reproduces your problem, and then we can start
doing some more serious debugging.
Ben Root
> Thanks,
> Martin
>

Benjamin Root wrote:
> 
> 
> 
> On Thu, Oct 10, 2013 at 9:05 AM, Martin MOKREJŠ <mmo...@gm... <mailto:mmo...@gm...>> wrote:
> 
> Hi,
> rendering some of my charts takes almost 50GB of RAM. I believe below is a stracktrace
> of one such situation when it already took 15GB. Would somebody comments on what is
> matplotlib doing at the very moment? Why the recursion?
> 
> The charts had to have 262422 data points in a 2D scatter plot, each point has assigned
> its own color. They are in batches so that there are 153 distinct colors but nevertheless,
> I assigned to each data point a color value. There are 153 legend items also (one color
> won't be used).
> 
> ^CTraceback (most recent call last):
> ...
> _figure.savefig(filename, dpi=100)
> File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1421, in savefig
> self.canvas.print_figure(*args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/backend_bases.py", line 2220, in print_figure
> **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 505, in print_png
> FigureCanvasAgg.draw(self)
> File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 451, in draw
> self.figure.draw(self.renderer)
> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
> draw(artist, renderer, *args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1034, in draw
> func(*args)
> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
> draw(artist, renderer, *args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line 2086, in draw
> a.draw(renderer)
> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
> draw(artist, renderer, *args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 718, in draw
> return Collection.draw(self, renderer)
> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
> draw(artist, renderer, *args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 276, in draw
> offsets, transOffset, self.get_facecolor(), self.get_edgecolor(),
> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 551, in get_edgecolor
> return self._edgecolors
> KeyboardInterrupt
> ^CError in atexit._run_exitfuncs:
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
> func(*targs, **kargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
> gc.collect()
> KeyboardInterrupt
> Error in sys.exitfunc:
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
> func(*targs, **kargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
> gc.collect()
> KeyboardInterrupt
> 
> ^C
> 
> 
> Clues what is the code doing? I use mpl-1.3.0.
> Thank you,
> Martin
> 
> 
> Unfortunately, that stacktrace isn't very useful. There is no recursion there, but rather the perfectly normal drawing of the figure object that has a child axes, which has child collections which have child artist objects.
> 
> Without the accompanying code, it would be difficult to determine where the memory hog is.
Could there be places where gc.collect() could be introduced? Are there places where matplotlib
could del() unnecessary objects right away? I think the problem is with huge lists or pythonic
dicts. I could save 10GB of RAM when I converted one python dict to a bsddb3 file having just
10MB on disk. I speculate matplotlib in that code keeps the data in some huge list or more likely
a dict and that is the same issue.
Are you sure you cannot see where a problem is? It happens (is visible) only with huge number of
dots, of course.
Thanks,
Martin

Can you provide a complete, standalone example that reproduces the 
problem. Otherwise all I can do is guess.
The usual culprit is forgetting to close figures after you're done with 
them.
Mike
On 10/10/2013 09:05 AM, Martin MOKREJŠ wrote:
> Hi,
> rendering some of my charts takes almost 50GB of RAM. I believe below is a stracktrace
> of one such situation when it already took 15GB. Would somebody comments on what is
> matplotlib doing at the very moment? Why the recursion?
>
> The charts had to have 262422 data points in a 2D scatter plot, each point has assigned
> its own color. They are in batches so that there are 153 distinct colors but nevertheless,
> I assigned to each data point a color value. There are 153 legend items also (one color
> won't be used).
>
> ^CTraceback (most recent call last):
> ...
> _figure.savefig(filename, dpi=100)
> File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1421, in savefig
> self.canvas.print_figure(*args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/backend_bases.py", line 2220, in print_figure
> **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 505, in print_png
> FigureCanvasAgg.draw(self)
> File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 451, in draw
> self.figure.draw(self.renderer)
> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
> draw(artist, renderer, *args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1034, in draw
> func(*args)
> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
> draw(artist, renderer, *args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line 2086, in draw
> a.draw(renderer)
> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
> draw(artist, renderer, *args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 718, in draw
> return Collection.draw(self, renderer)
> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
> draw(artist, renderer, *args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 276, in draw
> offsets, transOffset, self.get_facecolor(), self.get_edgecolor(),
> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 551, in get_edgecolor
> return self._edgecolors
> KeyboardInterrupt
> ^CError in atexit._run_exitfuncs:
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
> func(*targs, **kargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
> gc.collect()
> KeyboardInterrupt
> Error in sys.exitfunc:
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
> func(*targs, **kargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
> gc.collect()
> KeyboardInterrupt
>
> ^C
>
>
> Clues what is the code doing? I use mpl-1.3.0.
> Thank you,
> Martin
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
-- 
 _
|\/|o _|_ _. _ | | \.__ __|__|_|_ _ _ ._ _
| ||(_| |(_|(/_| |_/|(_)(/_|_ |_|_)(_)(_)| | |
http://www.droettboom.com

On Thu, Oct 10, 2013 at 9:05 AM, Martin MOKREJŠ <mmo...@gm...> wrote:
> Hi,
> rendering some of my charts takes almost 50GB of RAM. I believe below is
> a stracktrace
> of one such situation when it already took 15GB. Would somebody comments
> on what is
> matplotlib doing at the very moment? Why the recursion?
>
> The charts had to have 262422 data points in a 2D scatter plot, each
> point has assigned
> its own color. They are in batches so that there are 153 distinct colors
> but nevertheless,
> I assigned to each data point a color value. There are 153 legend items
> also (one color
> won't be used).
>
> ^CTraceback (most recent call last):
> ...
> _figure.savefig(filename, dpi=100)
> File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line
> 1421, in savefig
> self.canvas.print_figure(*args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/backend_bases.py",
> line 2220, in print_figure
> **kwargs)
> File
> "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py",
> line 505, in print_png
> FigureCanvasAgg.draw(self)
> File
> "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py",
> line 451, in draw
> self.figure.draw(self.renderer)
> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54,
> in draw_wrapper
> draw(artist, renderer, *args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line
> 1034, in draw
> func(*args)
> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54,
> in draw_wrapper
> draw(artist, renderer, *args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line 2086,
> in draw
> a.draw(renderer)
> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54,
> in draw_wrapper
> draw(artist, renderer, *args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py",
> line 718, in draw
> return Collection.draw(self, renderer)
> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54,
> in draw_wrapper
> draw(artist, renderer, *args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py",
> line 276, in draw
> offsets, transOffset, self.get_facecolor(), self.get_edgecolor(),
> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py",
> line 551, in get_edgecolor
> return self._edgecolors
> KeyboardInterrupt
> ^CError in atexit._run_exitfuncs:
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
> func(*targs, **kargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py",
> line 90, in destroy_all
> gc.collect()
> KeyboardInterrupt
> Error in sys.exitfunc:
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
> func(*targs, **kargs)
> File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py",
> line 90, in destroy_all
> gc.collect()
> KeyboardInterrupt
>
> ^C
>
>
> Clues what is the code doing? I use mpl-1.3.0.
> Thank you,
> Martin
>
>
Unfortunately, that stacktrace isn't very useful. There is no recursion
there, but rather the perfectly normal drawing of the figure object that
has a child axes, which has child collections which have child artist
objects.
Without the accompanying code, it would be difficult to determine where the
memory hog is.
Ben Root

Hi,
 rendering some of my charts takes almost 50GB of RAM. I believe below is a stracktrace
of one such situation when it already took 15GB. Would somebody comments on what is
matplotlib doing at the very moment? Why the recursion?
 The charts had to have 262422 data points in a 2D scatter plot, each point has assigned
its own color. They are in batches so that there are 153 distinct colors but nevertheless,
I assigned to each data point a color value. There are 153 legend items also (one color
won't be used).
^CTraceback (most recent call last):
...
 _figure.savefig(filename, dpi=100)
 File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1421, in savefig
 self.canvas.print_figure(*args, **kwargs)
 File "/usr/lib64/python2.7/site-packages/matplotlib/backend_bases.py", line 2220, in print_figure
 **kwargs)
 File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 505, in print_png
 FigureCanvasAgg.draw(self)
 File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 451, in draw
 self.figure.draw(self.renderer)
 File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
 draw(artist, renderer, *args, **kwargs)
 File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line 1034, in draw
 func(*args)
 File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
 draw(artist, renderer, *args, **kwargs)
 File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line 2086, in draw
 a.draw(renderer)
 File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
 draw(artist, renderer, *args, **kwargs)
 File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 718, in draw
 return Collection.draw(self, renderer)
 File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
 draw(artist, renderer, *args, **kwargs)
 File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 276, in draw
 offsets, transOffset, self.get_facecolor(), self.get_edgecolor(),
 File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 551, in get_edgecolor
 return self._edgecolors
KeyboardInterrupt
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
 File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
 func(*targs, **kargs)
 File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
 gc.collect()
KeyboardInterrupt
Error in sys.exitfunc:
Traceback (most recent call last):
 File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
 func(*targs, **kargs)
 File "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, in destroy_all
 gc.collect()
KeyboardInterrupt
^C
Clues what is the code doing? I use mpl-1.3.0.
Thank you,
Martin