matplotlib-devel Mailing List for matplotlib

Thanks. I just saw that maskedarray.putmask was gone and reached for 
the nearest thing. I'll update this to what you suggest.
Mike
Eric Firing wrote:
> md...@us... wrote:
>> Revision: 7016
>> 
>> http://matplotlib.svn.sourceforge.net/matplotlib/?rev=7016&view=rev
>> Author: mdboom
>> Date: 2009年03月31日 15:22:06 +0000 (2009年3月31日)
>>
>
> ...
>
>> Modified: branches/v0_98_5_maint/lib/matplotlib/transforms.py
>> ===================================================================
>> --- branches/v0_98_5_maint/lib/matplotlib/transforms.py 2009年03月31日 
>> 15:13:24 UTC (rev 7015)
>> +++ branches/v0_98_5_maint/lib/matplotlib/transforms.py 2009年03月31日 
>> 15:22:06 UTC (rev 7016)
>> @@ -975,8 +975,7 @@
>> if self._invalid:
>> points = self._transform.transform(self._bbox.get_points())
>> if ma.isMaskedArray(points):
>> - points.putmask(0.0)
>> - points = np.asarray(points)
>> + np.putmask(points, points.mask, 0.0)
>> self._points = points
>> self._invalid = 0
>> return self._points
>
> Mike,
>
> A cleaner version is this:
>
> points = points.filled(0.0)
>
> Or you can replace the conditional and the assignment with the single 
> line:
>
> points = np.ma.filled(points, 0.0)
>
> Example:
>
> In [6]:np.ma.filled([1,2,3], 0.0)
> Out[6]:array([1, 2, 3])
>
> In [7]:np.ma.filled(np.ma.array([1,2,3], mask=[False,True,False]), 0.0)
> Out[7]:array([1, 0, 3])
>
> The version you have actually can fail:
>
> In [10]:zz = np.ma.ones(5)
>
> In [11]:zz
> Out[11]:
> masked_array(data = [ 1. 1. 1. 1. 1.],
> mask = False,
> fill_value = 1e+20)
>
>
> In [12]:np.putmask(zz, zz.mask, 0)
> --------------------------------------------------------------------------- 
>
> ValueError Traceback (most recent call 
> last)
>
> /home/efiring/<ipython console> in <module>()
>
> ValueError: putmask: mask and data must be the same size
>
>
>
> Eric
-- 
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

md...@us... wrote:
> Revision: 7016
> http://matplotlib.svn.sourceforge.net/matplotlib/?rev=7016&view=rev
> Author: mdboom
> Date: 2009年03月31日 15:22:06 +0000 (2009年3月31日)
> 
...
> Modified: branches/v0_98_5_maint/lib/matplotlib/transforms.py
> ===================================================================
> --- branches/v0_98_5_maint/lib/matplotlib/transforms.py	2009年03月31日 15:13:24 UTC (rev 7015)
> +++ branches/v0_98_5_maint/lib/matplotlib/transforms.py	2009年03月31日 15:22:06 UTC (rev 7016)
> @@ -975,8 +975,7 @@
> if self._invalid:
> points = self._transform.transform(self._bbox.get_points())
> if ma.isMaskedArray(points):
> - points.putmask(0.0)
> - points = np.asarray(points)
> + np.putmask(points, points.mask, 0.0)
> self._points = points
> self._invalid = 0
> return self._points
Mike,
A cleaner version is this:
points = points.filled(0.0)
Or you can replace the conditional and the assignment with the single line:
points = np.ma.filled(points, 0.0)
Example:
In [6]:np.ma.filled([1,2,3], 0.0)
Out[6]:array([1, 2, 3])
In [7]:np.ma.filled(np.ma.array([1,2,3], mask=[False,True,False]), 0.0)
Out[7]:array([1, 0, 3])
The version you have actually can fail:
In [10]:zz = np.ma.ones(5)
In [11]:zz
Out[11]:
masked_array(data = [ 1. 1. 1. 1. 1.],
 mask = False,
 fill_value = 1e+20)
In [12]:np.putmask(zz, zz.mask, 0)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/home/efiring/<ipython console> in <module>()
ValueError: putmask: mask and data must be the same size
Eric

http://thread.gmane.org/gmane.comp.python.devel/102706
http://article.gmane.org/gmane.comp.python.devel/102742
The above two posts are worth reading for any among us who are 
interested in an eventual migration to a DVCS.
Eric (happy hg user)

On Mon, Mar 30, 2009 at 2:13 AM, Reinier Heeres <re...@he...> wrote:
>
> That sounds good. I assume you mean adding it to lib/mpl_toolkits in
> the matplotlib source? It would be great to do this soon.
>
> I also tried to create a separate mpl_toolkit installer with
> setuptools, but can't get that to work properly:
>
> In [2]: import mpl_toolkits.mplot3d
> /usr/lib/python2.5/site-packages/mpl_toolkits/__init__.py:2:
> UserWarning: Module mpl_toolkits was already imported from
> /usr/lib/python2.5/site-packages/mpl_toolkits/__init__.pyc, but
> /usr/lib/python2.5/site-packages/mplot3d-0.1-py2.5.egg is being added
> to sys.path
> __import__('pkg_resources').declare_namespace(__name__)
>
> This can probably be fixed, but I would prefer things to live in the
> matplotlib tree, and I'm willing to volunteer to support it.
>
Since this is pure python, my plan was to put it in the mpl source tree in
mpl_toolkits so that it gets included in all mpl installs, rather than
distribute it separately. This might put some additional burden on you
since we still use svn, but we have some notes on developing mpl via git
with a git/svn gateway
http://matplotlib.sourceforge.net/devel/coding_guide.html#id3
Have you done any work on bringing it up to compatibility with svn HEAD?
I'd like at least the basic demos to work with svn HEAD before I do the
import. I think the major missing piece there is the missing TextWithDash.
Thanks,
JDH

Hi,
I think that except for contourf3D everything is functioning fine
again, so I'm thinking about packaging/distributing it. (I will try to
fix contourf3D as well when I can find some time)
2009年3月3日 John Hunter <jd...@gm...>:
>
>
> On Mon, Mar 2, 2009 at 10:13 PM, Jonathan Taylor
> <jon...@ut...> wrote:
>>
>> Hi,
>>
>> I saw that 3D plotting was dropped from matplotlib since the last time
>> I used it. Unfortunately, it is pretty necessary for some of the work
>> I am doing. Thus, I have started the process of refactoring the code
>> to work with recent versions of matplotlib.
>>
>> Right now, it is still in very early stages and is quite flaky but I
>> do have some functionality. In particular, I am able to do a regular
>> 3d plot, a wireframe plot and a scatter plot. If this interests
>> anyone I am making the code available via git. Instructions are
>> available on my website at:
>
> That's great -- a number of people were very disappointed to see the
> functionality removed, even though it was primitive compared to a good 3D
> toolkit. The problem was, we could never find a core developer who was
> interested in taking it under his wing. Once you get this to a satisfactory
> point, I suggest you develop it as an mpl toolkit. That way, it will get
> installed with every mpl distro (the plain vanilla toolkits we ship, the
> complex ones like basemap are distributed separately) but without the
> implicit promise of full support until someone is willing to step up and
> offer to fully support it.
>
> JDH
>
That sounds good. I assume you mean adding it to lib/mpl_toolkits in
the matplotlib source? It would be great to do this soon.
I also tried to create a separate mpl_toolkit installer with
setuptools, but can't get that to work properly:
In [2]: import mpl_toolkits.mplot3d
/usr/lib/python2.5/site-packages/mpl_toolkits/__init__.py:2:
UserWarning: Module mpl_toolkits was already imported from
/usr/lib/python2.5/site-packages/mpl_toolkits/__init__.pyc, but
/usr/lib/python2.5/site-packages/mplot3d-0.1-py2.5.egg is being added
to sys.path
 __import__('pkg_resources').declare_namespace(__name__)
This can probably be fixed, but I would prefer things to live in the
matplotlib tree, and I'm willing to volunteer to support it.
Cheers,
-- 
Reinier Heeres
Waalstraat 17
2515 XK Den Haag
The Netherlands
Tel: +31 6 10852639

On Sun, Mar 29, 2009 at 1:14 AM, Karen Tracey <kmt...@gm...> wrote:
> On Sat, Mar 28, 2009 at 11:45 PM, Eric Firing <ef...@ha...> wrote:
>
>> Karen Tracey wrote:
>>
>> Hmm, the Ubuntu packaging for matplotlib seems to put a copy of
>>> matplotlibrc only in /etc, where, I gather, it will not ever be used by
>>> matplotlib? I guess one is supposed to copy it to one's home directory and
>>> do per-user customization there. For my case, where the code is running
>>> under Apache, I'd guess no matplotlibrc is being found so all defaults are
>>> being used.
>>>
>>
>> Karen,
>>
>> That seems a little odd; matplotlib doesn't look in /etc by default.
>> Although I run ubuntu, I have never used the ubuntu package, so I have not
>> run into this.
>>
>> The default matplotlibrc has been stripped down to a bare minimum:
>> everything but the default backend selection is commented out.
>>
>> You may have already discovered this, but in case you haven't, you can
>> find out where the active matplotlibrc is being found by using
>> matplotlib_fname():
>>
>> In [1]:import matplotlib
>>
>> In [2]:matplotlib.matplotlib_fname()
>>
>> Out[2]:'/usr/local/lib/python2.5/site-packages/matplotlib/mpl-data/matplotlibrc'
>>
>
> Thanks, I hadn't found that. It seems the Ubuntu packaging has changed
> things a bit:
>
> kmt@lbox:~$ python
> Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49)
> [GCC 4.3.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import matplotlib
> >>> matplotlib.__version__
> '0.98.3'
> >>> matplotlib.matplotlib_fname()
> '/etc/matplotlibrc'
> >>> matplotlib.get_data_path()
> '/usr/share/matplotlib/mpl-data'
> >>>
>
> Looking at the source code, I see in
> /usr/lib/python2.5/site-packages/matplotlib/__init__.py (which is actually a
> link to /usr/share/pyshared/matplotlib/__init__.py which I gather is a way
> of having packages that run under multiple Python versions only have one
> copy of the files installed), in matplotlib_fname, the very end is:
>
> path = '/etc' # guaranteed to exist or raise
> fname = os.path.join(path, 'matplotlibrc')
> if not os.path.exists(fname):
> warnings.warn('Could not find matplotlibrc; using defaults')
> return fname
>
> That is, it looks like they have hardcoded '/etc' where the
> lib/matplotlib/__init__.py file in SVN has a call to get_data_path(). Don't
> know why, but apparently if you run the Ubuntu repository version it's
> /etc/matplotlibrc that gets used (assuming nothing is found earlier in the
> search order). They like all config-type files to be under /etc maybe?
>
They are trying to do right by the linux file standard. /etc is really where
the global matplotlibrc belongs.
Darren

On Sat, Mar 28, 2009 at 11:45 PM, Eric Firing <ef...@ha...> wrote:
> Karen Tracey wrote:
>
> Hmm, the Ubuntu packaging for matplotlib seems to put a copy of
>> matplotlibrc only in /etc, where, I gather, it will not ever be used by
>> matplotlib? I guess one is supposed to copy it to one's home directory and
>> do per-user customization there. For my case, where the code is running
>> under Apache, I'd guess no matplotlibrc is being found so all defaults are
>> being used.
>>
>
> Karen,
>
> That seems a little odd; matplotlib doesn't look in /etc by default.
> Although I run ubuntu, I have never used the ubuntu package, so I have not
> run into this.
>
> The default matplotlibrc has been stripped down to a bare minimum:
> everything but the default backend selection is commented out.
>
> You may have already discovered this, but in case you haven't, you can find
> out where the active matplotlibrc is being found by using
> matplotlib_fname():
>
> In [1]:import matplotlib
>
> In [2]:matplotlib.matplotlib_fname()
>
> Out[2]:'/usr/local/lib/python2.5/site-packages/matplotlib/mpl-data/matplotlibrc'
>
Thanks, I hadn't found that. It seems the Ubuntu packaging has changed
things a bit:
kmt@lbox:~$ python
Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib
>>> matplotlib.__version__
'0.98.3'
>>> matplotlib.matplotlib_fname()
'/etc/matplotlibrc'
>>> matplotlib.get_data_path()
'/usr/share/matplotlib/mpl-data'
>>>
Looking at the source code, I see in
/usr/lib/python2.5/site-packages/matplotlib/__init__.py (which is actually a
link to /usr/share/pyshared/matplotlib/__init__.py which I gather is a way
of having packages that run under multiple Python versions only have one
copy of the files installed), in matplotlib_fname, the very end is:
 path = '/etc' # guaranteed to exist or raise
 fname = os.path.join(path, 'matplotlibrc')
 if not os.path.exists(fname):
 warnings.warn('Could not find matplotlibrc; using defaults')
 return fname
That is, it looks like they have hardcoded '/etc' where the
lib/matplotlib/__init__.py file in SVN has a call to get_data_path(). Don't
know why, but apparently if you run the Ubuntu repository version it's
/etc/matplotlibrc that gets used (assuming nothing is found earlier in the
search order). They like all config-type files to be under /etc maybe?
>
Thank you for the test script. I have added it to the "unit" subdirectory
> of matplotlib, after adding a short docstring. JDH may want to modify or
> move it.
>
>
You're welcome. Hope it's useful. I am glad the fix was relatively
simple.
Karen

Karen Tracey wrote:
> Hmm, the Ubuntu packaging for matplotlib seems to put a copy of 
> matplotlibrc only in /etc, where, I gather, it will not ever be used by 
> matplotlib? I guess one is supposed to copy it to one's home directory 
> and do per-user customization there. For my case, where the code is 
> running under Apache, I'd guess no matplotlibrc is being found so all 
> defaults are being used. 
Karen,
That seems a little odd; matplotlib doesn't look in /etc by default. 
Although I run ubuntu, I have never used the ubuntu package, so I have 
not run into this.
The default matplotlibrc has been stripped down to a bare minimum: 
everything but the default backend selection is commented out.
You may have already discovered this, but in case you haven't, you can 
find out where the active matplotlibrc is being found by using 
matplotlib_fname():
In [1]:import matplotlib
In [2]:matplotlib.matplotlib_fname()
Out[2]:'/usr/local/lib/python2.5/site-packages/matplotlib/mpl-data/matplotlibrc'
The docstring explains the search order:
 Return the path to the rc file
 Search order:
 * current working dir
 * environ var MATPLOTLIBRC
 * HOME/.matplotlib/matplotlibrc
 * MATPLOTLIBDATA/matplotlibrc
Thank you for the test script. I have added it to the "unit" 
subdirectory of matplotlib, after adding a short docstring. JDH may 
want to modify or move it.
Eric

On Fri, Mar 27, 2009 at 5:11 PM, John Hunter <jd...@gm...> wrote:
> On Fri, Mar 27, 2009 at 1:23 PM, Karen Tracey <kmt...@gm...> wrote:
>
>> With the above change to make the texd and _fontd caches per-instance, the
>> script has finished successfully 4 times. Seems pretty convincing evidence
>> that it fixes the problem.
>>
>
> OK, I made this change to svn r7008. We probably do not gain *that much*
> by caching across renderers anyhow.
>
>
Cool, thanks!
>
>>
>> Also, make sure you have disabled usetex in matplotlibrc, since the use of
>>> the filesystem for caching the tex datafiles is probably not thread safe.
>>> My guess is that the font cache on the file system is not thread safe
>>> either, but this may only affect the first run of mpl after a clean install.
>>>
>>
>> Not sure what matplotlibrc is? I found a file
>> /usr/share/matplotlib/matplotlib.conf that has usetex set to False. Is that
>> it? If so I guess it's been disabled all along, this is not something I
>> have fiddled with.
>>
>
> matplotlib.conf is an experimental config that is no longer in use (but it
> won't hurt you to have it lying around). See
> http://matplotlib.sourceforge.net/users/customizing.html for details on
> the matplotlibrc file
>
Hmm, the Ubuntu packaging for matplotlib seems to put a copy of matplotlibrc
only in /etc, where, I gather, it will not ever be used by matplotlib? I
guess one is supposed to copy it to one's home directory and do per-user
customization there. For my case, where the code is running under Apache,
I'd guess no matplotlibrc is being found so all defaults are being used.
>
> No need for data files -- just use np.random.rand to make up some random
> data for plotting. That keeps things simple and small, and you can post the
> script to the list, and hopefully we can find a home for it in our unit
> tests.
>
OK, I reduced it to a bare minimum, using random data. I removed stuff that
was in my code like customizing colors, highlighting a particular bar, and
even setting of the title and axis labels since it turns out none of that is
necessary to trip the exception. I'll append below a script that reliably
reproduces the error on my machine. 8 threads at 50 iterations seems sure
to hit it, fewer threads or fewer iterations may not. Hope this is helpful,
Karen
#! /usr/bin/python
import os
import threading
import traceback
import numpy as np
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
from matplotlib.figure import Figure
thread_count = 8
max_iterations = 50
exception_raised = False
def png_thread(tn):
 png_fname = 'out%d.png' % tn
 vals = 100 + 15 * np.random.randn(10000)
 i = 0
 excp = None
 global exception_raised
 while not exception_raised and i < max_iterations:
 i += 1
 png_f = open(png_fname, 'wb')
 try:
 fig = Figure()
 ax = fig.add_subplot(111)
 ax.hist(vals, 50)
 FigureCanvas(fig).print_png(png_f)
 except Exception, excp:
 pass
 png_f.close()
 if excp:
 print 'png_thread %d failed on iteration %d:' % (tn, i)
 print traceback.format_exc(excp)
 exception_raised = True
 else:
 print 'png_thread %d completed iteration %d.' % (tn, i)
 os.unlink(png_fname)
def main(tc):
 threads = []
 for i in range(tc):
 threads.append(threading.Thread(target=png_thread, args=(i+1,)))
 for t in threads:
 t.start()
 for t in threads:
 t.join()
 if not exception_raised:
 msg = 'Success! %d threads completed %d iterations with no
exceptions raised.'
 else:
 msg = 'Failed! Exception raised before %d threads completed %d
iterations.'
 print msg % (tc, max_iterations)
if __name__== "__main__":
 main(thread_count)

The subject says it all. Over the next few days, we will be updating
the conference website with additional information. So if you are
interested, please keep an eye on: http://conference.scipy.org/
Jarrod

On Fri, Mar 27, 2009 at 1:23 PM, Karen Tracey <kmt...@gm...> wrote:
>
> Yes, this change appears to fix the problem. I created a standalone script
> to recreate the problem. The script starts several threads. Each thread
> enters a loop generating a (different) png file. With the original
> matplotlib code, the threads seem unable to complete 50 iterations without
> the exception being raised. (I tried the script at least a dozen times, it
> never finished successfully with 50 iterations. Usually failed before 5,
> but once made it as high as 20, so I bumped it up to 50 and that seemed to
> ensure it would fail before finishing). With the above change to make the
> texd and _fontd caches per-instance, the script has finished successfully 4
> times. Seems pretty convincing evidence that it fixes the problem.
>
OK, I made this change to svn r7008. We probably do not gain *that much*
by caching across renderers anyhow.
>
>
> Also, make sure you have disabled usetex in matplotlibrc, since the use of
>> the filesystem for caching the tex datafiles is probably not thread safe.
>> My guess is that the font cache on the file system is not thread safe
>> either, but this may only affect the first run of mpl after a clean install.
>>
>
> Not sure what matplotlibrc is? I found a file
> /usr/share/matplotlib/matplotlib.conf that has usetex set to False. Is that
> it? If so I guess it's been disabled all along, this is not something I
> have fiddled with.
>
matplotlib.conf is an experimental config that is no longer in use (but it
won't hurt you to have it lying around). See
http://matplotlib.sourceforge.net/users/customizing.html for details on the
matplotlibrc file
>
> Understood, and I did make a standalone script (plus a set of pickle files
> for the data to graph) to recreate it. I can zip them up and send them
> somewhere, or open a ticket and attach it there, or whatever. Let me know
> what would be most convenient/helpful.
>
No need for data files -- just use np.random.rand to make up some random
data for plotting. That keeps things simple and small, and you can post the
script to the list, and hopefully we can find a home for it in our unit
tests.
Thanks,
JDH

Thanks for the responses everyone. More below inline...
On Fri, Mar 27, 2009 at 9:12 AM, John Hunter <jd...@gm...> wrote:
> On Fri, Mar 27, 2009 at 5:54 AM, Darren Dale <dsd...@gm...> wrote:
>
>>
>> It might not be too much trouble to protect RcParams and its data,
>> although I dont know how disruptive it would be to the mpl codebase and to
>> users for rcParams to begin returning copies of things like font lists.
>>
>
>
> In addition to the rc params, I suspect some of the caching we do at the
> module level, eg in agg
>
>
> class RendererAgg(RendererBase):
>
> texd = maxdict(50) # a cache of tex image rasters
> _fontd = maxdict(50)
>
> might be what is hurting here. Karen, could you try editing
> site-packages/matplotlib/backends/backend_agg.py and moving these two lines
> into the __init__method so they are at the instance level rather than the
> class level. Eg,
>
> def __init__(self, width, height, dpi):
> if __debug__: verbose.report('RendererAgg.__init__',
> 'debug-annoying')
> RendererBase.__init__(self)
> self.texd = maxdict(50) # a cache of tex image rasters
> self._fontd = maxdict(50)
>
> and see if that helps.
Yes, this change appears to fix the problem. I created a standalone script
to recreate the problem. The script starts several threads. Each thread
enters a loop generating a (different) png file. With the original
matplotlib code, the threads seem unable to complete 50 iterations without
the exception being raised. (I tried the script at least a dozen times, it
never finished successfully with 50 iterations. Usually failed before 5,
but once made it as high as 20, so I bumped it up to 50 and that seemed to
ensure it would fail before finishing). With the above change to make the
texd and _fontd caches per-instance, the script has finished successfully 4
times. Seems pretty convincing evidence that it fixes the problem.
>
Also, make sure you have disabled usetex in matplotlibrc, since the use of
> the filesystem for caching the tex datafiles is probably not thread safe.
> My guess is that the font cache on the file system is not thread safe
> either, but this may only affect the first run of mpl after a clean install.
>
Not sure what matplotlibrc is? I found a file
/usr/share/matplotlib/matplotlib.conf that has usetex set to False. Is that
it? If so I guess it's been disabled all along, this is not something I
have fiddled with.
> Also, as Eric suggests, a freestanding script which is thread enabled
> (preferably just using mpl and the standard threading library rather than
> django et al) which uses the agg backend that we could use for debugging
> would be very helpful.
>
Understood, and I did make a standalone script (plus a set of pickle files
for the data to graph) to recreate it. I can zip them up and send them
somewhere, or open a ticket and attach it there, or whatever. Let me know
what would be most convenient/helpful.
Thanks!
Karen

On Fri, Mar 27, 2009 at 1:08 PM, Gael Varoquaux <
gae...@no...> wrote:
> Here is a patch to fix a trivial problem I am having using the plot
> directive under recent versions of sphinx (0.6b1).
Thanks! Applies to svn r7006
JDH

Here is a patch to fix a trivial problem I am having using the plot
directive under recent versions of sphinx (0.6b1).
Ga

On Fri, Mar 27, 2009 at 5:54 AM, Darren Dale <dsd...@gm...> wrote:
>
> It might not be too much trouble to protect RcParams and its data, although
> I dont know how disruptive it would be to the mpl codebase and to users for
> rcParams to begin returning copies of things like font lists.
>
In addition to the rc params, I suspect some of the caching we do at the
module level, eg in agg
 class RendererAgg(RendererBase):
 texd = maxdict(50) # a cache of tex image rasters
 _fontd = maxdict(50)
might be what is hurting here. Karen, could you try editing
site-packages/matplotlib/backends/backend_agg.py and moving these two lines
into the __init__method so they are at the instance level rather than the
class level. Eg,
 def __init__(self, width, height, dpi):
 if __debug__: verbose.report('RendererAgg.__init__',
'debug-annoying')
 RendererBase.__init__(self)
 self.texd = maxdict(50) # a cache of tex image rasters
 self._fontd = maxdict(50)
and see if that helps. Also, make sure you have disabled usetex in
matplotlibrc, since the use of the filesystem for caching the tex datafiles
is probably not thread safe. My guess is that the font cache on the file
system is not thread safe either, but this may only affect the first run of
mpl after a clean install.
Also, as Eric suggests, a freestanding script which is thread enabled
(preferably just using mpl and the standard threading library rather than
django et al) which uses the agg backend that we could use for debugging
would be very helpful.
JDH

On Thu, Mar 26, 2009 at 8:10 PM, Eric Firing <ef...@ha...> wrote:
> Karen Tracey wrote:
> > I originally posted this to the user's list but got no response there.
> > As I think there's a bug in matplotlib here, I'm re-trying on the
> > development list. Here's what I sent to -users back on Mar 13:
>
>
>
> Karen,
>
> (I saw your question to the users group, and was hoping someone more
> knowledgeable than myself about threading problems would reply to you.)
>
>
> http://www.mail-archive.com/mat...@li.../msg06152.html
>
> I don't think there has been any progress on this question since it came
> up a year ago in the above thread. Evidently it is something that needs
> more attention. I would say that you have found a bug, and there are
> probably many more. There are global objects outside the pyplot
> interface; maybe they provide opportunities for threads to trip over
> each other. A reasonable goal would be to have matplotlib be thread-safe
> at least when the pure OO interface is used, and possibly with some
> additional restrictions such as "don't mess with rcParams when more than
> one thread might be running".
It might not be too much trouble to protect RcParams and its data, although
I dont know how disruptive it would be to the mpl codebase and to users for
rcParams to begin returning copies of things like font lists.
Darren

Karen Tracey wrote:
> I originally posted this to the user's list but got no response there. 
> As I think there's a bug in matplotlib here, I'm re-trying on the 
> development list. Here's what I sent to -users back on Mar 13:
Karen,
(I saw your question to the users group, and was hoping someone more 
knowledgeable than myself about threading problems would reply to you.)
http://www.mail-archive.com/mat...@li.../msg06152.html
I don't think there has been any progress on this question since it came 
up a year ago in the above thread. Evidently it is something that needs 
more attention. I would say that you have found a bug, and there are 
probably many more. There are global objects outside the pyplot 
interface; maybe they provide opportunities for threads to trip over 
each other. A reasonable goal would be to have matplotlib be thread-safe 
at least when the pure OO interface is used, and possibly with some 
additional restrictions such as "don't mess with rcParams when more than 
one thread might be running". I suspect this will take some time and 
effort to achieve--and I don't know who might be able to put in that 
time and effort. What might help would be a simple but brutal testing 
framework, independent of web servers etc., that would be likely to make 
mpl fail quickly unless it really is thread-safe. Maybe you can provide 
such a test program? And if you are willing to dive into mpl internals 
and come up with solutions for thread problems, that's even better.
For your immediate needs, I suggest that you use only the OO interface 
(which means that you won't import pylab or pyplot, and so won't use the 
figure() function), and/or use a mutex as you describe below. (I don't 
think that dumping "figure" in itself will help with the problem you 
have run into so far; you will still need the mutex. And if the mutex 
locks all your plotting, then you probably won't hit any other mpl 
threading problems.)
If you do keep using the figure() function (or any related aspect of the 
pyplot interface) then be sure to explicitly close each figure. 
Otherwise your process will chew up all available memory.
Eric
> 
> I am using matplotlib's object-oriented API to dynamically generate some 
> graphs served by a web site. The web site is built with Django and I 
> have generally followed the cookbook example I found here: 
> http://www.scipy.org/Cookbook/Matplotlib/Django for serving matplotlib 
> figures under Django. Specifically my code looks like this:
> 
> from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
> from matplotlib.figure import Figure
> 
> def generate_png(request, f, year, cid, pid, ic):
> 
> # ...snipped code that generates the data to graph...
> 
> fig = Figure()
> ax = fig.add_subplot(111)
> ax.set_title(fig_title)
> ax.set_xlabel("Score")
> ax.set_ylabel("Frequency")
> n, bins2, patches = ax.hist(vals, bins, facecolor='blue', 
> edgecolor='blue')
> if x is not None:
> patches[x].set_facecolor('red')
> patches[x].set_edgecolor('red')
> fig.legend((patches[x],), ('%s (%d)' % (cname, cval),), 'lower 
> left')
> canvas = FigureCanvas(fig)
> canvas.print_png(f)
> 
> # ... snip remainder ...
> 
> This works fine, except when I run it under a multi-threaded web server 
> (Apache with mod_wsgi in daemon mode with multi-threaded processes) it 
> sometimes (not always) fails with this traceback:
> 
> File 
> "/home/kmt/django/Django-1.1-alpha-1/django/core/handlers/base.py", line 
> 86, in get_response
> response = callback(request, *callback_args, **callback_kwargs)
> File "/home/kmt/software/web/xword/acpt/views.py", line 321, in get_png
> response = generate_png(request, f, year, cid, pid, ic)
> File "/home/kmt/software/web/xword/acpt/views.py", line 308, in 
> generate_png
> canvas.print_png(f)
> File 
> "/usr/lib/python2.5/site-packages/matplotlib/backends/backend_agg.py", 
> line 305, in print_png
> FigureCanvasAgg.draw(self)
> File 
> "/usr/lib/python2.5/site-packages/matplotlib/backends/backend_agg.py", 
> line 261, in draw
> self.figure.draw(self.renderer)
> File "/usr/lib/python2.5/site-packages/matplotlib/figure.py", line 765, 
> in draw
> legend.draw(renderer)
> File "/usr/lib/python2.5/site-packages/matplotlib/legend.py", line 215, 
> in draw
> t.draw(renderer)
> File "/usr/lib/python2.5/site-packages/matplotlib/text.py", line 329, 
> in draw
> ismath=self.is_math_text(line))
> File 
> "/usr/lib/python2.5/site-packages/matplotlib/backends/backend_agg.py", 
> line 113, in draw_text
> self._renderer.draw_text_image(font.get_image(), int(x), int(y) + 1, 
> angle, gc)
> RuntimeError: You must call .set_text() before .get_image()
> 
> I'm not at all familiar with the internals (truly I'm barely familiar 
> with the public APIs) of matplotlib but it appears from this exception 
> that internally there's a 'font' object being shared between threads 
> here, such that one thread can come in and change the font state 
> resulting in a subsequent error in a different thread that was also in 
> the middle of using that font object? If I protect that block of code 
> above with a thread lock so that only one thread is allowed in at a 
> time, the problem goes away. 
> 
> For reference I'm using the latest matplotlib available in the Ubuntu 
> Intrepid (8.10) repositories, which looks to be 0.98.3. In a brief scan 
> I didn't see anything relevant listed in the "what's new" page for 
> 0.98.4 (and can't find a "what's new in 0.98.5" on the matplotlib web 
> site though that is what is listed as most recent?). Nor can I find 
> anything that looks similar logged as a bug in the tracker.
> 
> Is there something (besides bracketing all access to the matplotlib code 
> with a thread mutex) that I should be doing to make my use of matplotlib 
> thread safe or does it seem like there's a multi-threading bug in 
> matplotlib here? 
> 
> Apologies if this is the wrong list and there is in fact something I 
> ought to be doing in my code (other than using a mutex) to prevent this 
> -- I haven't been able to find anything. My impression from various doc 
> I've read is the object-oriented API is supposed to be thread-safe. Is 
> that true? If so should I file a ticket for this?
> 
> Thanks for any feedback,
> Karen
> 
> 
> ------------------------------------------------------------------------
> 
> ------------------------------------------------------------------------------
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

I originally posted this to the user's list but got no response there. As I
think there's a bug in matplotlib here, I'm re-trying on the development
list. Here's what I sent to -users back on Mar 13:
I am using matplotlib's object-oriented API to dynamically generate some
graphs served by a web site. The web site is built with Django and I have
generally followed the cookbook example I found here:
http://www.scipy.org/Cookbook/Matplotlib/Django for serving matplotlib
figures under Django. Specifically my code looks like this:
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
from matplotlib.figure import Figure
def generate_png(request, f, year, cid, pid, ic):
 # ...snipped code that generates the data to graph...
 fig = Figure()
 ax = fig.add_subplot(111)
 ax.set_title(fig_title)
 ax.set_xlabel("Score")
 ax.set_ylabel("Frequency")
 n, bins2, patches = ax.hist(vals, bins, facecolor='blue',
edgecolor='blue')
 if x is not None:
 patches[x].set_facecolor('red')
 patches[x].set_edgecolor('red')
 fig.legend((patches[x],), ('%s (%d)' % (cname, cval),), 'lower
left')
 canvas = FigureCanvas(fig)
 canvas.print_png(f)
 # ... snip remainder ...
This works fine, except when I run it under a multi-threaded web server
(Apache with mod_wsgi in daemon mode with multi-threaded processes) it
sometimes (not always) fails with this traceback:
 File "/home/kmt/django/Django-1.1-alpha-1/django/core/handlers/base.py",
line 86, in get_response
 response = callback(request, *callback_args, **callback_kwargs)
 File "/home/kmt/software/web/xword/acpt/views.py", line 321, in get_png
 response = generate_png(request, f, year, cid, pid, ic)
 File "/home/kmt/software/web/xword/acpt/views.py", line 308, in
generate_png
 canvas.print_png(f)
 File "/usr/lib/python2.5/site-packages/matplotlib/backends/backend_agg.py",
line 305, in print_png
 FigureCanvasAgg.draw(self)
 File "/usr/lib/python2.5/site-packages/matplotlib/backends/backend_agg.py",
line 261, in draw
 self.figure.draw(self.renderer)
 File "/usr/lib/python2.5/site-packages/matplotlib/figure.py", line 765, in
draw
 legend.draw(renderer)
 File "/usr/lib/python2.5/site-packages/matplotlib/legend.py", line 215, in
draw
 t.draw(renderer)
 File "/usr/lib/python2.5/site-packages/matplotlib/text.py", line 329, in
draw
 ismath=self.is_math_text(line))
 File "/usr/lib/python2.5/site-packages/matplotlib/backends/backend_agg.py",
line 113, in draw_text
 self._renderer.draw_text_image(font.get_image(), int(x), int(y) + 1,
angle, gc)
RuntimeError: You must call .set_text() before .get_image()
I'm not at all familiar with the internals (truly I'm barely familiar with
the public APIs) of matplotlib but it appears from this exception that
internally there's a 'font' object being shared between threads here, such
that one thread can come in and change the font state resulting in a
subsequent error in a different thread that was also in the middle of using
that font object? If I protect that block of code above with a thread lock
so that only one thread is allowed in at a time, the problem goes away.
For reference I'm using the latest matplotlib available in the Ubuntu
Intrepid (8.10) repositories, which looks to be 0.98.3. In a brief scan I
didn't see anything relevant listed in the "what's new" page for 0.98.4 (and
can't find a "what's new in 0.98.5" on the matplotlib web site though that
is what is listed as most recent?). Nor can I find anything that looks
similar logged as a bug in the tracker.
Is there something (besides bracketing all access to the matplotlib code
with a thread mutex) that I should be doing to make my use of matplotlib
thread safe or does it seem like there's a multi-threading bug in matplotlib
here?
Apologies if this is the wrong list and there is in fact something I ought
to be doing in my code (other than using a mutex) to prevent this -- I
haven't been able to find anything. My impression from various doc I've
read is the object-oriented API is supposed to be thread-safe. Is that
true? If so should I file a ticket for this?
Thanks for any feedback,
Karen

All,
I will be out this morning with a sick child today. She is much better off this morning than yesterday and I fully expect to be back in tomorrow.
--James

Hi list,
I packaged some of python modules that I have been using as a mpl
toolkit. They are mostly helper classes to display images more
conveniently. Note that I put some of them in the matplotlib as
examples.
http://dl.getdropbox.com/u/178748/AxesGrid/htdocs/index.html
It consists of pure python modules, so the installation should be
straight forward. But I guess you need the svn version of mpl.
 > python setup.py install
While the code is currently poorly documented, and may not be suitable
for normal users, I'm releasing it in a hope that some others find
this useful. I tried to wrote a short summary of modules included.
Although it is far from complete (and written in poor english), I hope
this gives you some idea what you can do with this toolkit.
http://dl.getdropbox.com/u/178748/AxesGrid/htdocs/users/overview.html
Regards,
-JJ

On Wed, Mar 25, 2009 at 3:02 PM, Eric Firing <ef...@ha...> wrote:
> Darren Dale wrote:
>
>> I am experimenting with numpy masked arrays, and have a question about how
>> imshow handles them:
>>
>> from numpy import ma
>> from pylab import colorbar, imshow, show
>>
>> a=ma.array([[1,2,3],[4,5,6]],mask=[[0,0,1],[0,0,0]], fill_value=0)
>> imshow(a, interpolation='nearest')
>> colorbar()
>> show()
>>
>> With svn matplotlib, the missing value is treated as if identical to the
>> maximum value. I thought imshow would instead respect the masked array's
>>
> I don't see this with my installation from svn.
>
>> fill_value property by calling fix_invalid, and perhaps defaulting to the
>> min() or max() if fill_value is the default 999999. What is the intended
>> behavior?
>>
>
> What I see with your example is a white square for the masked value;
> actually, it is transparent, with alpha = 0. This is the intended default;
> if it is masked, don't paint anything. It is set in Colormap.__init__ and
> can be overridden by Colormap.set_bad().
>
I was using a greyscale colormap that painted the max value white, and I
confused no paint with max value. Personally, I think black would have been
a better default, but no matter. Thank you for the clarification.
>
> There is no intention to use the masked array fill value.
>
>
>> Relatedly, it looks like imshow and other functions like contour are badly
>> confused by NaNs, I thought they were supported?
>>
>
> I suspect we really should run the Z inputs through masked_invalid,
> especially for contour. The performance hit is minimal as a fraction of the
> total time. I will do this for contour. imshow has to be handled more
> carefully, so I don't want to do it in a hurry.
>
Ok.
Darren

Darren Dale wrote:
> I am experimenting with numpy masked arrays, and have a question about 
> how imshow handles them:
> 
> from numpy import ma
> from pylab import colorbar, imshow, show
> 
> a=ma.array([[1,2,3],[4,5,6]],mask=[[0,0,1],[0,0,0]], fill_value=0)
> imshow(a, interpolation='nearest')
> colorbar()
> show()
> 
> With svn matplotlib, the missing value is treated as if identical to the 
> maximum value. I thought imshow would instead respect the masked array's 
I don't see this with my installation from svn.
> fill_value property by calling fix_invalid, and perhaps defaulting to 
> the min() or max() if fill_value is the default 999999. What is the 
> intended behavior?
What I see with your example is a white square for the masked value; 
actually, it is transparent, with alpha = 0. This is the intended 
default; if it is masked, don't paint anything. It is set in 
Colormap.__init__ and can be overridden by Colormap.set_bad().
There is no intention to use the masked array fill value.
> 
> Relatedly, it looks like imshow and other functions like contour are 
> badly confused by NaNs, I thought they were supported?
I suspect we really should run the Z inputs through masked_invalid, 
especially for contour. The performance hit is minimal as a fraction of 
the total time. I will do this for contour. imshow has to be handled 
more carefully, so I don't want to do it in a hurry.
One of the general cleanups needed in mpl is clarity and consistency in 
argument validation. Part of this is a matter of clarity about API 
levels; we don't want to have to do full validation and acceptance of 
all possible input variations at every level.
Eric
> 
> Thanks,
> Darren

I am experimenting with numpy masked arrays, and have a question about how
imshow handles them:
from numpy import ma
from pylab import colorbar, imshow, show
a=ma.array([[1,2,3],[4,5,6]],mask=[[0,0,1],[0,0,0]], fill_value=0)
imshow(a, interpolation='nearest')
colorbar()
show()
With svn matplotlib, the missing value is treated as if identical to the
maximum value. I thought imshow would instead respect the masked array's
fill_value property by calling fix_invalid, and perhaps defaulting to the
min() or max() if fill_value is the default 999999. What is the intended
behavior?
Relatedly, it looks like imshow and other functions like contour are badly
confused by NaNs, I thought they were supported?
Thanks,
Darren

Hi,
Does anyone know if there is a reason that quadmesh objects don't have a
method for setting the geometry of the grid (i.e. the _coordinates
attribute)? If there's not a reason, I'll add one.
Now, this gets to a larger scale matplotlib API question. Should I add this
as a property (either coordinates or verts) which would be pythonic? Or do I
proliferate the use of getter and setter functions which is consistent? Or
do I meet in the middle and use set_* and get_* to implement the property?
Personally, I prefer the first one from purity, but recognize the need for a
consistent API. Looking over the code base right now, it seems pretty
organic, with a variety of all 3 of the approaches I mentioned being taken.
Thoughts?
Ryan
-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
Sent from: Norman Oklahoma United States.

>> I don't quite like my solution but it seems to work.
>> It passes over the figure instance when initializing the
>> MixedRenderer, and let the renderer change the dpi of the figure when
>> changing the backend.
>> I hope some other developer who better understands the dpi thing take
>> a look and come up with a better solution.
>
> I'll try to take a look at this later this afternoon. I agree that
> someone with more knowledge should take a look. The SVG backend seems
> to just ignore the dpi setting and forces 72 dpi across the board,
> which is why I was able to use it as a workaround.
I applied your patch, and it does allow the PDF backend to produced
mixed modes at various DPI.
The SVG backend complains about the change in call signature to
MixedModeRenderer.__init__(). SVG is the only other backend to use
MixedModeRenderer, so that would be an easy fix.
It seems surprising that dpi=72 is part of the renderer assumptions
somewhere along the way.
>From a design perspective, is it appropriate for the renderer to store
a reference to a figure? Many (all?) of the renderers seem entirely
decoupled from the figure.
-Eric
-Eric