matplotlib-devel Mailing List for matplotlib (Page 9)

On Wed, Nov 07, 2007 at 10:34:18AM -1000, Eric Firing wrote:
> 1) If an internal version of traits is to be used, how hard would it be 
> to patch it in such a way that it *could* be used externally?
I think pretty hard, but you would have to ask enthought's dev.
> 2) Does Gael's version already get around the slow initialization 
> problem? (I presume so--it was pkg-resources that was causing the 
> trouble, wasn't it?) 
As far as I can tell, yes. Setuptools is no longer a run time dependency,
just a compile time dependency. It is imported nowwhere in the code base.
> I think this is important.
I think it is _very_ important.
Gaël

On Wednesday 07 November 2007 09:25:51 am John Hunter wrote:
> On Nov 7, 2007 1:52 AM, Gael Varoquaux <gae...@no...> 
wrote:
> > On Tue, Nov 06, 2007 at 09:00:23PM -0500, Darren Dale wrote:
> > > I have not committed my work to svn yet. I wanted to get some feedback
> > > on points 1 and 2 first. Is it acceptable to use traits internally, but
> > > not expose it to the end user? I think the answer is yes, and that this
> > > is even a benefit. If we want to make it an external dependency we can
> > > strip the package out without impacting any user code.
> >
> > I agree that by itself this is a benefit. It might also be interesting to
> > discuss this usecase on the enthought mailing list. If you tell these
> > guys that this is an important usecase, that projects like ipython,
> > matplotlib, would like to use traits as an internal dependency, but would
> > still like to be able to use the benefit of traits when interfacing with
> > other libraries, there might be a solution.
>
> I am not wild on the idea of an "internal dependency". Since this is
> the first step in providing traitified mpl properties, and users will
> presumably want to be able to set event handlers on these properties,
> etc, it seems best to me if the users are exposed to a proper
> installation of enthought traits. Is there a reason not to treat
> traits like we do dateutil and pytz: check at runtime if it is
> installed and if not install it as enthought.traits using Gael's
> tarball? If traits is already installed and the version is too old,
> complain and fail figuring if users have installed it before, they
> have the wherewithall to upgrade.
>
> This gets enthought.traits (preferably with traits.ui included) on
> more platforms. Ideally we would set up our install to play nicely
> with packagers like debian, so they can get dateutil, pytz and
> enthought.traits through their normal dependency handlers. We are
> simply trying to keep the barrier of installation as low as possible
> for the typical mpl user.
I will try this approach tonight.

Darren Dale wrote:
> I have been working on updating the trunk to provide enthought.traits version 
> 2.6b1. backend_driver.py is running without exceptions using the traited 
> config package with the internal traits package.
> 
> Issues:
> 
> 1) there are lots of absolute package imports scattered throughout traits' 
> code. I worked around this by adding a line to matplotlib/__init__.py:
> 
> sys.path.append(os.path.split(__file__)[0])
> 
> This lets matplotlib access enthought.traits without modifying enthoughts code 
> (anymore than Gael had already done by stripping the pkg_resources imports).
> 
> 2) When I tried updating rc_traits.py to import matplotlib.enthought.traits 
> instead of enthought.traits (which isnt on the PYTHONPATH), I discovered a 
> problem:
> 
> enthought.traits.trait_errors.TraitError: The 'parents_items' trait of a 
> ViewElements instance must be a TraitListEvent, but a value of 
> <matplotlib.enthought.traits.trait_handlers.TraitListEvent object at 
> 0x850454c> was specified.
> 
> So traits would be a behind-the-scenes package, for internal mpl use only.
Regarding (1) and (2), and asking out of ignorance:
1) If an internal version of traits is to be used, how hard would it be 
to patch it in such a way that it *could* be used externally?
2) Does Gael's version already get around the slow initialization 
problem? (I presume so--it was pkg-resources that was causing the 
trouble, wasn't it?) I think this is important. Some mpl applications 
involve running simple scripts many times, so startup time matters. I 
would not want to see traits as an external dependency if that brought 
with it the startup lag--apart from all the other questions associated 
with making it an external dependency in any of its present forms.
> 
> 3) We can not include traits-3 without either adding setuptools as an external 
> dependency (which is already true for python-2.3 users) or monkey-patching 
> distutils. traits-3 includes some pyrex code, which standard distutils does 
> not recognize.
Is it a viable alternative to add pyrex as an external dependency? 
Pyrex.Distutils makes it easy to use pyrex modules via otherwise 
standard setup.py scripts. And pyrex itself is pure python, very easy 
to install.
Part of my motivation in bringing this up is that I have found pyrex 
extremely useful for quick speedups and for interfacing to larger chunks 
of C code. (And I have a strong personal bias towards C and pyrex for 
extension code in preference to C++ and any of the mechanisms used to 
provide bindings for C++ code. I find the former vastly more readable, 
understandable, and hence maintainable.) Pyrex (actually its variant, 
cython) is already essential to basemap, and I am hoping for increasing 
basemap integration, as well as judicious use of extension code to make 
the transforms branch lightening-fast.
Eric

On Wednesday 07 November 2007 02:31:47 pm Christopher Barker wrote:
> Darren Dale wrote:
> > Does the way matplotlib handles pytz and datetime present significant
> > complications as well?
>
> not too bad, but pytz and datetime are already pretty easy to install,
> self-contained packages.
>
> > But not all systems have a package manager, and not all package managers
> > have support for traits.
>
> I really don't think there is much choice here. Until (if) traits is
> widely available in an easy-to-install form (binaries for Windows and
> OS-X, "just works" with easy-install, rpms, and debs for Fedora, Ubuntu,
> Debian) MPL is going to have to keep a copy internally, just like it
> does for Agg, etc.
>
> The first step to the easy-to-install form is for:
>
> setup.py build
>
> to "just work" on all platforms that are setup right to build python
> extensions (have the right compiler, etc) -- is that the case yet?
It is the case on the platform I have tested. We will test others soon.

On Wednesday 07 November 2007 02:19:25 pm Eric Firing wrote:
> > But not all systems have a package manager, and not all package managers
> > have support for traits.
>
> Does *anything* or does *anyone* outside of enthought actually support
> traits now? I know Fernando has said he is sold on it for ipython, but
> I did not see it in the latest release.
I'm not sure. There was some talk about debs and rpms, but I dont know if or 
when traits would be supported by the package managers.
> While I greatly appreciate the time and effort you have put into
> developing the first mpl use of traits, Darren, I remain unconvinced
> that now is the time--if at all--to tie mpl irrevocably to traits.
I appreciate the acknowledgement, and your cautious approach toward changing 
mpl. I see some benefit in the new config package, even without traits. If we 
wanted, I think we could modify it so it doesnt depend on traits, but traits 
could be added in the future.
The addition of traits would only happen after the transforms branch has had 
time to settle. At that time, if we feel like we still need convincing that 
this is the right thing to do, we can create a traits branch and see how 
things pan out. By the time we are ready to roll out the final product, maybe 
everybody will be converting to py3k and we'll have a whole new can of worms 
to consider.
Darren

On Nov 7, 2007, at 12:08 PM, Darren Dale wrote:
>> I don't see the check-then-tarball as a feature, but as a significant
>> complication that I have to be aware of when I'm building MatPlotLib.
>
> Does the way matplotlib handles pytz and datetime present significant
> complications as well?
>
>> So for what it's worth, my vote is for limited cleverness in the
>> attempt to resolve dependencies. Let the compile fail.
>
> It follows then that matplotlib should not provide AGG, pyparsing, 
> pytz,
> datetime, subprocess ...
Well, OK, I think you are right: the included things do actually help 
very much.
Sorry.
I think it's a testament effectiveness of the MPL-included things that 
I am not always aware of the bits.
I don't let a package manager mess with installing AGG anymore. I just 
use the MPL-included one for MPL. Because of version drift, mostly.
So never mind: I like the MPL-provisioned things as "internal" 
dependencies, if it works the same as the current MPL-AGG datetime etc.
 - boyd
Boyd Waters
Scientific Programmer
National Radio Astronomy Observatory
Socorro, New Mexico
http://www.aoc.nrao.edu/~bwaters

Darren Dale wrote:
> Does the way matplotlib handles pytz and datetime present significant 
> complications as well?
not too bad, but pytz and datetime are already pretty easy to install, 
self-contained packages.
> But not all systems have a package manager, and not all package managers have 
> support for traits.
I really don't think there is much choice here. Until (if) traits is 
widely available in an easy-to-install form (binaries for Windows and 
OS-X, "just works" with easy-install, rpms, and debs for Fedora, Ubuntu, 
Debian) MPL is going to have to keep a copy internally, just like it 
does for Agg, etc.
The first step to the easy-to-install form is for:
setup.py build
to "just work" on all platforms that are setup right to build python 
extensions (have the right compiler, etc) -- is that the case yet? Once 
it is, it's not hard for various folks to build binaries for various 
platforms.
As for bundling -- if it's installed and working, py2exe and py2app 
should just work -- if they don't, then a little tweaking may be needed, 
which we can make into a recipe for py2app, and a Wiki page for py2exe 
(is there anything like recipes for py2exe?)
-Chris
-- 
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chr...@no...

On Nov 7, 2007, at 12:04 PM, John Hunter wrote:
> On Nov 7, 2007 12:01 PM, Boyd Waters <bw...@nr...> wrote:
>
>> Tell them they need a "real" enthought traits package. Then they can
>> use their package management to make it right.
>
> This is of course assuming their package management has
> enthought.traits, which by and large, it won't.
>
> I am sympathetic to this view, but our recent experience has been that
> getting enthought.traits to install properly even for matplotlib
> developers took a fair amount of work and consultation with the
> enthought dev team.
Ouch, was it really that hard? I don't remember it being that hard, 
but I am almost certainly forgetting something about the traits tweaks 
that were needed to get MatPlotLib built... maybe I didn't upgrade 
MPL after running into MPL build errors with traits. Hmm.
FWIW, here are the MacPorts for "standard" enthought traits that I 
cobbled up :

> But not all systems have a package manager, and not all package managers have 
> support for traits.
Does *anything* or does *anyone* outside of enthought actually support 
traits now? I know Fernando has said he is sold on it for ipython, but 
I did not see it in the latest release.
While I greatly appreciate the time and effort you have put into 
developing the first mpl use of traits, Darren, I remain unconvinced 
that now is the time--if at all--to tie mpl irrevocably to traits.
Eric

On Wednesday 07 November 2007 10:08:01 am william ratcliff wrote:
> And again, if MPL becomes "traitified", how will that effect users
> that need to roll executables for distribution?
We are considering changes that would require testing before we actually ma=
de=20
a release that would effect users.
> On Nov 7, 2007 9:39 AM, Gael Varoquaux <gae...@no...>=20
wrote:
> > On Wed, Nov 07, 2007 at 08:25:51AM -0600, John Hunter wrote:
> > > I am not wild on the idea of an "internal dependency". Since this is
> > > the first step in providing traitified mpl properties, and users will
> > > presumably want to be able to set event handlers on these properties,
> > > etc, it seems best to me if the users are exposed to a proper
> > > installation of enthought traits.
> >
> > I agree with that. This would clearly be a temporary solution waiting f=
or
> > better packaging of traits.
> >
> > > Is there a reason not to treat traits like we do dateutil and pytz:
> > > check at runtime if it is installed and if not install it as
> > > enthought.traits using Gael's tarball?
> >
> > This does not seem a terribly good solution. First of all, I assume you
> > mean install time rather than run-time, as I can't see this happening at
> > run-time for a question of permission. Package management is not such an
> > easy task, as the endeavour of setup_tools has shown. Of course the
> > solution for platforms with proper package managers, is to use this
> > package manager, but when there isn't one, try to provide one, even a
> > simple one, is a risky endeavour. If you are going to try to do this, I
> > would try to do it through setuptools, maybe by including the relevant
> > code from setuptools in MPL.
> >
> > Then there is the problem of compiling. Traits has a bit of compiled
> > code, and not everybody has a compiler installed. So you have to provide
> > binary packages for the major platforms. This also points towards eggs,
> > as eggs are binary packages (eggs are nothing more than a zipped python
> > package), and we are already building binary eggs for traits and Co.
> >
> > > This gets enthought.traits (preferably with traits.ui included) on
> > > more platforms.
> >
> > Yes, the big and important work is to get traits and traitsUI packaged
> > natively on more platforms. Ondrej Certik has started debian packages,
> > but he does not have much time to work on them, and they are progressing
> > slowly. Some people have offered packaging for Fedora, but the problem =
is
> > the same. If you know people who are willing to do some packaging work,
> > we need some help :->.
> >
> > > We are simply trying to keep the barrier of installation as low as
> > > possible for the typical mpl user.
> >
> > This is very important, indeed.
> >
> > Can you tell us a bit more what you plan to use traits for exactly (what
> > are the usecases) ? It might help me understand how to find a beautiful
> > solution to this chicken and eggs problem.
> >
> > Ga=EBl
> >
> >
> > -----------------------------------------------------------------------=
=2D-
> > This SF.net email is sponsored by: Splunk Inc.
> > Still grepping through log files to find problems? Stop.
> > Now Search log events and configuration files using AJAX and a browser.
> > Download your FREE copy of Splunk now >> http://get.splunk.com/
> > _______________________________________________
> > Matplotlib-devel mailing list
> > Mat...@li...
> > https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems? Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
=2D-=20
Darren S. Dale, Ph.D.
Staff Scientist
Cornell High Energy Synchrotron Source
Cornell University
275 Wilson Lab
Rt. 366 & Pine Tree Road
Ithaca, NY 14853
dar...@co...
office: (607) 255-3819
fax: (607) 255-9001
http://www.chess.cornell.edu

On Wednesday 07 November 2007 01:01:52 pm Boyd Waters wrote:
> On Nov 7, 2007, at 7:25 AM, John Hunter wrote:
> > Is there a reason not to treat
> > traits like we do dateutil and pytz: check at runtime if it is
> > installed and if not install it as enthought.traits using Gael's
> > tarball?
>
> I don't see the check-then-tarball as a feature, but as a significant
> complication that I have to be aware of when I'm building MatPlotLib.
Does the way matplotlib handles pytz and datetime present significant 
complications as well?
> So for what it's worth, my vote is for limited cleverness in the
> attempt to resolve dependencies. Let the compile fail.
It follows then that matplotlib should not provide AGG, pyparsing, pytz, 
datetime, subprocess ... We try to find a balance, checking at compile time 
(not runtime) for required dependencies and building what is needed. Who has 
time to spend an afternoon working through a series of compiler errors to 
track down the missing deps?
> Tell them they need a "real" enthought traits package. Then they can
> use their package management to make it right.
But not all systems have a package manager, and not all package managers have 
support for traits.
Darren

On Nov 7, 2007 12:01 PM, Boyd Waters <bw...@nr...> wrote:
> Tell them they need a "real" enthought traits package. Then they can
> use their package management to make it right.
This is of course assuming their package management has
enthought.traits, which by and large, it won't.
I am sympathetic to this view, but our recent experience has been that
getting enthought.traits to install properly even for matplotlib
developers took a fair amount of work and consultation with the
enthought dev team. My concern is that 50% of new and existing users
would simply not install or upgrade under these circumstances. The
difficultly getting scipy installed followed it for years (that is why
mpl never depended on it), and we is in danger of obtaining a similar
"difficult to install so I won't upgrade" reputation.
I would rather inconvenience a sophisticated developer like you
(embedding mpl in an app, following the dev list) primarily because
you can take it and work with it. For a newbie, having the install
fail and being pointed to a setuptools process they don't understand
that may also fail may be the difference between their using
matplotlib and not. If you could set a simple flag or environment
variable to turn off all the optional install "smarts" that should be
enough, no?

On Nov 7, 2007, at 7:25 AM, John Hunter wrote:
> Is there a reason not to treat
> traits like we do dateutil and pytz: check at runtime if it is
> installed and if not install it as enthought.traits using Gael's
> tarball?
I don't see the check-then-tarball as a feature, but as a significant 
complication that I have to be aware of when I'm building MatPlotLib.
I embed MatPlotLib inside a Macintosh application, so there is a 
significant amount of packaging and verification that I need to do to 
ensure a seamless installation for an end-user. I need to know more 
about Python culture and Python packages, I suppose, but for the 
moment I'm trying to keep things as straightforward as possible.
So for what it's worth, my vote is for limited cleverness in the 
attempt to resolve dependencies. Let the compile fail.
Tell them they need a "real" enthought traits package. Then they can 
use their package management to make it right.

And again, if MPL becomes "traitified", how will that effect users
that need to roll executables for distribution?
On Nov 7, 2007 9:39 AM, Gael Varoquaux <gae...@no...> wrot=
e:
> On Wed, Nov 07, 2007 at 08:25:51AM -0600, John Hunter wrote:
> > I am not wild on the idea of an "internal dependency". Since this is
> > the first step in providing traitified mpl properties, and users will
> > presumably want to be able to set event handlers on these properties,
> > etc, it seems best to me if the users are exposed to a proper
> > installation of enthought traits.
>
> I agree with that. This would clearly be a temporary solution waiting for
> better packaging of traits.
>
> > Is there a reason not to treat traits like we do dateutil and pytz:
> > check at runtime if it is installed and if not install it as
> > enthought.traits using Gael's tarball?
>
> This does not seem a terribly good solution. First of all, I assume you
> mean install time rather than run-time, as I can't see this happening at
> run-time for a question of permission. Package management is not such an
> easy task, as the endeavour of setup_tools has shown. Of course the
> solution for platforms with proper package managers, is to use this
> package manager, but when there isn't one, try to provide one, even a
> simple one, is a risky endeavour. If you are going to try to do this, I
> would try to do it through setuptools, maybe by including the relevant
> code from setuptools in MPL.
>
> Then there is the problem of compiling. Traits has a bit of compiled
> code, and not everybody has a compiler installed. So you have to provide
> binary packages for the major platforms. This also points towards eggs,
> as eggs are binary packages (eggs are nothing more than a zipped python
> package), and we are already building binary eggs for traits and Co.
>
> > This gets enthought.traits (preferably with traits.ui included) on
> > more platforms.
>
> Yes, the big and important work is to get traits and traitsUI packaged
> natively on more platforms. Ondrej Certik has started debian packages,
> but he does not have much time to work on them, and they are progressing
> slowly. Some people have offered packaging for Fedora, but the problem is
> the same. If you know people who are willing to do some packaging work,
> we need some help :->.
>
> > We are simply trying to keep the barrier of installation as low as
> > possible for the typical mpl user.
>
> This is very important, indeed.
>
> Can you tell us a bit more what you plan to use traits for exactly (what
> are the usecases) ? It might help me understand how to find a beautiful
> solution to this chicken and eggs problem.
>
> Ga=EBl
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems? Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>

On Wed, Nov 07, 2007 at 08:25:51AM -0600, John Hunter wrote:
> I am not wild on the idea of an "internal dependency". Since this is
> the first step in providing traitified mpl properties, and users will
> presumably want to be able to set event handlers on these properties,
> etc, it seems best to me if the users are exposed to a proper
> installation of enthought traits. 
I agree with that. This would clearly be a temporary solution waiting for
better packaging of traits.
> Is there a reason not to treat traits like we do dateutil and pytz:
> check at runtime if it is installed and if not install it as
> enthought.traits using Gael's tarball? 
This does not seem a terribly good solution. First of all, I assume you
mean install time rather than run-time, as I can't see this happening at
run-time for a question of permission. Package management is not such an
easy task, as the endeavour of setup_tools has shown. Of course the
solution for platforms with proper package managers, is to use this
package manager, but when there isn't one, try to provide one, even a
simple one, is a risky endeavour. If you are going to try to do this, I
would try to do it through setuptools, maybe by including the relevant
code from setuptools in MPL.
Then there is the problem of compiling. Traits has a bit of compiled
code, and not everybody has a compiler installed. So you have to provide
binary packages for the major platforms. This also points towards eggs,
as eggs are binary packages (eggs are nothing more than a zipped python
package), and we are already building binary eggs for traits and Co.
> This gets enthought.traits (preferably with traits.ui included) on
> more platforms. 
Yes, the big and important work is to get traits and traitsUI packaged
natively on more platforms. Ondrej Certik has started debian packages,
but he does not have much time to work on them, and they are progressing
slowly. Some people have offered packaging for Fedora, but the problem is
the same. If you know people who are willing to do some packaging work,
we need some help :->.
> We are simply trying to keep the barrier of installation as low as
> possible for the typical mpl user.
This is very important, indeed.
Can you tell us a bit more what you plan to use traits for exactly (what
are the usecases) ? It might help me understand how to find a beautiful
solution to this chicken and eggs problem.
Gaël

On Nov 7, 2007 1:52 AM, Gael Varoquaux <gae...@no...> wrote:
> On Tue, Nov 06, 2007 at 09:00:23PM -0500, Darren Dale wrote:
> > I have not committed my work to svn yet. I wanted to get some feedback on
> > points 1 and 2 first. Is it acceptable to use traits internally, but not
> > expose it to the end user? I think the answer is yes, and that this is even a
> > benefit. If we want to make it an external dependency we can strip the
> > package out without impacting any user code.
>
> I agree that by itself this is a benefit. It might also be interesting to
> discuss this usecase on the enthought mailing list. If you tell these
> guys that this is an important usecase, that projects like ipython,
> matplotlib, would like to use traits as an internal dependency, but would
> still like to be able to use the benefit of traits when interfacing with
> other libraries, there might be a solution.
I am not wild on the idea of an "internal dependency". Since this is
the first step in providing traitified mpl properties, and users will
presumably want to be able to set event handlers on these properties,
etc, it seems best to me if the users are exposed to a proper
installation of enthought traits. Is there a reason not to treat
traits like we do dateutil and pytz: check at runtime if it is
installed and if not install it as enthought.traits using Gael's
tarball? If traits is already installed and the version is too old,
complain and fail figuring if users have installed it before, they
have the wherewithall to upgrade.
This gets enthought.traits (preferably with traits.ui included) on
more platforms. Ideally we would set up our install to play nicely
with packagers like debian, so they can get dateutil, pytz and
enthought.traits through their normal dependency handlers. We are
simply trying to keep the barrier of installation as low as possible
for the typical mpl user.
JDH
JDH

I have done some preliminary testing with the new changes to 
mathtext. The STIX fonts and Arev Sans fonts (still my favorite) 
both work well and produce pdf files that are readable by standard 
readers (e.g., Preview) and Illustrator. I am using Mac OS X, with 
the TkAgg backend.
Nice work,
-Rob
----
Rob Hetland, Associate Professor
Dept. of Oceanography, Texas A&M University
http://pong.tamu.edu/~rob
phone: 979-458-0096, fax: 979-845-6331

Hi,
I'm sorry to ask for help as a new user by this maillist. I sent a message
to user list, but there is no answer over one week.
The problem is: for my code, If I press 'x' or 'y' while panning, the
motion still don't be constrained to the x or y axis respectively. The
feature is important to my work, but currently i have not yet gotten the
trick.
For your information, NavigationToolbar2GTKAgg is used, and pick event is
defined.
Any help will be appreciated.
-- 
sunzen
<<freedom & enjoyment>>

On Tue, Nov 06, 2007 at 09:00:23PM -0500, Darren Dale wrote:
> I have not committed my work to svn yet. I wanted to get some feedback on 
> points 1 and 2 first. Is it acceptable to use traits internally, but not 
> expose it to the end user? I think the answer is yes, and that this is even a 
> benefit. If we want to make it an external dependency we can strip the 
> package out without impacting any user code.
I agree that by itself this is a benefit. It might also be interesting to
discuss this usecase on the enthought mailing list. If you tell these
guys that this is an important usecase, that projects like ipython,
matplotlib, would like to use traits as an internal dependency, but would
still like to be able to use the benefit of traits when interfacing with
other libraries, there might be a solution.
Gaël

Mike,
On my machine, with pcolor from the trunk:
efiring@manini:~/test$ python pcolortest2.py
init: 2.0
fps: 0.287026406429
And substituting pcolormesh for pcolor:
init: 0.27
fps: 5.48245614035
Now that's more like it!
Using image can be another order of magnitude faster than pcolormesh 
(but with limitations, of course). I suspect nonuniform image code is 
intermediate, but it is a long time since I have tried it.
Eric
Michael Droettboom wrote:
> Attaching benchmark.
> 
> 
> ------------------------------------------------------------------------
> 
> from numpy.random import rand
> import matplotlib
> from matplotlib.pyplot import pcolor, savefig, show, ion, axis, draw, axes
> import time
> 
> ion()
> 
> t = time.clock()
> pcolor(rand(1000,100))
> print "init: ", time.clock() - t
> 
> frames = 25.0
> t = time.clock()
> for i in xrange(int(frames)):
> part = (1.0 - (i / frames) / 2.0)
> axes().set_ylim((0.0, 1000.0 * part))
> draw()
> print "fps:", frames / (time.clock() - t)
> 
> # show()
> 

I have been working on updating the trunk to provide enthought.traits version 
2.6b1. backend_driver.py is running without exceptions using the traited 
config package with the internal traits package.
Issues:
1) there are lots of absolute package imports scattered throughout traits' 
code. I worked around this by adding a line to matplotlib/__init__.py:
sys.path.append(os.path.split(__file__)[0])
This lets matplotlib access enthought.traits without modifying enthoughts code 
(anymore than Gael had already done by stripping the pkg_resources imports).
2) When I tried updating rc_traits.py to import matplotlib.enthought.traits 
instead of enthought.traits (which isnt on the PYTHONPATH), I discovered a 
problem:
enthought.traits.trait_errors.TraitError: The 'parents_items' trait of a 
ViewElements instance must be a TraitListEvent, but a value of 
<matplotlib.enthought.traits.trait_handlers.TraitListEvent object at 
0x850454c> was specified.
So traits would be a behind-the-scenes package, for internal mpl use only.
3) We can not include traits-3 without either adding setuptools as an external 
dependency (which is already true for python-2.3 users) or monkey-patching 
distutils. traits-3 includes some pyrex code, which standard distutils does 
not recognize.
I have not committed my work to svn yet. I wanted to get some feedback on 
points 1 and 2 first. Is it acceptable to use traits internally, but not 
expose it to the end user? I think the answer is yes, and that this is even a 
benefit. If we want to make it an external dependency we can strip the 
package out without impacting any user code.
Darren

Michael Droettboom wrote:
> Eric Firing wrote:
>> Mike,
>>
>> Thanks for the quick response.
>>
>> I was wrong as usual: the masked array overhead in your original 
>> version of the path initializer was actually small. I misinterpreted 
>> the kcachegrind display. Rats! I was hoping for a big gain. It 
>> looks like anything that makes a huge number of paths is going to be 
>> slow, no matter what we do to try to optimize the path initializer.
> 
> Do you think the overhead is just from Python object creation? On the 
> trunk, pcolor ultimately creates a large nested array, which on the 
> branch is then converted to a Python list of Path objects (each 
> containing two arrays). I can definitely imagine how that would be 
> slower...
I think the problem is at least partly the creation of a large number of 
numpy arrays; this is a known performance problem with numpy. It is 
also partly python function call overhead. In my benchmark, that path 
initializer gets called a *lot* of times, and every little bit of 
overhead adds up.
> 
>> A partial solution for pcolor is pcolormesh, using the quadmesh 
>> extension code, although that still has a bug. (Paul Kienzle was going 
>> to look into it.) Is the quadmesh extension compatible with your 
>> transforms branch?
> 
> I'm going to say 'no', only because I wasn't aware of it ;) Where is 
> that code? Trunk contains a "draw_quad_mesh" method in the Agg backend 
> which has been replaced by creating Path objects in Python and sending 
> them to a more generic draw_path_collection method.
That's it--"draw_quad_mesh" in Agg backend.
> 
>> My impression is that the transforms branch is going to be a big step 
>> forward, with performance improvements in some areas, at worst minor 
>> penalties in others--except for some problems like pcolor that need to 
>> be solved.
> 
> I hope you're right. In all honesty, I actually haven't seen many 
> performance improvements myself in what I've benchmarked. Certainly 
> there must be some degenerate synthetic benchmark I can devise to prove 
> its superiority <wink>. Seriously, I was focusing on generalization, 
> ease of adding new transforms, and reduction of interactions with 
> extension code -- all of which aren't certain to create performance 
> gains. I hope that through more pervasive use of numpy, however, some 
> of the performance lost can be made up -- and the reduction in lines of 
> code may make it easier to optimize more broadly.
I think a fair amount of extension code may remain necessary, but your 
reorganization should make it easier to have a clean interface to the 
extension code.
> 
>> In order to replace matlab in my application, a very fast interactive 
>> pcolor-type capability is absolutely essential. I think this simply 
>> has to be done via extension code, like quadmesh and the image codes. 
>> (Pcolor in the trunk isn't fast enough, either.) Unfortunately, I 
>> have found those codes hard to understand. Only the regular-grid 
>> image code is fully integrated into the trunk, and even it has a 
>> long-standing bug revealed by extreme zooming. The irregular-grid 
>> image routine might be a big help, but it has never been integrated. 
>> I don't remember which bugs it shares with quadmesh and image, if any.
> 
> We'll definitely have to find better ways of doing this stuff.
> 
> It would be really nice if all this could be done independently of the 
> backends. The primary reason for having the Path abstraction is so that 
> backends only need to understand one basic thing. Images (as they are 
> now) are an exception to that for the obvious optimization benefits. I 
> wonder if generating the Path objects in an extension would be a 
> worthwhile compromise to adding pcolor support in all the backends (hard 
> to say what the factor of improvement might be).
1) It might indeed be possible to speed up path object generation via 
fairly simple extension code using the numpy C API. We will have to 
look more closely to see where the time is being taken.
2) If we take Agg as the standard core for interactive backends, then it 
is not so crucial to optimize the non-interactive backends, although it 
would still be nice. Alternatively, it may be that the way to go for 
all very complex plots on all backends is to use an image for the 
complex part. Matlab did something like this, with their "zbuffer" 
versus "painters" renderer choice, but they fouled it up--with zbuffer, 
everything was bitmapped, including lines and fonts, so the result 
looked terrible, and the ps files were sometimes too big to print. (I 
know very little about options for images in svg, ps and pdf.) This was 
many years ago; I haven't run into this sort of problem with Matlab more 
recently.
> 
> If by interactive you mean "panning and zooming" performance, than the 
> branch is already at least twice as fast as the trunk (finally a 
> performance benchmark in favor of the branch!). I converted your 
> example into a benchmark (attached) and get these results:
> 
> trunk:
> 
> init: 1.39
> fps: 0.448591422932
> 
> branch:
> 
> init: 5.31
> fps: 1.08885017422
Good, this confirms my subjective impression. But for comparison, try 
the benchmark on the trunk with pcolormesh instead of pcolor.
> 
> Of course, if you mean "interactive" as in "updating the data", then, 
> yes, the branch has a long way to go to catch up to the trunk.
> 
Both aspects--data updating and pan/zoom--are important for interactive use.
Eric
> Cheers,
> Mike
> 
>> Michael Droettboom wrote:
>>> Eric Firing wrote:
>>>> Mike,
>>>>
>>>> I made a quick test and took a quick look, and I certainly see a 
>>>> ripe mango within reach. I don't know what your constraints and 
>>>> strategy are, but I thought I would give you the off-the-cuff idea 
>>>> before I forget what I did.
>>>>
>>>> The test was pcolortest.py, and the kcachegrind input is the .log file.
>>>>
>>>> The problem is the path initializer: it is converting everything to 
>>>> a masked array, which in the vast majority of cases is not needed, 
>>>> and is very costly.
>>>
>>> Thanks for finding this. I agree completely. I think that was 
>>> basically a typo that ended up "working", just suboptimally. The 
>>> input to the path constructor may be either a numpy array, an ma 
>>> array or a regular Python sequence. If it's the first two, it should 
>>> be left alone (if there is an array mask, it is dealt with later on 
>>> in the constructor), but if the latter, it should be converted to a 
>>> numpy array.
>>>
>>> What I meant to type was:
>>>
>>> if not ma.isMaskedArray(vertices):
>>> vertices = npy.asarray(vertices, npy.float_)
>>>
>>> The argument against just "npy.asarray(vertices, npy.float_)" is that 
>>> the mask needs to be preserved.
>>>
>>> If I understand correctly, that will be essentially a no-op when the 
>>> input is a numpy array, albeit with the overhead of some checks.
>>>
>>>> We need to think carefully about the levels of API, and what should 
>>>> be done at which levels. One possibility is that at the level of 
>>>> the path initializer, only ordinary ndarrays should be 
>>>> acceptable--any mask manipulations and compressions should already 
>>>> have been done. This would require a helper function to generate 
>>>> the codes for that case. Another is that the path initializer could 
>>>> get a flag telling it whether to check for masked arrays. And 
>>>> another is that a check for existance of a mask should be done at 
>>>> the start, and the mask processing done only if there is a mask.
>>>
>>> This option was the intent.
>>>
>>>> Yet another is that if a mask is needed, it be passed in as an 
>>>> optional 1-D array kwarg. An advantage of this is that the code 
>>>> that calls the path initializer may be in a better position to know 
>>>> what is needed to generate the 1-D mask (that is, a mask for each 
>>>> (x,y) point rather than for x and y separately)--that mask may 
>>>> already be sitting around.
>>>
>>> Many of these options I fear would significantly complicate the code. 
>>> One of the driving motivations for the refactoring is to allow 
>>> transformations to be combined more generally. Think of the case 
>>> where you have a polar plot with a logarithmic scale on the r-axis 
>>> (this wasn't ever possible in the trunk). The log scale means that 
>>> there is potential for negative masked values, but the polar part of 
>>> the transformation shouldn't have to know or care whether masked 
>>> values are being passed through. Requiring it to do so would need 
>>> the same checks currently performed in the Path constructor, but they 
>>> would be copied all over the code in every kind of new transformation.
>>>
>>> FWIW, there already is a deliberate "quarantining" of masked arrays 
>>> -- it happens where the logical elements of the plot hit the drawing 
>>> commands of the plot (the Path object). It could have been 
>>> implemented such that the backends must understand masked arrays and 
>>> draw accordingly, but it proved to be faster (based on the 
>>> simple_plot_fps.py benchmark) to convert to a non-masked array with 
>>> MOVETO codes upfront and reuse that. (Not surprising, given the 
>>> overhead of masked arrays). This means that masked arrays are not 
>>> used at all during panning and zooming operations where speed is 
>>> perhaps the most crucial.
>>>
>>>> Masked arrays are pretty clunky and slow. The maskedarray 
>>>> implementation by Pierre GM is nicer, more complete, and faster for 
>>>> many operations than numpy.ma, but it still adds a lot of overhead, 
>>>> especially for small arrays. (It needs to have its core in C; so 
>>>> far I have failed dismally in trying to understand how to do that 
>>>> without repeating the bulk of the ndarray code.)
>>>>
>>>> A related point: can you (or is it OK if I do it) change all the 
>>>> "import numpy.ma as ma" or whatever to "from matplotlib.numerix 
>>>> import npyma as ma"? The advantage is that it makes it easy to test 
>>>> the new version with either maskedarray or ma. This should be 
>>>> temporary; I am still hoping and expecting that maskedarray will 
>>>> replace ma in the core numpy distribution.
>>>
>>> That sounds like a very good idea. I'll go ahead and do this (on the 
>>> branch only).
>>>
>>> Cheers,
>>> Mike
>>>
>>
> 

Attaching benchmark.

Eric Firing wrote:
> Mike,
> 
> Thanks for the quick response.
> 
> I was wrong as usual: the masked array overhead in your original version 
> of the path initializer was actually small. I misinterpreted the 
> kcachegrind display. Rats! I was hoping for a big gain. It looks like 
> anything that makes a huge number of paths is going to be slow, no 
> matter what we do to try to optimize the path initializer.
Do you think the overhead is just from Python object creation? On the 
trunk, pcolor ultimately creates a large nested array, which on the 
branch is then converted to a Python list of Path objects (each 
containing two arrays). I can definitely imagine how that would be 
slower...
> A partial solution for pcolor is pcolormesh, using the quadmesh 
> extension code, although that still has a bug. (Paul Kienzle was going 
> to look into it.) Is the quadmesh extension compatible with your 
> transforms branch?
I'm going to say 'no', only because I wasn't aware of it ;) Where is 
that code? Trunk contains a "draw_quad_mesh" method in the Agg backend 
which has been replaced by creating Path objects in Python and sending 
them to a more generic draw_path_collection method.
> My impression is that the transforms branch is going to be a big step 
> forward, with performance improvements in some areas, at worst minor 
> penalties in others--except for some problems like pcolor that need to 
> be solved.
I hope you're right. In all honesty, I actually haven't seen many 
performance improvements myself in what I've benchmarked. Certainly 
there must be some degenerate synthetic benchmark I can devise to prove 
its superiority <wink>. Seriously, I was focusing on generalization, 
ease of adding new transforms, and reduction of interactions with 
extension code -- all of which aren't certain to create performance 
gains. I hope that through more pervasive use of numpy, however, some 
of the performance lost can be made up -- and the reduction in lines of 
code may make it easier to optimize more broadly.
> In order to replace matlab in my application, a very fast interactive 
> pcolor-type capability is absolutely essential. I think this simply has 
> to be done via extension code, like quadmesh and the image codes. 
> (Pcolor in the trunk isn't fast enough, either.) Unfortunately, I have 
> found those codes hard to understand. Only the regular-grid image code 
> is fully integrated into the trunk, and even it has a long-standing bug 
> revealed by extreme zooming. The irregular-grid image routine might be 
> a big help, but it has never been integrated. I don't remember which 
> bugs it shares with quadmesh and image, if any.
We'll definitely have to find better ways of doing this stuff.
It would be really nice if all this could be done independently of the 
backends. The primary reason for having the Path abstraction is so that 
backends only need to understand one basic thing. Images (as they are 
now) are an exception to that for the obvious optimization benefits. I 
wonder if generating the Path objects in an extension would be a 
worthwhile compromise to adding pcolor support in all the backends (hard 
to say what the factor of improvement might be).
If by interactive you mean "panning and zooming" performance, than the 
branch is already at least twice as fast as the trunk (finally a 
performance benchmark in favor of the branch!). I converted your 
example into a benchmark (attached) and get these results:
trunk:
init: 1.39
fps: 0.448591422932
branch:
init: 5.31
fps: 1.08885017422
Of course, if you mean "interactive" as in "updating the data", then, 
yes, the branch has a long way to go to catch up to the trunk.
Cheers,
Mike
> Michael Droettboom wrote:
>> Eric Firing wrote:
>>> Mike,
>>>
>>> I made a quick test and took a quick look, and I certainly see a ripe 
>>> mango within reach. I don't know what your constraints and strategy 
>>> are, but I thought I would give you the off-the-cuff idea before I 
>>> forget what I did.
>>>
>>> The test was pcolortest.py, and the kcachegrind input is the .log file.
>>>
>>> The problem is the path initializer: it is converting everything to a 
>>> masked array, which in the vast majority of cases is not needed, and 
>>> is very costly.
>>
>> Thanks for finding this. I agree completely. I think that was 
>> basically a typo that ended up "working", just suboptimally. The 
>> input to the path constructor may be either a numpy array, an ma array 
>> or a regular Python sequence. If it's the first two, it should be 
>> left alone (if there is an array mask, it is dealt with later on in 
>> the constructor), but if the latter, it should be converted to a numpy 
>> array.
>>
>> What I meant to type was:
>>
>> if not ma.isMaskedArray(vertices):
>> vertices = npy.asarray(vertices, npy.float_)
>>
>> The argument against just "npy.asarray(vertices, npy.float_)" is that 
>> the mask needs to be preserved.
>>
>> If I understand correctly, that will be essentially a no-op when the 
>> input is a numpy array, albeit with the overhead of some checks.
>>
>>> We need to think carefully about the levels of API, and what should 
>>> be done at which levels. One possibility is that at the level of the 
>>> path initializer, only ordinary ndarrays should be acceptable--any 
>>> mask manipulations and compressions should already have been done. 
>>> This would require a helper function to generate the codes for that 
>>> case. Another is that the path initializer could get a flag telling 
>>> it whether to check for masked arrays. And another is that a check 
>>> for existance of a mask should be done at the start, and the mask 
>>> processing done only if there is a mask.
>>
>> This option was the intent.
>>
>>> Yet another is that if a mask is needed, it be passed in as an 
>>> optional 1-D array kwarg. An advantage of this is that the code that 
>>> calls the path initializer may be in a better position to know what 
>>> is needed to generate the 1-D mask (that is, a mask for each (x,y) 
>>> point rather than for x and y separately)--that mask may already be 
>>> sitting around.
>>
>> Many of these options I fear would significantly complicate the code. 
>> One of the driving motivations for the refactoring is to allow 
>> transformations to be combined more generally. Think of the case 
>> where you have a polar plot with a logarithmic scale on the r-axis 
>> (this wasn't ever possible in the trunk). The log scale means that 
>> there is potential for negative masked values, but the polar part of 
>> the transformation shouldn't have to know or care whether masked 
>> values are being passed through. Requiring it to do so would need the 
>> same checks currently performed in the Path constructor, but they 
>> would be copied all over the code in every kind of new transformation.
>>
>> FWIW, there already is a deliberate "quarantining" of masked arrays -- 
>> it happens where the logical elements of the plot hit the drawing 
>> commands of the plot (the Path object). It could have been 
>> implemented such that the backends must understand masked arrays and 
>> draw accordingly, but it proved to be faster (based on the 
>> simple_plot_fps.py benchmark) to convert to a non-masked array with 
>> MOVETO codes upfront and reuse that. (Not surprising, given the 
>> overhead of masked arrays). This means that masked arrays are not 
>> used at all during panning and zooming operations where speed is 
>> perhaps the most crucial.
>>
>>> Masked arrays are pretty clunky and slow. The maskedarray 
>>> implementation by Pierre GM is nicer, more complete, and faster for 
>>> many operations than numpy.ma, but it still adds a lot of overhead, 
>>> especially for small arrays. (It needs to have its core in C; so far 
>>> I have failed dismally in trying to understand how to do that without 
>>> repeating the bulk of the ndarray code.)
>>>
>>> A related point: can you (or is it OK if I do it) change all the 
>>> "import numpy.ma as ma" or whatever to "from matplotlib.numerix 
>>> import npyma as ma"? The advantage is that it makes it easy to test 
>>> the new version with either maskedarray or ma. This should be 
>>> temporary; I am still hoping and expecting that maskedarray will 
>>> replace ma in the core numpy distribution.
>>
>> That sounds like a very good idea. I'll go ahead and do this (on the 
>> branch only).
>>
>> Cheers,
>> Mike
>>
> 
-- 
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

On Nov 6, 2007 2:09 PM, Michael Droettboom <md...@st...> wrote:
> No reason, other than the usual "hasn't been tested as much", and how
> that may affect the upcoming release. I anticipate more mis-mapped
> glyphs (I found some already, but I'm sure not all of them). But it
> won't get tested much unless people are nudged into using it ;) I'd
Release early, release often. We can always roll out a bug fix
release, if Charlie is willing....
Charlie, what say you to an MPL 0.92.0 release for Monday or Tuesday
next week? I will work on the release notes and updates to the web
page in the meantime...
JDH