matplotlib-devel Mailing List for matplotlib

On Fri, Nov 30, 2012 at 11:40 PM, Michiel de Hoon <mjl...@ya...> wrote:
> One package (Pysam) that I use a lot relies on Cython, and requires users to install Cython before they can install Pysam itself. With Cython, is that always the case? Will all users need to install Cython? Or is it sufficient if only matplotlib developers install Cython?
You can set things up so that end-users don't have to install cython.
You just convert the .pyx files to regular .c files before
distributing your package. Numpy itself uses cython, but end-users
don't notice or care. (It's something more of a hassle for developers
to do things this way, and cython is very easy to install, so I don't
know if it's worth it. But it's certainly possible.)
-n

One package (Pysam) that I use a lot relies on Cython, and requires users to install Cython before they can install Pysam itself. With Cython, is that always the case? Will all users need to install Cython? Or is it sufficient if only matplotlib developers install Cython?
Best,
-Michiel.
--- On Fri, 11/30/12, Chris Barker - NOAA Federal <chr...@no...> wrote:
> From: Chris Barker - NOAA Federal <chr...@no...>
> Subject: Re: [matplotlib-devel] Experiments in removing/replacing PyCXX
> To: "Michael Droettboom" <md...@st...>
> Cc: "Michiel de Hoon" <mjl...@ya...>, "mat...@li..." <mat...@li...>
> Date: Friday, November 30, 2012, 12:32 PM
> On Fri, Nov 30, 2012 at 6:06 AM,
> Michael Droettboom <md...@st...>
> wrote:
> 
> > If you read between the lines of what I was saying,
> that is basically
> > where I fall as well. There seems to be a lot of
> desire to use Cython
> > to make the code more accessible,
> 
> I'll add a beat to that drum -- I'm a big Cython fan.
> 
> > however, and I'm willing to consider
> > it if it can be shown to be superior to the raw C/API
> for this task --
> 
> I think there is NO QUESTION that Cython is superior to the
> C/API --
> why would you want to deal with the reference counting, etc
> yourself?
> Cython can handle the boiler plate code for you very cleanly
> an
> elegantly.
> 
> Something to keep in mind about Cython:
> 
> It can be used in multiple ways:
> 
> 1) Add static typing to what is essentially Python code to
> get better
> performance -- this may be what you mean by the "more
> accesible" part.
> A great use, but maybe, maybe, maybe not best for the core
> bits of
> MPL.
> 
> 2) Calling C/C++ code -- Cython is s great way to call C/C++
> code --
> it can handle the packing and unpacking of python types,
> reference
> counting, etc. for you, so much like using the C API, but a
> lot less
> tricky boiler plate code to write.
> 
> (2) is the use case that I'm arguing is NO QUESTION a better
> option
> than the C API.
> 
> Compared to SWIG, SIP (and I assume C_XX), the downside is
> that there
> is no auto-generation of wrappers (at least nothing mature).
> However,
> for the MPL case, we're not trying to wrap a large existing
> library,
> but rather particular code that is often written for MPL
> specifically,
> so hand-writing the Cython is a fine option.
> 
> So why not Ctypes, or??? I think the real strength of Cython
> in
> wrapping C code is that you can write a "thick" wrapper in
> an
> almost_python language. So if you want to vectorize a C
> function, for
> instance, you can write that bit in Cython very easily (and
> Cython's
> built-in understanding of numpy array is very helpful here).
> When you
> use ctypes, you need to write that in pure Python -- easy
> enough, but
> probably not very performant.
> 
> With SWIG, etc, you end up writing a fair bi tof C (or SWIG)
> code to
> handle the thicker bits of the wrapper -- so you're dealing
> with the
> raw CPython API, and , well, C. Cython really is an easier
> option.
> 
> I've found that for stuf that is less than very small (i.e.
> one or two
> loops through an array), writing the core code in native C
> or C++ can
> be easier, you know for sure you're not accidentally making
> expensive
> Python calls, etc. but using Cython to call it is still very
> helpful.
> 
> > I'm not sure it is -- I always seem to end up with
> things that are more
> > lines of code with more obscure workarounds than just
> coding in C directly.
> 
> Exactly -- but I don't think that applies to the CPython-API
> bits, but
> rather the core code -- so keep that in C.
> 
> In summary, I guess what I think is the power of Cython is
> the
> flexibility in where you draw the line between Python,
> Cython, and C
> -- you can pass pure Python through Cython, or you can do
> almost
> nothing with it but call a C function, and eveything in
> between.
> 
> > From my experience, I would prefer to write such
> extensions in C directly rather
> > than relying on Cython, SWIG, or Boost.Python, because
> those approaches would
> > lead to another dependency (for developers at least),
> 
> The dependency is pretty easy to deal with compared to the
> many others in MPL.
> 
> > and requires developers to
> > learn how to code in them. Which may not be very hard,
> but we may as well avoid > that if possible.
> 
> Here's where I disagree -- if we go pure C and C-API
> developers need
> to know the Python C-API -- that is actually a pretty big
> deal, and
> hard to get right. Knowing enough Cython to call some C code
> is a
> smaller lift for sure.
> 
> Anyway, I saw give it a shot -- I suspect you'll like it.
> 
> -Chris
> 
> 
> -- 
> 
> Christopher Barker, Ph.D.
> Oceanographer
> 
> Emergency Response Division
> NOAA/NOS/OR&R      
> (206) 526-6959  voice
> 7600 Sand Point Way NE  (206)
> 526-6329  fax
> Seattle, WA 98115    (206)
> 526-6317  main reception
> 
> Chr...@no...
> 

Let's try that again...
---------- Forwarded message ----------
From: Nathaniel Smith <nj...@po...>
Date: Fri, Nov 30, 2012 at 11:13 PM
Subject: Re: [matplotlib-devel] Travis numpy build failures on Python 3.x
To: Damon McDougall <dam...@gm...>
On Fri, Nov 30, 2012 at 10:25 PM, Damon McDougall
<dam...@gm...> wrote:
> We seem to have inherited these recently. I am questioning whether it
> is something caused by us or not. Can anybody build numpy/mpl under
> Python 3.x on their own machine successfully?
Not your bug.
Workaround:
 https://github.com/travis-ci/travis-cookbooks/issues/48#issuecomment-10843018
Also for context:
 https://github.com/numpy/numpy/issues/2761
 https://github.com/pypa/virtualenv/issues/359
-n

Forwarding to list again...
---------- Forwarded message ----------
From: Nathaniel Smith <nj...@po...>
Date: Fri, Nov 30, 2012 at 5:13 PM
Subject: Re: [matplotlib-devel] Travis numpy build failures on Python 3.x
To: Damon McDougall <dam...@gm...>
On Fri, Nov 30, 2012 at 10:25 PM, Damon McDougall
<dam...@gm...> wrote:
> We seem to have inherited these recently. I am questioning whether it
> is something caused by us or not. Can anybody build numpy/mpl under
> Python 3.x on their own machine successfully?
Not your bug.
Workaround:
 https://github.com/travis-ci/travis-cookbooks/issues/48#issuecomment-10843018
Also for context:
 https://github.com/numpy/numpy/issues/2761
 https://github.com/pypa/virtualenv/issues/359
-n
-- 
Damon McDougall
http://www.damon-is-a-geek.com
Institute for Computational Engineering Sciences
201 E. 24th St.
Stop C0200
The University of Texas at Austin
Austin, TX 78712-1229

Forwarding to the list...
---------- Forwarded message ----------
From: Thomas Kluyver <th...@kl...>
Date: Fri, Nov 30, 2012 at 4:35 PM
Subject: Re: [matplotlib-devel] Travis numpy build failures on Python 3.x
To: Damon McDougall <dam...@gm...>
On 30 November 2012 22:25, Damon McDougall <dam...@gm...> wrote:
>
> We seem to have inherited these recently. I am questioning whether it
> is something caused by us or not. Can anybody build numpy/mpl under
> Python 3.x on their own machine successfully?
The daily PPA builds are working fine:
https://code.launchpad.net/~takluyver/+recipe/matplotlib-daily
That uses the packaged version of numpy, though, rather than trying to
pip install it.
Thomas
-- 
Damon McDougall
http://www.damon-is-a-geek.com
Institute for Computational Engineering Sciences
201 E. 24th St.
Stop C0200
The University of Texas at Austin
Austin, TX 78712-1229

On Fri, Nov 30, 2012 at 4:25 PM, Damon McDougall
<dam...@gm...> wrote:
> We seem to have inherited these recently. I am questioning whether it
> is something caused by us or not. Can anybody build numpy/mpl under
> Python 3.x on their own machine successfully?
Looks like Jens found the problem: https://github.com/numpy/numpy/issues/2761
>
> --
> Damon McDougall
> http://www.damon-is-a-geek.com
> Institute for Computational Engineering Sciences
> 201 E. 24th St.
> Stop C0200
> The University of Texas at Austin
> Austin, TX 78712-1229
-- 
Damon McDougall
http://www.damon-is-a-geek.com
Institute for Computational Engineering Sciences
201 E. 24th St.
Stop C0200
The University of Texas at Austin
Austin, TX 78712-1229

We seem to have inherited these recently. I am questioning whether it
is something caused by us or not. Can anybody build numpy/mpl under
Python 3.x on their own machine successfully?
-- 
Damon McDougall
http://www.damon-is-a-geek.com
Institute for Computational Engineering Sciences
201 E. 24th St.
Stop C0200
The University of Texas at Austin
Austin, TX 78712-1229

On Fri, Nov 30, 2012 at 6:06 AM, Michael Droettboom <md...@st...> wrote:
> If you read between the lines of what I was saying, that is basically
> where I fall as well. There seems to be a lot of desire to use Cython
> to make the code more accessible,
I'll add a beat to that drum -- I'm a big Cython fan.
> however, and I'm willing to consider
> it if it can be shown to be superior to the raw C/API for this task --
I think there is NO QUESTION that Cython is superior to the C/API --
why would you want to deal with the reference counting, etc yourself?
Cython can handle the boiler plate code for you very cleanly an
elegantly.
Something to keep in mind about Cython:
It can be used in multiple ways:
1) Add static typing to what is essentially Python code to get better
performance -- this may be what you mean by the "more accesible" part.
A great use, but maybe, maybe, maybe not best for the core bits of
MPL.
2) Calling C/C++ code -- Cython is s great way to call C/C++ code --
it can handle the packing and unpacking of python types, reference
counting, etc. for you, so much like using the C API, but a lot less
tricky boiler plate code to write.
(2) is the use case that I'm arguing is NO QUESTION a better option
than the C API.
Compared to SWIG, SIP (and I assume C_XX), the downside is that there
is no auto-generation of wrappers (at least nothing mature). However,
for the MPL case, we're not trying to wrap a large existing library,
but rather particular code that is often written for MPL specifically,
so hand-writing the Cython is a fine option.
So why not Ctypes, or??? I think the real strength of Cython in
wrapping C code is that you can write a "thick" wrapper in an
almost_python language. So if you want to vectorize a C function, for
instance, you can write that bit in Cython very easily (and Cython's
built-in understanding of numpy array is very helpful here). When you
use ctypes, you need to write that in pure Python -- easy enough, but
probably not very performant.
With SWIG, etc, you end up writing a fair bi tof C (or SWIG) code to
handle the thicker bits of the wrapper -- so you're dealing with the
raw CPython API, and , well, C. Cython really is an easier option.
I've found that for stuf that is less than very small (i.e. one or two
loops through an array), writing the core code in native C or C++ can
be easier, you know for sure you're not accidentally making expensive
Python calls, etc. but using Cython to call it is still very helpful.
> I'm not sure it is -- I always seem to end up with things that are more
> lines of code with more obscure workarounds than just coding in C directly.
Exactly -- but I don't think that applies to the CPython-API bits, but
rather the core code -- so keep that in C.
In summary, I guess what I think is the power of Cython is the
flexibility in where you draw the line between Python, Cython, and C
-- you can pass pure Python through Cython, or you can do almost
nothing with it but call a C function, and eveything in between.
> From my experience, I would prefer to write such extensions in C directly rather
> than relying on Cython, SWIG, or Boost.Python, because those approaches would
> lead to another dependency (for developers at least),
The dependency is pretty easy to deal with compared to the many others in MPL.
> and requires developers to
> learn how to code in them. Which may not be very hard, but we may as well avoid > that if possible.
Here's where I disagree -- if we go pure C and C-API developers need
to know the Python C-API -- that is actually a pretty big deal, and
hard to get right. Knowing enough Cython to call some C code is a
smaller lift for sure.
Anyway, I saw give it a shot -- I suspect you'll like it.
-Chris
-- 
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chr...@no...

On 11/29/12 10:59 AM, Michael Droettboom wrote:
> I've not had
> much luck with Cython for this kind of thing in the past, but I know it
> is popular.
I'm curious about what problems you've run into and how long it was. In 
the past, Cython hasn't supported C++ very well, but the situation has 
greatly improved recently. See 
http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html for some 
details.
Thanks,
Jason

Thanks, Michiel.
If you read between the lines of what I was saying, that is basically 
where I fall as well. There seems to be a lot of desire to use Cython 
to make the code more accessible, however, and I'm willing to consider 
it if it can be shown to be superior to the raw C/API for this task -- 
I'm not sure it is -- I always seem to end up with things that are more 
lines of code with more obscure workarounds than just coding in C directly.
Cheers,
Mike
On 11/29/2012 08:47 PM, Michiel de Hoon wrote:
> Hi,
>
> The Mac OS X backend is entirely written in C (with some Objective-C elements where necessary). AFAICT, this is the largest C/C++ code in matplotlib. This backend was written from scratch without using Cython, SWIG, or Boost.Python. From my experience, I would prefer to write such extensions in C directly rather than relying on Cython, SWIG, or Boost.Python, because those approaches would lead to another dependency (for developers at least), and requires developers to learn how to code in them. Which may not be very hard, but we may as well avoid that if possible.
>
> I'd be happy to help out with the conversion of the other extensions from CXX to C. I would need some help though to use github appropriately.
>
> Best,
> -Michiel.
>
>
> --- On Thu, 11/29/12, Michael Droettboom <md...@st...> wrote:
>
>> From: Michael Droettboom <md...@st...>
>> Subject: [matplotlib-devel] Experiments in removing/replacing PyCXX
>> To: "mat...@li..." <mat...@li...>
>> Date: Thursday, November 29, 2012, 11:59 AM
>> Given the slow pace of development on
>> PyCXX, I know it has been the
>> desire of some here to remove our dependency on it.
>>
>> I thought a helpful starting point to evaluate the
>> alternatives would be
>> to restructure one of our extensions to not use PyCXX
>> anymore. I've
>> taken the PNG extension, which is reasonably straightforward
>> in that it
>> doesn't define any custom types, but does have some low
>> level C-wrapping
>> challenges, and separated out the Python-specific parts from
>> the
>> libpng-specific parts. The Python-specific parts are
>> now written using
>> the "raw" Python C/API. The other part still uses C++ (not
>> C) and does
>> throw exceptions, but doesn't use classes or templates or
>> anything else
>> that can be difficult to wrap. All of this is on my
>> "no_cxx" branch.
>>
>> Now here's the challenge: can we do better than this using
>> any of the
>> available wrapping tools? Cython, SWIG, Boost.Python
>> etc.? I've not had
>> much luck with Cython for this kind of thing in the past,
>> but I know it
>> is popular. Perhaps someone with more Cython
>> experience would want to
>> take a crack at this and then we could have something
>> concrete to compare...
>>
>> Cheers,
>> Mike
>>
>> ------------------------------------------------------------------------------
>> Keep yourself connected to Go Parallel:
>> VERIFY Test and improve your parallel project with help from
>> experts
>> and peers. http://goparallel.sourceforge.net
>> _______________________________________________
>> Matplotlib-devel mailing list
>> Mat...@li...
>> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>>

Note that we already use a decorator for a similar purpose
(allow_rasterization).
Also, please note that the "draw" method is not just for drawing things.
There are other things being done within the draw method, and I think some
of them still need to be done even though the artist is invisible.
My personal inclination on this issue is to refactor the "draw" method, the
only method being called during the drawing time. But, yes there are
sideeffects.
Regards,
-JJ
On Tue, Nov 27, 2012 at 3:40 AM, Ryan May <rm...@gm...> wrote:
> On Mon, Nov 26, 2012 at 12:23 PM, Eric Firing <ef...@ha...> wrote:
>
>> On 2012年11月26日 7:12 AM, Michael Droettboom wrote:
>> > The problem is that I don't think we can do this for all artists. Some
>> > may need to create groupings, or push and pop state even if they are
>> > "invisible". For instance, this is used in the SVG backend to create
>> > named groupings (possibly empty) that are referenced from Javascript to
>> > provide interactivity. I think I'd rather keep this to the contained
>> > solution in the PR and not try to generalize it beyond that.
>> >
>> > If we did want to generalize, this would only apply to "leaf node"
>> > artists, and not artists that simply exist to contain other artists --
>> > and conceivably we could implement that using either a decorator or
>> > explicit chaining to a base class, but in any event it would have to be
>> > a manual process to determine which artists this would apply to. We
>> > could insert a class in the heirarchy of "ConcreteArtist" (or somesuch)
>> > to handle this.
>>
>> I think we should be rather conservative about this sort of thing.
>> Sometimes it is better to just explicitly put the two lines in each
>> method than to come up with machinery to do it for you. Each level of
>> depth in an inheritance hierarchy or "meta" chain is an additional level
>> of complexity for someone reading the code. And if someone forgets to
>> put in those lines, the penalty is typically from small to nil; but if
>> they are put in automatically by fancy methods, and they are not really
>> wanted or something else goes wrong, it can make debugging painful.
>
>
> I think you and Mike are skirting around a key point here. You can always
> add the line if you need it, but if you don't need it (or can't use it), by
> use of a metaclass, there's no way to "opt out" so to speak.
>
> I'll also add that we don't need to add any more indirection (i.e. another
> Python function call) to our drawing stack--we really need to be doing
> everything possible to take every last millisecond out of the call to
> draw().
>
> Ryan
>
> --
> Ryan May
> Graduate Research Assistant
> School of Meteorology
> University of Oklahoma
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from 795ドル for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>
>

Hi,
The Mac OS X backend is entirely written in C (with some Objective-C elements where necessary). AFAICT, this is the largest C/C++ code in matplotlib. This backend was written from scratch without using Cython, SWIG, or Boost.Python. From my experience, I would prefer to write such extensions in C directly rather than relying on Cython, SWIG, or Boost.Python, because those approaches would lead to another dependency (for developers at least), and requires developers to learn how to code in them. Which may not be very hard, but we may as well avoid that if possible.
I'd be happy to help out with the conversion of the other extensions from CXX to C. I would need some help though to use github appropriately.
Best,
-Michiel.
--- On Thu, 11/29/12, Michael Droettboom <md...@st...> wrote:
> From: Michael Droettboom <md...@st...>
> Subject: [matplotlib-devel] Experiments in removing/replacing PyCXX
> To: "mat...@li..." <mat...@li...>
> Date: Thursday, November 29, 2012, 11:59 AM
> Given the slow pace of development on
> PyCXX, I know it has been the 
> desire of some here to remove our dependency on it.
> 
> I thought a helpful starting point to evaluate the
> alternatives would be 
> to restructure one of our extensions to not use PyCXX
> anymore. I've 
> taken the PNG extension, which is reasonably straightforward
> in that it 
> doesn't define any custom types, but does have some low
> level C-wrapping 
> challenges, and separated out the Python-specific parts from
> the 
> libpng-specific parts. The Python-specific parts are
> now written using 
> the "raw" Python C/API. The other part still uses C++ (not
> C) and does 
> throw exceptions, but doesn't use classes or templates or
> anything else 
> that can be difficult to wrap. All of this is on my
> "no_cxx" branch.
> 
> Now here's the challenge: can we do better than this using
> any of the 
> available wrapping tools? Cython, SWIG, Boost.Python
> etc.? I've not had 
> much luck with Cython for this kind of thing in the past,
> but I know it 
> is popular. Perhaps someone with more Cython
> experience would want to 
> take a crack at this and then we could have something
> concrete to compare...
> 
> Cheers,
> Mike
> 
> ------------------------------------------------------------------------------
> Keep yourself connected to Go Parallel: 
> VERIFY Test and improve your parallel project with help from
> experts 
> and peers. http://goparallel.sourceforge.net
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>