matplotlib-devel Mailing List for matplotlib

On Mon, Dec 3, 2012 at 2:21 PM, Nathaniel Smith <nj...@po...> wrote:
> For the file handle, I would just write
>
> cdef FILE *fp = fdopen(file_obj.fileno(), "w")
>
> and be done with it. This will work with any version of Python etc.
yeah, that makes sense -- though what if you want to be able to
read_to/write_from a file that is already open, and in the middle of
the file somewhere -- would that work?
I just posted a question to the Cython list, and indeed, it looks like
there is no easy answer to the file issue.
-Chris
-- 
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chr...@no...

On Mon, Dec 3, 2012 at 8:24 PM, Chris Barker - NOAA Federal
<chr...@no...> wrote:
> On Mon, Dec 3, 2012 at 11:59 AM, Michael Droettboom <md...@st...> wrote:
>> so there
>> are types in libpng, for example, that we don't actually know the size
>> of. They are different on different platforms. In C, you just include
>> the header. In Cython, I'd have to determine the size of the types in a
>> pre-compilation step, or manually determine their sizes and hard code
>> them for the platforms we care about.
>
> yeah -- this is a tricky problem, however, I think you can follow what
> you'd do in C -- i.e. presumable the header define their own data
> types: png_short or whatever. The actually definition is filled in by
> the pre-processor. So I wonder if you can declare those types in
> Cython, then have it write C code that uses those types, and it all
> gets cleared up at compile time -- maybe. The key is that when you
> declare stuff in Cython, that declaration is used to determine how to
> write the C code, I don't think the declarations themselves are
> translated.
Yeah, this isn't an issue in Cython, it's a totally standard thing
(though perhaps not well documented). When you write
 cdef extern from "png.h":
 ctypedef int png_short
or whatever, what you are saying is "the C compiler knows about a type
called png_short, which acts in an int-like fashion, so Cython, please
use your int rules when dealing with it". So this means that Cython
will know that if you return a png_short from a python function, it
should insert a call to PyInt_FromLong (or maybe PyInt_FromSsize_t? --
cython worries about these things so I don't have to). But Cython only
takes care of the Python<->C interface. It will leave the C compiler
to actually allocate the appropriate memory for png_shorts, perform C
arithmetic, coerce a png_short into a 'long' when necessary, etc.
It's kind of mind-bending to wrap your head around, and it definitely
does help to spend some time reading the C code that Cython spits out
to understand how the mapping works (it's both more and less magic
than it looks -- Python stuff gets carefully expanded, C stuff goes
through almost verbatim), but the end result works amazingly well.
>> It would at least make this a more fair comparison to have the Cython
>> code as Cythonic as possible. However, I couldn't find any ways around
>> using these particular APIs -- other than the Numpy stuff which probably
>> does have a more elegant solution in the form of Cython arrays and
>> memory views.
>
> yup -- that's what I noticed right away -- I"m note sure it there is
> easier handling of file handles.
For the file handle, I would just write
 cdef FILE *fp = fdopen(file_obj.fileno(), "w")
and be done with it. This will work with any version of Python etc.
-n

On Mon, Dec 3, 2012 at 11:59 AM, Michael Droettboom <md...@st...> wrote:
>>> but some of that complexity could be reduced by using Numpy arrays in place of the
>>> image buffer types that each of them contain
>> OR Cython arrays and/or memoryviews -- this is indeed a real strength of Cython.
>
> Sure, but when we return to Python, they should be Numpy arrays which
> have more methods etc. -- or am I missing something?
Cython makes it really easy to switch between ndarrays and
memoryviews, etc -- it's a question of what you want to work with in
your code, so you have write a function that takes numpy arrays and
returns numpy arrays, but uses a memoryview internally (and passes to
C code that way). But I'm not an expert on this, I'mve found that I'm
either doing simplestuff where using numpy arrays directly works fine,
or passing the pointer to the data array off to C:
def a_function_to_call_C( cnp.ndarray[double, ndim=2, mode="c" ] in_array ):
 """
 calls the_c_function, altering the array in-place
 """
 cdef int m, n
 m = in_array.size[0]
 m = in_array.size[1]
 the_c_function( &in_array[0], m, n )
>> It does support the C99 fixed-width integer types:
>> from libc.stdint cimport int16_t, int32_t,
>>
> The problem is that Cython can't actually read the C header,
yeah, this is a pity. There has been some work on auto-generating
Cython from C headers, though nothing mature. For my work, I've been
considering writing some simple pyd-generating code, just to make sure
my data types are inline with the C++ as it may change.
> so there
> are types in libpng, for example, that we don't actually know the size
> of. They are different on different platforms. In C, you just include
> the header. In Cython, I'd have to determine the size of the types in a
> pre-compilation step, or manually determine their sizes and hard code
> them for the platforms we care about.
yeah -- this is a tricky problem, however, I think you can follow what
you'd do in C -- i.e. presumable the header define their own data
types: png_short or whatever. The actually definition is filled in by
the pre-processor. So I wonder if you can declare those types in
Cython, then have it write C code that uses those types, and it all
gets cleared up at compile time -- maybe. The key is that when you
declare stuff in Cython, that declaration is used to determine how to
write the C code, I don't think the declarations themselves are
translated.
> It would at least make this a more fair comparison to have the Cython
> code as Cythonic as possible. However, I couldn't find any ways around
> using these particular APIs -- other than the Numpy stuff which probably
> does have a more elegant solution in the form of Cython arrays and
> memory views.
yup -- that's what I noticed right away -- I"m note sure it there is
easier handling of file handles.
> True. We do have two categories of stuff using PyCXX in matplotlib:
> things that (primarily) wrap third-party C/C++ libraries, and things
> that are actually doing algorithmic heavy lifting. It's quite possible
> we don't want the same solution for all.
And I'm not sure the wrappers all need to be written the same way, either.
-Chris
-- 
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chr...@no...

On 12/03/2012 01:12 PM, Chris Barker - NOAA Federal wrote:
> This argues against making the Cython source code a part of the matplotlib codebase.
>
> huh? are you suggesting that we use Cython to generate the glue, then
> hand-maintain that glue? I think that is a really, rally bad idea --
> generated code is ugly and hard to maintain, it is not designed to be
> human-readable, and we wouldn't get the advantages of bug-fixes
> further development in Cython.
>
> So -- if you use Cython, you want to keep using, and theat means the
> Cython source IS the source. I agree that it's a good idea to ship the
> generated code as well, so that no one that is not touching the Cython
> has to generate. Other than the slight mess from generated files
> showing up in diffs, etc, this really works just fine.
I agree with this approach.
>
> Any reason MPL couldn't continue with EXACTLY the same approach now
> used with C_XX -- it generates code as well, yes?
No -- PyCXX is just C++. Its killer feature is that it provides a 
fairly thin layer around the Python C/API that does implicit reference 
counting through the use of C++ constructors and destructors. I 
actually think it's a really elegant approach to the problem. The 
downside we're running into is that it's barely maintained, so using 
vanilla upstream as provided by packagers is not viable. An alternative 
to all of this discussion is to fork PyCXX and release as needed. The 
maintenance required is primarily when new versions of Python are 
released, so it wouldn't necessarily be a huge undertaking. However, I 
know some are reluctant to use a relatively unused tool.
>
> Michael Droettboom wrote:
>
>> For the PNG extension specifically, it was creating callbacks that can
>> be called from C and the setjmp magic that libpng requires. I think
>> it's possible to do it, but I was surprised at how non-obvious those
>> pieces of Cython were. I was really hoping by creating this experiment
>> that a Cython expert would step up and show the way ;)
> Did you not get the support you expected from the cython list? Anyway,
> there's no reason you can't keep stuff in C that's easier in C (or did
> C_XX make this easy?).
The support has been adequate, but the solutions aren't always an 
improvement over raw Python/C API (not just in terms of lines of code 
but in terms of the number of layers of abstraction and "magic" between 
the coder and what actually happens).
> I think making basic callbacks is actually
> pretty straightforward, but In don't know about the setjmp magic (I
> have no idea hat that means!).
It turned out to be not terrible once I figured out the correct incantation.
>
>> The Agg backend has more C++-specific challenges, particularly
>> instantiating very complex template expressions --
> I'm guessing you'd do the complex template stuff in C++ -- and let
> Cython see a more traditional static API.
Agreed -- I'm really only considering replacing the glue code provided 
by PyCXX, not the whole thing. matplotlib's C/C++ code has been around 
for a while and has been fairly vetted at this point, so I don't think a 
wholesale rewrite makes sense.
>
>> but some of that complexity could be reduced by using Numpy arrays in place of the
>> image buffer types that each of them contain
> OR Cython arrays and/or memoryviews -- this is indeed a real strength of Cython.
Sure, but when we return to Python, they should be Numpy arrays which 
have more methods etc. -- or am I missing something?
>> The Cython version isn't that much shorter than the C++ version.
> I think some things make sense to keep in C++, though I do see a fair
> bit of calls (in the C++) to the python API -- I'm surprised there
> isn't much code advantage, but anyway, the goal is more robust/easier
> to maintain, which may correlate with code-size, but not completely.
>
>> These declarations aren't exact matches to what one would find in the header file(s) >because Cython doesn't support exact-width data types etc.
> It does support the C99 fixed-width integer types:
>
> from libc.stdint cimport int16_t, int32_t,
>
> Or are you talking about something else?
The problem is that Cython can't actually read the C header, so there 
are types in libpng, for example, that we don't actually know the size 
of. They are different on different platforms. In C, you just include 
the header. In Cython, I'd have to determine the size of the types in a 
pre-compilation step, or manually determine their sizes and hard code 
them for the platforms we care about.
>
>> I'm not sure why some of the Python/C API calls I needed were not defined in Cython's include wrappers.
> I suspect that's an oversight -- for the most part, stuff has been
> added as it's needed.
>
> One other note -- from a quick glance at your Cython code, it looks
> like you did almost everything is Cython-that-will-compile-to-pure-C
> -- i.e. a lot of calls to the CPython API. But the whole point of
> Cython is that it makes those calls for you. So you can do type
> checking, and switching on types, and calling np.asarray(), etc, etc,
> etc, in Python, without calling the CPython api yourself. I know
> nothing of the PNG API, and am pretty week on the CPython API (and C
> for that matter), but I it's likely that the Cython code you've
> written could be much simplified.
It would at least make this a more fair comparison to have the Cython 
code as Cythonic as possible. However, I couldn't find any ways around 
using these particular APIs -- other than the Numpy stuff which probably 
does have a more elegant solution in the form of Cython arrays and 
memory views.
>
>
>> Once things compiled, due to my own mistake, calling the function segfaulted. Debugging
>> that segfault in gdb required, again, wading through the generated code. Using gdb on
>> hand-written code is *much* nicer.
> for sure -- there is a plug-in/add-on/something for using gdb on
> Cython code -- I haven't used it but I imagine it would help.
Ah. I wasn't aware of that. Thanks for pointing that out. I have the 
CPython plug-in for gdb and it's great.
>
> Ian Thomas wrote:	
>> I have never used Cython, but to me the code looks like an inelegant combination of
>> Python,C/C++ and some Cython-specific stuff.
> well, yes, it is that!
>
>> I can see the advantage of this approach for small sections of code, but I have strong > reservations about using it for complicated modules that have extensive use of
>> templated code and/or Standard Template Library collections (mpl has examples of
>> both of these).
> So far, I've found that Cython is good for:
> - The simple stuff -- basic loops through numpy arrays, etc.
> - wrapping/calling more complex C or C++
> -- essentially handling the reference counting and python type
> packing/unpacking of python types.
>
> So we find we do write some shim code in C++ to make the access to the
> core libraries Cython-friendly. We haven't dealt with complex
> templating, etc, but I'd guess if we did I'd keep that in C++. And
> since the resulting actual glue code is pretty simple, it makes the
> debugging easier.
>
>> Maybe rather than asking "if we switched to using Cython, would more participate", I
>> should be asking "among those that can participate in removing the PyCXX
>> dependency, what is the preferred approach?"
> I don't know that we need a one-sieze fits all approach -- perhaps
> some bits make the most sense to move to plain old C/C++, and some to
> Cython, either because of the nature of the code itself, or because of
> the experience/preference of the person that takes ownership of a
> particular problem.
>
True. We do have two categories of stuff using PyCXX in matplotlib: 
things that (primarily) wrap third-party C/C++ libraries, and things 
that are actually doing algorithmic heavy lifting. It's quite possible 
we don't want the same solution for all.
Cheers,
Mike

On Sat, Dec 1, 2012 at 6:44 AM, Michiel de Hoon
>
> Since the Python/C glue code is modified only very rarely, there may not be a need for regenerating the Python/C glue code by developers or users from a Cython source code.
True.
> In addition, it is much easier to maintain the Python/C glue code than to write it from scratch. Once you have the Python/C glue code, it's relatively straightforward to modify it by looking at the existing Python/C glue code.
>
not so true -- getting reference counting right, etc is difficult -- I
suppose once the glue code is robust, and all you are changing is a
bit of API to the C, maybe....
>
> This argues against making the Cython source code a part of the matplotlib codebase.
>
huh? are you suggesting that we use Cython to generate the glue, then
hand-maintain that glue? I think that is a really, rally bad idea --
generated code is ugly and hard to maintain, it is not designed to be
human-readable, and we wouldn't get the advantages of bug-fixes
further development in Cython.
So -- if you use Cython, you want to keep using, and theat means the
Cython source IS the source. I agree that it's a good idea to ship the
generated code as well, so that no one that is not touching the Cython
has to generate. Other than the slight mess from generated files
showing up in diffs, etc, this really works just fine.
Any reason MPL couldn't continue with EXACTLY the same approach now
used with C_XX -- it generates code as well, yes?
Michael Droettboom wrote:
> For the PNG extension specifically, it was creating callbacks that can
> be called from C and the setjmp magic that libpng requires. I think
> it's possible to do it, but I was surprised at how non-obvious those
> pieces of Cython were. I was really hoping by creating this experiment
> that a Cython expert would step up and show the way ;)
Did you not get the support you expected from the cython list? Anyway,
there's no reason you can't keep stuff in C that's easier in C (or did
C_XX make this easy?). I think making basic callbacks is actually
pretty straightforward, but In don't know about the setjmp magic (I
have no idea hat that means!).
> The Agg backend has more C++-specific challenges, particularly
> instantiating very complex template expressions --
I'm guessing you'd do the complex template stuff in C++ -- and let
Cython see a more traditional static API.
> but some of that complexity could be reduced by using Numpy arrays in place of the
> image buffer types that each of them contain
OR Cython arrays and/or memoryviews -- this is indeed a real strength of Cython.
> The Cython version isn't that much shorter than the C++ version.
I think some things make sense to keep in C++, though I do see a fair
bit of calls (in the C++) to the python API -- I'm surprised there
isn't much code advantage, but anyway, the goal is more robust/easier
to maintain, which may correlate with code-size, but not completely.
> These declarations aren't exact matches to what one would find in the header file(s) >because Cython doesn't support exact-width data types etc.
It does support the C99 fixed-width integer types:
from libc.stdint cimport int16_t, int32_t,
Or are you talking about something else?
> I'm not sure why some of the Python/C API calls I needed were not defined in Cython's include wrappers.
I suspect that's an oversight -- for the most part, stuff has been
added as it's needed.
One other note -- from a quick glance at your Cython code, it looks
like you did almost everything is Cython-that-will-compile-to-pure-C
-- i.e. a lot of calls to the CPython API. But the whole point of
Cython is that it makes those calls for you. So you can do type
checking, and switching on types, and calling np.asarray(), etc, etc,
etc, in Python, without calling the CPython api yourself. I know
nothing of the PNG API, and am pretty week on the CPython API (and C
for that matter), but I it's likely that the Cython code you've
written could be much simplified.
> Once things compiled, due to my own mistake, calling the function segfaulted. Debugging
> that segfault in gdb required, again, wading through the generated code. Using gdb on
> hand-written code is *much* nicer.
for sure -- there is a plug-in/add-on/something for using gdb on
Cython code -- I haven't used it but I imagine it would help.
Ian Thomas wrote:	
> I have never used Cython, but to me the code looks like an inelegant combination of
> Python,C/C++ and some Cython-specific stuff.
well, yes, it is that!
> I can see the advantage of this approach for small sections of code, but I have strong > reservations about using it for complicated modules that have extensive use of
> templated code and/or Standard Template Library collections (mpl has examples of
> both of these).
So far, I've found that Cython is good for:
 - The simple stuff -- basic loops through numpy arrays, etc.
 - wrapping/calling more complex C or C++
 -- essentially handling the reference counting and python type
packing/unpacking of python types.
So we find we do write some shim code in C++ to make the access to the
core libraries Cython-friendly. We haven't dealt with complex
templating, etc, but I'd guess if we did I'd keep that in C++. And
since the resulting actual glue code is pretty simple, it makes the
debugging easier.
> Maybe rather than asking "if we switched to using Cython, would more participate", I
> should be asking "among those that can participate in removing the PyCXX
> dependency, what is the preferred approach?"
I don't know that we need a one-sieze fits all approach -- perhaps
some bits make the most sense to move to plain old C/C++, and some to
Cython, either because of the nature of the code itself, or because of
the experience/preference of the person that takes ownership of a
particular problem.
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chr...@no...

On 12/03/2012 04:07 AM, Ian Thomas wrote:
> I vote for using the raw Python/C API. I've written a couple of PyCXX 
> extensions and whilst it is mostly convenient, PyCXX doesn't support 
> the use of numpy arrays so for them you have to use the Python/C API. 
> This means dealing with the reference counting yourself for numpy 
> arrays; extending this to do the reference counting for all python 
> objects is not onerous. Dealing with object lifetimes is 
> bread-and-butter work for C/C++ developers.
That matches my experience quite well.
> I have never used Cython, but to me the code looks like an inelegant 
> combination of Python, C/C++ and some Cython-specific stuff. I can 
> see the advantage of this approach for small sections of code, but I 
> have strong reservations about using it for complicated modules that 
> have extensive use of templated code and/or Standard Template Library 
> collections (mpl has examples of both of these).
Even for C libraries like libpng, which requires use of C function 
callbacks for some things, Cython is more convoluted, particularly when 
things go wrong and require debugging. (Running gdb over generated 
Cython code is not fun!) And in my view, writing code like that 
requires a pretty deep understanding of the Python/C API, C itself, and 
the rather complex transformations that Cython performs. Writing 
directly to the Python/C API only requires knowledge of the first two. 
And there's a large body of books/tutorials/debuggers/tools for C that 
don't really have equivalents for Cython.
>
> I agree that Cython opens us up to a larger body of contributors, but 
> I don't think that this is necessarily a good thing. I think this 
> really means opens us up to a larger body of Python/Cython 
> contributors, and is a view expressed from the Python side of the 
> fence and has the wrong emphasis. I am primarily a C++ developer is a 
> sea of Python developers, and rather than encourage other Python 
> contributors to dip their toes into C/C++ via Cython I think we should 
> be encouraging C/C++ contributors to do what they do best. We only 
> need a few C/C++ developers if we allow them to use their skills in 
> their preferred way, and they are used to interfacing to legacy APIs 
> and dealing with object lifetimes.
I think Cython is well suited to writing new algorithmic code to speed 
up hot spots in Python code. I don't think it's as well suited as glue 
between C and Python -- that was not a main goal of the original Pyrex 
project, IIRC. It feels kind of tacked on and not a very good fit to 
the problem. Most of the work to remove PyCXX use in matplotlib is 
either wrapping third-party libraries (where Cython doesn't really 
shine), or wrapping C/C++ code in our own tree that's already 
well-tested and vetted, and I wouldn't propose rewriting that in 
Cython. I'm only really considering rewriting the Python-to-C interface 
layer.
>
> OK, cards on the table. If we wanted to switch all of our PyCXX 
> modules to use the raw Python/C API, I would happily take on some of 
> the burden for making the changes and ongoing maintenance of such 
> modules. Particularly if, in return, I get some help with my 
> sometimes substandard Python! If we go down the Cython route I 
> couldn't make this offer; would our many Cython advocates take on the 
> responsibility of changing and maintaining my C++ code in this scenario?
That's a good way to look at this. I was definitely hoping that moving 
to Cython might open us up to more developers, but at the end of the 
day, the chosen tool should be the one preferred by those doing the 
work. Maybe rather than asking "if we switched to using Cython, would 
more participate", I should be asking "among those that can participate 
in removing the PyCXX dependency, what is the preferred approach?"
Cheers,
Mike

I vote for using the raw Python/C API. I've written a couple of PyCXX
extensions and whilst it is mostly convenient, PyCXX doesn't support the
use of numpy arrays so for them you have to use the Python/C API. This
means dealing with the reference counting yourself for numpy arrays;
extending this to do the reference counting for all python objects is not
onerous. Dealing with object lifetimes is bread-and-butter work for C/C++
developers.
I have never used Cython, but to me the code looks like an inelegant
combination of Python, C/C++ and some Cython-specific stuff. I can see the
advantage of this approach for small sections of code, but I have strong
reservations about using it for complicated modules that have extensive use
of templated code and/or Standard Template Library collections (mpl has
examples of both of these).
I agree that Cython opens us up to a larger body of contributors, but I
don't think that this is necessarily a good thing. I think this really
means opens us up to a larger body of Python/Cython contributors, and is a
view expressed from the Python side of the fence and has the wrong
emphasis. I am primarily a C++ developer is a sea of Python developers,
and rather than encourage other Python contributors to dip their toes into
C/C++ via Cython I think we should be encouraging C/C++ contributors to do
what they do best. We only need a few C/C++ developers if we allow them to
use their skills in their preferred way, and they are used to interfacing
to legacy APIs and dealing with object lifetimes.
OK, cards on the table. If we wanted to switch all of our PyCXX modules to
use the raw Python/C API, I would happily take on some of the burden for
making the changes and ongoing maintenance of such modules. Particularly
if, in return, I get some help with my sometimes substandard Python! If we
go down the Cython route I couldn't make this offer; would our many Cython
advocates take on the responsibility of changing and maintaining my C++
code in this scenario?
Ian Thomas

On Sun, Dec 2, 2012 at 8:06 PM, Michael Droettboom <md...@st...> wrote:
> I've pushed a fix to v1.2.x and master for this new problem
> (35ee2184111fb8f80027869d8ee309c7f4e5a467). Unfortunately, another rebase
> of your branches is in order in order to get this fix.
Still failing: https://travis-ci.org/matplotlib/matplotlib/jobs/3469141
>
> Mike
>
>
> On 12/02/2012 12:23 PM, Thomas Kluyver wrote:
>
> On 2 December 2012 17:02, Damon McDougall <dam...@gm...> wrote:
>>
>> > Still failing even with the workaround. Here's proof:
>> > https://github.com/matplotlib/matplotlib/pull/1549
>>
>> And looks like Thomas reported an issue too:
>> https://github.com/matplotlib/matplotlib/issues/1548
>
>
> This is a different problem, though (unless it's a really bizarre symptom of
> the other problem). Now it's an error in compiling matplotlib.
>
> Thomas
>
>
-- 
Damon McDougall
http://www.damon-is-a-geek.com
Institute for Computational Engineering Sciences
201 E. 24th St.
Stop C0200
The University of Texas at Austin
Austin, TX 78712-1229

I've pushed a fix to v1.2.x and master for this new problem 
(35ee2184111fb8f80027869d8ee309c7f4e5a467). Unfortunately, another 
rebase of your branches is in order in order to get this fix.
Mike
On 12/02/2012 12:23 PM, Thomas Kluyver wrote:
> On 2 December 2012 17:02, Damon McDougall <dam...@gm... 
> <mailto:dam...@gm...>> wrote:
>
> > Still failing even with the workaround. Here's proof:
> > https://github.com/matplotlib/matplotlib/pull/1549
>
> And looks like Thomas reported an issue too:
> https://github.com/matplotlib/matplotlib/issues/1548
>
>
> This is a different problem, though (unless it's a really bizarre 
> symptom of the other problem). Now it's an error in compiling matplotlib.
>
> Thomas