matplotlib-devel Mailing List for matplotlib

On 2012年12月06日 7:16 AM, Michael Droettboom wrote:
> I think this has been a very helpful and useful discussion.
>
> I'm going to attempt to summarize this discussion and propose some ways
> forward here.
>
> The impetus for this discussion is that PyCXX seems to be not adequately
> maintained. It is difficult to build matplotlib with "vanilla" PyCXX in
> certain configurations. (This history sort of predates this thread).
>
> So we have some options:
>
> 1) One way forward is to offer to take ownership of the PyCXX project.
> (I'm not using the "f" word here... I'd much prefer to just become more
> involved upstream somehow). I don't think this would be considerable
> additional work, as most of that work has been done in matplotlib for
> some time anyway. To the extent that it needs new features, it would be
> killer to add support for Numpy so Numpy no longer required manual
> reference counting. I had initially dismissed this approach, as I seem
> to be in the minority in liking PyCXX -- I happen to think it's
> fundamentally an extremely good approach to the problem: it helps with
> reference counting errors, but otherwise mostly stays out of the way.
> But I'd like to remove any one person as a bottleneck by choosing
> something that's more preferred all around.
>
> 2) Move to a different wrapping mechanism of some sort. While Cython is
> the clear choice for a third-party Python/C wrapping tool, it seems to
> be polarizing. (I won't attempt to repeat or summarize, but I think
> good points have been made on either side of the argument). I think
> it's ok to allow Cython to be used in matplotlib, given that we include
> both the Cython source and the generated C in the source repository such
> that matplotlib can be built without Cython installed. There are many
> other projects doing this that can provide best practices for us. I
> don't think, however, that we can or should require that all wrapping is
> done with Cython. I think we should allow raw Python/C API where it is
> most appropriate (and that is mainly in the case of wrapping third-party
> libraries, such as the png module and the macosx module which is already
> raw Python/C API). What I wouldn't want to see is the use of more than
> one wrapping tool, if only for reasons of proliferation of dependencies.
> (I count the Python/C API as "free" since it's always available
> anyway). I haven't seen in this discussion anyone really pushing for
> any of the alternatives (SWIG, Boost.Python, etc.) in any event.
>
> Note also, the goal is to deal with the PyCXX "problem", not rewrite
> large chunks of our existing and well-tested C/C++ code base in Cython,
> unless someone sees a real clear benefit to doing that for a particular
> module and is highly motivated to do the work. This is primarily about
> refactoring the code so that the interface layer between Python and C is
> separated and then replaced with either Cython or raw Python/C API using
> the most appropriate tool for the job.
>
> 3) Any other options...?
>
> Cheers,
> Mike
Mike,
That is an excellent summary. The options actually are not mutually 
exclusive; perhaps what we are considering is more in the line of 
nudging evolution in one direction or the other, not pushing for rapid 
extinction of a species.
Regarding PyCXX, I respect your opinion that it is a good match for what 
it does. To the limited extent that I can work with C++ at all, I don't 
have any problem with its use in mpl. I do share the concern about 
depending heavily on it, given that problems with it have cropped up, 
and you have been the only one willing and able to deal with those 
problems. Since PyCXX is a pure C++ construct, perhaps other C++ 
gurus--and it seems that we now have more than previously--would be 
willing to take a closer look at it, and reconsider whether they can 
relieve the single-person-bottleneck problem.
There is always a tradeoff between going to a higher-level language or 
library versus sticking with lower levels. Personally, I like C over 
C++ because the former is simple enough that I can generally figure out 
what is going on; but I like Cython over raw Python/C API because its 
internal complexity allows an external simplicity, hiding all sorts of 
things I really don't want to have to think about. Going to higher 
levels always brings the risk of dependency on a complex system, whether 
it be Cython or PyCXX or Agg, or even the C/C++ compiler itself. The 
cost/benefit ratios of such tradeoffs vary greatly with the situation, 
and from person to person, depending on training, experience, and 
personal quirks. So we just have to keep looking for the balance that 
is appropriate to the task, the times, and the people at hand. Your 
summary nicely facilitates that balancing act.
Eric

Just for completeness, there is also ctypes. I wrapped the freetype library (http://code.google.com/p/freetype-py/) using it and it is quite easy (and boring). But this only works for C (not C++).
Nicolas
On Dec 6, 2012, at 18:06 , Michael Droettboom wrote:
> I think this has been a very helpful and useful discussion.
> 
> I'm going to attempt to summarize this discussion and propose some ways 
> forward here.
> 
> The impetus for this discussion is that PyCXX seems to be not adequately 
> maintained. It is difficult to build matplotlib with "vanilla" PyCXX in 
> certain configurations. (This history sort of predates this thread).
> 
> So we have some options:
> 
> 1) One way forward is to offer to take ownership of the PyCXX project. 
> (I'm not using the "f" word here... I'd much prefer to just become more 
> involved upstream somehow). I don't think this would be considerable 
> additional work, as most of that work has been done in matplotlib for 
> some time anyway. To the extent that it needs new features, it would be 
> killer to add support for Numpy so Numpy no longer required manual 
> reference counting. I had initially dismissed this approach, as I seem 
> to be in the minority in liking PyCXX -- I happen to think it's 
> fundamentally an extremely good approach to the problem: it helps with 
> reference counting errors, but otherwise mostly stays out of the way. 
> But I'd like to remove any one person as a bottleneck by choosing 
> something that's more preferred all around.
> 
> 2) Move to a different wrapping mechanism of some sort. While Cython is 
> the clear choice for a third-party Python/C wrapping tool, it seems to 
> be polarizing. (I won't attempt to repeat or summarize, but I think 
> good points have been made on either side of the argument). I think 
> it's ok to allow Cython to be used in matplotlib, given that we include 
> both the Cython source and the generated C in the source repository such 
> that matplotlib can be built without Cython installed. There are many 
> other projects doing this that can provide best practices for us. I 
> don't think, however, that we can or should require that all wrapping is 
> done with Cython. I think we should allow raw Python/C API where it is 
> most appropriate (and that is mainly in the case of wrapping third-party 
> libraries, such as the png module and the macosx module which is already 
> raw Python/C API). What I wouldn't want to see is the use of more than 
> one wrapping tool, if only for reasons of proliferation of 
> dependencies. (I count the Python/C API as "free" since it's always 
> available anyway). I haven't seen in this discussion anyone really 
> pushing for any of the alternatives (SWIG, Boost.Python, etc.) in any event.
> 
> Note also, the goal is to deal with the PyCXX "problem", not rewrite 
> large chunks of our existing and well-tested C/C++ code base in Cython, 
> unless someone sees a real clear benefit to doing that for a particular 
> module and is highly motivated to do the work. This is primarily about 
> refactoring the code so that the interface layer between Python and C is 
> separated and then replaced with either Cython or raw Python/C API using 
> the most appropriate tool for the job.
> 
> 3) Any other options...?
> 

I think this has been a very helpful and useful discussion.
I'm going to attempt to summarize this discussion and propose some ways 
forward here.
The impetus for this discussion is that PyCXX seems to be not adequately 
maintained. It is difficult to build matplotlib with "vanilla" PyCXX in 
certain configurations. (This history sort of predates this thread).
So we have some options:
1) One way forward is to offer to take ownership of the PyCXX project. 
(I'm not using the "f" word here... I'd much prefer to just become more 
involved upstream somehow). I don't think this would be considerable 
additional work, as most of that work has been done in matplotlib for 
some time anyway. To the extent that it needs new features, it would be 
killer to add support for Numpy so Numpy no longer required manual 
reference counting. I had initially dismissed this approach, as I seem 
to be in the minority in liking PyCXX -- I happen to think it's 
fundamentally an extremely good approach to the problem: it helps with 
reference counting errors, but otherwise mostly stays out of the way. 
But I'd like to remove any one person as a bottleneck by choosing 
something that's more preferred all around.
2) Move to a different wrapping mechanism of some sort. While Cython is 
the clear choice for a third-party Python/C wrapping tool, it seems to 
be polarizing. (I won't attempt to repeat or summarize, but I think 
good points have been made on either side of the argument). I think 
it's ok to allow Cython to be used in matplotlib, given that we include 
both the Cython source and the generated C in the source repository such 
that matplotlib can be built without Cython installed. There are many 
other projects doing this that can provide best practices for us. I 
don't think, however, that we can or should require that all wrapping is 
done with Cython. I think we should allow raw Python/C API where it is 
most appropriate (and that is mainly in the case of wrapping third-party 
libraries, such as the png module and the macosx module which is already 
raw Python/C API). What I wouldn't want to see is the use of more than 
one wrapping tool, if only for reasons of proliferation of dependencies. 
 (I count the Python/C API as "free" since it's always available 
anyway). I haven't seen in this discussion anyone really pushing for 
any of the alternatives (SWIG, Boost.Python, etc.) in any event.
Note also, the goal is to deal with the PyCXX "problem", not rewrite 
large chunks of our existing and well-tested C/C++ code base in Cython, 
unless someone sees a real clear benefit to doing that for a particular 
module and is highly motivated to do the work. This is primarily about 
refactoring the code so that the interface layer between Python and C is 
separated and then replaced with either Cython or raw Python/C API using 
the most appropriate tool for the job.
3) Any other options...?
Cheers,
Mike
On 12/04/2012 05:33 PM, Eric Firing wrote:
> On 2012年12月04日 12:07 PM, Damon McDougall wrote:
>> On Mon, Dec 3, 2012 at 12:12 PM, Chris Barker - NOAA Federal
>> <chr...@no...> wrote:
>>> generated code is ugly and hard to maintain, it is not designed to be
>>> human-readable, and we wouldn't get the advantages of bug-fixes
>>> further development in Cython.
>> As far as I'm concerned, this is an argument against Cython.
> Nonsense. It is an argument against the idea of maintaining the
> generated code directly, rather than maintaining the cython source code
> and regenerating the C code as needed. That idea never made any sense
> in the first place. I doubt that anyone follows it. Chris already
> pointed this out. Would you maintain the assembly code generated by
> your C++ compiler? Do you consider the fact that this is unreadable and
> unmaintainable a reason to avoid using that compiler, and instead to
> code directly in assembly?
>
>> I've had to touch the C/C++/ObjC codebase. It was not automatically
>> generated by Cython and it's not that hard to read. There's almost
>> certainly a C/C++/ObjC expert around to help out. There's almost
>> certainly Cython experts to help out, too. There is almost certainly
>> *not* an expert in Cython-generated C code that is hard to read.
>>
> There doesn't need to be.
>
>> I vote raw Python/C API. Managing reference counters is not the
>> mundane task pythonistas make it out to be, in my opinion. If you know
>> ObjC, you've had to do your own reference counting. If you know C,
>> you've had to do your own memory management. If you know C++, you've
>> had to do your own new/delete (or destructor) management. I agree not
>> having to worry about reference counting is nice positive, but I don't
>> think it outweighs the negatives.
> You have completely misrepresented the negatives.
>
>> It seems to me that Cython is a 'middle-man' tool, with the added
>> downside of hard-to-maintain under-code.
>>
> Please, if you don't use Cython yourself, and therefore don't know it
> well, refrain from these sorts of criticisms. In normal cython use, one
> *never* modifies the code it generates. In developing with cython, one
> *might* read this code to find out what is going on, and especially to
> find out whether one inadvertently triggered a call to the python API by
> forgetting to declare a variable, for example. This is pretty easy,
> because the comments in the generated code show exactly which source
> line has generated each chunk of generated code. Context is included.
> It is very nicely done.
>
> Eric
>
> ------------------------------------------------------------------------------
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
> Remotely access PCs and mobile devices and provide instant support
> Improve your efficiency, and focus on delivering more value-add services
> Discover what IT Professionals Know. Rescue delivers
> http://p.sf.net/sfu/logmein_12329d2d
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

I think this has been a very helpful and useful discussion.
I'm going to attempt to summarize this discussion and propose some ways 
forward here.
The impetus for this discussion is that PyCXX seems to be not adequately 
maintained. It is difficult to build matplotlib with "vanilla" PyCXX in 
certain configurations. (This history sort of predates this thread).
So we have some options:
1) One way forward is to offer to take ownership of the PyCXX project. 
(I'm not using the "f" word here... I'd much prefer to just become more 
involved upstream somehow). I don't think this would be considerable 
additional work, as most of that work has been done in matplotlib for 
some time anyway. To the extent that it needs new features, it would be 
killer to add support for Numpy so Numpy no longer required manual 
reference counting. I had initially dismissed this approach, as I seem 
to be in the minority in liking PyCXX -- I happen to think it's 
fundamentally an extremely good approach to the problem: it helps with 
reference counting errors, but otherwise mostly stays out of the way. 
But I'd like to remove any one person as a bottleneck by choosing 
something that's more preferred all around.
2) Move to a different wrapping mechanism of some sort. While Cython is 
the clear choice for a third-party Python/C wrapping tool, it seems to 
be polarizing. (I won't attempt to repeat or summarize, but I think 
good points have been made on either side of the argument). I think 
it's ok to allow Cython to be used in matplotlib, given that we include 
both the Cython source and the generated C in the source repository such 
that matplotlib can be built without Cython installed. There are many 
other projects doing this that can provide best practices for us. I 
don't think, however, that we can or should require that all wrapping is 
done with Cython. I think we should allow raw Python/C API where it is 
most appropriate (and that is mainly in the case of wrapping third-party 
libraries, such as the png module and the macosx module which is already 
raw Python/C API). What I wouldn't want to see is the use of more than 
one wrapping tool, if only for reasons of proliferation of 
dependencies. (I count the Python/C API as "free" since it's always 
available anyway). I haven't seen in this discussion anyone really 
pushing for any of the alternatives (SWIG, Boost.Python, etc.) in any event.
Note also, the goal is to deal with the PyCXX "problem", not rewrite 
large chunks of our existing and well-tested C/C++ code base in Cython, 
unless someone sees a real clear benefit to doing that for a particular 
module and is highly motivated to do the work. This is primarily about 
refactoring the code so that the interface layer between Python and C is 
separated and then replaced with either Cython or raw Python/C API using 
the most appropriate tool for the job.
3) Any other options...?
Cheers,
Mike
On 12/04/2012 05:33 PM, Eric Firing wrote:
> On 2012年12月04日 12:07 PM, Damon McDougall wrote:
>> On Mon, Dec 3, 2012 at 12:12 PM, Chris Barker - NOAA Federal
>> <chr...@no...> wrote:
>>> generated code is ugly and hard to maintain, it is not designed to be
>>> human-readable, and we wouldn't get the advantages of bug-fixes
>>> further development in Cython.
>> As far as I'm concerned, this is an argument against Cython.
> Nonsense. It is an argument against the idea of maintaining the
> generated code directly, rather than maintaining the cython source code
> and regenerating the C code as needed. That idea never made any sense
> in the first place. I doubt that anyone follows it. Chris already
> pointed this out. Would you maintain the assembly code generated by
> your C++ compiler? Do you consider the fact that this is unreadable and
> unmaintainable a reason to avoid using that compiler, and instead to
> code directly in assembly?
>
>> I've had to touch the C/C++/ObjC codebase. It was not automatically
>> generated by Cython and it's not that hard to read. There's almost
>> certainly a C/C++/ObjC expert around to help out. There's almost
>> certainly Cython experts to help out, too. There is almost certainly
>> *not* an expert in Cython-generated C code that is hard to read.
>>
> There doesn't need to be.
>
>> I vote raw Python/C API. Managing reference counters is not the
>> mundane task pythonistas make it out to be, in my opinion. If you know
>> ObjC, you've had to do your own reference counting. If you know C,
>> you've had to do your own memory management. If you know C++, you've
>> had to do your own new/delete (or destructor) management. I agree not
>> having to worry about reference counting is nice positive, but I don't
>> think it outweighs the negatives.
> You have completely misrepresented the negatives.
>
>> It seems to me that Cython is a 'middle-man' tool, with the added
>> downside of hard-to-maintain under-code.
>>
> Please, if you don't use Cython yourself, and therefore don't know it
> well, refrain from these sorts of criticisms. In normal cython use, one
> *never* modifies the code it generates. In developing with cython, one
> *might* read this code to find out what is going on, and especially to
> find out whether one inadvertently triggered a call to the python API by
> forgetting to declare a variable, for example. This is pretty easy,
> because the comments in the generated code show exactly which source
> line has generated each chunk of generated code. Context is included.
> It is very nicely done.
>
> Eric
>
> ------------------------------------------------------------------------------
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
> Remotely access PCs and mobile devices and provide instant support
> Improve your efficiency, and focus on delivering more value-add services
> Discover what IT Professionals Know. Rescue delivers
> http://p.sf.net/sfu/logmein_12329d2d
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel