matplotlib-devel Mailing List for matplotlib

On 2012年12月04日 12:07 PM, Damon McDougall wrote:
> On Mon, Dec 3, 2012 at 12:12 PM, Chris Barker - NOAA Federal
> <chr...@no...> wrote:
>> generated code is ugly and hard to maintain, it is not designed to be
>> human-readable, and we wouldn't get the advantages of bug-fixes
>> further development in Cython.
>
> As far as I'm concerned, this is an argument against Cython.
Nonsense. It is an argument against the idea of maintaining the 
generated code directly, rather than maintaining the cython source code 
and regenerating the C code as needed. That idea never made any sense 
in the first place. I doubt that anyone follows it. Chris already 
pointed this out. Would you maintain the assembly code generated by 
your C++ compiler? Do you consider the fact that this is unreadable and 
unmaintainable a reason to avoid using that compiler, and instead to 
code directly in assembly?
>
> I've had to touch the C/C++/ObjC codebase. It was not automatically
> generated by Cython and it's not that hard to read. There's almost
> certainly a C/C++/ObjC expert around to help out. There's almost
> certainly Cython experts to help out, too. There is almost certainly
> *not* an expert in Cython-generated C code that is hard to read.
>
There doesn't need to be.
> I vote raw Python/C API. Managing reference counters is not the
> mundane task pythonistas make it out to be, in my opinion. If you know
> ObjC, you've had to do your own reference counting. If you know C,
> you've had to do your own memory management. If you know C++, you've
> had to do your own new/delete (or destructor) management. I agree not
> having to worry about reference counting is nice positive, but I don't
> think it outweighs the negatives.
You have completely misrepresented the negatives.
>
> It seems to me that Cython is a 'middle-man' tool, with the added
> downside of hard-to-maintain under-code.
>
Please, if you don't use Cython yourself, and therefore don't know it 
well, refrain from these sorts of criticisms. In normal cython use, one 
*never* modifies the code it generates. In developing with cython, one 
*might* read this code to find out what is going on, and especially to 
find out whether one inadvertently triggered a call to the python API by 
forgetting to declare a variable, for example. This is pretty easy, 
because the comments in the generated code show exactly which source 
line has generated each chunk of generated code. Context is included. 
It is very nicely done.
Eric

On Tue, Dec 4, 2012 at 4:07 PM, Damon McDougall
<dam...@gm...>wrote:
> On Mon, Dec 3, 2012 at 12:12 PM, Chris Barker - NOAA Federal
> <chr...@no...> wrote:
> > generated code is ugly and hard to maintain, it is not designed to be
> > human-readable, and we wouldn't get the advantages of bug-fixes
> > further development in Cython.
>
> As far as I'm concerned, this is an argument against Cython.
>
> I've had to touch the C/C++/ObjC codebase. It was not automatically
> generated by Cython and it's not that hard to read. There's almost
> certainly a C/C++/ObjC expert around to help out. There's almost
> certainly Cython experts to help out, too. There is almost certainly
> *not* an expert in Cython-generated C code that is hard to read.
>
You've had to touch the C/C++/ObjC because that's the only source that
exists; in this case that's the C *is* the implementation of the wrapper.
 If we go Cython, the cython source is all that is maintained. It may be
useful to glance at generated code, but no-one should be tweaking it by
hand--the Cython source, and only the Cython source, represents the
implementation of the wrapper.
Ryan
-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

On Mon, Dec 3, 2012 at 12:12 PM, Chris Barker - NOAA Federal
<chr...@no...> wrote:
> generated code is ugly and hard to maintain, it is not designed to be
> human-readable, and we wouldn't get the advantages of bug-fixes
> further development in Cython.
As far as I'm concerned, this is an argument against Cython.
I've had to touch the C/C++/ObjC codebase. It was not automatically
generated by Cython and it's not that hard to read. There's almost
certainly a C/C++/ObjC expert around to help out. There's almost
certainly Cython experts to help out, too. There is almost certainly
*not* an expert in Cython-generated C code that is hard to read.
I vote raw Python/C API. Managing reference counters is not the
mundane task pythonistas make it out to be, in my opinion. If you know
ObjC, you've had to do your own reference counting. If you know C,
you've had to do your own memory management. If you know C++, you've
had to do your own new/delete (or destructor) management. I agree not
having to worry about reference counting is nice positive, but I don't
think it outweighs the negatives.
It seems to me that Cython is a 'middle-man' tool, with the added
downside of hard-to-maintain under-code.
-- 
Damon McDougall
http://www.damon-is-a-geek.com
Institute for Computational Engineering Sciences
201 E. 24th St.
Stop C0200
The University of Texas at Austin
Austin, TX 78712-1229

On Tue, Dec 4, 2012 at 1:10 PM, Michael Droettboom <md...@st...> wrote:
> I think I see what's happened. I accidentally committed a #define in
> there when I was experimenting last week with removing deprecated Numpy
> APIs. It didn't cause things to break for me, but it looks like it could
> break things for more recent Numpy's. I've just gone ahead and reverted my
> change. Let me know if that fixes things for you when you get a chance.
>
> Cheers,
> Mike
>
>
Looks like that was the problem. The build is now successful.
Thanks!
Ben Root

I think I see what's happened. I accidentally committed a #define in 
there when I was experimenting last week with removing deprecated Numpy 
APIs. It didn't cause things to break for me, but it looks like it 
could break things for more recent Numpy's. I've just gone ahead and 
reverted my change. Let me know if that fixes things for you when you 
get a chance.
Cheers,
Mike
On 12/04/2012 10:50 AM, Benjamin Root wrote:
>
>
> On Tue, Dec 4, 2012 at 10:43 AM, Michael Droettboom <md...@st... 
> <mailto:md...@st...>> wrote:
>
> It looks like we're using the "old" Numpy API there. Did you
> recently update Numpy by any chance? I hadn't realised these APIs
> had been turned off yet, but maybe they are in git master. In any
> event, we should update these to the new APIs (NPY_UBYTE instead
> of PyArray_UBYTE etc.).
>
> Cheers,
> Mike
>
>
> Not since Nov. 5th (which was a fix for a bug I reported in numpy 
> master. So, I was using numpy 1.8.0 dev branch.
>
> Cheers!
> Ben Root
>

On Tue, Dec 4, 2012 at 10:43 AM, Michael Droettboom <md...@st...> wrote:
> It looks like we're using the "old" Numpy API there. Did you recently
> update Numpy by any chance? I hadn't realised these APIs had been turned
> off yet, but maybe they are in git master. In any event, we should update
> these to the new APIs (NPY_UBYTE instead of PyArray_UBYTE etc.).
>
> Cheers,
> Mike
>
>
Not since Nov. 5th (which was a fix for a bug I reported in numpy master.
So, I was using numpy 1.8.0 dev branch.
Cheers!
Ben Root

It looks like we're using the "old" Numpy API there. Did you recently 
update Numpy by any chance? I hadn't realised these APIs had been 
turned off yet, but maybe they are in git master. In any event, we 
should update these to the new APIs (NPY_UBYTE instead of PyArray_UBYTE 
etc.).
Cheers,
Mike
On 12/04/2012 09:46 AM, Benjamin Root wrote:
> I can't seem to build v1.2.x branch right now on CentOS6. This has 
> not been a problem before. I get the following error message while 
> trying to build the freetype2 stuff:
>
> creating build/temp.linux-x86_64-2.7/CXX
> gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall 
> -fPIC -DPY_ARRAY_UNIQUE_SYMBOL=MPL_ARRAY_API -DPYCXX_ISO_CPP_LIB=1 
> -I/usr/local/include -I/usr/include 
> -I/nas/home/broot/centos6/lib/python2.7/site-packages/numpy/core/include 
> -I/usr/include/freetype2 -I/usr/local/include -I/usr/include -I. 
> -I/home/broot/.local_centos6/include/python2.7 -c src/ft2font.cpp -o 
> build/temp.linux-x86_64-2.7/src/ft2font.o
> src/ft2font.cpp: In member function 'Py::Object 
> FT2Image::py_as_array(const Py::Tuple&)':
> src/ft2font.cpp:388: error: 'PyArray_UBYTE' was not declared in this scope
> src/ft2font.cpp: In member function 'Py::Object FT2Font::get_path()':
> src/ft2font.cpp:626: error: 'PyArray_DOUBLE' was not declared in this 
> scope
> src/ft2font.cpp:632: error: 'PyArray_UINT8' was not declared in this scope
> error: command 'gcc' failed with exit status 1
>
>
> ------------------------------------------------------------------------------
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
> Remotely access PCs and mobile devices and provide instant support
> Improve your efficiency, and focus on delivering more value-add services
> Discover what IT Professionals Know. Rescue delivers
> http://p.sf.net/sfu/logmein_12329d2d
>
>
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

I can't seem to build v1.2.x branch right now on CentOS6. This has not
been a problem before. I get the following error message while trying to
build the freetype2 stuff:
creating build/temp.linux-x86_64-2.7/CXX
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall
-fPIC -DPY_ARRAY_UNIQUE_SYMBOL=MPL_ARRAY_API -DPYCXX_ISO_CPP_LIB=1
-I/usr/local/include -I/usr/include
-I/nas/home/broot/centos6/lib/python2.7/site-packages/numpy/core/include
-I/usr/include/freetype2 -I/usr/local/include -I/usr/include -I.
-I/home/broot/.local_centos6/include/python2.7 -c src/ft2font.cpp -o
build/temp.linux-x86_64-2.7/src/ft2font.o
src/ft2font.cpp: In member function ‘Py::Object FT2Image::py_as_array(const
Py::Tuple&)’:
src/ft2font.cpp:388: error: ‘PyArray_UBYTE’ was not declared in this scope
src/ft2font.cpp: In member function ‘Py::Object FT2Font::get_path()’:
src/ft2font.cpp:626: error: ‘PyArray_DOUBLE’ was not declared in this scope
src/ft2font.cpp:632: error: ‘PyArray_UINT8’ was not declared in this scope
error: command 'gcc' failed with exit status 1

Also -- this feedback is really helpful when writing some comments in 
the wrappers as to why certain things are the way they are... I'll make 
sure to include rationales for raw file fast path and the need to open 
the files on the Python side.
Mike
On 12/04/2012 08:45 AM, Michael Droettboom wrote:
> On 12/03/2012 08:01 PM, Chris Barker - NOAA Federal wrote:
>> On Mon, Dec 3, 2012 at 4:16 PM, Nathaniel Smith <nj...@po...> wrote:
>>
>>> Yeah, this is a general problem with the Python file API, trying to
>>> hook it up to stdio is not at all an easy thing. A better version of
>>> this code would skip that altogether like:
>>>
>>> cdef void write_to_pyfile(png_structp s, png_bytep data, png_size_t count):
>>> fobj = <object>png_get_io_ptr(s)
>>> pydata = PyString_FromStringAndSize(data, count)
>>> fobj.write(pydata)
>> Good point -- not at all Cython-specific, but do you need libpng (or
>> whatever) to write to the file? can you just get a buffer with the
>> encoded data and write it on the Python side? Particularly if the user
>> wants to pass in an open file object. This might be a better API for
>> folks that might want stream an image right through a web app, too.
> You need to support both: raw C FILE objects for speed, and writing to a
> Python file-like object for flexibility. The code in master already
> does this (albeit with PyCXX), and the code on my "No CXX" branch does
> this as well with Cython.
>> As a lot of Python APIs take either a file name or a file-like object,
>> perhaps it would make sense to push that distinction down to the
>> Cython level:
>> -- if it's a filename, open it with raw C
> Unfortunately, as stated in detail in my last e-mail, that doesn't work
> with Unicode paths.
>
>> -- if it's a file-like object, have libpng write to a buffer (bytes
>> object) , and pass that to the file-like object in Python
> libpng does one better and allows us to stream directly to a callback
> which can then write to a Python object. This prevents double
> allocation of memory.
>
>> anyway, not really a Cython issue, but that second object sure would
>> be easy on Cython....
>>
> Yeah -- once I figured out how to make a real C callback function from
> Cython, the contents of the callback function itself is pretty easy to
> write.
>
> Mike
>
> ------------------------------------------------------------------------------
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
> Remotely access PCs and mobile devices and provide instant support
> Improve your efficiency, and focus on delivering more value-add services
> Discover what IT Professionals Know. Rescue delivers
> http://p.sf.net/sfu/logmein_12329d2d
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

On 12/03/2012 08:01 PM, Chris Barker - NOAA Federal wrote:
> On Mon, Dec 3, 2012 at 4:16 PM, Nathaniel Smith <nj...@po...> wrote:
>
>> Yeah, this is a general problem with the Python file API, trying to
>> hook it up to stdio is not at all an easy thing. A better version of
>> this code would skip that altogether like:
>>
>> cdef void write_to_pyfile(png_structp s, png_bytep data, png_size_t count):
>> fobj = <object>png_get_io_ptr(s)
>> pydata = PyString_FromStringAndSize(data, count)
>> fobj.write(pydata)
> Good point -- not at all Cython-specific, but do you need libpng (or
> whatever) to write to the file? can you just get a buffer with the
> encoded data and write it on the Python side? Particularly if the user
> wants to pass in an open file object. This might be a better API for
> folks that might want stream an image right through a web app, too.
You need to support both: raw C FILE objects for speed, and writing to a 
Python file-like object for flexibility. The code in master already 
does this (albeit with PyCXX), and the code on my "No CXX" branch does 
this as well with Cython.
>
> As a lot of Python APIs take either a file name or a file-like object,
> perhaps it would make sense to push that distinction down to the
> Cython level:
> -- if it's a filename, open it with raw C
Unfortunately, as stated in detail in my last e-mail, that doesn't work 
with Unicode paths.
> -- if it's a file-like object, have libpng write to a buffer (bytes
> object) , and pass that to the file-like object in Python
libpng does one better and allows us to stream directly to a callback 
which can then write to a Python object. This prevents double 
allocation of memory.
>
> anyway, not really a Cython issue, but that second object sure would
> be easy on Cython....
>
Yeah -- once I figured out how to make a real C callback function from 
Cython, the contents of the callback function itself is pretty easy to 
write.
Mike

On 12/03/2012 07:16 PM, Nathaniel Smith wrote:
> On Mon, Dec 3, 2012 at 11:50 PM, Chris Barker - NOAA Federal
> <chr...@no...> wrote:
>> On Mon, Dec 3, 2012 at 2:21 PM, Nathaniel Smith <nj...@po...> wrote:
>>> For the file handle, I would just write
>>>
>>> cdef FILE *fp = fdopen(file_obj.fileno(), "w")
>>>
>>> and be done with it. This will work with any version of Python etc.
>> yeah, that makes sense -- though what if you want to be able to
>> read_to/write_from a file that is already open, and in the middle of
>> the file somewhere -- would that work?
>>
>> I just posted a question to the Cython list, and indeed, it looks like
>> there is no easy answer to the file issue.
> Yeah, this is a general problem with the Python file API, trying to
> hook it up to stdio is not at all an easy thing. A better version of
> this code would skip that altogether like:
>
> cdef void write_to_pyfile(png_structp s, png_bytep data, png_size_t count):
> fobj = <object>png_get_io_ptr(s)
> pydata = PyString_FromStringAndSize(data, count)
> fobj.write(pydata)
>
> cdef void flush_pyfile(png_structp s):
> # Not sure if this is even needed
> fobj = <object>png_get_io_ptr(s)
> fobj.flush()
>
> # in write_png:
> write_png_c(<png_byte*>pix_buffer, width, height,
> NULL, <void*>file_obj, write_to_pyfile, flush_pyfile, dpi)
This is what my original version already does in the event that the 
file_obj is not a "real" file. In practice, you need to support both 
methods -- the callback approach is many times slower than writing 
directly to a regular old FILE object, because there is overhead both at 
the libpng and Python level, and there's no way to select a good buffer 
size.
>
> But this is a separate issue :-) (and needs further fiddling to make
> exception handling work).
>
> Or if you're only going to work on real OS-level file objects anyway,
> you might as well just accept a filename as a string and fopen() it
> locally. Having Python do the fopen just makes your life harder for no
> reason.
There's actually a very good reason. It is difficult to deal with 
Unicode in file paths from C in a portable way. On Windows, for 
example, if the user's name contains non-ascii characters, you can't 
write to the home directory using fopen, etc. It's doable with some 
care by using platform-specific C APIs etc., but CPython has already 
done all of the hard work for us, so it's easiest just to leverage that 
by opening the file from Python.
Mike

On 12/03/2012 07:00 PM, Chris Barker - NOAA Federal wrote:
> On Mon, Dec 3, 2012 at 12:24 PM, Chris Barker - NOAA Federal
> <chr...@no...> wrote:
>
>>>>> but some of that complexity could be reduced by using Numpy arrays in place
>>> It would at least make this a more fair comparison to have the Cython
>>> code as Cythonic as possible. However, I couldn't find any ways around
>>> using these particular APIs -- other than the Numpy stuff which probably
>>> does have a more elegant solution in the form of Cython arrays and
>>> memory views.
> OK -- so I poked at it, and this is my (very untested) version of
> write_png (I left out the py3 stuff, though it does look like it may
> be required for file handling...
>
> Letting Cython unpack the numpy array is the real win. Maybe having it
> this simple won't work for MPL, but this is what my code tends to look
> like.
>
>
> def write_png(cnp.ndarray[cnp.uint32, ndim=2, mode="c" ] buff not None,
> file_obj,
> double dpi=0.0):
>
> cdef png_uint_32 width = buff.size[0]
> cdef png_uint_32 height = buff.size[1]
>
> if PyFile_CheckExact(file_obj):
> cdef FILE *fp = fdopen(file_obj.fileno(), "w")
> fp = PyFile_AsFile(file_obj)
> write_png_c(buff[0,0], width, height, fp,
> NULL, NULL, NULL, dpi)
> return
> else:
> raise TypeError("write_png only works with real PyFileObject")
>
>
> NOTE: that could be:
>
> cnp.ndarray[cnp.uint8, ndim=3, mode="c" ]
>
> I'm not sure how MPL stores image buffers.
>
> or you could accept any object, then call:
>
> np.view()
The buffer comes in both ways, so the latter solution seems like the 
thing to do.
Thanks for working this through. This sort of thing is very helpful.
We can also, of course, maintain the existing code that allows writing 
to an arbitrary file-like object, but this fast path (where it is a 
"real" file) is very important. It's significantly faster than calling 
methods on Python objects.
Mike

On 2012年12月03日 4:54 AM, Michael Droettboom wrote:
> I think Cython is well suited to writing new algorithmic code to speed
> up hot spots in Python code. I don't think it's as well suited as glue
> between C and Python -- that was not a main goal of the original Pyrex
> project, IIRC. It feels kind of tacked on and not a very good fit to
> the problem.
Not entirely relevant to the PyCXX discussion, but to avoid misleading 
others reading this discussion, I must strongly disagree with your 
assertion about Cython's usefulness for wrapping C libraries or small 
chunks of C. I think this has always been a primary function of Cython 
and Pyrex, as far back as I have been aware of them. I wrote the raw 
interface to our contouring code, and I have written cython interfaces 
to various chunks of C outside of mpl; and cython makes it much easier 
for a non-professional programmer such as myself. So I am not arguing 
that Cython should be the choice for removing PyCXX, but for 
non-wizards, it can work very well as glue. It is much more 
approachable than any alternative of which I am aware. For Fortran, of 
course, f2py plays this glue code generation role.
Eric

On Mon, Dec 3, 2012 at 4:16 PM, Nathaniel Smith <nj...@po...> wrote:
> Yeah, this is a general problem with the Python file API, trying to
> hook it up to stdio is not at all an easy thing. A better version of
> this code would skip that altogether like:
>
> cdef void write_to_pyfile(png_structp s, png_bytep data, png_size_t count):
> fobj = <object>png_get_io_ptr(s)
> pydata = PyString_FromStringAndSize(data, count)
> fobj.write(pydata)
Good point -- not at all Cython-specific, but do you need libpng (or
whatever) to write to the file? can you just get a buffer with the
encoded data and write it on the Python side? Particularly if the user
wants to pass in an open file object. This might be a better API for
folks that might want stream an image right through a web app, too.
As a lot of Python APIs take either a file name or a file-like object,
perhaps it would make sense to push that distinction down to the
Cython level:
 -- if it's a filename, open it with raw C
 -- if it's a file-like object, have libpng write to a buffer (bytes
object) , and pass that to the file-like object in Python
anyway, not really a Cython issue, but that second object sure would
be easy on Cython....
-Chris
-- 
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chr...@no...

On Mon, Dec 3, 2012 at 11:50 PM, Chris Barker - NOAA Federal
<chr...@no...> wrote:
> On Mon, Dec 3, 2012 at 2:21 PM, Nathaniel Smith <nj...@po...> wrote:
>> For the file handle, I would just write
>>
>> cdef FILE *fp = fdopen(file_obj.fileno(), "w")
>>
>> and be done with it. This will work with any version of Python etc.
>
> yeah, that makes sense -- though what if you want to be able to
> read_to/write_from a file that is already open, and in the middle of
> the file somewhere -- would that work?
>
> I just posted a question to the Cython list, and indeed, it looks like
> there is no easy answer to the file issue.
Yeah, this is a general problem with the Python file API, trying to
hook it up to stdio is not at all an easy thing. A better version of
this code would skip that altogether like:
cdef void write_to_pyfile(png_structp s, png_bytep data, png_size_t count):
 fobj = <object>png_get_io_ptr(s)
 pydata = PyString_FromStringAndSize(data, count)
 fobj.write(pydata)
cdef void flush_pyfile(png_structp s):
 # Not sure if this is even needed
 fobj = <object>png_get_io_ptr(s)
 fobj.flush()
# in write_png:
write_png_c(<png_byte*>pix_buffer, width, height,
 NULL, <void*>file_obj, write_to_pyfile, flush_pyfile, dpi)
But this is a separate issue :-) (and needs further fiddling to make
exception handling work).
Or if you're only going to work on real OS-level file objects anyway,
you might as well just accept a filename as a string and fopen() it
locally. Having Python do the fopen just makes your life harder for no
reason.
-n

On Mon, Dec 3, 2012 at 12:24 PM, Chris Barker - NOAA Federal
<chr...@no...> wrote:
>>>> but some of that complexity could be reduced by using Numpy arrays in place
>> It would at least make this a more fair comparison to have the Cython
>> code as Cythonic as possible. However, I couldn't find any ways around
>> using these particular APIs -- other than the Numpy stuff which probably
>> does have a more elegant solution in the form of Cython arrays and
>> memory views.
OK -- so I poked at it, and this is my (very untested) version of
write_png (I left out the py3 stuff, though it does look like it may
be required for file handling...
Letting Cython unpack the numpy array is the real win. Maybe having it
this simple won't work for MPL, but this is what my code tends to look
like.
def write_png(cnp.ndarray[cnp.uint32, ndim=2, mode="c" ] buff not None,
 file_obj,
 double dpi=0.0):
 cdef png_uint_32 width = buff.size[0]
 cdef png_uint_32 height = buff.size[1]
 if PyFile_CheckExact(file_obj):
 cdef FILE *fp = fdopen(file_obj.fileno(), "w")
 fp = PyFile_AsFile(file_obj)
 write_png_c(buff[0,0], width, height, fp,
 NULL, NULL, NULL, dpi)
 return
 else:
 raise TypeError("write_png only works with real PyFileObject")
NOTE: that could be:
cnp.ndarray[cnp.uint8, ndim=3, mode="c" ]
I'm not sure how MPL stores image buffers.
or you could accept any object, then call:
np.view()
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chr...@no...