matplotlib-devel Mailing List for matplotlib

On Thursday 12 July 2007 5:57:27 pm Eric Firing wrote:
> Carl Worth wrote:
> [...]
>
> > But yeah, I also understand that there are licensing concerns.
> > Anything else? Are there cairo performance concerns? If so, I'd love
> > to hear about them so we can fix them.
>
> Backend cairo.ps took 0.87 minutes to complete
> Backend ps took 0.66 minutes to complete
>
> So the mpl Cairo ps output is also relatively slow. I don't know
> whether this is inherent in Cairo, or whether it could be optimized away
> in the mpl Cairo backend. And, it may be caused by a few outliers
> among the examples; I have not tried to compare times for individual plots.
Probably mpl's cairo backend can be optimized. The PS backend is heavily 
optimized, making use of the newer API to draw lines and markers 
(RendererPS.draw_lines and draw_markers).

Darren Dale wrote:
> On Thursday 12 July 2007 05:11:31 pm Carl Worth wrote:
>> On 2007年7月12日 10:31:19 -1000, Eric Firing wrote:
>>> if only cairo would make eps files
>> Isn't EPS a trivially change compared to PS? Something like a modified
>> header and the addition of bounding-box information?
> 
> The following is listed in the Encapsulated Postscript File Format 
> Specification, 
> http://partners.adobe.com/public/developer/en/ps/5002.EPSF_Spec.pdf:
> 
> "There are some PostScript language operators plus statusdict and userdict 
> operators that are intended for system-level jobs or page descriptions that 
> are not appropriate in an EPS file. In addition to all operators in statusdict 
> and the operators in userdict for establishing an imageable area, the 
> following operators must not be used in an EPS file:
> banddevice exitserver initmatrix setshared
> clear framedevice quit startjob
> cleardictstack grestoreall renderbands
> copypage initclip setglobal
> erasepage initgraphics setpagedevice
Aha! Cairo ps output is full of "initclip" commands, so it can't be 
converted to eps without substantial reworking.
This is a big limitation. To put a figure in a LaTeX document, for 
example, eps usually is what you need. So until Cairo solves this 
problem and provides eps, it is locked out of a large domain of graphics 
use.
Eric

Carl Worth wrote:
[...]
> But yeah, I also understand that there are licensing concerns.
> Anything else? Are there cairo performance concerns? If so, I'd love
> to hear about them so we can fix them.
Backend cairo.ps took 0.87 minutes to complete
Backend ps took 0.66 minutes to complete
So the mpl Cairo ps output is also relatively slow. I don't know 
whether this is inherent in Cairo, or whether it could be optimized away 
in the mpl Cairo backend. And, it may be caused by a few outliers 
among the examples; I have not tried to compare times for individual plots.
Eric

Carl Worth wrote:
> On 2007年7月12日 10:31:19 -1000, Eric Firing wrote:
>> if only cairo would make eps files
> 
> Isn't EPS a trivially change compared to PS? Something like a modified
> header and the addition of bounding-box information?
Carl,
My knowledge of graphics topics is sketchy, but yes, in many cases the 
difference between PS and EPS is just a few lines. This is true for 
Matlab files--ps and eps are identical except for something like 3 
lines. This works *provided* the ps file doesn't have anything in it 
that is not allowed in eps. I don't recall the constraints, apart from 
the file being single-page, but I know there are some. I suspect this 
is not a problem for cairo output, but I haven't tried to check.
> 
> It sounds like something that should be really easy to change in
> cairo.
Yes, so much so that I don't understand why it is not already done. (I 
know you are a cairo dev--is there not a postscript specialist in the 
cairo dev community?)
> 
> Maybe you could take a cairo-generated .ps file and manually munge it
> into the .eps that you'd like? If you did that, I'd be glad to write
> the necessary cairo code to get the same result, (and come up with
> whatever tiny new API is necessary to instruct cairo to do that).
If no one else steps forward, I can try to look at it fairly soon--but I 
am really not the right person to do it.
> 
> As for the rest of what you say. From my point of view, yes, using
> cairo and its multi-backend capabilities seems to make a fair amount
> more sense than inventing a new system with multiple backends. (Not a
> criticism against the original MPL stuff---cairo 1.0 has only existed
> since August 2005).
> 
> But yeah, I also understand that there are licensing concerns.
> Anything else? Are there cairo performance concerns? If so, I'd love
> to hear about them so we can fix them.
Well, I think that for screen display and for png generation, Agg is 
significantly faster. Backend_driver.py gives
Backend agg took 0.76 minutes to complete
Backend cairo.png took 0.95 minutes to complete
Backend template took 0.46 minutes to complete
I haven't tried to test it, but I suspect that for some types of plots 
the optimizations in Agg and the agg backend will make a much bigger 
difference than the averages above would indicate.
Eric
> And even if you don't use cairo directly, I will offer the suggestion
> of its drawing model as being very good, (using source and mask
> pattern objects and just 5 different drawing operations: stroke, fill,
> paint, show_text/glyphs, and mask).
> 
> Whatever you decide to go with, have fun!
> 
> -Carl

On Thursday 12 July 2007 05:11:31 pm Carl Worth wrote:
> On 2007年7月12日 10:31:19 -1000, Eric Firing wrote:
> > if only cairo would make eps files
>
> Isn't EPS a trivially change compared to PS? Something like a modified
> header and the addition of bounding-box information?
The following is listed in the Encapsulated Postscript File Format=20
Specification, =20
http://partners.adobe.com/public/developer/en/ps/5002.EPSF_Spec.pdf:
"There are some PostScript language operators plus statusdict and userdict=
=20
operators that are intended for system-level jobs or page descriptions that=
=20
are not appropriate in an EPS =EF=AC=81le. In addition to all operators in =
statusdict=20
and the operators in userdict for establishing an imageable area, the=20
following operators must not be used in an EPS =EF=AC=81le:
 banddevice exitserver initmatrix setshared
 clear framedevice quit startjob
 cleardictstack grestoreall renderbands
 copypage initclip setglobal
 erasepage initgraphics setpagedevice
If used properly, the following operators are allowed in an EPS =EF=AC=81le=
=2E However,=20
use of any of these must comply with the rules in Appendix I of the=20
PostScript Language Reference Manual, Second Edition. Improper use can caus=
e=20
unpredictable results.
 nulldevice sethalftone setscreen unde=EF=AC=81nef=
ont
 setgstate setmatrix settransfer
"

On 2007年7月12日 10:31:19 -1000, Eric Firing wrote:
> if only cairo would make eps files
Isn't EPS a trivially change compared to PS? Something like a modified
header and the addition of bounding-box information?
It sounds like something that should be really easy to change in
cairo.
Maybe you could take a cairo-generated .ps file and manually munge it
into the .eps that you'd like? If you did that, I'd be glad to write
the necessary cairo code to get the same result, (and come up with
whatever tiny new API is necessary to instruct cairo to do that).
As for the rest of what you say. From my point of view, yes, using
cairo and its multi-backend capabilities seems to make a fair amount
more sense than inventing a new system with multiple backends. (Not a
criticism against the original MPL stuff---cairo 1.0 has only existed
since August 2005).
But yeah, I also understand that there are licensing concerns.
Anything else? Are there cairo performance concerns? If so, I'd love
to hear about them so we can fix them.
And even if you don't use cairo directly, I will offer the suggestion
of its drawing model as being very good, (using source and mask
pattern objects and just 5 different drawing operations: stroke, fill,
paint, show_text/glyphs, and mask).
Whatever you decide to go with, have fun!
-Carl

John Hunter wrote:
> On 7/12/07, Eric Firing <ef...@ha...> wrote:
>> John Hunter wrote:
> 
>> Do all the backend devices or libraries we *need* now support compound
>> paths (1) at all, and (2) in a consistent enough way to solve the
>> contouring problem as well as to make something like polygon collections
>> more efficient?
>>
>> I guess this is really part of a larger question about the degree to
>> which the various graphics drawing models have converged with a rich
>> intersection of capabilities, including arbitrary text rotation, precise
>> text positioning, arcs, bezier curves, etc.
> 
> Yes, I think they do, though we will encounter some differences
> between spline handling that might make some of the path stuff tricky.
> We need agg, cairo, svg, ps and pdf, all of which use vector
> graphics. We would need to sacrifice native GDK and WX, which would
 From a purely technical standpoint--getting the desired output 
formats--perhaps cairo could substitute for pdf and svg, and also for ps 
if only cairo would make eps files. So, even if eventually you want to 
restore native pdf, svg, and ps backends, it could be done at a later 
stage than the initial chainsaw sculpture.
> inconvenience some but these users could certainly get by with GTK* or
> WX*. One reason a few still use native GTK is because it is faster,
> but this would likely disappear with some intelligent design. One
> might envision just three or four fundamental objects which the
> backends need to understand.
> 
> Path
> CompoundPath
> Image
> Text
> 
> and maybe use a stateful graphics context (linewidth, facecolor, etc...)
> 
> numpify everything, traitify everything and rebuild the
> transformation architecture from scratch. Borrowing some language
> from ipython, this would truly be a "chainsaw" branch.
> 
> I'm halfway inclined to take a crack at it......
Great! I wonder whether your company could be convinced to give you 
some slack for that... It might be worth a bit of thought and planning 
to see how parts of the work could be divided up so that you would not 
have to do everything yourself.
To minimize the interval during which a separate branch would be needed, 
maybe some initial traitification should be done without a fork.
Eric
> 
> JDH

Michael Droettboom wrote:
> cntr.c is creating the inner and outer 
> polygons correctly, just not connecting them with a slit, right?
> 
To answer my own question, unfortunately, this isn't always the case. 
There is also a bug where if a masked region is inside of the *inner* 
polygon of a contour, the inner contour is incorrect. So, other 
advantages aside, adding support for compound polygons won't magically 
fix any of the contour problems.
Cheers,
Mike

On 7/12/07, Carl Worth <cw...@cw...> wrote:
> What's the "GTK drawing model" here that is problematic? Is that
> pre-cairo stuff? And can using cairo directly help at all?
Yes, this is pre cairo and unfortunately, no I don't think cairo can
help. When I wrote MPL, it was GTK only and I used GDK for rendering.
 The backend canvas was a gtk.DrawingArea calls and the renderer was a
gdk.Drawable which defined methods like draw_rectangle and
draw_polygon
 http://www.pygtk.org/pygtk2reference/class-gdkdrawable.html
As I added other backends, backend_ps first, I abstracted the
gtk.DrawingArea, gdk.Drawable and gdk.GC to the FigureCanvas, Renderer
and GraphicsContext respectively which are are core backend base
classes, and then for each backend did a concrete implemention of the
gdk.Drawable API, etc. This was back in 2001. If I were smarter or
had done more research at the time, I would have chosen a different
drawing mode, eg DisplayPDF or something like that, but I guess I was
too busy coding .... (reminds me of a joke we used to tell in the lab
"Why spend and hour in the library when you can get the same result in
the lab in a year?").
So the fact that GTK now uses Cairo doesn't help us too much, since
all of our backends implement the gdk.Drawable API more or less. What
we are discussing here is a clean room reimplementation of our backend
API and drawing model.
JDH

On 7/12/07, Eric Firing <ef...@ha...> wrote:
> John Hunter wrote:
> Do all the backend devices or libraries we *need* now support compound
> paths (1) at all, and (2) in a consistent enough way to solve the
> contouring problem as well as to make something like polygon collections
> more efficient?
>
> I guess this is really part of a larger question about the degree to
> which the various graphics drawing models have converged with a rich
> intersection of capabilities, including arbitrary text rotation, precise
> text positioning, arcs, bezier curves, etc.
Yes, I think they do, though we will encounter some differences
between spline handling that might make some of the path stuff tricky.
 We need agg, cairo, svg, ps and pdf, all of which use vector
graphics. We would need to sacrifice native GDK and WX, which would
inconvenience some but these users could certainly get by with GTK* or
WX*. One reason a few still use native GTK is because it is faster,
but this would likely disappear with some intelligent design. One
might envision just three or four fundamental objects which the
backends need to understand.
 Path
 CompoundPath
 Image
 Text
and maybe use a stateful graphics context (linewidth, facecolor, etc...)
numpify everything, traitify everything and rebuild the
transformation architecture from scratch. Borrowing some language
from ipython, this would truly be a "chainsaw" branch.
I'm halfway inclined to take a crack at it......
JDH

On 2007年7月12日 09:29:06 -1000, Eric Firing wrote:
> > So one reason why the backends are a bit of a kludge is because we
> > have tried to throw some extra stuff into the GTK drawing model as an
> > afterthough. We could redo all the collection stuff w/ compounds
> > paths.
What's the "GTK drawing model" here that is problematic? Is that
pre-cairo stuff? And can using cairo directly help at all?
-Carl

John Hunter wrote:
[...]
> So one reason why the backends are a bit of a kludge is because we
> have tried to throw some extra stuff into the GTK drawing model as an
> afterthough. We could redo all the collection stuff w/ compounds
> paths.
John,
Do all the backend devices or libraries we *need* now support compound 
paths (1) at all, and (2) in a consistent enough way to solve the 
contouring problem as well as to make something like polygon collections 
more efficient?
I guess this is really part of a larger question about the degree to 
which the various graphics drawing models have converged with a rich 
intersection of capabilities, including arbitrary text rotation, precise 
text positioning, arcs, bezier curves, etc.
Eric

John Hunter wrote:
> On 7/12/07, Michael Droettboom <md...@st...> wrote:
> 
>> It looks like that the backend API isn't really set up to do compound
>> paths. The PolygonCollection class appears to be just a list of simple
>> polygons, rather than something that could be used for compound paths.
>> (Correct me if I'm wrong.) Is the only reason for wanting branch cuts
>> in contours because matplotlib doesn't support compound paths? Is there
>> a reason matplotlib doesn't support compound paths? I suspect you guys
>> have been down that "path" [haha] already. I would anticipate problems
>> if different backends have different winding rules, but perhaps there
>> are some ways to around that? This could be useful in general in
>> matplotlib...
> 
> Actually we haven't. In fact, paths have been a fairly recent
> addition to mpl. The drawing model is based on GTK, which did not
> have paths at the time. The renderer API is basically the GTK API,
> and all the other backends simply implement the GTK drawing model. In
> the early days, there was only a GTK backend....
> 
> So one reason why the backends are a bit of a kludge is because we
> have tried to throw some extra stuff into the GTK drawing model as an
> afterthough. We could redo all the collection stuff w/ compounds
> paths.
This is the sort of thing I was getting at by suggesting that if a big 
project is possible, rethinking the drawing model and refactoring 
accordingly would be a candidate. I think there is quite a bit of scope 
for fundamentally improving mpl internals (and maybe even performance), 
and reducing future maintenance costs, by doing this. It is at the same 
level as, and maybe complementary to, redoing the transforms framework.
With respect to contouring, yes, a branch cut is a kludge--a way to fake 
a compound path. It would be much cleaner if the cuts could be 
eliminated. With a little extra record-keeping to mark the parts of 
paths that bound masked regions and boundaries, this would also solve 
the problem that the polygon boundaries don't always coincide with the 
contour lines that are calculated with cntr.c in non-filled mode.
Eric
> 
> JDH

On 7/12/07, Michael Droettboom <md...@st...> wrote:
> It looks like that the backend API isn't really set up to do compound
> paths. The PolygonCollection class appears to be just a list of simple
> polygons, rather than something that could be used for compound paths.
> (Correct me if I'm wrong.) Is the only reason for wanting branch cuts
> in contours because matplotlib doesn't support compound paths? Is there
> a reason matplotlib doesn't support compound paths? I suspect you guys
> have been down that "path" [haha] already. I would anticipate problems
> if different backends have different winding rules, but perhaps there
> are some ways to around that? This could be useful in general in
> matplotlib...
Actually we haven't. In fact, paths have been a fairly recent
addition to mpl. The drawing model is based on GTK, which did not
have paths at the time. The renderer API is basically the GTK API,
and all the other backends simply implement the GTK drawing model. In
the early days, there was only a GTK backend....
So one reason why the backends are a bit of a kludge is because we
have tried to throw some extra stuff into the GTK drawing model as an
afterthough. We could redo all the collection stuff w/ compounds
paths.
JDH

Eric,
Thanks for the scripts -- they make it quite clear what the problem is.
As you know, cntr.c is a fairly opaque chunk of code, and I'm not having 
much luck so far tracking down why the branch-cutting between the outer 
and inner polygons is not working.
In the meantime, I have some more general questions.
It looks like that the backend API isn't really set up to do compound 
paths. The PolygonCollection class appears to be just a list of simple 
polygons, rather than something that could be used for compound paths. 
(Correct me if I'm wrong.) Is the only reason for wanting branch cuts 
in contours because matplotlib doesn't support compound paths? Is there 
a reason matplotlib doesn't support compound paths? I suspect you guys 
have been down that "path" [haha] already. I would anticipate problems 
if different backends have different winding rules, but perhaps there 
are some ways to around that? This could be useful in general in 
matplotlib...
Another option is to remove the mask from the contours in a 
post-processing step (i.e. don't worry about trying to do it as part of 
the main two passes in cntr.c). The result may be a bit of a 
Frankenstein's monster, admittedly, but it is an option of last resort 
if we choose to take it. cntr.c is creating the inner and outer 
polygons correctly, just not connecting them with a slit, right?
In the meantime, I'll continue to look for the root cause.
Cheers,
Mike
Eric Firing wrote:
> Mike,
>
> The attached file masked_interior.py illustrates masking failure in a 
> very simple case; you can see masking working in the plot on the left, 
> where a contour intersects the masked region, but when that contour 
> level is removed the masked region is getting filled in.
>
> The file contourf_demo.py is slightly modified from the one in the mpl 
> examples directory, and shows a failure of masking in a more 
> complicated setting. The masked region at the lower-left corner is 
> correct, but the masked region in the middle of the plot is getting 
> filled with gray instead of being left blank.
>
> In cntr.c there is a function, print_Csite, that may be helpful for 
> debugging the simplest case, where the array size is not too large.
>
> Note that the code path for filled contours is quite different, and 
> more complicated, than for line contours--and in fact, even neglecting 
> branch cuts, the two code paths don't always yield the same contours.
>
> cntr.c is somewhat unusual among contour algorithms in that it works 
> with rectangles without subdividing them into triangles.
>
> Eric
> ------------------------------------------------------------------------
>
> #!/usr/bin/env python
> '''
> This is a very simple illustration of what is causing
> the problem with masked interior regions for filled contours.
> The exterior contour and the interior contour are being
> separated into two polygons instead of being joined by the
> branch cut.
> '''
> import sys
> from pylab import *
> import matplotlib.numerix.npyma as ma
> rc('figure', dpi=120)
>
> x = y = arange(5)
> X, Y = meshgrid(x, y)
> Z = Y
>
> ax = (0,4,0,4)
>
> badmask = zeros(shape(X))
>
> badmask[2, 2] = 1
>
> Z = ma.array(Z, mask=badmask)
> print Z
>
> subplot(2,1,1)
> CS = contourf(X, Y, Z, (-0.1, 0.8, 2.0, 6.0),
> colors=('r', 'g', 'b'))
>
> CS.collections[0].set_edgecolor('c')
> CS.collections[0].set_linewidth(6)
> CS.collections[1].set_edgecolor('y')
> CS.collections[1].set_linewidth(2)
> CS.collections[2].set_edgecolor('m')
> CS.collections[2].set_linewidth(4)
> title('Masked correctly')
>
> subplot(2,1,2)
> CS = contourf(X, Y, Z, (-0.1, 0.8, 6.0),
> colors=('r', 'b'))
>
> CS.collections[0].set_edgecolor('c')
> CS.collections[0].set_linewidth(6)
> CS.collections[1].set_edgecolor('y')
> CS.collections[1].set_linewidth(2)
> title('Wrong: masked region filled')
> print "len(CS.collections)", len(CS.collections)
>
> colls = CS.collections
>
> # We want to look at the actual polygons individually.
> for i, coll in enumerate(colls):
> print "level ", i
> for j, T in enumerate(coll._verts):
> print "polygon ", j
> print T
>
> # The second collection has two polygons, but should only have
> # one; the two should be joined by a branch cut.
>
> show()
>
> 
> ------------------------------------------------------------------------
>
> #!/usr/bin/env python
> from pylab import *
> import matplotlib.numerix.npyma as ma
> origin = 'lower'
> #origin = 'upper'
>
> test_masking = True # There is a bug in filled contour masking.
>
> if test_masking:
> # Use a coarse grid so only a few masked points are needed.
> delta = 0.5
> else:
> delta = 0.025
>
> x = y = arange(-3.0, 3.01, delta)
> X, Y = meshgrid(x, y)
> Z1 = bivariate_normal(X, Y, 1.0, 1.0, 0.0, 0.0)
> Z2 = bivariate_normal(X, Y, 1.5, 0.5, 1, 1)
> Z = 10 * (Z1 - Z2)
>
> # interior badmask doesn't work yet for filled contours
> if test_masking:
> badmask = zeros(shape(Z))
>
> badmask[5,5] = 1
> badmask[5,6] = 1
> Z[5,5] = 0
> Z[5,6] = 0
>
> badmask[0,0] = 1
> Z[0,0] = 0
> Z = ma.array(Z, mask=badmask)
>
> # We are using automatic selection of contour levels;
> # this is usually not such a good idea, because they don't
> # occur on nice boundaries, but we do it here for purposes
> # of illustration.
> CS = contourf(X, Y, Z, 10, # [-1, -0.1, 0, 0.1],
> #alpha=0.5,
> cmap=cm.bone,
> origin=origin)
>
> # Note that in the following, we explicitly pass in a subset of
> # the contour levels used for the filled contours. Alternatively,
> # We could pass in additional levels to provide extra resolution.
>
> CS2 = contour(X, Y, Z, CS.levels[::2],
> colors = 'r',
> origin=origin,
> hold='on')
>
> title('Nonsense')
> xlabel('word length anomaly')
> ylabel('sentence length anomaly')
>
> # Make a colorbar for the ContourSet returned by the contourf call.
> cbar = colorbar(CS)
> cbar.ax.set_ylabel('verbosity coefficient')
> # Add the contour line levels to the colorbar
> cbar.add_lines(CS2)
>
> figure()
>
> # Now make a contour plot with the levels specified,
> # and with the colormap generated automatically from a list
> # of colors.
> levels = [-2, -1.5, -1, -0.5, 0, 0.5, 1, 1.5]
> CS3 = contourf(X, Y, Z, levels,
> colors = ('r', 'g', 'b'),
> origin=origin)
>
> CS4 = contour(X, Y, Z, levels,
> colors = ('k',),
> linewidths = (3,),
> origin = origin)
> title('Listed colors')
> clabel(CS4, fmt = '%2.1f', colors = 'w', fontsize=14)
> colorbar(CS3)
>
> #savefig('contourf_demo')
> show()
>
> 

On 7/11/07, Darren Dale <dd...@co...> wrote:
> I thought they were going to include both serifs and sans-serifs, but I might
> be wrong. I'm on the list to receive the beta package when it is available,
> which could be any day now.
Definitely within 6 months, heh
JDH