matplotlib-devel Mailing List for matplotlib

Actually, it just occurred to me that you could use 'defs' and 'use' to 
define characters as paths and reuse them. And that's something that 
works today in virtually any SVG renderer.
It totally blitzes the logical representation of the text (i.e. 
searching and copying as text etc.), but if it was only done for math 
text, then that's probably not a big deal. IMHO, math text is where 
matching the exact font is most important, where a reasonable substitute 
is often not installed, and where searching doesn't make a lot of 
sense. In any case, editing the SVG in something like Inkscape will be 
harder, since the text has become raw paths.
Thanks to ft2font, it was fairly easy to get outlines of math characters 
embedded in the SVG file. I've put up an example mathtext_demo.py SVG here:
 ftp://ftp.stsci.edu/tmp/mdroe/mathtext_demo.svg
I'll commit this code once it's cleaned up and better tested etc. if 
it's agreed this is something we want to do. I think in any case this 
should be optional -- it gives the file some portability at the expense 
of editability.
Cheers,
Mike
Carl Worth wrote:
> On 2007年7月10日 13:43:49 -0400, Michael Droettboom wrote:
> 
>> "major SVG renderers" support seems to be the issue at first glance.
>> None of the big open source options -- Firefox, inkscape, rsvg -- seem
>> to support it.
>> 
>
> That's my understanding as well.
>
> 
>> Until there's something readily available to test with, there's probably
>> not much point.
>> 
>
> It's a classic chicken-and-egg. I know the cairo-svg maintainer is
> hoping to add SVGFont stuff to cairo's SVG output to try to boostrap
> past this problem.
> 

Jouni K. Seppänen wrote:
> "John Hunter" <jd...@gm...> writes:
>
> 
>> On 7/10/07, Michael Droettboom <md...@st...> wrote:
>> 
> I'm seeing a bug on OS X, whose file system is by default
> case-preserving but not case-sensitive:
>
> 458 -> fontdictObject = self.embedTTF(
> 459 filename, self.used_characters[filename])
>
> Here self.used_characters has a key starting with '/Users/jks/...' but
> filename is '/users/jks/...'. I'm not sure how to fix this cleanly. 
> I though os.path.normcase would help, but it doesn't:
>
> 
Thanks for that. The Ps backend has the same problem. I'll do a little 
research and see what a common solution to this might be.
Cheers,
Mike

"John Hunter" <jd...@gm...> writes:
> On 7/10/07, Michael Droettboom <md...@st...> wrote:
>
>> The comparison is just over the PDF files, old way (Truetype embedding)
>> vs. new way (Type 3 subsetting).
>
> This must be dominated by some weird outliers. I'm seeing great
> results with the canonical "simple_plot"
>
> -rw-r--r-- 1 johnh research 19352 Jul 10 12:11 new.ps
> -rw-r--r-- 1 johnh research 144227 Jul 10 12:11 old.ps
Probably the difference is just that you are comparing PS files while
Michael was comparing PDF files, where the internal gzip compression
reduces the file size. Even in PDF files you can get large gains if you
use several different fonts (which can easily happen with math formulas)
or you have very big fonts (such as some fonts that come with Mac OS X).
I'm seeing a bug on OS X, whose file system is by default
case-preserving but not case-sensitive:
458 -> fontdictObject = self.embedTTF(
459 filename, self.used_characters[filename])
Here self.used_characters has a key starting with '/Users/jks/...' but
filename is '/users/jks/...'. I'm not sure how to fix this cleanly. 
I though os.path.normcase would help, but it doesn't:
>>> normcase('/users'); normcase('/Users')
'/users'
'/Users'
> In any case, excellent work!
Yes, very good! This is a much-needed feature for the PS and PDF
backends.
-- 
Jouni K. Seppänen
http://www.iki.fi/jks

Michael Droettboom wrote:
> Eric Firing wrote:
>> I second that! I greatly appreciate your contributions, first in 
>> chasing down memory leaks and now in reducing file sizes by embedding 
>> fonts.
> It's been fun.
Mike,
Good--what's next? You're ready for more fun, I hope. If you are 
looking for brain-benders, I know of two bugs lurking deep in extension 
code (one in cntr.c, the other in the Agg quadmesh rendering) that have 
completely eluded me. There is also a bug in the Agg image rendering 
that shows up at high magnification. If you are interested in grand 
strategy, let's discuss what a suitable target might be--maybe some 
refactoring and consolidation of backend code to reduce duplication, and 
make optimizations applicable to all (referring to recent work by 
Allan). A related idea that has been languishing is to consolidate the 
various image-like functionality, again so as to make maximum use of the 
different optimizations and options that are now spread among different 
functions. John probably has better ideas about what to attack.
> 
> Now, Eric, I'm just waiting for you to tell me how this latest batch 
> reveals another bug on Ubuntu ;) (with all due respect to Ubuntu)
So, you use RHEL4 at work and ubuntu at home; has the former actually 
managed to put together a set of relatively bug-free versions of gui 
toolkits and other libraries? Or is it a choice between old bugs and 
newer bugs? Given all the different constantly-changing libraries, 
compilers, etc. involved in something like mpl, I am sometimes amazed 
that it works at all--and worried that at any moment it may cease to work.
Eric
> 
> Cheers,
> Mike

Darren Dale <dd...@co...> writes:
>> > svn: PROPFIND of '/svnroot/matplotlib/trunk/matplotlib': Could not
>> > resolve hostname `svn.sourceforge.net': Host not found
>>
>> I've been experiencing this since yesterday, and was waiting to see if
>> it clears up by today...
>
> I ended up checking out the entire repository again. It seems that the working 
> address is now matplotlib.svn.sourceforge.net/svn/matplotlib, instead of 
> svn.sourceforge.net/svn/matplotlib.
Yes, here is the announcement with some instructions from
sourceforge.net:
https://sourceforge.net/docs/E09#notice
-- 
Jouni K. Seppänen
http://www.iki.fi/jks

Eric Firing wrote:
> I second that! I greatly appreciate your contributions, first in 
> chasing down memory leaks and now in reducing file sizes by embedding 
> fonts.
It's been fun.
Now, Eric, I'm just waiting for you to tell me how this latest batch 
reveals another bug on Ubuntu ;) (with all due respect to Ubuntu)
Cheers,
Mike

John Hunter wrote:
> On 7/10/07, Michael Droettboom <md...@st...> wrote:
> 
>> The comparison is just over the PDF files, old way (Truetype embedding)
>> vs. new way (Type 3 subsetting).
[...]
> 
> In any case, excellent work!
> 
> JDH
> 
Mike,
I second that! I greatly appreciate your contributions, first in 
chasing down memory leaks and now in reducing file sizes by embedding fonts.
Eric

On 2007年7月10日 13:43:49 -0400, Michael Droettboom wrote:
> "major SVG renderers" support seems to be the issue at first glance.
> None of the big open source options -- Firefox, inkscape, rsvg -- seem
> to support it.
That's my understanding as well.
> Until there's something readily available to test with, there's probably
> not much point.
It's a classic chicken-and-egg. I know the cairo-svg maintainer is
hoping to add SVGFont stuff to cairo's SVG output to try to boostrap
past this problem.
-Carl

John Hunter wrote:
> On 7/10/07, Michael Droettboom <md...@st...> wrote:
>> Has anyone looked into embedding fonts in SVG files?
>
> It's definitely come up before. Paul Barret may have had a look at
> it. I'm not sure why we ended up not doing it. You can search the
> archives if you are interested. If it isn't too hard, and major SVG
> renderers support it, it seems like a great idea.
"major SVG renderers" support seems to be the issue at first glance. 
None of the big open source options -- Firefox, inkscape, rsvg -- seem 
to support it. Please let me know if I'm missing something. Firefox 
has even pushed SVG fonts off of the Firefox 3 timeline:
http://weblogs.mozillazine.org/tor/archives/2007/03/svg_priorities_in_firefox_3.html
Until there's something readily available to test with, there's probably 
not much point.
Cheers,
Mike

On 7/10/07, Michael Droettboom <md...@st...> wrote:
> Has anyone looked into embedding fonts in SVG files? That might
> alleviate some of the installation problems that were recently mentioned
> on the matplotlib-users list.
>
> The relevant spec:
>
> http://www.w3.org/TR/SVG11/fonts.html
>
> If there are no plans under way, I may have a crack at it, since I'm
> sort of on an "embedding fonts" kick lately anyway... ;)
It's definitely come up before. Paul Barret may have had a look at
it. I'm not sure why we ended up not doing it. You can search the
archives if you are interested. If it isn't too hard, and major SVG
renderers support it, it seems like a great idea.
JDH

On 7/10/07, Michael Droettboom <md...@st...> wrote:
> The comparison is just over the PDF files, old way (Truetype embedding)
> vs. new way (Type 3 subsetting).
> The ratios are different because I chose to highlight the examples that
> are quite "texty". That wasn't a deliberate attempt to mislead, it's
> just because this change is related to fonts. Most of the other
> examples use a single font, and the plotting content itself dominates
> file size.
This must be dominated by some weird outliers. I'm seeing great
results with the canonical "simple_plot"
johnh@flag:examples> python simple_plot.py -dPS
johnh@flag:examples> mv simple_plot.ps new.ps
johnh@flag:examples> PYTHONPATH=/my/old/site-packages python simple_plot.py -dPS
johnh@flag:examples> mv simple_plot.ps old.ps
johnh@flag:examples> ls -l old.ps new.ps
-rw-r--r-- 1 johnh research 19352 Jul 10 12:11 new.ps
-rw-r--r-- 1 johnh research 144227 Jul 10 12:11 old.ps
Though for some reason my 90.1 install is picking up Vera Serif and my
svn install is picking up Vera Sans., which is mysterious but
unrelated to your work
In any case, excellent work!
JDH

Has anyone looked into embedding fonts in SVG files? That might 
alleviate some of the installation problems that were recently mentioned 
on the matplotlib-users list.
The relevant spec:
http://www.w3.org/TR/SVG11/fonts.html
If there are no plans under way, I may have a crack at it, since I'm 
sort of on an "embedding fonts" kick lately anyway... ;)
Cheers,
Mike

John Hunter wrote:
> On 7/10/07, Michael Droettboom <md...@st...> wrote:
>>
>> fonts_demo_kw.py: 201744 -> 37326
>> mathtext_demo.py: 129306 -> 26179
>> unicode_demo.py: 45303 -> 20084
>> over all demos in backend_driver.py: 5856001 -> 3390460
>
> When you say "over all demos" do you mean just over the PS or PDF
> depending on which you are testing? I'm a bit confused because the
> single examples you show show between a 5 and 20 fold improvement, but
> the overall number is less than 2 fold. So I wonder if you are
> including the other backend driver PNG, SVG, etc... output.... Just
> curious.
The comparison is just over the PDF files, old way (Truetype embedding) 
vs. new way (Type 3 subsetting). 
The ratios are different because I chose to highlight the examples that 
are quite "texty". That wasn't a deliberate attempt to mislead, it's 
just because this change is related to fonts. Most of the other 
examples use a single font, and the plotting content itself dominates 
file size.
Cheers,
Mike

On 7/10/07, Michael Droettboom <md...@st...> wrote:
> I just committed changes that add TTF subsetting to the PDF backend. It
> is completely analogous to the font subsetting recently added to the PS
> backend.
>
> I have added a configuration option, pdf.fonttype, to choose either
> "Type3" or "Truetype" font output. This may be removed in the future
> once the "Type3" stuff has been sufficiently tested.
>
> Some results:
>
> fonts_demo_kw.py: 201744 -> 37326
> mathtext_demo.py: 129306 -> 26179
> unicode_demo.py: 45303 -> 20084
> over all demos in backend_driver.py: 5856001 -> 3390460
When you say "over all demos" do you mean just over the PS or PDF
depending on which you are testing? I'm a bit confused because the
single examples you show show between a 5 and 20 fold improvement, but
the overall number is less than 2 fold. So I wonder if you are
including the other backend driver PNG, SVG, etc... output.... Just
curious.
JDH

I just committed changes that add TTF subsetting to the PDF backend. It 
is completely analogous to the font subsetting recently added to the PS 
backend.
I have added a configuration option, pdf.fonttype, to choose either 
"Type3" or "Truetype" font output. This may be removed in the future 
once the "Type3" stuff has been sufficiently tested.
Some results:
fonts_demo_kw.py: 201744 -> 37326
mathtext_demo.py: 129306 -> 26179
unicode_demo.py: 45303 -> 20084
over all demos in backend_driver.py: 5856001 -> 3390460
The differences aren't as dramatic as with Postscript, but IMHO they are 
still large enough to be worthwhile.
Again, please help by testing with your own favorite PDF tools.
Gory details about composite characters follow -->
In this new code, composite characters (such as a character composed of 
a letter and an accent) aren't handled as they should be. According to 
the PDF spec, a PDF-1.2 (Acrobat 3.x) Type 3 font can reference other 
glyphs with the "Do" command, to avoid duplicating the components of a 
composite glyph. I was able to get this to work with Acrobat 7, but 
xpdf-3.0 and ggv 2.8.0 both choked on the file. Therefore, I decided to 
err on the side of compatibility by including each component of a 
composite character inline where it is used. This makes the PDF files 
larger than they would otherwise have to be. However, it should only be 
a real problem if a plot contains an inordinate amount of different 
accented characters.
Cheers,
Mike

John Hunter wrote:
> On 7/10/07, Darren Dale <dd...@co...> wrote:
>
> 
>> I ended up checking out the entire repository again. It seems that the working
>> address is now matplotlib.svn.sourceforge.net/svn/matplotlib, instead of
>> svn.sourceforge.net/svn/matplotlib.
>> 
>
>
> Nothing is working for me
>
> -bash-2.05b$ cd python/svn/
> -bash-2.05b$ svn co
> http://matplotlib.svn.sourceforge.net/svn/matplotlib
> --username=jdh2358 --password=mypasswd
> svn: PROPFIND request failed on '/svn/matplotlib'
> svn: PROPFIND of '/svn/matplotlib': 405 Method Not Allowed
> (http://matplotlib.svn.sourceforge.net)
> -bash-2.05b$ svn co
> https://matplotlib.svn.sourceforge.net/svn/matplotlib
> --username=jdh2358 --password=mypasswd
> svn: PROPFIND request failed on '/svn/matplotlib'
> svn: PROPFIND of '/svn/matplotlib': 405 Method Not Allowed
> (https://matplotlib.svn.sourceforge.net)
> -bash-2.05b$ svn co
> https://matplotlib.svn.sourceforge.net/svn/matplotlib svn: PROPFIND
> request failed on '/svn/matplotlib'
> svn: PROPFIND of '/svn/matplotlib': 405 Method Not Allowed
> (https://matplotlib.svn.sourceforge.net)
> -bash-2.05b$ svn co
> https://svn.sourceforge.net/svnroot/matplotlib/trunk matplotlib
> svn: PROPFIND request failed on '/svnroot/matplotlib/trunk'
> svn: PROPFIND of '/svnroot/matplotlib/trunk': Could not resolve
> hostname `svn.sourceforge.net': Host not found
> (https://svn.sourceforge.net)
>
> Can you see anything wrong with what I am doing?
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
> 
svn co
https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/trunk/matplotlib
works for me.
Nils
 

On 7/10/07, Darren Dale <dd...@co...> wrote:
> I ended up checking out the entire repository again. It seems that the working
> address is now matplotlib.svn.sourceforge.net/svn/matplotlib, instead of
> svn.sourceforge.net/svn/matplotlib.
Nothing is working for me
-bash-2.05b$ cd python/svn/
-bash-2.05b$ svn co
http://matplotlib.svn.sourceforge.net/svn/matplotlib
--username=jdh2358 --password=mypasswd
svn: PROPFIND request failed on '/svn/matplotlib'
svn: PROPFIND of '/svn/matplotlib': 405 Method Not Allowed
(http://matplotlib.svn.sourceforge.net)
-bash-2.05b$ svn co
https://matplotlib.svn.sourceforge.net/svn/matplotlib
--username=jdh2358 --password=mypasswd
svn: PROPFIND request failed on '/svn/matplotlib'
svn: PROPFIND of '/svn/matplotlib': 405 Method Not Allowed
(https://matplotlib.svn.sourceforge.net)
-bash-2.05b$ svn co
https://matplotlib.svn.sourceforge.net/svn/matplotlib svn: PROPFIND
request failed on '/svn/matplotlib'
svn: PROPFIND of '/svn/matplotlib': 405 Method Not Allowed
(https://matplotlib.svn.sourceforge.net)
-bash-2.05b$ svn co
https://svn.sourceforge.net/svnroot/matplotlib/trunk matplotlib
svn: PROPFIND request failed on '/svnroot/matplotlib/trunk'
svn: PROPFIND of '/svnroot/matplotlib/trunk': Could not resolve
hostname `svn.sourceforge.net': Host not found
(https://svn.sourceforge.net)
Can you see anything wrong with what I am doing?

On Tuesday 10 July 2007 10:05:40 am John Hunter wrote:
> On 7/10/07, Nils Wagner <nw...@ia...> wrote:
> > svn: PROPFIND request failed on '/svnroot/matplotlib/trunk/matplotlib'
> > svn: PROPFIND of '/svnroot/matplotlib/trunk/matplotlib': Could not
> > resolve hostname `svn.sourceforge.net': Host not found
> > (https://svn.sourceforge.net)
> >
> > Is this a temporary problem ?
>
> I've been experiencing this since yesterday, and was waiting to see if
> it clears up by today...
I ended up checking out the entire repository again. It seems that the working 
address is now matplotlib.svn.sourceforge.net/svn/matplotlib, instead of 
svn.sourceforge.net/svn/matplotlib.
Darren

On 7/10/07, Nils Wagner <nw...@ia...> wrote:
> svn: PROPFIND request failed on '/svnroot/matplotlib/trunk/matplotlib'
> svn: PROPFIND of '/svnroot/matplotlib/trunk/matplotlib': Could not
> resolve hostname `svn.sourceforge.net': Host not found
> (https://svn.sourceforge.net)
>
> Is this a temporary problem ?
I've been experiencing this since yesterday, and was waiting to see if
it clears up by today...
JDH

svn: PROPFIND request failed on '/svnroot/matplotlib/trunk/matplotlib'
svn: PROPFIND of '/svnroot/matplotlib/trunk/matplotlib': Could not
resolve hostname `svn.sourceforge.net': Host not found
(https://svn.sourceforge.net)
Is this a temporary problem ?
Nils

On Tuesday 10 July 2007 07:51:25 am Michael Droettboom wrote:
> I was able to duplicate your bug on my Ubuntu Feisty box.
>
> There is a bug against evince filed here:
>
> http://bugzilla.gnome.org/show_bug.cgi?id=303365
>
> It appears that evince always renders an entire page at a time. There
> are some reports that are very similar to yours -- but on the other
> hand, it doesn't explain why some files work and others don't. It also
> doesn't explain why gv is also be breaking for you, unless it does the
> same thing.
>
> But I suppose we've ruled out matplotlib as the culprit for now, unless
> a number of other tools/versions also have problems.
I was able to zoom to 400% with ggv, but not with the kde ps viewer. The same 
behavior occurs with a simple postscript file created with latex and dvips. I 
think we can safely conclude that it is an issue with the renderer(s), not 
mpl and not your recent improvements.
Darren

I was able to duplicate your bug on my Ubuntu Feisty box.
There is a bug against evince filed here:
http://bugzilla.gnome.org/show_bug.cgi?id=303365
It appears that evince always renders an entire page at a time. There 
are some reports that are very similar to yours -- but on the other 
hand, it doesn't explain why some files work and others don't. It also 
doesn't explain why gv is also be breaking for you, unless it does the 
same thing.
But I suppose we've ruled out matplotlib as the culprit for now, unless 
a number of other tools/versions also have problems.
Cheers,
Mike
Eric Firing wrote:
> Michael Droettboom wrote:
>> Here's simple_plot.ps generated with the old PS backend.
>
> Crash! Again, with 400% magnification. So now you really are off the 
> hook.
>
> efiring@manini:~/programs/py/mpl/tests$ evince simple_plot_ps.ps
> ERROR: /unknownerror in --%op_show_continue--
> Operand stack:
>
> Execution stack:
> %interp_exit .runexec2 --nostringval-- --nostringval-- 
> --nostringval-- 2 %stopped_push --nostringval-- 
> --nostringval-- --nostringval-- false 1 %stopped_push 1 
> 3 %oparray_pop 1 3 %oparray_pop 1 3 %oparray_pop 1 3 
> %oparray_pop .runexec2 --nostringval-- --nostringval-- 
> --nostringval-- 2 %stopped_push --nostringval--
> Dictionary stack:
> --dict:1126/1686(ro)(G)-- --dict:0/20(G)-- --dict:103/200(L)-- 
> --dict:6/7(L)--
> Current allocation mode is local
> ESP Ghostscript 815.04: Unrecoverable error, exit code 1
>
> ** (evince:7887): WARNING **: Interpreter failed.
>
>
> Eric
>

ah...@cs... wrote:
> John (and others),
> 
> I've made a quick change to the svg backend to cull the data, see attached
> patch. I haven't tested it extensively, but it seems OK. It culls both
> lines and polygons out of svg output.
Allan,
Looks good, and the basic idea should be generally applicable. It can 
be sped up (probably), and the code made more compact, by using numpy 
arrays; but this can wait.
> 
> About making it general across backends: I've looked a bit at how things
> get rendered, and here is what I understand: We can only do the culling in
> pixel coordinates, since the height/width of the figure is in pixel units.
> 
> The transform from unit to pixel coordinates either happens in
> Line2D.draw() (such as SVG), or in the backend's draw function (such as
> for agg). Therefore we could only generalize for the case where the
> transform is in line2d, and even then, it looks simpler to do it in the
> backend to me since that is the only place we conveniently have all the
> pixel information we need. Maybe I have misunderstood, though.
The relevant transformation is between data coordinates and pixel 
coordinates, so provided the transformation is from one rectilinear 
system to another, it should be possible to do the culling in either 
system. One way to do it would be by inserting nans in the x and y 
arrays (I am thinking in terms of numpy array-based algorithms), since 
nan is already supported in backends as a do-not-draw marker.
This brings up the larger question of why we have "newstyle" (do 
transformation in backend) and oldstyle (transform before calling 
backend) at all, and whether we still need both. My guess is that we 
don't; by consistently using numpy arrays for x and y, and by using the 
transformation function that works with these arrays, we should be able 
to do the transform equally efficiently in the backend or earlier. 
Making the backends consistent would take a bit of work, and perhaps 
some culling of backends that are neither widely used nor actively 
maintained.
Eric

John (and others),
I've made a quick change to the svg backend to cull the data, see attached
patch. I haven't tested it extensively, but it seems OK. It culls both
lines and polygons out of svg output.
About making it general across backends: I've looked a bit at how things
get rendered, and here is what I understand: We can only do the culling in
pixel coordinates, since the height/width of the figure is in pixel units.
The transform from unit to pixel coordinates either happens in
Line2D.draw() (such as SVG), or in the backend's draw function (such as
for agg). Therefore we could only generalize for the case where the
transform is in line2d, and even then, it looks simpler to do it in the
backend to me since that is the only place we conveniently have all the
pixel information we need. Maybe I have misunderstood, though.
Also, I noticed I forgot to comment out a degugging cout from my last agg
patch, near the end of draw_lines. Oops, sorry, you should fix that in
svn.
Allan
By the way, Here's what I used to test the svg patch, using the -dSVG switch:
import pylab
from scipy import *
pylab.plot(sin(arange(1000)*pi/100.0))
pylab.plot(rand(1000)*2, '.')
pylab.axis([300, 800, -0.5, 1])
pylab.savefig('test')