matplotlib-devel Mailing List for matplotlib (Page 4)

John Hunter wrote:
> On Sun, Jan 3, 2010 at 3:54 PM, Eric Firing <ef...@ha...> wrote:
>> The only problem is that lines.color is the default for LineCollection and
>> Line2D, both of which are fairly separate from Axes, so having them default
>> to rcParams['axes.color_cycle'][0] seems a little odd.
> 
> Yes, I was worrying about the same thing while cooking dinner :-) A
> line can be added to a Figure or an Axes, so perhaps the default color
> should not come from there. What about a module level attribute
> matplotlib.colors.cycle by an rc param "colors.cycle" and then the
> default Line2D and LineCollection color can default to
> matplotlib.colors.cycle[0].
We could, but I don't see any advantage over leaving it the way I made 
it a few minutes ago, which leaves lines.color independent of the 
axes-level cycle. It seems to me that color-cycling is quite different 
from setting a default color for something like Line2D, and there is no 
reason they should be connected via the first element of a list. Maybe 
this would need to be revisited as part of the API layer redesign that 
you sketched out once, but that we still haven't gotten around to advancing.
The idea of a rename to colors.cycle has some appeal, except that as it 
is used, it really is Axes-specific, and appears uniquely at the Axes level.
Eric

On Sun, Jan 3, 2010 at 3:54 PM, Eric Firing <ef...@ha...> wrote:
> The only problem is that lines.color is the default for LineCollection and
> Line2D, both of which are fairly separate from Axes, so having them default
> to rcParams['axes.color_cycle'][0] seems a little odd.
Yes, I was worrying about the same thing while cooking dinner :-) A
line can be added to a Figure or an Axes, so perhaps the default color
should not come from there. What about a module level attribute
matplotlib.colors.cycle by an rc param "colors.cycle" and then the
default Line2D and LineCollection color can default to
matplotlib.colors.cycle[0].

Hi,
With the recent regression fixed in Pygments, the doc auto-builder is
closer to completing successfully. However, there's a new bug. The build
ends with:
LaTeX Warning: File `/home/mpl-chslave/slave-py25/build_docs/build/doc/build/pl
ot_directive/mpl_examples/pylab_examples/barb_demo.pdf' not found on input line
 28650.
!pdfTeX error: pdflatex (file /home/mpl-chslave/slave-py25/build_docs/build/doc
/build/plot_directive/mpl_examples/pylab_examples/barb_demo.pdf): cannot find i
mage file
 ==> Fatal error occurred, no output PDF file produced!
Anyone have a clue what is going on here?
Feel free to check in the attempted solution and the buildbot will
automatically try it. (See the "
<http://mpl-buildbot.code.astraw.com/builders/Ubuntu%208.04%2C%20Python%202.5%2C%20amd64>Ubuntu
8.04, Python 2.5, amd64 (docs)" column at
http://mpl-buildbot.code.astraw.com/waterfall )
Waiting for this will take up to 20 minutes before the build is
triggered (10 minute polling on the svn repo, and a 10 minute timeout
before any build is started in case another commit comes in). So you can
also force a build by clicking the column title ("Ubuntu 8.04, Python
2.5, amd64 (docs)" and then clicking the "Force Build" button.
It will be great to get the buildbot automatically building the svn docs
successfully.
-Andrew

On Sun, Jan 3, 2010 at 4:23 AM, David Cournapeau <cou...@gm...> wrote:
> On Sun, Jan 3, 2010 at 8:05 PM, Nathaniel Smith <nj...@po...> wrote:
>> What I do -- and documented for people in my lab to do -- is set up
>> one virtualenv in my user account, and use it as my default python. (I
>> 'activate' it from my login scripts.) The advantage of this is that
>> easy_install (or pip) just works, without any hassle about permissions
>> etc.
>
> It just works if you happen to be able to build everything from
> sources. That alone means you ignore the majority of users I intend to
> target.
>
> No other community (except maybe Ruby) push those isolated install
> solutions as a general deployment solutions. If it were such a great
> idea, other people would have picked up those solutions.
AFAICT, R works more-or-less identically (once I convinced it to use a
per-user library directory); install.packages() builds from source,
and doesn't automatically pull in and build random C library
dependencies.
I'm not advocating the 'every app in its own world' model that
virtualenv's designers had min mind, but virtualenv is very useful to
give each user their own world. Normally I only use a fraction of
virtualenv's power this way, but sometimes it's handy that they've
solved the more general problem -- I can easily move my environment
out of the way and rebuild if I've done something stupid, or
experiment with new python versions in isolation, or whatever. And
when you *do* have to reproduce some old environment -- if only to
test that the new improved environment gives the same results -- then
it's *really* handy.
>> This should be easier, but I think the basic approach is sound.
>> "Integration with the package system" is useless; the advantage of
>> distribution packages is that distributions can provide a single
>> coherent system with consistent version numbers across all packages,
>> etc., and the only way to "integrate" with that is to, well, get the
>> packages into the distribution.
>
> Another way is to provide our own repository for a few major
> distributions, with automatically built packages. This is how most
> open source providers work. Miguel de Icaza explains this well:
>
> http://tirania.org/blog/archive/2007/Jan-26.html
>
> I hope we will be able to reuse much of the opensuse build service
> infrastructure.
Sure, I'm aware of the opensuse build service, have built third-party
packages for my projects, etc. It's a good attempt, but also has a lot
of problems, and when talking about scientific software it's totally
useless to me :-). First, I don't have root on our compute cluster.
Second, even if I did I'd be very leery about installing third-party
packages because there is no guarantee that the version numbering will
be consistent between the third-party repo and the real distro repo --
suppose that the distro packages 0.1, then the third party packages
0.2, then the distro packages 0.3, will upgrades be seamless? What if
the third party screws up the version numbering at some point? Debian
has "epochs" to deal with this, but third-parties can't use them and
maintain compatibility. What if the person making the third party
packages is not an expert on these random distros that they don't even
use? Will bug reporting tools work properly? Distros are complicated.
Third, while we shouldn't advocate that people screw up backwards
compatibility, version skew is a real issue. If I need one version of
a package and my lab-mate needs another and we have submissions due
tomorrow, then filing bugs is a great idea but not a solution. Fourth,
even if we had expert maintainers taking care of all these third-party
packages and all my concerns were answered, I couldn't convince our
sysadmin of that; he's the one who'd have to clean up if something
went wrong we don't have a big budget for overtime.
Let's be honest -- scientists, on the whole, suck at IT
infrastructure, and small individual packages are not going to be very
expertly put together. IMHO any real solution should take this into
account, keep them sandboxed from the rest of the system, and focus on
providing the most friendly and seamless sandbox possible.
>> On another note, I hope toydist will provide a "source prepare" step,
>> that allows arbitrary code to be run on the source tree. (For, e.g.,
>> cython->C conversion, ad-hoc template languages, etc.) IME this is a
>> very common pain point with distutils; there is just no good way to do
>> it, and it has to be supported in the distribution utility in order to
>> get everything right. In particular:
>> -- Generated files should never be written to the source tree
>> itself, but only the build directory
>> -- Building from a source checkout should run the "source prepare"
>> step automatically
>> -- Building a source distribution should also run the "source
>> prepare" step, and stash the results in such a way that when later
>> building the source distribution, this step can be skipped. This is a
>> common requirement for user convenience, and necessary if you want to
>> avoid arbitrary code execution during builds.
>
> Build directories are hard to implement right. I don't think toydist
> will support this directly. IMO, those advanced builds warrant a real
> build tool - one main goal of toydist is to make integration with waf
> or scons much easier. Both waf and scons have the concept of a build
> directory, which should do everything you described.
Maybe I was unclear -- proper build directory handling is nice,
Cython/Pyrex's distutils integration get it wrong (not their fault,
distutils is just impossible to do anything sensible with, as you've
said), and I've never found build directories hard to implement
(perhaps I'm missing something). But what I'm really talking about is
having a "pre-build" step that integrates properly with the source and
binary packaging stages, and that's not something waf or scons have
any particular support for, AFAIK.
-- Nathaniel

>> 2) It was pointed out that there is a strange connection between the color
>> cycle and the lines.color rcParam. This connection looks to me like a bit
>> of legacy that can be dropped with little risk of pain in user land, since
>> the default would still be to have the initial color in the cycle (blue) be
>> the same as the default lines.color (blue). The proposed difference is that
>> setting lines.color would have no effect on the color_cycle, and vice-versa.
>> John, I think the present connection dates all the way back to your work
>> around svn r500 or so; please tell me if there is some compelling reason to
>> keep it. It appears to me that breaking the connection would make both the
>> code and the actual mpl behavior simpler and less surprising, with no loss
>> of useful functionality.
> 
> I think deprecating lines.color and making the first element of the
> cycle be the default color makes the most sense.
The only problem is that lines.color is the default for LineCollection 
and Line2D, both of which are fairly separate from Axes, so having them 
default to rcParams['axes.color_cycle'][0] seems a little odd.
Eric

On Sun, Jan 3, 2010 at 2:39 PM, Eric Firing <ef...@ha...> wrote:
> 1) Should the the color_cycle be in the axes group? Although it affects
> lines, it is defined only at the Axes level, and affects only lines drawn by
> plot.
>
> Alternative: since it affects only plot, should there be a new "plot" group
> to make that explicit, so it would be rcParams['plot.color_cycle']?
I think it is better to do at the Axes level, because even if plot is
the only command to do this now, I could imagine others commands
wanting to use it (fill_between, bar?). I have had people asking me
just for access to the list so they could use it for some purpose or
another, so having the list easily accessible is desirable.
> 2) It was pointed out that there is a strange connection between the color
> cycle and the lines.color rcParam. This connection looks to me like a bit
> of legacy that can be dropped with little risk of pain in user land, since
> the default would still be to have the initial color in the cycle (blue) be
> the same as the default lines.color (blue). The proposed difference is that
> setting lines.color would have no effect on the color_cycle, and vice-versa.
> John, I think the present connection dates all the way back to your work
> around svn r500 or so; please tell me if there is some compelling reason to
> keep it. It appears to me that breaking the connection would make both the
> code and the actual mpl behavior simpler and less surprising, with no loss
> of useful functionality.
I think deprecating lines.color and making the first element of the
cycle be the default color makes the most sense.
> 3) Would it make sense to add color_cycle to the Axes API, so that it can be
> set directly for a given axes as an alternative to going through the rc
> mechanism? Right now it can be set via a function, but must be set before
> Axes creation; this doesn't make sense to me. It would make more sense as
> an Axes property that could be set at any time, and would apply to
> subsequent calls to plot.
It might be best to make this a class level attribute, that way people
can tweak it globally or at the instance level, and have access to the
list *before* the Axes instance is created. The downside of this is it
falls outside of the usual properties pattern that people can access
via the normal introspection facilities.
The linestyles cycle question is as you say a good bit harder, and
would best be served by having some sort of configurable color scheme,
where one could set grayscale or something like that to get default
black/white/gray colors and styles for the cycle.
JDH

On Sun, Jan 3, 2010 at 2:41 PM, Gökhan Sever <gok...@gm...> wrote:
> You seemed like forgetting to check-in the qt4_editor_options.svg, because I
> get file not found error:
>
> I[2]: Cannot open file
> '.../matplotlib/lib/matplotlib/mpl-data/images/qt4_editor_options.svg',
> because: No such file or directory
Oops, just added. Thanks for the head's up.
JDH

John Hunter wrote:
> On Sat, Jan 2, 2010 at 12:34 AM, Eric Firing <ef...@ha...> wrote:
>> Jae-Joon Lee wrote:
>>> The error happens because of the *.rst files under doc/examples that
>>> are not in sync with examples/*.py.
>>> Removing that directory (doc/examples) will solve the problem (the
>>> directory will be repopulated when you run make.py again). Here is a
>>> related link.
>>>
>>> http://old.nabble.com/python-make.py-html-failure-tt26894350.html
>> Thank you for the quick response. That was it, exactly. There are
>> still some example script failures, but nothing that stops the build in
>> its tracks.
>>
>>> Maybe we need something like "python make.py clean"?
>> Yes, if the generated files can't all land in the "build" directory (as
>> would seem preferable), then the next-best thing would be to have
>> make.py able to do a thorough cleaning.
> 
> I just do svn-clean whenever I have a problem
> 
> http://svn.collab.net/repos/svn/trunk/contrib/client-side/svn-clean
I didn't know about that. It seems strange that svn has no such command 
built in. Mercurial has "hg purge", so I was expecting to find 
something similar in svn. It seems like a pretty basic operation.
Do you know why the plot directive (I assume that is the cause of the 
problem) cannot put all generated output in build? Or is this a more 
general Sphinx wart?
Eric

On Sun, Jan 3, 2010 at 12:35 PM, John Hunter <jd...@gm...> wrote:
> On Sun, Jan 3, 2010 at 9:28 AM, Darren Dale <dsd...@gm...> wrote:
>
> > Your patch file is backwards, it would revert your changes if applied
> > to the updated code. It doesn't matter, the patch is so small that the
> > changes can just be cut and pasted. Unfortunately, however, I will not
> > have time to review your contribution closely enough to consider
> > committing it for at least two weeks, maybe more (conference and long
> > hours at work). I'll look into it when I get a chance, if someone else
> > doesn't beat me to it.
>
> Hey Pierre, Darren,
>
> I took a stab at this. I put the helper code in a backends.qt4_editor
> package, and put the toolbar button right after the configure subplots
> button. Thanks for the patch and the license change Pierre, and sorry
> it took us so long to incorporate it. Let me know if any of the
> reorganizations are a problem for you.
>
> JDH
>
>
John,
You seemed like forgetting to check-in the qt4_editor_options.svg, because I
get file not found error:
I[2]: Cannot open file
'.../matplotlib/lib/matplotlib/mpl-data/images/qt4_editor_options.svg',
because: No such file or directory
Besides, thanks for including this improvement in the trunk. It was long
time waited :)
>
> ------------------------------------------------------------------------------
> This SF.Net email is sponsored by the Verizon Developer Community
> Take advantage of Verizon's best-in-class app development support
> A streamlined, 14 day to market process makes app distribution fast and
> easy
> Join now and get one step closer to millions of Verizon customers
> http://p.sf.net/sfu/verizon-dev2dev
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>
-- 
Gökhan

http://www.mail-archive.com/mat...@li.../msg14772.html
The above thread includes two reasonable requests:
1) add color cycle as an rcParam
2) add a line style cycle as well
The first of these is easier to implement, although it requires a bit 
more than the patch provided. Questions:
1) Should the the color_cycle be in the axes group? Although it affects 
lines, it is defined only at the Axes level, and affects only lines 
drawn by plot.
Alternative: since it affects only plot, should there be a new "plot" 
group to make that explicit, so it would be rcParams['plot.color_cycle']?
2) It was pointed out that there is a strange connection between the 
color cycle and the lines.color rcParam. This connection looks to me 
like a bit of legacy that can be dropped with little risk of pain in 
user land, since the default would still be to have the initial color in 
the cycle (blue) be the same as the default lines.color (blue). The 
proposed difference is that setting lines.color would have no effect on 
the color_cycle, and vice-versa. John, I think the present connection 
dates all the way back to your work around svn r500 or so; please tell 
me if there is some compelling reason to keep it. It appears to me that 
breaking the connection would make both the code and the actual mpl 
behavior simpler and less surprising, with no loss of useful functionality.
3) Would it make sense to add color_cycle to the Axes API, so that it 
can be set directly for a given axes as an alternative to going through 
the rc mechanism? Right now it can be set via a function, but must be 
set before Axes creation; this doesn't make sense to me. It would make 
more sense as an Axes property that could be set at any time, and would 
apply to subsequent calls to plot.
My sense is that as a matter of design strategy, whenever possible, one 
should be able to use the API, via properties, methods, or functions, to 
 locally set any options for which rcParams gives the global default 
values.
Eric

On Sat, Jan 2, 2010 at 12:34 AM, Eric Firing <ef...@ha...> wrote:
> Jae-Joon Lee wrote:
>> The error happens because of the *.rst files under doc/examples that
>> are not in sync with examples/*.py.
>> Removing that directory (doc/examples) will solve the problem (the
>> directory will be repopulated when you run make.py again). Here is a
>> related link.
>>
>> http://old.nabble.com/python-make.py-html-failure-tt26894350.html
>
> Thank you for the quick response. That was it, exactly. There are
> still some example script failures, but nothing that stops the build in
> its tracks.
>
>>
>> Maybe we need something like "python make.py clean"?
>
> Yes, if the generated files can't all land in the "build" directory (as
> would seem preferable), then the next-best thing would be to have
> make.py able to do a thorough cleaning.
I just do svn-clean whenever I have a problem
http://svn.collab.net/repos/svn/trunk/contrib/client-side/svn-clean
JDH

2010年1月3日 John Hunter <jd...@gm...>:
> On Sun, Jan 3, 2010 at 9:28 AM, Darren Dale <dsd...@gm...> wrote:
>
>> Your patch file is backwards, it would revert your changes if applied
>> to the updated code. It doesn't matter, the patch is so small that the
>> changes can just be cut and pasted. Unfortunately, however, I will not
>> have time to review your contribution closely enough to consider
>> committing it for at least two weeks, maybe more (conference and long
>> hours at work). I'll look into it when I get a chance, if someone else
>> doesn't beat me to it.
>
> Hey Pierre, Darren,
>
> I took a stab at this. I put the helper code in a backends.qt4_editor
> package, and put the toolbar button right after the configure subplots
> button. Thanks for the patch and the license change Pierre, and sorry
> it took us so long to incorporate it. Let me know if any of the
> reorganizations are a problem for you.
>
> JDH
>
That is perfect!
Thanks very much John.
Pierre

2010年1月3日 Darren Dale <dsd...@gm...>:
> On Sun, Jan 3, 2010 at 9:11 AM, Pierre Raybaut <co...@py...> wrote:
>> 2009年12月1日 Darren Dale <dsd...@gm...>:
>>> On Tue, Apr 28, 2009 at 8:18 AM, Pierre Raybaut <co...@py...> wrote:
>>>> Hi all,
>>>>
>>>> I would like to contribute to matplotlib with this enhancement for the
>>>> PyQt4 backend: the idea is to add a toolbar button to configure figure
>>>> options (axes, curves, ...).
>>>>
>>>> It's based on a tiny module called formlayout to generate PyQt4 form
>>>> dialog automatically.
>>>>
>>>> Some screenshots:
>>>> http://code.google.com/p/formlayout/
>>>>
>>>> So, if you're interested (all the following is GPL2):
>>>>
>>>> *matplotlib patch*
>>>
>>> Would you please submit an actual patch? I don't know exactly where
>>> you intend these changes to be placed.
>>>
>>>> In FigureManagerQT.__init__, added:
>>>> self.canvas.axes = self.canvas.figure.add_subplot(111)
>>>
>>> What is the purpose of this change? What if I didn't want such an axes
>>> on my canvas? What if I want to layout my own axes([.2,.2,.75,.75]) or
>>> add_subplot(311)? I don't think these changes can be accepted in the
>>> current form, they don't appear to integrate well with the standard
>>> behavior of the library.
>>>
>>> Darren
>>
>> Ok, I admit that it was very difficult to fix this -- it took me a lot
>> of seconds without knowing very well matplotlib... ;-)
>
> [...]
>
>> Here is a zip file containing all you need (patch for backend_qt4.py,
>> two scripts to be copied in backends/, and one .svg image to be copied
>> to mpl-data/images/)
>
> [...]
>
> Your patch file is backwards, it would revert your changes if applied
> to the updated code.
Sorry for this... I admit that I did this maybe too quickly.
> It doesn't matter, the patch is so small that the
> changes can just be cut and pasted. Unfortunately, however, I will not
> have time to review your contribution closely enough to consider
> committing it for at least two weeks, maybe more (conference and long
> hours at work). I'll look into it when I get a chance, if someone else
> doesn't beat me to it.
>
> Darren
No problem, this is already part of Spyder's matplotlib patch for six
months now, and I'm only using matplotlib through Spyder, so there is
absolutely no hurry for me. Since the beginning I was just trying to
be useful, to share something. I simply thought that this kind of
feature was missing in matplotlib: matplotlib produces good looking
figures but -compared to MATLAB- there is a serious lack of
interactivity (and performance -- but matplotlib is at worst as slow
as MATLAB, so it doesn't matter much).
Pierre

On Sun, Jan 3, 2010 at 9:28 AM, Darren Dale <dsd...@gm...> wrote:
> Your patch file is backwards, it would revert your changes if applied
> to the updated code. It doesn't matter, the patch is so small that the
> changes can just be cut and pasted. Unfortunately, however, I will not
> have time to review your contribution closely enough to consider
> committing it for at least two weeks, maybe more (conference and long
> hours at work). I'll look into it when I get a chance, if someone else
> doesn't beat me to it.
Hey Pierre, Darren,
I took a stab at this. I put the helper code in a backends.qt4_editor
package, and put the toolbar button right after the configure subplots
button. Thanks for the patch and the license change Pierre, and sorry
it took us so long to incorporate it. Let me know if any of the
reorganizations are a problem for you.
JDH

On Sun, Jan 3, 2010 at 9:11 AM, Pierre Raybaut <co...@py...> wrote:
> 2009年12月1日 Darren Dale <dsd...@gm...>:
>> On Tue, Apr 28, 2009 at 8:18 AM, Pierre Raybaut <co...@py...> wrote:
>>> Hi all,
>>>
>>> I would like to contribute to matplotlib with this enhancement for the
>>> PyQt4 backend: the idea is to add a toolbar button to configure figure
>>> options (axes, curves, ...).
>>>
>>> It's based on a tiny module called formlayout to generate PyQt4 form
>>> dialog automatically.
>>>
>>> Some screenshots:
>>> http://code.google.com/p/formlayout/
>>>
>>> So, if you're interested (all the following is GPL2):
>>>
>>> *matplotlib patch*
>>
>> Would you please submit an actual patch? I don't know exactly where
>> you intend these changes to be placed.
>>
>>> In FigureManagerQT.__init__, added:
>>> self.canvas.axes = self.canvas.figure.add_subplot(111)
>>
>> What is the purpose of this change? What if I didn't want such an axes
>> on my canvas? What if I want to layout my own axes([.2,.2,.75,.75]) or
>> add_subplot(311)? I don't think these changes can be accepted in the
>> current form, they don't appear to integrate well with the standard
>> behavior of the library.
>>
>> Darren
>
> Ok, I admit that it was very difficult to fix this -- it took me a lot
> of seconds without knowing very well matplotlib... ;-)
[...]
> Here is a zip file containing all you need (patch for backend_qt4.py,
> two scripts to be copied in backends/, and one .svg image to be copied
> to mpl-data/images/)
[...]
Your patch file is backwards, it would revert your changes if applied
to the updated code. It doesn't matter, the patch is so small that the
changes can just be cut and pasted. Unfortunately, however, I will not
have time to review your contribution closely enough to consider
committing it for at least two weeks, maybe more (conference and long
hours at work). I'll look into it when I get a chance, if someone else
doesn't beat me to it.
Darren

2009年12月1日 Darren Dale <dsd...@gm...>:
> On Tue, Apr 28, 2009 at 8:18 AM, Pierre Raybaut <co...@py...> wrote:
>> Hi all,
>>
>> I would like to contribute to matplotlib with this enhancement for the
>> PyQt4 backend: the idea is to add a toolbar button to configure figure
>> options (axes, curves, ...).
>>
>> It's based on a tiny module called formlayout to generate PyQt4 form
>> dialog automatically.
>>
>> Some screenshots:
>> http://code.google.com/p/formlayout/
>>
>> So, if you're interested (all the following is GPL2):
>>
>> *matplotlib patch*
>
> Would you please submit an actual patch? I don't know exactly where
> you intend these changes to be placed.
>
>> In FigureManagerQT.__init__, added:
>> self.canvas.axes = self.canvas.figure.add_subplot(111)
>
> What is the purpose of this change? What if I didn't want such an axes
> on my canvas? What if I want to layout my own axes([.2,.2,.75,.75]) or
> add_subplot(311)? I don't think these changes can be accepted in the
> current form, they don't appear to integrate well with the standard
> behavior of the library.
>
> Darren
Ok, I admit that it was very difficult to fix this -- it took me a lot
of seconds without knowing very well matplotlib... ;-)
Anyway, now, you can't say no any longer :-)
Here is a zip file containing all you need (patch for backend_qt4.py,
two scripts to be copied in backends/, and one .svg image to be copied
to mpl-data/images/) -- I can't make it clearer than this...
Cheers,
Pierre

On Sun, Jan 3, 2010 at 8:05 PM, Nathaniel Smith <nj...@po...> wrote:
> On Tue, Dec 29, 2009 at 6:34 AM, David Cournapeau <cou...@gm...> wrote:
>> Buildout, virtualenv all work by sandboxing from the system python:
>> each of them do not see each other, which may be useful for
>> development, but as a deployment solution to the casual user who may
>> not be familiar with python, it is useless. A scientist who installs
>> numpy, scipy, etc... to try things out want to have everything
>> available in one python interpreter, and does not want to jump to
>> different virtualenvs and whatnot to try different packages.
>
> What I do -- and documented for people in my lab to do -- is set up
> one virtualenv in my user account, and use it as my default python. (I
> 'activate' it from my login scripts.) The advantage of this is that
> easy_install (or pip) just works, without any hassle about permissions
> etc.
It just works if you happen to be able to build everything from
sources. That alone means you ignore the majority of users I intend to
target.
No other community (except maybe Ruby) push those isolated install
solutions as a general deployment solutions. If it were such a great
idea, other people would have picked up those solutions.
> This should be easier, but I think the basic approach is sound.
> "Integration with the package system" is useless; the advantage of
> distribution packages is that distributions can provide a single
> coherent system with consistent version numbers across all packages,
> etc., and the only way to "integrate" with that is to, well, get the
> packages into the distribution.
Another way is to provide our own repository for a few major
distributions, with automatically built packages. This is how most
open source providers work. Miguel de Icaza explains this well:
http://tirania.org/blog/archive/2007/Jan-26.html
I hope we will be able to reuse much of the opensuse build service
infrastructure.
>
> On another note, I hope toydist will provide a "source prepare" step,
> that allows arbitrary code to be run on the source tree. (For, e.g.,
> cython->C conversion, ad-hoc template languages, etc.) IME this is a
> very common pain point with distutils; there is just no good way to do
> it, and it has to be supported in the distribution utility in order to
> get everything right. In particular:
> -- Generated files should never be written to the source tree
> itself, but only the build directory
> -- Building from a source checkout should run the "source prepare"
> step automatically
> -- Building a source distribution should also run the "source
> prepare" step, and stash the results in such a way that when later
> building the source distribution, this step can be skipped. This is a
> common requirement for user convenience, and necessary if you want to
> avoid arbitrary code execution during builds.
Build directories are hard to implement right. I don't think toydist
will support this directly. IMO, those advanced builds warrant a real
build tool - one main goal of toydist is to make integration with waf
or scons much easier. Both waf and scons have the concept of a build
directory, which should do everything you described.
David

On Tue, Dec 29, 2009 at 6:34 AM, David Cournapeau <cou...@gm...> wrote:
> Buildout, virtualenv all work by sandboxing from the system python:
> each of them do not see each other, which may be useful for
> development, but as a deployment solution to the casual user who may
> not be familiar with python, it is useless. A scientist who installs
> numpy, scipy, etc... to try things out want to have everything
> available in one python interpreter, and does not want to jump to
> different virtualenvs and whatnot to try different packages.
What I do -- and documented for people in my lab to do -- is set up
one virtualenv in my user account, and use it as my default python. (I
'activate' it from my login scripts.) The advantage of this is that
easy_install (or pip) just works, without any hassle about permissions
etc. This should be easier, but I think the basic approach is sound.
"Integration with the package system" is useless; the advantage of
distribution packages is that distributions can provide a single
coherent system with consistent version numbers across all packages,
etc., and the only way to "integrate" with that is to, well, get the
packages into the distribution.
On another note, I hope toydist will provide a "source prepare" step,
that allows arbitrary code to be run on the source tree. (For, e.g.,
cython->C conversion, ad-hoc template languages, etc.) IME this is a
very common pain point with distutils; there is just no good way to do
it, and it has to be supported in the distribution utility in order to
get everything right. In particular:
 -- Generated files should never be written to the source tree
itself, but only the build directory
 -- Building from a source checkout should run the "source prepare"
step automatically
 -- Building a source distribution should also run the "source
prepare" step, and stash the results in such a way that when later
building the source distribution, this step can be skipped. This is a
common requirement for user convenience, and necessary if you want to
avoid arbitrary code execution during builds.
And if you just set up the distribution util so that the only place
you can specify arbitrary code execution is in the "source prepare"
step, then even people who know nothing about packaging will
automatically get all of the above right.
Cheers,
-- Nathaniel

On Sun, Jan 3, 2010 at 3:27 AM, Andrew Straw <str...@as...> wrote:
>>
> Typically, the dependencies only depend on the smallest subset of what
> they require (if they don't need lapack, they'd only depend on
> python-numpy-core in your example), but yes, if there's an unsatisfiable
> condition, then apt-get will raise an error and abort. In practice, this
> system seems to work quite well, IMO.
Yes, but:
 - debian dependency resolution is complex. I think many people don't
realize how complex the problem really is (AFAIK, any correct scheme
to resolve dependencies in debian requires an algorithm which is
NP-complete )
 - introducing a lot of variants significantly slow down the whole thing.
I think it worths thinking whether our problems warrant such a complexity.
>
> Anyhow, here's the full Debian documentation:
> http://www.debian.org/doc/debian-policy/ch-relationships.html
This is not the part I am afraid of. This is:
http://people.debian.org/~dburrows/model.pdf
cheers,
David

On Sun, Jan 03, 2010 at 03:05:54AM -0800, Nathaniel Smith wrote:
> What I do -- and documented for people in my lab to do -- is set up
> one virtualenv in my user account, and use it as my default python. (I
> 'activate' it from my login scripts.) The advantage of this is that
> easy_install (or pip) just works, without any hassle about permissions
> etc. This should be easier, but I think the basic approach is sound.
> "Integration with the package system" is useless; the advantage of
> distribution packages is that distributions can provide a single
> coherent system with consistent version numbers across all packages,
> etc., and the only way to "integrate" with that is to, well, get the
> packages into the distribution.
That works because either you use packages that don't have much hard-core
compiled dependencies, or these are already installed.
Think about installing VTK or ITK this way, even something simpler such
as umfpack. I think that you would loose most of your users. In my lab,
I do lose users on such packages actually.
Beside, what you are describing is possible without package isolation, it
is simply the use of a per-user local site-packages, which now semi
automatic in python2.6 using the '.local' directory. I do agree that, in
a research lab, this is a best practice.
Gaël

Andrew Straw wrote:
> Andrew Straw wrote:
> 
>> In fact, does anyone know what could be wrong? The last lines of the
>> LaTeX output are below.
>> 
>> 
> OK, this cropped up on another buildbot for another project of mine --
> it looks with a Sphinx dependency, Pygments 1.2.1 (just released), there
> is some issue that wasn't there with Pygments 1.1.1. I'll try to isolate
> the issue and report it to the appropriate tracker, but at this point I
> don't think it's an MPL issue.
> 
For the record, I reported it here:
http://dev.pocoo.org/projects/pygments/ticket/463

Andrew Straw wrote:
> In fact, does anyone know what could be wrong? The last lines of the
> LaTeX output are below.
> 
OK, this cropped up on another buildbot for another project of mine --
it looks with a Sphinx dependency, Pygments 1.2.1 (just released), there
is some issue that wasn't there with Pygments 1.1.1. I'll try to isolate
the issue and report it to the appropriate tracker, but at this point I
don't think it's an MPL issue.
-Andrew

David Cournapeau wrote:
> On Sat, Jan 2, 2010 at 4:58 PM, Gael Varoquaux
> <gae...@no...> wrote:
> 
>> On Sat, Jan 02, 2010 at 11:32:00AM +0900, David Cournapeau wrote:
>> 
>>> [snip]
>>> - supporting different variants of the same package in the
>>> dependency graph at install time
>>> 
>>> [snip]
>>> 
>>> The second issue is more challenging. It complicates the dependency
>>> handling quite a bit, and may cause difficult situations to happen at
>>> dependency resolution time. This becomes particularly messy if you mix
>>> packages you build yourself with packages grabbed from a repository. I
>>> wonder if there is a simpler solution which would give a similar
>>> feature set.
>>> 
>> AFAICT, in Debian, the same feature is given via virtual packages: you
>> would have:
>> 
>
> I don't think virtual-packages entirely fix the issue. AFAIK, virtual
> packages have two uses:
> - handle dependencies where several packages may resolve one
> particular dependency in an equivalent way (one good example is
> LAPACK: both liblapack and libatlas provides the lapack feature)
> - closer to this discussion, you can build several variants of the
> same package, and each variant would resolve the dependency on a
> virtual package handling the commonalities. For example, say we have
> two numpy packages, one built with lapack (python-numpy-full), the
> other without (python-numpy-core). What happens when a package foo
> depends on numpy-full, but numpy-core is installed ?
When you type "apt-get install my_new_package", the version resolution
system will tell you that apt-get will remove python-numpy-core and will
install python-numpy-full in the act of installing my_new_package.
> AFAICS, this can
> only work as long as the set containing every variant can be ordered
> (in the conventional set ordering sense), and the dependency can be
> satisfied by the smallest one.
> 
Typically, the dependencies only depend on the smallest subset of what
they require (if they don't need lapack, they'd only depend on
python-numpy-core in your example), but yes, if there's an unsatisfiable
condition, then apt-get will raise an error and abort. In practice, this
system seems to work quite well, IMO.
Anyhow, here's the full Debian documentation:
http://www.debian.org/doc/debian-policy/ch-relationships.html

2010年1月2日 David Cournapeau <cou...@gm...>:
> On Fri, Jan 1, 2010 at 10:43 PM, Pierre Raybaut <co...@py...> wrote:
>> Hi David,
>>
>> Following your announcement for the 'toydist' module, I think that
>> your project is very promising: this is certainly a great idea and it
>> will be very controversial but that's because people expectactions are
>> great on this matter (distutils is so disappointing indeed).
>>
>> Anyway, if I may be useful, I'll gladly contribute to it.
>> In time, I could change the whole Python(x,y) packaging system (which
>> is currently quite ugly... but easy/quick to manage/maintain) to
>> use/promote this new module.
>
> That would be a good way to test toydist on a real, complex package. I
> am not familiar at all with python(x,y) internals. Do you have some
> explanation I could look at somewhere ?
Honestly, let's assume that there is currently no packaging system...
it would not be very far from the truth. I did it when I was young and
naive regarding Python. Actually I almost did it without having
writing any code in Python (approx. two months after earing about the
Python language for the first time) : it's an ugly collection of
AutoIt, NSIS and PHP scripts -- most of the tasks are automated like
updating the generated website pages and so on.
So I'm not proud at all, but it was easy and very quick to do as it
is, and it's still quite easy to maintain. But, it's not satisfying in
terms of code "purity" -- I've been wanting to rewrite all this in
Python for a year and a half but since the features are there, there
is no real motivation to do the work (in other words, Python(x,y)
users would not see the difference, at least at the beginning).
An other thing: Python(x,y) plugins are not built from source but from
existing binaries (it's a pity I know, but it was incredibly faster to
do this way). For example, eggs or distutils .exe may be converted in
Python(x,y) plugins directly (same internal directory structure).
So it may be different from the idea you had in mind (it's not like
EPD which is entirely generated from source, AFAIK).
> In the meantime, I will try to clean-up the code to have a first
> experimental release.
>
Ok, keep up the good work!
Cheers,
Pierre

On Sat, Jan 2, 2010 at 4:58 PM, Gael Varoquaux
<gae...@no...> wrote:
> On Sat, Jan 02, 2010 at 11:32:00AM +0900, David Cournapeau wrote:
>> [snip]
>>  - supporting different variants of the same package in the
>> dependency graph at install time
>
>> [snip]
>
>> The second issue is more challenging. It complicates the dependency
>> handling quite a bit, and may cause difficult situations to happen at
>> dependency resolution time. This becomes particularly messy if you mix
>> packages you build yourself with packages grabbed from a repository. I
>> wonder if there is a simpler solution which would give a similar
>> feature set.
>
>
> AFAICT, in Debian, the same feature is given via virtual packages: you
> would have:
I don't think virtual-packages entirely fix the issue. AFAIK, virtual
packages have two uses:
 - handle dependencies where several packages may resolve one
particular dependency in an equivalent way (one good example is
LAPACK: both liblapack and libatlas provides the lapack feature)
 - closer to this discussion, you can build several variants of the
same package, and each variant would resolve the dependency on a
virtual package handling the commonalities. For example, say we have
two numpy packages, one built with lapack (python-numpy-full), the
other without (python-numpy-core). What happens when a package foo
depends on numpy-full, but numpy-core is installed ? AFAICS, this can
only work as long as the set containing every variant can be ordered
(in the conventional set ordering sense), and the dependency can be
satisfied by the smallest one.
cheers,
David