Hi all, With the integration of mathtex into matplotlib nearing completion (just bug fixes really) I think now is a good time to be considering the best way to include mathtex into matplotlib. This has already been discussed on the mathtex mailing list, with Michael proposing a few ways of doing this. However, I am not an expert by any means when it comes to Python set-up/configuring. (As anyone who has looked at setup.py in mathtex will have seen.) While including the source is not difficult (it can be done directly or using svn:external) getting it built/configured is. Lets say that mathtex was dumped into lib/, how would one go about configuring and installing it from setup.py in matplotlib. Although I am sure that just executing a shell command would do it I am sure there must be a 'better' option for this type of 'package chaining'. Does anyone have any suggestions or recommendations? Regards, Freddie.
There may be a setuptools solution here, but if there is, I'm not the one to know ;) matplotlib is for the most part ignorant of setuptools, and it's probably reasonable to keep it that way. Anyway, since the mathtex setup infrastructure is based on what matplotlib was already doing, there's a common convention we can exploit. Essentially, the matplotlib setup.py builds up a list of extension modules (ext_modules) and packages and then passes those lists to distutils for building. So, in theory, all mathtex needs to do is provide a function that will add extension modules and packages to those lists (basically like all of the build_* methods in setupext.py). So basically, matplotlib's setup.py would import lib/mathtex/setupext.py (by filename) and call a method in it. Lots of details I'm missing, but that should provide a general framework. Another issue this raises is whether to build the FT2Font and png modules twice, once as part of matplotlib, and once as part of mathtex. Once mathtex is a truly external dependency for matplotlib, I don't see a way around this, so maybe we should just pretend we're already there, despite the duplication. If we want to be clever, I could see mathtex being smart about imports: try importing its local copies of its libraries and failing that import matplotlib's. I'm not entirely sure about that idea, but I sort of feel "hacky-if-you-do, hacky-if-you-don't" here ;) I see the code maintenance problem of this duplication (i.e. making sure bugfixes to FT2Font make it into matplotlib and mathtex) almost as a separate issue. We know the solution to that: break out the freetype wrappers into its own project (which then both matplotlib and mathtex would rely on) -- but that's probably outside of the scope of this GSoC project. Please try to use svn:externals if you can -- that will make pulling updates from mathtex easier. I've never used it cross-repository like this, so there may be unforeseen issues. It also just occurred to me that we might want to take another step in preparation for mathtex as an external dependency: make it optional. That is, if importing mathtex fails, be able to render regular text, and warn if trying to render math text. Mike Freddie Witherden wrote: > Hi all, > > With the integration of mathtex into matplotlib nearing completion > (just bug fixes really) I think now is a good time to be considering > the best way to include mathtex into matplotlib. > > This has already been discussed on the mathtex mailing list, with > Michael proposing a few ways of doing this. However, I am not an > expert by any means when it comes to Python set-up/configuring. (As > anyone who has looked at setup.py in mathtex will have seen.) > > While including the source is not difficult (it can be done directly > or using svn:external) getting it built/configured is. Lets say that > mathtex was dumped into lib/, how would one go about configuring and > installing it from setup.py in matplotlib. > > Although I am sure that just executing a shell command would do it I > am sure there must be a 'better' option for this type of 'package > chaining'. > > Does anyone have any suggestions or recommendations? > > Regards, Freddie. > > ------------------------------------------------------------------------------ > _______________________________________________ > Matplotlib-devel mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-devel > -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA
Hi, > Anyway, since the mathtex setup infrastructure is based on what > matplotlib was already doing, there's a common convention we can > exploit. Essentially, the matplotlib setup.py builds up a list of > extension modules (ext_modules) and packages and then passes those > lists to distutils for building. So, in theory, all mathtex needs > to do is provide a function that will add extension modules and > packages to those lists (basically like all of the build_* methods > in setupext.py). So basically, matplotlib's setup.py would import > lib/mathtex/setupext.py (by filename) and call a method in it. Lots > of details I'm missing, but that should provide a general framework. Not sure about the extension modules + lists stuff, although setuptools is not my area of expertise. Sadly executing the lib/ mathtex/setup.py file isn't really an option either, so I'll need to think on it. Might need to ask for some help though. > > Another issue this raises is whether to build the FT2Font and png > modules twice, once as part of matplotlib, and once as part of > mathtex. Once mathtex is a truly external dependency for > matplotlib, I don't see a way around this, so maybe we should just > pretend we're already there, despite the duplication. If we want to > be clever, I could see mathtex being smart about imports: try > importing its local copies of its libraries and failing that import > matplotlib's. I'm not entirely sure about that idea, but I sort of > feel "hacky-if-you-do, hacky-if-you-don't" here ;) I don't think it hurts having two FT2Font wrappers lying around on a system. > Please try to use svn:externals if you can -- that will make pulling > updates from mathtex easier. I've never used it cross-repository > like this, so there may be unforeseen issues. svn:externals seems to work quite well and is currently used in the mathtex branch to pull mathtex. > It also just occurred to me that we might want to take another step > in preparation for mathtex as an external dependency: make it > optional. That is, if importing mathtex fails, be able to render > regular text, and warn if trying to render math text. Shouldn't be too hard to add, I'll write in backend support today. Regards, Freddie.
Hi, I was thinking about the problem of including mathtex in matplotlib earlier and came up with an alternative means of 'solving' the problem. Instead of hacking setup.py to install mathtex on the behalf of matplotlib it may be easier to leave it up to the user/packager to install mathtex. While simplifying the code (not as many changed are need to the setup files) it also eliminates the problems associated with one package installing another package (matplotlib installing mathtex). This could be done either through ones distribution package manger, standalone (getting the mathtex source) or just by following the instructions in lib/mathtex/INSTALL. (Checking out matplotlib also checks out the mathtex source into lib/mathtex.) However, I am interested if this solution is acceptable to the matplotlib developers. It may also be worth pointing out that mathtex is an optional dependency of matplotlib and is not required for matplotlib to function. Regards, Freddie.
I think we need (at least as a transitional stopgap) a single "python setup.py install" to install both matplotlib and mathtex. Once distributions catch up (which could take more than a year, depending on cycles), we can consider being more loosely coupled. There are already examples of installing subpackages from matplotlib (pytz and dateutil) for example, and each of the extensions (_backend_agg etc.) have examples in setupext.py showing how to dynamically add extensions and packages to the list of things that matplotlib's setup.py will install and build. Are there fundamental differences in how that works vs. how mathtex needs to work that is a stumbling block? I must admit I haven't thought it all the way through, but I'm surprised that the existing examples there don't provide a template for how to deal with mathtex. That said, I know that distutils can be rather, um, labyrinthine. Cheers, Mike Freddie Witherden wrote: > Hi, > > I was thinking about the problem of including mathtex in matplotlib earlier > and came up with an alternative means of 'solving' the problem. > > Instead of hacking setup.py to install mathtex on the behalf of matplotlib it > may be easier to leave it up to the user/packager to install mathtex. > > While simplifying the code (not as many changed are need to the setup files) it > also eliminates the problems associated with one package installing another > package (matplotlib installing mathtex). > > This could be done either through ones distribution package manger, standalone > (getting the mathtex source) or just by following the instructions in > lib/mathtex/INSTALL. (Checking out matplotlib also checks out the mathtex > source into lib/mathtex.) > > However, I am interested if this solution is acceptable to the matplotlib > developers. > > It may also be worth pointing out that mathtex is an optional dependency of > matplotlib and is not required for matplotlib to function. > > Regards, Freddie. > > ------------------------------------------------------------------------------ > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus on > what you do best, core application coding. Discover what's new with > Crystal Reports now. http://p.sf.net/sfu/bobj-july > _______________________________________________ > Matplotlib-devel mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-devel > -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA
On Wed, Jul 29, 2009 at 7:52 AM, Michael Droettboom<md...@st...> wrote: > I think we need (at least as a transitional stopgap) a single "python > setup.py install" to install both matplotlib and mathtex. Once > distributions catch up (which could take more than a year, depending on > cycles), we can consider being more loosely coupled. > > There are already examples of installing subpackages from matplotlib > (pytz and dateutil) for example, and each of the extensions > (_backend_agg etc.) have examples in setupext.py showing how to > dynamically add extensions and packages to the list of things that > matplotlib's setup.py will install and build. Are there fundamental > differences in how that works vs. how mathtex needs to work that is a > stumbling block? I must admit I haven't thought it all the way through, > but I'm surprised that the existing examples there don't provide a > template for how to deal with mathtex. That said, I know that distutils > can be rather, um, labyrinthine. I second this. I would like to see the new mathtext (and png and freetype wrappers as necessary, possibly in mathtext) live in the matplotlib trunk under lib, the same place we put pytz and dateutil. It can have its own setup and release cycle, but I think it would make matters simpler to have it there. If there is compelling reason to have it live in a separate svn repo, we can always grab it externally, as scipy does for our sphinx extensions. We can use the mpl configure setup.cfg to either no-install/force-install/auto-install as we do for these other packages. This works fine at build time, but is a little more complicated when distributing binaries. Eg for win32 binaries, we currently force these packages on, which is sometimes a problem. We might be able to be a little smarter about building win32 installers, so the user can select which subpackages to install, but I am not holding my breath on this one. I am unwilling to rely on any setuptools solutions at this point, as I've always viewed them with suspicion (maybe because I rarely found easy_install to be easy) and the setuptools community is in a bit of disarray, with the lead developer unable to adequately maintain the package and unwilling to let those who want to take it over do so since he feels they are not qualified, so a fork is in progress: http://www.mail-archive.com/dis...@py.../msg08111.html JDH
On 2009年07月29日 08:17, John Hunter wrote: > On Wed, Jul 29, 2009 at 7:52 AM, Michael Droettboom<md...@st...> wrote: >> I think we need (at least as a transitional stopgap) a single "python >> setup.py install" to install both matplotlib and mathtex. Once >> distributions catch up (which could take more than a year, depending on >> cycles), we can consider being more loosely coupled. >> >> There are already examples of installing subpackages from matplotlib >> (pytz and dateutil) for example, and each of the extensions >> (_backend_agg etc.) have examples in setupext.py showing how to >> dynamically add extensions and packages to the list of things that >> matplotlib's setup.py will install and build. Are there fundamental >> differences in how that works vs. how mathtex needs to work that is a >> stumbling block? I must admit I haven't thought it all the way through, >> but I'm surprised that the existing examples there don't provide a >> template for how to deal with mathtex. That said, I know that distutils >> can be rather, um, labyrinthine. > > I second this. I would like to see the new mathtext (and png and > freetype wrappers as necessary, possibly in mathtext) live in the > matplotlib trunk under lib, the same place we put pytz and dateutil. > It can have its own setup and release cycle, but I think it would make > matters simpler to have it there. If there is compelling reason to > have it live in a separate svn repo, we can always grab it externally, > as scipy does for our sphinx extensions. Instead of using the matplotlib package as a catch-all for these external packages, could you instead keep them all nominally separate and release a tarball that includes all of them side by side? You can even have an installation script that will run the setups for each. Even better, this installation script wouldn't use distutils, so you don't have to hack all of this "optional package" stuff into it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Thu, Jul 23, 2009 at 8:36 AM, Freddie Witherden <fr...@wi...>wrote: > Hi all, > > With the integration of mathtex into matplotlib nearing completion > (just bug fixes really) I think now is a good time to be considering > the best way to include mathtex into matplotlib. > > This has already been discussed on the mathtex mailing list, with > Michael proposing a few ways of doing this. However, I am not an > expert by any means when it comes to Python set-up/configuring. (As > anyone who has looked at setup.py in mathtex will have seen.) > > While including the source is not difficult (it can be done directly > or using svn:external) getting it built/configured is. Lets say that > mathtex was dumped into lib/, how would one go about configuring and > installing it from setup.py in matplotlib. > > Although I am sure that just executing a shell command would do it I > am sure there must be a 'better' option for this type of 'package > chaining'. > > Does anyone have any suggestions or recommendations? > > Regards, Freddie. > > > ------------------------------------------------------------------------------ > _______________________________________________ > Matplotlib-devel mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-devel > Could you please the advantages of using mathtext? How does it compare and contrast to the existing interface? Any pointers to read before getting more lost? Thanks, -- Gökhan
Hi, On 23 Jul 2009, at 22:46, Gökhan SEVER wrote: > Could you please the advantages of using mathtext? How does it > compare and contrast to the existing interface? The interface itself is mostly the same, although there are a few enhancements to make it easier to use. To main advantages of using mathtex are that, being an external project, it is likely to experience a wider audience than the current code in matplotlib. This should result in a package that is better maintained with more features. Currently, however, it is almost exactly the same as the mathtext code in matplotlib. > Any pointers to read before getting more lost? There is the project code site here: http://code.google.com/p/mathtex/ while my project blog is here: http://gsoc-mathtex.blogspot.com/ Regards, Freddie.
> I second this. I would like to see the new mathtext (and png and > freetype wrappers as necessary, possibly in mathtext) live in the > matplotlib trunk under lib, the same place we put pytz and dateutil. > It can have its own setup and release cycle, but I think it would make > matters simpler to have it there. If there is compelling reason to > have it live in a separate svn repo, we can always grab it externally, > as scipy does for our sphinx extensions. > > We can use the mpl configure setup.cfg to either > no-install/force-install/auto-install as we do for these other > packages. This works fine at build time, but is a little more > complicated when distributing binaries. Eg for win32 binaries, we > currently force these packages on, which is sometimes a problem. We > might be able to be a little smarter about building win32 installers, > so the user can select which subpackages to install, but I am not > holding my breath on this one. After thinking about the problem a little bit I have gone for the following solution. I added support for mathtex to use the wrappers provided by matplotlib for FT2Font and _png. It is likely we will want to keep them in sync anyway -- so that should not be an issue. However, I am having trouble working around the fact that mathtex is two levels deep (lib/mathtex/mathtex). I can either get the setup.py file to install either just the setup* stuff from mathtex or install mathtex as mathtex/mathtex. Not sure how to work around this. I have committed what I have thus far to the branch. Regards, Freddie.
On 2009年07月29日 15:29, Robert Kern wrote: > On 2009年07月29日 08:17, John Hunter wrote: >> On Wed, Jul 29, 2009 at 7:52 AM, Michael Droettboom<md...@st...> wrote: >>> I think we need (at least as a transitional stopgap) a single "python >>> setup.py install" to install both matplotlib and mathtex. Once >>> distributions catch up (which could take more than a year, depending on >>> cycles), we can consider being more loosely coupled. >>> >>> There are already examples of installing subpackages from matplotlib >>> (pytz and dateutil) for example, and each of the extensions >>> (_backend_agg etc.) have examples in setupext.py showing how to >>> dynamically add extensions and packages to the list of things that >>> matplotlib's setup.py will install and build. Are there fundamental >>> differences in how that works vs. how mathtex needs to work that is a >>> stumbling block? I must admit I haven't thought it all the way through, >>> but I'm surprised that the existing examples there don't provide a >>> template for how to deal with mathtex. That said, I know that distutils >>> can be rather, um, labyrinthine. >> I second this. I would like to see the new mathtext (and png and >> freetype wrappers as necessary, possibly in mathtext) live in the >> matplotlib trunk under lib, the same place we put pytz and dateutil. >> It can have its own setup and release cycle, but I think it would make >> matters simpler to have it there. If there is compelling reason to >> have it live in a separate svn repo, we can always grab it externally, >> as scipy does for our sphinx extensions. > > Instead of using the matplotlib package as a catch-all for these external > packages, could you instead keep them all nominally separate and release a > tarball that includes all of them side by side? You can even have an > installation script that will run the setups for each. Even better, this > installation script wouldn't use distutils, so you don't have to hack all of > this "optional package" stuff into it. Furthermore, you could put numpy in there, too. Maybe even recipes for downloading and building the C dependencies for those who need it. Basically, by freeing yourself from the shackles of distutils, you can make the "new-to-Python-just-give-me-matplotlib" experience much less painful while improving the situation for downstream packagers like Linux distributions, EPD, and python(x,y) at the same time. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Wed, Jul 29, 2009 at 5:05 PM, Robert Kern<rob...@gm...> wrote: > Furthermore, you could put numpy in there, too. Maybe even recipes for > downloading and building the C dependencies for those who need it. Basically, by > freeing yourself from the shackles of distutils, you can make the > "new-to-Python-just-give-me-matplotlib" experience much less painful while > improving the situation for downstream packagers like Linux distributions, EPD, > and python(x,y) at the same time. For linux, unix and os x src installs, this is imminently viable. We already have a start on this in the mpl/releases dir to autofetch the deps and build them, but this is for the lower level deps, like freetype, png and zlib. We could easily extend the approach to pytz and dateutil, and provide some enhancements (remove make, use python instead, have some sort of configure file support). The bugaboo is the win32 installer and OSX binary. If we don't build this stuff in, suddenly the dependency list grows from "just install numpy" to "install pytz, dateutil, numpy and mathtext" which is a very different picture. As I alluded to in my earlier post, I assume there is some way to build smart windows installers (eg along the lines of the ETS install tool or native win32 solutions) but this is a question of manpower. None of our core developers are primarily win32 users, and we often struggle to get anyone to even test a win32 release candidate, much less do the heavy lifting to build a smart installer. The ETS and PythonXY solutions have gotten so good that maybe I should just give up worrying making life easy for win32 users, since these solutions already satisfy the needs of someone who wants it to work with one or two clicks. I already aggressively point them to these solutions on our installing page. But windows users still make up about half of our known installations, so I don't want to be cavalier in dismissing the importance of making it easy for them.
On Wed, Jul 29, 2009 at 4:32 PM, Freddie Witherden<fr...@wi...> wrote: > After thinking about the problem a little bit I have gone for the following > solution. I added support for mathtex to use the wrappers provided by > matplotlib for FT2Font and _png. It is likely we will want to keep them in > sync anyway -- so that should not be an issue. Isn't this a bit backwards? The point of the GSOC project is to make the mathtext project available as a smaller, lighter, more widely accessible project. If it depends on mpl, what have we achieved except complexity? Don't we want the ft2font and png to be provided by mathtext, and let mpl depend on it to get the stuff there? Admittedly, mathtex may not be the right name (mpltext?) but conceptually it seems like we want the low level stuff in mathtex. Or am I missing your point here? > However, I am having trouble working around the fact that mathtex is two > levels deep (lib/mathtex/mathtex). I can either get the setup.py file to > install either just the setup* stuff from mathtex or install mathtex as > mathtex/mathtex. Not sure how to work around this. You can punt on the setup/dependency issue for now. In the branch, you can assume mathtex exists and is installed. We can worry about how to handle the setup and distribution issues once we have the package organization rationalized. A simple set of install instructions for developers with very explicit commands to check out the branch, install the deps, and test would help us test as you progress. JDH
Him On 30 Jul 2009, at 02:27, John Hunter wrote: > Isn't this a bit backwards? The point of the GSOC project is to make > the mathtext project available as a smaller, lighter, more widely > accessible project. If it depends on mpl, what have we achieved > except complexity? Don't we want the ft2font and png to be provided > by mathtext, and let mpl depend on it to get the stuff there? > Admittedly, mathtex may not be the right name (mpltext?) but > conceptually it seems like we want the low level stuff in mathtex. Or > am I missing your point here? I should have made myself clearer. Mathtex will attempt to firstly import its own versions of FT2Font and friends. Should this fail it will then attempt to import those in matplotlib. So yes, mathtex still has all of the required dependencies, nothing has changed there. This means that if you have matplotlib installed you can use/install mathtex without compiling a single C-code extension. It also means that when matplotlib wishes to install mathtex it only need bother with copying over .py files -- the same way it does for pytz. > You can punt on the setup/dependency issue for now. In the branch, > you can assume mathtex exists and is installed. We can worry about > how to handle the setup and distribution issues once we have the > package organization rationalized. A simple set of install > instructions for developers with very explicit commands to check out > the branch, install the deps, and test would help us test as you > progress. Okay, I'll revert to the previous way of doing so (cd'ing to the directory where mathtex is checked out and running setup.py there). Regards, Freddie.