Do we need namespace packages in toolkits? I recently added gtktools and exceltools to toolkits, and got a very hard to debug error: In [1]: import matplotlib In [2]: matplotlib.__file__ Out[2]: '/home/titan/johnh/dev/lib/python2.4/site-packages/matplotlib/__init__.pyc' In [3]: matplotlib.rcParams['axes.axisbelow'] Out[3]: False In [4]: import matplotlib.toolkits.gtktools In [5]: matplotlib.__file__ Out[5]: '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/matplotlib/__init__.pyc' In [6]: matplotlib.rcParams['axes.axisbelow'] ------------------------------------------------------------ Traceback (most recent call last): File "<ipython console>", line 1, in ? KeyError: 'axes.axisbelow' Notice that the matplotlib module which was previously imported got changed by the import of the toolkit which declared itself a namespace package. Now this may be the result of mpl not using namespace packages correctly (eg matplotlib/__init__.py should be empty save for the namespace package or something like that) but I gotta tell ya, this looks just plain wrong. Had I not known that namespace packages were funky like this, it could have taken me a long time to trace this bug. I am going to comment out the toolkits namespace package code until I hear persuasive arguments to the contrary, and until we fix the rest of mpl to work properly with them. JDH
On Dec 13, 2007 11:18 AM, John Hunter <jd...@gm...> wrote: > Do we need namespace packages in toolkits? I recently added gtktools > and exceltools to toolkits, and got a very hard to debug error: Welcome to the wonderful world of setuptools' under-the-covers magic :) Cheers, f
Fernando Perez wrote: > On Dec 13, 2007 11:18 AM, John Hunter <jd...@gm...> wrote: >> Do we need namespace packages in toolkits? I recently added gtktools >> and exceltools to toolkits, and got a very hard to debug error: > > Welcome to the wonderful world of setuptools' under-the-covers magic :) Fernando, Was this also the culprit that you and someone from enthought identified as causing the slow startup when using traits? This was a while back when I was complaining about a big startup time increase with Darren's traits-enabled configuration scheme. Eric
On Dec 13, 2007 11:47 AM, Eric Firing <ef...@ha...> wrote: > > Fernando Perez wrote: > > On Dec 13, 2007 11:18 AM, John Hunter <jd...@gm...> wrote: > >> Do we need namespace packages in toolkits? I recently added gtktools > >> and exceltools to toolkits, and got a very hard to debug error: > > > > Welcome to the wonderful world of setuptools' under-the-covers magic :) > > Fernando, > > Was this also the culprit that you and someone from enthought identified > as causing the slow startup when using traits? This was a while back > when I was complaining about a big startup time increase with Darren's > traits-enabled configuration scheme. Kind of, in a general sense. As best as we understood it from simple profiling runs, the issue was that when NS packages are enabled, the number of filesystem calls made in path searches explodes. That's obviously always going to be a problem, but it's particularly concerning for those of us (like John and myself) who live in environments where most things are network-mounted disks (NFS, typically). Tens of thousands of path searches over the network can really cause a hit. But setuptools is complex enough that I don't want to badmouth it without a very precise analysis of the real root causes of the problems, which I don't have time for right now. I just know that using it well will require some understanding of its behavior, which is a bit 'magic' at times. Cheers, f
John Hunter wrote: > Do we need namespace packages in toolkits? I recently added gtktools > and exceltools to toolkits, and got a very hard to debug error: > > In [1]: import matplotlib > > In [2]: matplotlib.__file__ > Out[2]: '/home/titan/johnh/dev/lib/python2.4/site-packages/matplotlib/__init__.pyc' > > In [3]: matplotlib.rcParams['axes.axisbelow'] > Out[3]: False > > In [4]: import matplotlib.toolkits.gtktools > > In [5]: matplotlib.__file__ > Out[5]: '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/matplotlib/__init__.pyc' > > In [6]: matplotlib.rcParams['axes.axisbelow'] > ------------------------------------------------------------ > Traceback (most recent call last): > File "<ipython console>", line 1, in ? > KeyError: 'axes.axisbelow' > > Notice that the matplotlib module which was previously imported got > changed by the import of the toolkit which declared itself a namespace > package. Now this may be the result of mpl not using namespace > packages correctly (eg matplotlib/__init__.py should be empty save for > the namespace package or something like that) Probably. In setuptools 0.7, they can have stuff in them, but all of them must be identical. If you are using an unmodified matplotlib/__init__.py in the main matplotlib package, then the "if 0:" is preventing the declaration of the namespace package, and that may be contributing. Also, neither matplotlib nor matplotlib.toolkits is named as a namespace_package in matplotlib's setup.py, and both have to be. Similarly, basemap (for example) only lists matplotlib.toolkits but not matplotlib. > but I gotta tell ya, > this looks just plain wrong. Had I not known that namespace packages > were funky like this, it could have taken me a long time to trace this > bug. Well, when you do exactly what you were told not to do, this is what happens. Don't make me break out the "Doctor, it hurts when I do this!" bromide. If you don't want to bother to learn to use namespace packages correctly, that's perfectly fine; just don't use them. But certainly don't blame the tools for your mistake. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Thursday 13 December 2007 2:22:13 pm Robert Kern wrote: > John Hunter wrote: > > Do we need namespace packages in toolkits? I recently added gtktools > > and exceltools to toolkits, and got a very hard to debug error: > > > > In [1]: import matplotlib > > > > In [2]: matplotlib.__file__ > > Out[2]: > > '/home/titan/johnh/dev/lib/python2.4/site-packages/matplotlib/__init__.py > >c' > > > > In [3]: matplotlib.rcParams['axes.axisbelow'] > > Out[3]: False > > > > In [4]: import matplotlib.toolkits.gtktools > > > > In [5]: matplotlib.__file__ > > Out[5]: > > '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/matplotlib/__ini > >t__.pyc' > > > > In [6]: matplotlib.rcParams['axes.axisbelow'] > > ------------------------------------------------------------ > > Traceback (most recent call last): > > File "<ipython console>", line 1, in ? > > KeyError: 'axes.axisbelow' > > > > Notice that the matplotlib module which was previously imported got > > changed by the import of the toolkit which declared itself a namespace > > package. Now this may be the result of mpl not using namespace > > packages correctly (eg matplotlib/__init__.py should be empty save for > > the namespace package or something like that) > > Probably. In setuptools 0.7, they can have stuff in them, but all of them > must be identical. How would one learn about this new feature? According to setuptools.txt from the 0.7 svn checkout, the namespace package's __init__.py must still be empty save the namespace declaration. > If you are using an unmodified matplotlib/__init__.py in > the main matplotlib package, then the "if 0:" is preventing the declaration > of the namespace package, and that may be contributing. > > Also, neither matplotlib nor matplotlib.toolkits is named as a > namespace_package in matplotlib's setup.py, and both have to be. Similarly, > basemap (for example) only lists matplotlib.toolkits but not matplotlib. If I understand correctly, even after fixing these two problems, namespace packages would not work since matplotlib/__init__.py contains lots of meaningful code. Based on the remarks at http://mail.python.org/pipermail/distutils-sig/2007-October/008346.html and the setuptools svn commit log, it looks like development on version 0.7 is not a priority and a release in the next year is unlikely. So is it fair to say that for the foreseeable future, matplotlib is not compatible with namespace packages unless we cleaned out __init__.py? Darren
Darren Dale wrote: > On Thursday 13 December 2007 2:22:13 pm Robert Kern wrote: >> John Hunter wrote: >>> Do we need namespace packages in toolkits? I recently added gtktools >>> and exceltools to toolkits, and got a very hard to debug error: >>> >>> In [1]: import matplotlib >>> >>> In [2]: matplotlib.__file__ >>> Out[2]: >>> '/home/titan/johnh/dev/lib/python2.4/site-packages/matplotlib/__init__.py >>> c' >>> >>> In [3]: matplotlib.rcParams['axes.axisbelow'] >>> Out[3]: False >>> >>> In [4]: import matplotlib.toolkits.gtktools >>> >>> In [5]: matplotlib.__file__ >>> Out[5]: >>> '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/matplotlib/__ini >>> t__.pyc' >>> >>> In [6]: matplotlib.rcParams['axes.axisbelow'] >>> ------------------------------------------------------------ >>> Traceback (most recent call last): >>> File "<ipython console>", line 1, in ? >>> KeyError: 'axes.axisbelow' >>> >>> Notice that the matplotlib module which was previously imported got >>> changed by the import of the toolkit which declared itself a namespace >>> package. Now this may be the result of mpl not using namespace >>> packages correctly (eg matplotlib/__init__.py should be empty save for >>> the namespace package or something like that) >> Probably. In setuptools 0.7, they can have stuff in them, but all of them >> must be identical. > > How would one learn about this new feature? According to setuptools.txt from > the 0.7 svn checkout, the namespace package's __init__.py must still be empty > save the namespace declaration. It's a logical consequence of how namespace packages are loaded in 0.7 as has been explained on the mailing list whenever the subject of namespace packages in 0.7 is brought up. It's still not recommended, but it can be made to work. The remaining objection in the documentation is about installing the package using --single-version-externally-managed for distro packagers. Using --svem, the __init__.py files are not installed. Since a distro package would have a mynamespace-common package (or something similar) that *just* provides this __init__.py file for all of the mynamespace.foo packages, it can provide the single copy of the meaningful code. But it's still not a good idea. >> If you are using an unmodified matplotlib/__init__.py in >> the main matplotlib package, then the "if 0:" is preventing the declaration >> of the namespace package, and that may be contributing. >> >> Also, neither matplotlib nor matplotlib.toolkits is named as a >> namespace_package in matplotlib's setup.py, and both have to be. Similarly, >> basemap (for example) only lists matplotlib.toolkits but not matplotlib. > > If I understand correctly, even after fixing these two problems, namespace > packages would not work since matplotlib/__init__.py contains lots of > meaningful code. Based on the remarks at > http://mail.python.org/pipermail/distutils-sig/2007-October/008346.html and > the setuptools svn commit log, it looks like development on version 0.7 is > not a priority and a release in the next year is unlikely. So is it fair to > say that for the foreseeable future, matplotlib is not compatible with > namespace packages unless we cleaned out __init__.py? Yes. Phillip explained this in February and suggested a way around the issue: http://www.mail-archive.com/dis...@py.../msg02908.html -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco