Hi all,
I just had this crash in mpl:
/usr/lib/python2.3/site-packages/matplotlib/texmanager.py in
get_dvipng_version(self)
299 def get_dvipng_version(self):
300 if self.dvipngVersion is not None: return
self.dvipngVersion
--> 301 sin, sout = os.popen2('dvipng --version')
302 for line in sout.readlines():
303 if line.startswith('dvipng '):
/usr/lib/python2.3/os.py in popen2(cmd, mode, bufsize)
614 def popen2(cmd, mode="t", bufsize=-1):
615 import popen2
--> 616 stdout, stdin = popen2.popen2(cmd, bufsize)
617 return stdin, stdout
618 __all__.append("popen2")
/usr/lib/python2.3/popen2.py in popen2(cmd, bufsize, mode)
145 specified, it sets the buffer size for the I/O pipes.
The file objects
146 (child_stdout, child_stdin) are returned."""
--> 147 inst = Popen3(cmd, False, bufsize)
148 return inst.fromchild, inst.tochild
149
/usr/lib/python2.3/popen2.py in __init__(self, cmd, capturestderr, bufsize)
40 if capturestderr:
41 errout, errin = os.pipe()
---> 42 self.pid = os.fork()
43 if self.pid == 0:
44 # Child
OSError: [Errno 12] Cannot allocate memory
This was in a long-running session with very large memory allocations,
but it dawned on me that get_dvipng_version(self) should cache its
return value. There's no point in forcing a popen() call every single
time, is there?
I don't know the texmanager code, so I don't want to touch it myself.
But if the idea is OK, a simple property tag is enough to cache:
In [3]: def cached():
...: try:
...: return cached.cachedvalue
...: except AttributeError:
...: cached.cachedvalue=42
...: return cached.cachedvalue
...:
In [4]: cached()
Out[4]: 42
I also just saw pylab crash when a user was trying to run with $PWD
being something he didn't have write access to. Are there any checks in
the code to fall back to /tmp or something sensible if texmanager can't
write the temp files it needs? Sorry for not giving a traceback, but I
only saw this on someone else's screen while helping them, and for some
odd reason it's not happening on my box.
Cheers,
f
>>>>> "Fernando" == Fernando Perez <Fer...@co...> writes:
Fernando> This was in a long-running session with very large
Fernando> memory allocations, but it dawned on me that
Fernando> get_dvipng_version(self) should cache its return value.
Fernando> There's no point in forcing a popen() call every single
Fernando> time, is there?
It already does cache the version
def get_dvipng_version(self):
if self.dvipngVersion is not None: return self.dvipngVersion
sin, sout = os.popen2('dvipng --version')
...snip...
I think the reason it is failing in your case is that is each renderer
creation creates a new TexManager instance. The cache should be on a
class basis and not on a per instance basis (slaps self on head).
Moving
self.dvipngVersion = None
to the class level and out of __init__ should suffice. I made this
change, and similar changes in texmanager and backend_agg, so that
caching is shared between instances. Update from CVS and let me know
if everything still works :-)
Fernando> I also just saw pylab crash when a user was trying to
Fernando> run with $PWD being something he didn't have write
Fernando> access to. Are there any checks in the code to fall
Fernando> back to /tmp or something sensible if texmanager can't
Fernando> write the temp files it needs? Sorry for not giving a
Fernando> traceback, but I only saw this on someone else's screen
Fernando> while helping them, and for some odd reason it's not
Fernando> happening on my box.
This is probably failing on the tex/latex temporary files. I spent
some time initially trying to figure out how to get these to go into
~/.tex.cache but didn't succeed. If anyone knows how to direct
tex/latex to put the various *.aux, *.log, etc, files in a specified
directory, pass it my way.
Thanks,
JDH
John Hunter wrote:
>>>>>>"Fernando" == Fernando Perez <Fer...@co...> writes:
>
>
> Fernando> This was in a long-running session with very large
> Fernando> memory allocations, but it dawned on me that
> Fernando> get_dvipng_version(self) should cache its return value.
> Fernando> There's no point in forcing a popen() call every single
> Fernando> time, is there?
>
> It already does cache the version
>
> def get_dvipng_version(self):
> if self.dvipngVersion is not None: return self.dvipngVersion
> sin, sout = os.popen2('dvipng --version')
> ...snip...
>
> I think the reason it is failing in your case is that is each renderer
> creation creates a new TexManager instance. The cache should be on a
> class basis and not on a per instance basis (slaps self on head).
> Moving
>
> self.dvipngVersion = None
>
> to the class level and out of __init__ should suffice. I made this
> change, and similar changes in texmanager and backend_agg, so that
> caching is shared between instances. Update from CVS and let me know
> if everything still works :-)
swamped at the moment, but I'll let you know if I see problems when I
get a chance.
> Fernando> I also just saw pylab crash when a user was trying to
> Fernando> run with $PWD being something he didn't have write
> Fernando> access to. Are there any checks in the code to fall
> Fernando> back to /tmp or something sensible if texmanager can't
> Fernando> write the temp files it needs? Sorry for not giving a
> Fernando> traceback, but I only saw this on someone else's screen
> Fernando> while helping them, and for some odd reason it's not
> Fernando> happening on my box.
>
> This is probably failing on the tex/latex temporary files. I spent
> some time initially trying to figure out how to get these to go into
> ~/.tex.cache but didn't succeed. If anyone knows how to direct
> tex/latex to put the various *.aux, *.log, etc, files in a specified
> directory, pass it my way.
I don't really know, sorry. My texpertise is pretty limited, as you know :)
Thanks for the fixes!
f
John Hunter wrote: >This is probably failing on the tex/latex temporary files. I spent >some time initially trying to figure out how to get these to go into >~/.tex.cache but didn't succeed. If anyone knows how to direct >tex/latex to put the various *.aux, *.log, etc, files in a specified >directory, pass it my way. > > From "man tex" on FC3, the section on environment variables: TEXMFOUTPUT Normally, TeX puts its output files in the current directory. If any output file cannot be opened there, it tries to open it in the directory specified in the environment variable TEXM- FOUTPUT. There is no default value for that variable. For example, if you say tex paper and the current directory is not writable, if TEXMFOUTPUT has the value /tmp, TeX attempts to create /tmp/paper.log (and /tmp/paper.dvi, if any output is produced.)
>>>>> "Stephen" == Stephen Walton <ste...@cs...> writes:
Stephen> TEXMFOUTPUT Normally, TeX puts its output files in
Stephen> the current directory. If any output file cannot be
Stephen> opened there, it tries to open it in the directory
Stephen> specified in the environment variable TEXM- FOUTPUT.
Stephen> There is no default value for that variable. For
Stephen> example, if you say tex paper and the current directory
Stephen> is not writable, if TEXMFOUTPUT has the value /tmp, TeX
Stephen> attempts to create /tmp/paper.log (and /tmp/paper.dvi, if
Stephen> any output is produced.)
Hmm, that looks familiar -- I think I did come across that. But this
will be a bit of a pain to set from matplotlib, since we are making
calls to os.popen. We might have to do something like
os.popen('export TEXMFOUTPUT=/some/path; run some command')
and then deal with platform and shell dependent differences in how to
set environment variables. Is there a better way? Me thinks it is
better to make this a FAQ and advise people working in non-writable
dirs to set this var themselves.
Thanks for the help...
JDH
John Hunter wrote:
>We might have to do something like
>
> os.popen('export TEXMFOUTPUT=/some/path; run some command')
>
>and then deal with platform and shell dependent differences in how to
>set environment variables. Is there a better way? Me thinks it is
>better to make this a FAQ and advise people working in non-writable
>dirs to set this var themselves.
>
I agree with this last. On Unix, you could do
os.popen('/bin/sh -c "export TEXMFOUTPUT=/some/path; run some command"')
and be pretty sure it would work on all systems. But Windows would be a
pain. I agree with the FAQ solution. This seems like a very rare
problem to me, to be honest.
Stephen Walton wrote:
> John Hunter wrote:
>
>
>>We might have to do something like
>>
>> os.popen('export TEXMFOUTPUT=/some/path; run some command')
>>
>>and then deal with platform and shell dependent differences in how to
>>set environment variables. Is there a better way? Me thinks it is
>>better to make this a FAQ and advise people working in non-writable
>>dirs to set this var themselves.
>>
>
> I agree with this last. On Unix, you could do
>
> os.popen('/bin/sh -c "export TEXMFOUTPUT=/some/path; run some command"')
>
> and be pretty sure it would work on all systems. But Windows would be a
> pain. I agree with the FAQ solution. This seems like a very rare
> problem to me, to be honest.
It may be simpler than that:
planck[~/test]> cat env.py
#!/usr/bin/env python
import os
print 'before'
os.system('env | grep TEXMFOUTPUT')
os.environ['TEXMFOUTPUT'] = '/some/path'
print 'after'
os.system('env | grep TEXMFOUTPUT')
planck[~/test]> ./env.py
before
after
TEXMFOUTPUT=/some/path
planck[~/test]>
Cheers,
f
Fernando Perez wrote: > It may be simpler than that: So os.environ does change the environment under Python and its children. The remaining question is what should TEXMFOUTPUT be? /home/user/.tex.cache will not work across all platforms.
Stephen Walton wrote: > Fernando Perez wrote: > > >>It may be simpler than that: > > > So os.environ does change the environment under Python and its > children. The remaining question is what should TEXMFOUTPUT be? > /home/user/.tex.cache will not work across all platforms. Well, it could be something like $HOME/.tex.cache, where $HOME can be determined via a routine like the below (this is what ipython uses to try and guess a sensible value for $HOME): #---------------------------------------------------------------------------- class HomeDirError(Error): pass def get_home_dir(): """Return the closest possible equivalent to a 'home' directory. We first try $HOME. Absent that, on NT it's $HOMEDRIVE\$HOMEPATH. Currently only Posix and NT are implemented, a HomeDirError exception is raised for all other OSes. """ #' try: return os.environ['HOME'] except KeyError: if os.name == 'posix': raise HomeDirError,'undefined $HOME, IPython can not proceed.' elif os.name == 'nt': # For some strange reason, win9x returns 'nt' for os.name. try: return os.path.join(os.environ['HOMEDRIVE'],os.environ['HOMEPATH']) except: try: # Use the registry to get the 'My Documents' folder. import _winreg as wreg key = wreg.OpenKey(wreg.HKEY_CURRENT_USER, "Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders") homedir = wreg.QueryValueEx(key,'Personal')[0] key.Close() return homedir except: return 'C:\\' elif os.name == 'dos': # Desperate, may do absurd things in classic MacOS. May work under DOS. return 'C:\\' else: raise HomeDirError,'support for your operating system not implemented.' Cheers, f
Fernando Perez wrote: > Well, it could be something like $HOME/.tex.cache, where $HOME can be > determined via a routine like the below (this is what ipython uses to > try and guess a sensible value for $HOME): I *like* it.
Stephen Walton wrote: > Fernando Perez wrote: > > >>Well, it could be something like $HOME/.tex.cache, where $HOME can be >>determined via a routine like the below (this is what ipython uses to >>try and guess a sensible value for $HOME): > > > I *like* it. Though I'd personally vote for matplotlib holding $HOME/.matplotlib/ as a directory, and putting in there a tex.cache dir, the matplotlibrc file, and anything else it may need in the future. I *really* don't like the idea that matplotlib will begin to put a bunch of differently named things under $HOME with various .foo names. Ipython also started with .ipythonrc, and I quickly moved to an .ipython/ directory, where I stuff anything I need. It's future-proof, clean, and gives an easy way for users to clean up if needed, without having to guess 'how many dot files/directories does this thing create'? So if you were to ask for my opinion, I'd vote +100 on matplotlib moving to a single directory for holding *ALL* user and configuration data, which would default to $HOME/.matplotlib, and which could be overridden if the $MATPLOTLIBDIR environment variable is defined. That's exactly how ipython works, so it must be the perfect solution :) Cheers, f
On Wednesday 22 June 2005 06:35 pm, Fernando Perez wrote: > Stephen Walton wrote: > > Fernando Perez wrote: > >>Well, it could be something like $HOME/.tex.cache, where $HOME can be > >>determined via a routine like the below (this is what ipython uses to > >>try and guess a sensible value for $HOME): > > > > I *like* it. > > Though I'd personally vote for matplotlib holding $HOME/.matplotlib/ as a > directory, and putting in there a tex.cache dir, the matplotlibrc file, and > anything else it may need in the future. I second that notion. In fact, I have been meaning to suggest this on the list for some time now. Thank for reminding me. > I *really* don't like the idea that matplotlib will begin to put a bunch of > differently named things under $HOME with various .foo names. Ipython also > started with .ipythonrc, and I quickly moved to an .ipython/ directory, > where I stuff anything I need. It's future-proof, clean, and gives an easy > way for users to clean up if needed, without having to guess 'how many dot > files/directories does this thing create'? > > So if you were to ask for my opinion, I'd vote +100 on matplotlib moving to > a single directory for holding *ALL* user and configuration data, which > would default to $HOME/.matplotlib, and which could be overridden if the > $MATPLOTLIBDIR environment variable is defined. That's exactly how ipython > works, so it must be the perfect solution :) +101 Darren
>>>>> "Fernando" == Fernando Perez <Fer...@co...> writes:
Fernando> Stephen Walton wrote:
>> Fernando Perez wrote:
>>> Well, it could be something like $HOME/.tex.cache, where $HOME
>>> can be determined via a routine like the below (this is what
>>> ipython uses to try and guess a sensible value for $HOME):
>> I *like* it.
Fernando> Though I'd personally vote for matplotlib holding
Fernando> $HOME/.matplotlib/ as a directory, and putting in there
Fernando> a tex.cache dir, the matplotlibrc file, and anything
Fernando> else it may need in the future.
I am in the process of moving all of matplotlib's config files and
outputs to HOME/.matplotlib.
I want to issue a warning to users if I find a .matplotlibrc file (the
new name is matplotlibrc (no hidden "dot") and the default location is
HOME/.matplotlib/matplotlibrc. As before, you can put an rc file in
the current dir.
I am using warnings.warn to issue the warning
home = get_home()
oldname = os.path.join( home, '.matplotlibrc')
if os.path.exists(oldname):
warnings.warn('Old rc filename "%s" found and ignored; new default rc file name is HOME/.matplotlib/matplotlibrc"'%oldname)
But this results in a warning that looks like this
> python ~/python/projects/matplotlib/examples/subplot_demo.py
/usr/lib/python2.4/site-packages/matplotlib/__init__.py:737: UserWarning: Old rc filename "/home/jdhunter/.matplotlibrc" found and ignored; new default rc file name is HOME/.matplotlib/matplotlibrc"
warnings.warn('Old rc filename "%s" found and ignored; new default rc file name is HOME/.matplotlib/matplotlibrc"'%oldname)
which I find hard to read. I can add a stacklevel=0 argument to warn
which is slightly more legible
> python ~/python/projects/matplotlib/examples/subplot_demo.py
/usr/lib/python2.4/warnings.py:41: UserWarning: Old rc filename "/home/jdhunter/.matplotlibrc" found and ignored; new default rc file name is HOME/.matplotlib/matplotlibrc"
lineno = caller.f_lineno
but still has the annoying "lineno = caller.f_lineno"
what I am really after is a simple print >> sys.stderr. Is there
anything wrong with using sys.stderr for this kind of thing, error
mavens? I know when we discussed this many moons ago we converged on
using exceptions and the warnings module everywhere. But now I am a
bit confounded by the inability to make the warnings print pretty
messages that the typical user will read and understand. Or is there
a way to tell warning.warn to just print the error message with no
lineno/traceback info?
JDH
John Hunter wrote: >>>>>>"Fernando" == Fernando Perez <Fer...@co...> writes: > > > Fernando> Stephen Walton wrote: > >> Fernando Perez wrote: > >>> Well, it could be something like $HOME/.tex.cache, where $HOME > >>> can be determined via a routine like the below (this is what > >>> ipython uses to try and guess a sensible value for $HOME): > >> I *like* it. > > Fernando> Though I'd personally vote for matplotlib holding > Fernando> $HOME/.matplotlib/ as a directory, and putting in there > Fernando> a tex.cache dir, the matplotlibrc file, and anything > Fernando> else it may need in the future. > > I am in the process of moving all of matplotlib's config files and > outputs to HOME/.matplotlib. Great! > what I am really after is a simple print >> sys.stderr. Is there > anything wrong with using sys.stderr for this kind of thing, error > mavens? I know when we discussed this many moons ago we converged on > using exceptions and the warnings module everywhere. But now I am a > bit confounded by the inability to make the warnings print pretty > messages that the typical user will read and understand. Or is there > a way to tell warning.warn to just print the error message with no > lineno/traceback info? Dunno, sorry. In ipython, I have a trivial warn() routine which is just a print >> sys.stderr wrapper. I've never used the stdlib's warnings module. Cheers, f
On Wed, 2005年06月22日 at 11:45 -0600, Fernando Perez wrote: > os.environ['TEXMFOUTPUT'] = '/some/path' According to the online docs (http://docs.python.org/lib/os-procinfo.html) setting os.environ isn't safe/available for all platforms. You can use the subprocess module to set the environment of a subprocess under python 2.4 but I don't think there's a simple way to do this and capture the output for earlier versions. Nick
On Jun 21, 2005, at 1:09 PM, John Hunter wrote: > Fernando> I also just saw pylab crash when a user was trying to > Fernando> run with $PWD being something he didn't have write > Fernando> access to. Are there any checks in the code to fall > Fernando> back to /tmp or something sensible if texmanager can't > Fernando> write the temp files it needs? Sorry for not giving a > Fernando> traceback, but I only saw this on someone else's screen > Fernando> while helping them, and for some odd reason it's not > Fernando> happening on my box. > > This is probably failing on the tex/latex temporary files. I spent > some time initially trying to figure out how to get these to go into > ~/.tex.cache but didn't succeed. If anyone knows how to direct > tex/latex to put the various *.aux, *.log, etc, files in a specified > directory, pass it my way. I get this when mpl complains about not being able to create a .ttffont.cache (or whatever it's called) file. I'll submit a bug on the tracker later. Cheers! Andrew
Nicholas Young wrote: > On Wed, 2005年06月22日 at 11:45 -0600, Fernando Perez wrote: > >>os.environ['TEXMFOUTPUT'] = '/some/path' > > > According to the online docs > (http://docs.python.org/lib/os-procinfo.html) setting os.environ isn't > safe/available for all platforms. You can use the subprocess module to > set the environment of a subprocess under python 2.4 but I don't think > there's a simple way to do this and capture the output for earlier > versions. Well, after reading that I get that os.environ _is_ writable everywhere, it's just that it may leak memory in OSX/BSD. What's not always available is the putenv() call, but python will find its way around it if needed. Since this would be a once-only call, I think that leaking a few bytes is an acceptable price to pay to prevent a crash if the user happens to be positioned on a non-writable dir. Cheers, f
On Thu, 2005年06月23日 at 10:58 -0600, Fernando Perez wrote: > Nicholas Young wrote: > > On Wed, 2005年06月22日 at 11:45 -0600, Fernando Perez wrote: > > > >>os.environ['TEXMFOUTPUT'] = '/some/path' > > > > > > According to the online docs > > (http://docs.python.org/lib/os-procinfo.html) setting os.environ isn't > > safe/available for all platforms. You can use the subprocess module to > > set the environment of a subprocess under python 2.4 but I don't think > > there's a simple way to do this and capture the output for earlier > > versions. > > Well, after reading that I get that os.environ _is_ writable everywhere, it's > just that it may leak memory in OSX/BSD. What's not always available is the > putenv() call, but python will find its way around it if needed. To quote "If putenv() is not provided, this mapping may be passed to the appropriate process-creation functions to cause child processes to use a modified environment.". To me this implies that you have to pass os.environ to a process-creation function supporting the env keyword (os.execve, etc.) none of which seem to support capturing output. On the other hand does anyone actually run mpl on a platform which doesn't support putenv? > Since this would be a once-only call, I think that leaking a few bytes is an > acceptable price to pay to prevent a crash if the user happens to be > positioned on a non-writable dir. After reading an online copy of the freebsd man putenv it actually reads "Successive calls to setenv() or putenv() assigning a differently sized value to the same name will result in a memory leak." so setting this once wouldn't be a problem. So I was wrong and setting os.environ seems reasonable in this case - but it seems sensible to be aware of the potential for causing problems if anyone trys to get mpl working on an odd platform. Nick