Hi all, I recently run into a problem with the .matplotlib directory. I run a script as a daemon, that in its turn runs several scripts to create graphs (often the same script with different input parameters), dependent on an outside trigger. Recently, I found that these script crashed with an OSError during the 'import matplotlib' phase, at the point where it tries to check that the .matplotlib directory is writeable. The exact traceback is appended below. Searching on the mailing list, I found someone had the same problem almost a year ago ( http://sourceforge.net/mailarchive/message.php? msg_id=44964AD1.605%40yahoo.com ), but it appears that that post doesn't have a response. A wider search resulted in this post, http://osdir.com/ml/python.peak/ 2006-06/msg00019.html , where the response suggests these problems are caused by all the things mpl does in its __init__ file. A few notes on my setup: - the .matplotlib is a symbolic link to a .python/matplotlib directory, since I'd like to keep my python resources together (eg, .python also contains an ipython directory). I've currently changed the symlink to a proper directory, but haven't had a chance to test this new setup. Anyway, afaik, symlinks should work properly - the scripts that are run from the daemon script, are run in the background, and quite often shortly after each other. Whether that creates a deadlock situation or something, I don't know. Obviously, I'm using a non-interactive backend, agg in this case. - the problem only showed after I upgraded from mpl version 0.87 to 0.9. - interactively, everything runs fine. - everything runs under a locally installed python, which is called with the appropriate path from the script - python version 2.4.4, numpy version 1.0 Does anyone have a suggestion what went wrong, and how to properly fix all this? As a temporary measure, I now ignore the _is_writeable (p) result in __init__.py (the directory, and therefore the symlink, were/are writeable when this occurred, as far as I can tell). I like mpl very much, but this problem may render it much less usable for me; a temporary fix would be fine for now, but it actually feels like a bug to me. Thanks a lot, Evert full traceback (note: replaced directories with ~) Traceback (most recent call last): File "~/myscript.py", line 64, in ? sys.exit(main()) File "~/myscript.py", line 59, in main return run(parser) File "~/myscript.py", line 27, in run sigmaclip=parser['sigmaclip'], invert=True, compass=True) File "~/python/modules/fitsimage/FITSImage_numpy.py", line 777, in plot import matplotlib as mpl File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 1019, in ? rcParams = rc_params() File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 976, in rc_params fname = matplotlib_fname() File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 922, in matplotlib_fname fname = os.path.join(get_configdir(), 'matplotlibrc') File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 273, in wrapper ret = func(*args, **kwargs) File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", line 329, in _get_configdir os.mkdir(p) OSError: [Errno 17] File exists: '<...>/.matplotlib'
One way this could happen is if the .matplotlib directory exists but is not writable; such a case would give the error traceback you see with little clue as to what and where the problem really is. So, __init__.py certainly could be improved. (Offhand, I don't even know why the existence of such a writable config directory should be required.) In the meantime, to solve your particular problem: my guess is that when you run a script as a daemon, and it runs more scripts, the environment is not what you think it is (or what it is when you run interactively), so __init__.py is not looking where you think it is. Either that, or the uid/gid of the script are not the same as yours, so the script can't write to your .matplotlib directory even though you can. Instead of ignoring the _is_writable(p) result, you might want to try printing out os.path.abspath(p) to verify where p really is. Eric Evert Rol wrote: > Hi all, > > I recently run into a problem with the .matplotlib directory. I run a > script as a daemon, that in its turn runs several scripts to create > graphs (often the same script with different input parameters), > dependent on an outside trigger. Recently, I found that these script > crashed with an OSError during the 'import matplotlib' phase, at the > point where it tries to check that the .matplotlib directory is > writeable. The exact traceback is appended below. > > Searching on the mailing list, I found someone had the same problem > almost a year ago ( http://sourceforge.net/mailarchive/message.php? > msg_id=44964AD1.605%40yahoo.com ), but it appears that that post > doesn't have a response. > A wider search resulted in this post, http://osdir.com/ml/python.peak/ > 2006-06/msg00019.html , where the response suggests these problems > are caused by all the things mpl does in its __init__ file. > > A few notes on my setup: > - the .matplotlib is a symbolic link to a .python/matplotlib > directory, since I'd like to keep my python resources together > (eg, .python also contains an ipython directory). I've currently > changed the symlink to a proper directory, but haven't had a chance > to test this new setup. Anyway, afaik, symlinks should work properly > - the scripts that are run from the daemon script, are run in the > background, and quite often shortly after each other. Whether that > creates a deadlock situation or something, I don't know. Obviously, > I'm using a non-interactive backend, agg in this case. > - the problem only showed after I upgraded from mpl version 0.87 to 0.9. > - interactively, everything runs fine. > - everything runs under a locally installed python, which is called > with the appropriate path from the script > - python version 2.4.4, numpy version 1.0 > > > Does anyone have a suggestion what went wrong, and how to properly > fix all this? As a temporary measure, I now ignore the _is_writeable > (p) result in __init__.py (the directory, and therefore the symlink, > were/are writeable when this occurred, as far as I can tell). > > I like mpl very much, but this problem may render it much less usable > for me; a temporary fix would be fine for now, but it actually feels > like a bug to me. > > Thanks a lot, > > Evert > > > full traceback (note: replaced directories with ~) > > Traceback (most recent call last): > File "~/myscript.py", line 64, in ? > sys.exit(main()) > File "~/myscript.py", line 59, in main > return run(parser) > File "~/myscript.py", line 27, in run > sigmaclip=parser['sigmaclip'], invert=True, compass=True) > File "~/python/modules/fitsimage/FITSImage_numpy.py", line 777, in > plot > import matplotlib as mpl > File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", > line 1019, in ? > rcParams = rc_params() > File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", > line 976, in rc_params > fname = matplotlib_fname() > File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", > line 922, in matplotlib_fname > fname = os.path.join(get_configdir(), 'matplotlibrc') > File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", > line 273, in wrapper > ret = func(*args, **kwargs) > File "~/sw/lib/python2.4/site-packages/matplotlib/__init__.py", > line 329, in _get_configdir > os.mkdir(p) > OSError: [Errno 17] File exists: '<...>/.matplotlib' > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > Matplotlib-users mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-users
On Monday 30 April 2007 02:31:59 pm Eric Firing wrote: > One way this could happen is if the .matplotlib directory exists but is > not writable; such a case would give the error traceback you see with > little clue as to what and where the problem really is. So, __init__.py > certainly could be improved. (Offhand, I don't even know why the > existence of such a writable config directory should be required.) It is writeable because we use it for caching fonts and tex-related stuff. Don't most programs require a writeable config directory? My home directory is filled with them. Darren
Hi Eric, Thanks for your response. > One way this could happen is if the .matplotlib directory exists > but is not writable; such a case would give the error traceback you > see with little clue as to what and where the problem really is. > So, __init__.py certainly could be improved. (Offhand, I don't > even know why the existence of such a writable config directory > should be required.) I'm pretty sure the directory is writeable, as my default settings are 755 for new files/dirs (umask=022). Unless, indeed, some process changed that. But when I looked after the crash, the permissions were writeable/executable. And yes, I was somewhat surprised by the writeable requirement of the config dir as well. > In the meantime, to solve your particular problem: my guess is that > when you run a script as a daemon, and it runs more scripts, the > environment is not what you think it is (or what it is when you run > interactively), so __init__.py is not looking where you think it > is. Either that, or the uid/gid of the script are not the same as > yours, so the script can't write to your .matplotlib directory even > though you can. Instead of ignoring the _is_writable(p) result, > you might want to try printing out os.path.abspath(p) to verify > where p really is. The OSError does show it to be the correct directory, unless there's a way to fool that traceback. Also, I haven't set MATPLOTLIBRC anywhere, so I guess it picks up .matplotlibrc in my HOME (which I assume is set automatically by the system, even for daemon scripts). I'll put in the os.path.abspath output anyway. I wouldn't know 100% about the uid/gid, but everything is done under my user account (I don't have access to any other account on this system anyway), so might be hard to have that changed). Thanks again for the response. Cheers, Evert >> Hi all, >> I recently run into a problem with the .matplotlib directory. I >> run a script as a daemon, that in its turn runs several scripts >> to create graphs (often the same script with different input >> parameters), dependent on an outside trigger. Recently, I found >> that these script crashed with an OSError during the 'import >> matplotlib' phase, at the point where it tries to check that >> the .matplotlib directory is writeable. The exact traceback is >> appended below. >> Searching on the mailing list, I found someone had the same >> problem almost a year ago ( http://sourceforge.net/mailarchive/ >> message.php? msg_id=44964AD1.605%40yahoo.com ), but it appears >> that that post doesn't have a response. >> A wider search resulted in this post, http://osdir.com/ml/ >> python.peak/ 2006-06/msg00019.html , where the response suggests >> these problems are caused by all the things mpl does in its >> __init__ file. >> A few notes on my setup: >> - the .matplotlib is a symbolic link to a .python/matplotlib >> directory, since I'd like to keep my python resources together >> (eg, .python also contains an ipython directory). I've currently >> changed the symlink to a proper directory, but haven't had a >> chance to test this new setup. Anyway, afaik, symlinks should >> work properly >> - the scripts that are run from the daemon script, are run in the >> background, and quite often shortly after each other. Whether >> that creates a deadlock situation or something, I don't know. >> Obviously, I'm using a non-interactive backend, agg in this case. >> - the problem only showed after I upgraded from mpl version 0.87 >> to 0.9. >> - interactively, everything runs fine. >> - everything runs under a locally installed python, which is >> called with the appropriate path from the script >> - python version 2.4.4, numpy version 1.0 >> Does anyone have a suggestion what went wrong, and how to >> properly fix all this? As a temporary measure, I now ignore the >> _is_writeable (p) result in __init__.py (the directory, and >> therefore the symlink, were/are writeable when this occurred, as >> far as I can tell). >> I like mpl very much, but this problem may render it much less >> usable for me; a temporary fix would be fine for now, but it >> actually feels like a bug to me. >> Thanks a lot, >> Evert >> full traceback (note: replaced directories with ~) >> Traceback (most recent call last): >> File "~/myscript.py", line 64, in ? >> sys.exit(main()) >> File "~/myscript.py", line 59, in main >> return run(parser) >> File "~/myscript.py", line 27, in run >> sigmaclip=parser['sigmaclip'], invert=True, compass=True) >> File "~/python/modules/fitsimage/FITSImage_numpy.py", line 777, >> in plot >> import matplotlib as mpl >> File "~/sw/lib/python2.4/site-packages/matplotlib/ >> __init__.py", line 1019, in ? >> rcParams = rc_params() >> File "~/sw/lib/python2.4/site-packages/matplotlib/ >> __init__.py", line 976, in rc_params >> fname = matplotlib_fname() >> File "~/sw/lib/python2.4/site-packages/matplotlib/ >> __init__.py", line 922, in matplotlib_fname >> fname = os.path.join(get_configdir(), 'matplotlibrc') >> File "~/sw/lib/python2.4/site-packages/matplotlib/ >> __init__.py", line 273, in wrapper >> ret = func(*args, **kwargs) >> File "~/sw/lib/python2.4/site-packages/matplotlib/ >> __init__.py", line 329, in _get_configdir >> os.mkdir(p) >> OSError: [Errno 17] File exists: '<...>/.matplotlib'
Darren Dale wrote: > On Monday 30 April 2007 02:31:59 pm Eric Firing wrote: >> One way this could happen is if the .matplotlib directory exists but is >> not writable; such a case would give the error traceback you see with >> little clue as to what and where the problem really is. So, __init__.py >> certainly could be improved. (Offhand, I don't even know why the >> existence of such a writable config directory should be required.) > > It is writeable because we use it for caching fonts and tex-related stuff. > Don't most programs require a writeable config directory? My home directory > is filled with them. This occurred to me a couple minutes after I hit "send". Yes, this sort of thing is common, but I wouldn't say "most programs"; and it is exactly the sort of thing that can cause trouble and confusion when running a program from something like a cron job or modpython instead of the usual command line. Eric