Matplotlib is quite nice, but I keep running into problems with the actual display interface. In some cases the plot window freezes and such when using it from a debugger, or sometimes pylab.show() just never returns. Matlab's plotting interface just never gave me so much trouble. I think all these problems could be fixed if the display interface were turned into a separate process that communicates with the Python program using pipes or some other IPC mechanism. I used this technique in a (C/C++) image debugging utility program I wrote (http://www.billbaxter.com/projects/imdebug) and it works quite well. The displayer is a separate process using a memory mapped file to communicate with the main program. When the user's code calls the "display_image" function, the data to display is copied to the memory mapped file, the displayer process is sent a signal, and then the displayer reads the data from the memory mapped file and displays it. Because the displayer is a separate process, it never hangs, even if the main app crashes or gets stuck at a break point or in an infinite loop. And the displayer always owns its display loop completely, so no nasty issues with who gets to run the wx.App event loop and when, or with the single-gui-thread limitations you have with multithreaded GUI apps. Has anyone given thought to making matplotlib work in such a manner? It would be hell to do in C or C++ I think, but with Python's extensive RPC libs I bet it wouldn't be so bad. Also, once you have that sort of remote operation set up, you can think about crazy things like just using a web browser as the display interface. Or remotely displaying graphs on other machines, if you have some reason to want to do that. --bb
Matplotlib is quite nice, but I keep running into problems with the actual display interface. In some cases the plot window freezes and such when using it from a debugger, or sometimes pylab.show() just never returns. Matlab's plotting interface just never gave me so much trouble. I think all these problems could be fixed if the display interface were turned into a separate process that communicates with the Python program using pipes or some other IPC mechanism. I used this technique in a (C/C++) image debugging utility program I wrote (http://www.billbaxter.com/projects/imdebug) and it works quite well. The displayer is a separate process using a memory mapped file to communicate with the main program. When the user's code calls the "display_image" function, the data to display is copied to the memory mapped file, the displayer process is sent a signal, and then the displayer reads the data from the memory mapped file and displays it. Because the displayer is a separate process, it never hangs, even if the main app crashes or gets stuck at a break point or in an infinite loop. And the displayer always owns its display loop completely, so no nasty issues with who gets to run the wx.App event loop and when, or with the single-gui-thread limitations you have with multithreaded GUI apps. Has anyone given thought to making matplotlib work in such a manner? It would be hell to do in C or C++ I think, but with Python's RPC libs I bet it wouldn't be so bad. --bb
Bill Baxter wrote: > I think all these problems could be fixed if the display interface > were turned into a separate process Have you tried ipython? http://ipython.scipy.org/
Hi Stephen, Yep, ipython is not bad, but it is not really a replacement for a real IDE. IPython also seems to act a little wanky with graphs to me. For instance my plots seem to get drawn interactively (read: very SLOOOOOWLY) when I use the special -pylab mode. Maybe I'm not configuring ipython properly, but part of my point is that end users shouldn't have to think about it, and shouldn't need some special -pylab mode. It should just work. And I think it would if the displayer's guts were in a completely separate process, acting as a graph display server. The only complication I can see is for callbacks from mouse and keyboard events that occur on the graphs (but does matplotlib even support that yet?-- I only saw it mentioned on the web page). Those events would still need to find their way into callbacks in the original process. But that's doable too, I think. Just use a separate thread for communication with the graph display server. And would perhaps be even less painful than dealing with the wx event loop. Anyway, it's more of a 'food for thought' suggestion than anything else. It's not like I'm going to have time to implement it (though it seems like it would be a fun project if I did have the time). I am curious as to what the current thinking is about tacking such event loop issues, though. Surely folks don't think that "use ipython" is the be-all-and-end-all ultimate solution. --bb On 7/26/06, Stephen Walton <ste...@cs...> wrote: > Bill Baxter wrote: > > I think all these problems could be fixed if the display interface > > were turned into a separate process > Have you tried ipython? > > http://ipython.scipy.org/
On Jul 24, 2006, at 9:16 PM, Bill Baxter wrote: > I think all these problems could be fixed if the display interface > were turned into a separate process that communicates with the Python > program using pipes or some other IPC mechanism. I used this > technique in a (C/C++) image debugging utility program I wrote > (http://www.billbaxter.com/projects/imdebug) and it works quite well. The biggest problem I see with this approach is that matplotlib has a "display list" drawing model (as opposed to the "big matrix of pixels" model). An example of what I mean is that when someone resizes a {Gtk,Tk,Wx}Agg figure, the entire plot is re-rendered from scratch to the new pixel dimensions. It could be a failure of imagination, but I can't see how you could move the display interface to a subprocess without moving the rest of matplotlib along with it. I just can't see a clear line along which to separate out the "graph display server" part. > Has anyone given thought to making matplotlib work in such a manner? > It would be hell to do in C or C++ I think, but with Python's > extensive RPC libs I bet it wouldn't be so bad. Python has extensive RPC libraries?!?! ;-) In seriousness, RPC is an acknowledged weakness of Python's standard library. For example, Python gives you a sockets API that is much nicer to use than the BSD sockets API or WinSock, but which still makes you worry about all of the platform-specific idiosyncrasies of socket programming. It's my understanding that this is one of the big reasons Twisted came into being. As another example, before Python 2.4 added the "subprocess" module, there was no portable way to spawn and communicate with subprocesses... you'd have to worry about the shell of whatever platform you were on mangling the commandline, you couldn't reliably interrupt subprocesses on win32 (no os.kill() to send a SIGINT with), etc. The point of this long-winded email is that you probably could build some kind of RPC system to run matplotlib remotely, but I believe it would be an awful lot of work. In terms of manhours, my opinion is that you'd come out ahead if you just focused on debugging your current problems. I suspect that the matplotlib developers and irregular contributers like myself will more able to help with that debugging than we would be able to help with writing a display server... but please don't think that I'm speaking for everyone! Ken
Hi Ken, Thanks for the reply. On 7/31/06, Ken McIvor <mc...@ii...> wrote: > On Jul 24, 2006, at 9:16 PM, Bill Baxter wrote: > > I think all these problems could be fixed if the display interface > > were turned into a separate process that communicates with the Python > > program using pipes or some other IPC mechanism. I used this > > technique in a (C/C++) image debugging utility program I wrote > > (http://www.billbaxter.com/projects/imdebug) and it works quite well. > > The biggest problem I see with this approach is that matplotlib has a > "display list" drawing model (as opposed to the "big matrix of > pixels" model). An example of what I mean is that when someone > resizes a {Gtk,Tk,Wx}Agg figure, the entire plot is re-rendered from > scratch to the new pixel dimensions. It could be a failure of > imagination, but I can't see how you could move the display interface > to a subprocess without moving the rest of matplotlib along with it. > I just can't see a clear line along which to separate out the "graph > display server" part. What's the problem with sending the "display list" or with having most of matplotlib also exist in the separate process? As long as evaluating that display list doesn't involve making extensive callbacks into user code, then it shouldn't be a problem. Having matplotlib live on both sides of the process boundary is ok in my opinion, because it can be made to work that way. But having to have the *users's* app on both sides is a problem. Or having to do so many RPC callbacks back to users' code that a single rendering becomes intolerably slow. Conceptually, in terms of the "clear line", I think maybe a special 'axes' class would get you a lot of what is needed. That would act like a proxy for 'axes' actually living in another process. Anyway, clearly matplotlib shouldn't always use remote display. If you're using for embedded graphing then that would be silly. It would be more for general interactive use and debugging purposes. > > Has anyone given thought to making matplotlib work in such a manner? > > It would be hell to do in C or C++ I think, but with Python's > > extensive RPC libs I bet it wouldn't be so bad. > > Python has extensive RPC libraries?!?! ;-) Heh, well compared to C++ at least. :-) My only experience with it was with doing some very simple operations using xmlrpc. But compared with how hard that would have been for me to do in C++ I was impressed. > The point of this long-winded email is that you probably could build > some kind of RPC system to run matplotlib remotely, but I believe it > would be an awful lot of work. In terms of manhours, my opinion is > that you'd come out ahead if you just focused on debugging your > current problems. Ok. It's good to know practically the best way to achieve "good enough" but I like to think and discuss about how to achieve "best possible", also. :-) Regards --bb