matplotlib-devel Mailing List for matplotlib

I decided to revisit this briefly and found another pretty easy change
that gives a fairly significant speed boost for a large number of
points. Basically, I just used two list comprehensions instead of
looping.
On Mon, Dec 13, 2010 at 8:10 AM, Benjamin Root <ben...@ou...> wrote:
> On Tue, Nov 30, 2010 at 4:53 PM, Benjamin Root <ben...@ou...> wrote:
>>
>> On Wednesday, November 17, 2010, Benjamin Root <ben...@ou...> wrote:
>> > On Tue, Nov 16, 2010 at 5:20 PM, J P <jp...@gm...> wrote:
>> >
>> >
>> > Hi all, here's my first patch for matplotlib. Someone noticed at Stack
>> > Overflow that the plot_surface function in mplot3d wasn't especially fast
>> > for a lot of points (and small rstrides/cstrides) and using shading and a
>> > single color. I found some parts of the code that weren't vectorized. These
>> > are my changes so far.
>> >
>> > Summary of changes:
>> > 1. Changed from double looping over aranges to using xrange
>> > 2. Made the normalization of the normals and their dot product with the
>> > vector [-1,-1,0.5] to find the shading a vectorized operation.
>> > 3. Changed a list comprehension which calculated the colors using an
>> > iterative approach to using the already built-in vectorization of the
>> > Normalization class and using the np.outer function. The result is a numpy
>> > array rather than a list which actually speeds up things down the line.
>> > 4. removed the corners array from plot_surface which wasn't ever used or
>> > returned. It didn't really slow things down, but I'm thinking that it is
>> > cruft.
>> >
>> > For change number two, I made a separate function that generates the
>> > shades, but feel free to move that around if you prefer.. or maybe it should
>> > be a function that begins with a _ because it shouldn't be used externally.
>> > These changes give varying levels of speed improvement depending on the
>> > number of points and the rstrides/cstrides arguments. With larger numbers of
>> > points and small rstrides/cstrides, these changes can more than halve the
>> > running time. I have found no difference in output after my changes.
>> >
>> > I know there is more work to be done within the plot_surface function
>> > and I'll submit more changes soon.
>> >
>> > Justin
>> >
>> >
>> > Justin,
>> >
>> > Thank you for your efforts to improve the performance of mplot3d. I
>> > will take a look at the patches you have submitted and test them out. What
>> > I am probably going to do is break down the patches into smaller pieces and
>> > incorporate them piece-by-piece.
>> >
>> > I tried refactoring plot_surface once before to mixed results. The key
>> > feature I was trying to gain was compatibility with masked arrays. I wanted
>> > to duplicate the behavior of contourf and pcolor where masked out portions
>> > of the surface would not be created. I managed to get it to work for that
>> > particular use-case, but I broke a bunch of other use-cases along the way.
>> > I am curious to see if your patches make this easier/harder to do.
>> >
>> > Thank you for your efforts and thank you for using matplotlib!
>> >
>> > Ben Root
>> >
>> >
>>
>> I have looked through the patches, and there are definite
>> improvements. I have broken the work into four separate patches. The
>> first patch is essentially code cleanup and utilization of xrange
>> (plot_surface_cleanup.patch). This patch can be back-ported without
>> concern (although it doesn't fix any bug per se).
>>
>> The second patch does the vectorization of the shades. The original
>> patch that was submitted had some edge cases, but I have found that
>> just simply converting that for-loop that creates the shades into a
>> list comprehension (and then pass into np.array) yielded almost
>> identical speedups without changing the current code path. (Note: I
>> am a minimalist when it comes to patches). This is in
>> plot_surface_vectshading.patch.
>>
>> The third patch improves the calculation of the normals in
>> plot_surface by pre-allocating the arrays for calculating the vectors
>> and doing a final call to np.cross rather than appending on a list. I
>> deviated slightly from the original patch by calling "which" as
>> "which_pt", adding a couple of comments, and also added an else
>> condition to create a "normal" with an empty list. This last part is
>> to keep the code path as similar as it was before. It was
>> theoretically possible to utilize a variable called normal elsewhere
>> without all the same conditions holding, so this guarantees that
>> normal exists, which was the case before. This patch is
>> plot_surface_vectnorm.patch.
>>
>> Finally, the fourth patch utilizes numpy array functionality for
>> calculating the vertexes. This patch also forgoes the use of
>> transposed arrays. I took the original patch a step further and
>> eliminated the array transpose line earlier in the plot_surface
>> function. The array transpose was not being properly utilized here,
>> and I saw no speed penalty/speedup either way, so in favor of simpler
>> code, I eliminated its use. This patch is
>> plot_surface_vectvertex.patch.
>>
>> Of the four patches, the speedups are in mostly found in the second
>> patch (100% speedup). The first patch does provide noticeable
>> improvements. There is also a slight improvement with the third
>> patch. I am up in the air regarding speed improvements with the
>> fourth patch, but I wonder if there might be some memory improvements
>> here, or if any speedup is being hidden by the for-loop that the
>> vectorization is done in.
>>
>> I will let these patches be mulled over before applying them. Thanks
>> to JP for submitting the original patch.
>>
>> Ben Root
>
>
> Re-pinging, as I haven't heard any opinions on this. The key question is
> should any of these patch be put into the maintenance branch or should it
> only be in the development branch?
>
> Thanks,
> Ben Root
>
> ------------------------------------------------------------------------------
> Oracle to DB2 Conversion Guide: Learn learn about native support for PL/SQL,
> new data types, scalar functions, improved concurrency, built-in packages,
> OCI, SQL*Plus, data movement tools, best practices and more.
> http://p.sf.net/sfu/oracle-sfdev2dev
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>
>

I am on RHEL5 with QT 4.2.1 installed. I am stuck with that version of QT
and can't go up.
I have PyQt 4.4.4 with SIP 4.7.8. That seems to be one of the few
combinations that would build with this version of QT. It took forever for
me to find sources for the other versions. 
I tried installing matplotlib 1.0 and 0.99 but if I set the backend to
Qt4Agg, I get an error when I try "show()": 'ImportError: Warning:
formlayout requires PyQt4 >v4.3'
Can anyone help with this?
--
Enrico Ng

On 12/14/10 11:25 PM, Ludwig Schwardt wrote:
> - The only GUI backends worth building on Mac OS X are TkAgg and the
> native macosx one, in my humble opinion.
wx works just fine on OS-X, though I'm pretty sure it's no longer a 
build-time dependency (no compiled code)
> - I have been successfully using the Apple "system" Python since Mac
> OS 10.5 to run numpy/scipy/matplotlib/ipython and never encountered
> any major build or usage problems. I agree that Mac OS 10.4 and
> earlier needed a Python reinstallation, but do not see why it is
> currently still a strong requirement.
I'm not sure how strong, and I don't know about numpy issues, but there 
are some other key issues:
1) not a native readline
2) Apple has NEVER updated any of their Python installs
3) You can't re-distribute it (py2app)
4) I thought there was a TK issue, but I'm not a TK user.
5) It'll only work on a given OS-X version (i.e. you can't distribute a 
binary that will work on multiple versions of OS-X -- at least not older 
ones)
Anyway, for years, the MacPython community has tried to establish a 
standard for the binaries that people build -- Python on the Mac is 
something of a nightmare, what with Apple's Python, python.org's build, 
fink, macports, etc....
The ONLY reasonable versions to build binaries for are the python.org 
builds, unless you want to build a LOT of different binaries. Note that 
Robin Dunn has managed to build wxPython binaries that work on both 
python.org and Apple pythons -- neat trick, that -- but not many people 
seem to want to do that kludge.
> As improved documentation I can contribute (yet another :-)) tutorial
> for building and installing matplotlib and friends on a clean Mac OS
> 10.6 system, with a minimum of downloaded packages and using the
> standard system stuff as far as possible.
That would still be nice -- I'm still not sure if you can count on X11 
being installed, though -- do you know?
-Chris
-- 
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chr...@no...

Hi,
Weighing in on the Mac build issue:
- The only GUI backends worth building on Mac OS X are TkAgg and the
native macosx one, in my humble opinion. Sticking to them will prevent
the kind of pain Kynn described. These backends are autodetected by
default during the build process and you only land in trouble if you
explicitly enable the rest. My suggestion is therefore to do the
default "python setup.py install".
- I have been successfully using the Apple "system" Python since Mac
OS 10.5 to run numpy/scipy/matplotlib/ipython and never encountered
any major build or usage problems. I agree that Mac OS 10.4 and
earlier needed a Python reinstallation, but do not see why it is
currently still a strong requirement. Ben, do you know off-hand what
issues numpy has been having with Apple Python?
As improved documentation I can contribute (yet another :-)) tutorial
for building and installing matplotlib and friends on a clean Mac OS
10.6 system, with a minimum of downloaded packages and using the
standard system stuff as far as possible. I fine-tuned the
instructions on several iMacs at work. I will just check that it still
works with the latest packages.
Regards,
Ludwig
> Message: 4
> Date: 2010年11月18日 16:57:10 -0600
> From: Benjamin Root <ben...@ou...>
>
> There are a variety of issues depending on your Mac system that needs to be
> sorted out to determine the best way to go about installing everything. The
> particular sticking point is that Apple supplied their own interperater
> rather than the standard python interpreater. Unfortunately, this causes
> problems with numpy (and thus matplotlib).

On Tue, Dec 14, 2010 at 5:26 PM, Russell Owen <ro...@uw...> wrote:
> On Dec 14, 2010, at 2:38 PM, Benjamin Root wrote:
>
> On Tue, Dec 14, 2010 at 4:22 PM, Benjamin Root <ben...@ou...> wrote:
>
>>
>>
>> On Tue, Dec 14, 2010 at 4:06 PM, Russell Owen <ro...@uw...> wrote:
>>
>>> On Dec 14, 2010, at 1:50 PM, Benjamin Root wrote:
>>>
>>> On Tue, Dec 14, 2010 at 3:18 PM, Russell E. Owen <ro...@uw...> wrote:
>>>
>>>> I tried the test script on linux using matplotlib svn trunk rev8840
>>>> (which appears to include your patch) and found a leak that starts out
>>>> small but gets systematically larger. This is with Python 2.6.5 and
>>>> Tkinter built against Tcl/Tk 8.4.13.
>>>>
>>>> Is anyone else seeing this?
>>>>
>>>> time rss memory mem/sec
>>>> (sec) (kb) (kb/sec)
>>>> 0.0 36816 nan
>>>> 5.0 36860 nan
>>>> 10.0 36860 0.0
>>>> 15.1 36860 0.0
>>>> 20.1 36860 0.0
>>>> 25.1 36896 1.8
>>>> ...
>>>> 326.5 36896 0.1
>>>> 331.6 36972 0.3
>>>> ...
>>>> 401.9 36972 0.3
>>>> 406.9 36980 0.3
>>>> ...
>>>>
>>>> 602.8 37684 1.4
>>>> 607.8 37700 1.4
>>>>
>>>> This is different behavior than on Mac OS X, but still alarming.
>>>>
>>>> -- Russell
>>>>
>>>>
>>> Russell,
>>>
>>> I am curious, I recently ran into problems with mixing builds of numpy
>>> and matplotlib at various levels of revisions. I eventually had to do a
>>> complete clean rebuild. Note, what I mean by a complete clean rebuild is
>>> that I removed the numpy and matplotlib source directories and re-checkout
>>> the codes from source and then rebuild. I would be curious if the problem
>>> persists after that.
>>>
>>>
>>> An interesting question. I can say that this was a clean build of
>>> matplotlib since I ran it "in place" (in build/lib.linux-x86_64-2.6/) after
>>> building it rather than installing it in site-packages to avoid messing up
>>> other users (on the linux system this was a shared python). I modified the
>>> script to print matplotlib.__file__ to make sure I was running the right
>>> version. I doubt it was a clean build of numpy. But considering no numerics
>>> are occurring in this example (it is literally just creating an Axes on a
>>> Canvas and calling canvas.draw() repeatedly) do you think numpy could
>>> possibly come into this?
>>>
>>>
>> It is quite possible. Numpy is used extensively throughout the matplotlib
>> system, regardless of whether you are using it or not. Also, the fact that
>> you are on a shared system is interesting. For those situations, try doing
>>
>> "python setupegg.py develop --user"
>>
>> for the build command. What this does is builds everything in-place (the
>> build directory symbolically links to those files), and then uses your
>> ~/.local directory to install an egg.lnk file to point back to the build
>> directory. This should have higher search precedence than the system
>> install of matplotlib and numpy.
>>
>> Note, I had mixed success with this approach on a RHEL (5?) system
>> recently. I don't know how recently ~/.local became widely accepted among
>> various distros.
>>
>>
>>> Might you run the script on your system with the clean build?
>>>
>>> -- Russell
>>>
>>>
>> I will give it a shot (once my other analysis programs are done for the
>> day).
>>
>> Ben Root
>>
>
> I ran your script from the url you posted earlier. My build does not show
> any leak for about a minute. I am fairly certain that what we have here is
> a build mixing issue or an issue with an unknown combination of libraries
> interacting. Could you post your build logs?
>
> Ben Root
>
>
> I tried your suggestion -- installing numpy from scratch (deleting the old
> numpy first) and then building matplotlib fresh. Here are my build logs:
> <http://www.astro.washington.edu/users/rowen/python/numpy_build_log.txt>
> <
> http://www.astro.washington.edu/users/rowen/python/matplotlib_build_log.txt
> >
>
> Much like last time (with a fresh matplotlib) the test program shows no
> leak at first, then it starts growing to alarming levels (though it took
> longer to start leaking this time than it did last time):
>
>
Interesting behavior to report. I did a completely clean rebuild on my
other computer (Ubuntu 10.10) and the first matplotlib-related program that
was used was this memory check program. It exhibited a memory leak for the
very first run. However, after ending that program and starting it back up,
the memory levels were lower and very steady.
I also noticed for the runs where the memory levels were steady, I could
make the memory levels go slightly up if I interacted with the figure window
(moving it and such).
I did do a long run (600+ seconds) and I did get a memory leak at around 600
second mark. I did *not* have the same problem, though, where the next run
exhibits an immediate memory leak. Instead, the memory levels were
consistent with previous runs.
I will take a peek at your build logs tomorrow. There is something fishy
happening, but I am not convinced that it is entirely an issue with
matplotlib (although that first run behavior is bad...).
Ben Root