matplotlib-users Mailing List for matplotlib

There has to be something else in play here. I'll try to keep this
short, but the summary is this: I can get the transparency to look
right, but only if 1) I put "straight" alpha in image, not
premultiplied, and 2) I hack agg to remove specificially every
instance of the lines of code that you refer to above.
Why this is, I don't know. Hopefully I'm still misusing something.
However, it behaves as if the clipping of alpha in the agg library is
corrupting the alpha channel. I also submit that I could have broken
some other transparency capabilities of matplotlib, because I don't
know what other routines use what I hacked....I did check a few
transparent polygons and such though, and everything seemed to be
fine.
I know that the agg library has been around for quite a long time, so
that also means that such a basic bug is unlikely.
I've reattached the (slightly modified) script that reproduces the
problem, along with a sample image that it uses. The only change to
the script is right at the top, where a different image is read, a
quick statement is placed to add an alpha channel if there is not
already one, and I'm attempting to use premultiplied alphas. I've
also attached a screenshot of the output. Notice that in this case,
both "transparent" images look wrong.
Now, if I 1) hack agg to remove the alpha clipping, and 2) modify the
one line in the attached python script so that I use straight alpha,
everything looks right. Specifically, I removed every instance of the
code below from xxxx, rebuilt all of the matplotlib .so's, and
specifically replaced _image.so and _backend_agg.so in my matplotlib
distribution.
 if(fg[order_type::A] > base_mask) fg[order_type::A]
= base_mask;
 if(fg[order_type::R] > fg[order_type::A])
fg[order_type::R] = fg[order_type::A];
 if(fg[order_type::G] > fg[order_type::A])
fg[order_type::G] = fg[order_type::A];
 if(fg[order_type::B] > fg[order_type::A])
fg[order_type::B] = fg[order_type::A];
On Wed, Oct 19, 2011 at 2:34 PM, Daniel Hyams <dh...@gm...> wrote:
>
> Ah, thanks so much Michael! That explanation helps a great deal; I
> was always considering things in "straight alpha" format, not even
> knowing that there was alternative.
>
> I'll play with this tonight; I don't see any problem getting the thing
> working, though, now that I know what agg expects to see...
>
> And yes, alpha support in the image class would be very helpful ;)
>
> On Wed, Oct 19, 2011 at 2:16 PM, Michael Droettboom <md...@st...> wrote:
> > You are right that Agg is doing the resizing here. Agg expects
> > premultiplied alpha. See [1] for information about what that means.
> >
> > [1] http://en.wikipedia.org/wiki/Alpha_compositing
> >
> > After Agg interpolates the pixel values, to prevent oversaturation it
> > truncates all values to be less than alpha (which makes sense if everything
> > is assumed to be premultiplied alpha). Arguably, the bug here is that
> > nearest neighbor (which doesn't have to do any blending) doesn't perform the
> > truncation step -- then both would look "wrong".
> >
> > It happens in this code snippet in span_image_filter_rgba: (base_mask is
> > 255)
> >
> >         if(fg[order_type::A] > base_mask)     fg[order_type::A]
> > = base_mask;
> >         if(fg[order_type::R] > fg[order_type::A]) fg[order_type::R]
> > = fg[order_type::A];
> >         if(fg[order_type::G] > fg[order_type::A]) fg[order_type::G]
> > = fg[order_type::A];
> >         if(fg[order_type::B] > fg[order_type::A]) fg[order_type::B]
> > = fg[order_type::A];
> >
> > So, the solution to make a partially transparent image is to not do:
> >
> >   pix[:,:,3] = 127
> >
> > but instead, do
> >
> >   pix *= 0.5
> >
> > Of course, the real fix here is to support alpha blending properly in the
> > image class, then the user wouldn't have to deal with such details. A bug
> > should probably be filed in the matplotlib issue tracker for this.
> >
> > Mike
> >
> > On 10/19/2011 12:23 PM, Daniel Hyams wrote:
> >
> > [Sorry, I keep getting tripped up with HTML mail....resent in ascii,
> > and resaved one of the attachment png's to make it smaller.]
> >
> >
> > Example script attached (PIL required). Basically, if I impose a
> > specific value into an image's alpha channel and use any interpolation
> > scheme other than 'nearest', there appears gray all where the figure
> > didn't have any color to begin with.  I've also attached a screenshot
> > of the output of the script on my machine.
> >
> > Hopefully I'm doing something wrongly?
> >
> > I chased the problem and managed to hack in a solution that fixes the
> > problem, but it's extremely inefficient...basically, in matplotlib's
> > image.py, routine BboxImage.make_image, you can create two images
> > there....one with no alpha channel (call it imRGB) and one with (call
> > it imRGBA). Go through all of the routine, doing exactly the same
> > things to both of the images *except* for the interpolation, which is
> > set to 'nearest' for imRGBA. Then, rip the colors out of imRGB, the
> > alpha channel off of imRGBA, and put them together....go through all
> > of the routine again with this composited image, and it works. I
> > know...I told you it was bad ;)
> >
> > The problem seems to be in the "resize" call in that routine...resize,
> > which calls into C code, does not appear to handle things correctly
> > when the alpha is anything other than 255's across the board. It
> > might be a problem in the agg routines, but hopefully it is just maybe
> > a misuse of the agg routines.
> >
> > The behavior seems to be backend independent as far as I could test (I
> > tried with wxagg and tk backends). I am using mpl 1.0.0 on Windows if
> > it matters.
> >
> >
> > --
> > Daniel Hyams
> > dh...@gm...
> >
> > ------------------------------------------------------------------------------
> > All the data continuously generated in your IT infrastructure contains a
> > definitive record of customers, application performance, security
> > threats, fraudulent activity and more. Splunk takes this data and makes
> > sense of it. Business sense. IT sense. Common sense.
> > http://p.sf.net/sfu/splunk-d2d-oct
> >
> > _______________________________________________
> > Matplotlib-users mailing list
> > Mat...@li...
> > https://lists.sourceforge.net/lists/listinfo/matplotlib-users
> >
> >
> > ------------------------------------------------------------------------------
> > The demand for IT networking professionals continues to grow, and the
> > demand for specialized networking skills is growing even more rapidly.
> > Take a complimentary Learning@Ciosco Self-Assessment and learn
> > about Cisco certifications, training, and career opportunities.
> > http://p.sf.net/sfu/cisco-dev2dev
> > _______________________________________________
> > Matplotlib-users mailing list
> > Mat...@li...
> > https://lists.sourceforge.net/lists/listinfo/matplotlib-users
> >
> >
>
>
>
> --
> Daniel Hyams
> dh...@gm...
--
Daniel Hyams
dh...@gm...

thank you, I am not quite sure how to 'draw' the shapefile but making those
changes and removing the shapefile has sped the processing up considerably!
Thank you for your help
On Wed, Oct 19, 2011 at 11:42 PM, Jeff Whitaker <js...@fa...> wrote:
> On 10/18/11 8:55 PM, questions anon wrote:
>
> Thanks Jeff, that certainly speeds it up! But when I take them out of the
> loop and place them elsewhere they are no longer added to the map.
> Is there someway I can call them in the loop but still get it to run
> quickly?
> Thanks
>
>
> Just the Basemap instance creation and the transformation of coordinates to
> projection space should be hoisted out of the loop
>
>
> map =
> Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
>
> llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
> x,y=map(*N.meshgrid(LON,LAT))
>
> you can leave the other statements in.
>
> If you still have memory issues, bring the readshapefile call out, and draw
> the shapes whose coordinates are stored in the instance variable
> map.DSE_REGIONS manually in the loop.
>
> -Jeff
>
>
> On Fri, Oct 14, 2011 at 10:54 PM, Jeff Whitaker <js...@fa...>wrote:
>
> On 10/12/11 8:20 PM, questions anon wrote:
>
> Hi All,
> I keep receiving a memory error when processing many netcdf files. I
> assumed it had something to do with how I loop things and maybe needed to
> close things off properly but I recently received an error that made me
> think it might be because of matplotlib.
>
> In the code below I am looping through a bunch of netcdf files (each file
> is hourly data for one month) and within each netcdf file I am outputting a
> *png file every three hours. This works for one netcdf file (therefore one
> month) but when it begins to process the next netcdf file I receive a memory
> error (see below). Since I have tidied some of my code up it seems to
> process partly into the second file but then I still receive the memory
> error.
> I have tried a few suggestions such as:
> -Combining the dataset using MFDataset (using NETCDF4) is not an option
> because the files do not have unlimited dimension.
> - gc.collect() but that just results in a *GEOS_ERROR: bad allocation
> error*.
> -only open LAT and LON once (which worked)
>
> System Details:
> Python 2.7.2 |EPD 7.1-2 (32-bit)| (default, Jul 3 2011, 15:13:59) [MSC
> v.1500 32 bit (Intel)] on win32
>
> Any feedback will be greatly appreciated as I seem to keep ending up with
> memory errors when working with netcdf files this even happens if I am using
> a much better computer.
>
> *Most recent error: *
> Traceback (most recent call last):
> File "C:\plot_netcdf_merc_multiplot_across_multifolders_TSFC.py", line
> 78, in <module>
> plt.savefig((os.path.join(outputfolder, 'TSFC'+date_string+'UTC.png')))
> File "C:\Python27\lib\site-packages\matplotlib\pyplot.py", line 363, in
> savefig
> return fig.savefig(*args, **kwargs)
> File "C:\Python27\lib\site-packages\matplotlib\figure.py", line 1084, in
> savefig
> self.canvas.print_figure(*args, **kwargs)
> File
> "C:\Python27\lib\site-packages\matplotlib\backends\backend_wxagg.py", line
> 100, in print_figure
> FigureCanvasAgg.print_figure(self, filename, *args, **kwargs)
> File "C:\Python27\lib\site-packages\matplotlib\backend_bases.py", line
> 1923, in print_figure
> **kwargs)
> File "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
> line 438, in print_png
> FigureCanvasAgg.draw(self)
> File "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
> line 393, in draw
> self.renderer = self.get_renderer()
> File "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
> line 404, in get_renderer
> self.renderer = RendererAgg(w, h, self.figure.dpi)
> File "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
> line 59, in __init__
> self._renderer = _RendererAgg(int(width), int(height), dpi,
> debug=False)
> RuntimeError: Could not allocate memory for image
>
> *Error when I added gc.collect()*
> GEOS_ERROR: bad allocation
>
> *Old error (before adding gc.collect() )*
> *Traceback (most recent call last):
> File "d:/plot_netcdf_merc_multiplot_across_multifolders__memoryerror.py",
> line 44, in <module>
> TSFC=ncfile.variables['T_SFC'][1::3]
> File "netCDF4.pyx", line 2473, in netCDF4.Variable.__getitem__
> (netCDF4.c:23094)
> MemoryError*
>
>
>
> from netCDF4 import Dataset
> import numpy as N
> import matplotlib.pyplot as plt
> from mpl_toolkits.basemap import Basemap
> from netcdftime import utime
> from datetime import datetime
> import os
> import gc
>
>
> shapefile1="E:/
>
> griddeddatasamples/GIS/DSE_REGIONS"
> MainFolder=r"E:/griddeddatasamples/GriddedData/InputsforValidation/T_SFC/"
> OutputFolder=r"E:/griddeddatasamples/GriddedData/OutputsforValidation"
> fileforlatlon=Dataset("E:/griddeddatasamples/GriddedData/InputsforValidation/T_SFC/TSFC_1974_01/IDZ00026_VIC_ADFD_T_SFC.nc",
> 'r+', 'NETCDF4')
> LAT=fileforlatlon.variables['latitude'][:]
> LON=fileforlatlon.variables['longitude'][:]
>
> for (path, dirs, files) in os.walk(MainFolder):
> for dir in dirs:
> print dir
> path=path+'/'
> for ncfile in files:
> if ncfile[-3:]=='.nc':
> print "dealing with ncfiles:", ncfile
> ncfile=os.path.join(path,ncfile)
> ncfile=Dataset(ncfile, 'r+', 'NETCDF4')
> TSFC=ncfile.variables['T_SFC'][1::3]
> TIME=ncfile.variables['time'][1::3]
> ncfile.close()
> gc.collect()
>
> for TSFC, TIME in zip((TSFC[:]),(TIME[:])):
> cdftime=utime('seconds since 1970年01月01日 00:00:00')
> ncfiletime=cdftime.num2date(TIME)
> print ncfiletime
> timestr=str(ncfiletime)
> d = datetime.strptime(timestr, '%Y-%m-%d %H:%M:%S')
> date_string = d.strftime('%Y%m%d_%H%M')
>
> map =
> Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
>
> llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
> x,y=map(*N.meshgrid(LON,LAT))
> map.drawcoastlines(linewidth=0.5)
> map.readshapefile(shapefile1, 'DSE_REGIONS')
> map.drawstates()
>
> plt.title('Surface temperature at %s UTC'%ncfiletime)
> ticks=[-5,0,5,10,15,20,25,30,35,40,45,50]
> CS = map.contourf(x,y,TSFC, ticks, cmap=plt.cm.jet)
> l,b,w,h =0.1,0.1,0.8,0.8
> cax = plt.axes([l+w+0.025, b, 0.025, h], )
> cbar=plt.colorbar(CS, cax=cax, drawedges=True)
>
> plt.savefig((os.path.join(OutputFolder,
> 'TSFC'+date_string+'UTC.png')))
> plt.close()
> gc.collect()
>
>
> Try moving these lines
>
>
> map =
> Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
>
> llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
> x,y=map(*N.meshgrid(LON,LAT))
> map.drawcoastlines(linewidth=0.5)
> map.readshapefile(shapefile1, 'DSE_REGIONS')
> map.drawstates()
>
> out of the loop.
>
> -Jeff
>
>
>
>

On Fri, Oct 14, 2011 at 3:42 AM, Nils Wagner
<nw...@ia...>wrote:
> Hi all,
>
> what is the native "data" coordinate system for Arrows in
> a polar plot ?
>
> How do I add arrows to a polar plot ?
>
> An example would be appreciated.
> fig = figure(figsize=(12,12))
> ax = fig.add_subplot(111, polar=True)
>
> Nils
>
>
I don't know if it is an existing feature or not. If it isn't, then I think
a feature request should be filed. If it is, then a request should be filed
anyway for an example to be added to the gallery.
Ben Root

Ah, thanks so much Michael! That explanation helps a great deal; I
was always considering things in "straight alpha" format, not even
knowing that there was alternative.
I'll play with this tonight; I don't see any problem getting the thing
working, though, now that I know what agg expects to see...
And yes, alpha support in the image class would be very helpful ;)
On Wed, Oct 19, 2011 at 2:16 PM, Michael Droettboom <md...@st...> wrote:
> You are right that Agg is doing the resizing here. Agg expects
> premultiplied alpha. See [1] for information about what that means.
>
> [1] http://en.wikipedia.org/wiki/Alpha_compositing
>
> After Agg interpolates the pixel values, to prevent oversaturation it
> truncates all values to be less than alpha (which makes sense if everything
> is assumed to be premultiplied alpha). Arguably, the bug here is that
> nearest neighbor (which doesn't have to do any blending) doesn't perform the
> truncation step -- then both would look "wrong".
>
> It happens in this code snippet in span_image_filter_rgba: (base_mask is
> 255)
>
>         if(fg[order_type::A] > base_mask)     fg[order_type::A]
> = base_mask;
>         if(fg[order_type::R] > fg[order_type::A]) fg[order_type::R]
> = fg[order_type::A];
>         if(fg[order_type::G] > fg[order_type::A]) fg[order_type::G]
> = fg[order_type::A];
>         if(fg[order_type::B] > fg[order_type::A]) fg[order_type::B]
> = fg[order_type::A];
>
> So, the solution to make a partially transparent image is to not do:
>
>   pix[:,:,3] = 127
>
> but instead, do
>
>   pix *= 0.5
>
> Of course, the real fix here is to support alpha blending properly in the
> image class, then the user wouldn't have to deal with such details. A bug
> should probably be filed in the matplotlib issue tracker for this.
>
> Mike
>
> On 10/19/2011 12:23 PM, Daniel Hyams wrote:
>
> [Sorry, I keep getting tripped up with HTML mail....resent in ascii,
> and resaved one of the attachment png's to make it smaller.]
>
>
> Example script attached (PIL required). Basically, if I impose a
> specific value into an image's alpha channel and use any interpolation
> scheme other than 'nearest', there appears gray all where the figure
> didn't have any color to begin with.  I've also attached a screenshot
> of the output of the script on my machine.
>
> Hopefully I'm doing something wrongly?
>
> I chased the problem and managed to hack in a solution that fixes the
> problem, but it's extremely inefficient...basically, in matplotlib's
> image.py, routine BboxImage.make_image, you can create two images
> there....one with no alpha channel (call it imRGB) and one with (call
> it imRGBA). Go through all of the routine, doing exactly the same
> things to both of the images *except* for the interpolation, which is
> set to 'nearest' for imRGBA. Then, rip the colors out of imRGB, the
> alpha channel off of imRGBA, and put them together....go through all
> of the routine again with this composited image, and it works. I
> know...I told you it was bad ;)
>
> The problem seems to be in the "resize" call in that routine...resize,
> which calls into C code, does not appear to handle things correctly
> when the alpha is anything other than 255's across the board. It
> might be a problem in the agg routines, but hopefully it is just maybe
> a misuse of the agg routines.
>
> The behavior seems to be backend independent as far as I could test (I
> tried with wxagg and tk backends). I am using mpl 1.0.0 on Windows if
> it matters.
>
>
> --
> Daniel Hyams
> dh...@gm...
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2d-oct
>
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>
>
> ------------------------------------------------------------------------------
> The demand for IT networking professionals continues to grow, and the
> demand for specialized networking skills is growing even more rapidly.
> Take a complimentary Learning@Ciosco Self-Assessment and learn
> about Cisco certifications, training, and career opportunities.
> http://p.sf.net/sfu/cisco-dev2dev
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>
>
-- 
Daniel Hyams
dh...@gm...

<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style id="owaParaStyle">P {
	MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
</style>
</head>
<body fPStyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">
<div>
<p>Dear all,</p>
<p>&nbsp;</p>
<p>I am trying to plot a wind field using the quiver function and would like to color the wind arrows according to the wind strength. I have gone through the online matplotlib material, but have not been able to work out how to do it so far.</p>
<p>&nbsp;</p>
<p>Would it be possible for someone to give me&nbsp;a hint on how&nbsp;to color the arrows according to the wind strength?</p>
<p>&nbsp;</p>
<p>I would much appreciate any help with this problem.</p>
<p>&nbsp;</p>
<p>Best wishes,</p>
<p>&nbsp;</p>
<p>David</p>
<p>&nbsp;</p>
<div style="FONT-FAMILY: Tahoma; FONT-SIZE: 13px">
<p><font size="2">&nbsp;</p>
</font></div>
</div>
</div>
</body>
</html>

You are right that Agg is doing the resizing here. Agg expects 
premultiplied alpha. See [1] for information about what that means.
[1] http://en.wikipedia.org/wiki/Alpha_compositing
After Agg interpolates the pixel values, to prevent oversaturation it 
truncates all values to be less than alpha (which makes sense if 
everything is assumed to be premultiplied alpha). Arguably, the bug 
here is that nearest neighbor (which doesn't have to do any blending) 
doesn't perform the truncation step -- then both would look "wrong".
It happens in this code snippet in span_image_filter_rgba: (base_mask is 
255)
 if(fg[order_type::A] > base_mask) 
fg[order_type::A] = base_mask;
 if(fg[order_type::R] > fg[order_type::A]) 
fg[order_type::R] = fg[order_type::A];
 if(fg[order_type::G] > fg[order_type::A]) 
fg[order_type::G] = fg[order_type::A];
 if(fg[order_type::B] > fg[order_type::A]) 
fg[order_type::B] = fg[order_type::A];
So, the solution to make a partially transparent image is to not do:
 pix[:,:,3] = 127
but instead, do
 pix *= 0.5
Of course, the real fix here is to support alpha blending properly in 
the image class, then the user wouldn't have to deal with such details. 
A bug should probably be filed in the matplotlib issue tracker for this.
Mike
On 10/19/2011 12:23 PM, Daniel Hyams wrote:
> [Sorry, I keep getting tripped up with HTML mail....resent in ascii,
> and resaved one of the attachment png's to make it smaller.]
>
>
> Example script attached (PIL required). Basically, if I impose a
> specific value into an image's alpha channel and use any interpolation
> scheme other than 'nearest', there appears gray all where the figure
> didn't have any color to begin with. I've also attached a screenshot
> of the output of the script on my machine.
>
> Hopefully I'm doing something wrongly?
>
> I chased the problem and managed to hack in a solution that fixes the
> problem, but it's extremely inefficient...basically, in matplotlib's
> image.py, routine BboxImage.make_image, you can create two images
> there....one with no alpha channel (call it imRGB) and one with (call
> it imRGBA). Go through all of the routine, doing exactly the same
> things to both of the images *except* for the interpolation, which is
> set to 'nearest' for imRGBA. Then, rip the colors out of imRGB, the
> alpha channel off of imRGBA, and put them together....go through all
> of the routine again with this composited image, and it works. I
> know...I told you it was bad ;)
>
> The problem seems to be in the "resize" call in that routine...resize,
> which calls into C code, does not appear to handle things correctly
> when the alpha is anything other than 255's across the board. It
> might be a problem in the agg routines, but hopefully it is just maybe
> a misuse of the agg routines.
>
> The behavior seems to be backend independent as far as I could test (I
> tried with wxagg and tk backends). I am using mpl 1.0.0 on Windows if
> it matters.
>
>
> --
> Daniel Hyams
> dh...@gm...
>
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2d-oct
>
>
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users

[Sorry, I keep getting tripped up with HTML mail....resent in ascii,
and resaved one of the attachment png's to make it smaller.]
Example script attached (PIL required). Basically, if I impose a
specific value into an image's alpha channel and use any interpolation
scheme other than 'nearest', there appears gray all where the figure
didn't have any color to begin with.  I've also attached a screenshot
of the output of the script on my machine.
Hopefully I'm doing something wrongly?
I chased the problem and managed to hack in a solution that fixes the
problem, but it's extremely inefficient...basically, in matplotlib's
image.py, routine BboxImage.make_image, you can create two images
there....one with no alpha channel (call it imRGB) and one with (call
it imRGBA). Go through all of the routine, doing exactly the same
things to both of the images *except* for the interpolation, which is
set to 'nearest' for imRGBA. Then, rip the colors out of imRGB, the
alpha channel off of imRGBA, and put them together....go through all
of the routine again with this composited image, and it works. I
know...I told you it was bad ;)
The problem seems to be in the "resize" call in that routine...resize,
which calls into C code, does not appear to handle things correctly
when the alpha is anything other than 255's across the board. It
might be a problem in the agg routines, but hopefully it is just maybe
a misuse of the agg routines.
The behavior seems to be backend independent as far as I could test (I
tried with wxagg and tk backends). I am using mpl 1.0.0 on Windows if
it matters.
--
Daniel Hyams
dh...@gm...

On 10/18/11 8:55 PM, questions anon wrote:
> Thanks Jeff, that certainly speeds it up! But when I take them out of 
> the loop and place them elsewhere they are no longer added to the map.
> Is there someway I can call them in the loop but still get it to run 
> quickly?
> Thanks
Just the Basemap instance creation and the transformation of coordinates 
to projection space should be hoisted out of the loop
 map = 
Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
 
llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
 x,y=map(*N.meshgrid(LON,LAT))
you can leave the other statements in.
If you still have memory issues, bring the readshapefile call out, and 
draw the shapes whose coordinates are stored in the instance variable 
map.DSE_REGIONS manually in the loop.
-Jeff
>
> On Fri, Oct 14, 2011 at 10:54 PM, Jeff Whitaker <js...@fa... 
> <mailto:js...@fa...>> wrote:
>
> On 10/12/11 8:20 PM, questions anon wrote:
>> Hi All,
>> I keep receiving a memory error when processing many netcdf
>> files. I assumed it had something to do with how I loop things
>> and maybe needed to close things off properly but I recently
>> received an error that made me think it might be because of
>> matplotlib.
>>
>> In the code below I am looping through a bunch of netcdf files
>> (each file is hourly data for one month) and within each netcdf
>> file I am outputting a *png file every three hours. This works
>> for one netcdf file (therefore one month) but when it begins to
>> process the next netcdf file I receive a memory error (see
>> below). Since I have tidied some of my code up it seems to
>> process partly into the second file but then I still receive the
>> memory error.
>> I have tried a few suggestions such as:
>> -Combining the dataset using MFDataset (using NETCDF4) is not an
>> option because the files do not have unlimited dimension.
>> - gc.collect() but that just results in a /GEOS_ERROR: bad
>> allocation error/.
>> -only open LAT and LON once (which worked)
>>
>> System Details:
>> Python 2.7.2 |EPD 7.1-2 (32-bit)| (default, Jul 3 2011,
>> 15:13:59) [MSC v.1500 32 bit (Intel)] on win32
>>
>> Any feedback will be greatly appreciated as I seem to keep ending
>> up with memory errors when working with netcdf files this even
>> happens if I am using a much better computer.
>>
>> *Most recent error: *
>> Traceback (most recent call last):
>> File
>> "C:\plot_netcdf_merc_multiplot_across_multifolders_TSFC.py", line
>> 78, in <module>
>> plt.savefig((os.path.join(outputfolder,
>> 'TSFC'+date_string+'UTC.png')))
>> File "C:\Python27\lib\site-packages\matplotlib\pyplot.py", line
>> 363, in savefig
>> return fig.savefig(*args, **kwargs)
>> File "C:\Python27\lib\site-packages\matplotlib\figure.py", line
>> 1084, in savefig
>> self.canvas.print_figure(*args, **kwargs)
>> File
>> "C:\Python27\lib\site-packages\matplotlib\backends\backend_wxagg.py",
>> line 100, in print_figure
>> FigureCanvasAgg.print_figure(self, filename, *args, **kwargs)
>> File
>> "C:\Python27\lib\site-packages\matplotlib\backend_bases.py", line
>> 1923, in print_figure
>> **kwargs)
>> File
>> "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
>> line 438, in print_png
>> FigureCanvasAgg.draw(self)
>> File
>> "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
>> line 393, in draw
>> self.renderer = self.get_renderer()
>> File
>> "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
>> line 404, in get_renderer
>> self.renderer = RendererAgg(w, h, self.figure.dpi)
>> File
>> "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
>> line 59, in __init__
>> self._renderer = _RendererAgg(int(width), int(height), dpi,
>> debug=False)
>> RuntimeError: Could not allocate memory for image
>>
>> *Error when I added gc.collect()*
>> GEOS_ERROR: bad allocation
>>
>> *Old error (before adding gc.collect() )*
>> /Traceback (most recent call last):
>> File
>> "d:/plot_netcdf_merc_multiplot_across_multifolders__memoryerror.py",
>> line 44, in <module>
>> TSFC=ncfile.variables['T_SFC'][1::3]
>> File "netCDF4.pyx", line 2473, in netCDF4.Variable.__getitem__
>> (netCDF4.c:23094)
>> MemoryError/
>>
>>
>>
>> from netCDF4 import Dataset
>> import numpy as N
>> import matplotlib.pyplot as plt
>> from mpl_toolkits.basemap import Basemap
>> from netcdftime import utime
>> from datetime import datetime
>> import os
>> import gc
>>
>>
>> shapefile1="E:/
>>
>> griddeddatasamples/GIS/DSE_REGIONS"
>> MainFolder=r"E:/griddeddatasamples/GriddedData/InputsforValidation/T_SFC/"
>> OutputFolder=r"E:/griddeddatasamples/GriddedData/OutputsforValidation"
>> fileforlatlon=Dataset("E:/griddeddatasamples/GriddedData/InputsforValidation/T_SFC/TSFC_1974_01/IDZ00026_VIC_ADFD_T_SFC.nc",
>> 'r+', 'NETCDF4')
>> LAT=fileforlatlon.variables['latitude'][:]
>> LON=fileforlatlon.variables['longitude'][:]
>>
>> for (path, dirs, files) in os.walk(MainFolder):
>> for dir in dirs:
>> print dir
>> path=path+'/'
>> for ncfile in files:
>> if ncfile[-3:]=='.nc':
>> print "dealing with ncfiles:", ncfile
>> ncfile=os.path.join(path,ncfile)
>> ncfile=Dataset(ncfile, 'r+', 'NETCDF4')
>> TSFC=ncfile.variables['T_SFC'][1::3]
>> TIME=ncfile.variables['time'][1::3]
>> ncfile.close()
>> gc.collect()
>>
>> for TSFC, TIME in zip((TSFC[:]),(TIME[:])):
>> cdftime=utime('seconds since 1970年01月01日
>> 00:00:00')
>> ncfiletime=cdftime.num2date(TIME)
>> print ncfiletime
>> timestr=str(ncfiletime)
>> d = datetime.strptime(timestr, '%Y-%m-%d
>> %H:%M:%S')
>> date_string = d.strftime('%Y%m%d_%H%M')
>>
>> map =
>> Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
>> 
>> llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
>> x,y=map(*N.meshgrid(LON,LAT))
>> map.drawcoastlines(linewidth=0.5)
>> map.readshapefile(shapefile1, 'DSE_REGIONS')
>> map.drawstates()
>>
>> plt.title('Surface temperature at %s
>> UTC'%ncfiletime)
>> ticks=[-5,0,5,10,15,20,25,30,35,40,45,50]
>> CS = map.contourf(x,y,TSFC, ticks,
>> cmap=plt.cm.jet)
>> l,b,w,h =0.1,0.1,0.8,0.8
>> cax = plt.axes([l+w+0.025, b, 0.025, h], )
>> cbar=plt.colorbar(CS, cax=cax, drawedges=True)
>>
>> plt.savefig((os.path.join(OutputFolder,
>> 'TSFC'+date_string+'UTC.png')))
>> plt.close()
>> gc.collect()
>>
>
> Try moving these lines
>
>
> map =
> Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
> 
> llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
> x,y=map(*N.meshgrid(LON,LAT))
> map.drawcoastlines(linewidth=0.5)
> map.readshapefile(shapefile1, 'DSE_REGIONS')
> map.drawstates()
>
> out of the loop.
>
> -Jeff
>
>

Thanks for the comprehensive explanation.
So it would seem it's really only the
http://matplotlib.sourceforge.net/users/whats_new.html page that is
misleading on this.
Many thanks!
-- 
View this message in context: http://old.nabble.com/QT-draw-issue-in-1.1.0-and-PyQt4v2-missing--tp32676093p32680797.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

On 10/18/2011 09:44 PM, RuiDC wrote:
> Benjamin Root-2 wrote:
> I think that might have been a little unclear. You should only need
> to select 'PyQt4' or 'PySide'. If PyQt4 is selected, then (I think)
> the v2 is automatically tested for internally.
>
> hmm, please explain where you think it is used internally, as the code
> seem to expect that this param can be set when there is no env variable
> 'QT_API' set, in backends\qt4_compat.py:
After quite a bit of thrashing around, we settled on the present system. 
 The rcParam deliberately does *not* set the API version if PyQt4 is 
used. Whatever your Qt4 comes up with when the backend is imported is 
what is used. That way there is no conflict with applications that 
import pyqt4, and perhaps set the API, before importing mpl. If you want 
to use the v2 api, you can import pyqt4 and set the API yourself before 
importing mpl. Or, if you want the API to be forced to v2 by mpl when 
the backend is imported, you can do what ETS does, which is to set the 
QT_API environment variable to "pyqt".
This moderately ugly mess was necessitated by the need to be compatible 
with ipython, ETS, and existing user code, while dealing with two 
versions of pyqt4 and one of pyside, and with different default versions 
of pyqt4 for python 2 and python 3 (which we will be supporting in the 
not-too-distant future, I believe.)
Eric
>
> # Available APIs.
> QT_API_PYQT = 'PyQt4' # API is not set here; Python 2.x default is V 1
> QT_API_PYQTv2 = 'PyQt4v2' # forced to Version 2 API
> QT_API_PYSIDE = 'PySide' # only supports Version 2 API
>
> ETS = dict(pyqt=QT_API_PYQTv2, pyside=QT_API_PYSIDE)
>
> # If the ETS QT_API environment variable is set, use it. Note that
> # ETS requires the version 2 of PyQt4, which is not the platform
> # default for Python 2.x.
>
> QT_API_ENV = os.environ.get('QT_API')
> if QT_API_ENV is not None:
> try:
> QT_API = ETS[QT_API_ENV]
> except KeyError:
> raise RuntimeError(
> 'Unrecognized environment variable %r, valid values are: %r or %r' %
> (QT_API_ENV, 'pyqt', 'pyside'))
> else:
> # No ETS environment, so use rcParams.
> QT_API = rcParams['backend.qt4']
>
> but trying to set it, via code or matplotlibrc results in: Unrecognized
> backend.qt4 string "PyQt4v2": valid strings are ['PySide', 'PyQt4']
> "%s"\n\t%s' % (val, cnt, line, fname, msg))
> ------------------------------------------------------------------------
> View this message in context: Re: QT draw issue in 1.1.0 and PyQt4v2
> missing?
> <http://old.nabble.com/QT-draw-issue-in-1.1.0-and-PyQt4v2-missing--tp32676093p32680414.html>
> Sent from the matplotlib - users mailing list archive
> <http://old.nabble.com/matplotlib---users-f2906.html> at Nabble.com.
>
>
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2d-oct
>
>
>
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users

efiring wrote:
> 
> https://github.com/matplotlib/matplotlib/pull/539
> I think this pull request fixes it.
> Eric
> 
Great, thanks for confirming and fixing!
Whilst waiting for this to make it into a release, I've hacked this to
achieve the same effect (so I don't have to patch + distribute the mpl
code):
 def do_draw_hack(self):
 fig = self.figure
 fig.canvas.draw()
 fig.draw(fig.canvas.get_renderer())
 fig.canvas.update()
-- 
View this message in context: http://old.nabble.com/QT-draw-issue-in-1.1.0-and-PyQt4v2-missing--tp32676093p32680445.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

Benjamin Root-2 wrote:
> 
> I think that might have been a little unclear. You should only need to
> select 'PyQt4' or 'PySide'. If PyQt4 is selected, then (I think) the v2
> is
> automatically tested for internally.
> 
hmm, please explain where you think it is used internally, as the code seem
to expect that this param can be set when there is no env variable 'QT_API'
set, in backends\qt4_compat.py:
# Available APIs.
QT_API_PYQT = 'PyQt4' # API is not set here; Python 2.x default is V 1
QT_API_PYQTv2 = 'PyQt4v2' # forced to Version 2 API
QT_API_PYSIDE = 'PySide' # only supports Version 2 API
ETS = dict(pyqt=QT_API_PYQTv2, pyside=QT_API_PYSIDE)
# If the ETS QT_API environment variable is set, use it. Note that
# ETS requires the version 2 of PyQt4, which is not the platform
# default for Python 2.x.
QT_API_ENV = os.environ.get('QT_API')
if QT_API_ENV is not None:
 try:
 QT_API = ETS[QT_API_ENV]
 except KeyError:
 raise RuntimeError(
 'Unrecognized environment variable %r, valid values are: %r or %r'
%
 (QT_API_ENV, 'pyqt', 'pyside'))
else:
 # No ETS environment, so use rcParams.
 QT_API = rcParams['backend.qt4']
but trying to set it, via code or matplotlibrc results in:
 Unrecognized backend.qt4 string "PyQt4v2": valid strings are
['PySide', 'PyQt4']
 "%s"\n\t%s' % (val, cnt, line, fname, msg))
-- 
View this message in context: http://old.nabble.com/QT-draw-issue-in-1.1.0-and-PyQt4v2-missing--tp32676093p32680414.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

Thanks Jeff, that certainly speeds it up! But when I take them out of the
loop and place them elsewhere they are no longer added to the map.
Is there someway I can call them in the loop but still get it to run
quickly?
Thanks
On Fri, Oct 14, 2011 at 10:54 PM, Jeff Whitaker <js...@fa...> wrote:
> On 10/12/11 8:20 PM, questions anon wrote:
>
> Hi All,
> I keep receiving a memory error when processing many netcdf files. I
> assumed it had something to do with how I loop things and maybe needed to
> close things off properly but I recently received an error that made me
> think it might be because of matplotlib.
>
> In the code below I am looping through a bunch of netcdf files (each file
> is hourly data for one month) and within each netcdf file I am outputting a
> *png file every three hours. This works for one netcdf file (therefore one
> month) but when it begins to process the next netcdf file I receive a memory
> error (see below). Since I have tidied some of my code up it seems to
> process partly into the second file but then I still receive the memory
> error.
> I have tried a few suggestions such as:
> -Combining the dataset using MFDataset (using NETCDF4) is not an option
> because the files do not have unlimited dimension.
> - gc.collect() but that just results in a *GEOS_ERROR: bad allocation
> error*.
> -only open LAT and LON once (which worked)
>
> System Details:
> Python 2.7.2 |EPD 7.1-2 (32-bit)| (default, Jul 3 2011, 15:13:59) [MSC
> v.1500 32 bit (Intel)] on win32
>
> Any feedback will be greatly appreciated as I seem to keep ending up with
> memory errors when working with netcdf files this even happens if I am using
> a much better computer.
>
> *Most recent error: *
> Traceback (most recent call last):
> File "C:\plot_netcdf_merc_multiplot_across_multifolders_TSFC.py", line
> 78, in <module>
> plt.savefig((os.path.join(outputfolder, 'TSFC'+date_string+'UTC.png')))
> File "C:\Python27\lib\site-packages\matplotlib\pyplot.py", line 363, in
> savefig
> return fig.savefig(*args, **kwargs)
> File "C:\Python27\lib\site-packages\matplotlib\figure.py", line 1084, in
> savefig
> self.canvas.print_figure(*args, **kwargs)
> File
> "C:\Python27\lib\site-packages\matplotlib\backends\backend_wxagg.py", line
> 100, in print_figure
> FigureCanvasAgg.print_figure(self, filename, *args, **kwargs)
> File "C:\Python27\lib\site-packages\matplotlib\backend_bases.py", line
> 1923, in print_figure
> **kwargs)
> File "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
> line 438, in print_png
> FigureCanvasAgg.draw(self)
> File "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
> line 393, in draw
> self.renderer = self.get_renderer()
> File "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
> line 404, in get_renderer
> self.renderer = RendererAgg(w, h, self.figure.dpi)
> File "C:\Python27\lib\site-packages\matplotlib\backends\backend_agg.py",
> line 59, in __init__
> self._renderer = _RendererAgg(int(width), int(height), dpi,
> debug=False)
> RuntimeError: Could not allocate memory for image
>
> *Error when I added gc.collect()*
> GEOS_ERROR: bad allocation
>
> *Old error (before adding gc.collect() )*
> *Traceback (most recent call last):
> File "d:/plot_netcdf_merc_multiplot_across_multifolders__memoryerror.py",
> line 44, in <module>
> TSFC=ncfile.variables['T_SFC'][1::3]
> File "netCDF4.pyx", line 2473, in netCDF4.Variable.__getitem__
> (netCDF4.c:23094)
> MemoryError*
>
>
>
> from netCDF4 import Dataset
> import numpy as N
> import matplotlib.pyplot as plt
> from mpl_toolkits.basemap import Basemap
> from netcdftime import utime
> from datetime import datetime
> import os
> import gc
>
>
> shapefile1="E:/
>>
>> griddeddatasamples/GIS/DSE_REGIONS"
>> MainFolder=r"E:/griddeddatasamples/GriddedData/InputsforValidation/T_SFC/"
>> OutputFolder=r"E:/griddeddatasamples/GriddedData/OutputsforValidation"
>> fileforlatlon=Dataset("E:/griddeddatasamples/GriddedData/InputsforValidation/T_SFC/TSFC_1974_01/IDZ00026_VIC_ADFD_T_SFC.nc",
>> 'r+', 'NETCDF4')
>> LAT=fileforlatlon.variables['latitude'][:]
>> LON=fileforlatlon.variables['longitude'][:]
>>
>> for (path, dirs, files) in os.walk(MainFolder):
>> for dir in dirs:
>> print dir
>> path=path+'/'
>> for ncfile in files:
>> if ncfile[-3:]=='.nc':
>> print "dealing with ncfiles:", ncfile
>> ncfile=os.path.join(path,ncfile)
>> ncfile=Dataset(ncfile, 'r+', 'NETCDF4')
>> TSFC=ncfile.variables['T_SFC'][1::3]
>> TIME=ncfile.variables['time'][1::3]
>> ncfile.close()
>> gc.collect()
>>
>> for TSFC, TIME in zip((TSFC[:]),(TIME[:])):
>> cdftime=utime('seconds since 1970年01月01日 00:00:00')
>> ncfiletime=cdftime.num2date(TIME)
>> print ncfiletime
>> timestr=str(ncfiletime)
>> d = datetime.strptime(timestr, '%Y-%m-%d %H:%M:%S')
>> date_string = d.strftime('%Y%m%d_%H%M')
>>
>> map =
>> Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
>>
>> llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
>> x,y=map(*N.meshgrid(LON,LAT))
>> map.drawcoastlines(linewidth=0.5)
>> map.readshapefile(shapefile1, 'DSE_REGIONS')
>> map.drawstates()
>>
>> plt.title('Surface temperature at %s UTC'%ncfiletime)
>> ticks=[-5,0,5,10,15,20,25,30,35,40,45,50]
>> CS = map.contourf(x,y,TSFC, ticks, cmap=plt.cm.jet)
>> l,b,w,h =0.1,0.1,0.8,0.8
>> cax = plt.axes([l+w+0.025, b, 0.025, h], )
>> cbar=plt.colorbar(CS, cax=cax, drawedges=True)
>>
>> plt.savefig((os.path.join(OutputFolder,
>> 'TSFC'+date_string+'UTC.png')))
>> plt.close()
>> gc.collect()
>>
>>
> Try moving these lines
>
>
> map =
> Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
>
> llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')
> x,y=map(*N.meshgrid(LON,LAT))
> map.drawcoastlines(linewidth=0.5)
> map.readshapefile(shapefile1, 'DSE_REGIONS')
> map.drawstates()
>
> out of the loop.
>
> -Jeff
>