Hi, folks! I'm again encountering the problem - imshow generating a
MemoryError exception trying to image a very large array - discussed in this
thread I started almost a year and a half ago.
Question 1) has anything changed in MPL in that time interval which would
provide an "easy" solution?
Question 2) is there some way I can add pieces of the array incrementally to
the image into their proper place, i.e., modify the following code:
ax.imshow(image[0:ny/2+1, 0:nx/2+1]) # upper left corner of image
ax.hold(True)
ax.imshow(argW[ny/2+1:-1, 0:nx/2+1]) # lower left corner of image
ax.imshow(argW[0:ny/2+1, nx/2+1:-1]) # upper right corner of image
ax.imshow(argW[ny/2+1:-1, nx/2+1:-1]) # lower right corner of image
so that each subsequent imshow doesn't cover up the previous imshow and is
placed in the right place relative to each of the other pieces?
Question 3) Would such incremental addition work to get around the memory
limit, or does the fact (if the following statement is in fact correct) that
eventually the entire too-large image needs to be handled doom this strategy
to failure?
Question 4) would I have this problem if I was running 64 bit (i.e., OS, as
well as 64 bit builds of Python, numpy, MPL, etc.), i.e., is it most likely
a memory addressing problem?
Question 5) can anyone suggest any other work-around(s)?
Thanks!
DG
On Sat, Sep 6, 2008 at 4:00 PM, David Goldsmith <d_l...@ya...>wrote:
> Ah, Ich verstehe now. I'll try RGBA-ing it; in the meantime, let me know
> if the colormapping conversion gets changed to 32 bit. Thanks again!
>
> DG
>
> --- On Sat, 9/6/08, Eric Firing <ef...@ha...> wrote:
>
> > From: Eric Firing <ef...@ha...>
> > Subject: Re: [Matplotlib-users] imshow size limitations?
> > To: d_l...@ya...
> > Cc: mat...@li...
> > Date: Saturday, September 6, 2008, 3:13 PM
> > David Goldsmith wrote:
> > > Thanks, Eric!
> > >
> > > --- On Sat, 9/6/08, Eric Firing
> > <ef...@ha...> wrote:
> > >
> > > -- snip OP --
> > >
> > >> It looks to me like you simply ran out of
> > memory--this is
> > >> not an imshow
> > >> problem as such. Your array is about 1e8
> > elements, and as
> > >> floats that
> > >> would be close to a GB--just for that array alone.
> > Do you
> > >
> > > Well, I anticipated that, so I do initialize the
> > storage for the numpy array as numpy.uint8 and have
> > confirmed that the data in the array returned by the
> > function which creates it remains numpy.uint8, so it should
> > "only" be ~100MB (indeed, the .na file into which
> > I tofile it is 85,430 KB, just as it should be for a 10800 x
> > 8100 array of uint8 elements). And the ax.imshow statement
> > doesn't (directly) cause the crash (but I don't know
> > that it isn't making either a float copy or an in-place
> > conversion of the array). So, AFAIK, right up until the
> > statement:
> > >
> > > canvas.print_figure('HiResHex')
> > >
> > > the data being imaged are all numpy.uint8 type.
> >
> > Yes, but it looks to me like they are still getting
> > color-mapped, and
> > this requires conversion to numpy.float. This may be a bad
> > aspect of
> > the mpl design, but it is quite deeply embedded. I suspect
> > the best we
> > could do would be to use float32 instead of float64;
> > certainly for color
> > mapping one does not need 64 bits.
> >
> > Using numpy.uint8 helps only if you are specifying RGBA
> > directly,
> > bypassing the colormapping.
> >
> > >
> > >> really need
> > >> all that resolution?
> > >
> > > Well, there's the rub: I fancy myself a fractal
> > "artist" and I have
> > > access to an HP DesignJet 500ps plotter with a maximum
> > resolution of
> > > 1200 x 600 dpi. For the size images I'm trying to
> > make (I'm hoping to go
> > > even bigger than 36" x 27", but I figured
> > that as a good starting point)
> > > even I regard _that_ resolution as too much - I was
> > thinking of 300 x
> > > 300 dpi (which is its "normal" resolution)
> > as certainly worthy of giving
> > > a try. :-)
> >
> > >> If you do, you will probably have to
> > >> get a much
> > >> more capable machine.
> > >
> > > Possible, but I was hoping to generate at least one
> > "proof" first to determine how hard I'd need
> > to try.
> > >
> > >> Otherwise, you need to knock down
> > >> the size of
> > >> that array before trying to plot or otherwise
> > manipulate
> > >> it.
> > >
> > > Forgive me, but I'd like a more detailed
> > explanation as to why: I
> > > have
> > > ample (~35 GB, just on my built-in disc, much more
> > than that on external
> > > discs) harddisc space - isn't there some way to
> > leverage that?
> >
> > I don't know enough about virtual memory
> > implementations--especially on
> > Win or Mac--to say. In practice, I suspect you would find
> > that as soon
> > as you are doing major swapping during a calculation, you
> > will thrash
> > the disk until you run out of patience.
> >
> >
> > >> With respect to imshow, probably you can get it to
> > handle
> > >> larger images
> > >
> > > Again, imshow doesn't appear to be the culprit
> > (contrary to my
> > > original subject line), rather it would appear to be
> > > canvas.print_figure. (While I'm on the subject of
> > canvas.print_figure,
> > > isn't there some way for MPL to "splash"
> > the image directly to the
> > > screen, without first having to write to a file? I
> > didn't ask this
> > > before because I did eventually want to write the
> > image to a file, but I
> > > would prefer to do so only after I've had a look
> > at it.)
> >
> > It is imshow in the sense that most of the action in mpl
> > doesn't happen
> > when you call imshow or plot or whatever--they just set
> > things up. The
> > real work is done in the backend when you display with
> > show() or write
> > to a file.
> >
> >
> > >> if you feed them in as NxMx4 numpy.uint8 RGBA
> > arrays--but I
> > >> doubt this
> > >> is going to be enough, or the right approach, for
> > your
> > >> present situation.
> > >
> > > Right: I don't see how that would be better than
> > having a single 8
> > > bit
> > > datum at each point w/ color being determined from a
> > color map (which is
> > > how I'd prefer to do it anyway).
> >
> > The way it is better is that it avoids a major operation,
> > including the
> > generation of the double-precision array. The rgba array
> > can go
> > straight to agg.
> >
> > Eric
> >
> >
> > > Thanks again,
> > >
> > > DG
> > >> Eric
> > >>
> > >>> Platform Details: MPL 0.91.2 (sorry, I
> > didn't
> > >> realize I was running such an old version, maybe I
> > just need
> > >> to upgrade?), Python 2.5.2, Windows XP 2002 SP3,
> > 504MB
> > >> physical RAM, 1294MB VM Page size (1000MB init.,
> > 5000MB max)
> > >>> Thanks!
> > >>>
> > >>> DG
>
>
>
>
--- On Sun, 2/28/10, David Goldsmith <d_l...@ya...> wrote: > > >> > Question 2) is there some way I can add > > pieces of the > > >> array incrementally to > > >> > the image into their proper place, > i.e., > > modify the > > >> following code: > > >> > > > >> > ax.imshow(image[0:ny/2+1, > 0:nx/2+1]) # > > upper > > >> left corner of image > > >> > ax.hold(True) > > >> > ax.imshow(argW[ny/2+1:-1, > 0:nx/2+1]) # > > lower > > >> left corner of image > > >> > ax.imshow(argW[0:ny/2+1, > nx/2+1:-1]) # > > upper > > >> right corner of image > > >> > ax.imshow(argW[ny/2+1:-1, > nx/2+1:-1]) > > # lower > > >> right corner of image > > >> > > >> Try the extents keyword argument. It let's > you > > specify the > > >> corners of > > >> the image in data coordinates. > > >> > > >> Ryan > > > > > > Hi, Ryan, thanks! Can you be a little more > specific > > as to how I should try that? I tried: > > > > > > ax.imshow(argW[0:ny/2+1, 0:nx/2+1], cmap_name, > > extent=(0,nx/2,ny/2,0)) > > > ax.hold(True) > > > ax.imshow(argW[ny/2+1:-1, 0:nx/2+1], cmap_name, > > extent=(0,nx/2,ny,ny/2)) > > > ax.imshow(argW[0:ny/2+1, nx/2+1:-1], cmap_name, > > extent=(nx/2,nx,ny/2,0)) > > > ax.imshow(argW[ny/2+1:-1, nx/2+1:-1], cmap_name, > > extent=(nx/2,nx,ny,ny/2)) > > > > > > which didn't work (I only got one "corner" - the > last > > one, I think - i.e., I think it's still just putting > > subsequent images on top of prior ones). > > > > Based on just a quick look, I'd make sure: > > > > 1) To set the x and y limits appropriately: > > > > ax.set_xlim(0, nx) > > ax.set_ylim(ny, 0) > > I'll try it out and report back. Nope, still only getting the last corner. Let me give a little more of my code: import numpy as N from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas from matplotlib.figure import Figure w, h, DPI = (8.2, 6.2, 50) fig = Figure(figsize=(w, h), dpi=DPI, frameon=False) ax = fig.add_subplot(1,1,1) canvas = FigureCanvas(fig) nx, xmin, xmax = (int(w*DPI), -0.5, 0.5) ny, ymin, ymax = (int(h*DPI), 0.6, 1.2) W = N.zeros((ny,nx),N.complex) ## Generate non-trivial W; unnecessary for code to run, I think argW = N.angle(W) ax.set_xlim(0, nx) ax.set_ylim(ny, 0) ax.imshow(argW[0:ny/2, 0:nx/2], extent=(0, nx/2 - 1, ny/2 - 1, 0)) ax.hold(True) ax.imshow(argW[ny/2:-1, 0:nx/2], extent=(0, nx/2 - 1, ny, ny/2)) ax.imshow(argW[0:ny/2, nx/2:-1], extent=(nx/2, nx, ny/2 - 1, 0)) ax.imshow(argW[ny/2:-1, nx/2:-1], extent=(nx/2, nx, ny, ny/2)) canvas.print_figure(fid, dpi=DPI) DG
David Goldsmith wrote: > --- On Sun, 2/28/10, David Goldsmith <d_l...@ya...> wrote: > > >>>>>> Question 2) is there some way I can add >>>>>> >>> pieces of the >>> >>>>> array incrementally to >>>>> >>>>>> the image into their proper place, >>>>>> >> i.e., >> >>> modify the >>> >>>>> following code: >>>>> >>>>>> ax.imshow(image[0:ny/2+1, >>>>>> >> 0:nx/2+1]) # >> >>> upper >>> >>>>> left corner of image >>>>> >>>>>> ax.hold(True) >>>>>> ax.imshow(argW[ny/2+1:-1, >>>>>> >> 0:nx/2+1]) # >> >>> lower >>> >>>>> left corner of image >>>>> >>>>>> ax.imshow(argW[0:ny/2+1, >>>>>> >> nx/2+1:-1]) # >> >>> upper >>> >>>>> right corner of image >>>>> >>>>>> ax.imshow(argW[ny/2+1:-1, >>>>>> >> nx/2+1:-1]) >> >>> # lower >>> >>>>> right corner of image >>>>> >>>>> Try the extents keyword argument. It let's >>>>> >> you >> >>> specify the >>> >>>>> corners of >>>>> the image in data coordinates. >>>>> >>>>> Ryan >>>>> >>>> Hi, Ryan, thanks! Can you be a little more >>>> >> specific >> >>> as to how I should try that? I tried: >>> >>>> ax.imshow(argW[0:ny/2+1, 0:nx/2+1], cmap_name, >>>> >>> extent=(0,nx/2,ny/2,0)) >>> >>>> ax.hold(True) >>>> ax.imshow(argW[ny/2+1:-1, 0:nx/2+1], cmap_name, >>>> >>> extent=(0,nx/2,ny,ny/2)) >>> >>>> ax.imshow(argW[0:ny/2+1, nx/2+1:-1], cmap_name, >>>> >>> extent=(nx/2,nx,ny/2,0)) >>> >>>> ax.imshow(argW[ny/2+1:-1, nx/2+1:-1], cmap_name, >>>> >>> extent=(nx/2,nx,ny,ny/2)) >>> >>>> which didn't work (I only got one "corner" - the >>>> >> last >> >>> one, I think - i.e., I think it's still just putting >>> subsequent images on top of prior ones). >>> >>> Based on just a quick look, I'd make sure: >>> >>> 1) To set the x and y limits appropriately: >>> >>> ax.set_xlim(0, nx) >>> ax.set_ylim(ny, 0) >>> >> I'll try it out and report back. >> > > Nope, still only getting the last corner. Let me give a little more of my code: > David: Just add ax.set_xlim(0,nx) ax.set_ylim(0,ny) after you imshow calls. The axes limits are being automatically set to match your last invocation of imshow. -Jeff > import numpy as N > from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas > from matplotlib.figure import Figure > > w, h, DPI = (8.2, 6.2, 50) > fig = Figure(figsize=(w, h), > dpi=DPI, > frameon=False) > ax = fig.add_subplot(1,1,1) > canvas = FigureCanvas(fig) > > nx, xmin, xmax = (int(w*DPI), -0.5, 0.5) > ny, ymin, ymax = (int(h*DPI), 0.6, 1.2) > W = N.zeros((ny,nx),N.complex) > > ## Generate non-trivial W; unnecessary for code to run, I think > > argW = N.angle(W) > > ax.set_xlim(0, nx) > ax.set_ylim(ny, 0) > ax.imshow(argW[0:ny/2, 0:nx/2], extent=(0, nx/2 - 1, ny/2 - 1, 0)) > ax.hold(True) > ax.imshow(argW[ny/2:-1, 0:nx/2], extent=(0, nx/2 - 1, ny, ny/2)) > ax.imshow(argW[0:ny/2, nx/2:-1], extent=(nx/2, nx, ny/2 - 1, 0)) > ax.imshow(argW[ny/2:-1, nx/2:-1], extent=(nx/2, nx, ny, ny/2)) > canvas.print_figure(fid, dpi=DPI) > > DG >
On Sat, Feb 27, 2010 at 2:23 PM, David Goldsmith <d_l...@ya...> wrote: > Question 2) is there some way I can add pieces of the array incrementally to > the image into their proper place, i.e., modify the following code: > > ax.imshow(image[0:ny/2+1, 0:nx/2+1]) # upper left corner of image > ax.hold(True) > ax.imshow(argW[ny/2+1:-1, 0:nx/2+1]) # lower left corner of image > ax.imshow(argW[0:ny/2+1, nx/2+1:-1]) # upper right corner of image > ax.imshow(argW[ny/2+1:-1, nx/2+1:-1]) # lower right corner of image Try the extents keyword argument. It let's you specify the corners of the image in data coordinates. Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma
On Sun, Feb 28, 2010 at 12:24 PM, David Goldsmith <d_l...@ya...> wrote: > --- On Sat, 2/27/10, Ryan May <rm...@gm...> wrote: > >> From: Ryan May <rm...@gm...> >> Subject: Re: [Matplotlib-users] imshow size limitations? >> To: "David Goldsmith" <d_l...@ya...> >> Cc: "matplotlib-users" <mat...@li...> >> Date: Saturday, February 27, 2010, 7:58 PM >> On Sat, Feb 27, 2010 at 2:23 PM, >> David Goldsmith >> <d_l...@ya...> >> wrote: >> > Question 2) is there some way I can add pieces of the >> array incrementally to >> > the image into their proper place, i.e., modify the >> following code: >> > >> > ax.imshow(image[0:ny/2+1, 0:nx/2+1]) # upper >> left corner of image >> > ax.hold(True) >> > ax.imshow(argW[ny/2+1:-1, 0:nx/2+1]) # lower >> left corner of image >> > ax.imshow(argW[0:ny/2+1, nx/2+1:-1]) # upper >> right corner of image >> > ax.imshow(argW[ny/2+1:-1, nx/2+1:-1]) # lower >> right corner of image >> >> Try the extents keyword argument. It let's you specify the >> corners of >> the image in data coordinates. >> >> Ryan > > Hi, Ryan, thanks! Can you be a little more specific as to how I should try that? I tried: > > ax.imshow(argW[0:ny/2+1, 0:nx/2+1], cmap_name, extent=(0,nx/2,ny/2,0)) > ax.hold(True) > ax.imshow(argW[ny/2+1:-1, 0:nx/2+1], cmap_name, extent=(0,nx/2,ny,ny/2)) > ax.imshow(argW[0:ny/2+1, nx/2+1:-1], cmap_name, extent=(nx/2,nx,ny/2,0)) > ax.imshow(argW[ny/2+1:-1, nx/2+1:-1], cmap_name, extent=(nx/2,nx,ny,ny/2)) > > which didn't work (I only got one "corner" - the last one, I think - i.e., I think it's still just putting subsequent images on top of prior ones). (Putting back on list) Based on just a quick look, I'd make sure: 1) To set the x and y limits appropriately: ax.set_xlim(0, nx) ax.set_ylim(ny, 0) 2) Make sure to use the same colormapping limits, by using an instance of normalize: # Can also import Normalize from matplotlib.colors norm = plt.Normalize(datamin, datamax) ax.imshow(argW[0:ny/2+1, 0:nx/2+1], cmap_name, extent=(0,nx/2,ny/2,0), norm=norm) I'm pretty sure #1 is your problem in seeing, but #2 would potentially cause a funky looking image. Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma
--- On Sun, 2/28/10, Ryan May <rm...@gm...> wrote: > David Goldsmith > <d_l...@ya...> > wrote: > > --- On Sat, 2/27/10, Ryan May <rm...@gm...> > wrote: > >> David Goldsmith > >> <d_l...@ya...> > >> wrote: > >> > Question 2) is there some way I can add > pieces of the > >> array incrementally to > >> > the image into their proper place, i.e., > modify the > >> following code: > >> > > >> > ax.imshow(image[0:ny/2+1, 0:nx/2+1]) # > upper > >> left corner of image > >> > ax.hold(True) > >> > ax.imshow(argW[ny/2+1:-1, 0:nx/2+1]) # > lower > >> left corner of image > >> > ax.imshow(argW[0:ny/2+1, nx/2+1:-1]) # > upper > >> right corner of image > >> > ax.imshow(argW[ny/2+1:-1, nx/2+1:-1]) > # lower > >> right corner of image > >> > >> Try the extents keyword argument. It let's you > specify the > >> corners of > >> the image in data coordinates. > >> > >> Ryan > > > > Hi, Ryan, thanks! Can you be a little more specific > as to how I should try that? I tried: > > > > ax.imshow(argW[0:ny/2+1, 0:nx/2+1], cmap_name, > extent=(0,nx/2,ny/2,0)) > > ax.hold(True) > > ax.imshow(argW[ny/2+1:-1, 0:nx/2+1], cmap_name, > extent=(0,nx/2,ny,ny/2)) > > ax.imshow(argW[0:ny/2+1, nx/2+1:-1], cmap_name, > extent=(nx/2,nx,ny/2,0)) > > ax.imshow(argW[ny/2+1:-1, nx/2+1:-1], cmap_name, > extent=(nx/2,nx,ny,ny/2)) > > > > which didn't work (I only got one "corner" - the last > one, I think - i.e., I think it's still just putting > subsequent images on top of prior ones). > > (Putting back on list) Sorry, my (unintentioned) bad. :-( > Based on just a quick look, I'd make sure: > > 1) To set the x and y limits appropriately: > > ax.set_xlim(0, nx) > ax.set_ylim(ny, 0) OK, thanks! > 2) Make sure to use the same colormapping limits, by using > an instance > of normalize: > > # Can also import Normalize from matplotlib.colors > norm = plt.Normalize(datamin, datamax) > ax.imshow(argW[0:ny/2+1, 0:nx/2+1], cmap_name, > extent=(0,nx/2,ny/2,0), > norm=norm) > > I'm pretty sure #1 is your problem in seeing, but #2 would > potentially > cause a funky looking image. Right, thanks, I was worried about that, but I thought "one problem at a time." ;-) I'll try it out and report back. DG
2010年2月27日 David Goldsmith <d_l...@ya...>: > ax.imshow(image[0:ny/2+1, 0:nx/2+1]) # upper left corner of image > ax.imshow(argW[ny/2+1:-1, 0:nx/2+1]) # lower left corner of image > ax.imshow(argW[0:ny/2+1, nx/2+1:-1]) # upper right corner of image > ax.imshow(argW[ny/2+1:-1, nx/2+1:-1]) # lower right corner of image Some tiny improvement: ax.imshow(argW[:ny/2+1, :nx/2+1]) ax.imshow(argW[ny/2+1:, :nx/2+1]) ax.imshow(argW[:ny/2+1, nx/2+1:]) ax.imshow(argW[ny/2+1:, nx/2+1:]) The main advantage is that you do not cut off the last pixel row/column by indicing [:-1], which will run until the last index *before* the index -1. >>> a = numpy.asarray([1, 2, 3]) >>> a[:-1] array([1, 2]) Friedrich