matplotlib-users Mailing List for matplotlib

Dear All,
I am using Python 3.4.1 on Debian. Matplotlib works fine for me, but when I
try to save the animation using moviewriter I get error.
My program file:
************************** short test
program**************************************
#!/usr/bin/env python3
from matplotlib import animation as anim
from matplotlib import pyplot as plt
import numpy as np
fig = plt.figure() # the basic canvas
plt.axis([-1,1,-1,1]) # set the x-axis
abcissa, = plt.plot([],[], color='black')
 # initialise the x-axis
graph, = plt.plot([], [], color='blue')
 # initialise the graph
t, = plt.plot([],[], color='red')
def init():
 x = np.arange(-1,1,0.01)# the x-values
 y = np.sin(np.pi/2 * x)
 abcissa.set_data(x, 0) # draw the x-axis
 graph.set_data(x, y) # draw the graph
 return abcissa, graph
def animate(i):
 p = -1 + i*0.02 # the pt p where the tangent is to be drawn
 tx = [p-1,p+1]
 ty = [np.sin(np.pi/2 * p) - np.pi/2 * np.cos(np.pi/2 *p),
 np.sin(np.pi/2 * p) + np.pi/2 * np.cos(np.pi/2 *p)]
 t.set_data(tx, ty)
 return t,
 
ani = anim.FuncAnimation(fig, animate, init_func=init,
 frames=100, interval=30, blit=True)
##plt.show()
Writer = anim.writers['ffmpeg']
writer = Writer(fps=15, bitrate=1800)
ani.save('test.mp4', writer=writer)
********************end of
program*************************************************
plt.show() shows correctly; on trying to save as above I get the following
error:
**************error from python
******************************************************
Traceback (most recent call last):
 File "/home/partha/নথিপত্র/স্বরচিত/আমারপ্রোগ্রাম/test.py", line 48, in
<module>
 ani.save('test.mp4', writer=writer)
 File "/usr/lib/python3/dist-packages/matplotlib/animation.py", line 719,
in save
 writer.grab_frame(**savefig_kwargs)
 File "/usr/lib/python3/dist-packages/matplotlib/animation.py", line 205,
in grab_frame
 dpi=self.dpi, **savefig_kwargs)
 File "/usr/lib/python3/dist-packages/matplotlib/figure.py", line 1422, in
savefig
 self.canvas.print_figure(*args, **kwargs)
 File "/usr/lib/python3/dist-packages/matplotlib/backend_bases.py", line
2225, in print_figure
 self.figure.dpi = origDPI
 File "/usr/lib/python3/dist-packages/matplotlib/figure.py", line 385, in
_set_dpi
 self.dpi_scale_trans.clear().scale(dpi, dpi)
 File "/usr/lib/python3/dist-packages/matplotlib/transforms.py", line 1786,
in clear
 self._mtx = np.identity(3)
 File "/usr/lib/python3/dist-packages/numpy/core/numeric.py", line 2053, in
identity
 from numpy import eye
 File "<frozen importlib._bootstrap>", line 2280, in _handle_fromlist
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0:
invalid start byte
*************end of
error****************************************************
As you can see from the first line of the error, my locale is set to bn_IN
(Bengali, India).
Any help in trying to resolve this problem would be highly appreciated.
With my thanks and regards,
partha
--
View this message in context: http://matplotlib.1069221.n5.nabble.com/matplotlib-error-on-trying-to-use-moviewriter-tp43916.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

Hi Ben and Ryan,
I will try to figure out as it works. 
Thank you.
Regards, 
Raf
-----Original Message-----
From: ben...@gm... on behalf of Benjamin Root
Sent: Tue 9/9/2014 3:25 PM
To: Ryan Nelson
Cc: Raffaele Quarta; Matplotlib Users
Subject: Re: [Matplotlib-users] Plotting large file (NetCDF)
 
Most of the time, you will not need to use meshgrid. Take advantage of
numpy's broadcasting feature:
http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html
It saves *significantly* on memory and processing time. Most of
Matplotlib's plotting functions work well with broadcastable inputs, so
that is a great way to save on memory. NumPy's ogrid is also a neat tool
for generating broadcastable grids.
When I get a chance, I'll look through the script for any other obvious
savers.
Cheers!
Ben Root
On Tue, Sep 9, 2014 at 9:02 AM, Ryan Nelson <rne...@gm...> wrote:
> Raffaele,
>
> As Ben pointed out, you might be creating a lot of in memory Numpy arrays
> that you probably don't need/want.
>
> For example, I think (?) slicing all of the variable below:
> lons = fh.variables['lon'][:]
> is making a copy of all that (mmap'ed) data as a Numpy array in memory.
> Get rid of the slice ([:]). Of course, these variables are not Numpy
> arrays, so you'll have to change some of your code. For example:
> lon_0 = lons.mean()
> Will have to become:
> lon_0 = np.mean( lons )
>
> If lats and lons are very large sets of data, then meshgrid will make two
> very, very large arrays in memory.
> For example, try this:
> np.meshgrid(np.arange(5), np.arange(5))
> The output is two much larger arrays:
> [array([[0, 1, 2, 3, 4],
> [0, 1, 2, 3, 4],
> [0, 1, 2, 3, 4],
> [0, 1, 2, 3, 4],
> [0, 1, 2, 3, 4]]),
> array([[0, 0, 0, 0, 0],
> [1, 1, 1, 1, 1],
> [2, 2, 2, 2, 2],
> [3, 3, 3, 3, 3],
> [4, 4, 4, 4, 4]])]
> I don't know Basemap at all, so I don't know if this is necessary. You
> might be able to force the meshgrid output into a memmap file, but I don't
> know how to do that right now. Perhaps someone else has some suggestions.
>
> Hope that helps.
>
> Ryan
>
>
>
>
> On Tue, Sep 9, 2014 at 4:07 AM, Raffaele Quarta <
> raf...@li...> wrote:
>
>> Hi Jody and Ben,
>>
>> thanks for your answers.
>> I tried to use pcolormesh instead of pcolor and the result is very good!
>> For what concern with the memory system problem, I wasn't able to solve it.
>> When I tried to use the bigger file, I got the same problem. Attached you
>> will find the script that I'm using to make the plot. May be, I didn't
>> understand very well how can I use the mmap function.
>>
>> Regards,
>>
>> Raffaele.
>>
>>
>> -----Original Message-----
>> From: Jody Klymak [mailto:jk...@uv... <jk...@uv...>]
>> Sent: Mon 9/8/2014 5:46 PM
>> To: Benjamin Root
>> Cc: Raffaele Quarta; Matplotlib Users
>> Subject: Re: [Matplotlib-users] Plotting large file (NetCDF)
>>
>> It looks like you are calling `pcolor`. Can I suggest you try
>> `pcolormesh`? ii
>>
>> 75 Mb is not a big file!
>>
>> Cheers, Jody
>>
>>
>> On Sep 8, 2014, at 7:38 AM, Benjamin Root <ben...@ou...> wrote:
>>
>> > (Keeping this on the mailing list so that others can benefit)
>> >
>> > What might be happening is that you are keeping around too many numpy
>> arrays in memory than you actually need. Take advantage of memmapping,
>> which most netcdf tools provide by default. This keeps the data on disk
>> rather than in RAM. Second, for very large images, I would suggest either
>> pcolormesh() or just simply imshow() instead of pcolor() as they are more
>> way more efficient than pcolor(). In addition, it sounds like you are
>> dealing with re-sampled data ("at different zoom levels"). Does this mean
>> that you are re-running contour on re-sampled data? I am not sure what the
>> benefit of doing that is if one could just simply do the contour once at
>> the highest resolution.
>> >
>> > Without seeing any code, though, I can only provide generic suggestions.
>> >
>> > Cheers!
>> > Ben Root
>> >
>> >
>> > On Mon, Sep 8, 2014 at 10:12 AM, Raffaele Quarta <
>> raf...@li...> wrote:
>> > Hi Ben,
>> >
>> > sorry for the few details that I gave to you. I'm trying to make a
>> contour plot of a variable at different zoom levels by using high
>> resolution data. The aim is to obtain .PNG output images. Actually, I'm
>> working with big data (NetCDF file, dimension is about 75Mb). The current
>> Matplotlib version on my UBUNTU 14.04 machine is the 1.3.1 one. My system
>> has a RAM capacity of 8Gb.
>> > Actually, I'm dealing with memory system problems when I try to make a
>> plot. I got the error message as follow:
>> >
>> > --------------------------------------------
>> > cs = m.pcolor(xi,yi,np.squeeze(t))
>> > File "/usr/lib/pymodules/python2.7/mpl_toolkits/basemap/__init__.py",
>> line 521, in with_transform
>> > return plotfunc(self,x,y,data,*args,**kwargs)
>> > File "/usr/lib/pymodules/python2.7/mpl_toolkits/basemap/__init__.py",
>> line 3375, in pcolor
>> > x = ma.masked_values(np.where(x > 1.e20,1.e20,x), 1.e20)
>> > File "/usr/lib/python2.7/dist-packages/numpy/ma/core.py", line 2195,
>> in masked_values
>> > condition = umath.less_equal(mabs(xnew - value), atol + rtol *
>> mabs(value))
>> > MemoryError
>> > --------------------------------------------
>> >
>> > Otherwise, when I try to make a plot of smaller file (such as 5Mb), it
>> works very well. I believe that it's not something of wrong in the script.
>> It might be a memory system problem.
>> > I hope that my message is more clear now.
>> >
>> > Thanks for the help.
>> >
>> > Regards,
>> >
>> > Raffaele
>> >
>> > -----------------------------------------
>> >
>> > Sent: Mon 9/8/2014 3:19 PM
>> > To: Raffaele Quarta
>> > Cc: Matplotlib Users
>> > Subject: Re: [Matplotlib-users] Plotting large file (NetCDF)
>> >
>> >
>> >
>> > You will need to be more specific... much more specific. What kind of
>> plot
>> > are you making? How big is your data? What version of matplotlib are you
>> > using? How much RAM do you have available compared to the amount of data
>> > (most slowdowns are actually due to swap-thrashing issues). Matplotlib
>> can
>> > be used for large data, but there exists some speciality tools for the
>> > truly large datasets. The solution depends on the situation.
>> >
>> > Ben Root
>> >
>> > On Mon, Sep 8, 2014 at 7:45 AM, Raffaele Quarta <
>> raf...@li...>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > I'm working with NetCDF format. When I try to make a plot of very
>> large
>> > > file, I have to wait for a long time for plotting. How can I solve
>> this?
>> > > Isn't there a solution for this problem?
>> > >
>> > > Raffaele
>> > >
>> > > --
>> > > This email was Virus checked by Astaro Security Gateway.
>> http://www.sophos.com
>> > >
>> > >
>> > >
>> > >
>> ------------------------------------------------------------------------------
>> > > Want excitement?
>> > > Manually upgrade your production database.
>> > > When you want reliability, choose Perforce
>> > > Perforce version control. Predictably reliable.
>> > >
>> > >
>> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
>> > > _______________________________________________
>> > > Matplotlib-users mailing list
>> > > Mat...@li...
>> > > https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>> > >
>> > >
>> >
>> > --
>> > This email was Virus checked by Astaro Security Gateway.
>> http://www.sophos.com
>> >
>> >
>> >
>> >
>> ------------------------------------------------------------------------------
>> > Want excitement?
>> > Manually upgrade your production database.
>> > When you want reliability, choose Perforce
>> > Perforce version control. Predictably reliable.
>> >
>> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk_______________________________________________
>> > Matplotlib-users mailing list
>> > Mat...@li...
>> > https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>>
>> --
>> Jody Klymak
>> http://web.uvic.ca/~jklymak/
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Want excitement?
>> Manually upgrade your production database.
>> When you want reliability, choose Perforce.
>> Perforce version control. Predictably reliable.
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Matplotlib-users mailing list
>> Mat...@li...
>> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> Want excitement?
> Manually upgrade your production database.
> When you want reliability, choose Perforce.
> Perforce version control. Predictably reliable.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>
>

Most of the time, you will not need to use meshgrid. Take advantage of
numpy's broadcasting feature:
http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html
It saves *significantly* on memory and processing time. Most of
Matplotlib's plotting functions work well with broadcastable inputs, so
that is a great way to save on memory. NumPy's ogrid is also a neat tool
for generating broadcastable grids.
When I get a chance, I'll look through the script for any other obvious
savers.
Cheers!
Ben Root
On Tue, Sep 9, 2014 at 9:02 AM, Ryan Nelson <rne...@gm...> wrote:
> Raffaele,
>
> As Ben pointed out, you might be creating a lot of in memory Numpy arrays
> that you probably don't need/want.
>
> For example, I think (?) slicing all of the variable below:
> lons = fh.variables['lon'][:]
> is making a copy of all that (mmap'ed) data as a Numpy array in memory.
> Get rid of the slice ([:]). Of course, these variables are not Numpy
> arrays, so you'll have to change some of your code. For example:
> lon_0 = lons.mean()
> Will have to become:
> lon_0 = np.mean( lons )
>
> If lats and lons are very large sets of data, then meshgrid will make two
> very, very large arrays in memory.
> For example, try this:
> np.meshgrid(np.arange(5), np.arange(5))
> The output is two much larger arrays:
> [array([[0, 1, 2, 3, 4],
> [0, 1, 2, 3, 4],
> [0, 1, 2, 3, 4],
> [0, 1, 2, 3, 4],
> [0, 1, 2, 3, 4]]),
> array([[0, 0, 0, 0, 0],
> [1, 1, 1, 1, 1],
> [2, 2, 2, 2, 2],
> [3, 3, 3, 3, 3],
> [4, 4, 4, 4, 4]])]
> I don't know Basemap at all, so I don't know if this is necessary. You
> might be able to force the meshgrid output into a memmap file, but I don't
> know how to do that right now. Perhaps someone else has some suggestions.
>
> Hope that helps.
>
> Ryan
>
>
>
>
> On Tue, Sep 9, 2014 at 4:07 AM, Raffaele Quarta <
> raf...@li...> wrote:
>
>> Hi Jody and Ben,
>>
>> thanks for your answers.
>> I tried to use pcolormesh instead of pcolor and the result is very good!
>> For what concern with the memory system problem, I wasn't able to solve it.
>> When I tried to use the bigger file, I got the same problem. Attached you
>> will find the script that I'm using to make the plot. May be, I didn't
>> understand very well how can I use the mmap function.
>>
>> Regards,
>>
>> Raffaele.
>>
>>
>> -----Original Message-----
>> From: Jody Klymak [mailto:jk...@uv... <jk...@uv...>]
>> Sent: Mon 9/8/2014 5:46 PM
>> To: Benjamin Root
>> Cc: Raffaele Quarta; Matplotlib Users
>> Subject: Re: [Matplotlib-users] Plotting large file (NetCDF)
>>
>> It looks like you are calling `pcolor`. Can I suggest you try
>> `pcolormesh`? ii
>>
>> 75 Mb is not a big file!
>>
>> Cheers, Jody
>>
>>
>> On Sep 8, 2014, at 7:38 AM, Benjamin Root <ben...@ou...> wrote:
>>
>> > (Keeping this on the mailing list so that others can benefit)
>> >
>> > What might be happening is that you are keeping around too many numpy
>> arrays in memory than you actually need. Take advantage of memmapping,
>> which most netcdf tools provide by default. This keeps the data on disk
>> rather than in RAM. Second, for very large images, I would suggest either
>> pcolormesh() or just simply imshow() instead of pcolor() as they are more
>> way more efficient than pcolor(). In addition, it sounds like you are
>> dealing with re-sampled data ("at different zoom levels"). Does this mean
>> that you are re-running contour on re-sampled data? I am not sure what the
>> benefit of doing that is if one could just simply do the contour once at
>> the highest resolution.
>> >
>> > Without seeing any code, though, I can only provide generic suggestions.
>> >
>> > Cheers!
>> > Ben Root
>> >
>> >
>> > On Mon, Sep 8, 2014 at 10:12 AM, Raffaele Quarta <
>> raf...@li...> wrote:
>> > Hi Ben,
>> >
>> > sorry for the few details that I gave to you. I'm trying to make a
>> contour plot of a variable at different zoom levels by using high
>> resolution data. The aim is to obtain .PNG output images. Actually, I'm
>> working with big data (NetCDF file, dimension is about 75Mb). The current
>> Matplotlib version on my UBUNTU 14.04 machine is the 1.3.1 one. My system
>> has a RAM capacity of 8Gb.
>> > Actually, I'm dealing with memory system problems when I try to make a
>> plot. I got the error message as follow:
>> >
>> > --------------------------------------------
>> > cs = m.pcolor(xi,yi,np.squeeze(t))
>> > File "/usr/lib/pymodules/python2.7/mpl_toolkits/basemap/__init__.py",
>> line 521, in with_transform
>> > return plotfunc(self,x,y,data,*args,**kwargs)
>> > File "/usr/lib/pymodules/python2.7/mpl_toolkits/basemap/__init__.py",
>> line 3375, in pcolor
>> > x = ma.masked_values(np.where(x > 1.e20,1.e20,x), 1.e20)
>> > File "/usr/lib/python2.7/dist-packages/numpy/ma/core.py", line 2195,
>> in masked_values
>> > condition = umath.less_equal(mabs(xnew - value), atol + rtol *
>> mabs(value))
>> > MemoryError
>> > --------------------------------------------
>> >
>> > Otherwise, when I try to make a plot of smaller file (such as 5Mb), it
>> works very well. I believe that it's not something of wrong in the script.
>> It might be a memory system problem.
>> > I hope that my message is more clear now.
>> >
>> > Thanks for the help.
>> >
>> > Regards,
>> >
>> > Raffaele
>> >
>> > -----------------------------------------
>> >
>> > Sent: Mon 9/8/2014 3:19 PM
>> > To: Raffaele Quarta
>> > Cc: Matplotlib Users
>> > Subject: Re: [Matplotlib-users] Plotting large file (NetCDF)
>> >
>> >
>> >
>> > You will need to be more specific... much more specific. What kind of
>> plot
>> > are you making? How big is your data? What version of matplotlib are you
>> > using? How much RAM do you have available compared to the amount of data
>> > (most slowdowns are actually due to swap-thrashing issues). Matplotlib
>> can
>> > be used for large data, but there exists some speciality tools for the
>> > truly large datasets. The solution depends on the situation.
>> >
>> > Ben Root
>> >
>> > On Mon, Sep 8, 2014 at 7:45 AM, Raffaele Quarta <
>> raf...@li...>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > I'm working with NetCDF format. When I try to make a plot of very
>> large
>> > > file, I have to wait for a long time for plotting. How can I solve
>> this?
>> > > Isn't there a solution for this problem?
>> > >
>> > > Raffaele
>> > >
>> > > --
>> > > This email was Virus checked by Astaro Security Gateway.
>> http://www.sophos.com
>> > >
>> > >
>> > >
>> > >
>> ------------------------------------------------------------------------------
>> > > Want excitement?
>> > > Manually upgrade your production database.
>> > > When you want reliability, choose Perforce
>> > > Perforce version control. Predictably reliable.
>> > >
>> > >
>> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
>> > > _______________________________________________
>> > > Matplotlib-users mailing list
>> > > Mat...@li...
>> > > https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>> > >
>> > >
>> >
>> > --
>> > This email was Virus checked by Astaro Security Gateway.
>> http://www.sophos.com
>> >
>> >
>> >
>> >
>> ------------------------------------------------------------------------------
>> > Want excitement?
>> > Manually upgrade your production database.
>> > When you want reliability, choose Perforce
>> > Perforce version control. Predictably reliable.
>> >
>> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk_______________________________________________
>> > Matplotlib-users mailing list
>> > Mat...@li...
>> > https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>>
>> --
>> Jody Klymak
>> http://web.uvic.ca/~jklymak/
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Want excitement?
>> Manually upgrade your production database.
>> When you want reliability, choose Perforce.
>> Perforce version control. Predictably reliable.
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Matplotlib-users mailing list
>> Mat...@li...
>> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> Want excitement?
> Manually upgrade your production database.
> When you want reliability, choose Perforce.
> Perforce version control. Predictably reliable.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>
>

Raffaele,
As Ben pointed out, you might be creating a lot of in memory Numpy arrays
that you probably don't need/want.
For example, I think (?) slicing all of the variable below:
lons = fh.variables['lon'][:]
is making a copy of all that (mmap'ed) data as a Numpy array in memory. Get
rid of the slice ([:]). Of course, these variables are not Numpy arrays, so
you'll have to change some of your code. For example:
lon_0 = lons.mean()
Will have to become:
lon_0 = np.mean( lons )
If lats and lons are very large sets of data, then meshgrid will make two
very, very large arrays in memory.
For example, try this:
np.meshgrid(np.arange(5), np.arange(5))
The output is two much larger arrays:
[array([[0, 1, 2, 3, 4],
 [0, 1, 2, 3, 4],
 [0, 1, 2, 3, 4],
 [0, 1, 2, 3, 4],
 [0, 1, 2, 3, 4]]),
array([[0, 0, 0, 0, 0],
 [1, 1, 1, 1, 1],
 [2, 2, 2, 2, 2],
 [3, 3, 3, 3, 3],
 [4, 4, 4, 4, 4]])]
I don't know Basemap at all, so I don't know if this is necessary. You
might be able to force the meshgrid output into a memmap file, but I don't
know how to do that right now. Perhaps someone else has some suggestions.
Hope that helps.
Ryan
On Tue, Sep 9, 2014 at 4:07 AM, Raffaele Quarta <raf...@li...>
wrote:
> Hi Jody and Ben,
>
> thanks for your answers.
> I tried to use pcolormesh instead of pcolor and the result is very good!
> For what concern with the memory system problem, I wasn't able to solve it.
> When I tried to use the bigger file, I got the same problem. Attached you
> will find the script that I'm using to make the plot. May be, I didn't
> understand very well how can I use the mmap function.
>
> Regards,
>
> Raffaele.
>
>
> -----Original Message-----
> From: Jody Klymak [mailto:jk...@uv... <jk...@uv...>]
> Sent: Mon 9/8/2014 5:46 PM
> To: Benjamin Root
> Cc: Raffaele Quarta; Matplotlib Users
> Subject: Re: [Matplotlib-users] Plotting large file (NetCDF)
>
> It looks like you are calling `pcolor`. Can I suggest you try
> `pcolormesh`? ii
>
> 75 Mb is not a big file!
>
> Cheers, Jody
>
>
> On Sep 8, 2014, at 7:38 AM, Benjamin Root <ben...@ou...> wrote:
>
> > (Keeping this on the mailing list so that others can benefit)
> >
> > What might be happening is that you are keeping around too many numpy
> arrays in memory than you actually need. Take advantage of memmapping,
> which most netcdf tools provide by default. This keeps the data on disk
> rather than in RAM. Second, for very large images, I would suggest either
> pcolormesh() or just simply imshow() instead of pcolor() as they are more
> way more efficient than pcolor(). In addition, it sounds like you are
> dealing with re-sampled data ("at different zoom levels"). Does this mean
> that you are re-running contour on re-sampled data? I am not sure what the
> benefit of doing that is if one could just simply do the contour once at
> the highest resolution.
> >
> > Without seeing any code, though, I can only provide generic suggestions.
> >
> > Cheers!
> > Ben Root
> >
> >
> > On Mon, Sep 8, 2014 at 10:12 AM, Raffaele Quarta <
> raf...@li...> wrote:
> > Hi Ben,
> >
> > sorry for the few details that I gave to you. I'm trying to make a
> contour plot of a variable at different zoom levels by using high
> resolution data. The aim is to obtain .PNG output images. Actually, I'm
> working with big data (NetCDF file, dimension is about 75Mb). The current
> Matplotlib version on my UBUNTU 14.04 machine is the 1.3.1 one. My system
> has a RAM capacity of 8Gb.
> > Actually, I'm dealing with memory system problems when I try to make a
> plot. I got the error message as follow:
> >
> > --------------------------------------------
> > cs = m.pcolor(xi,yi,np.squeeze(t))
> > File "/usr/lib/pymodules/python2.7/mpl_toolkits/basemap/__init__.py",
> line 521, in with_transform
> > return plotfunc(self,x,y,data,*args,**kwargs)
> > File "/usr/lib/pymodules/python2.7/mpl_toolkits/basemap/__init__.py",
> line 3375, in pcolor
> > x = ma.masked_values(np.where(x > 1.e20,1.e20,x), 1.e20)
> > File "/usr/lib/python2.7/dist-packages/numpy/ma/core.py", line 2195,
> in masked_values
> > condition = umath.less_equal(mabs(xnew - value), atol + rtol *
> mabs(value))
> > MemoryError
> > --------------------------------------------
> >
> > Otherwise, when I try to make a plot of smaller file (such as 5Mb), it
> works very well. I believe that it's not something of wrong in the script.
> It might be a memory system problem.
> > I hope that my message is more clear now.
> >
> > Thanks for the help.
> >
> > Regards,
> >
> > Raffaele
> >
> > -----------------------------------------
> >
> > Sent: Mon 9/8/2014 3:19 PM
> > To: Raffaele Quarta
> > Cc: Matplotlib Users
> > Subject: Re: [Matplotlib-users] Plotting large file (NetCDF)
> >
> >
> >
> > You will need to be more specific... much more specific. What kind of
> plot
> > are you making? How big is your data? What version of matplotlib are you
> > using? How much RAM do you have available compared to the amount of data
> > (most slowdowns are actually due to swap-thrashing issues). Matplotlib
> can
> > be used for large data, but there exists some speciality tools for the
> > truly large datasets. The solution depends on the situation.
> >
> > Ben Root
> >
> > On Mon, Sep 8, 2014 at 7:45 AM, Raffaele Quarta <
> raf...@li...>
> > wrote:
> >
> > > Hi,
> > >
> > > I'm working with NetCDF format. When I try to make a plot of very large
> > > file, I have to wait for a long time for plotting. How can I solve
> this?
> > > Isn't there a solution for this problem?
> > >
> > > Raffaele
> > >
> > > --
> > > This email was Virus checked by Astaro Security Gateway.
> http://www.sophos.com
> > >
> > >
> > >
> > >
> ------------------------------------------------------------------------------
> > > Want excitement?
> > > Manually upgrade your production database.
> > > When you want reliability, choose Perforce
> > > Perforce version control. Predictably reliable.
> > >
> > >
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> > > _______________________________________________
> > > Matplotlib-users mailing list
> > > Mat...@li...
> > > https://lists.sourceforge.net/lists/listinfo/matplotlib-users
> > >
> > >
> >
> > --
> > This email was Virus checked by Astaro Security Gateway.
> http://www.sophos.com
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > Want excitement?
> > Manually upgrade your production database.
> > When you want reliability, choose Perforce
> > Perforce version control. Predictably reliable.
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk_______________________________________________
> > Matplotlib-users mailing list
> > Mat...@li...
> > https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>
> --
> Jody Klymak
> http://web.uvic.ca/~jklymak/
>
>
>
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Want excitement?
> Manually upgrade your production database.
> When you want reliability, choose Perforce.
> Perforce version control. Predictably reliable.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
>
>

Hi Jody and Ben,
thanks for your answers. 
I tried to use pcolormesh instead of pcolor and the result is very good! For what concern with the memory system problem, I wasn't able to solve it. When I tried to use the bigger file, I got the same problem. Attached you will find the script that I'm using to make the plot. May be, I didn't understand very well how can I use the mmap function. 
Regards,
Raffaele.
-----Original Message-----
From: Jody Klymak [mailto:jk...@uv...]
Sent: Mon 9/8/2014 5:46 PM
To: Benjamin Root
Cc: Raffaele Quarta; Matplotlib Users
Subject: Re: [Matplotlib-users] Plotting large file (NetCDF)
 
It looks like you are calling `pcolor`. Can I suggest you try `pcolormesh`? ii
75 Mb is not a big file!
Cheers, Jody
On Sep 8, 2014, at 7:38 AM, Benjamin Root <ben...@ou...> wrote:
> (Keeping this on the mailing list so that others can benefit)
> 
> What might be happening is that you are keeping around too many numpy arrays in memory than you actually need. Take advantage of memmapping, which most netcdf tools provide by default. This keeps the data on disk rather than in RAM. Second, for very large images, I would suggest either pcolormesh() or just simply imshow() instead of pcolor() as they are more way more efficient than pcolor(). In addition, it sounds like you are dealing with re-sampled data ("at different zoom levels"). Does this mean that you are re-running contour on re-sampled data? I am not sure what the benefit of doing that is if one could just simply do the contour once at the highest resolution.
> 
> Without seeing any code, though, I can only provide generic suggestions.
> 
> Cheers!
> Ben Root
> 
> 
> On Mon, Sep 8, 2014 at 10:12 AM, Raffaele Quarta <raf...@li...> wrote:
> Hi Ben,
> 
> sorry for the few details that I gave to you. I'm trying to make a contour plot of a variable at different zoom levels by using high resolution data. The aim is to obtain .PNG output images. Actually, I'm working with big data (NetCDF file, dimension is about 75Mb). The current Matplotlib version on my UBUNTU 14.04 machine is the 1.3.1 one. My system has a RAM capacity of 8Gb.
> Actually, I'm dealing with memory system problems when I try to make a plot. I got the error message as follow:
> 
> --------------------------------------------
> cs = m.pcolor(xi,yi,np.squeeze(t))
> File "/usr/lib/pymodules/python2.7/mpl_toolkits/basemap/__init__.py", line 521, in with_transform
> return plotfunc(self,x,y,data,*args,**kwargs)
> File "/usr/lib/pymodules/python2.7/mpl_toolkits/basemap/__init__.py", line 3375, in pcolor
> x = ma.masked_values(np.where(x > 1.e20,1.e20,x), 1.e20)
> File "/usr/lib/python2.7/dist-packages/numpy/ma/core.py", line 2195, in masked_values
> condition = umath.less_equal(mabs(xnew - value), atol + rtol * mabs(value))
> MemoryError 
> --------------------------------------------
> 
> Otherwise, when I try to make a plot of smaller file (such as 5Mb), it works very well. I believe that it's not something of wrong in the script. It might be a memory system problem.
> I hope that my message is more clear now.
> 
> Thanks for the help.
> 
> Regards,
> 
> Raffaele
> 
> -----------------------------------------
> 
> Sent: Mon 9/8/2014 3:19 PM
> To: Raffaele Quarta
> Cc: Matplotlib Users
> Subject: Re: [Matplotlib-users] Plotting large file (NetCDF)
> 
> 
> 
> You will need to be more specific... much more specific. What kind of plot
> are you making? How big is your data? What version of matplotlib are you
> using? How much RAM do you have available compared to the amount of data
> (most slowdowns are actually due to swap-thrashing issues). Matplotlib can
> be used for large data, but there exists some speciality tools for the
> truly large datasets. The solution depends on the situation.
> 
> Ben Root
> 
> On Mon, Sep 8, 2014 at 7:45 AM, Raffaele Quarta <raf...@li...>
> wrote:
> 
> > Hi,
> >
> > I'm working with NetCDF format. When I try to make a plot of very large
> > file, I have to wait for a long time for plotting. How can I solve this?
> > Isn't there a solution for this problem?
> >
> > Raffaele
> >
> > --
> > This email was Virus checked by Astaro Security Gateway. http://www.sophos.com
> >
> >
> >
> > ------------------------------------------------------------------------------
> > Want excitement?
> > Manually upgrade your production database.
> > When you want reliability, choose Perforce
> > Perforce version control. Predictably reliable.
> >
> > http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Matplotlib-users mailing list
> > Mat...@li...
> > https://lists.sourceforge.net/lists/listinfo/matplotlib-users
> >
> >
> 
> --
> This email was Virus checked by Astaro Security Gateway. http://www.sophos.com
> 
> 
> 
> ------------------------------------------------------------------------------
> Want excitement?
> Manually upgrade your production database.
> When you want reliability, choose Perforce
> Perforce version control. Predictably reliable.
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk_______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
--
Jody Klymak 
http://web.uvic.ca/~jklymak/