Python memory problem

Question

i started programming on python but i have a memory problem (sorry for my bad english). I made a while loop on my algorithm, but on every cicle, the program consummes a lot of memory. I have 3Gb of RAM an AMD 64 x2 processor, and Windows 7 64 bits.

For every cicle, it consummes about 800 Mb of RAM, it's too much i think. Part of my code is here

from sympy import Symbol, diff, flatten 
import numpy as np
from numpy import linalg
from math import log, sqrt, cos, pi
import matplotlib.pyplot as plt
L = 7 #numero de variables
X = [Symbol('x%d' % i) for i in range(L*(L+1))] #Las variables simbolicas
XX = [X[i] for i in xrange(L)]
LAM = [] 
# Parametros
Pr = 10
Eps = 0
Ome = 5
LL = 0.5
b = 2
Gam = 0.2*2*(pi**2)
ran1 = xrange(L)
ran2 = xrange(L*L) 
ran3 = xrange(0,L*(L-1)+1,L)
ran4 = xrange(L,2*L,1)
dt = 0.01
TMAX = 60
def f(x,R,Tau):
 return [Pr*((1 + Eps*cos(Ome*Tau))*x[2] - LL*x[0] - (1 -LL*x[5])) , \
 Pr*((1 + Eps*cos(Ome*Tau))*x[3] - LL*x[1] - (1 - LL)*x[6]),\
 R*x[0] - x[2] - x[0]*x[4],R*x[1] - x[3] - x[1]*x[4],(x[0]*x[2] + x[1]*x[3])/2 - b*x[4],\
 (1/Gam)*(x[0] - x[5]),(1/Gam)*(x[1] - x[6])]
def Jacobian(f,x): #num son los numeros en el que se evalua la matriz jacobiana, x las variables y f la funcion
 return [[diff(f[i],x[n]) for i in ran1] for n in ran1]
def Y(x):
 return[[x[i+j] for j in ran3] for i in ran4]
 #Ahora la multiplicacion de Y traspuesto por Jacobian traspuesto
def JY(r,Tau):
 J = flatten((np.dot(np.array(Jacobian(f(XX,r,Tau),XX)),np.array(Y(X)))).T)
 return [J[i] for i in ran2]
def Func(x,r,Tau): #Expandemos las funciones en un solo arreglo
 FFF = []
 map(lambda g: FFF.append(g),f(XX,r,Tau))
 map(lambda g: FFF.append(g),JY(r,Tau))
 return map(lambda f: f.evalf(subs={X[j]:x[j] for j in xrange(L*(L+1))}),FFF)
def RKutta(xi,r):
 i = 1
 while i <= int(TMAX/dt):
 Tau = 0
 YY = xi
 k1 = np.array(Func(YY,r,Tau))*dt
 k2 = (np.array(Func(YY + k1/2,r,Tau/2)))*dt
 k3 = (np.array(Func(YY + k2/2,r,Tau/2)))*dt
 k4 = (np.array(Func(YY + k3,r,Tau)))*dt
 xi = YY + (k1/6) + (k2/3) + (k3/3) + (k4/6)
 Tau = Tau + dt
 i = i + 1
 return [xi[j] for j in xrange(len(xi))]
def lyap(xxi):
 u = [i for i in flatten(np.random.rand(1,L))]
 PhiT = (np.array([[float(xxi[i+j]) for j in ran3] for i in ran4])).T
 PU = np.dot(PhiT,u)
 summ = 0
 jj = 0
 while jj < len(PU):
 summ += (float(PU[jj]))**2
 jj = jj + 1
 lam = log(sqrt(summ))/TMAX
 return lam
R = 46.5
Rmax = 48.5
Rstep = 0.5
while R <= Rmax:
 xi = [5,5,5,5,5,5,5] #Condiciones Iniciales
 for i in ran2:
 xi.append(None)
 for i in ran4: 
 for j in ran3: 
 if (i+j+1)%(L+1) == 0:
 xi[i+j] = 1
 else:
 xi[i+j] = 0
 #Ahora el Runge Kutta para integrar todo el sistema
 #Y.append([r for r in xx])
 # savetxt('butterfly.txt', Y, fmt="%12.6G")
 #print Y
 XI = RKutta(xi,R)
 lamb = lyap(XI)
 LAM.append([R,lamb])
 print [R,lamb]
 R = R + Rstep
#print LAM
#x = [LAM[i][0] for i in xrange(len(LAM))]
#y = [LAM[i][1] for i in xrange(len(LAM))]
np.savetxt('lyap3.txt', LAM, fmt="%12.6G")
#plt.axis([10,30,-3,3]);
#plt.scatter(x,y)
#plt.show()

I don't know where the problem could be. Maybe at the Runge Kutta steps or an architecture problem. The memory don't seem to be cleaned at every step and i'm not storing anything, just a pair of numbers at the end of the code. I hope i expressed myself well.

#

OK, i edited this and posted the whole code, i hope someone can help :) . I changed a lot of things, but i still have the memory problem. Each cicle uses about 600 Mb of RAM.

#

Thanks in advance

• Laurence Gonsalves: Your code isn't syntactically correct Python because the indentation is messed up. What's supposed to be inside that while loop? Also, what are the values of L, R, LL?
• user395760: You indent only be 1 space per nesting? You are an evil, evil person.
• PerroNoob: L is just the number of variables, in this case, 7. I had another with 3 variables but it still consummes a lot of memory.
• Adam Rosenfield: You're doing some pretty heavy math there. Have you considered using a numerical library such as numpy?
• PerroNoob: Everything is inside the first while, is just how i copied it into this forum, the program runs well

Answer 1

It's a little tricky to follow the code without context, seeing as you've apparently used numpy in multiple places (both as np and without prefix), and evalf might be from sympy.. but we don't see your imports.

At a very vague guess, some of your list comprehensions build temporary lists that stick around longer than expected. You could perhaps convert those into generators. Another technique is using map() or similar as much as possible.

I also notice a bit of unpythonic index iteration where it's not needed. Func first builds a list called FFF, one item at a time (fairly expensive), then iterates through it by index for no real reason. Use [f(item) for item in seq] rather than [f(seq[i]) for i in xrange(len(seq))], or better yet, map(f, seq), and in this case, try not building the temporary list at all.

• PerroNoob: I think it's a great idea, but i don't know how i could use map for evalf since it is like FFF.evalf() and not evalf( )
• PerroNoob: I just did it, i had to use lambda functions. Thanks for your answer :)

Answer 2

What's L? Much of your code uses O(L^2) storage, so if L is large that will be it.

• PerroNoob: L is just 7, the number of variables

Answer 3

Seems like you are instantiating very large lists. Look if you can't replace some of the list comprehensions with iterators.

• PerroNoob: How's that? I replaced some 'for' cicles with 'whiles' if that is what you mean :O
• Paulo Scardine: @David Winchester: no, look at python itertools for sample implementations of iterators. See if your calculations can't be made "on the go" instead of "acumulating large lists and processing later".
• PerroNoob: I tried to make a lot of functions before the while cicle, so i didn't acummulate large lists, the memory usage was a lot less but the problem is that the calculations were too slow, so i put everything inside the big cicle again.
• PerroNoob: M... there's some solution? :/
• littlegreen: @Paulo: do you mean that instead of itertools he should define the lists in NumPy? Can you explain how?