Memory error with numpy. array

Question

I get a memory error when using numpy.arange with large numbers. My code is as follows:

import numpy as np
list = np.arange(0, 10**15, 10**3)
profit_list = []
for diff in list:
 x = do_some_calculation
 profit_list.append(x)

What can be a replacement so I can avoid getting the memory error?

Answer 1

If you replace list1 with a generator, that is, you do

for diff in range(10**15, 10**3):
 x = do_some_calculation
 profit_list.append(x)

then that will no longer cause MemoryErrors as you no longer initiate the full list. In this world, though, profit_list will probably by causing issues instead, as you are trying to add 10^12 items to that. Again, you can probably get around that by not storing the values explicitly, but rather yield them as you need them, using generators.

1: Side note: Don't use list as a variable name as it shadows a built-in.

• user123: I was also trying to plot(difficulty_list,profit_list). Would this be done the same way?
• fuglede: I don't believe there is any way out of that, as matplotlib will act on numpy arrays; why would you ever want to plot 10^12 points in the first place?
• DSM: You should think about the size of the numbers you're working with, user123. Setting aside the memory problems, for me, it takes about 20 ms to make an ndarray with 10^7 elements. Let's assume that scales linearly (in the real world it'll slow down even more). That means it'll take more than half an hour just to build it. And that's happening at C speed. It takes me about 10 ns just to execute pass, which means that whatever your loop does, it'll take orders of magnitude longer than 10^12 * 10 * 10^-9ns ~ 3 hours. At best you should work with millions of numbers, not trillions.