Java: Garbage Collection

Question

I'm having an out of memory error. I have a large range of inputs (2^40), that is too large to hold at once. Each input is a String[].

Instead, I thought I'd run my test program on each input, write the results to a file, then discard the input. The length of the longest input is 42, so that isn't an error causing the overflow. I don't think I understand garbage collection.

PowerSet is like a lazy list - the result isn't calculated until .next() is called. Each result returns one subset of baseSet. baseSet is a String[] with length 40. runTests does some analysis on the input and writes it to a file.

PowerSet argSetSet = powerset(baseSet); 
while (argSetSet.hasNext()) {
 runTests(argSetSet.next()); //saves output to file
}

This causes an out of memory error. But I'm not saving the result of argSetSet.next() anywhere, so why should this occur? I don't need to store any data from next() or runTests(). How can I make sure it's all being garbage collected?

If I comment out runTests(), it runs fine.

• McDowell: What is the exact error message? Is it an OutOfMemoryError, which suggests too many objects? Or a StackOverflowError, which suggests a recursion bug?

Answer 1

There isn't enough code to understand what is going on, primarily PowerSet, but PowerSet has to calculate the String array to return the next method. It could be that it is holding on to that object.

The memory issue is either in the runTests method or the PowerSet class. It isn't in the code you posted.

Answer 2

Attach a profiler like jvisualvm and investigate where your memory goes. You might be surprised :)

Answer 3

What's in baseSet? I'm guessing that's using up a ton of memory. This is probably exacerbated when PowerSet uses baseSet internally.

Answer 4

The fact that you aren't storing the results of .next() anywhere isn't relevant, what is relevant is what .next() is actually doing.

Are you setting the heap size to a non-default size? What settings are you using to start the JVM? The default heap size of the JVM is only 64M, so one trillion entries certainly cannot fit into that space.

Answer 5

[Clippy-like icon] It appears you are calculating the powerset of a very large set. Do you want to increase your heap size?

What concerns me is that you're saying that this is a lazy list, which means that the entire powerset isn't actually in memory, but only a piece of it is in memory when you call .next(). HOWEVER, depending on what .next() actually returns (the size of the array) then the default heap size is most likely not enough.

You can increase the size of the heap by specifying -Xmx1024m (setting the maximum value size of the heap to 1gb). Obviously you can tweak that size, but this will allow you to test to see if it scales. This is not an end-all solution, but it should at least give you some runway.

• Nick Heiner: the max size of the array that next() returns is 40.
• Malaxeur: Okay yeah, that's definitely off. I'd recommend downloading a profiler like yourkit (yourkit.com) and try out their Java Profiler. You can see where the memory is being held. Judging from your comments, you're holding on to the arrays or the values of the strings in memory within runTests. Not sure if you want to expose that method to us, but look for places where you put it in a map/list/set and don't clear a map or you use static collections.