Video Export memory leak? #57

ariesity commented 2019年08月15日 12:42:39 +02:00 (Migrated from github.com)

I have a processing sketch that generates races of 12 stickmen and each race is fixed for 120 seconds at 25fps, the video is recorded and a new race is generated and this goes on indefinitely until I manually stop it. After 1100 videos, the sketch will bomb out and say out of heap memory error occuring at videoexport.saveframe().

Is there a memory leak somewhere? I have closed off the video and null the videoExport instance after each run and it continues to error out with out of heap memory error after 1100 videos.

I have a processing sketch that generates races of 12 stickmen and each race is fixed for 120 seconds at 25fps, the video is recorded and a new race is generated and this goes on indefinitely until I manually stop it. After 1100 videos, the sketch will bomb out and say out of heap memory error occuring at videoexport.saveframe(). Is there a memory leak somewhere? I have closed off the video and null the videoExport instance after each run and it continues to error out with out of heap memory error after 1100 videos.

hamoid commented 2019年08月15日 14:36:56 +02:00 (Migrated from github.com)

That's impressive. 3.3 million frames! :)

Which OS?

You could check the memory situation by using https://visualvm.github.io/ ...

You could also check with a system monitor if the ffmpeg processes that are launched end or do they stay running forever.

Once it crashes, can you immediately start it again? Or do you need to restart the computer?

Just curious: how many videos do you plan on making?

That's impressive. 3.3 million frames! :) Which OS? You could check the memory situation by using https://visualvm.github.io/ ... You could also check with a system monitor if the ffmpeg processes that are launched end or do they stay running forever. Once it crashes, can you immediately start it again? Or do you need to restart the computer? Just curious: how many videos do you plan on making?

ariesity commented 2019年08月18日 03:30:30 +02:00 (Migrated from github.com)

Firstly to answer your questions.

1. I am running it headless under Ubuntu 18.04 server.

2. I have tried to use visualvm but can't really figure out the problem.

3. The ffmpeg processes terminate properly.

4. Once it crashes, I can just stop the processing sketch and restart again, no need to restart the computer.

5. I am intending to make as many videos as a 6 TB harddisk can contain, probably about 1 million videos.

I am not too good at Java programming, I have some questions here.

pixelsByte = new byte[img.pixelWidth * img.pixelHeight * 3];

under the dispose method, it simply calls endmovie. However, the references to pixelsByte is not dereferenced. If I create a new VideoExport object in draw cycle and calls dispose at the endof the draw cycle, will the GC recollect the entire memory used by the VideoExport project including the only one big allocation for the image?

Then since the VideoExport object is attached to the PApplet object, does it mean all the new VideoExport objects will never be dereferenced unless the processing sketch stops, meaning that the processing sketch could be attached to 1000s of these VideoExport objects.

Anyway, I am trying not to create a new object in each draw cycle but to reuse the existing object, hope this works.

Update: Seems to be a memory leak issue with PImage.
https://forum.processing.org/one/topic/pimage-memory-leak-example.html

Firstly to answer your questions. 1) I am running it headless under Ubuntu 18.04 server. 2) I have tried to use visualvm but can't really figure out the problem. 3) The ffmpeg processes terminate properly. 4) Once it crashes, I can just stop the processing sketch and restart again, no need to restart the computer. 5) I am intending to make as many videos as a 6 TB harddisk can contain, probably about 1 million videos. I am not too good at Java programming, I have some questions here. pixelsByte = new byte[img.pixelWidth * img.pixelHeight * 3]; under the dispose method, it simply calls endmovie. However, the references to pixelsByte is not dereferenced. If I create a new VideoExport object in draw cycle and calls dispose at the endof the draw cycle, will the GC recollect the entire memory used by the VideoExport project including the only one big allocation for the image? Then since the VideoExport object is attached to the PApplet object, does it mean all the new VideoExport objects will never be dereferenced unless the processing sketch stops, meaning that the processing sketch could be attached to 1000s of these VideoExport objects. Anyway, I am trying not to create a new object in each draw cycle but to reuse the existing object, hope this works. Update: Seems to be a memory leak issue with PImage. https://forum.processing.org/one/topic/pimage-memory-leak-example.html

hamoid commented 2019年08月18日 11:58:37 +02:00 (Migrated from github.com)

Hi, unfortunately I'm too busy the next two weeks for looking into the program logic. I can check after the end of the month.

Did you see this example for creating multiple movies? https://github.com/hamoid/video_export_processing/blob/master/examples/multipleMovies/multipleMovies.pde
It's not necessary to call new VideoExport... Does that help?

ps. the PImage memory leak you linked is very old. The dates are wrong in that post. Maybe 10 years old? I doubt it affects Processing 3.

Hi, unfortunately I'm too busy the next two weeks for looking into the program logic. I can check after the end of the month. Did you see this example for creating multiple movies? https://github.com/hamoid/video_export_processing/blob/master/examples/multipleMovies/multipleMovies.pde It's not necessary to call new VideoExport... Does that help? ps. the PImage memory leak you linked is very old. The dates are wrong in that post. Maybe 10 years old? I doubt it affects Processing 3.

ariesity commented 2019年08月19日 13:09:32 +02:00 (Migrated from github.com)

The processing sketch has at this time created 1297 videos and the heap is still healthy. I am currently using only one videoExport object throughout the entire processing sketch.

So I guess the slight enhancement required on the videoExport class is to dereference the pixelByte object at the dispose method so that the gc can at least recollect the memory space. I have tried changing the source, but I am unable to compile it into a library :P

The processing sketch has at this time created 1297 videos and the heap is still healthy. I am currently using only one videoExport object throughout the entire processing sketch. So I guess the slight enhancement required on the videoExport class is to dereference the pixelByte object at the dispose method so that the gc can at least recollect the memory space. I have tried changing the source, but I am unable to compile it into a library :P

hamoid commented 2019年08月24日 22:30:56 +02:00 (Migrated from github.com)

Feel free to send a pull request, or to post here the line that I should change :)

Feel free to send a pull request, or to post here the line that I should change :)