Photo scripting: crash @ 2GB - Doom9's Forum

Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.


Reload this Page Photo scripting: crash @ 2GB

(追記) (追記ここまで)
Old 21st October 2005, 17:57 #1 | Link
Registered User

Join Date: Oct 2005
Posts: 11
Photo scripting: crash @ 2GB

Hi there,

since weeks I am struggling with the following, thank you already for reading this post and any advise you may have.

I want to create a video from a set of, say, 200 digicam photos (jpeg), each has a resolution of 2048x1536 pixels and about 500K. In my script, I create a clip c per photo using ImageSource() and add some zoom in/out motion using Animate with BilinearResize. I then append the clips, either using each time v=v+c (no fade) or v=Dissolve(v, c) (classic fade) or v=MaskTransition(v, c) (masked fade). Of course, I can give you all the details if you wish.

The resulting script plays correctly in MediaPlayer, however not in real-time, so it must be syntactically and functionally correct. But when I feed it into any encoder (Dr. DivX, TMPGEnc, QuEnc, NuEnc, HCEnc...) I notice (taskmgr) that during the encoding process more and more memory is allocated (some 20 - 30 MB per photo) and when we reach 2GB ... BOOOOM... the encoder application just plainly disappears without a trace.

I tried various versions of Avisynth 2.5.x without success

My questions:
- Is there anything wrong with the script, am I doing something not allowed?
- How would YOUR script look like?
- Is there a way to reduce memory usage by changing the script logic?
- Is there a way to explicitely free memory?
- Is there a known memory leak in AviSynth?
- Is there a known limitation at 2GB?

Again, thanks for reading and your advise.

gamez.
gamez is offline Reply With Quote
Old 21st October 2005, 18:05 #2 | Link
Registered User

Join Date: Feb 2004
Location: NTSC R1
Posts: 2,046
__________________
unskinnyboy is offline Reply With Quote
Old 21st October 2005, 18:13 #3 | Link
Registered User

Join Date: Oct 2005
Posts: 11
I have added 2 attachments, hope it works.
Please remove ".txt" and put your own photos...
The masks are borrowed from the program DVDslideshowGUI.

See you,
gamez
Attached Files
File Type: txt builtin.avs.txt (4.5 KB, 237 views)
File Type: txt myvideo.avs.txt (35.2 KB, 247 views)
gamez is offline Reply With Quote
Old 21st October 2005, 18:50 #4 | Link
Registered User

Join Date: Feb 2004
Location: NTSC R1
Posts: 2,046
Attachments are still awaiting approval, but I guess what you probably need here is to use the SetMemoryMax(int) function in your script and reduce the memory allocated to AVISynth.

where, int = value in MB that defines the amount of memory that will be allocated to a script when it is loaded.
__________________
unskinnyboy is offline Reply With Quote
Old 21st October 2005, 18:51 #5 | Link
Confused

Join Date: Apr 2003
Location: Euroland
Posts: 2,820
Quote:
Originally Posted by gamez
Is there a known limitation at 2GB?
Yup, afaik windoze cant allocate more than 2GB per process!


Bye
Soulhunter is offline Reply With Quote
Old 22nd October 2005, 00:49 #6 | Link
Registered User

Join Date: Sep 2005
Location: 100011110010001000001 10000011111111000001
Posts: 221
Quote:
Originally Posted by gamez
- Is there a way to reduce memory usage by changing the script logic?
Since your script is a linear series of commands, you can split it in as many pieces (separate .avs scripts) are needed to get each step's required memory below 2GB. Each step must return a video that the next step will use as input. That way you only have to include an extra AVISource command for each step after the first.

If you want to automate the execution of the chain, you may try DubMan.

Quote:
- Is there a way to explicitely free memory?
I have been trying it for some time but did not find a way to do this by scripting; I guess there is no way.
This is a feature request for next versions of Avisynth
__________________
AVSLib, a free extension library for Avisynth. Current version: 1.1.0 (beta), 14/05/2007.
[ Home page | Download page ]

Last edited by gzarkadas; 22nd October 2005 at 00:51.
gzarkadas is offline Reply With Quote
Old 24th October 2005, 15:19 #7 | Link
C64

Join Date: Apr 2002
Location: Austria
Posts: 830
Maybe you consider using Zoom?

www.avisynth.org\warpenterprises\#zoom
WarpEnterprises is offline Reply With Quote
Old 24th October 2005, 21:54 #8 | Link
Registered User

Join Date: Oct 2005
Posts: 11
@WarpEnterprises

Thanks a lot for your hint, I installed your plugin right away.

Sorry, somehow I can't manage to get the Zoom parms right; could you be so kind to help me translate the syntax, it will be piece of cake for you to understand how I use Animate today in my ZoomIn/ZoomOut functions (see attached scripts).

Thank you!!!


@gzarkadas

Will try your advice as well, thanks!


gamez
gamez is offline Reply With Quote
Old 25th October 2005, 07:05 #9 | Link
C64

Join Date: Apr 2002
Location: Austria
Posts: 830
I will try to translate...

Meanwhile still another option which may suit exactly your needs:
vcmohan made some nice filters which are much easier to use (for many effects and transitions):
www.avisynth.org\warpenterprises\#EffectsMany
www.avisynth.org\warpenterprises\#HollywoodSQ
www.avisynth.org\warpenterprises\#TransAll
WarpEnterprises is offline Reply With Quote
Old 25th October 2005, 11:42 #10 | Link
Registered User

Join Date: Oct 2005
Posts: 11
Those effects are indeed breathtaking, really stunning.
I surely will study them as soon as I have sorted out my memory consumption problems and as it seems today they are more related to the zooming of one picture than to the fade-over transitions between the pictures.
gamez is offline Reply With Quote
Old 26th October 2005, 20:54 #11 | Link
Registered User

Join Date: Oct 2005
Posts: 11
@WarpEnterprises

Hello again,

I've spend some time to find the required maths that enable my scripts to use your zoom plugin instead of Animate/BilinearResize; for my function ZoomIn it goes like this:

Quote:
function ZoomIn(clip v, int x, int y, int left, int top, int right, int bottom)
{
# compute zoom factor for x
w1 = float(v.width)
w2 = float(right-left)
fx = string(x/w1) + "+n*" + string(x*(w1-w2)/(framecount(v)*w1*w2))

# compute zoom factor for y
h1 = float(v.height)
h2 = float(bottom-top)
fy = string(y/h1) + "+n*" + string(y*(h1-h2)/(framecount(v)*h1*h2))

# compute source center for x
w2 = float(right+left)
sx = string(w1/2) + (w2 > w1 ?
\ "+n*" + string((w2-w1)/(2*framecount(v))) :
\ "-n*" + string((w1-w2)/(2*framecount(v))))

# compute source center for y
h2 = float(bottom+top)
sy = string(h1/2) + (h2 > h1 ?
\ "+n*" + string((h2-h1)/(2*framecount(v))) :
\ "-n*" + string((h1-h2)/(2*framecount(v))))

return v.ConvertToRGB32.zoom(
\ factorx=fx, factory=fy, srcx=sx, srcy=sy,
\ width=x, height=y).ConvertToYV12
}
For ZoomOut, it's just the other way round, no big deal. Unfortunately, I have to tell you that I've found the following (sorry, I don't want to criticize, it's just the things I have noticed...)

- when the zoom focus is different from the picture's center, the resulting zoom motion looks quite strange in my eyes, it does not zoom linearily to the target area but in a kind of curved or contorted motion - this is probably due to the way how the x/y-factors interact with the source coordinates
- the version using zoom is three times slower than before
- the quality of the resulting resized photos using BilinearResize is significantly superior than the ones resized with zoom - as you mention yourself in the accompanying textfile
- although the memory allocation is indeed different than with Animate/BilinearResize, my "main" script with the 200 photos still does not succeed to complete the encoding process, it even aborts earlier than before

I am very sorry that I was not able to use your function to my advantage, I'm sure it has nice features useful under other circumstances.

If you have any other idea, please let me know!

gamez
gamez is offline Reply With Quote
Old 26th October 2005, 21:17 #12 | Link
C64

Join Date: Apr 2002
Location: Austria
Posts: 830
To your memory problem: did you try the script with VirtualDub? It has better error messaging and you can check when the memory climbs (e.g. if you step frame-by-frame or only in preview mode or if the memory gets freed again if you do nothing).
WarpEnterprises is offline Reply With Quote
Old 26th October 2005, 22:49 #13 | Link
easily bamboozled user

Join Date: Sep 2002
Location: Atlanta
Posts: 373
You know, no PC application was ever supposed to hit the 2GB address space limit while Win32 was still around. Yet you've just managed to do it. Props.
Prettz is offline Reply With Quote
Old 27th October 2005, 07:44 #14 | Link
C64

Join Date: Apr 2002
Location: Austria
Posts: 830
But there is no obvious reason why this amount of memory is actually needed.
It looks like there is a missing memory deallocation or the like.
WarpEnterprises is offline Reply With Quote
Old 3rd November 2005, 00:24 #15 | Link
Registered User

Join Date: Sep 2005
Location: 100011110010001000001 10000011111111000001
Posts: 221
@ gamez,

I have made some further investigation on this and I believe I have found the "suspect" (which I must admit I was suspecting some time ago, when I was crashing-at-approx-2GB regularly trying to move 12 polygons on spiral paths in a single script until I gave up and made a script chain to accomplish it; see here).

The "suspect" is Overlay(); just one call inside the transition function but this is enough (since it is repeated about 200 times) to blow the script out of memory.

Incidentally, this also means there is currently no way to encode and store the script as is in a full video file in one single step. It must be broken in pieces of say 25-40 photos at the most. I used 25, but this is because the test-case photos I have used are actually 2,25 MB each; with yours (0,5MB) I believe you could go up to 40 per piece.

If you are still working on this, I can post an almost-ready to run set of scripts, which I have used as test-case (I will need a couple of days though to document it properly).
__________________
AVSLib, a free extension library for Avisynth. Current version: 1.1.0 (beta), 14/05/2007.
[ Home page | Download page ]

Last edited by gzarkadas; 3rd November 2005 at 00:26.
gzarkadas is offline Reply With Quote
Old 3rd November 2005, 03:51 #16 | Link
Registered User

Join Date: Apr 2005
Posts: 1,339
I had a "similar" (sorta) problem on some workstations at work that were trying to render some complex scenes. Even though they had plenty (4GB) of RAM, they kept running out of memory. There's a Boot.ini function /3GB that can help if you have SP2 installed (yuck). I was able to get the machines to render by installing an app called FSAutoStart, which will shut down all the un-needed Windows Services and applications, reclaim the RAM, run the APP, then restart all the programs. It works quite well, and it's easy to install and uninstall if need be. I use it at home as well to give my machine that "freshly booted feeling".

http://www.kensalter.com/forum/forum.asp?FORUM_ID=14

There's also a free command line app called Memdefrag which you can run at timed intervals to recover the RAM. http://web.telia.com/~u32102551/

Finally, you can try O&O Software Clevercache which will optimize your memory dynamically and (supposedly) speed up your system. They have a 30 day eval.

Last edited by Pookie; 3rd November 2005 at 03:56.
Pookie is offline Reply With Quote
Old 4th November 2005, 01:22 #17 | Link
Registered User

Join Date: Sep 2005
Location: 100011110010001000001 10000011111111000001
Posts: 221
My test case

Ok, here is my test case.
The attached scripts implement the test case; the initial "builtin.avs" attached by gamez is also needed to run the normal chain (see below).
At the end are the results that IMHO point to a leak in the implementation of the Overlay() filter.
By the way the scripts are also a nice example of using arrays:
  • to make your life better (and the code listings much shorter) in general,
  • to create a for...next loop in Avisynth.
What the script-set does in brief is to code all of the images and intermediate bitmaps as array data and them to execute a for...next loop to combine them to the final result.
The breakdown to pieces of 25 images per step (8 steps in total) is hard-coded inside the mv.avs script's body. The global variable chain_part informs the script about what part of the chain to execute. It has to be set for each step from the command line (see section 1.2 below).

1. Prerequisites

1.1. Required software:
  1. Avisynth, version 2.5.6
  2. AVSLib, version 1.0.0.
  3. The 1.0.1 patch version of AVSlib (from the above location). The patch must be installed after the installation of AVSLib 1.0.0. The patch offers (for Avisynth 2.5.6 and later only) arrays of practically unlimited length; it is also needed for the scripts to execute.
  4. Configuration of your system so that at least 2.2 GB of total (ram+swap) memory are available (the "Limit" field at Task Manager's dialog box should report > 2200000)
  5. VirtualDub, version 1.5.10 or later
  6. XviD 1.0.3 Codec
The later two are needed for exactly reproducing my setup and comparing the results; I assume any selected video compression method will work for encoding the results.

1.2. Script edits / creation of environment
  1. The path on the first line of the mv.avs (main) script must change to the location where you have installed the AVSLib loader (avslib-h.avsi).
  2. From a command prompt you must cd to the scripts' directory and issue the following two commands:
    Code:
    echo return 0 > mv_chain.ini
    echo return 0 > mv_built.ini
    This will setup the main script for normal operation at step 0 of the script chain.
  3. For later steps (1,2,...,7) you must issue a
    Code:
    	echo return X > mv_chain.ini
    command (X = 1,2,...,7) before invoking the main script again.
  4. For each successive step you must save the output video as: mvout0.avi, mvout1.avi, mvout2.avi, ... and so on up to the last step.
  5. If you are gamez:

    i. Edit the line (no #6 inside mv.avs):
    Code:
    global __debug__ = true
    to
    Code:
    global __debug__ = false
    ii. Check the "bmps" and "bmp_ids" arrays in mvinc.avsi; they may need some adjustment because I think I have missed three items during copying and pasting from your sources. This is the only thing that may make the output to deviate from the intensions of your initial script.
  6. If you are not gamez:

    i. Create a folder and put inside:
    a. 402 JPEG photos with dimensions at least those reported at the top message of this forum thread, named {anything}001.jpg, ..., {anything}402.jpg
    b. one JPEG photo with same dimensions and name "z.jpg" (for the transition effect).
  7. Edit lines #19 and #75 in mvinc.avsi to point at the above folder (don't forget to include {anything} in a1., above, at the end of the path).
  8. Edit the "chain_last" function in mv.avs (lines #48 - 58) to insert an audio of your choice (this will only be called at step 7 of the script chain).

2. Execution

i. As said above, before each step, issue this command (at the command prompt, with active dir the script's dir):
echo return X > mv_chain.ini
Substitute 0,1,2,3,4,5,6,7 for X. Save the output video as mvoutX.avi.

ii. Executing the normal (with effects and the call to Overlay()) chain is the default, if all steps above have been made. If after playing with debug chains you want to revert to normal execution, issue this command (at the command prompt):
echo return 0 > mv_built.ini

iii.To execute a debug chain, issue one of the three commands below:
echo return 1 > mv_built.ini (uses Layer instead of Overlay)
echo return 2 > mv_built.ini (in addition to the case above, does not use Animate)
echo return 3 > mv_built.ini (in addition to the cases above, does not use the transition mask effect)

iii.For inspecting memory consumption, record the total memory before and after opening and encoding (saving as AVI) the script in VirtualDub.

3. Results

With my setup:
1. images of ~2.2 MB each
2. VirtualDub 1.5.10 as renderer
3. XviD 1.0.3 Codec for video compression, with settings:
a. profile / level: AS @ L5
b. target quantizer: 1.00
c. quantization type: MPEG
d. quarter pixel: true
e. global motion compensation: true
f. Motion search precision: 6
g. Use chroma motion: true
h. Trellis quantization: true
4. PC: Pentium-M 1.7GHz / 1GB RAM / ATI Mob. Radeon 9600 AGP, 350MHz, 64MB / XP-Pro, SP2 / avisynth.dll 2.5.6.0

The following results are obtained:

1. Executing the normal chain, required ~ 1.34 GB memory to fully encode the first part (step 0).

2. Executing the first debug chain (Overlay replaced by Layer in the transition function), required ~ 0.52 GB to fully encode the first part (step 0)

3. Executing the other debug chains (without Overlay + Animate / without Overlay + Animate + Transition) required similar amount of memory with the first debug chain.

4. This means ~ 0.82 extra GB are due to Overlay() alone (for about 4000 frames, ie ~ 200 KB per frame).

5. The memory consumption between opening the script and the appearance of the first frame at VirtualDub's input window was about the same for all cases (approx. 0.3 GB). The build-up of the great difference in consumption between the normal and the debug chains is performed during the serving of frames to the encoder.

6. The extra memory required to encode the normal chain is 160% of the debug chains; this is too much and thus it is probably a sign of a memory leak somewhere inside the Overlay() code.
Attached Files
File Type: txt mv.avs.txt (2.1 KB, 171 views)
File Type: txt mvinc.avsi.txt (5.5 KB, 141 views)
File Type: txt builtin_dbg.avs.txt (4.6 KB, 138 views)
File Type: txt builtin_dbg2.avs.txt (4.4 KB, 128 views)
File Type: txt builtin_dbg3.avs.txt (4.2 KB, 136 views)
__________________
AVSLib, a free extension library for Avisynth. Current version: 1.1.0 (beta), 14/05/2007.
[ Home page | Download page ]
gzarkadas is offline Reply With Quote
Old 4th November 2005, 11:11 #18 | Link
Avisynth Developer

Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,167
Problem is an interaction between Animate and Overlay. Animate env->Invoke's a chain of filters per different parameter frame. It maintains a small cache of these chains. Each instance of Overlay malloc's 3 YV24 full size work frames. The way the scripts are written causes huge numbers of instances of whole filter chains to concurrently exist each with several Overlay instances. Carefull restructuring of the scripts to avoid recursive calls to Animate may help.
IanB is offline Reply With Quote
Old 4th November 2005, 15:24 #19 | Link
Registered User

Join Date: Sep 2005
Location: 100011110010001000001 10000011111111000001
Posts: 221
Quote:
Originally Posted by IanB
Problem is an interaction between Animate and Overlay. Animate env->Invoke's a chain of filters per different parameter frame. It maintains a small cache of these chains. Each instance of Overlay malloc's 3 YV24 full size work frames. The way the scripts are written causes huge numbers of instances of whole filter chains to concurrently exist each with several Overlay instances. Carefull restructuring of the scripts to avoid recursive calls to Animate may help.
Ok, we 'll live with it. After all we can divide and conquer and there are also the VirtualDub's job files to assemble all pieces together. Thus, we are not doomed, we are happy doom-ers instead .

However:
1. the Animate + Overlay combination is a damn useful one.
2. the initial script of gamez which also suffered from the same memory problems does not make recursive calls. It is a linear flow of function calls which result in the following linear combination of calls:
Animate (BilinearResize)
Overlay
Animate (BilinearResize)
Overlay
...
etc.
Now, if what you are saying is that what appears as a linear script flow in the source is in fact a huge recursive call list during execution because each filter pull frames from the previous and thus all up to the top, then simply the task under question (open a bunch of images and make an animated slideshow with a simple transition) cannot be accomplished by a linear-flow script for bunch size > 30 - 50 (depending on their size) in Win32.

I will then give up any animation attempts with Animate and stick to ScriptClip which (at least this is my conclusion up to now) is a much less memory-demanding filter even when combined with Overlay.

I must admit though, that writing scripts as a huge F(frame) function in order to be memory-efficient and manage to accomplish conseptually simple-to-medium level tasks such as a ~50 pictures slideshow with a bit of zooming is not a pleasant situation from the user's perspective. Higher-level primitives are always more easy to comprehend and use, especially from non-developers.

Is there a hope for a better (regarding memory use) Animate / Overlay support in future versions of Avisynth ?
__________________
AVSLib, a free extension library for Avisynth. Current version: 1.1.0 (beta), 14/05/2007.
[ Home page | Download page ]
gzarkadas is offline Reply With Quote
Old 9th December 2005, 06:52 #20 | Link
Registered User

Join Date: Mar 2005
Posts: 65
Interestingly enough, I'm also having a problem with a memory leak, but I'm not using animate or overlay! It is related to my earlier post about getting around avisynth's simultaneous avi issue. To recap, I'm using a script to load all the files into virtualdub and frameserving out. I also generate trims in avs and use that to access the original clips from out of the big concatenated "file" coming from VD.

From there, I'm de-interlacing with or without slow motion, adding titles, and basically dissolving everything together.

My problem is that there seems to be a memory leak in that every time it switches to a new clip a chunk of ram gets taken out and never gets released. It wouldn't be a problem if it was smaller, but it totals a half-hour. I can split it in half, at which point it goes from a start of about 200megs (seems reasonable with all that's going on) up to around 750megs! If I try to do the whole thing, virtualdub will choke during compression. This also happens when just playing a file. If you seek to the middle, it starts leaking from there, etc.

If anyone wants to check out my scripts, look here:
http://www.yoyoing.com/shawn/videofiles/avisynth/

It is basically _edit.avs that is the main file with the edits and functions. _trims.avs contains the trim points. _append.syl is the script for virtualdub.

I need to experiment more, but it seems weird that this is happening considering I'm not doing anything particularly crazy for the actual edting itself...

Shawn
ShawnFumo is offline Reply With Quote
Reply
Page 1 of 2 1 2

Thread Tools Search this Thread
Show Printable Version Show Printable Version
Email this Page Email this Page
Search this Thread:

Advanced Search
Display Modes
Linear Mode Linear Mode

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 19:11.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2025, vBulletin Solutions Inc.

AltStyle によって変換されたページ (->オリジナル) /