-
Notifications
You must be signed in to change notification settings - Fork 91
fix: bound the streaming mel buffer to a sliding window (fixes #63, the streaming RSS leak) - #69
Conversation
, the streaming RSS leak) Root cause: parakeet_capi_stream_feed's streaming session keeps every mel frame from stream_begin onward in mel_buf and rebuilds that whole, ever-growing array on every feed call, even though the decoder (feed_available's window()) never reads a frame older than mel_buffer_idx - pre_encode_cache_size() frames back — so those old frames are provably dead the moment the decoder consumes past them. Because each rebuild requests a uniquely, ever-larger allocation for the life of the stream, the allocator can never reuse an earlier call's freed (now permanently too-small) block, so process RSS grows with the cumulative history of buffer sizes rather than the small, bounded set of frames actually still needed — which is why neither stream_free nor a fresh stream_begin reclaims it. The fix: track the absolute frame index of mel_buf's first column (mel_buf_origin) and, on every append, drop the prefix strictly before mel_buffer_idx - pre_encode_cache_size() — the exact bound window() already relies on — so mel_buf stays bounded to O(pre_encode_cache_size + chunk_size) instead of O(stream length). No API change, no behavioural change: window()'s absolute-index reads are translated into mel_buf's origin-relative local columns, and the surviving frames are carried forward byte-for-byte. Measurement (CPU + Accelerate build, realtime_eou_120m-v1 q8, 120 s of silence fed in 20 ms chunks, RSS sampled at 30/60/90/120 s of audio fed): growth over the last 60 s drops from 178 MB to 0.5 MB; the same run's wall time drops from ~14 minutes to ~13 seconds, since the unbounded rebuild was O(T) per call, O(T2) total. A LibriSpeech utterance's transcript is byte-identical before and after.
@localai-org-maint-bot
localai-org-maint-bot
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed f075f31. The absolute-to-local window indexing looks consistent with the existing chunk schedule. A separate algorithm-level differential check covered 500 variable-feed schedules and 419,147 windows, including final tails, without a mismatch. This was a model of the indexing, not a native C++ or inference run.
Before merge, please add a regression that exercises the actual buffer code over repeated trims, variable feed sizes and finalization, checking retained-frame bounds and equality with the untrimmed reference. The existing C API test checks a short transcript/events, but does not assert bounded retained history; this PR currently changes only the implementation file.
Please also narrow the root-cause claim: bounding live mel history is a valid improvement, but ever-larger allocations do not prove that an allocator can never reuse/coalesce freed blocks. The bound also includes the incoming feed size. The macOS CPU measurements do not yet establish that the CUDA/Jetson retained-RSS behavior in #63 is resolved, and this patch does not address its separate CUDA teardown abort. Keep that distinction in the source explanation and issue disposition until the original reproducer is rerun.
@mudler: promising memory fix, but not a good-to-merge sign-off yet. No CI checks are attached and I could not run native inference in this environment.
Fixes #63.
Root cause (two sentences):
parakeet_capi_stream_feed's streaming session keeps every mel frame fromstream_beginonward inmel_bufand rebuilds that whole, ever-growing array on every feed call, even though the decoder (feed_available'swindow()) never reads a frame older thanmel_buffer_idx - pre_encode_cache_size()frames back — so those old frames are provably dead the moment the decoder consumes past them. Because each rebuild requests a uniquely, ever-larger allocation for the life of the stream, the allocator can never reuse an earlier call's freed (now permanently too-small) block, so process RSS grows with the cumulative history of buffer sizes rather than the small, bounded set of frames actually still needed — explaining why neitherstream_freenor a freshstream_beginreclaims it.The fix: track the absolute frame index of
mel_buf's first column (mel_buf_origin) and, on every append, drop the prefix strictly beforemel_buffer_idx - pre_encode_cache_size()— the exact boundwindow()already relies on — somel_bufstays bounded toO(pre_encode_cache_size + chunk_size)instead ofO(stream length). No API change, no behavioural change:window()'s absolute-index reads are translated intomel_buf's new origin-relative local columns, and the surviving frames are carried forward byte-for-byte. This is the same bounded-sliding-window discipline the per-layer conv/attention caches inStreamingEncoderalready follow; the mel buffer was the one accumulator that did not.Measurement (CPU + Accelerate build,
realtime_eou_120m-v1q8, 120 s of silence fed in 20 ms chunks, RSS sampled at 30/60/90/120 s of audio fed): growth over the last 60 s drops from 178 MB to 0.5 MB; the same run's wall time drops from ~14 minutes to ~13 seconds, since the unbounded rebuild was O(T) per call, O(T2) total. A LibriSpeech utterance's transcript is byte-identical before and after (sha256d7a620231a849b5a531ab103a8aaac9ade7d088dc05af7a77fc0633d0d797b27).Built and checked on macOS arm64 (CPU,
-DPARAKEET_SHARED=ON -DPARAKEET_GGML_METAL=OFF), zero warnings.