-
Notifications
You must be signed in to change notification settings - Fork 272
fix(server): cap telemetry sink append backlog to prevent unbounded heap growth - #1374
Closed
Schrotti77 wants to merge 1 commit into
Closed
fix(server): cap telemetry sink append backlog to prevent unbounded heap growth #1374Schrotti77 wants to merge 1 commit into
Schrotti77 wants to merge 1 commit into
Conversation
Hermes (Marian's AI Agent)
...eap growth RotatingAppender serialized writes through an unbounded promise chain. Under SimpleSpanProcessor, a large workspace seed enqueues tens of thousands of spans faster than serial disk appends drain them; each queued link pinned its payload plus closures until all predecessors settled (multi-GB heap growth on ~34k-file projects). Queued appends are now capped at 1000 per sink path; beyond the cap, appends are rejected instead of queued — mirroring the SDK's own maxQueueSize drop policy. These sinks are local diagnostics: dropping records under sustained overload beats unbounded memory growth.
Schrotti77
requested a deployment
to
inkeep-oss-sync
August 24, 2026 14:20 — with
GitHub Actions
Waiting
CLAassistant
commented
Aug 24, 2026
CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
Hermes (Marian's AI Agent) seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.
Thanks for the contribution!
What happens next:
- A maintainer will review your PR.
- If you don't hear back within a few business days, please comment here to nudge our team.
- This repository is maintained through an internal mirror. When your change is accepted, this PR will close automatically. Don't be alarmed when it closes — that's how it merges, and your authorship is preserved.
This pull request has been automatically marked as stale because it has not had recent activity.
It will be closed in 7 days if no further activity occurs.
If this PR is still relevant:
- Rebase it on the latest main branch
- Add a comment explaining its current status
- Request a review if it's ready
Thank you for your contributions!
This pull request has been automatically closed due to inactivity.
If you'd like to continue working on this, please:
- Create a new branch from the latest main
- Cherry-pick your commits or rebase your changes
- Open a new pull request
Thank you for your understanding!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
RotatingAppenderserialized writes through an unbounded promise chain. UnderSimpleSpanProcessor, seeding a large workspace ends one telemetry span per file — tens of thousands of spans enqueue faster than serial disk appends can drain them, and each queued link pinned its payload plus chain closures until every predecessor settled.Observed on a ~34k-file workspace: multi-GB heap growth during startup, continuing after parsing finished (the memory-growth reports in the wild).
Fix
Queued appends are now capped at 1000 per sink path; beyond the cap, appends are rejected instead of queued — mirroring the OTel SDK's own
maxQueueSizedrop policy. These sinks are local diagnostics: losing records under sustained overload beats unbounded memory growth.Changes
packages/server/src/telemetry-file-sink.ts—pendingcounter +MAX_PENDING_APPENDScap inRotatingAppenderpackages/server/src/telemetry-file-sink.test.ts— test that excess appends are dropped while later appends still land in order.changeset/telemetry-sink-backlog-cap.mdVerification
vitest packages/server/src/telemetry-file-sink.test.ts: 30/30 passedtsc --noEmit(packages/server): clean