Skip to content

Navigation Menu

Sign in
Sign up

Design question: why not use model metadata for contextWindow + built-in compaction? #256

Answered by PsiACE
Lynskylate asked this question in Q&A
Discussion options

Context

I noticed that Bub currently:

  1. Has no contextWindow metadata — It doesn't track each model's max context length (unlike pi, where models.json contains contextWindow per model)
  2. Has no proactive compaction — It relies entirely on the API returning a context-length error, then does a reactive auto-handoff (anchor-based truncation, retrying once)
  3. Uses handoff for truncation, not summarization — When context overflows, older tape entries before the anchor are simply dropped, not summarized

Current behavior

AgentSettings:
 - max_tokens: 16384 (output limit only, not context window)
 
On context overflow:
 - is_context_length_error() regex-matches the API error
 - tape.handoff() creates a new anchor → older messages are silently dropped
 - MAX_AUTO_HANDOFF_RETRIES = 1

Question

Is this by design? Specifically:

  1. Model metadata: any-llm-sdk doesn't appear to expose contextWindow for providers/models. Is there a plan to add this? Or is the philosophy that Bub should be model-agnostic and not rely on hardcoded limits?

  2. Built-in compaction: Would the project consider built-in summarization-based compaction (like pi's /compact), or is the anchor-based handoff considered sufficient? The current approach drops history rather than preserving it in compressed form.

  3. Prevention vs recovery: Is the preference for "try first, recover on error" over "track context usage proactively"? The latter would allow Bub to trim/compact before hitting the API limit.

I'm happy to contribute if there's alignment on a direction. Thanks!

You must be logged in to vote

Thanks for the thoughtful suggestion. We don’t think Bub should try to track each model’s context window directly: the effective limit depends on both the provider and the deployment. Projects such as Pi and models.dev—do maintain similar metadata, but adopting that machinery would add complexity. For now, we assume users understand the constraints of the model they choose.

For recovery and compaction, we’d like to keep anchors as a general mechanism that remains useful across a broad range of cases,our current setup already follows this idea: prompts and skills guide the model to create an anchor proactively before it reaches the context window or another limit. It isn’t perfect, but it ...

Replies: 1 comment 1 reply

Comment options

Thanks for the thoughtful suggestion. We don’t think Bub should try to track each model’s context window directly: the effective limit depends on both the provider and the deployment. Projects such as Pi and models.dev—do maintain similar metadata, but adopting that machinery would add complexity. For now, we assume users understand the constraints of the model they choose.

For recovery and compaction, we’d like to keep anchors as a general mechanism that remains useful across a broad range of cases,our current setup already follows this idea: prompts and skills guide the model to create an anchor proactively before it reaches the context window or another limit. It isn’t perfect, but it works well for most small tasks, a lightweight management aid could still be valuable.

You must be logged in to vote
1 reply
Comment options

Sorry for the delayed reply. Also, I think adding some necessary info related to task progress might help, so I’ve moved this to the discussion. If you have any thoughts, please feel free to comment here.

Answer selected by Lynskylate
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
Converted from issue

This discussion was converted from issue #246 on July 12, 2026 18:50.

AltStyle によって変換されたページ (->オリジナル) /