Skip to content

Navigation Menu

Sign in
Sign up

fix(llm): Claude provider strips tool calls and results from the conversation #3389

Open

Description

Any GAIA agent running against a Claude model loses its own tool calls on the way out. The
provider rebuilds every outbound message as {"role", "content"}, so tool_calls is
discarded — and an assistant turn that carried tool calls but no narration is dropped
entirely. Tool results are then forwarded with role="tool", which Anthropic does not
accept.

The effect is that the model never sees its own tool use in the conversation. It cannot
tell what it already ran, so it re-runs work, loses the thread on multi-step tasks, and has
no basis for batching calls. None of this errors — the request succeeds with a quietly
truncated history, which is why it has gone unnoticed.

The Lemonade/OpenAI path is unaffected.

🔍 Technical details

src/gaia/llm/providers/claude.py:231, _split_system:

if content is None or content == "":
 logger.debug("Dropping empty %s message for Claude request", role)
 continue
cleaned.append({"role": role, "content": content})

Two independent losses:

  1. Whole-turn drop. _build_assistant_message sets content=None when the model
    returned tool calls and no text. That turn is skipped, tool calls included.
  2. Silent key strip. The cleaned.append keeps only role and content. A turn with
    both narration and tool calls survives the first check and still loses its
    tool_calls. This is the worse of the two because it looks fine.

There is no outbound translation to Anthropic's shape anywhere in the provider. The three
tool_use references (claude.py:59, :358, :411) are all inbound — parsing Claude's
response back into OpenAI form. The reverse direction does not exist.

Anthropic expects:

  • assistant turn: content: [{"type":"tool_use","id":...,"name":...,"input":{...}}]
  • result turn: role="user" with content: [{"type":"tool_result","tool_use_id":...,"content":...}]

tests/unit/test_claude_provider.py covers the outbound direction only
(test_tool_use_reencoded_as_lemonade_sentinel); no case feeds the provider an
assistant-message-with-tool_calls or a role=tool message, which is why neither loss is
caught.

Found while investigating #3387 (GAIA never emits a parallel tool call). That benchmark
bypassed this provider, so the two are independent — but this one would discard fan-out
even after #3387 is fixed.

Acceptance criteria

  • Assistant turns carrying tool_calls are translated to Anthropic tool_use blocks
  • role="tool" messages become role="user" with a tool_result block referencing the
    originating tool_use_id
  • An assistant turn with tool calls and no text is no longer dropped
  • Multi-call turns preserve all calls and their ids
  • Tests feeding the provider an assistant-with-tool_calls and a role=tool message,
    asserting the outbound payload shape
  • A multi-step agent run on the Claude path shows prior tool use present in the request

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working llmLLM backend changes p2low priority

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    AltStyle によって変換されたページ (->オリジナル) /