diff --git a/docs/guides/gaia.mdx b/docs/guides/gaia.mdx index 3af0ea89f..3daf2881f 100644 --- a/docs/guides/gaia.mdx +++ b/docs/guides/gaia.mdx @@ -21,7 +21,7 @@ GAIA is the agent a new user meets first. One agent handles the everyday work yo - **Data exploration** — load a CSV or Excel file into a local SQL scratchpad table and query it, so "which region grew fastest" becomes a real query rather than a guess from a truncated preview. - **Web research** — search the web, fetch and read pages, download files. - **Files** — read, search by name or content, and walk directory trees inside your allowed scope. Writing and editing are available too, behind an explicit confirmation (see [Tools that need your approval](#tools-that-need-your-approval)). -- **Images** — describe a local image or answer a question about it, using the local vision model. +- **Images** — describe a local image or answer a question about it, using the local vision model. It can also *generate* one from a description with local Stable Diffusion; that swaps the resident model, so the reply after an image takes an extra moment while the chat model reloads. - **Skills** — short playbooks loaded into the agent's own prompt. One ships always-on; the rest are opt-in. See [Skills](#skills) below for what this release ships. ## Requirements @@ -85,7 +85,7 @@ It answers **200** when ready and **503** when not, with a `hint` naming the fix ## Tool surface -The default construction registers **67 tools** — the 55 of the Chat agent's `full` profile, plus 7 skill-library tools, 4 code-search tools, and the `load_tools` escape hatch. Grouped by what they're for: +The default construction registers **70 tools** — the 55 of the Chat agent's `full` profile, plus 7 skill-library tools, 4 code-search tools, 3 image-generation tools, and the `load_tools` escape hatch. Grouped by what they're for: | Area | Tools | |---|---| @@ -94,7 +94,7 @@ The default construction registers **67 tools** — the 55 of the Chat agent's ` | Data | `analyze_data_file`, `create_table`, `insert_data`, `query_data`, `list_tables`, `drop_table` | | Web | `search_web`, `fetch_page`, `fetch_webpage`, `open_url`, `download_file`, `bookmark`, `search_documentation` | | Memory | `remember`, `recall`, `forget`, `update_memory`, `search_past_conversations` | -| Images | `analyze_image`, `answer_question_about_image` | +| Images | `analyze_image`, `answer_question_about_image`, `generate_image`, `list_sd_models`, `get_generation_history` | | Desktop & system | `take_screenshot`, `list_windows`, `get_system_info`, `notify_desktop`, `read_clipboard`, `write_clipboard`, `text_to_speech`, `run_shell_command` | | Skills | `list_skills`, `search_skill_hub`, `install_skill`, `remove_skill`, `load_skill`, `unload_skill`, `skill_status` | | Code search | `index_codebase`, `search_code_index`, `get_index_status`, `clear_code_index` | @@ -228,4 +228,4 @@ The daemon redirects the sidecar's output to `~/.gaia/agents/gaia/logs/sidecar-< - **No skill sets declared** — the skill machinery is wired and tested, and `gaia-voice` ships always-on, but no *set* does; see [Skills](#skills). - **Narrowing file scope requires embedding the agent** — no sidecar flag for `allowed_paths` yet. - **No server-side confirmation flow** — over `/query`, all six [confirmation-gated tools](#tools-that-need-your-approval) end the run with a refusal instead of prompting over the stream. -- **No image generation** — deliberately, so the chat model is never evicted mid-conversation. +- **Image generation evicts the chat model** — the SD model has to be resident to draw, so the first reply after an image waits for the chat model to reload. Ask for pictures at a natural pause, not mid-thread. diff --git a/docs/guides/starter-skills.mdx b/docs/guides/starter-skills.mdx index 4ba54ad17..f77514117 100644 --- a/docs/guides/starter-skills.mdx +++ b/docs/guides/starter-skills.mdx @@ -1,6 +1,6 @@ --- title: "Starter Skills" -description: "Ten ready-made SKILL.md skills to install, read, and fork into your own — the fastest way to see what GAIA can be told to do." +description: "Ready-made SKILL.md skills to install, read, and fork into your own — the fastest way to see what GAIA can be told to do." icon: "puzzle-piece" --- @@ -16,8 +16,8 @@ Markdown procedure. That's the whole format. It is not code, not an agent, and not a plugin — it is a written-down way of doing something, in a shape an agent can load. -The point of shipping ten of them is not that GAIA does ten things. It is that -**you can describe a new one in an afternoon**. Each skill below is a worked +The point of shipping a pack of them is not that GAIA does exactly these things. +It is that **you can describe a new one in an afternoon**. Each skill below is a worked example of a different platform primitive, and each is meant to be copied and edited rather than used verbatim. @@ -78,7 +78,7 @@ it registers the web, RAG, scratchpad, memory, file, and shell tools that these skills consume. A skill loaded into a narrower agent still works, but any tool it names that the agent lacks is logged as unavailable. -## The ten skills +## The skills ### research-report @@ -209,6 +209,21 @@ function, declared in `metadata.gaia.tools` and registered as - **Note** — it fetches through GAIA's `WebClient`, so private and loopback addresses are refused. +### image-gen + +Turns "draw me a red bicycle" into a prompt worth generating, runs it through +local Stable Diffusion, and iterates on the previous image instead of starting +over. Written around the fact that generation is slow enough that you only get +a couple of attempts. + +- **Demonstrates** — a local non-text model in a skill, and iterating from + session history rather than regenerating from scratch. +- **Consumes** — `generate_image`, `list_sd_models`, `get_generation_history`. +- **Configure** — a house style clause, if you always want the same look. +- **Note** — these tools are only registered on the flagship `gaia` agent, which + turns `enable_sd_tools` on. Generating evicts the chat model to make room for + the SD model, so the next reply reloads it; `ChatAgent` keeps the flag off. + ## Fork one This is the part that matters. Copying a skill and editing it is the whole diff --git a/hub/agents/chat/python/gaia_agent_chat/agent.py b/hub/agents/chat/python/gaia_agent_chat/agent.py index 214851666..0ebcec1ed 100644 --- a/hub/agents/chat/python/gaia_agent_chat/agent.py +++ b/hub/agents/chat/python/gaia_agent_chat/agent.py @@ -750,12 +750,17 @@ def _post_process_tool_result( return super()._post_process_tool_result(tool_name, _tool_args, tool_result) def _get_mixin_prompts(self) -> list[str]: - """Auto-discover mixin prompts, but exclude SD unless actually initialized.""" - prompts = super()._get_mixin_prompts() - # Remove SD prompt if SD was not explicitly initialized (saves ~1000 tokens) - if not hasattr(self, "sd_default_model"): - prompts = [p for p in prompts if "Stable Diffusion" not in p] - return prompts + """Auto-discover mixin prompts, minus SD's. + + ``SDToolsMixin.get_sd_system_prompt`` opens with "You are an expert + image generation assistant" and runs ~5K chars. It was written for the + standalone SD agent, where that persona was the whole job. Auto- + discovery pulls it in for any class composing the mixin, so on a + general-purpose agent it front-loads the prompt with an identity that + is wrong for every other turn. The procedure lives in the ``image-gen`` + skill instead, which renders only when a turn calls for it. + """ + return [p for p in super()._get_mixin_prompts() if "Stable Diffusion" not in p] def _get_system_prompt(self) -> str: """Generate the system prompt for the Chat Agent.""" @@ -1477,12 +1482,30 @@ def execute_python_file( # Only registered when explicitly enabled via config.enable_sd_tools=True. # Off by default to prevent image generation being called for document Q&A. if getattr(self.config, "enable_sd_tools", False): + from gaia.config import GAIA_CONFIG_DIR + + # Absolute, under the user's home. The mixin default is relative to + # cwd, which for a daemon-launched sidecar is the package directory. + sd_output_dir = GAIA_CONFIG_DIR / "cache" / "sd" / "images" try: - self.init_sd() - logger.debug("SD tools registered (generate_image, list_sd_models)") - except Exception as _sd_err: + self.init_sd(output_dir=str(sd_output_dir)) logger.debug( - "SD tools not available (SD model not loaded): %s", _sd_err + "SD tools registered (generate_image, list_sd_models, " + "get_generation_history), output=%s", + sd_output_dir, + ) + except OSError as _sd_err: + # Only the output-dir mkdir can fail here — the SD client makes + # no network call at construction — so a down server is not a + # trigger. Anything other than OSError is a bug and propagates. + logger.warning( + "Image generation unavailable: could not create the SD " + "output directory %s (%s). Fix that directory's " + "permissions, or point GAIA_CONFIG_DIR somewhere writable. " + "Every other tool is unaffected.", + sd_output_dir, + _sd_err, + exc_info=True, ) # ── Phase 3: Web & System tools ────────────────────────────────────────── diff --git a/hub/agents/chat/python/gaia_agent_chat/tool_bundles.py b/hub/agents/chat/python/gaia_agent_chat/tool_bundles.py index 00b396dda..a9e2e437e 100644 --- a/hub/agents/chat/python/gaia_agent_chat/tool_bundles.py +++ b/hub/agents/chat/python/gaia_agent_chat/tool_bundles.py @@ -377,6 +377,13 @@ members=frozenset({"analyze_image", "answer_question_about_image"}), description="Analyze images and answer questions about them (VLM).", ), + ToolBundle( + name="image_gen", + members=frozenset( + {"generate_image", "list_sd_models", "get_generation_history"} + ), + description="Generate images from a text prompt (Stable Diffusion).", + ), ToolBundle( name="memory", members=frozenset( @@ -399,7 +406,7 @@ # Bundle members a healthy ``full`` registry may legitimately lack. Handed to # ToolLoader as ``optional_tools`` so ``validate_registry`` tolerates exactly -# these and still fails loudly on a typo or a deleted tool. Three structural +# these and still fails loudly on a typo or a deleted tool. Four structural # reasons, not "it might be missing, who knows": # # 1. Environment-conditional registration -- ``search_documentation`` needs npx @@ -412,6 +419,8 @@ # a degraded-but-running agent has a store and no memory tools. Selection # already skips CORE names absent from the registry; without this, validation # turned that survivable state into a hard ValueError on the first turn. +# 4. Config-gated registration -- the SD tools register only under +# ``enable_sd_tools``, which only GaiaAgentConfig turns on. # # The CI drift guard checks the other direction against the flagship registry, # where every one of these IS present, so a rename still fails the build. @@ -434,6 +443,9 @@ "search_skill_hub", "install_skill", "remove_skill", + "generate_image", + "list_sd_models", + "get_generation_history", } ) diff --git a/hub/agents/gaia/npm/CHANGELOG.md b/hub/agents/gaia/npm/CHANGELOG.md index a1c23bfb1..02d453a0c 100644 --- a/hub/agents/gaia/npm/CHANGELOG.md +++ b/hub/agents/gaia/npm/CHANGELOG.md @@ -14,6 +14,14 @@ the terminal UI meant building it from source. ### Added +- **Image generation, reachable out of the box.** "Draw me a red bicycle" now + generates a PNG with local Stable Diffusion and reports the path; previously + the tools existed behind a flag nothing turned on, so the agent just said it + couldn't. Adds `generate_image`, `list_sd_models`, and `get_generation_history` + (67 tools → 70) plus an `image_gen` bundle so per-turn selection can find them. + Generating swaps the resident model, so the next reply waits for the chat model + to reload. The `image-gen` starter skill covers prompt expansion and iterating + on the previous image. - **`503` from `/query` at session capacity.** When every retained session slot is busy and none is idle enough to evict, starting a new session returns `503` with the reason in `detail` — retryable, distinct from a @@ -32,7 +40,7 @@ the terminal UI meant building it from source. overrides the match threshold, and an embedder outage disables it for the session (every body renders — capability is never lost to a failed match). - **Per-turn tool selection, now on by default for the flagship `full` - profile.** The model is sent at most 26 of its 67 tools on any one call — a + profile.** The model is sent at most 26 of its 70 tools on any one call — a fixed core plus whichever cohesion bundles the query matched — instead of the whole registry every time. No capability is lost: `load_tools` is an escape hatch the model calls mid-turn to pull in a bundle the selector missed. diff --git a/hub/agents/gaia/npm/SKILL.md b/hub/agents/gaia/npm/SKILL.md index 472b1e2c6..066f0a0af 100644 --- a/hub/agents/gaia/npm/SKILL.md +++ b/hub/agents/gaia/npm/SKILL.md @@ -327,7 +327,7 @@ Rules a client must respect: Read this before you design a workflow around it. This section is about the HTTP surface — the agent's other transport can collect an approval; see SPEC §5.5. -Six of the agent's 67 tools mutate the machine and need explicit approval before +Six of the agent's 70 tools mutate the machine and need explicit approval before they run. Four sit in the base `TOOLS_REQUIRING_CONFIRMATION` set — **`write_file`**, **`edit_file`**, **`run_shell_command`**, and **`execute_python_file`** — and the flagship adds two of its own, diff --git a/hub/agents/gaia/python/gaia-agent.yaml b/hub/agents/gaia/python/gaia-agent.yaml index 54feda1c8..9e9ac1f3b 100644 --- a/hub/agents/gaia/python/gaia-agent.yaml +++ b/hub/agents/gaia/python/gaia-agent.yaml @@ -17,10 +17,11 @@ icon: sparkles # the agent and compares — a hand-edit that disagrees with the registry fails CI. # 55 from ChatAgent's "full" profile + 7 skill-library tools # (gaia_agent.skill_tools.SKILL_LIBRARY_TOOL_NAMES) + 4 code-index tools +# + 3 Stable Diffusion tools (enable_sd_tools is on for this agent) # + the load_tools escape hatch, which registers because dynamic_tools is on. # This is the REGISTERED size — what the agent can do. Dynamic tool loading # means a single turn only shows the model a subset of it. -tools_count: 67 +tools_count: 70 language: python min_gaia_version: "0.23.0" diff --git a/hub/agents/gaia/python/gaia_agent/__init__.py b/hub/agents/gaia/python/gaia_agent/__init__.py index 63aaa7cc3..59d3e96af 100644 --- a/hub/agents/gaia/python/gaia_agent/__init__.py +++ b/hub/agents/gaia/python/gaia_agent/__init__.py @@ -74,7 +74,7 @@ def build_gaia(): icon="sparkles", # Must equal the real registry size for the default construction, and # the manifest's own tools_count. Drift-guarded by tests/test_gaia_agent.py. - tools_count=67, + tools_count=70, # ChatAgent loads MCP servers dynamically, so the Settings "Active for" # panel must list this agent for MCP-server connectors. consumes_mcp_servers=True, diff --git a/hub/agents/gaia/python/gaia_agent/agent.py b/hub/agents/gaia/python/gaia_agent/agent.py index 111d8bdd4..108a3311b 100644 --- a/hub/agents/gaia/python/gaia_agent/agent.py +++ b/hub/agents/gaia/python/gaia_agent/agent.py @@ -184,10 +184,12 @@ class GaiaAgentConfig(ChatAgentConfig): skill_discovery: bool = True skill_discovery_threshold: Optional[float] = None - # Image generation stays off: it pulls a second resident model, and evicting - # the chat model to draw a picture is not a trade a document agent should - # make silently. - enable_sd_tools: bool = False + # On for the flagship only. It does pull a second resident model and evict + # the chat model — a cost a document agent should not pay silently, so + # ChatAgent keeps it off — but this is the general-purpose surface, and off + # here means "draw me a picture" has no answer at all. The image-gen skill + # carries the eviction cost into the procedure. + enable_sd_tools: bool = True rag_documents: List[str] = field(default_factory=list) diff --git a/hub/agents/gaia/python/tests/test_sd_tools_do_not_rewrite_the_prompt.py b/hub/agents/gaia/python/tests/test_sd_tools_do_not_rewrite_the_prompt.py new file mode 100644 index 000000000..e3c1f8866 --- /dev/null +++ b/hub/agents/gaia/python/tests/test_sd_tools_do_not_rewrite_the_prompt.py @@ -0,0 +1,133 @@ +# Copyright(C) 2025-2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT +"""Turning image generation on must not change who the agent thinks it is. + +``SDToolsMixin`` ships a ``get_sd_system_prompt`` fragment that opens with +"You are an expert image generation assistant" and runs ~5K chars. Base-agent +prompt composition auto-discovers every ``get_*_system_prompt`` method on the +instance, so once ``init_sd`` runs that fragment lands at the FRONT of the +flagship's prompt — measured at +4,971 chars (a 40% increase) and a persona +that is wrong for every non-image turn. + +The capability still has to be reachable, so the fix is not "leave SD off": it +is that the procedure belongs in the ``image-gen`` skill, which renders only on +turns that need it. These tests pin both halves — tools present, persona absent. +""" + +from __future__ import annotations + +import contextlib + +import pytest +from gaia_agent.agent import GaiaAgent, GaiaAgentConfig + +from gaia.agents.base.tools import _TOOL_REGISTRY + +SD_TOOLS = {"generate_image", "list_sd_models", "get_generation_history"} + + +@contextlib.contextmanager +def _isolated_registry(): + saved = dict(_TOOL_REGISTRY) + _TOOL_REGISTRY.clear() + try: + yield + finally: + _TOOL_REGISTRY.clear() + _TOOL_REGISTRY.update(saved) + + +@pytest.fixture(scope="module") +def flagship(): + """The built agent, plus a SNAPSHOT of the registry it was built with. + + ``select`` scores against whatever registry it is handed, and sibling test + modules clear the process-global ``_TOOL_REGISTRY`` in their own fixtures. + Reading it live makes selection assertions depend on test order — a smaller + registry lets everything fit under the cap, so a negative assertion passes + or fails according to what ran first. The snapshot pins it. + """ + with _isolated_registry(), pytest.MonkeyPatch.context() as mp: + mp.setenv("GAIA_MEMORY_DISABLED", "1") + agent = GaiaAgent(config=GaiaAgentConfig(silent_mode=True)) + agent._registry_snapshot = dict(agent._tools_registry) + yield agent + + +def _select_fresh(agent, query): + """First-turn selection for ``query``, independent of test order. + + Skips when the embedder is unreachable: scoring a real query against real + tool descriptions needs live embeddings, and a plain CI runner has no + Lemonade. Keyed off the loader's own ``session_disabled`` flag rather than + a bare ``None``, so a genuine selection regression still fails here. + """ + agent.tool_loader.reset_session() + selected = agent.tool_loader.select(query, agent._registry_snapshot) + if selected is None: + if agent.tool_loader.session_disabled: + pytest.skip("semantic tool selection needs a reachable embedder") + pytest.fail("select() returned None with the session still enabled") + return selected + + +def test_image_generation_is_reachable_out_of_the_box(flagship): + """The point of the change: a default flagship can actually draw. + + PR #2995 removed the standalone SD agent on the basis that image + generation stayed available behind ``enable_sd_tools`` — but nothing + turned that flag on, so no user could reach it. + """ + assert GaiaAgentConfig().enable_sd_tools is True + assert SD_TOOLS <= set(flagship._registry_snapshot) + + +def test_the_sd_persona_stays_out_of_the_system_prompt(flagship): + """The regression: SD tools on must not rewrite the agent's identity.""" + prompt = flagship.system_prompt + + assert "expert image generation assistant" not in prompt + # The distinctive junk from the SD prompt's "proven quality boosters" list; + # its presence means the whole ~5K fragment came along. + assert "Aqua Vista" not in prompt + + +def test_the_capability_is_still_advertised_as_a_bundle(flagship): + """Suppressing the persona must not make the tools undiscoverable. + + The one-line bundle entry is how a semantic miss is recovered, and it is + the entire intended prompt cost of this capability. + """ + assert "- image_gen:" in flagship.system_prompt + + +@pytest.mark.parametrize( + "query", + [ + "draw me a picture of a red bicycle", + "generate an image of a mountain at sunset", + ], +) +def test_an_image_request_selects_the_tools_without_any_skill_loaded(flagship, query): + """Registering the tools is not the same as the model being shown them. + + The flagship runs per-turn semantic tool selection, so a tool outside the + turn's selected set is invisible to the model. If this fails the capability + is only reachable through the escape hatch, which the model has to think to + use. + """ + selected = _select_fresh(flagship, query) + + assert SD_TOOLS <= set(selected) + + +def test_a_document_question_does_not_drag_in_the_image_tools(flagship): + """The other half: breadth must not cost every unrelated turn. + + Must be a FRESH conversation. Selected tools stay loaded across turns + until the cap evicts them, so asking this after an image request measures + that (correct) stickiness rather than the first-turn match. + """ + selected = _select_fresh(flagship, "what does my document say about revenue") + + assert not SD_TOOLS & set(selected) diff --git a/hub/skills/README.md b/hub/skills/README.md index 51c8719fd..7a19e70d2 100644 --- a/hub/skills/README.md +++ b/hub/skills/README.md @@ -5,10 +5,11 @@ a procedure written for a model to follow, plus the tools it needs. Unlike an agent, it does not run on its own, so it is a separate lane with its own package format and publish contract ([#2467](https://github.com/amd/gaia/issues/2467)). -This directory is the AMD **starter pack** (#893): thirteen worked examples, each +This directory is the AMD **starter pack** (#893): fourteen worked examples, each demonstrating a different platform primitive (RAG, scratchpad SQL, memory, -browser, file I/O). They are meant to be copied and edited, not used verbatim — -see the [starter skills guide](https://amd-gaia.ai/docs/guides/starter-skills). +browser, file I/O, image generation). They are meant to be copied and edited, +not used verbatim — see the +[starter skills guide](https://amd-gaia.ai/docs/guides/starter-skills). ## Package format diff --git a/hub/skills/image-gen/SKILL.md b/hub/skills/image-gen/SKILL.md new file mode 100644 index 000000000..339c24c21 --- /dev/null +++ b/hub/skills/image-gen/SKILL.md @@ -0,0 +1,120 @@ +--- +name: image-gen +description: Turn a description into an image file with local Stable Diffusion, then iterate on it. Use when the user says draw, sketch, paint, render, "make a picture of", "generate an image", or asks for concept art, a thumbnail, a logo idea, a wallpaper, or a mockup — and when they want the last image changed rather than replaced. +license: MIT +version: 1.0.0 +metadata: + gaia: + security_tier: community + tools_required: + - generate_image + - list_sd_models + - get_generation_history + provenance: + source: starter-pack +--- + +# Image Generation + +Generation runs locally and is slow — tens of seconds to minutes per image, and +the first call for a model downloads gigabytes. That changes the job: you get +few attempts, so spend the thinking *before* the call rather than firing off +four variations and picking one. + +It also costs the conversation. Drawing loads the image model in place of the +chat model, so the reply after an image pauses while the chat model comes back. +Generate when the user actually asked for a picture — not to illustrate an +answer they did not ask to have illustrated. + +## Check what is loaded before you promise anything + +Call `list_sd_models()` first. It tells you which models exist and what each +costs, and the reported `default_model` is the one you get if you pass no +`model`. Do not assume a specific model is resident — naming one the machine +has not pulled turns a 20-second request into a multi-gigabyte download the +user did not agree to. + +Tell the user the estimate before a slow model, not after: SDXL-Base-1.0 at +1024x1024 is on the order of minutes, the Turbo models are seconds. + +## Build the prompt for them + +A user asking for "a cat" has a picture in their head that "a cat" will not +produce. Expand it yourself rather than interrogating them — one round of +questions is fine, three is a worse experience than a decent first image. + +A usable prompt names, roughly in this order: **subject**, **what it is doing +or how it is arranged**, **setting**, **style**, **lighting or mood**. So +"a red bicycle" becomes "a red bicycle leaning against a brick wall, morning +sunlight, shallow depth of field, photographic". + +Then say the expanded prompt back to the user with the result. They cannot +correct a prompt they never saw, and "make it warmer" is only meaningful if +they know what you asked for. + +## The default is a few-step model — do not over-tune it + +SDXL-Turbo is the default and it is distilled to converge in about **4 steps** +with **CFG around 1.0**. The knobs that matter on a normal model do nothing +useful here: + +- Raising `steps` to 30 costs seven times the wall clock and does not improve + the image. +- Raising `cfg_scale` degrades it — Turbo models are trained for guidance-free + sampling. +- Long negative-prompt boilerplate ("blurry, low quality, watermark, extra + fingers...") is wasted. Spend those words describing what you *do* want. + +Leave `steps`, `cfg_scale`, and `size` unset unless you have a reason; the tool +fills in the right values per model. Reach for `SDXL-Base-1.0` only when the +user explicitly wants photorealism and has accepted the wait. + +## Iterate instead of starting over + +`get_generation_history()` returns this session's generations with the exact +prompt, model, size and seed of each. When the user says "same but at sunset" +or "make it wider", read the previous entry, change the one thing they asked +about, and keep everything else — including the `seed`. Reusing the seed is +what makes the second image recognisably the same picture rather than an +unrelated one that happens to match the words. + +Rewriting the prompt from scratch throws away everything that was already +working, and the user has to re-explain the parts they liked. + +## When it fails, say what failed + +`generate_image` returns `{"status": "error", "error": ...}` rather than +raising. Read it and pass the actual message to the user. + +**Do not quietly retry with a different model, a smaller size, or fewer +steps.** A user who asked for a photorealistic 1024px render and silently +received a 512px Turbo sketch has been given the wrong thing and told nothing. +If a fallback would genuinely help, propose it and let them choose. + +The common failures and what to say: + +- **Cannot reach Lemonade Server** — inference is not running. Tell them to + start it; nothing here works until it is up. +- **Timed out** — usually the first use of a model, downloading several GB. + The server is fine. Tell them to pre-fetch it (`lemonade-server pull + `) and retry, rather than restarting anything. +- **Invalid model or size** — you passed something outside the supported set. + Call `list_sd_models()` and pick from what it returned. + +## Reporting a generated image + +Give them the path. It is the only part of the result they can act on: + +> Saved to `~/.gaia/cache/sd/images/a_red_bicycle_..._SDXL-Turbo_....png` (18s). +> +> Prompt used: "a red bicycle leaning against a brick wall, morning sunlight, +> shallow depth of field, photographic" — say the word if you want it warmer, +> wider, or at a different time of day. + +Never describe an image you did not generate, and never claim a file exists +because the call was made — check `status` first. + +## Fork this + +Pin the style clause in step two to your own house look (brand palette, flat +vector, isometric) and the skill stops needing to be told it every time. diff --git a/src/gaia/sd/mixin.py b/src/gaia/sd/mixin.py index 372665be1..ca9290d67 100644 --- a/src/gaia/sd/mixin.py +++ b/src/gaia/sd/mixin.py @@ -2,7 +2,7 @@ SDToolsMixin - Stable Diffusion image generation tools for GAIA agents. Provides tools to generate images using the Lemonade Server SD endpoint. -Supports 4 SD models: SD-Turbo (fast, default), SDXL-Turbo, SD-1.5, and +Supports 4 SD models: SDXL-Turbo (the default), SD-Turbo (faster), SD-1.5, and SDXL-Base-1.0 (photorealistic) running on Ryzen AI. Example: @@ -56,6 +56,12 @@ class SDToolsMixin: Constants SD_MODELS and SD_SIZES are duplicated from LemonadeClient for convenience. Primary source of truth is LemonadeClient, but having them here allows direct access via SDToolsMixin.SD_MODELS for better API ergonomics. + + ``get_sd_system_prompt`` is opt-in. ChatAgent drops it so the ~5K-char + "expert image generation assistant" persona does not front-load every + turn; that guidance lives in the ``image-gen`` skill instead. A + standalone agent composing this mixin gets no SD prompt unless its own + ``_get_system_prompt`` returns one. """ # Supported configurations (duplicated from LemonadeClient for API convenience) @@ -144,7 +150,7 @@ def init_sd( }, "model": { "type": "str", - "description": "SD model: SD-Turbo (fast, default), SDXL-Turbo (better), SDXL-Base-1.0 (photorealistic, slow), SD-1.5", + "description": "SD model: SDXL-Turbo (default), SD-Turbo (faster, lower quality), SDXL-Base-1.0 (photorealistic, slow), SD-1.5. Omit to use the default.", "required": False, }, "size": { @@ -191,7 +197,7 @@ def list_sd_models() -> Dict[str, Any]: "models": [ { "name": "SD-Turbo", - "description": "Very fast, 512x512, 4 steps (default)", + "description": "Very fast, 512x512, 4 steps", "recommended_steps": 4, "recommended_size": "512x512", "speed": "~13s", @@ -336,11 +342,11 @@ def _generate_image( if "already loaded" in str(e).lower(): logger.debug(f"Model already loaded: {model}") else: - # Connection error or other failure - return error - error_msg = str(e) - if "Connection" in error_msg or "connect" in error_msg.lower(): - error_msg = "Cannot connect to Lemonade Server. Is it running?" - return {"status": "error", "error": error_msg} + logger.error("Failed to load SD model %s: %s", model, e) + return { + "status": "error", + "error": self._describe_client_error(e, model=model), + } # Start progress for generation with timer (show_timer not supported by all consoles) if console and hasattr(console, "start_progress"): @@ -434,14 +440,12 @@ def _generate_image( if console and hasattr(console, "stop_progress"): console.stop_progress() - error_msg = str(e) - if "Connection" in error_msg or "connect" in error_msg.lower(): - error_msg = "Cannot connect to Lemonade Server. Is it running?" + error_msg = self._describe_client_error(e, model=model) if console and hasattr(console, "print_error"): console.print_error(error_msg) - logger.error(error_msg) + logger.error("SD generation failed for %s: %s", model, e) return {"status": "error", "error": error_msg} except Exception as e: @@ -456,6 +460,34 @@ def _generate_image( logger.error(error_msg, exc_info=True) return {"status": "error", "error": error_msg} + @staticmethod + def _describe_client_error(error: Exception, model: str) -> str: + """Turn a Lemonade client error into something the user can act on. + + Order matters: a ``requests`` read-timeout carries "HTTPConnectionPool" + in its text, so a substring test for "connect" reports a live server as + unreachable and sends the user off to restart something that was fine. + Timeouts are checked first. + """ + raw = str(error) + lowered = raw.lower() + + if "timed out" in lowered or "timeout" in lowered: + return ( + f"Timed out waiting for {model}; the server is running but did " + "not answer in time. First use of an SD model both downloads " + "and loads several GB. Pre-fetch it with `lemonade-server pull " + f"{model}`, confirm it loads with `lemonade-server load " + f"{model}`, then retry. ({raw})" + ) + if "connection refused" in lowered or "failed to establish" in lowered: + return ( + "Cannot reach Lemonade Server. Start it with " + "`lemonade-server serve`, or set LEMONADE_BASE_URL to a running " + f"server. ({raw})" + ) + return f"Image generation failed for {model}: {raw}" + def _estimate_generation_time(self, model: str, size: str) -> str: """ Estimate generation time based on model and size. diff --git a/tests/unit/test_sd_error_messages.py b/tests/unit/test_sd_error_messages.py new file mode 100644 index 000000000..704750184 --- /dev/null +++ b/tests/unit/test_sd_error_messages.py @@ -0,0 +1,64 @@ +# Copyright(C) 2025-2026 Advanced Micro Devices, Inc. All rights reserved. +# SPDX-License-Identifier: MIT +"""Image-generation errors must name the failure the user actually hit. + +The bug this pins: ``requests`` puts "HTTPConnectionPool" in the text of a +*read timeout* as well as a refused connection, so a substring test for +"connect" reported a healthy server as unreachable. A first-use model download +runs to several GB and routinely outlasts the request window, so the message a +user saw most often was the one telling them to restart a server that was fine. +""" + +from __future__ import annotations + +import pytest + +from gaia.llm.lemonade_client import LemonadeClientError +from gaia.sd.mixin import SDToolsMixin + +_TIMEOUT = LemonadeClientError( + "Request failed: HTTPConnectionPool(host='localhost', port=13305): " + "Read timed out. (read timeout=600)" +) +_REFUSED = LemonadeClientError( + "Request failed: HTTPConnectionPool(host='localhost', port=19999): " + "Max retries exceeded with url: /api/v1/load (Caused by NewConnectionError(" + "\"HTTPConnection(host='localhost', port=19999): Failed to establish a new " + 'connection: [WinError 10061] No connection could be made"))' +) + + +def test_a_download_timeout_is_not_reported_as_an_unreachable_server(): + """The regression: both errors carry 'HTTPConnectionPool'.""" + message = SDToolsMixin._describe_client_error(_TIMEOUT, model="SDXL-Turbo") + + assert "timed out" in message.lower() + assert "SDXL-Turbo" in message, "the user cannot pre-fetch an unnamed model" + assert "pull" in message, "a timeout must point at the fix, not just the symptom" + # The wrong diagnosis, in the words that would send the user to restart. + assert "cannot reach" not in message.lower() + + +def test_a_refused_connection_says_the_server_is_not_running(): + message = SDToolsMixin._describe_client_error(_REFUSED, model="SDXL-Turbo") + + assert "cannot reach" in message.lower() + assert "lemonade-server serve" in message + assert "timed out" not in message.lower() + + +@pytest.mark.parametrize("error", [_TIMEOUT, _REFUSED], ids=["timeout", "refused"]) +def test_the_raw_error_survives_for_debugging(error): + """A friendlier message must not delete the detail a bug report needs.""" + assert "13305" in SDToolsMixin._describe_client_error( + error, model="SDXL-Turbo" + ) or "19999" in SDToolsMixin._describe_client_error(error, model="SDXL-Turbo") + + +def test_an_unrecognized_failure_is_passed_through_named(): + message = SDToolsMixin._describe_client_error( + LemonadeClientError("out of VRAM"), model="SDXL-Base-1.0" + ) + + assert "out of VRAM" in message + assert "SDXL-Base-1.0" in message diff --git a/tests/unit/test_starter_skills.py b/tests/unit/test_starter_skills.py index 43316b281..5083bc92c 100644 --- a/tests/unit/test_starter_skills.py +++ b/tests/unit/test_starter_skills.py @@ -142,7 +142,7 @@ def test_starter_skill_permissions_resolve_against_the_real_catalog(skill_dir: P @pytest.fixture(scope="module") -def registry_tool_names() -> frozenset[str]: +def registry_tool_names(tmp_path_factory) -> frozenset[str]: """Tool names the mixins a starter skill may target actually register. Registrars are invoked on bare stubs — they only close over ``self`` inside @@ -159,6 +159,7 @@ def registry_tool_names() -> frozenset[str]: from gaia.agents.tools.rag_tools import RAGToolsMixin from gaia.agents.tools.scratchpad_tools import ScratchpadToolsMixin from gaia.agents.tools.shell_tools import ShellToolsMixin + from gaia.sd.mixin import SDToolsMixin class _Stub: """Enough surface for the registrars. @@ -192,6 +193,12 @@ def _get_memory_store(self): _TOOL_REGISTRY.clear() for mixin, method in registrars: getattr(mixin, method)(_Stub()) + # SD registers inside ``init_sd`` rather than a ``register_*`` method, + # so it needs the initializer — and an explicit output_dir, or it + # mkdirs ``.gaia/`` into the developer's cwd. No server is contacted. + SDToolsMixin.init_sd( + _Stub(), output_dir=str(tmp_path_factory.mktemp("sd-images")) + ) names = frozenset(_TOOL_REGISTRY) finally: _TOOL_REGISTRY.clear() @@ -225,9 +232,15 @@ def _chat_agent_inline_tools() -> frozenset[str]: def test_registry_fixture_actually_registered_something(registry_tool_names): """Guards the guard: an empty set would make the check below vacuous.""" - assert {"search_web", "fetch_page", "query_documents", "recall"} <= ( - registry_tool_names - ) + assert { + "search_web", + "fetch_page", + "query_documents", + "recall", + # SD registers via init_sd, not a register_* method — if that call + # starts failing quietly, image-gen's honesty check goes vacuous. + "generate_image", + } <= registry_tool_names @pytest.mark.parametrize("skill_dir", STARTER_DIRS, ids=_ids(STARTER_DIRS))

AltStyle によって変換されたページ (->オリジナル) /