Skip to content

Navigation Menu

Sign in
Sign up

feat(skills): make image generation reachable from the flagship #3073

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kovtcharov-amd wants to merge 4 commits into main
base: main
Choose a base branch
Loading
from feat/image-gen-skill
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/guides/gaia.mdx
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ GAIA is the agent a new user meets first. One agent handles the everyday work yo
- **Data exploration** β€” load a CSV or Excel file into a local SQL scratchpad table and query it, so "which region grew fastest" becomes a real query rather than a guess from a truncated preview.
- **Web research** β€” search the web, fetch and read pages, download files.
- **Files** β€” read, search by name or content, and walk directory trees inside your allowed scope. Writing and editing are available too, behind an explicit confirmation (see [Tools that need your approval](#tools-that-need-your-approval)).
- **Images** β€” describe a local image or answer a question about it, using the local vision model.
- **Images** β€” describe a local image or answer a question about it, using the local vision model. It can also *generate* one from a description with local Stable Diffusion; that swaps the resident model, so the reply after an image takes an extra moment while the chat model reloads.
- **Skills** β€” short playbooks loaded into the agent's own prompt. One ships always-on; the rest are opt-in. See [Skills](#skills) below for what this release ships.

## Requirements
Expand Down Expand Up @@ -85,7 +85,7 @@ It answers **200** when ready and **503** when not, with a `hint` naming the fix

## Tool surface

The default construction registers **67 tools** β€” the 55 of the Chat agent's `full` profile, plus 7 skill-library tools, 4 code-search tools, and the `load_tools` escape hatch. Grouped by what they're for:
The default construction registers **70 tools** β€” the 55 of the Chat agent's `full` profile, plus 7 skill-library tools, 4 code-search tools, 3 image-generation tools, and the `load_tools` escape hatch. Grouped by what they're for:

| Area | Tools |
|---|---|
Expand All @@ -94,7 +94,7 @@ The default construction registers **67 tools** β€” the 55 of the Chat agent's `
| Data | `analyze_data_file`, `create_table`, `insert_data`, `query_data`, `list_tables`, `drop_table` |
| Web | `search_web`, `fetch_page`, `fetch_webpage`, `open_url`, `download_file`, `bookmark`, `search_documentation` |
| Memory | `remember`, `recall`, `forget`, `update_memory`, `search_past_conversations` |
| Images | `analyze_image`, `answer_question_about_image` |
| Images | `analyze_image`, `answer_question_about_image`, `generate_image`, `list_sd_models`, `get_generation_history` |
| Desktop & system | `take_screenshot`, `list_windows`, `get_system_info`, `notify_desktop`, `read_clipboard`, `write_clipboard`, `text_to_speech`, `run_shell_command` |
| Skills | `list_skills`, `search_skill_hub`, `install_skill`, `remove_skill`, `load_skill`, `unload_skill`, `skill_status` |
| Code search | `index_codebase`, `search_code_index`, `get_index_status`, `clear_code_index` |
Expand Down Expand Up @@ -228,4 +228,4 @@ The daemon redirects the sidecar's output to `~/.gaia/agents/gaia/logs/sidecar-<
- **No skill sets declared** β€” the skill machinery is wired and tested, and `gaia-voice` ships always-on, but no *set* does; see [Skills](#skills).
- **Narrowing file scope requires embedding the agent** β€” no sidecar flag for `allowed_paths` yet.
- **No server-side confirmation flow** β€” over `/query`, all six [confirmation-gated tools](#tools-that-need-your-approval) end the run with a refusal instead of prompting over the stream.
- **No image generation** β€” deliberately, so the chat model is never evicted mid-conversation.
- **Image generation evicts the chat model** β€” the SD model has to be resident to draw, so the first reply after an image waits for the chat model to reload. Ask for pictures at a natural pause, not mid-thread.
23 changes: 19 additions & 4 deletions docs/guides/starter-skills.mdx
View file Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Starter Skills"
description: "Ten ready-made SKILL.md skills to install, read, and fork into your own β€” the fastest way to see what GAIA can be told to do."
description: "Ready-made SKILL.md skills to install, read, and fork into your own β€” the fastest way to see what GAIA can be told to do."
icon: "puzzle-piece"
---

Expand All @@ -16,8 +16,8 @@ Markdown procedure. That's the whole format. It is not code, not an agent, and
not a plugin β€” it is a written-down way of doing something, in a shape an agent
can load.

The point of shipping ten of them is not that GAIA does ten things. It is that
**you can describe a new one in an afternoon**. Each skill below is a worked
The point of shipping a pack of them is not that GAIA does exactly these things.
It is that **you can describe a new one in an afternoon**. Each skill below is a worked
example of a different platform primitive, and each is meant to be copied and
edited rather than used verbatim.

Expand Down Expand Up @@ -78,7 +78,7 @@ it registers the web, RAG, scratchpad, memory, file, and shell tools that these
skills consume. A skill loaded into a narrower agent still works, but any tool
it names that the agent lacks is logged as unavailable.

## The ten skills
## The skills

### research-report

Expand Down Expand Up @@ -209,6 +209,21 @@ function, declared in `metadata.gaia.tools` and registered as
- **Note** β€” it fetches through GAIA's `WebClient`, so private and loopback
addresses are refused.

### image-gen

Turns "draw me a red bicycle" into a prompt worth generating, runs it through
local Stable Diffusion, and iterates on the previous image instead of starting
over. Written around the fact that generation is slow enough that you only get
a couple of attempts.

- **Demonstrates** β€” a local non-text model in a skill, and iterating from
session history rather than regenerating from scratch.
- **Consumes** β€” `generate_image`, `list_sd_models`, `get_generation_history`.
- **Configure** β€” a house style clause, if you always want the same look.
- **Note** β€” these tools are only registered on the flagship `gaia` agent, which
turns `enable_sd_tools` on. Generating evicts the chat model to make room for
the SD model, so the next reply reloads it; `ChatAgent` keeps the flag off.

## Fork one

This is the part that matters. Copying a skill and editing it is the whole
Expand Down
43 changes: 33 additions & 10 deletions hub/agents/chat/python/gaia_agent_chat/agent.py
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -750,12 +750,17 @@ def _post_process_tool_result(
return super()._post_process_tool_result(tool_name, _tool_args, tool_result)

def _get_mixin_prompts(self) -> list[str]:
"""Auto-discover mixin prompts, but exclude SD unless actually initialized."""
prompts = super()._get_mixin_prompts()
# Remove SD prompt if SD was not explicitly initialized (saves ~1000 tokens)
if not hasattr(self, "sd_default_model"):
prompts = [p for p in prompts if "Stable Diffusion" not in p]
return prompts
"""Auto-discover mixin prompts, minus SD's.

``SDToolsMixin.get_sd_system_prompt`` opens with "You are an expert
image generation assistant" and runs ~5K chars. It was written for the
standalone SD agent, where that persona was the whole job. Auto-
discovery pulls it in for any class composing the mixin, so on a
general-purpose agent it front-loads the prompt with an identity that
is wrong for every other turn. The procedure lives in the ``image-gen``
skill instead, which renders only when a turn calls for it.
"""
return [p for p in super()._get_mixin_prompts() if "Stable Diffusion" not in p]

def _get_system_prompt(self) -> str:
"""Generate the system prompt for the Chat Agent."""
Expand Down Expand Up @@ -1477,12 +1482,30 @@ def execute_python_file(
# Only registered when explicitly enabled via config.enable_sd_tools=True.
# Off by default to prevent image generation being called for document Q&A.
if getattr(self.config, "enable_sd_tools", False):
from gaia.config import GAIA_CONFIG_DIR

# Absolute, under the user's home. The mixin default is relative to
# cwd, which for a daemon-launched sidecar is the package directory.
sd_output_dir = GAIA_CONFIG_DIR / "cache" / "sd" / "images"
try:
self.init_sd()
logger.debug("SD tools registered (generate_image, list_sd_models)")
except Exception as _sd_err:
self.init_sd(output_dir=str(sd_output_dir))
logger.debug(
"SD tools not available (SD model not loaded): %s", _sd_err
"SD tools registered (generate_image, list_sd_models, "
"get_generation_history), output=%s",
sd_output_dir,
)
except OSError as _sd_err:
# Only the output-dir mkdir can fail here β€” the SD client makes
# no network call at construction β€” so a down server is not a
# trigger. Anything other than OSError is a bug and propagates.
logger.warning(
"Image generation unavailable: could not create the SD "
"output directory %s (%s). Fix that directory's "
"permissions, or point GAIA_CONFIG_DIR somewhere writable. "
"Every other tool is unaffected.",
sd_output_dir,
_sd_err,
exc_info=True,
)

# ── Phase 3: Web & System tools ──────────────────────────────────────────
Expand Down
14 changes: 13 additions & 1 deletion hub/agents/chat/python/gaia_agent_chat/tool_bundles.py
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -377,6 +377,13 @@
members=frozenset({"analyze_image", "answer_question_about_image"}),
description="Analyze images and answer questions about them (VLM).",
),
ToolBundle(
name="image_gen",
members=frozenset(
{"generate_image", "list_sd_models", "get_generation_history"}
),
description="Generate images from a text prompt (Stable Diffusion).",
),
ToolBundle(
name="memory",
members=frozenset(
Expand All @@ -399,7 +406,7 @@

# Bundle members a healthy ``full`` registry may legitimately lack. Handed to
# ToolLoader as ``optional_tools`` so ``validate_registry`` tolerates exactly
# these and still fails loudly on a typo or a deleted tool. Three structural
# these and still fails loudly on a typo or a deleted tool. Four structural
# reasons, not "it might be missing, who knows":
#
# 1. Environment-conditional registration -- ``search_documentation`` needs npx
Expand All @@ -412,6 +419,8 @@
# a degraded-but-running agent has a store and no memory tools. Selection
# already skips CORE names absent from the registry; without this, validation
# turned that survivable state into a hard ValueError on the first turn.
# 4. Config-gated registration -- the SD tools register only under
# ``enable_sd_tools``, which only GaiaAgentConfig turns on.
#
# The CI drift guard checks the other direction against the flagship registry,
# where every one of these IS present, so a rename still fails the build.
Expand All @@ -434,6 +443,9 @@
"search_skill_hub",
"install_skill",
"remove_skill",
"generate_image",
"list_sd_models",
"get_generation_history",
}
)

Expand Down
10 changes: 9 additions & 1 deletion hub/agents/gaia/npm/CHANGELOG.md
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,14 @@ the terminal UI meant building it from source.

### Added

- **Image generation, reachable out of the box.** "Draw me a red bicycle" now
generates a PNG with local Stable Diffusion and reports the path; previously
the tools existed behind a flag nothing turned on, so the agent just said it
couldn't. Adds `generate_image`, `list_sd_models`, and `get_generation_history`
(67 tools β†’ 70) plus an `image_gen` bundle so per-turn selection can find them.
Generating swaps the resident model, so the next reply waits for the chat model
to reload. The `image-gen` starter skill covers prompt expansion and iterating
on the previous image.
- **`503` from `/query` at session capacity.** When every retained session
slot is busy and none is idle enough to evict, starting a new session
returns `503` with the reason in `detail` β€” retryable, distinct from a
Expand All @@ -32,7 +40,7 @@ the terminal UI meant building it from source.
overrides the match threshold, and an embedder outage disables it for the
session (every body renders β€” capability is never lost to a failed match).
- **Per-turn tool selection, now on by default for the flagship `full`
profile.** The model is sent at most 26 of its 67 tools on any one call β€” a
profile.** The model is sent at most 26 of its 70 tools on any one call β€” a
fixed core plus whichever cohesion bundles the query matched β€” instead of the
whole registry every time. No capability is lost: `load_tools` is an escape
hatch the model calls mid-turn to pull in a bundle the selector missed.
Expand Down
2 changes: 1 addition & 1 deletion hub/agents/gaia/npm/SKILL.md
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -327,7 +327,7 @@ Rules a client must respect:
Read this before you design a workflow around it. This section is about the HTTP
surface β€” the agent's other transport can collect an approval; see SPEC Β§5.5.

Six of the agent's 67 tools mutate the machine and need explicit approval before
Six of the agent's 70 tools mutate the machine and need explicit approval before
they run. Four sit in the base `TOOLS_REQUIRING_CONFIRMATION` set β€”
**`write_file`**, **`edit_file`**, **`run_shell_command`**, and
**`execute_python_file`** β€” and the flagship adds two of its own,
Expand Down
3 changes: 2 additions & 1 deletion hub/agents/gaia/python/gaia-agent.yaml
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,11 @@ icon: sparkles
# the agent and compares β€” a hand-edit that disagrees with the registry fails CI.
# 55 from ChatAgent's "full" profile + 7 skill-library tools
# (gaia_agent.skill_tools.SKILL_LIBRARY_TOOL_NAMES) + 4 code-index tools
# + 3 Stable Diffusion tools (enable_sd_tools is on for this agent)
# + the load_tools escape hatch, which registers because dynamic_tools is on.
# This is the REGISTERED size β€” what the agent can do. Dynamic tool loading
# means a single turn only shows the model a subset of it.
tools_count: 67
tools_count: 70

language: python
min_gaia_version: "0.23.0"
Expand Down
2 changes: 1 addition & 1 deletion hub/agents/gaia/python/gaia_agent/__init__.py
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ def build_gaia():
icon="sparkles",
# Must equal the real registry size for the default construction, and
# the manifest's own tools_count. Drift-guarded by tests/test_gaia_agent.py.
tools_count=67,
tools_count=70,
# ChatAgent loads MCP servers dynamically, so the Settings "Active for"
# panel must list this agent for MCP-server connectors.
consumes_mcp_servers=True,
Expand Down
10 changes: 6 additions & 4 deletions hub/agents/gaia/python/gaia_agent/agent.py
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,10 +184,12 @@ class GaiaAgentConfig(ChatAgentConfig):
skill_discovery: bool = True
skill_discovery_threshold: Optional[float] = None

# Image generation stays off: it pulls a second resident model, and evicting
# the chat model to draw a picture is not a trade a document agent should
# make silently.
enable_sd_tools: bool = False
# On for the flagship only. It does pull a second resident model and evict
# the chat model β€” a cost a document agent should not pay silently, so
# ChatAgent keeps it off β€” but this is the general-purpose surface, and off
# here means "draw me a picture" has no answer at all. The image-gen skill
# carries the eviction cost into the procedure.
enable_sd_tools: bool = True

rag_documents: List[str] = field(default_factory=list)

Expand Down
Loading
Loading

AltStyle γ«γ‚ˆγ£γ¦ε€‰ζ›γ•γ‚ŒγŸγƒšγƒΌγ‚Έ (->γ‚ͺγƒͺγ‚ΈγƒŠγƒ«) /