Workspace & product surface teardown — diff, highlighting, and missing capabilities #3996

Workspace & product surface teardown — actionable findings

A teardown of AO's non-terminal surfaces — diff/files panel, code highlighting, and missing capabilities — with concrete, actionable improvements. Every claim is verified against code in this repo (not docs). The terminal stack has its own thread: #3991.

Diff/Files panel: it exists; the gap is narrower than it looks

AO ships a real diff viewer (SessionFilesView.tsx + service/session/workspace_files.go): changed-files list with per-file +/- badges, unified/split views, lazy per-file diff, virtualization, Web Worker parsing, intra-line LCS highlighting, SSE invalidation, and file/line feedback routed back to the agent. That last part is stronger than the alternative model.

What's missing (full side-by-side + priority order in issue #3982):

1. Git-state sections — staged / unstaged / committed / untracked + a commit list. AO shows one "working tree vs base/HEAD" list and can't separate committed from dirty work.
2. Aggregate workspace summary — +files/+adds/-dels totals cheap enough for many sidebar rows.
3. In-app commit/push (editing optional).
4. Ahead/behind + push/pull counts in the panel.

Code highlighting: unify and cover the diff panel

• Chat already highlights fenced code and patches via HighlightedCode (lowlight engine behind lib/code-highlight.ts).
• The Files panel diff renders plain text — no syntax highlighting — even though HighlightedCode and the shared grammar cache already exist.
• code-highlight.ts documents why lowlight was chosen: the renderer CSP is script-src 'self' with no wasm-unsafe-eval, which blocks a WASM-based highlighter's engine. So swapping to a TextMate/WASM engine means either its slower JS engine + heavier grammars, or a CSP relaxation — a real tradeoff, not a free win.

Actionable: (1) route Files-panel diff lines through the same HighlightedCode engine (per-file language) so the diff viewer reads like the chat timeline; (2) only revisit the highlighter engine itself if the CSP is relaxed or the JS-engine cost is measured and justified.

Missing capabilities (biggest first)

Automations — scheduled agent sessions

Nothing exists today. The portable shape: a daemon ticker reads an indexed next_run_at; spawn via the existing session_manager.Spawn; a runs table with a unique (automation_id, scheduled_for) for idempotency; RRule stored as text (CLI takes --cron as sugar, converts server-side); at-least-once semantics with next_run_at always advancing; a reconciler for crashed-mid-flight runs. No per-agent dispatch code — reuse the same spawn path the desktop uses.

SDK + MCP server over the daemon API

AO has a CLI but no SDK and no MCP server. Both are thin: an MCP server maps 1:1 onto the existing REST routes (workspaces.create, agents.create, terminals.read/send, ...), and an SDK is the same typed client already generated (frontend/src/api/schema.ts) published for external use. Unlocks "agents drive AO," which is the product's thesis.

Listening-port auto-detection for the browser preview

Today ao preview is explicit: static index.html, a given URL, or a .ao/launch.json dev server. It does not discover a port the agent's already-running dev server just bound. A daemon-side process-tree→port scanner (lsof/procfs shape) feeding a "Detected ports" list would remove the "you must know the port" step.

Terminal splits + presets

Tabs already exist (ShellTerminalsView.tsx / ShellTerminalTab.tsx). Split panes and saved layouts (reopen an agent+shell arrangement with one keystroke) do not.

Per-project setup/teardown scripts

.ao/launch.json covers dev-server run config; there are no setup/teardown hooks (env setup, dep install) run per workspace on spawn/cleanup.

Open-in-IDE handoff

Already tracked (#3118); spec-level details (per-file + line-accurate open, cross-platform detection/launch commands, scratch-session path resolution, main-process launch boundary) added as a comment there.

Task board + Linear sync

Tracker intake exists (observe/trackerintake/observer.go auto-spawns one worker per eligible issue). What's missing is a human-facing board (statuses/priorities/assignees/due dates) and two-way Linear sync. This may be a deliberate product-direction difference, but the board is a concrete candidate if task tracking becomes a need.

Process & tooling

• Planning template (docs/plans/): Context → Goals/Non-Goals → Schema → API surface → "holes vs real code" table → Phases → Key Invariants → Open Questions → Verification checklist → Critical File Paths. AO has the seed (4 files); adopting this shape per-feature pays compounding dividends.
• CLI output contract: --json (default when driven by an agent) / --quiet (IDs only) / a per-command display() + a table() helper — makes the CLI machine-drivable. Plus an interactive arrow-key help browser (zero-dep raw-mode ANSI) for bare ao on a TTY.
• Ratchet tests: source-level grep tests that fail when a forbidden pattern reappears — e.g. "no base64 / per-chunk utf8 on the PTY data path," "no sync subprocesses in the serving loop." Cheap way to lock in the load-bearing rules in docs/architecture.md.

What I'd do first (value / effort)

1. Port auto-detection for the preview — small, high UX value.
2. Diff-panel syntax highlighting via HighlightedCode — small, already have the engine.
3. Automations — the biggest genuinely-missing feature.
4. SDK + MCP server — thin over the existing API, unlocks agents-driving-AO.
5. Adopt the planning template — process change, zero code.
6. Ratchet tests — lock in the rules with cheap tests.

Agent integration: agents as data, not code

A comparable codebase models each agent as a declarative manifest, not a hand-written adapter:

• source: builtin | user — a user source means custom agents with zero code change.
• kind: terminal | chat.
• command + promptCommand + resumeCommand (id-based resume) + nonInteractiveCommand (one-shot headless run, locked down: read-only/plan modes, default-deny sandboxes).
• taskPromptTemplate + contextPromptTemplate{System,User} (Mustache templates) — the system context (AGENTS.md, stable config) and per-launch context (user prompt, linked issues/PRs, attachments) are composed separately and cacheably.

AO has 26 code adapters; the definitional surface lives in Go. Actionable: extract a declarative agent-definition schema (even if it compiles down to the existing ports adapter interface) so a new CLI agent is a JSON/YAML entry — command, prompt/resume/headless commands, templates — not a new adapter. The nonInteractiveCommand concept (headless, read-only, default-deny) also maps directly onto AO's automations and the reviewer adapters, which today carry hand-written "experimental, user-approved" flags instead of a declared headless mode.

Built-in skills: teach agents to drive AO

A comparable codebase ships a product skill pack that the agent auto-discovers, so the agent can operate the product itself:

• orchestrate — create workspaces, launch workers, read terminals, and track a coordinator table (Task / Dependencies / Workspace / Host / Terminal / Status / Result).
• automate — turn a recurring chore into a scheduled automation (draft prompt, confirm RRULE + target, create via CLI, review the first run).
• feedback — file structured feedback; doctor — diagnose a broken install; setup / standup / contribute — env / onboarding / contributing.

AO already has skill infrastructure (Pi), so this is content, not plumbing: an ao:*-style skill pack that teaches the agent to drive AO (spawn sessions, read terminals, schedule, file feedback) via the CLI/MCP. Pairs with the SDK/MCP finding above — the skills are the ergonomic layer on top of the machine surface.

Release engineering: channel-specific rolling pointers

A comparable codebase documents a footgun we share: GitHub's /releases/latest does not filter by tag prefix, so two release streams (desktop + CLI + canary) publishing to one repo means the next CLI release shadows desktop auto-update. Their fix is a channel taxonomy with rolling per-channel tags:

And --prerelease is reserved for actual canary builds, not used as a shadowing workaround. AO already has electron-updater + the "exactly one publisher" rule + artifact verification; if desktop canary/CLI streams ever coexist, adopting the rolling-pointer pattern now avoids the shadowing bug entirely.

Notifications: client-owned playback

A comparable codebase moved notification playback out of the backend/main process: the host only ingests normalized lifecycle events and broadcasts; the client resolves those to visible panes, decides suppression (already looking at it?), plays audio, shows OS notifications, and handles click-to-focus. Identity/status transitions are pure functions; the hook endpoint stays low-capability so a hook can't spoof system copy.

AO has dashboard notifications + Electron toasts, so this is a refinement, not a new feature: keep the daemon as the event source, move "mute / ringtone / suppress-if-focused / click-to-focus" into a small renderer-side controller with testable pure transitions. Concrete wins: no per-agent hook naming drift (Start / Stop / PermissionRequest normalized from many hook names), and no stuck transient statuses when a terminal/session exits.

Chat drivers: per-harness best input, not one wire format

A comparable codebase made a decision worth surfacing for AO's Chat stack: drive Claude Code via its agent SDK directly, not through an ACP bridge. The reasoning:

• ACP flattens the harness's vocabulary down to ACP v1 — losing tool_use_result real output objects, canUseTool option titles, parent_tool_use_id subagent nesting, and the 18-value terminal reasons.
• The vendor SDK gives the full typed message union in-process; the harness's own process model stays theirs, and there's no extra JSON-RPC subprocess in between.
• ACP is kept as the generic adapter (for a harness that is natively ACP), under the same seam.

AO's Chat drivers use ACP (claude-agent-acp, codex app-server). Actionable: keep ACP as the generic seam, but consider an SDK-direct adapter for Claude (or Codex) where fidelity loss matters, behind the same adapter interface. Two smaller patterns worth stealing regardless:

1. Parse every adapter emission at the boundary — fail loudly in the driver, not in the renderer.
2. Keep adapters dumb about infrastructure — start / prompt / cancelTurn / respondToApproval / setMode / dispose, emitting only item / delta / turn / session events.

Spawn context composition: stable system vs per-launch user

A comparable codebase composes launch context from heterogeneous sources (user prompt, linked issues/PRs/tasks, attachments, agent instructions) into a LaunchContext → buildLaunchSpec → executeAgentLaunch, with a system/user split + cache hint: stable context (AGENTS.md, repo docs) goes in cacheable system blocks; per-launch content (prompt, linked work) goes in the user message. Sources are declared (user-prompt / github-issue / github-pr / task / attachment / agent-instructions), each with displayName + description + required query.

AO's spawn config is a flat prompt + project config. Actionable: a structured launch-context composition (declared sources, stable-vs-per-launch split, cacheable system block) would make the existing agent-instructions/rules config composable instead of a concatenated string, and would slot directly into the automations + reviewer launch paths (same builder everywhere — matching the "one AgentLaunchRequest builder" idea above).

Shared AI component library

A comparable codebase centralizes its chat/tool-call UI primitives into a shared component set (file-diff block, code block, clickable file path, read-file tool, show-code), shared across desktop / web / mobile. AO hand-builds these per surface (ChatTimelineItems, SessionFilesView, ...). Actionable: extract the stable chat primitives (tool-call card, code block, file-diff block, file-path chip) into a shared package so desktop and mobile stop re-implementing the same rendering. Lower priority than the capabilities above, but it compounds as more tool cards are added.

Custom themes: import / export / editor

AO ships a light/dark/system style + preference selector (GeneralSettingsSection.tsx, site-theme/tokens.css). A comparable codebase treats themes as user-authored files — build / edit / import / export, with a CLI (settings theme get/set/list/import/export/remove). Small gap, but "import a theme file" is cheap and unlocks a community-themes path later.

Slack / Linear → workspace triggers

A comparable codebase spins up workspaces from a Slack message or Linear issue. AO has no third-party trigger surfaces. Lower priority than automations/SDK, but it pairs with the MCP surface: the same create-workspace primitive, exposed to an integration.

Deliberate divergences — do NOT copy

For balance, these looked attractive but are wrong for AO's constraints (or already solved differently):

• Cloud / remote workspaces, relay, orgs/teams, multi-tenant Postgres/Neon/Electric. A comparable product is cloud-capable and multi-tenant; AO is deliberately single-user local with a loopback daemon. Copying the relay/org layer would violate AO's core rules.
• tRPC end-to-end types. That's a TypeScript-everywhere answer; AO's Go daemon can't use it, and the OpenAPI codegen (npm run api → frontend/src/api/schema.ts) already delivers the equivalent contract + drift-checking in CI.
• Bun / Turborepo / Biome. A function of being an all-TS monorepo; AO's go build / go test -race + npm run lint is the equivalent.
• "Any terminal agent" breadth over structured Chat depth. A real tradeoff: terminal-first gets breadth cheaply; AO's ACP Chat drivers + TUI↔Chat handoff + approval model gets depth. Don't regress Chat depth to chase breadth — the findings above (agent-as-data for custom agents, SDK-direct for fidelity) capture the good parts without the regression.

Lifecycle: delete as a saga + archived history + a status board

Two concrete lifecycle patterns worth stealing:

Delete as a saga with a single commit point. A comparable codebase orders workspace delete as: (0) preflight — reversible checks (git clean? throw CONFLICT before touching state); (1) teardown script; (2) commit point — the authoritative delete (in AO's case, the SQLite row); (3) local cleanup — kill PTYs, git worktree remove --force, git branch -D, drop the row — best-effort, every failure a warning. Everything before the commit point is reversible; after it, orphans are cheap and a future sweeper cleans them. The phases stay separate in code so a future change (retry, reconcile, tombstone) lands at one seam. This maps directly onto AO's session/worktree teardown (which already has conservative guardrails like "never force-delete dirty worktrees") and gives a clean ordering discipline.

Archived (soft-delete) history + a status board. Instead of hard-deleting, archive the row (archivedAt + archiveReason) so merged/deleted workspaces remain as history, and render a Kanban grouped by derived status (Idle / Working / Needs attention / Needs review / Merged / Deleted) with URL-synced filters. AO already derives status from durable facts (its load-bearing rule) — so a board-by-derived-status is a natural fit, and the archived-history column ("what merged this week", "what got deleted") is a concrete gap. Session tombstones + a status-column board would make the existing derived-status pipeline visible as a review surface, not just a sidebar badge.

SDK: generate it from the existing OpenAPI spec

The SDK finding above has a proven implementation path: the comparable codebase's SDK is generated from its OpenAPI spec (resource classes + an APIPromise Promise-subclass that lazily parses responses). AO already produces an OpenAPI spec (npm run api → backend/internal/httpd/apispec/openapi.yaml) and a typed client (frontend/src/api/schema.ts). A published SDK is the same artifact, generated for external consumption — no hand-written client to maintain. The APIPromise "Promise subclass that parses lazily + _thenUnwrap for typed transforms" pattern is a nice-to-have, not a requirement.

Process: inventory consumers before moving a source of truth

A comparable codebase, before moving its workspace table from cloud to host-local, wrote a usage inventory (who actually reads this table?) and discovered the cloud list endpoint had zero real consumers — mobile/api never read it, and several columns were write-only tags never read anywhere. That let them delete the cloud path outright instead of maintaining a sync layer. AO's equivalent: before adding or migrating a storage surface, enumerate the actual readers/writers first. The "Tracker lane exists but nothing consumes it" note in STATUS.md is exactly the class of thing this catches early.

Daemon single-flight: adopt instead of spawn

A comparable codebase hit the exact "two app instances on one machine" problem AO tracks in #3805 (concurrent daemons reconciling the same data dir), and solved it with a pattern worth porting:

• Manifest file written by the child once listening: { pid, endpoint, authToken, startedAt } — everything needed to adopt it. (AO's ~/.ao/running.json PID+port handshake is already this shape.)
• Atomic cross-process spawn lock (exclusive-create lockfile) held only during spawn + health-check — the second instance waits, then adopts the healthy daemon instead of racing to spawn a duplicate.
• Health check by endpoint + secret before adopt.
• Ownership-aware teardown: never kill or de-manifest a daemon another live instance spawned.

Net effect: one daemon per data dir across stable/canary/dev instances, no WAL/socket contention, no mutual reap. AO already has daemon-owner.ts (attach vs re-link); the missing pieces are the atomic lockfile and "adopt by manifest instead of spawn" semantics.

Optimistic delete

Deleting a workspace/session feels slow because the UI waits for the whole teardown (kill terminals → teardown script → git worktree remove --force → DB cleanup, which can take seconds). A comparable codebase makes the row disappear immediately (optimistic update) and runs deletion in the background, rolling back the optimistic removal on failure. AO's session/worktree delete would benefit from the same: hide the row, run the saga in the background, restore it on error. Pairs with the "delete as a saga" finding above.

Terminal-agent binding: which agent is alive in which terminal

A comparable codebase tracks, in-memory, which agent is currently alive in which terminal via a tiny store: one binding per terminalId (agent swap overwrites), delete on exit (absence = the only signal), tie-break by lastEventAt. AO has the richer agent adapters + the TerminalSwitchAgentButton surface, so the shape is familiar — the note is the simplicity: no DB, no migration, primitives only (findActive + getOrCreate), callers compose with the existing terminal write path. Useful if the "which agent is this pane" question ever needs a cheap, correct answer without a schema.

Process: a canonical ticket/issue template

A comparable repo codifies a three-section ticket format: Context (2–4 outcome-focused sentences) / References (source, who, link, date) / Implementation notes (Files path:line + why, Approach paragraph, Related code, Gotchas). The "implementation notes" section is deliberately agent-groomed and left empty until a grooming pass. AO's bug-triage skill has a shape; standardizing the ticket template across issues would make them uniformly triage-ready.

Chat transcript invariants worth adopting

A comparable codebase's chat protocol is built on a small set of invariants that outrank convenience; several are worth checking against AO's conversation model:

• Full-snapshot upsert, no patch format — every durable event carries the complete item; the only client mutation is items.set(id, item). This single decision makes reconnect, replay, and multi-client trivial.
• One pure reducer serves live streaming, reconnect replay, and history pagination — so the three paths can't drift.
• Deltas are optional and droppable — a client that ignores deltas still converges via snapshots; the final snapshot is authoritative over accumulated deltas.
• The transcript never shrinks — edit-a-past-message forks a session (forkedFromSessionId), it never truncates.
• declined / canceled / stale are statuses, not errors — a refused tool renders as a normal settled row; a lost approval marks stale instead of hanging forever.
• Queued prompts — a prompt arriving mid-turn is queued FIFO and delivered at the turn boundary; the session stays running until the queue drains, making "infer idle from absence of a running turn" unrepresentable.
• In-band stop — cancel is a command, never a socket close; a dropped socket means nothing about user intent.

AO already has compaction, rollback, and controller-generation fencing, so this is a vocabulary/consistency checklist more than new machinery — but the "full snapshot + one reducer + droppable deltas" trinity is the strongest single idea to adopt for any new streamed surface.

Meta-skills for the agent's own workflow

A comparable repo ships skills that shape how the agent works with the human, not what the product does: a decide skill that walks the user through decisions one at a time (context → trade-off → one mutually-exclusive question → "Logged: <decision>"), and a redesign skill for reviewing completed code one change at a time. Each ends in a | # | Decision | Choice | table. This is orthogonal to the ao:* product-skill pack above — it's a lightweight way to make interactive decision-making deterministic and logged, which any of AO's agent workflows could reuse.

Shell readiness: byte-level prompt detection (OSC 133)

A comparable codebase detects "the shell is at a prompt, ready for input" with a byte-level scanner for the OSC 133;A semantic-prompt marker (\x1b]133;A ... \x07, the FinalTerm standard) that zsh/bash/fish wrappers inject. Key properties:

• Pure-ASCII marker → byte-level matching, no per-chunk UTF-8 decode hop; output stays opaque bytes end-to-end.
• Holds matching bytes back during a partial match, flushes them on mismatch (a stray \x1b] never eats output), and handles the marker spanning chunk boundaries.

Why this matters for AO: activity_state already distinguishes waiting_input (empty prompt) from blocked (pending approval), and the TUI send path must know when the agent is actually at a prompt. Heuristic idle detection can misread a long-running silent command as "ready". A shell-injected OSC 133 marker is a reliable, byte-level signal — the same class of signal AO already uses for hooks, but for the prompt itself. Worth considering as a richer input to the lifecycle reducer's activity state.

Pane/split data model (the implementation detail behind splits)

The splits finding above has a concrete, proven data model:

• Flatten the tree to Tab → Split → Pane (no tabs-within-tabs). The layout tree holds only paneId strings; pane data lives in a flat map keyed by id.
• N-ary splits with relative weights, not percentages — [1,1,1] = thirds, [3,2] = 60/40. Weights don't sum to anything; CSS flex-grow: weight renders them directly; resize converts pixels→weights (only the two adjacent panes change); "equalize" is just set-all-to-1. Sidesteps the 33.33 + 33.33 + 33.34 rounding problem entirely.
• Derived titles via a registry getTitle(context) + optional titleOverride.
• pinned flag for preview/replace semantics: unpinned panes (e.g. single-click file preview) are replaced in-place; pinning (double-click/edit) makes them persist.

This is the whole "splits done right" recipe — adopt it if/when AO adds split panes.

Chat delta coalescing

A comparable chat runtime batches streamed deltas per session through a Coalescer (flush on a cadence, dispose on unsubscribe) so the wire carries ≤N frames/sec per session rather than one frame per emit. AO's chat streams via SSE; the same per-session coalescing (batch deltas, flush on interval, drop-only-if-coalesced) is a cheap backpressure win for chatty tool output, and it's independent of the "full snapshot is authoritative" invariant above.

Daemon diagnostics: probe for a degraded macOS trustd bootstrap

A comparable codebase ships a trustd-probe because a degraded Mach bootstrap (after logout/login, or an updater relaunch) makes Go binaries fail with x509: OSStatus -26276 and headless Chromium abort with bootstrap_check_in error 141 — while Node/curl succeed (they use their own TLS stack), so the failure is invisible to most tooling. The reliable probe is security verify-cert (exercises the platform verifier). The probe fails open (healthy) so it never triggers a session-destroying respawn, but logs inconclusive results loudly.

AO is a Go daemon that shells out to gh and runs Chromium — the exact surface this bites. Adding a trustd check to ao doctor (or the daemon's readiness path) would turn an opaque "gh fails after logout/login" into an explicit diagnostic.

MCP server design (when it's built)

The SDK/MCP finding above has a concrete, current design worth copying when AO builds its MCP server:

• Tool surface: ~30 flat tools across agents / automations / hosts / projects / tasks / terminals / workspaces — one tool per REST route.
• Tasks extension (io.modelcontextprotocol/tasks) is the ergonomics centerpiece: agents_run / automations_run return a task handle; clients drive tasks/get (status + transcript) and tasks/update (follow-up input) instead of polling terminals_read. That turns "run an agent" from a blocking call into an addressable, steppable handle.
• MRTR (input_required) for mid-call confirmation on destructive verbs (workspaces_delete, automations_delete) and host disambiguation — no bidirectional stream needed.
• Stateless Streamable HTTP, tools-only capabilities, no session state.
• Cacheable list results (ttlMs) with a long TTL for the static tool catalog, so client prompt caches stay stable across reconnects.

Onboarding: a usage-audit skill (the "10x" pattern)

A comparable codebase ships a skill that does a read-only usage audit — runs list on every subsystem in parallel, tolerating failures — then presents a scorecard of the 3–5 highest-impact features the user isn't using (one-line payoff each), and walks through setting each one up one at a time (pitch → "Set up now / Tell me more / Skip", actually doing the setup after confirmation, never printing instructions as a substitute). This is a strong onboarding/activation loop for AO: audit the user's sessions/automations/skills usage and walk them into the features they're missing — an ao:* skill, not new product surface.

Diagnostic + feedback skills

Two more skill patterns worth folding into AO's agent workflows:

• doctor — "diagnose first, change one thing at a time, verify after each change": snapshot (read-only, parallel, tolerate failures) → match a known-signatures table (symptom → fix) → propose the fix and get go-ahead → verify by re-running the failing action → escalate with evidence. Complements AO's existing ao doctor binary with a walkthrough layer.
• feedback — classify (bug / feature / general), draft a title + "what happened / what you want" in the user's voice, offer screenshot/diagnostics, and never include repo contents, terminal output, or logs without explicit consent. Refines AO's bug-triage skill with the consent boundary.

Agent permission defaults: never ship YOLO flags as defaults; migrate on hardening

A comparable codebase's own migration code exposes a real safety lesson: their old built-in agent defaults were claude --dangerously-skip-permissions, codex --dangerously-bypass-approvals-and-sandbox, gemini --yolo, copilot --allow-all, cursor-agent --yolo. They later swapped in safer defaults and wrote a one-time migration to backfill the safer values for users already exposed to the old ones.

AO's reviewer adapters are in exactly this territory (STATUS.md: some reviewers "retain their native approval prompts instead of receiving broad unattended flags"). The lesson, applied: (1) never make a broad permission-bypass flag the default launch command; (2) when a default is hardened, ship a migration that rewrites existing user configs off the unsafe value, not just the new default for fresh installs; (3) keep the legacy unsafe command strings as a frozen reference (like their LEGACY_BUILTIN_TERMINAL_AGENT_OVERRIDES table) so the migration is auditable.