-
Notifications
You must be signed in to change notification settings - Fork 162
fix(release): make the v0.23.1 tag pass its own publish gates - #3354
fix(release): make the v0.23.1 tag pass its own publish gates #3354kovtcharov wants to merge 2 commits into
Conversation
Tagging v0.23.1 today fails publish.yml twice before it builds anything. The docs navbar still advertised v0.23.0 with Lemonade 11.5.0 while version.py declared 0.23.1/11.8.1, and publish.yml hard-fails a tag whose label does not name it. The release notes were missing the "What's New" section validate_release_notes.py requires, so that step exited 1 too. The notes also handed users two commands that do not exist in any GAIA binary: `gaia install gaia` is the Lemonade installer and rejects an agent id, and `gaia list` is not a subcommand at all. The working pair is `gaia hub install gaia --trust` / `gaia hub list`, which is what the TUI already tells users to run. Both validators only ever ran against a pushed tag, which is why a release PR could merge untaggable. tests/unit/test_docs_json_release.py now checks docs.json against version.py at PR time, and docs.yml runs the release-note validator for the declared version.
Verdict: Approve with suggestions
This makes the v0.23.1 tag pass the gates that would otherwise fail it mid-release, and fixes release notes that handed users two commands no GAIA binary accepts. Both corrections check out: the version/navbar/notes now agree, and the replacement commands match what the CLI actually parses and what the README already tells people to run.
One gap worth closing before merge, and it's small: the new docs.json guard is a unit test, and the unit-test workflow doesn't run on documentation-only changes. So the exact drift this PR is guarding against — someone editing the navbar label in a docs-only pull request — still slips through to the tag, which is the moment this PR is trying to stop being the moment of discovery. Adding the two documentation paths to the unit-test workflow's trigger list closes it; that file already does this for one other doc.
Two smaller things: the new "What's New" entry is a prose block where every other entry in the file is a single bullet, and it repeats the same fact the opening paragraph and the bug-fix entry already state — which also pushes the notes past the release process's own word budget for a patch release. And both new gates now fail loudly whenever the declared version has no notes file yet, which is intended for a release pull request but will also redden unrelated documentation changes if a version bump ever lands ahead of its notes.
Real-world evidence
No evidence bundle was produced for this run (evidence-bundle.md is absent) — this PR touches docs, CI, and a test, none of which is an Agent UI or inference surface, so nothing was deferred to the strix-halo lane. The PR description carries its own CLI transcripts, and I re-ran the two release gates in this checkout:
$ python util/validate_release_notes.py docs/releases/v0.23.1.mdx --tag v0.23.1
Validating docs/releases/v0.23.1.mdx...
✅ 1 file(s) validated successfully
exit=0
publish.yml's inline docs.json check, reproduced against this tree: navbar label v0.23.1 · Lemonade 11.8.1, release page releases/v0.23.1 present, notes file present — all pass. I also confirmed against the CLI's own parser that gaia list is not a subcommand and gaia install accepts no agent id, so the command correction in the notes is real and not cosmetic.
Not run here: pytest isn't installed in this environment, so I verified the five new assertions by reproducing their logic rather than executing them. The PR reports them passing.
🔍 Technical details
🟡 The new docs.json guard can't fire on the PR shape it most needs to catch
tests/unit/test_docs_json_release.py is the only check on docs.json ↔ version.py, but test_unit.yml's paths filter (.github/workflows/test_unit.yml:15-26 and 31-42) covers src/** and tests/** only — not docs/docs.json. docs.yml runs the notes validator but never touches docs.json.
Practical effect: a release PR is covered (it bumps src/gaia/version.py, so src/** matches). A docs-only PR that edits or drops the navbar label is not — and that lands on main untested until some later src/** PR fails a test whose subject it didn't touch, or until publish.yml fails against the tag. That workflow already carries precedent for exactly this: it lists docs/guides/custom-agent.mdx because a docs-only change can break a unit test. Same reasoning applies here:
# tests/unit/test_doc_examples_runnable.py execs this guide's own code # blocks, so a docs-only change can still break the guard. - 'docs/guides/custom-agent.mdx' # tests/unit/test_docs_json_release.py pins docs.json + the release notes # to src/gaia/version.py, so a docs-only change can break it too. - 'docs/docs.json' - 'docs/releases/**'
(needs adding to both the push and pull_request blocks).
🟢 ## What's New is a prose block where the house style is one bullet
docs/releases/v0.23.1.mdx:23-27. Every other entry in the file is a single bullet; this one is an H3 plus a paragraph, and it restates what line 8 and line 33 already say — third telling of the same fact, with gaia hub list named twice in one sentence. It also takes the narrative section to 358 words against the release skill's 350-word patch cap (awk '/^## (Bug Fixes|...)/{exit}' ... | wc -w).
The validator only needs the heading to exist, so a bullet satisfies it:
## What's New
- **The flagship GAIA agent installs from the terminal hub** — `gaia hub install gaia --trust` downloads it and starts its sidecar, and `gaia hub list` now shows it next to `email`.
That lands the section at ~310 words and drops the duplicate command mention.
🟢 Both new gates go red repo-wide if a version bump ever precedes its notes
.github/workflows/docs.yml:62-70 and test_release_notes_file_exists_for_current_version both key off the declared version, so version.py = 0.24.0 with no docs/releases/v0.24.0.mdx fails every docs PR and every unit-test run, not just the release PR. The current release flow bumps version and notes in one commit, so this is fine today — but the repo used to bump to a dev version separately (8aeefad6 "release: v0.17.3 notes + bump to 0.17.4 for development"). Worth a line in the release skill noting the bump and the notes must now land together, so a future maintainer doesn't rediscover it as mystery CI failures.
Strengths
- The unit test mirrors
publish.yml:126-168assertion-for-assertion (navbar label,releases/v<version>page, notes file), andtest_version_label_carries_both_versionscloses a hole the workflow itself has — two labels each half-right would pass the tag gate but ship a wrong navbar. - Failure messages name
publish.ymlas the downstream consequence, so whoever hits the red test knows what it is protecting rather than just what string mismatched. - The command correction is verified, not guessed:
gaia install(src/gaia/cli.py:3013) takes no positional agent id,listis absent from the top-level subparsers, andgaia hub install <id> --trust(src/gaia/cli.py:2689-2705) is the pairtui/README.md:111-113already documents. - Adding
util/validate_release_notes.pytodocs.yml's trigger paths means a change to the validator re-validates the current notes rather than going unexercised until a tag.
The guard added in the previous commit could not fire on the PR shape it exists to catch. test_unit.yml's paths filter covers src/** and tests/**, so a docs-only PR that edits or drops the navbar label never runs the test that pins it to version.py — which is exactly how v0.23.1 became untaggable. Add docs/docs.json and docs/releases/** to both trigger blocks, following the docs/guides/custom-agent.mdx precedent already in that file. The v0.23.1 "What's New" entry becomes a single bullet like every other entry in the file, dropping a third telling of what the opening paragraph and the bug-fix entry already say (narrative section: 358 -> 333 words, against the release skill's 350-word patch cap). Both new gates resolve the notes path from the declared __version__, so a bump that lands ahead of its notes now reddens every docs and unit run. The release skill gains a hard rule that the two must land together, rather than the gate being loosened to tolerate the gap.
kovtcharov
commented
Sep 4, 2026
All three addressed in ccfffaf8.
The guard now covers docs-only PRs. docs/docs.json and docs/releases/** are in both the push and pull_request blocks of test_unit.yml, commented the same way as the docs/guides/custom-agent.mdx line above them. You were right that this was the gap that mattered — the test I added was blind to the exact PR shape that produced the bug.
What's New is one bullet now. Took your suggested edit. The narrative section is 333 words against the 350-word patch cap.
On the version-bump question, I kept the gates strict and wrote the constraint down instead of loosening them. A bump with no notes is not a benign intermediate state — it is the untaggable state this PR exists to make loud, and CLAUDE.md's fail-loudly rule cuts against adding a tolerance to the check that catches it. The release skill's Phase 2 already commits version.py and the notes in one git add, so today's flow never hits it; what was missing was anyone saying why it has to stay that way. .claude/skills/gaia-release/SKILL.md now carries it as a hard rule naming the old separate-bump pattern and what breaks if it comes back.
What I consciously did not do: make test_release_notes_file_exists_for_current_version skip when the notes are absent. That would mean a bump could sit on main with no notes and nothing would say so until someone pushed the tag — the same late-discovery this PR is trying to end, just moved one step further out.
🔍 Technical details
Verified on ccfffaf8:
$ python -m pytest tests/unit/test_docs_json_release.py -q
5 passed in 3.10s
# regression probe, docs.json reverted via `git apply -R` (not stash):
FAILED test_navbar_label_matches_gaia_version
FAILED test_navbar_label_matches_lemonade_version
FAILED test_version_label_carries_both_versions
3 failed, 2 passed
$ python util/validate_release_notes.py docs/releases/v0.23.1.mdx --tag v0.23.1
✅ 1 file(s) validated successfully
$ awk '/^## (Bug Fixes|Known Issues|Contributors|Full Changelog)/{exit} {print}' \
docs/releases/v0.23.1.mdx | wc -w
333
$ python util/lint.py --all
Black PASS · isort PASS · Flake8 PASS · Pylint [X] 9 errors
The 9 Pylint errors are unchanged from the base: Windows-only os.killpg / os.getpgid / os.geteuid no-member false positives in daemon/sidecars/{ledger,manager}.py and installer/lemonade_installer.py, none of which this PR touches.
test_unit.yml's paths after the change, parsed from the file so both blocks are confirmed identical:
push: ['src/**', 'tests/**', 'util/check_pe_resources.py', 'installer/tui/**',
'setup.py', 'pyproject.toml', '.github/workflows/test_unit.yml',
'docs/guides/custom-agent.mdx', 'docs/docs.json', 'docs/releases/**']
pr: (identical)
Summary
v0.23.1cannot be tagged frommaintoday —publish.yml'svalidatejob rejects it twice before anything is built. This makes the tag pass, and adds the PR-time checks whose absence let the release PR merge untaggable.Why
The release PR bumped
version.pyto0.23.1/ Lemonade11.8.1but left the docs navbar readingv0.23.0 · Lemonade 11.5.0, and shipped release notes with no## What's Newsection. Both are hard gates inpublish.yml, and neither runs on a PR — they only fire against a pushed tag, so the first time anyone finds out is mid-release.The notes also hand new users two commands that do not exist in any GAIA binary:
gaia listis not a subcommand, andgaia installis the Lemonade installer, which rejects a trailing agent id. A user following the v0.23.1 notes to install the flagship agent gets an argparse error. The working commands aregaia hub install gaia --trust/gaia hub list— the same pair the TUI already tells users to run.Linked issue
Closes #3353
Refs #1128
Changes
publish.ymlunchanged.tests/unit/test_docs_json_release.pychecksdocs.jsonagainstversion.pyat PR time;docs.ymlruns the release-note validator for the declared version. These are the two gates that previously existed only behind a tag.Test plan
python util/validate_release_notes.py docs/releases/v0.23.1.mdx --tag v0.23.1→ exits 0 (was 1)python -m pytest tests/unit/test_docs_json_release.py -q→ 5 passed; revertingdocs.jsonalone fails 3 of thempublish.yml's inlinedocs.jsoncheck executed verbatim withTAG_NAME=v0.23.1→ exits 0docs.ymlstep extracted from the workflow and run against a simulated bump to0.24.0→ exits 1python util/lint.py --all→ Black / isort / Flake8 pass; the 9 Pylint errors are pre-existing Windows-onlyos.killpg/os.geteuidfalse positives in files this PR does not touch (identical count on the unmodified base)Evidence
CLI — the commands the notes told users to run (
.venv\Scripts\python.exe -m gaia.cli):Release-notes gate, before → after:
publish.yml's docs.json gate, run verbatim withTAG_NAME=v0.23.1:New PR-time gates:
Not in this PR
update-release-branch.ymlandrelease_components.ymlstill fire on the same tag independently of validation and approval —update-release-branch.ymlforce-pushesreleaseand exits 1 on pre-release tags;release_components.ymlredeploys the production Cloudflare Worker before any approval. Chaining those aftergithub-releaseis a separate PR.approve-publishhard-requiresemail-eval, which is currently failing on Anthropic billing (credit balance is too low). Even with these validators fixed the tag stalls before approval until that is cleared — an ops action, not a code change (eval: route remaining direct-Anthropic-API eval paths through Claude Code (subscription auth) #1344 ).Checklist