-
Notifications
You must be signed in to change notification settings - Fork 162
fix(ci): run the full suite on stacked PRs, not just PRs into main - #3372
fix(ci): run the full suite on stacked PRs, not just PRs into main #3372kovtcharov-amd wants to merge 1 commit into
Conversation
`on.pull_request.branches` matches the PR's BASE branch, not its head, so `branches: [ main ]` meant "only when merging into main". Every PR stacked on a feature branch fell outside the filter and the jobs were never created — nothing failed, nothing showed as skipped, the checks page was simply missing them. #2599 merged email-agent changes with zero email tests, zero unit tests and zero lint behind a green page. Drops the filter from the 34 workflows that carried it. `paths:` filters and `push:` triggers are untouched, so a workflow still only runs for the code it covers and still only runs on main for pushes. `util/check_workflow_triggers.py` (wired into `lint.py --all`) fails the build if the filter comes back; lint.yml now watches `.github/workflows/**` so a workflows-only PR actually runs that gate.
Verdict: Approve with suggestions
This removes the base-branch filter that was silently skipping CI on every stacked PR, and adds a lint gate so the filter can't come back. It's a real fix to a real hole, it's tested, and I reproduced the before/after myself — merge it.
One thing worth doing before or right after merge: the new guard only watches the ordinary pull-request trigger, not the privileged variant that the PR-review bot itself runs on. If someone adds the same filter there, reviews would quietly stop appearing on stacked PRs — the exact failure this PR exists to prevent, in the one workflow nobody would notice going quiet. It's a small addition to the checker.
Two smaller notes: one comment in the eval workflow still describes the filter that was just deleted, and the heavy self-hosted eval queue will now see traffic from stacked PRs too (the draft guard already absorbs most of that, so this is a heads-up rather than a problem).
Real-world evidence
No evidence-bundle.md was produced for this run, and this change touches no user-facing GAIA surface — it's CI config, a dev lint check, tests, and docs — so no screenshot or agent evidence applies. I verified the PR's own claims first-hand instead of relying on the description:
$ python util/check_workflow_triggers.py
[OK] 72 workflow pull_request triggers validated.
$ python -m pytest tests/unit/test_check_workflow_triggers.py -q
12 passed
The stacked-PR A/B in the test plan is real and holds up. Both throwaway PRs exist on the same base branch, and the check counts today are even more decisive than the description claims:
$ gh pr checks 3303 # control, no fix
10 checks — build, label, auto-merge, and Claude bot jobs only.
No lint, no unit tests, no email suite.
$ gh pr checks 3304 # treatment, this fix
72 checks — including Dependency Review, Email Agent Unit Tests,
RAG Unit Tests, Skill Audit, Example Agents Unit Tests, Audit Dependencies.
That is the bug and the fix, demonstrated on live PRs. The evidence supports the approve verdict.
🔍 Technical details
Issues
🟡 The guard doesn't cover pull_request_target.branches (util/check_workflow_triggers.py:651)
_pull_request_trigger() only reads triggers.get("pull_request"). pull_request_target has identical base-branch semantics, and .github/workflows/claude.yml — the PR review/re-review bot — is driven entirely by it (claude.yml:9). A future branches: [ main ] there would silence PR review on stacked PRs with a green checks page, which is #2767 recurring in the workflow least likely to be missed quickly. auto-label.yml:7 is the other pull_request_target consumer. Neither carries the filter today, so this is a hole in the regression guard, not a live bug.
branches-ignore is the same key in mirror form and is also unchecked (no current uses).
Shape of the fix — parameterize the helper and loop over both events:
PR_EVENTS = ("pull_request", "pull_request_target") FILTER_KEYS = ("branches", "branches-ignore") def _trigger(workflow: Any, event: str) -> Dict[str, Any] | None: if not isinstance(workflow, dict): return None triggers = workflow.get("on", workflow.get(True)) if not isinstance(triggers, dict): return None trigger = triggers.get(event) return trigger if isinstance(trigger, dict) else None # in run_check(): for event in PR_EVENTS: trigger = _trigger(workflow, event) if trigger is None: continue for key in FILTER_KEYS: if key in trigger: errors.append(f"{path.name}: `on.{event}.{key}: {trigger[key]}` ...")
Worth adding TestRejected cases for pull_request_target + branches and for branches-ignore alongside the existing five/six.
🟢 Stale comment describes the filter this PR deleted (.github/workflows/test_eval_agent_gemma_consolidation.yml:150)
# path below. The `types:` list follows test_email_agent_eval.yml;
# `ready_for_review` is what runs the gate on a PR that was opened as a
# draft (see the draft clause in the job `if:`).
🟢 Self-hosted eval queue now takes stacked-PR traffic (.github/workflows/test_eval_agent_gemma_consolidation.yml:179)
That workflow uses concurrency: group: lemonade-eval with cancel-in-progress: false on a single-slot [self-hosted, Windows, strix-halo, lemonade-eval] pool, and its paths: include hub/agents/*/python/**. Stacked PRs previously couldn't enqueue at all; now they can, and nothing cancels a superseded run. The draft == false guard at line 239 keeps most of the 42 open PRs out, and test_sd/test_npu_embedder/test_lemonade_server/build-electron-apps carry the same guard — so this is likely fine, just worth watching queue depth after merge rather than changing anything now.
Strengths
- The guard is the point, and it's wired correctly. Widening
lint.yml'spathsto.github/workflows/**(bothpush:andpull_request:) is the non-obvious half — without it the new check would be invisible on exactly the PRs that trip it. The inline comment says why, next to thecheck_dependabot.py/dependabot.ymlprecedent it mirrors. check_workflow_triggers()inutil/lint.py:806mirrorscheck_dependabot()line-for-line — same nested-import fallback, sameCheckResultshape, same failure message pointing at the issue. The[N/11]→[N/12]renumber is complete across all twelve.- Tests cover the traps that actually bite here:
"on"resolving to booleanTrueunder YAML 1.1, barepull_request:parsing toNone,push:filters deliberately left alone, and aTestRealRepositorycase that runs the check against the live.github/workflows/. The unparseable-file and missing-directory cases both return1rather than passing vacuously, which is what keeps the guard from being a no-op. - Scope is clean. Every hunk in the 34 workflows is a one-line replacement; no
paths:filter moved, nopush:trigger touched, and the replacement comment carries the issue number so the next person doesn't re-add it.
Before: a pull request based on any branch other than
mainran no unit tests, no lint and no agent suites, and its checks page looked healthy the whole time — nothing failed, nothing showed as skipped, the jobs simply never existed. #2599 merged email-agent changes that way, with zero email tests, zero unit tests and zero lint behind a green page. After: those workflows run on every pull request regardless of base, so a stacked PR is held to the same bar as one opened againstmain. This matters right now because 42 open PRs are sitting on CI results that can't be trusted until it lands.The cause is that
on.pull_request.branchesmatches the PR's base branch, not its head, sobranches: [ main ]meant "only when merging into main". 34 workflows carried it, includinglint,test_unit,test_securityanddependency-review.Nothing is left main-only. Scoping stays where it belongs — every
paths:filter is preserved byte-for-byte, so a workflow still only runs for the code it covers, and the 5 workflows whosebranches:filter sits underpush:are untouched, since restricting pushes tomainis correct. A PR into thereleasebranch would now run CI too; that is the desired behaviour and costs nothing today, becausereleaseis force-pushed from a tag byupdate-release-branch.ymland has never once been a PR target. Excluding it would just recreate this bug for the first backport anyone opens.To stop it regressing,
util/check_workflow_triggers.pyfails the build if the filter reappears, andlint.ymlnow watches.github/workflows/**so a workflows-only PR actually runs that gate — without it the guard would itself be invisible on the change most likely to trip it.Fixes #2767
Test plan
Run Code Quality Checks,Unit Tests,Test Email Agent,Skill AuditandDependency Review. Both were drafts, so every job then short-circuited on its existingdraft == falseguard — the point is that the check runs exist at all, which in the control they do not.python util/check_workflow_triggers.pypasses on all 72 workflows, and fails with the file name and issue number when the filter is reintroduced.python -m pytest tests/unit/test_check_workflow_triggers.py— 12 passed.python util/lint.py— black, isort, flake8, security suppressions, agent conventions, dependabot, workflow triggers, doc versions all pass. (Pylint reports 9 pre-existingos.killpg/os.geteuidno-member errors on untouched files; those are Windows-only false positives and CI runs Linux.)yaml.safe_load; confirmed all 35push:triggers are still scoped tomainand nopaths:filter changed.