@josephjclark

The problem

Lightning and kit (worker + CLI) ship independently, but they share a contract - provisioner API, deploy v1/v2, the worker WebSocket protocol - that nothing currently exercises end-to-end against the heads of both repos.

Lightning's integration suite runs against a pinned @openfn/ws-worker and @openfn/cli. Kit's suite uses the in-repo lightning-mock. Both are useful, neither catches the case where a new worker changes a behaviour that real Lightning depends on.

The recent example: worker #1306 trimmed final_state from run:complete for single-leaf runs as an optimisation, while Lightning #4531 had separately added storage for final_dataclip_id without rewiring the sync-mode webhook consumer to fall back to it. Each PR was fine on its own; together, sync-mode webhook responses started coming back empty. Fixed in kit #1333 and pin-bumped in Lightning #4561. Lightning's existing tests would have caught it - they just weren't running against the worker that shipped.

This issue is about closing that pre-publish gap.

What we want

One suite of integration tests, in a standalone repo, run three ways:

1. test-repo ↔ production - the harness running against pinned versions on both sides.
2. kit main ↔ production Lightning - triggered from kit release tags (or main?).
3. Lightning main ↔ production CLI/worker - triggered from Lightning release tags (or again, main).

There's a working prototype at OpenFn/project-integration-tests covering project-merge stories. It has a runner abstraction that lets the same test file execute "do operation X" via either the CLI or a Lightning mix task - the load-bearing pattern that makes "tests as contract" work. The merge stories are a narrow start; this epic extends the harness to cover the run-execution path that actually broke.

Why a standalone repo and not a folder in either

• You can pin "suite vN" against arbitrary historical heads of kit and Lightning to prove regression windows. Hard to do cleanly when tests live in one of the trees.
• The contract belongs to neither side, so the tests probably shouldn't either.
• A single set of workflows that both repos call into, instead of two copies that drift.

A small downside/cost is that contributors occasionally hitting a CI failure on a repo they don't have checked out - is real but acceptable, provided the originating PR shows the failure loudly.

Deployment-time variant

The same suite can run against the actually-configured versions before a worker/Lightning pod rolls out, not just @latest. If we parameterise the workflows by image tag / npm version, this becomes a precondition we can run pre-rollout. Stretch goal for this epic.

But more importantly the test suite should be usable by anyone on their local machines allowing for bisecting or specific version targeting.

Scope

1. Harness foundations

   • Add worker-process support to the harness - a fixture that boots @openfn/ws-worker at a chosen ref (npm version, git sha, or local path) against the test's Lightning, waits for it to connect and start claiming, surfaces logs on failure, and tears down cleanly. Existing cli / lightning invocations stay as they are; this is a new component that runs alongside them so run-execution stories have a worker to dispatch to.
   • Parameterise workflows by kit ref, Lightning ref, and (stretch) npm/image version so all three run modes share one suite.
   • Cross-repo triggers from kit and Lightning release branches.

2. Canary sync test

   • One end-to-end story exercising the sync-mode webhook reply path: boot Lightning (via the existing setup), boot a worker against it (via the new fixture), POST to a webhook_reply: :after_completion workflow, assert on the response body rather than just the status.
   • Wired into both repos' release flows.
   • Pinning to the kit/Lightning refs from the recent regression should make it fail; pinning to the fix PRs should make it pass. That's the the 'bisect' test.

3. Lightning-side dev ergonomics

   • Default the Lightning test suite to install worker + CLI at @latest rather than hard-pinning in assets/package.json / install_runtime.ex. Pair with a lockfile so installs are reproducible.
   • Lightweight nudge (dev-startup check or git hook) when the installed @latest drifts from npm @latest, so the warning is local and contextual rather than CI noise.
   • Goal: devs working on anything that touches the worker/CLI surface are running a current version by default, without anyone having to remember to bump a pin.

4. Notification

   • Failures on the standalone repo need to surface on the originating PR/release. A status icon alone isn't enough.
   • Fallback if needed: post-tag job on each repo that refuses to release if the last cross-repo run failed (gh API query).

Not in this epic

• Lightning #4559 - decoupling handle_delayed_response from worker-supplied finalState. Removes this regression class entirely; complementary, not a substitute. Tracked there.

@doc-handoc-han on Jun 15, 2026

From my understanding, both repos have their own tests, but neither catches bugs that only appear when a real Lightning and a real worker/kit run together and this because none of the test suites run both simultaneously. So bugs that need both real repos to be caught get missed (main culprit here is kit running against a mock).

Proposal

A cross-repo integration test suite that runs a real Lightning instance against a real worker and asserts what external users would observe(Public APIs). After thinking through for a while, I think this should be achievalble without a separate integrations repo trying to hold the contract/test suites between lightning and kit/worker. We can actually have t...

View full comment