Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

feat(ws-worker): support multi-root @local adaptor resolution#1397

Merged
josephjclark merged 2 commits into
OpenFn:multi-adaptor-repos from
jeremi:feat/multi-root-local-adaptors
Jun 1, 2026
Merged

feat(ws-worker): support multi-root @local adaptor resolution #1397
josephjclark merged 2 commits into
OpenFn:multi-adaptor-repos from
jeremi:feat/multi-root-local-adaptors

Conversation

@jeremi

@jeremi jeremi commented May 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Lets @openfn/ws-worker resolve @local adaptors against a colon-separated OPENFN_ADAPTORS_REPO, so a private adaptor monorepo can be loaded alongside the canonical OpenFn adaptors monorepo without forking.

  • Single-path values keep working unchanged (a one-element root list).
  • Order is precedence: when more than one root contains packages/<shortName>/package.json, the earlier root wins.
  • Whitespace and empty segments in the colon list are trimmed.
  • When no root contains the adaptor, the worker falls back to the first root's resolved candidate path. The directory does not exist, so the runtime still surfaces a recognisable "missing local adaptor" path, and single-path callers see the same behaviour as before this PR (the path was returned without an existence check).

Companion to Lightning PR

This is the executor-side half of OpenFn/lightning#4714. Without this patch, colon-separated OPENFN_ADAPTORS_REPO only widens what Lightning's AdaptorRegistry exposes (picker UI, metadata, version validation); the worker still tries to load the literal joined path and dies with ENOENT. The two PRs ship together so multi-root @Local execution works end-to-end.

The first-wins semantics here mirror Lightning's AdaptorRegistry: a private adaptor repo listed before the canonical OpenFn monorepo will override individual adaptors locally, and the registry view and the worker's execution path stay consistent.

Test plan

  • pnpm --filter @openfn/ws-worker test — 30/30 convert-lightning-plan tests green, including 5 new cases:
    • resolves @local against a single existing root
    • walks colon-separated roots in order, first match wins
    • ignores roots that do not contain the adaptor
    • trims whitespace and drops empty segments
    • falls back to the first root when no root has the adaptor
  • pnpm --filter @openfn/ws-worker test:types clean
  • pnpm prettier --check clean for the touched files
  • Live verification inside a Lightning dev container with the companion Lightning PR applied: a workflow with @openfn/language-publicschema@local was resolved by the patched worker to /private-adaptors/packages/publicschema/dist/index.js (the first root in the colon list), and the runtime logged Resolved adaptor @openfn/language-publicschema to version 0.0.1.

Changeset

minor bump for @openfn/ws-worker.

 

OPENFN_ADAPTORS_REPO (and the --monorepo-dir / -m flag) now accept a
colon-separated list of monorepo roots. When a job pins an adaptor to
@Local, the worker walks the configured roots in order and resolves to
the first root whose `packages/<shortName>/package.json` exists. This
matches Lightning's AdaptorRegistry precedence so the registry view and
the worker's execution path agree on which root supplies a given
adaptor.
Single-path values keep behaving exactly as before. When no root
contains the adaptor the worker still surfaces a candidate path under
the first root, so the runtime emits a clean "missing adaptor" error
rather than crashing on a malformed colon-joined string.
This unblocks the multi-root flow on the Lightning side, where the
AdaptorRegistry already accepts the colon-separated form but the worker
was rejecting it with ENOENT on @Local execution.

@josephjclark josephjclark left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @jeremi, just taking a closer look at this. The solution is basically fine but it strikes me that this approach won't work for windows users, who have paths like c:/repo/adaptors.

I think a comma separated list will do just as well, and be a bit more accessible to other users

): { plan: ExecutionPlan; options: WorkerRunOptions; input: Lazy<State> } => {
const { collectionsVersion, monorepoPath } = options;

// monorepoPath is a colon-separated list of monorepo roots, mirroring how

@josephjclark josephjclark May 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The most obvious problem that strikes me here is that this will break on windows machines.

It's convenient that the colon approach mimics a unix path. And I'm not aware we have many windows users doing this stuff - although occasionally someone crops up.

Anyway I would prefer a comma or semi-colon separated list here so that it's safer

return candidate;
}
}
// Fall back to the first root's resolved candidate path. The directory

@josephjclark josephjclark May 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Way too many comments here, and in the wrong place. But I can tidy that up before merge to main. I presume these are AI generated?

import { Job } from '@openfn/lexicon';

// Builds a temporary monorepo root with package.json files in each named adaptor.
const makeMonorepo = (adaptors: string[]) => {

@josephjclark josephjclark May 11, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not wild about the file IO stuff in these tests to be honest. Mostly in this repo we use the mock-fs library to mock out the file system. If you're using an AI agent can you ask it to regenerate like that?

 

Colon collides with Windows drive letters (c:/repo); comma matches
Lightning's parsing of OPENFN_ADAPTORS_REPO. Single-path callers are
unchanged.

jeremi commented May 26, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @josephjclark — addressed in 66e889b:

  • Switched the split separator from : to , so Windows drive-letter paths like c:/repo stay usable; single-path callers (the common case) are unchanged.
  • Updated the comment block, CLI help text, changeset blurb, and the dedicated multi-root tests (titles + literals).
  • Companion change on the registry side: feat: support multiple local adaptor repos in registry lightning#4714

Ready for another look.

Copy link
Copy Markdown
Collaborator

Alright @jeremi this looks great.

One more issue I see: we'll also need to update the CLI so that it can support comma-separated paths on OPENFN_ADAPTORS_REPO. That need to be done before I can release this.

Are you able to take a look at that, or shall try to resolve it for you?

@josephjclark josephjclark changed the base branch from main to multi-adaptor-repos June 1, 2026 10:46
@josephjclark josephjclark merged commit ff1b1b6 into OpenFn:multi-adaptor-repos Jun 1, 2026
6 of 7 checks passed
josephjclark added a commit that referenced this pull request Jun 2, 2026 
* feat(ws-worker): support multi-root @Local adaptor resolution (#1397)
* feat(ws-worker): support multi-root @Local adaptor resolution
OPENFN_ADAPTORS_REPO (and the --monorepo-dir / -m flag) now accept a
colon-separated list of monorepo roots. When a job pins an adaptor to
@Local, the worker walks the configured roots in order and resolves to
the first root whose `packages/<shortName>/package.json` exists. This
matches Lightning's AdaptorRegistry precedence so the registry view and
the worker's execution path agree on which root supplies a given
adaptor.
Single-path values keep behaving exactly as before. When no root
contains the adaptor the worker still surfaces a candidate path under
the first root, so the runtime emits a clean "missing adaptor" error
rather than crashing on a malformed colon-joined string.
This unblocks the multi-root flow on the Lightning side, where the
AdaptorRegistry already accepts the colon-separated form but the worker
was rejecting it with ENOENT on @Local execution.
* fix(ws-worker): use comma to separate multi-root adaptor paths
Colon collides with Windows drive letters (c:/repo); comma matches
Lightning's parsing of OPENFN_ADAPTORS_REPO. Single-path callers are
unchanged.
* simplify changeset
* update cli
* versions
---------
Co-authored-by: Jeremi Joslin <jeremi@newlogic.com>
josephjclark added a commit that referenced this pull request Jun 9, 2026 
* feat(ws-worker): support multi-root @Local adaptor resolution (#1397)
* feat(ws-worker): support multi-root @Local adaptor resolution
OPENFN_ADAPTORS_REPO (and the --monorepo-dir / -m flag) now accept a
colon-separated list of monorepo roots. When a job pins an adaptor to
@Local, the worker walks the configured roots in order and resolves to
the first root whose `packages/<shortName>/package.json` exists. This
matches Lightning's AdaptorRegistry precedence so the registry view and
the worker's execution path agree on which root supplies a given
adaptor.
Single-path values keep behaving exactly as before. When no root
contains the adaptor the worker still surfaces a candidate path under
the first root, so the runtime emits a clean "missing adaptor" error
rather than crashing on a malformed colon-joined string.
This unblocks the multi-root flow on the Lightning side, where the
AdaptorRegistry already accepts the colon-separated form but the worker
was rejecting it with ENOENT on @Local execution.
* fix(ws-worker): use comma to separate multi-root adaptor paths
Colon collides with Windows drive letters (c:/repo); comma matches
Lightning's parsing of OPENFN_ADAPTORS_REPO. Single-path callers are
unchanged.
* simplify changeset
* update cli
* versions
---------
Co-authored-by: Jeremi Joslin <jeremi@newlogic.com>
josephjclark added a commit that referenced this pull request Jun 9, 2026 
* feat(ws-worker): support multi-root @Local adaptor resolution (#1397)
* feat(ws-worker): support multi-root @Local adaptor resolution
OPENFN_ADAPTORS_REPO (and the --monorepo-dir / -m flag) now accept a
colon-separated list of monorepo roots. When a job pins an adaptor to
@Local, the worker walks the configured roots in order and resolves to
the first root whose `packages/<shortName>/package.json` exists. This
matches Lightning's AdaptorRegistry precedence so the registry view and
the worker's execution path agree on which root supplies a given
adaptor.
Single-path values keep behaving exactly as before. When no root
contains the adaptor the worker still surfaces a candidate path under
the first root, so the runtime emits a clean "missing adaptor" error
rather than crashing on a malformed colon-joined string.
This unblocks the multi-root flow on the Lightning side, where the
AdaptorRegistry already accepts the colon-separated form but the worker
was rejecting it with ENOENT on @Local execution.
* fix(ws-worker): use comma to separate multi-root adaptor paths
Colon collides with Windows drive letters (c:/repo); comma matches
Lightning's parsing of OPENFN_ADAPTORS_REPO. Single-path callers are
unchanged.
* simplify changeset
* update cli
* versions
---------
Co-authored-by: Jeremi Joslin <jeremi@newlogic.com>

jeremi commented Jun 16, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @josephjclark !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@josephjclark josephjclark josephjclark approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

AltStyle によって変換されたページ (->オリジナル) /