Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Replace job-iteration with ActiveJob::Continuable#1461

Open
kwent wants to merge 3 commits into
Shopify:main from
kwent:features/custom-csv-parsing
Open

Replace job-iteration with ActiveJob::Continuable #1461
kwent wants to merge 3 commits into
Shopify:main from
kwent:features/custom-csv-parsing

Conversation

@kwent

@kwent kwent commented Apr 28, 2026

Copy link
Copy Markdown

Summary

Closes #1398.

Drop the job-iteration gem dependency in favor of Rails 8.1's built-in ActiveJob::Continuable for cursor-based iteration and resumption. This is a new minor version that bumps the minimum Rails to 8.1 while preserving all existing behavior.

What changed

  • Removed job-iteration dependency, replaced include JobIteration::Iteration with include ActiveJob::Continuable
  • Rewrote TaskJobConcern to use a single Continuable step :iterate block that handles all collection types
  • Added 6 custom enumerator classes replacing job-iteration's enumerator builders:
    • ActiveRecordRecordEnumerator — cursor-based AR record iteration with multi-column/composite PK support
    • ActiveRecordBatchEnumerator — cursor-based AR batch iteration
    • ArrayEnumerator, CsvRowEnumerator, CsvBatchEnumerator, OnceEnumerator
  • Added MaintenanceTasks.max_job_runtime config (default 5 minutes) replacing JobIteration.max_job_runtime
  • Two interruption paths preserved:
    • Queue stopping → Continuable's Interrupt exception → auto re-enqueue via retry_job
    • App stopping (pause/cancel) → @run.stopping? check → manual state persistence
  • Bumped minimum Rails from 7.1 to 8.1
  • Removed Rails 7.x/8.0 CI gemfiles
  • Updated README references

Key design insight

ActiveJob::Continuation::Interrupt inherits from Exception (not StandardError), so the existing rescue_from StandardError error handling flows through cleanly without conflicting with Continuable's interrupt mechanism.

checkpoint! override

Continuable's checkpoint! directly checks queue_adapter.stopping?. To support max_job_runtime, we override checkpoint! on the class level (inside the included block) to also check elapsed time. This is necessary because ActiveJob::Continuable sits above TaskJobConcern in the MRO when included in the included block.

Test plan

  • All 288 existing unit/integration tests pass
  • RuboCop clean (0 offenses)
  • System tests pass
  • Verify pause/resume cycle works end-to-end
  • Verify CSV task with cursor resumption
  • Verify multi-column cursor AR tasks
  • Verify throttled tasks back off and re-enqueue

kwent commented Apr 28, 2026

Copy link
Copy Markdown
Author

I have signed the CLA!

github-actions[bot] reacted with thumbs down emoji github-actions[bot] reacted with eyes emoji

1 similar comment

kwent commented Apr 28, 2026

Copy link
Copy Markdown
Author

I have signed the CLA!

github-actions[bot] reacted with thumbs up emoji github-actions[bot] reacted with eyes emoji

kwent added 3 commits May 11, 2026 18:04
Drop the job-iteration gem dependency in favor of Rails 8.1's built-in
ActiveJob::Continuable for cursor-based iteration and resumption.
- Bump minimum Rails to 8.1, remove job-iteration dependency
- Rewrite TaskJobConcern to use a single Continuable `step :iterate`
- Add custom enumerator classes (ActiveRecordRecordEnumerator,
 ActiveRecordBatchEnumerator, ArrayEnumerator, CsvRowEnumerator,
 CsvBatchEnumerator, OnceEnumerator) replacing job-iteration's
 enumerator builders
- Override checkpoint! for time-based max_job_runtime interruption
- Handle two interruption paths: queue stopping (Continuable Interrupt)
 and app stopping (pause/cancel via Run status)
- Add MaintenanceTasks.max_job_runtime config (replaces
 JobIteration.max_job_runtime)
- Update CI matrix to drop Rails 7.x/8.0 gemfiles
- Update README references from job-iteration to Continuable
Closes Shopify#1398 
Verify Continuable actually re-enqueues via retry_job when interrupted.
- Extract ActiveRecordCursor module shared by both AR enumerators
 (eliminates 4 duplicated private methods)
- Cache ordered scope in build_scope to avoid rebuilding Arel per batch
- @collection_enum ivar → local var (only used within step block)
- check_throttle: use find idiom instead of each + early return
@kwent kwent force-pushed the features/custom-csv-parsing branch from 79c7208 to a50ed28 Compare May 12, 2026 01:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Consider supporting ActiveJob::Continuation as an alternative to job-iteration

1 participant

AltStyle によって変換されたページ (->オリジナル) /