Skip to content

Navigation Menu

Sign in
Sign up

[core] Allow custom primary-key compaction rewriters - #9762

Open
jordepic wants to merge 2 commits into
apache:master from
jordepic:compaction-rewriter-factory
Open

[core] Allow custom primary-key compaction rewriters #9762
jordepic wants to merge 2 commits into
apache:master from
jordepic:compaction-rewriter-factory

Conversation

@jordepic

@jordepic jordepic commented Sep 12, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Purpose

Primary-key compaction chooses its rewriter inside a private factory. An external engine currently has to reconstruct the writer setup to replace the file read/merge/write work with its own implementation.

Add TableWrite.withCompactRewriterFactory(...), backed by the existing CompactRewriter boundary. Each partition/bucket receives its own rewriter, and the factory receives Paimon's normal rewriter so implementations can delegate unsupported operations. Paimon retains input selection, background scheduling, and commit coordination. This PR adds the extension point; it does not add a native compaction engine.

write.withCompactRewriterFactory((partition, bucket, defaultRewriter) ->
 new CustomCompactRewriter(partition, bucket, defaultRewriter));

The API documents ownership and recovery: the returned rewriter owns the default, failed creation closes the default, and the factory must be installed before bucket creation and again on recovered writers. Write-only mode does not invoke the factory; append, postpone, and primary-key clustering writers reject it. The Java writing guide includes the integration contract.

The shared MinIO test image now uses quay.io/minio/minio with the existing release tag. Docker Hub returns pull-access-denied for the old image, causing the S3 setup failures on this PR and master. Quay is the registry shown in MinIO's container instructions.

cc @JingsongLi — feedback on exposing this boundary for external/native engines would be appreciated.

Tests

24 focused Core tests passed on JDK 11, with normal Checkstyle, Spotless, and enforcer checks enabled:

mvn -pl paimon-core -am -Pflink1 \
 -DwildcardSuites=none \
 -Dtest=CompactRewriterFactoryTest,MergeTreeCompactManagerFactoryTest,TableWriteTest,KeyValueFileStoreWriteTest \
 -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false test

Coverage includes real rewrites and metadata upgrades across reopened writers; fallback selection for none/input/lookup/full-compaction and deletion vectors; final table contents; rewriter closure; write-only behavior; unsupported append writers; late installation; and cleanup/suppressed exceptions when factory creation fails.

The failing S3FileIOTest also passes on JDK 11 against the Quay image: 21 tests, zero failures/errors/skips, with normal Checkstyle, Spotless, and enforcer checks enabled.

mvn -pl paimon-filesystems/paimon-s3-impl -am -Pflink1 \
 -DwildcardSuites=none -Dtest=S3FileIOTest \
 -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false test

Expose a per-bucket rewriter factory through the table write API so external engines can replace file rewriting while retaining Paimon compaction scheduling and commit coordination. Supply the default rewriter for selective fallback and define ownership, recovery, and initialization rules.
Validate real rewrites and upgrades across reopened writers, producer and deletion-vector fallback selection, closure, write-only behavior, unsupported append writers, late configuration, and failed factory cleanup. All 24 focused Core tests pass on JDK 11 with the normal build checks.

Copy link
Copy Markdown
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

AltStyle によって変換されたページ (->オリジナル) /