-
Notifications
You must be signed in to change notification settings - Fork 1.4k
[core] Allow custom primary-key compaction rewriters - #9762
Open
jordepic wants to merge 2 commits into
Open
Conversation
Expose a per-bucket rewriter factory through the table write API so external engines can replace file rewriting while retaining Paimon compaction scheduling and commit coordination. Supply the default rewriter for selective fallback and define ownership, recovery, and initialization rules. Validate real rewrites and upgrades across reopened writers, producer and deletion-vector fallback selection, closure, write-only behavior, unsupported append writers, late configuration, and failed factory cleanup. All 24 focused Core tests pass on JDK 11 with the normal build checks.
jordepic
commented
Sep 12, 2026
Contributor
Author
Thank you! Hoping to use this in https://github.com/datafusion-contrib/StreamFusion/tree/main
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
Purpose
Primary-key compaction chooses its rewriter inside a private factory. An external engine currently has to reconstruct the writer setup to replace the file read/merge/write work with its own implementation.
Add
TableWrite.withCompactRewriterFactory(...), backed by the existingCompactRewriterboundary. Each partition/bucket receives its own rewriter, and the factory receives Paimon's normal rewriter so implementations can delegate unsupported operations. Paimon retains input selection, background scheduling, and commit coordination. This PR adds the extension point; it does not add a native compaction engine.The API documents ownership and recovery: the returned rewriter owns the default, failed creation closes the default, and the factory must be installed before bucket creation and again on recovered writers. Write-only mode does not invoke the factory; append, postpone, and primary-key clustering writers reject it. The Java writing guide includes the integration contract.
The shared MinIO test image now uses
quay.io/minio/miniowith the existing release tag. Docker Hub returns pull-access-denied for the old image, causing the S3 setup failures on this PR and master. Quay is the registry shown in MinIO's container instructions.cc @JingsongLi — feedback on exposing this boundary for external/native engines would be appreciated.
Tests
24 focused Core tests passed on JDK 11, with normal Checkstyle, Spotless, and enforcer checks enabled:
mvn -pl paimon-core -am -Pflink1 \ -DwildcardSuites=none \ -Dtest=CompactRewriterFactoryTest,MergeTreeCompactManagerFactoryTest,TableWriteTest,KeyValueFileStoreWriteTest \ -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false testCoverage includes real rewrites and metadata upgrades across reopened writers; fallback selection for none/input/lookup/full-compaction and deletion vectors; final table contents; rewriter closure; write-only behavior; unsupported append writers; late installation; and cleanup/suppressed exceptions when factory creation fails.
The failing
S3FileIOTestalso passes on JDK 11 against the Quay image: 21 tests, zero failures/errors/skips, with normal Checkstyle, Spotless, and enforcer checks enabled.mvn -pl paimon-filesystems/paimon-s3-impl -am -Pflink1 \ -DwildcardSuites=none -Dtest=S3FileIOTest \ -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false test