-
Notifications
You must be signed in to change notification settings - Fork 29.4k
[SPARK-59457][SQL] RowLevelOperationTable should delegate partitioning() to the wrapped table - #58756
Open
pan3793 wants to merge 1 commit into
Open
[SPARK-59457][SQL] RowLevelOperationTable should delegate partitioning() to the wrapped table #58756pan3793 wants to merge 1 commit into
pan3793 wants to merge 1 commit into
Conversation
...g() to the wrapped table The read relation of a row-level rewrite (UPDATE, MERGE, group-based DELETE) wraps the table in RowLevelOperationTable, which did not override partitioning(), so the default empty array was returned. Since SPARK-55596 (4.2.0) PushDownUtils.getPartitionPredicateSchema reads it for those scans, found no transforms, and never derived a PartitionPredicate for any row-level operation. Delegate partitioning() to the wrapped table. The in-memory row-level fixture now pushes V2 predicates iteratively, and a group-based UPDATE test checks that the second-pass PartitionPredicate reaches the scan and that only the matching partitions are replaced. Assisted-by: Claude Fable 5.1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
RowLevelOperationTable, the internal wrapper a row-level rewrite (UPDATE, MERGE, group-basedDELETE) reads the table through, delegates
name,columns,capabilitiesandconstraintsto the wrapped table but not
partitioning(), so it returned the default empty array. It nowdelegates
partitioning()too.The in-memory row-level test table now pushes V2 predicates iteratively, so a group-based scan
can receive a second-pass
PartitionPredicate.Why are the changes needed?
Since SPARK-55596 (4.2.0),
PushDownUtils.getPartitionPredicateSchemareads the partitioningof the read relation's table to derive
PartitionPredicates, in the group-based row-levelscan planning rule, in
V2ScanRelationPushDownfor delta-based scans, and in runtime filterpushdown. For every row-level operation the table it sees is the wrapper, so the schema was
always empty and no
PartitionPredicatewas ever derived for a row-level scan.The wrapper has been this way since SPARK-38085 (3.3.0), but nothing read its partitioning
before 4.2.0, so the gap was invisible until then.
Does this PR introduce any user-facing change?
No. A connector that opts into iterative pushdown now receives
PartitionPredicates forUPDATE, MERGE and DELETE scans the same way it does for SELECT.
How was this patch tested?
New test: a group-based UPDATE on a partitioned table pushes a second-pass
PartitionPredicateto the row-level scan, and only the matching partitions are replaced.It fails without the fix.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5.1