-
Notifications
You must be signed in to change notification settings - Fork 373
fix: validate map constructor row lengths and null short-circuiting - #5846
Open
sunchao wants to merge 2 commits into
Open
fix: validate map constructor row lengths and null short-circuiting #5846sunchao wants to merge 2 commits into
sunchao wants to merge 2 commits into
Conversation
@github-actions
github-actions
Bot
added
bug
Something isn't working
area:expressions
Expression evaluation
labels
Sep 10, 2026
sunchao
marked this pull request as ready for review
September 11, 2026 06:25
rich7420
rich7420
approved these changes
Sep 11, 2026
@rich7420
rich7420
left a comment
Contributor
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM , @sunchao thanks for the patch!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
Which issue does this PR close?
Follow-up to #5045, addressing additional correctness failures in
map_from_arrays.Rationale for this change
map_from_arrayspairs each row's keys with that same row's values. Spark requires both arrays to have equal lengths. Comet currently checks the combined data for a batch, allowing mismatches in individual rows to cancel out:[1][10, 20]{1: 10}[2, 3, 4][30, 40]{2: 20, 3: 30, 4: 40}Both columns contain four elements in total, so construction succeeds even though neither row is valid. The value
20crosses into the second row's map. Spark rejects this input with a length-mismatch error; Comet should do the same.There is also an evaluation-order problem. When the keys array is null, Spark returns a null map immediately. For example, consider a batch containing
k = 0andk = 1:With ANSI mode enabled, Spark returns null for
k = 0without evaluating1 / 0. Comet can evaluate the values expression for that row and fail the entire query, even though no map needs to be constructed.What changes are included in this PR?
The native path now checks key/value lengths within each non-null row before constructing maps. The same rule applies when one operand is a constant array repeated across the batch, so combining constants and columns cannot hide a mismatch.
Map construction also follows Spark's null-handling order: check the keys first, evaluate the values only for rows with non-null keys, and construct maps only where both arrays exist. The existing constructor handles the resulting valid inputs. This fixes the two examples above without expanding the set of supported map types or casts. Evaluating nondeterministic children exactly once remains a separate issue in #5781.
How are these changes tested?
On the original PR revision, the Linux Rust test job passed, including native coverage for row-length mismatches, constant/column combinations, nulls, empty batches, and sliced inputs. All six new Spark regressions passed in the Spark 4.1 expression job, with JVM codegen dispatch enabled and disabled.
That run's only failed test was a shared decimal codegen coverage assertion, now corrected by upstream #5849 and included in this branch. The exact assertion failed locally before that correction and passed afterward. Full Spark 4.1 reactor compilation, Scalastyle, and Spotless passed on the updated branch. See fresh CI for revision
f3ee8d91for the complete test matrix.