Skip to content

Navigation Menu

Sign in
Sign up

fix: validate map constructor row lengths and null short-circuiting - #5846

Open
sunchao wants to merge 2 commits into
apache:main from
sunchao:codex/upstream-map-correctness
Open

fix: validate map constructor row lengths and null short-circuiting #5846
sunchao wants to merge 2 commits into
apache:main from
sunchao:codex/upstream-map-correctness

Conversation

@sunchao

@sunchao sunchao commented Sep 10, 2026
edited
Loading

Copy link
Copy Markdown
Member

Which issue does this PR close?

Follow-up to #5045, addressing additional correctness failures in map_from_arrays.

Rationale for this change

map_from_arrays pairs each row's keys with that same row's values. Spark requires both arrays to have equal lengths. Comet currently checks the combined data for a batch, allowing mismatches in individual rows to cancel out:

Row Keys Values Current Comet result
1 [1] [10, 20] {1: 10}
2 [2, 3, 4] [30, 40] {2: 20, 3: 30, 4: 40}

Both columns contain four elements in total, so construction succeeds even though neither row is valid. The value 20 crosses into the second row's map. Spark rejects this input with a length-mismatch error; Comet should do the same.

There is also an evaluation-order problem. When the keys array is null, Spark returns a null map immediately. For example, consider a batch containing k = 0 and k = 1:

SELECT map_from_arrays(
 CASE WHEN k = 0 THEN CAST(NULL AS ARRAY<INT>) ELSE array(1) END,
 array(1 / k))
FROM input

With ANSI mode enabled, Spark returns null for k = 0 without evaluating 1 / 0. Comet can evaluate the values expression for that row and fail the entire query, even though no map needs to be constructed.

What changes are included in this PR?

The native path now checks key/value lengths within each non-null row before constructing maps. The same rule applies when one operand is a constant array repeated across the batch, so combining constants and columns cannot hide a mismatch.

Map construction also follows Spark's null-handling order: check the keys first, evaluate the values only for rows with non-null keys, and construct maps only where both arrays exist. The existing constructor handles the resulting valid inputs. This fixes the two examples above without expanding the set of supported map types or casts. Evaluating nondeterministic children exactly once remains a separate issue in #5781.

How are these changes tested?

On the original PR revision, the Linux Rust test job passed, including native coverage for row-length mismatches, constant/column combinations, nulls, empty batches, and sliced inputs. All six new Spark regressions passed in the Spark 4.1 expression job, with JVM codegen dispatch enabled and disabled.

That run's only failed test was a shared decimal codegen coverage assertion, now corrected by upstream #5849 and included in this branch. The exact assertion failed locally before that correction and passed afterward. Full Spark 4.1 reactor compilation, Scalastyle, and Spotless passed on the updated branch. See fresh CI for revision f3ee8d91 for the complete test matrix.

@github-actions github-actions Bot added bug Something isn't working area:expressions Expression evaluation labels Sep 10, 2026
sunchao marked this pull request as ready for review September 11, 2026 06:25

@rich7420 rich7420 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM , @sunchao thanks for the patch!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@rich7420 rich7420 rich7420 approved these changes
Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

area:expressions Expression evaluation bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

AltStyle によって変換されたページ (->オリジナル) /