-
Notifications
You must be signed in to change notification settings - Fork 29.4k
[SPARK-54966][PYTHON] Factor out legacy pandas-to-Arrow column conversion - #58751
Open
Spenserrrr wants to merge 1 commit into
Open
[SPARK-54966][PYTHON] Factor out legacy pandas-to-Arrow column conversion #58751Spenserrrr wants to merge 1 commit into
Spenserrrr wants to merge 1 commit into
Conversation
...sion Separate batch handling in from_pandas from single-Series dispatch in convert and the existing implementation in convert_legacy. Clarify the legacy error-handling flag and update the worker callers and tests.
Spenserrrr
marked this pull request as ready for review
September 11, 2026 21:59
Spenserrrr
commented
Sep 11, 2026
Contributor
Author
Hi @Yicong-Huang @zhengruifeng! This PR separates pandas-to-Arrow batch processing from column conversion. Right now, from_pandas() handles batch orchestration, and convert() selects the per-column strategy. This prepares the column layer for subsequent convert_numpy and convert_pyarrow implementations. Could you take a look when you have time? Thanks!
cc @gaogaotiantian
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
What changes were proposed in this pull request?
Separate
PandasToArrowConversioninto three layers:from_pandas(): batch handling and RecordBatch assembly.convert(): per-column strategy dispatch.convert_legacy(): the existing_create_converter_from_pandas()fix-ups, Arrow construction, cast fallback, and error translation.The name
convert_legacyidentifies the existing conversion implementation retained during migration. Rename the separateis_legacyflag touse_legacy_error_handlingto distinguish its exception/message policy from strategy selection, and update callers and tests.Why are the changes needed?
Planned follow-ups are type-by-type migration from
convert_legacyintoconvert_numpy, following the read leg's approach, and addingconvert_pyarrowfor Arrow-backed pandas Series (SPARK-54970) as part of the zero-copy pandas-UDF work. Establishing the batch/column boundary now lets those strategies evolve independently of batch handling while retaining the existing fallback.Does this PR introduce any user-facing change?
No; this is an internal refactor.
How was this patch tested?
Existing conversion tests with pandas 2.3/PyArrow 24 and pandas 3/PyArrow 25, plus baseline-parity and error-handling probes.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-6)