-
-
Notifications
You must be signed in to change notification settings - Fork 18.9k
PARTIAL FIX: Improve leading zeros preservation with dtype=str for dict-based dtypes #62242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Open
Changes from all commits
Commits
Show all changes
20 commits
Select commit
Hold shift + click to select a range
b766c15
Create test_preserve_leading_zeros.py
dxdc ea08043
Update test_preserve_leading_zeros.py
dxdc 206cb69
Update test_preserve_leading_zeros.py
dxdc 247d514
Update test_preserve_leading_zeros.py
dxdc 3e9f04e
Update test_preserve_leading_zeros.py
dxdc c86f33f
Update test_preserve_leading_zeros.py
dxdc 81b50db
Update test_preserve_leading_zeros.py
dxdc 5990400
Update test_preserve_leading_zeros.py
dxdc d9f6983
draft column_types parsing
dxdc 0ebca38
update tests
dxdc 22bd4e3
Update v3.0.0.rst
dxdc 22e4129
Update test_preserve_leading_zeros.py
dxdc 74f01a7
use numpy dtype mapping
dxdc 9d68f91
change warnings behavior
dxdc a324f4a
Update arrow_parser_wrapper.py
dxdc 8cda906
Update arrow_parser_wrapper.py
dxdc df9e96d
Update arrow_parser_wrapper.py
dxdc d2925bc
use native to_pyarrow_type
dxdc d6f3a5c
remove unused dep
dxdc 5df2f20
Update arrow_parser_wrapper.py
dxdc File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
71 changes: 71 additions & 0 deletions
pandas/tests/io/parser/test_preserve_leading_zeros.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| from io import StringIO | ||
|
|
||
| import pytest | ||
|
|
||
|
|
||
| def test_leading_zeros_preserved_with_dtype_str(all_parsers, request): | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this doesn't merit its own file. try to find plausibly-related tests to put it with There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It seemsl ike the |
||
| # GH#57666: pyarrow engine strips leading zeros when dtype=str is passed | ||
| # GH#61618: further discussion on ensuring string dtype preservation across engines | ||
|
|
||
| parser = all_parsers | ||
| engine_name = getattr(parser, "engine", "unknown") | ||
|
|
||
| data = """col1,col2,col3,col4 | ||
| AB,000388907,abc,0150 | ||
| CD,101044572,def,0150 | ||
| EF,000023607,ghi,0205 | ||
| GH,100102040,jkl,0205""" | ||
|
|
||
| result = parser.read_csv( | ||
| StringIO(data), | ||
| dtype=str, | ||
| ) | ||
|
|
||
| try: | ||
| assert result.shape == (4, 4) | ||
| assert list(result.columns) == ["col1", "col2", "col3", "col4"] | ||
|
|
||
| assert result.loc[0, "col2"] == "000388907", "lost zeros in col2 row 0" | ||
| assert result.loc[2, "col2"] == "000023607", "lost zeros in col2 row 2" | ||
| assert result.loc[0, "col4"] == "0150", "lost zeros in col4 row 0" | ||
| assert result.loc[2, "col4"] == "0205", "lost zeros in col4 row 2" | ||
|
|
||
| except AssertionError as exc: | ||
| if engine_name == "pyarrow": | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. take a look at how we handle xfails elsewhere. we check and add the marker before the meat of the test There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jbrockmendel I considered that option, but it doesn't seem appropriate for this case. The tests only fail for the pyarrow engine, and only because there is an underlying flaw in the pyarrow read logic. Is there another preferred way to handle this? |
||
| # Temporary workaround for GH#57666 | ||
| # Remove once type preservation is fixed in pyarrow engine. | ||
| request.node.add_marker( | ||
| pytest.mark.xfail(reason=f"failed assertions: {exc}", strict=False) | ||
| ) | ||
| raise | ||
|
|
||
|
|
||
| def test_leading_zeros_preserved_with_dtype_dict(all_parsers): | ||
| # GH#57666: pyarrow engine strips leading zeros when dtype=str is passed | ||
| # GH#61618: further discussion on ensuring string dtype preservation across engines | ||
|
|
||
| parser = all_parsers | ||
|
|
||
| data = """col1,col2,col3,col4 | ||
| AB,000388907,199,0150 | ||
| CD,101044572,200,0150 | ||
| EF,000023607,201,0205 | ||
| GH,100102040,202,0205""" | ||
|
|
||
| result = parser.read_csv( | ||
| StringIO(data), | ||
| dtype={"col2": str, "col3": int, "col4": str}, | ||
| ) | ||
|
|
||
| assert result.shape == (4, 4) | ||
| assert list(result.columns) == ["col1", "col2", "col3", "col4"] | ||
|
|
||
| assert result.loc[0, "col2"] == "000388907", "lost zeros in col2 row 0" | ||
| assert result.loc[2, "col2"] == "000023607", "lost zeros in col2 row 2" | ||
| assert result.loc[0, "col4"] == "0150", "lost zeros in col4 row 0" | ||
| assert result.loc[2, "col4"] == "0205", "lost zeros in col4 row 2" | ||
|
|
||
| assert result.loc[0, "col3"] == 199 | ||
| assert result.loc[1, "col3"] == 200 | ||
| assert result.loc[2, "col3"] == 201 | ||
| assert result.loc[3, "col3"] == 202 | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.