Commit 22df68e

authored

BUG: read_csv with chained fsspec TAR file and compression="infer" (#60100)

1 parent f9f72d1 commit 22df68eCopy full SHA for 22df68e

File tree

+18

-0

lines changed

+18

-0

lines changed

Lines changed: 1 addition & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -784,6 +784,7 @@ Other`
`784`	`784`	- Bug in :meth:`Series.dt` methods in :class:`ArrowDtype` that were returning incorrect values. (:issue:`57355`)
`785`	`785`	- Bug in :meth:`Series.rank` that doesn't preserve missing values for nullable integers when ``na_option='keep'``. (:issue:`56976`)
`786`	`786`	- Bug in :meth:`Series.replace` and :meth:`DataFrame.replace` inconsistently replacing matching instances when ``regex=True`` and missing values are present. (:issue:`56599`)
	`787`	+- Bug in :meth:`read_csv` where chained fsspec TAR file and ``compression="infer"`` fails with ``tarfile.ReadError`` (:issue:`60028`)
`787`	`788`	- Bug in Dataframe Interchange Protocol implementation was returning incorrect results for data buffers' associated dtype, for string and datetime columns (:issue:`54781`)
`788`	`789`	- Bug in ``Series.list`` methods not preserving the original :class:`Index`. (:issue:`58425`)
`789`	`790`

Lines changed: 3 additions & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -584,6 +584,9 @@ def infer_compression(`
`584`	`584`	`# Infer compression`
`585`	`585`	`if compression == "infer":`
`586`	`586`	`# Convert all path types (e.g. pathlib.Path) to strings`
	`587`	`+ if isinstance(filepath_or_buffer, str) and "::" in filepath_or_buffer:`
	`588`	`+ # chained URLs contain ::`
	`589`	`+ filepath_or_buffer = filepath_or_buffer.split("::")[0]`
`587`	`590`	`filepath_or_buffer = stringify_path(filepath_or_buffer, convert_file_like=True)`
`588`	`591`	`if not isinstance(filepath_or_buffer, str):`
`589`	`592`	`# Cannot infer compression of a buffer, assume no compression`

10 KB

Binary file not shown.

Lines changed: 14 additions & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -25,6 +25,7 @@`
`25`	`25`	`WASM,`
`26`	`26`	`is_platform_windows,`
`27`	`27`	`)`
	`28`	`+import pandas.util._test_decorators as td`
`28`	`29`
`29`	`30`	`import pandas as pd`
`30`	`31`	`import pandas._testing as tm`
`@@ -642,6 +643,19 @@ def close(self):`
`642`	`643`	`handles.created_handles.append(TestError())`
`643`	`644`
`644`	`645`
	`646`	`+@td.skip_if_no("fsspec", min_version="202310")`
	`647`	`+@pytest.mark.parametrize("compression", [None, "infer"])`
	`648`	`+def test_read_csv_chained_url_no_error(compression):`
	`649`	`+ # GH 60100`
	`650`	`+ tar_file_path = "pandas/tests/io/data/tar/test-csv.tar"`
	`651`	`+ chained_file_url = f"tar://test.csv::file://{tar_file_path}"`
	`652`	`+`
	`653`	`+ result = pd.read_csv(chained_file_url, compression=compression, sep=";")`
	`654`	`+ expected = pd.DataFrame({"1": {0: 3}, "2": {0: 4}})`
	`655`	`+`
	`656`	`+ tm.assert_frame_equal(expected, result)`
	`657`	`+`
	`658`	`+`
`645`	`659`	`@pytest.mark.parametrize(`
`646`	`660`	`"reader",`
`647`	`661`	`[`

Comments

(0)