-
Notifications
You must be signed in to change notification settings - Fork 4.3k
[Python] Update pandas->pyarrow conversion for the .values deprecation in pandas #51302
Open
Description
Pandas has deprecated certain aspects of .values (when it looses information), which our pandas->pyarrow conversion currently relies upon, so in the tests with pandas nightly we are seeing a lot of warnings because of that (https://github.com/ursacomputing/crossbow/actions/runs/34548523730/job/103106277981).
Small illustration:
>>> import pandas as pd >>> import pyarrow as pa >>> ser = pd.Series(pd.date_range("2025-01-01", periods=3, tz="UTC")) >>> pa.array(ser) <python-input-4>:1: Pandas4Warning: Series.values returning an ndarray that drops timezone information for DatetimeTZDtype is deprecated. In a future version, this will return the underlying DatetimeArray instead. Use 'Series.to_numpy()' to get a NumPy array, or 'Series.array' to get the ExtensionArray. <pyarrow.lib.TimestampArray object at 0x7ff624fb1a20> [ 2025-01-01 00:00:00.000000Z, 2025-01-02 00:00:00.000000Z, 2025-01-03 00:00:00.000000Z ]
This is caused by our handling of a Series object in pa.array(..):
arrow/python/pyarrow/array.pxi
Lines 5396 to 5407 in 07be48c
cdef object get_values(object obj, bint* is_series):
if pandas_api.is_series(obj) or pandas_api.is_index(obj):
result = pandas_api.get_values(obj)
is_series[0] = True
elif isinstance(obj, np.ndarray):
result = obj
is_series[0] = False
else:
result = pandas_api.series(obj, copy=False).values
is_series[0] = False
return result
and
arrow/python/pyarrow/pandas-shim.pxi
Lines 231 to 242 in 07be48c
cpdef get_values(self, obj):
"""
Get the underlying array values of a pandas Series or Index in the
format (np.ndarray or pandas ExtensionArray) as we need them.
Assumes obj is a pandas Series or Index.
"""
self._check_import()
if isinstance(obj.dtype, (self.pd.api.types.IntervalDtype,
self.pd.api.types.PeriodDtype)):
return obj.array
return obj.values
So we already have some custom handling for Period/Interval, but that will also be needed for DatetimeTZDtype