-
Notifications
You must be signed in to change notification settings - Fork 4.3k
Is pyarrow.dataset still unstable? #50797
The docstring of python/pyarrow/dataset.py says, "Dataset is currently unstable. APIs subject to change without notice." However, the documentation has already removed the warning that it is experimental since version 12.0. Is the docstring still up to date?
All reactions
Replies: 2 comments
I don't have the historical context but the docstring message appears on several files:
python/pyarrow/dataset.py:"""Dataset is currently unstable. APIs subject to change without notice."""
python/pyarrow/_dataset.pyx:"""Dataset is currently unstable. APIs subject to change without notice."""
python/pyarrow/_dataset.pxd:"""Dataset is currently unstable. APIs subject to change without notice."""
The original one is from 7 years ago. Given this is not part of the documentation and only a docstring file comment it probably is just a stray comment. At this point if we modify the API it would probably go over a Deprecation warning as we do with other APIs.
I'd create an issue and submit a PR removing the docstring, we can ping a couple more people to validate on the review.
All reactions
-
👍 1
I am not maintainer but I believe pyarrow dataset relies on C++ dataset that relies on Acero which is experimental and sadly feels abandoned.
I believe Acreo should remain experimental until it handles ordered datasets and time series correctly and natively. Right now dataset ordering can only be maintained with obscure combo of single threaded scanning custom naming convention(for alphabetical ordering) and ordered sinks.
It feels like Acero is just a POC to prove columnar memory format is suitable for execution and its development stopped once it was established.