-
Notifications
You must be signed in to change notification settings - Fork 4.3k
I have this snippet of code:
from pathlib import Path import pyarrow as pa import pyarrow.dataset as ds tempdir = Path(__file__).parent / 'temp' table = pa.table({"a": range(1024)}) batches = table.to_batches(max_chunksize=2) ds.write_dataset(batches, tempdir, format="parquet", preserve_order=True, use_threads=True)
I was expecting it to write all the batches, but it is only writing the first batch:
$ ls temp part-0.parquet
Is this the expected output? I'm working with Python 3.11 and pyarrow 21.0.0
$ ipython Python 3.11.11 (main, Feb 5 2025, 19:11:07) [Clang 19.1.6 ] Type 'copyright', 'credits' or 'license' for more information IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for help. Tip: You can change the editing mode of IPython to behave more like vi, or emacs. In [1]: import pyarrow as pa In [2]: pa.__version__ Out[2]: '21.0.0' In [3]:
All reactions
Replies: 1 comment 1 reply
Hi @mesejo. What's the contents of part-0.parquet? I'd expect it to contain all 1024 rows. write_dataset doesn't partition data into separate files for each input batch. Maybe you want to use the max_rows_per_file parameter if you're trying to reduce the size of files?
All reactions
1 reply
Indeed, the file contains all the data.
All reactions
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment