-
Notifications
You must be signed in to change notification settings - Fork 152
dataset write api #1025
why is dataset is designed to write a single parquet file, i mean it throws an error if write to non empty directory into s3, but usually tables consists of many files, am i did something wrong? but i do not see an option to write multiple parquet files into a directory using dataset
All reactions
PR review capacity is pretty limited at the moment but PRs are always welcome. Supporting all options at once would probably be the better approach rather than picking and choosing.
Replies: 1 comment 7 replies
Hi @dbelozerovx1, arrow-java's Dataset implementation supports writing a dataset out to multiple files like you expect, see https://github.com/apache/arrow-java/blob/main/dataset/src/test/java/org/apache/arrow/dataset/file/TestDatasetFileWriter.java#L66 for an example. If you want to share the code you have that isn't working, I can take a look.
All reactions
I see now, thanks.
I think there are two options,
- The JNI bindings need to be extended to support setting custom DatasetWriteOptions. This might let you do what you want though I'm not totally sure.
- (Probably easier) Create a new reader for each file you want to write and call into
DatasetFileWriter.writeonce for each reader
All reactions
first one seems better since its generally supports whats already done on cpp side, i can provide a basic contribution with existing_data_behaviour param and compression codec(the second thing i need the most) and try to contribute, but theres many other options maybe you have plans to support em all?
All reactions
my suggestion is
options.existing_data_behavior =
arrow::dataset::ExistingDataBehavior::kOverwriteOrIgnore;
and compressionCodec allow as a param to be passed from java
All reactions
PR review capacity is pretty limited at the moment but PRs are always welcome. Supporting all options at once would probably be the better approach rather than picking and choosing.
All reactions
-
👍 1