I am exploring data write into glue Table (Iceberg Table format). I have been using saveAsTable method mentioned as option1 . However is there any difference between two methods. Iceberg stores metadata in s3. Is wrtiting to path will not alow to update partitions in. glue metastore? I am using emr to write to data into s3.
result_df.write \
.mode("overwrite") \
.format("iceberg") \
.option("write-format", "parquet") \
.option("fanout-enabled", "false") \
.option("format-version", "2") \
.option("write.distribution-mode", "hash") \
.option("write.merge.mode", "copy-on-write") \
.partitionBy("source", "region", "country", "dataset_date") \
.saveAsTable("your_database.your_table")
Or for path-based table:
result_df.write \
.mode("overwrite") \
.format("iceberg") \
.option("path", s3_output_path) \
.option("write-format", "parquet") \
.option("fanout-enabled", "false") \
.option("format-version", "2") \
.option("write.distribution-mode", "hash") \
.option("write.merge.mode", "copy-on-write") \
.partitionBy("source", "region", "country", "dataset_date") \
.save()
asked Jun 26, 2025 at 15:21
user3858193
1,5685 gold badges23 silver badges60 bronze badges