data-lake

This repo contains deployment of a simple data lakehouse.

Components:

• MinIO s3 storage
• Apache Polaris Catalog
• Query engines:
  • Spark
  • Trino
  • DuckDB
• Dagu task orchestrator
  • scheduled iceberg maintenance
  • schema migrations
• Sample micro-batching data ingestion from NATS using python (pyiceberg)

Notes/TODOs:

• This is local deployment with example credentials committed to repo. For running in prod they need to be replaced with secrets,
• Multiple users with limited permissions need to be used in s3 and polaris,
• MinIO is deprecated, replacement needed,
• Custom partitioning for tables,
• Maintenance tasks: track commit failure and retry,

Running instructions

Build necessary images:

podman build . -f Dockerfile.bash -t dagu-worker-bash:0.1
podman build . -f Dockerfile.spark -t dagu-worker-spark:0.1
podman build . -f Dockerfile.writer -t writer-python:0.1

Start all components:

sudo bash lake.sh up

Stop all components:

sudo bash lake.sh down

Event writer dev:

pip3 install nats-py pyiceberg[pyarrow,s3fs]

Querying data

Duckdb:

sudo podman run -it --rm --network lake -v ./config/duckdb/init.sql:/tmp/init.sql docker.io/duckdb/duckdb:1.4.4 duckdb -init /tmp/init.sql
# select count(*) from mycatalog.raw_data.event;

Spark:

sudo podman run -it --rm --network lake -e SPARK_CONFIG=/opt/spark/conf/spark-defaults.conf -v ./config/spark/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf dagu-worker-spark:0.1 /opt/spark/bin/spark-sql --master local[*]
# select count(*) from spark_catalog.raw_data.event;

Trino:

sudo podman run -d --name trino --network lake -v ./config/trino:/etc/trino:ro docker.io/trinodb/trino:479
sudo podman run -it --rm --network lake docker.io/trinodb/trino:479 trino --catalog iceberg --server trino:8080
# select count(*) from raw_data.event;