Data Proxy

Data Proxy synchronises BigQuery tables to a PostgreSQL database (pg_duckdb) and exposes them through a PostgREST REST API, enforcing row-level security against a backend-managed access policy table, keyed by JWT identity claims.

BigQuery is the authoritative data store. PostgreSQL is a disposable, eventually consistent read cache.

• Architecture — service objectives, pipeline components, standalone and HA diagrams.
• Backups — encrypted backups of access_policy tables, and how to restore one.
• Development — running the full pipeline on Kubernetes with Minikube and Helm.
• Environment Variables — configuration read by the producer, worker, and finalizer.
• Helm Chart — installing and configuring the chart, including HA mode.
• Security — creating users and row-level security.
• Sync — the sync.json file, table fields, and strategy selection.
• Using the API — querying tables through PostgREST: filtering, ordering, pagination, counting.

This project is licensed under the Apache License 2.0. See LICENSE for the full text.