Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.
The leader in Customer Data Infrastructure
Flink CDC is a streaming data integration tool
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
Privacy and Security focused Segment-alternative, in Golang and React
A list of useful resources to learn Data Engineering from scratch
Memphis.dev is a highly scalable and effortless data streaming platform
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
An open-source data logging library for machine learning models and data pipelines. 📚 Provides visibility into data quality & model performance over time. 🛡️ Supports privacy-preserving data collection, ensuring safety & robustness. 📈
Lightweight and powerful workflow engine for enterprise & small teams. Single binary with Web UI. 100% open source. No vendor lock-in. It natively supports running containers and executing commands over SSH. Offline or air-gapped environment ready.
The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
A lightweight stream processing library for Go
CLI task management & automation tool
BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.
🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
Superlinked is a Python framework for AI Engineers building high-performance search & recommendation applications that combine structured and unstructured data.
Example end to end data engineering project.
Add a description, image, and links to the data-pipeline topic page so that developers can more easily learn about it.
To associate your repository with the data-pipeline topic, visit your repo's landing page and select "manage topics."