Upserts, Deletes And Incremental Processing on Big Data.
bigdata stream-processing data-integration datalake apachespark hudi apachehudi incremental-processing apacheflink
-
Updated
Jun 14, 2026 - Java
#
incremental-processing
Here are 12 public repositories matching this topic...
Language: All
Filter by language
Sort: Most stars
Sort options
汇总Apache Hudi相关资料
bigdata apache stream-processing data-integration datalake hudi apachehudi incremental-processing hudi-resources
-
Updated
Mar 31, 2026
Incremental processing and maintaining data freshness with CocoIndex and LanceDB
-
Updated
Jan 14, 2026 - Python
An AI Email Intelligence Platform Real-time email intelligence with multi-provider AI fallback, semantic search, OAuth integration. Handles incremental sync and streaming with 70% cold start reduction.
oauth2 websockets gmail-api fastapi incremental-processing ai-automation trpc-api real-time-ai llm-orchestration multi-model-ai
-
Updated
Jun 5, 2026 - TypeScript
A modern banking data pipeline built with Dagster and DBT!
python data-engineering dbt databricks data-quality data-lineage scd2 dagster incremental-processing duckdb modendatastack
-
Updated
Jan 31, 2026 - Python
Reusable data matching system with incremental processing for efficient reuse of historical results.
-
Updated
May 6, 2026 - Python
A production-grade cryptocurrency data pipeline built on GCP that ingests real-time market data from the CoinGecko API, implements Medallion architecture (Raw → Staging → Curated), and supports idempotent backfill and metadata-driven incremental processing for reliable, scalable analytics.
cryptocurrency data-engineering data-pipeline api-integration metadata-driven gcp-bigquery incremental-processing medallion-architecture idempotent-pipeline
-
Updated
Mar 25, 2026 - Python
End-to-end data engineering project built in Snowflake using a Medallion Architecture (Bronze, Silver, Gold). The project demonstrates ELT pipeline design, data ingestion from AWS S3, data cleaning and transformation, incremental processing, and dimensional modelling using a star schema.
sql aws-s3 snowflake data-warehouse data-engineering business-intelligence data-quality star-schema etl-pipeline data-modelling analytics-engineering dimensional-modeling incremental-processing cloud-data-platform elt-pipeline medallion-architecture
-
Updated
May 9, 2026
End-to-end data engineering pipeline on Databricks with Delta Lake, incremental watermarking, and Dockerized Airflow orchestration (Postgres-backed, SLA-enabled).
docker airflow spark etl data-engineering databricks delta-lake incremental-processing medallion-architecture
-
Updated
Feb 21, 2026 - Jupyter Notebook
Apache Hudi is a data lake platform that provides incremental data processing primitives including upserts and incremental queries. It manages storage of large analytical datasets on distributed file systems with ACID transactions, timeline-based versioning, and integrations for Spark, Flink, and Hive.
-
Updated
Jun 5, 2026
End-to-end data pipeline that ingests job market data from the Adzuna API and processes it using a Medallion Architecture (Bronze→Silver→Gold). Includes data quality validation, incremental loading into DuckDB, and Airflow orchestration. Generates analytics-ready datasets for role demand, salary trends, and skill demand across multiple countries.
airflow data-engineering parquet etl-pipeline incremental-processing job-market-analysis duckdb api-ingestion medallion-architecture
-
Updated
Mar 1, 2026 - Python
Monitor AI agent memory, skills, and behavior in a live terminal HUD for Hermes.
apple terminal progress cocoapods stream-processing openai data-integration datalake ai-agents apachespark hudi apachehudi incremental-processing llm chatgpt anthropic claude-code moltbot openclaw hermes-agent
-
Updated
Jun 14, 2026 - Python
Improve this page
Add a description, image, and links to the incremental-processing topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the incremental-processing topic, visit your repo's landing page and select "manage topics."