Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

yuhexiong/kafka-data-pipeline-structured-flink-java

Folders and files

NameName
Last commit message
Last commit date

Latest commit

History

10 Commits

Repository files navigation

Kafka Data Pipeline Flink

(also provided Traditional Chinese version document README-CH.md.)

Data pipeline written by flink to transfer Kafka to Kafka, Doris and also merge the two data sources.

Overview

  • Platform: JDK 11
  • Build Tool: Apache Maven v3.9.6
  • Data Processing Framework: Flink v1.18.1

Run

Use IntelliJ IDEA
IntelliJ IDEA

Entry

1. KafkaToKafka

Transfer all Product messages from topic-source in Kafka (localhost:9092) to topic-sink in Kafka (localhost:9092).

  • Kafka Topic topic-source Message
{
 "id": "12345",
 "name": "Wireless Mouse",
 "category": "Electronics",
 "manufacturer": "TechCorp",
 "description": "A sleek, ergonomic wireless mouse with advanced optical tracking.",
 "cost": 29.99
}

2. KafkaToDoris

Convert all message in topic-product in Kafka (localhost:9092) and insert it into the Doris (localhost:9030) database database.product.

  • Kafka Topic topic-product Message
{
 "id": "12345",
 "name": "Wireless Mouse",
 "category": "Electronics",
 "manufacturer": "TechCorp",
 "description": "A sleek, ergonomic wireless mouse with advanced optical tracking.",
 "cost": 29.99
}
  • Doris Table database.product
| id | name | category | manufacturer | description | cost |
|---------|-----------------|-------------|---------------|-----------------------------------|-------|
| 12345 | Wireless Mouse | Electronics | TechCorp | A sleek, ergonomic wireless mouse | 29.99 |

3. TwoKafkaToDoris

Combine message topic-product in Kafka (localhost:9092) with message from topic-sale. Then, transfer the resulting data into the Doris (localhost:9030) database database.sale_report.

  • Kafka Topic topic-product Message
{
 "id": "12345",
 "name": "Wireless Mouse",
 "category": "Electronics",
 "manufacturer": "TechCorp",
 "description": "A sleek, ergonomic wireless mouse with advanced optical tracking.",
 "cost": 29.99
}
  • Kafka Topic topic-sale Message
{
 "id": "A98765",
 "productId": "12345",
 "unit": 3,
 "unitPrice": 49.99,
 "totalPrice": 149.97,
 "saleDate": "2024年11月28日"
}
  • Doris Table database.sale_report
| sale_id | product_id | unit | unit_price | total_price | sale_date | product_name | product_unit_cost | profit |
|-----------|------------|------|------------|-------------|----------------------|---------------|-------------------|--------|
| A98765 | 12345 | 3 | 49.99 | 149.97 | 2024年11月28日T08:00:00 | Wireless Mouse| 29.99 | 60.00 |

About

Data pipeline from Kafka to Kafka, Doris and Doris to Kafka using Flink Java.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

AltStyle によって変換されたページ (->オリジナル) /