|
| 1 | +# Kafka Data Pipeline Flink |
| 2 | + |
| 3 | +使用 Flink 編寫的資料管道,用於將 Kafka 資料傳輸到 Kafka、Doris,也支援合併兩個資料來源。 |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +- 平台:JDK 11 |
| 8 | +- 建構工具:Apache Maven v3.9.6 |
| 9 | +- 資料處理框架:Flink v1.18.1 |
| 10 | + |
| 11 | +## Run |
| 12 | +使用 IntelliJ IDEA |
| 13 | + |
| 14 | + |
| 15 | +## Entry |
| 16 | + |
| 17 | +### 1. KafkaToKafka |
| 18 | + |
| 19 | +將 Kafka (localhost:9092) 中 `topic-source` 的所有產品訊息傳輸到 Kafka 中的 `topic-sink`。 |
| 20 | + |
| 21 | +- Kafka 主題 `topic-source` 訊息範例 |
| 22 | +```json |
| 23 | +{ |
| 24 | + "id": "12345", |
| 25 | + "name": "Wireless Mouse", |
| 26 | + "category": "Electronics", |
| 27 | + "manufacturer": "TechCorp", |
| 28 | + "description": "A sleek, ergonomic wireless mouse with advanced optical tracking.", |
| 29 | + "cost": 29.99 |
| 30 | +} |
| 31 | +``` |
| 32 | + |
| 33 | +### 2. KafkaToDoris |
| 34 | + |
| 35 | +將 Kafka (localhost:9092) 中 `topic-product` 的所有訊息轉換後插入 Doris (localhost:9030) 資料庫 `database.product` 中。 |
| 36 | + |
| 37 | +- Kafka 主題 `topic-product` 訊息範例 |
| 38 | +```json |
| 39 | +{ |
| 40 | + "id": "12345", |
| 41 | + "name": "Wireless Mouse", |
| 42 | + "category": "Electronics", |
| 43 | + "manufacturer": "TechCorp", |
| 44 | + "description": "A sleek, ergonomic wireless mouse with advanced optical tracking.", |
| 45 | + "cost": 29.99 |
| 46 | +} |
| 47 | +``` |
| 48 | + |
| 49 | +- Doris 資料表 `database.product` |
| 50 | +``` |
| 51 | +| id | name | category | manufacturer | description | cost | |
| 52 | +|---------|-----------------|-------------|---------------|-----------------------------------|-------| |
| 53 | +| 12345 | Wireless Mouse | Electronics | TechCorp | A sleek, ergonomic wireless mouse | 29.99 | |
| 54 | +``` |
| 55 | + |
| 56 | +### 3. TwoKafkaToDoris |
| 57 | + |
| 58 | +將 Kafka (localhost:9092) 中 `topic-product` 與 `topic-sale` 的訊息結合後,傳輸到 Doris (localhost:9030) 資料庫 `database.sale_report` 中。 |
| 59 | + |
| 60 | +- Kafka 主題 `topic-product` 訊息範例 |
| 61 | +```json |
| 62 | +{ |
| 63 | + "id": "12345", |
| 64 | + "name": "Wireless Mouse", |
| 65 | + "category": "Electronics", |
| 66 | + "manufacturer": "TechCorp", |
| 67 | + "description": "A sleek, ergonomic wireless mouse with advanced optical tracking.", |
| 68 | + "cost": 29.99 |
| 69 | +} |
| 70 | +``` |
| 71 | + |
| 72 | +- Kafka 主題 `topic-sale` 訊息範例 |
| 73 | +```json |
| 74 | +{ |
| 75 | + "id": "A98765", |
| 76 | + "productId": "12345", |
| 77 | + "unit": 3, |
| 78 | + "unitPrice": 49.99, |
| 79 | + "totalPrice": 149.97, |
| 80 | + "saleDate": "2024年11月28日" |
| 81 | +} |
| 82 | +``` |
| 83 | + |
| 84 | +- Doris 資料表 `database.sale_report` |
| 85 | +``` |
| 86 | +| sale_id | product_id | unit | unit_price | total_price | sale_date | product_name | product_unit_cost | profit | |
| 87 | +|-----------|------------|------|------------|-------------|----------------------|---------------|-------------------|--------| |
| 88 | +| A98765 | 12345 | 3 | 49.99 | 149.97 | 2024年11月28日T08:00:00 | Wireless Mouse| 29.99 | 60.00 | |
| 89 | +``` |
0 commit comments