You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: readme.md
+13-5Lines changed: 13 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,15 +5,17 @@ Examples will include analytic models built with TensorFlow, Keras, H2O, Python,
5
5
6
6
![Kafka Open Source Ecosystem for a Scalable Mission Critical Machine Learning Infrastructure](http://www.kai-waehner.de/blog/wp-content/uploads/2017/10/Apache_Kafka_Ecosystem_Kafka_Streams_Machine_Learning.png"Kafka Open Source Ecosystem for a Scalable Mission Critical Machine Learning Infrastructure")
7
7
8
-
9
8
## Material (Blogs Posts, Slides, Videos)
9
+
10
10
Here is some material about this topic if you want to read and listen to the theory instead of just doing hands-on:
11
+
11
12
- Blog Post: [How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka](https://www.confluent.io/blog/build-deploy-scalable-machine-learning-production-apache-kafka/)
12
13
- Slide Deck: [Apache Kafka + Machine Learning => Intelligent Real Time Applications](https://www.slideshare.net/KaiWaehner/apache-kafka-streams-machine-learning-deep-learning)
13
14
- Slide Deck: [Deep Learning at Extreme Scale (in the Cloud) with the Apache Kafka Open Source Ecosystem](https://www.slideshare.net/KaiWaehner/deep-learning-at-extreme-scale-in-the-cloud-with-the-apache-kafka-open-source-ecosystem)
14
15
- Video Recording: [Deep Learning in Mission Critical and Scalable Real Time Applications with Open Source Frameworks](https://vimeo.com/jaxtv/review/256406763/7fbf4213be)
15
16
- Blog Post: [Using Apache Kafka to Drive Cutting-Edge Machine Learning - Hybrid ML Architectures, AutoML, and more...](https://www.confluent.io/blog/using-apache-kafka-drive-cutting-edge-machine-learning)
16
17
- Blog Post: [Machine Learning with Python, Jupyter, KSQL and TensorFlow](https://www.confluent.io/blog/machine-learning-with-python-jupyter-ksql-tensorflow)
18
+
- Blog Post: [Streaming Machine Learning with Tiered Storage and Without a Data Lake](https://www.confluent.io/blog/streaming-machine-learning-with-tiered-storage/)
17
19
18
20
## Use Cases and Technologies
19
21
@@ -35,22 +37,28 @@ Here is some material about this topic if you want to read and listen to the the
35
37
36
38
##### Some other Github projects exist already with more ML + Kafka content:
37
39
40
+
The most exciting and powerful example first:
41
+
[Streaming Machine Learning at Scale from 100000 IoT Devices with HiveMQ, Apache Kafka and TensorFLow](https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference)
42
+
43
+
Here some more demos:
44
+
38
45
- Deep Learning UDF for KSQL: [Streaming Anomaly Detection of MQTT IoT Sensor Data using an Autoencoder](https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot)
39
46
- End-to-End ML Integration Demo: [Continuous Health Checks with Anomaly Detection using KSQL, Kafka Connect, Deep Learning and Elasticsearch](https://github.com/kaiwaehner/ksql-fork-with-deep-learning-function)
40
47
- TensorFlow Serving + gRPC + Kafka Streams on Github => Stream Processing and RPC / Request-Response concepts combined: [Model inference with Apache Kafka, Kafka Streams and a TensorFlow model deployed on a TensorFlow Serving model server](https://github.com/kaiwaehner/tensorflow-serving-java-grpc-kafka-streams)
48
+
- Solving the impedance mismatch between Data Scientist and Production Engineer: [Python, Jupyter, TensorFlow, Keras, Apache Kafka, KSQL](https://github.com/kaiwaehner/python-jupyter-apache-kafka-ksql-tensorflow-keras)
41
49
42
50
## Requirements, Installation and Usage
43
51
The code is developed and tested on Mac and Linux operating systems. As Kafka does not support and work well on Windows, this is not tested at all.
44
52
45
53
Java 8 and Maven 3 are required. Maven will download all required dependencies.
46
54
47
-
Just download the project and run
55
+
Just download the project and run
48
56
49
57
mvn clean package
50
-
51
-
You can do this in main directory or each module separately.
52
58
53
-
Apache Kafka 2.1 is currently used. The code is also compatible with Kafka and Kafka Streams 1.1 and 2.0.
59
+
You can do this in main directory or each module separately.
60
+
61
+
Apache Kafka 2.5 is currently used. The code is also compatible with Kafka and Kafka Streams 1.1 and 2.x.
54
62
55
63
**Please make sure to run the Maven build without any changes first.** If it works without errors, you can change library versions, Java version, etc. and see if it still works or if you need to adjust code.
0 commit comments