Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

AWS-Big-Data-Projects/Iot-and-Big-Data-Application-using-aws-and-apache-kafka

Folders and files

NameName
Last commit message
Last commit date

Latest commit

History

5 Commits

Repository files navigation

Iot-and-Big-Data-Application-using-aws-and-apache-kafka

Internet of things , Big Data Analytics using Aws-kafka, spark and other aws services.

image

image

To run spark job on your local machine, you need to setup Kafka and create a producer first, see http://kafka.apache.org/documentation.html#quickstart and then run the example $ bin/spark-submit --jars \ external/kafka-assembly/target/scala-*/spark-streaming-kafka-assembly-*.jar \ kafka-direct-iot-sql.py \ localhost:9092 test

Steps

1.Define function to process RDDs of the json DStream to convert them to DataFrame and run SQL queries

2.Process each RDD of the DStream coming in from Kafka

3.Set number of simulated messages to generate

4.Generate JSON output

About

Iot,Big Data Analytics using Apache-kafka,spark and other aws services

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

AltStyle によって変換されたページ (->オリジナル) /