A curated list of awesome System Design (A.K.A. Distributed Systems) resources.

• Updated Jun 27, 2024

DE직무에 필요한 모든 것

• Updated May 22, 2025
• Jupyter Notebook

HadoopOffice - Analyze Office documents using the Hadoop ecosystem (Spark/Flink/Hive)

• Updated Sep 29, 2025
• Java

IBIS is a workflow creation-engine that abstracts the Hadoop internals of ingesting RDBMS data.

• Updated Apr 13, 2022
• Python

Life-cycle: Internal working of HDFS, SQOOP, HIVE, SPARK, HBASE, KAFKA with code.

• Updated Sep 10, 2019
• Shell

Hadoop3.2 single/cluster mode with web terminal gotty, spark, jupyter pyspark, hive, eco etc.

• Updated Nov 7, 2019
• Shell

Instructions on setting up Hadoop, HDFS, java, sbt, kafka, scala, spark and flume on Ubuntu 18.04

• Updated Jul 17, 2021
• Shell

Dockerfile for running Apache Knox (http://knox.apache.org/) in Docker

• Updated Mar 21, 2022
• Dockerfile

This project sets up a Hadoop High Availability (HA) Cluster using Docker Compose with three master nodes and two worker nodes for fault-tolerant big data processing. It includes Zookeeper & JournalNodes for automatic NameNode failover, ensuring scalability & reliability.

• Updated May 26, 2025
• Shell

The goal of this project is to identify the flood-prone areas with probabilities of flood in counties in a future date, using Spark MLLib.

• Updated Jan 20, 2020
• Scala

Analysis of YouTube Data using Hadoop Mapreduce framework in Java.

• Updated Jan 30, 2022
• Java

Helm chart for Apache Knox

• Updated Sep 28, 2022
• Mustache

SparkSQL Quick Start Tutorial

• Updated Oct 7, 2017
• Scala

Built a Large Scale Distributed Data Processing system for Streaming Analytics using Hadoop Ecosystem (Apache Spark and HDFS), in Cloud for real-time spatial analytics.

• Updated Jun 4, 2021
• Scala

EMR 5.25.0 cluster single node Hadoop docker image. With Amazon Linux, Hadoop 2.8.5 and Hive 2.3.5

• Updated Jan 6, 2020
• Shell

Big Data is Stored and analyzed of various Customer using Hadoop and other tools like Hive, Zookeeper, Hbase and sqoop and all details of the customer is analyzed then result are given.This result is very useful for companies.

• Updated Feb 10, 2021

Comprehensive Big Data Engineering learning repository featuring hands-on projects with Hadoop, Spark, Kafka, Docker, Airflow, and Azure Cloud. Includes end-to-end data pipelines, real-time streaming, and distributed processing implementations.

• Updated Oct 7, 2025
• Jupyter Notebook

[BigData] one year weblog analysis using PIG

• Updated Jan 11, 2018
• PigLatin

This project focuses on analyzing movie data using Pyspark tailored for efficient data processing on Hadoop Distributed File System (HDFS)

• Updated May 6, 2024
• Jupyter Notebook

Practise programs in hadoop ecosystem for refrence

• Updated Sep 14, 2018