hadoop-hdfs

Star

Here are 323 public repositories matching this topic...

Language: All

Filter by language

All 323 Java 90 Python 62 Jupyter Notebook 41 Shell 25 Scala 11 JavaScript 8 Jinja 6 HiveQL 5 Dockerfile 4 R 3

Sort: Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

seaweedfs

seaweedfs / seaweedfs

Star 25.7k

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, xDC replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. Enterprise version is at seaweedfs.com.

kubernetes distributed-systems fuse replication cloud-drive s3 posix s3-storage hdfs distributed-storage distributed-file-system erasure-coding object-storage blob-storage seaweedfs hadoop-hdfs tiered-file-system

Updated Sep 2, 2025
Go

OBenner / data-engineering-interview-questions

Star 1.4k

More than 2000+ Data engineer interview questions.

aws airflow sql kafka spark cassandra hive hadoop avro azure impala interview hbase data-structures data-engineering flume interview-questions nifi flink hadoop-hdfs

Updated Jan 26, 2025

Morphl-AI / MorphL-Community-Edition

Star 262

MorphL Community Edition uses big data and machine learning to predict user behaviors in digital products and services with the end goal of increasing KPIs (click-through rates, conversion rates, etc.) through personalization

kubernetes machine-learning cassandra pipeline artificial-intelligence pyspark user-experience data-driven-design conversion-rate-optimization front-end-development product-development hadoop-hdfs morphl-platform

Updated Oct 2, 2019
Python

linkedin / dynamometer

Star 132

A tool for scale and performance testing of HDFS with a specific focus on the NameNode.

testing hadoop scale performance-metrics hdfs testing-tools performance-analysis hdfs-dfs performance-test performance-testing hadoop-filesystem hadoop-framework scale-up hadoop-hdfs

Updated Jan 11, 2024
Java

Data-Engineering-Project-with-HDFS-and-Kafka

AhmetFurkanDEMIR / Data-Engineering-Project-with-HDFS-and-Kafka

Sponsor

Star 115

Data Engineering Project with Hadoop HDFS and Kafka

python docker data kafka hadoop docker-compose data-engineering kafka-consumer hdfs kafka-producer hdfs-dfs hadoop-filesystem data-engineer hadoop-hdfs hdfs-client python-hdfs-client pipline data-engineering-pipeline kafka-ui kafkaui

Updated Nov 4, 2023
Python

big_data

groda / big_data

Star 81

Big Data essentials: Hadoop, MapReduce, Spark. Explore tutorials and demos in Jupyter notebooks—most are self-contained and live, ready to run with a click.

docker big-data spark apache-spark hadoop bigdata jupyter-notebook pyspark hadoop-cluster mapreduce gutenberg-ebooks hadoop-mapreduce spark-sql mrjob bigtop hadoop-hdfs testdfsio mapreduce-bash apache-sedona

Updated Jul 27, 2025
Jupyter Notebook

IBM / sparksql-for-hbase

Star 69

Learn how to use Spark SQL and HSpark connector package to create / query data tables that reside in HBase region servers

sql spark apache-spark nosql hbase ibmcode hadoop-hdfs

Updated Oct 18, 2022

vim89 / datapipelines-essentials-python

Star 55

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations