Apache Spark

Apache Spark is an open source distributed general-purpose cluster-computing framework. It provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.

Here are 9,290 public repositories matching this topic...

Language: All

Filter by language

All 9,290 Python 1,984 Scala 1,938 Jupyter Notebook 1,929 Java 1,193 Shell 338 HTML 260 JavaScript 181 Dockerfile 127 TypeScript 62 R 56

Sort: Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

apache / spark

Star 42.1k

Apache Spark - A unified analytics engine for large-scale data processing

python java r scala sql big-data spark jdbc

Updated Oct 16, 2025
Scala

DataTalksClub / data-engineering-zoomcamp

Star 33.1k

Data Engineering Zoomcamp is a free nine-week course that covers the fundamentals of data engineering.

docker kafka spark data-engineering dbt kestra

Updated Oct 15, 2025
Jupyter Notebook

donnemartin / data-science-ipython-notebooks

Star 28.6k

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

python aws data-science machine-learning caffe theano big-data spark deep-learning hadoop tensorflow numpy scikit-learn keras pandas kaggle scipy matplotlib mapreduce

Updated Mar 20, 2024
Python

getredash / redash

Star 27.9k

Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.

visualization javascript mysql python bigquery bi spark dashboard athena analytics postgresql business-intelligence redash redshift databricks hacktoberfest spark-sql

Updated Oct 15, 2025
Python

yeasy / docker_practice

Star 25.6k

Learn and understand Docker&Container technologies, with real DevOps practice!

linux docker kubernetes devops spark book container mesos swarm cloud-computing

Updated Dec 26, 2024
Go

heibaiying / BigData-Notes

Star 16.7k

大数据入门指南 ⭐

phoenix scala kafka big-data spark yarn hive hadoop storm bigdata hbase zookeeper hdfs mapreduce flume azkaban sqoop

Updated Jan 5, 2024
Java

FavioVazquez / ds-cheatsheets

Star 15.7k

List of Data Science Cheatsheets to rule the world

python r programming spark jupyter datascience cheatsheet

Updated Jul 18, 2024

GaiZhenbiao / ChuanhuChatGPT

Star 15.4k

GUI for ChatGPT API and many LLMs. Supports agents, file-based QA, GPT finetuning and query with web search. All with a neat UI.

spark chatbot gemini llama minimax moss gemma claude ernie midjourney chatgpt-api chatglm stablelm ollama qwen dalle3 inspurai

Updated Aug 15, 2025
Python

flink-learning

zhisheng17 / flink-learning

Star 14.9k

flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例,还有 Flink 落地应用的大型项目案例(PVUV、日志存储、百亿数据实时去重、监控告警)分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》

mysql redis elasticsearch streaming kafka spark influxdb rabbitmq clickhouse hbase stream-processing opentsdb loki flink rocketmq

Updated Mar 12, 2025
Java

aalansehaiyang / technology-talk

Star 14.6k

【大厂面试专栏】一份Java程序员需要的技术指南,这里有面试题、系统架构、职场锦囊、主流中间件等,让你成为更牛的自己!

git java kafka spark spring es6 hbase springboot dubbo mycat

Updated Jul 21, 2025

horovod / horovod

Star 14.6k

Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.

machine-learning spark deep-learning uber mxnet tensorflow mpi keras pytorch machinelearning baidu deeplearning ray

Updated Sep 23, 2025
Python

apache / doris

Star 14.4k

Apache Doris is an easy-to-use, high performance and unified analytics database.

agent bigquery real-time sql database ai spark snowflake olap query-engine redshift dbt elt iceberg hudi delta-lake lakehouse paimon

Updated Oct 16, 2025
Java

deeplearning4j / deeplearning4j

Star 14.1k

Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learn...

python java clojure scala spark hadoop gpu intellij linear-algebra artificial-intelligence deeplearning neural-nets dl4j matrix-library deeplearning4j

Updated Oct 13, 2025
Java

wangzhiwubigdata / God-Of-BigData

Star 10.3k

专注大数据学习面试,大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...

kafka spark hive hadoop bigdata hbase zookeeper hdfs flume flink azkaban

Updated Aug 7, 2023

mage-ai / mage-ai

Star 8.5k

🧙 Build, run, and manage data pipelines for integrating and transforming data.

python data-science data machine-learning sql spark pipeline etl pipelines orchestration artificial-intelligence data-engineering data-integration dbt elt transformation data-pipelines reverse-etl

Updated Oct 13, 2025
Python

tobymao / sqlglot

Star 8.4k

Python SQL Parser and Transpiler

mysql python bigquery parser postgres sql spark presto hive clickhouse sqlite snowflake optimizer transpiler redshift databricks tsql trino sqlparser duckdb

Updated Oct 16, 2025
Python

delta-io / delta

Star 8.3k

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

big-data spark analytics acid delta-lake

Updated Oct 15, 2025
Scala

h2oai / h2o-3

Star 7.3k

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.