fsdp

Star

Here are 79 public repositories matching this topic...

Language: All

Filter by language

All 79 Python 62 Jupyter Notebook 8 HTML 3 C++ 1 JavaScript 1 Shell 1 TypeScript 1

Sort: Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

sgl-project / SpecForge

Star 1.2k

Train speculative decoding models effortlessly and port them smoothly to SGLang serving.

training eagle pytorch llm fsdp sglang eagle3

Updated Sep 7, 2026
Python

LambdaLabsML / distributed-training-guide

Star 630

Best practices & guides on how to write distributed pytorch training code

gpu cluster mpi cuda slurm pytorch sharding kuberentes distributed-training nccl gpu-cluster deepspeed fsdp lambdalabs

Updated Oct 22, 2025
Python

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

rust cuda pytorch triton quantization knowledge-distillation inference-engine jax kv-cache ml-systems llm mechanistic-interpretability fsdp flash-attention speculative-decoding paged-attention

Updated Jun 5, 2026
Python

meta-pytorch / torchft

Star 538

Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)

training distributed-systems fault-tolerance consensus ml torch pytorch hsdp fsdp localsgd diloco

Updated Aug 28, 2026
Python

tsiendragon / qwen-image-finetune

Star 253

Repo for Qwen Image Finetune

lora image-to-image diffusion-models fsdp peft-fine-tuning-llm flux-kontext qwen-image-edit image-edit-model qwen-image-edit-2509

Updated Aug 22, 2026
Jupyter Notebook

lightseekorg / TorchSpec

Sponsor

Star 243

A PyTorch native library for training speculative decoding models

pytorch mooncake llm vllm fsdp tokenspeed eagle3 lightseek dspark

Updated Sep 4, 2026
Python

traceopt-ai / traceml

Star 186

Open-source performance diagnostics for PyTorch training runs.

machine-learning deep-learning gpu cuda slurm pytorch dataloader performance-analysis profiling performance-monitoring ray ddp memory-leak observability distributed-training mlops pytorch-lightning hugging-face fsdp llm-training

Updated Sep 7, 2026
Python

saforem2 / ezpz

Star 41

Write once, run anywhere; ezpz 🍋

python training machine-learning ai launcher mpi slurm torch pytorch parallelism rich mpi4py distributed-training fsdp

Updated Aug 31, 2026
Python

debnsuma / ray-for-developers

Star 31

A comprehensive hands-on guide to building production-grade distributed applications with Ray - from distributed training and multimodal data processing to inference and reinforcement learning.

machine-learning deep-learning distributed-computing pytorch ray ddp model-serving distributed-training multimodal mlops fsdp

Updated Feb 12, 2026
Jupyter Notebook

ruimalheiro / gradient-garden

Star 27

Research platform for model training, evaluation, and experimentation across architectures, benchmarks, and recipes.

benchmarking machine-learning research deep-learning evaluation transformer experimentation lora ddp knowledge-distillation distributed-training finetuning sft dpo mixture-of-experts pretraining llm instruction-tuning fsdp causal-lm

Updated Sep 3, 2026
Python

GURPREETKAURJETHRA / Meta-LLAMA3-GenAI-UseCases-End-To-End-Implementation-Guides

Star 26

META LLAMA3 GENAI Real World UseCases End To End Implementation Guide

pytorch fine-tuning sagemaker rag huggingface streamlit prompt-tuning generativeai langchain-python chromadb fsdp qlora ollama llama3 llama3-prompts llama3-rag llama3-finetune llama3-meta-ai llama3-70b-8192

Updated Sep 24, 2024
Jupyter Notebook

AlibabaPAI / FlashModels

Star 12

Fast and easy distributed model training examples.

deep-learning pytorch zero data-parallelism model-parallelism distributed-training xla tensor-parallelism llm fsdp sequence-parallelism

Updated Nov 26, 2024
Python

BillusA1111 / billus-model-skill-library

Star 11

Billus 大模型技能库:面向 LLM、VL、多模态与图像生成模型的训练、调参、剪枝、量化与工程化技能库

training skill diffusion codex vl peft multimodal huggingface deepspeed llm fsdp qlora

Updated Mar 12, 2026
Python

acforvs / build-your-trace

Star 10

Interactive exercises for understanding compute, communication, and dependencies in distributed LLM training traces.

cuda pytorch distributed-training nccl fsdp llm-training gpu-profiling

Updated Aug 26, 2026
JavaScript

liangyuwang / Tinytron

Star 10

A minimal, hackable pre-training stack for GPT-style language models

pytorch megatron fsdp

Updated Aug 3, 2026
Python

Forge-researchlab / forge-kernels

Star 9

Forge kernels — fused Triton kernels for faster, leaner LLM fine-tuning. One-call patching into Hugging Face models via forge.patch(model), with FSDP2 multi-GPU support and a growing set of kernels and architectures. Every speedup backed by a committed benchmark. Apache-2.0.

cuda transformers pytorch triton lora gemma gpu-kernels fine-tuning gpu-optimization huggingface llm fsdp qwen

Updated Aug 15, 2026
Python

arawxx / FSDP-Distributed-Training-of-ConvNextV2-on-CIFAR10

Star 8

A script for training the ConvNextV2 on CIFAR10 dataset using the FSDP technique for a distributed training scheme.

deep-learning distributed-computing pytorch distributed distributed-training distributed-learning convnext convnextv2 fsdp fully-sharded-data-parallel

Updated Dec 11, 2023
Python

zigzagcai / DeepSeekV3

Star 8

Simple and efficient implementation of 671B DeepSeek V3 that trainable with FSDP+EP and minimal requirement of 256x A100/H100, targeted for HuggingFace ecosystem

reinforcement-learning fsdp expert-parallel

Updated Jan 15, 2026
Python

liangyuwang / train-large-model-from-scratch

Star 7

A minimal, hackable pre-training stack for GPT-style language models

gpt megatron llms fsdp

Updated Feb 21, 2026
Python

SulRash / minLLMTrain

Star 6

Minimal yet high performant code for pretraining llms. Attempts to implement some SOTA features. Implements training through: Deepspeed, Megatron-LM, and FSDP. WIP

huggingface pretraining deepspeed megatron-lm llm fsdp

Updated Feb 6, 2024
Python

Add this topic to your repo

To associate your repository with the fsdp topic, visit your repo's landing page and select "manage topics."

Learn more

Navigation Menu