Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

transformers moe data-parallelism distributed-optimizers model-parallelism megatron mixture-of-experts pipeline-parallelism huggingface-transformers megatron-lm tensor-parallelism large-scale-language-modeling 3d-parallelism zero-1 sequence-parallelism

• Updated Dec 14, 2023
• Python

GPT-2 (124M) fixed-work distributed training benchmark on NYU BigPurple (Slurm) scaling ×ばつ V100 across 2 nodes using DeepSpeed ZeRO-1 + FP16/AMP. Built a reproducible harness that writes training_metrics.json + RUN_COMPLETE.txt + launcher metadata per run, plus NCCL topology/log artifacts and Nsight Systems traces/summaries (NVTX + NCCL ranges).

performance hpc amp slurm pytorch reproducibility distributed-training mixed-precision gpt2 deepspeed zero-1 gpu-benchmarking

• Updated Feb 4, 2026
• Python