目的:基于DeepSpeed,突破硬件限制,实现大模型高效训练。
- cifar示例
- 分布式数据并行DDP的训练pipeline
- MoE用法
- 学习率调度器的配置
- ZeRO零冗余优化器的配置
- pipeline_parallelism示例
- 流水并行的训练pipeline
- 流水模型的保存、加载、指标评估
- TensorBoard可视化
-
Notifications
You must be signed in to change notification settings - Fork 5
DeepSpeed教程 & 示例注释 & 学习笔记 (大模型高效训练)
License
Notifications
You must be signed in to change notification settings
bobo0810/LearnDeepSpeed
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Latest commit | ||||
Repository files navigation
About
DeepSpeed教程 & 示例注释 & 学习笔记 (大模型高效训练)
Topics
Resources
License
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published