This repository provides code to experiment with training LLMs on the Moreh's MoAI Platform. With MoAI platform you can scale to thousands of GPU/NPUs by automatic parallelization and optimization, without any code changes.

We currently provide 4 LLMs; Qwen1.5, Llama2, Mistral, GPT and Baichuan2.