This repo is a tiny language model mimicking LLMs. I would like to create LLM-like model from scratch. To some degree LLM could also be abbreviated to little language model. :D

• Collect Data
• Clean & Augment Data
• Make Tokenizer
• Make Model Based on Transformer
• Pretrain Model
• LLM Alignment

• Transfer to downstream tasks.
• SFT Training
• Distribute Data Parallel
• Distill model
• RAG
• Agent

1. LLM数据工程3——数据收集魔法:获取顶级训练数据的方法
2. 掌握 LLM 技术:数据预处理