AI Models Banner

A collection of language model experiments built from scratch — each one exploring a different technique, architecture, or dataset.

The goal is hands-on understanding: no pre-built pipelines, just raw implementation, training, and deployment.

All content is suspect — this repo is created by someone who knows nothing... yet.

GPT-style causal transformer built from scratch with PyTorch, trained on the STACK construction estimating platform support knowledge base. Implements BPE tokenization, multi-head self-attention, and causal language modelling with a ~5.8M parameter model targeting Raspberry Pi deployment via GGUF/Ollama.

Tokenization

• T1a — BPE 2K
• T1b — BPE 5K
• T1c — SentencePiece 8K
• T1d — Byte-level BPE 10K
• T1e — Morfessor (morphological)

Data Preparation

• T2a — Synthesized Q&A pairs
• T2b — EDA augmentation
• T2c — EDA + Q&A

Architecture

• T3a — Seq2Seq (encoder-decoder)
• T3b — BERT retrieval

All models are built to run on a Raspberry Pi cluster using the Raspberry Pi AI HAT+ 2 (Hailo-10H, 10 TOPS).