mvGPT

Minimum Viable GPT — a small decoder-only transformer coded from scratch and readable end to end. Roughly 14.6M parameters, trained on TinyStories, carried through the whole modern LLM pipeline: pretraining → SFT / LoRA → DPO → evaluation, plus KV-cache inference. Every stage is a single, heavily commented Python file you can read in one sitting.

Watch the walkthroughs

These two videos are the guided tour — start here, then dig into the pages.

Architecture — how the model is built

Watch on YouTube Read the architecture page

Pretraining — training it from scratch

Watch on YouTube Read the pretraining page

Full series

Both videos are part of the mvGPT playlist .

The pipeline

Each stage is one file that loads the previous stage's checkpoint and writes its own. You can stop anywhere and still have a working model.

%%{init: {'theme':'dark','flowchart':{'htmlLabels':false},'themeVariables':{'fontFamily':'Helvetica, Arial, sans-serif','lineColor':'#8b949e','edgeLabelBackground':'#161b22'}}}%%
flowchart LR
    D["TinyStories<br/>corpus"] --> PT["pretrain.py<br/>next-token LM"]
    PT --> B(["ckpt_base.pt"])
    B --> SFT["sft.py<br/>instruction tuning"]
    B --> LoRA["lora.py<br/>low-rank adapters"]
    SFT --> S(["ckpt_sft.pt"])
    LoRA --> L(["ckpt_lora.pt"])
    S --> DPO["dpo.py<br/>preference optimization"]
    DPO --> P(["ckpt_dpo.pt"])
    B --> EV["eval.py<br/>compare stages"]
    S --> EV
    L --> EV
    P --> EV

    classDef data fill:#16324f,stroke:#5aa9f8,color:#e6edf3;
    classDef stage fill:#4a2230,stroke:#ff9eb1,color:#e6edf3;
    classDef ckpt fill:#16402a,stroke:#5fd98a,color:#e6edf3;
    class D data;
    class PT,SFT,LoRA,DPO,EV stage;
    class B,S,L,P ckpt;

• Architecture — the transformer, layer by layer, with data-flow and parameter diagrams.
• Pretraining — next-token prediction on TinyStories.
• Fine-tuning — instruction tuning with full SFT and with LoRA.
• Preference Optimization — DPO from programmatic rewards.
• Inference & KV Cache — why generation is O(T²), and the fix.
• Evaluation — comparing every stage side by side.

Model at a glance

Property	Value
Parameters	14,585,856 (~14.6M)
Vocabulary	10,000 (word-level tokenizer)
Context length	256 tokens
Layers	6
Attention heads	6
Embedding dimension	384
MLP hidden dimension	1,536 (4×)
Activation	GELU
Normalization	Pre-norm LayerNorm
Weight tying	token embedding ↔ output head
Dataset	TinyStories / TinyStories-Instruct

Defined once in GPTConfig (base_model.py), which fixes every parameter shape so a checkpoint only loads into a model built with the same config.

Quickstart

git clone https://github.com/nenuadrian/mvGPT
cd mvGPT

python -m venv .venv && source .venv/bin/activate
pip install torch numpy requests

# Stage 1: download TinyStories, build the tokenizer, pretrain (10 min budget)
python pretrain.py 10

# Stage 2: instruction fine-tuning (5 min budget)
python sft.py 5

# Compare what each stage learned
python eval.py

Every training script takes a wall-clock budget in minutes as its argument, so you can get a feel for the model on a laptop and scale up the time on a GPU.

Device

The code auto-selects CUDA, then Apple Silicon (MPS), then CPU — no flags needed. TinyStories is small enough that even CPU pretraining produces fluent toy stories within minutes.