Learning roadmap

Archived curriculum scaffold. The active study queue is docs/learning/new-things.md.

The path to learning the whole stack without drowning in complexity. This is the curriculum; model_guide.md, lora_guide.md, and browser_notes.md are the implementation detail.

For external course/company-doc reading, use industry_learning_roadmap.md. It makes Stanford CS336 the course spine and maps company docs from Hugging Face, Ai2, Anthropic, OpenAI, Poolside, Apple, Meta, Qwen, DeepSeek, Mistral, and Lamina back to concrete TinyGPT work.

This roadmap is also mirrored into the swe-interview-prep fleet project as 19 FSRS-tracked ml-* concepts (swe-interview-prep/docs/TINYGPT_LEARNING_PATH.md).

The 9 phases

Phase 1 — Core ML foundations

Goal: understand what the model is optimizing. Learn: vectors, matrices, dot products, matmul, softmax, cross entropy, gradient descent, backpropagation, Adam/AdamW, train/val split, overfitting. Do: implement softmax and cross entropy from scratch; train logistic regression or a tiny MLP. Ready when: you can explain forward pass, loss, backward pass, and optimizer step without handwaving. External sources:

• 3Blue1Brown — Essence of Linear Algebra (chs 1-4: vectors, matmul)
• 3Blue1Brown — Backpropagation
• Karpathy — Spelled-out intro to neural networks and backpropagation (builds a tiny autograd by hand) Concepts: ml-math, ml-gradient-descent, ml-backprop, ml-softmax-xent, ml-adamw

Phase 2 — Language modeling basics

Goal: understand next-token prediction. Learn: tokenization, byte/char-level tokens, context windows, causal masking, sampling, temperature, top-k, perplexity. Do: build a bigram model and a char-level MLP; generate text; measure train/val loss. Key idea: a language model is not “answering” — it is predicting the next token. External sources:

• 3Blue1Brown — But what is a GPT? (25 min big-picture)
• Karpathy — makemore series (bigram → char-level MLP → transformer, notebook-by-notebook)
• GPT-1/2/3 papers: Radford et al. 2018, Radford et al. 2019, Brown et al. 2020 Concepts: ml-tokenization, ml-language-modeling, ml-sampling

Phase 3 — Transformer / GPT internals

Goal: understand the architecture. Learn: embeddings, position embeddings, self-attention, causal attention, multi-head attention, layernorm, residuals, MLP blocks, GELU, logits. Do: build one transformer block, then a full TinyGPT; train 0.8M on a small corpus; overfit 10 KB; generate samples. External sources:

• Attention is All You Need — Vaswani et al. 2017 (the original transformer)
• The Annotated Transformer (Harvard NLP) (Vaswani paper, code-annotated)
• Jay Alammar — The Illustrated Transformer (the visual canon)
• Karpathy — Let’s build GPT, from scratch
• RoPE — Su et al. 2021 (rotary position embeddings — what modern models use instead of learned positions)
• Implemented at: python_ref/model.py, wasm/src/attention.cpp, webgpu/attention_fa2.wgsl. Concepts: ml-embeddings, ml-self-attention, ml-multi-head, ml-transformer-block

Phase 4 — Training & debugging

Goal: know why training fails. Learn: initialization, learning rate, gradient clipping, NaNs, exploding gradients, under/overfitting, batch size effects, validation loss, checkpointing, seed reproducibility. Do: build the test suite — loss near ln(vocab), tiny overfit, gradient check, checkpoint reload, fixed-seed generation. This is where most beginners lie to themselves. If loss is not falling on tiny data, the model is broken — do not scale. Concepts: ml-training, ml-checkpointing

Phase 5 — LoRA & PEFT

Goal: understand parameter-efficient adaptation. Learn: full fine-tuning, frozen base models, adapters, LoRA, rank, alpha, target modules, adapter checkpoints, memorization, style-transfer limits. Do: build a LoRALinear; inject into q_proj/v_proj; freeze base; train adapter only; save/load adapter; compare base vs LoRA outputs. External sources:

• LoRA — Hu et al. 2021 (the original)
• QLoRA — Dettmers et al. 2023 (int4 base + fp16 LoRA)
• DoRA — Liu et al. 2024 (magnitude + direction decomposition)
• Implemented at: python_ref/lora.py; mechanics + maths in lora_guide.md. Concepts: ml-lora

Phase 6 — Data engineering for style adaptation

Goal: prepare clean text that teaches the model something specific. Learn: text cleaning, deduplication, train/val split, dataset hashing, prompt formatting, continuation/rewrite/title/Q&A examples, memorization tests. Do: produce a clean JSONL dataset (300–1,000 examples) of structured tasks. Concepts: ml-data-engineering

Phase 7 — Browser systems

Goal: move training/inference into the browser safely. Learn: TypedArray, ArrayBuffer, Web Workers, transferable objects, WASM, Emscripten, WASM SIMD, OPFS, IndexedDB, storage quota, feature detection. Do: a browser app that trains a tiny model in a Worker; UI stays responsive; checkpoint survives reload. External sources:

• MDN — WebAssembly concepts
• MDN — Using Web Workers
• MDN — OPFS
• emscripten.org (pthreads + SAB build flags)
• Implemented at: browser/src/worker.ts, wasm/src/, browser_notes.md. Concepts: ml-browser-runtime

Phase 8 — WebGPU

Goal: accelerate tensor-heavy operations. Learn: WGSL, GPU buffers, bind groups, compute pipelines, command encoders, workgroups, matmul/softmax/attention kernels, device limits, buffer sharding. Do: WebGPU matmul → CPU parity → linear forward → linear backward → attention scores → softmax → attention output → optimizer. Start with matmul only. External sources:

• WebGPU Fundamentals (compute articles)
• WebGPU spec (W3C)
• WGSL spec
• FlashAttention — Dao et al. 2022 and FlashAttention-2 — Dao 2023 (the online-softmax + recomputation trick we use in WGSL)
• Implemented at: webgpu/matmul_blocked.wgsl, webgpu/attention_fa2.wgsl; perf walk-through in perf_quest.md. Concepts: ml-webgpu

Phase 9 — Evaluation & safety

Goal: know what the model is actually doing. Learn: validation loss, held-out prompts, qualitative evaluation, memorization testing, style similarity, hallucination, copying risk, data leakage. Do: a suite producing base / prompt-only / LoRA / LoRA+retrieval outputs. See validation_report.md (evaluation-and-safety appendix). Concepts: ml-evaluation

Self-test — the exit exam

The roadmap works if you can answer these from first principles. Each links to where the answer lives; if a link doesn’t make it click, that doc is the gap to fix.

1. Why does attention need the max-subtraction in softmax for numerical stability? (and why is online softmax a structural, not numerical, win?) → online_softmax_in_attention.md
2. How does the backward pass for attention work? (recompute P from the saved L = m + log l) → fa2_backward_notes.md, study_guide.md §7
3. Why does register blocking improve matmul until it hurts occupancy? (4×4 vs 8×8 register spill) → study_guide.md §8
4. What caused the Memory64 ABI issue? (memory.grow detaches SAB views mid-kernel; host plumbing differs) → study_guide.md §4
5. Why does parity testing matter more than standalone kernel benchmarks? → study_guide.md §9
6. What does 2.5% loss drift actually mean? → learn/essential-vs-optimization.md (loss-drift section)
7. Which parts are mathematically essential vs engineering optimization? → learn/essential-vs-optimization.md
8. Could you reimplement the core transformer training loop without AI? → python_ref/model.py + Phases 1–4 / sessions 1–8
9. Could you debug a wrong-gradient bug from first principles? → study_guide.md §9 (worked example), §2
10. Could you explain the WebGPU execution model clearly to another engineer? → learn/webgpu-execution-model.md

12-week schedule

Traps to avoid

1. Starting with the browser. Right order: PyTorch reference → WASM reference → WebGPU acceleration.
2. Too large a model. First targets: 0.8M from scratch; 5M–15M frozen base for LoRA. Not 100M+.
3. Confusing style with intelligence. LoRA learns tone, format, phrasing, argument rhythm, vocabulary — not truth, judgment, or reliable reasoning.
4. Not testing memorization. Always test: training prefix → generated continuation. If it copies exact passages, reduce training or improve data.
5. Skipping baselines. Always compare base / few-shot / LoRA / LoRA+retrieval. If LoRA does not beat prompting, the adapter wasn’t worth it.

Final build order

1.  Byte-level 0.8M TinyGPT in Python
2.  Train from scratch on tiny text
3.  Save / load checkpoints
4.  Pretrain a 5M–15M base model outside the browser
5.  Add LoRA on q_proj / v_proj
6.  Fine-tune on clean blog-style examples
7.  Evaluate base vs prompt vs LoRA vs retrieval
8.  Port inference/training loop to a browser Worker
9.  Add the WASM backend
10. Add WebGPU matmul — only after correctness

Project deliverables

1. Python TinyGPT reference
2. Python LoRA fine-tuning reference
3. Clean blog/style dataset builder
4. Browser TinyGPT demo
5. WASM backend
6. Adapter checkpoint format
7. Evaluation harness
8. WebGPU matmul prototype
9. Learning notes explaining every component