TinyGPT — study queue

Short stubs for every genuinely-novel topic in the codebase. Fill in Why here: yourself after internalising each topic.

Backpropagation & gradient descent

• What: compute the loss gradient w.r.t. every parameter via the chain rule (reverse-mode autodiff), then step each parameter against its gradient.
• Why here: TBD
• Source: https://www.youtube.com/watch?v=VMj-3S1tku0 (Karpathy — spelled-out intro, builds a tiny autograd by hand)

Next-token language modeling & sampling

• What: a language model predicts a probability distribution over the next token; generation repeatedly samples from it, shaped by temperature and top-k truncation.
• Why here: TBD
• Source: https://github.com/karpathy/makemore

Memorization testing (style-adaptation eval)

• What: feed a training-set prefix and check whether the model copies the exact continuation; distinguishes learned style from memorised text, alongside base/prompt-only/LoRA baselines.
• Why here: TBD
• Source: https://arxiv.org/abs/2012.07805 (Carlini et al. — extracting training data from LMs)

Transformer / scaled-dot-product attention

• What: sequence model where each token attends to all previous tokens via learned Q/K/V projections; output is a weighted sum of values.
• Why here: TBD
• Gotcha (from code): per-thread private array<f32, 1024> for the score row at ctx=1024 is 4 KB — Apple WebGPU spills it to global memory.
• Source: https://nlp.seas.harvard.edu/annotated-transformer/

Flash Attention 2 (FA2)

• What: tiled attention algorithm that walks K/V in blocks and accumulates softmax online, never materialising the full T×T score matrix.
• Why here: TBD
• Gotcha (from code): tile sizes Br = Bc = 16 are hard-coded in webgpu/attention_fa2.wgsl; going wider hits register pressure on Apple GPU.
• Source: https://arxiv.org/abs/2307.08691

Online softmax

• What: single-pass numerically-stable softmax that carries a running max and denominator, updating them as each new element arrives.
• Why here: TBD
• Source: https://arxiv.org/abs/1805.02867

Byte-Pair Encoding (BPE) tokenization

• What: greedy compression algorithm that iteratively merges the most-frequent byte pair until a target vocab size is reached; tokens are subword units.
• Why here: TBD
• Source: https://huggingface.co/learn/nlp-course/chapter6/5

WebAssembly (WASM) + Emscripten

• What: portable binary instruction format; Emscripten compiles C++ to .wasm + a JS glue file that wires browser APIs (memory, threads) to the C runtime.
• Why here: TBD
• Gotcha (from code): WASM Memory64 (tinygpt64.wasm) hits a pthread × WebAssembly.Memory.grow race in the browser — the fix is over-allocating INITIAL_MEMORY=256 MB so growth never fires during training.
• Source: https://emscripten.org/docs/porting/pthreads.html

WASM SIMD + multi-threading

• What: WASM SIMD extends the ISA with 128-bit vector ops; multi-threading uses SharedArrayBuffer + Atomics to run pthreads across Web Workers.
• Why here: TBD
• Source: https://github.com/WebAssembly/simd/blob/master/proposals/simd/SIMD.md

WebGPU + WGSL

• What: next-gen browser GPU API (successor to WebGL) with an explicit pipeline model; WGSL is its shader language.
• Why here: TBD
• Gotcha (from code): var<storage, read> vs buffer: { type: "storage" } mismatch — Apple WebGPU silently returns wrong data instead of a validation error; caused vec4 matmul to diverge loss to 88.67 in end-to-end training.
• Source: https://gpuweb.github.io/gpuweb/

Tiled / register-blocked matrix multiplication (GPU)

• What: partition operand matrices into shared-memory tiles to reduce global memory traffic; register blocking accumulates partial sums in registers to further reduce shared-memory reads.
• Why here: TBD
• Gotcha (from code): 8×8 register blocking beat 4×4 in theory but lost at every measured size — exceeded Apple’s per-thread register budget and forced spill.
• Source: https://siboehm.com/articles/22/CUDA-MMM

fp16 / mixed-precision

• What: 16-bit floating point; mixed-precision trains in fp16 for speed while keeping a fp32 master copy of weights to avoid underflow/overflow.
• Why here: TBD
• Gotcha (from code): f16-packed weights were 1.7× faster standalone but slower when stacked on tiled matmul — tiling had already amortised the global-read savings.
• Source: https://arxiv.org/abs/1710.03740

Sparse Autoencoder (SAE) / mechanistic interpretability

• What: a learned over-complete dictionary trained to reconstruct a model’s hidden states with a sparsity penalty; used to decompose polysemantic neurons into monosemantic features.
• Why here: TBD
• Source: https://transformer-circuits.pub/2023/monosemantic-features/index.html

Logit lens

• What: interpretability probe that projects each intermediate residual-stream state through the final unembedding to see what token the model “would predict” at that depth.
• Why here: TBD
• Source: https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

Gradient checkpointing (activation recomputation)

• What: trade compute for memory by discarding intermediate activations during the forward pass and recomputing them during the backward pass.
• Why here: TBD
• Source: https://arxiv.org/abs/1604.06174

AdamW optimizer

• What: Adam with decoupled weight decay (decay applied to weights directly, not through the gradient); standard default for transformer training.
• Why here: TBD
• Gotcha (from code): browser default LR was 3e-3 (10× too hot vs Python reference 3e-4); training plateaued at loss ~2.45 and looked like a modelling ceiling.
• Source: https://arxiv.org/abs/1711.05101

LoRA (Low-Rank Adaptation)

• What: fine-tuning method that freezes pre-trained weights and injects trainable low-rank matrices (A·B) into each projection; only A and B are updated.
• Why here: TBD
• Source: https://arxiv.org/abs/2106.09685

Speculative decoding (draft + verify)

• What: a small fast draft model proposes K tokens; the big target verifies them in ONE batched forward and accepts the longest matching prefix — lossless vs the target’s own sampling, just faster wall-clock.
• Why here: TBD
• Gotcha (from code): on Metal, greedy spec-decode is NOT byte-identical to single-token decode — batched-verify vs single-token attention aren’t bit-reproducible across batch shapes, so a near-tie argmax occasionally flips (still a valid greedy decode; verified against the uncached reference). serve speedup is content-dependent (acceptance 2.3 dense → 3.3 structured) and masking-bound under --grammar. “Seed-carry” (feed the last burst token as the next verify’s first) avoids a wasted second target forward.
• Source: https://arxiv.org/abs/2211.17192 (Leviathan et al., 2023)

Continual learning / online adapter refresh

• What: instead of freezing a model at ship time, capture real-usage signal (corrections, edits, retries) and periodically retrain a LoRA on it so the model improves from production; the on-device version keeps all data local.
• Why here: TBD
• Source: https://docs.trajectory.ai/introduction · see docs/prds/continual-learning-loop.md