← TinyGPT · docs · devlog · roadmap · speedup
source: docs/learn.md · view on GitHub ↗

Learn TinyGPT — a guided path for software engineers new to AI

Legacy guided code walkthrough for the original Python/WASM/WebGPU learning arc. The active learning index is docs/learn/README.md; active project work starts from PROJECT_STATUS.md, docs/NEXT.md, and docs/factory/.

You can write code. You have never built or trained a neural network. This guide takes you, in order, through everything in this repository until none of it is a black box.

It does not re-teach what others teach better. For each concept it gives you:

Learn it — a link to the best free explainer (watch / read it). In the repo — the exact file and function where TinyGPT implements it. See it work — the command that runs that code and proves it.

The repo is a real, working GPT — trained from scratch, fine-tuned with LoRA, ported to hand-written WebAssembly, accelerated with WebGPU, and wrapped in a browser app. By the end you will understand all of it, both the AI and the systems engineering.

Time: ~12–20 hours if you watch every linked video. You do not have to do it in one sitting — the parts are self-contained.

Contents

0. Orientation — the repo in 10 minutes

What TinyGPT is. A GPT-2-shaped transformer the user can size from ~360k parameters (Small) up to ~470M (Behemoth, Memory64). The same architecture as ChatGPT, ten-to-five-hundred-thousand× smaller depending on preset. It is intentionally readable — every part has a test. It will never say anything clever at the smallest sizes; the point is that you can understand a complete one. The bundled demo (Medium, ~840k params) produces character-aware pseudo-Shakespeare after about four minutes of in-browser training.

The guiding rule of the codebase: never trust a component until a test pins it down. Every layer has a check that fails loudly if it is wrong — which is also why this guide can always end a section with “see it work”.

The repo map:

DirectoryWhat’s in it
configs/The exact model / training / LoRA settings, as JSON
python_ref/The reference implementation in PyTorch — read this first
wasm/The same maths re-written in C++, compiled to WebAssembly
webgpu/A matrix-multiply kernel for the GPU
browser/The web app that trains a model in your browser
tests/The correctness tests for every layer above
docs/This guide, plus a deeper spec for each phase

Read order for the code: python_ref/ first (it is the clearest), then wasm/, then webgpu/ and browser/. The whole project was built in that order on purpose — a correct reference first, then ports of it.

Set up your environment (you will need these as you go):

# Python reference — needs Python 3.10+
python -m venv python_ref/.venv && source python_ref/.venv/bin/activate
pip install -r python_ref/requirements.txt

# C++ / WASM parts — needs a C/C++ compiler (clang or gcc); Emscripten later
# Browser app — needs Node.js

The companion doc notes.md is the retrospective — what was built and what each experiment showed. Skim it now for the shape of the project; come back to it at the end when it will all make sense.

1. The math you actually need

You do not need a maths degree. You need to be comfortable with five ideas. Treat this as vocabulary, not a course.

1.1 Vectors, matrices, and matrix multiplication

A neural network is mostly arrays of numbers multiplied together. A “vector” is a 1-D array; a “matrix” is a 2-D array; “matmul” is the one operation that combines them.

1.2 The dot product

Multiply two vectors element-wise, sum the result — one number that measures how aligned two vectors are. Attention (Part 3) is built entirely on this.

1.3 softmax

Turns a list of arbitrary numbers (“scores”) into a list of probabilities that sum to 1. The model produces scores; softmax makes them a probability distribution over the 256 possible next bytes.

1.4 Loss, and cross-entropy

“Loss” is a single number measuring how wrong the model is — training is just making it smaller. For predicting one of 256 bytes, the loss is cross-entropy: it is low when the model put high probability on the correct byte.

1.5 Gradients, backpropagation, gradient descent

This is the heart of learning. A gradient tells you which direction to nudge each number to make the loss smaller. Backpropagation is the algorithm that computes every gradient efficiently. Gradient descent is the loop: nudge, re-measure, repeat. AdamW is the specific, well-tuned nudging rule used here.

✓ You’ve got this when you can say, in your own words: what a loss is, what a gradient is, and what one training step does (measure loss → compute gradients → nudge every weight a little).

2. Language modeling — what the model is doing

2.1 The one job: predict the next token

A GPT is not answering questions or reasoning. It does exactly one thing: given some text, predict the next token. Everything else (chat, code) is that one trick, repeated.

2.2 Tokenization — text becomes numbers

A model only sees numbers. A tokenizer maps text to integers. Real GPTs use “BPE”; TinyGPT uses the simplest possible scheme — one byte = one token, so the vocabulary is exactly 256. No tokenizer to train, no edge cases.

2.3 Context window and the causal rule

The model sees a fixed window of recent tokens (here, 128). When predicting position t, it may look only at positions ≤ t — never the future. That “causal” rule is what makes left-to-right generation possible.

2.4 Sampling — turning predictions into text

The model outputs a probability for every possible next byte. To generate, you sample one, append it, and repeat. temperature controls boldness (low = safe and repetitive, high = wild); top_k restricts the choice to the k most likely bytes.

✓ You’ve got this when you can explain why “a language model predicts the next token” and “a language model reasons” are very different claims.

3. The transformer

This is the architecture itself. Do this part with two tabs open: the reading below, and python_ref/model.py — it is ~230 lines and you should read all of it here.

First, the big picture. Read Jay Alammar — The Illustrated Transformer, and keep Transformer Explainer open in a tab — it is a live GPT-2 you can poke, and it makes the next four ideas concrete.

3.1 Embeddings — a number becomes a vector

A token id (0–255) indexes into a learned table, producing a d_model-length vector. A second table does the same for the token’s position. The two are added: now the model knows what the token is and where it sits.

3.2 Self-attention — tokens look at each other

The one place tokens exchange information. Each token asks a question (a “query” vector), every token offers a key; the dot product of query·key scores how relevant each other token is; softmax turns the scores into weights; the token pulls in a weighted blend of every token’s “value” vector.

3.3 The supporting cast

Read these straight from python_ref/model.py — each is a few lines:

3.4 The whole model

A TransformerBlock is attention + MLP, each wrapped in LayerNorm and a residual. Stack four of them, add a final LayerNorm, and project back to 256 scores with the tied output head (it reuses the embedding table). That is TinyGPT.

3.5 Training it

Training is the gradient-descent loop from Part 1, wrapped around this model: get a batch of text, predict next tokens, measure cross-entropy loss, backpropagate, AdamW step, repeat.

✓ You’ve got this when you can trace one token’s vector through python_ref/model.py: embedding → 4 blocks → final norm → 256 scores.

4. LoRA — cheap fine-tuning

Once a model is trained, LoRA adapts it to a new style without retraining the whole thing. It freezes the original weights and trains two tiny matrices alongside each chosen layer. Here the adapter is 0.96% of the model’s size.

✓ You’ve got this when you can explain why LoRA trains <1% of the parameters and still changes the model’s behaviour.

5. Running it in the browser — the systems half

This is the half almost no tutorial covers, and it is what makes this repo unusual. A browser cannot run PyTorch. To train a model in a browser tab you have to rebuild the compute yourself and respect the browser’s constraints.

5.1 WebAssembly — running C++ in the browser

WebAssembly (WASM) is a fast, low-level bytecode browsers can run. You write C/C++ and compile it to WASM with Emscripten. TinyGPT’s kernels are written in plain C++ for exactly this.

5.2 Verifying a backward pass you wrote yourself

If you hand-write a gradient, how do you know it is right? You nudge each input by a tiny amount and measure the slope of the loss — the “finite-difference” gradient check. If your formula and the measured slope disagree, your formula is wrong.

5.3 Web Workers — not freezing the page

JavaScript runs on one thread. If you train a model on it, the whole tab freezes. A Web Worker is a separate thread; TinyGPT runs all training there and the UI thread only draws.

5.4 Sharing memory across the JS ↔ WASM boundary

WASM has its own block of memory. JavaScript reaches into it through typed-array “views” and raw pointers. This is the seam where browser ML actually lives.

5.5 WebGPU — using the GPU

WebGPU is the browser API for running compute on the GPU. TinyGPT uses it for the heaviest operation, matrix multiply, written as a compute shader in WGSL.

✓ You’ve got this when you can explain why training must run in a Worker, and what a “finite-difference gradient check” proves.

6. See the whole thing run

You have read it. Now watch every layer execute. If all of this passes, you have understood a complete, working version of the modern LLM stack.

# --- the AI half (Parts 1-4) -----------------------------------------
source python_ref/.venv/bin/activate
python tests/test_phase1.py          # model, training, sampling — 8 tests
python tests/test_lora.py            # LoRA fine-tuning — 6 tests
python python_ref/train.py --overfit # watch a real loss curve fall

# --- the systems half (Part 5) ---------------------------------------
bash wasm/build_native.sh            # the C++ kernels + model, all verified

# --- the browser app -------------------------------------------------
bash wasm/build_wasm.sh              # compile C++ -> WebAssembly (needs Emscripten)
cd browser && npm install && npm run dev
# open the printed URL, click "Start training" — a GPT trains in your tab

For the full story of what each test proved — with the numbers — read notes.md. For a per-phase deep spec, see the other files in docs/. The phase/week breakdown is in archive/learning_roadmap.md.

Going deeper

When you want to go beyond TinyGPT — to models that actually produce good text:

You started as a software engineer who had never built a neural network. If you have followed this path, you can now read — and have tested — every line of a real GPT, and the browser engineering that runs it. That was the whole goal.