TinyGPT learning corpus

A reading map for the docs/learn/ directory. Three reading paths depending on what you want.

Start here for ground-up learning: Curriculum overview. It is the 10-module path from functions and loss to transformers, post-training, evals, rewards, and the self-improving factory.

Start here for Mac-local scope: Mac-local AI mastery map — the living agenda: everything buildable on a Mac, what’s already covered, and the single-machine ↔ distributed boundary.

For the active owner learning sequence tied to current factory work, use ../learning-pipeline.md. It orders eval design, post-training data, SFT/LoRA, preference tuning, verifiable rewards, RLVR/OAPL, failure analysis, and public reporting around the SQL/factory loop.

I want to learn ML from scratch

Read these in order — the curriculum is designed as a single arc from basic math to modern transformer training. Each session is self-contained but builds on the last.

• Curriculum overview — the 10-module ground-up roadmap, mastery gates, exercises, and project anchors.
• Session 1 — From a line to a learned line
• Session 2 — Gradient descent
• Session 3 — Non-linearities
• Session 4 — ML paradigms
• Session 5 — Scaling
• Session 6 — Tokenization + embeddings
• Session 7 — Behavior learning
• Session 8 — Training mechanics

I want the modern-LLM mechanics reference

These document the architectural + algorithmic choices that show up in current LLMs. They’re for someone who knows the basics and wants the “why” behind specific designs (RoPE, GQA, MoE, etc.).

• LLM mechanics fundamentals — RoPE, GQA, attention variants, MoE, expert routing
• Mathematically essential vs engineering optimization — the project split: the math that defines the model’s function (oracle = python_ref/model.py) vs the optimization layer that only makes it faster/smaller; and loss drift, the number that polices the boundary
• The WebGPU execution model — device/queue, pipeline, dispatch, workgroups, invocations, bind groups, the memory hierarchy; how our matmul + attention shaders map onto it (read before the .wgsl files)

Interview-grade topic maps (what / why-it-matters-here / external source / repo anchor — for senior/staff prep):

• Speech & systems topics — voice-pipeline latency, WER, speech-to-speech, fine-tune debugging, feature selection, queues vs websockets, FSDP2
• Advanced LLM training & post-training — ZeRO/FSDP2 depth, precision (bf16/fp8), gradient checkpointing, MoE training, data curation, RLHF/DPO/GRPO/reward-modeling/distillation
• Advanced LLM inference & serving — roofline, KV cache + paging, batching, speculative decoding, quantization, FlashAttention, attention variants, long context, serving architecture
• Advanced architecture, RAG/agents, eval & system design — modern decoder block, attention-as-matmuls whiteboard, RAG, agents, LLM-as-judge, perplexity, contamination, ML system-design rounds, classic-ML depth
• Qwen3-VL mRoPE + DeepStack — vision-language attention specifics; relevant to Pace’s VLM pillar
• App Intents comparison — how Pace’s action surface relates to macOS App Intents
• Apple on-device Foundation Models — where they fit (and don’t) — the bridge we built, the measured verdict (can’t ground actions; 4096-ctx can’t hold a tool catalog; not faster), and the decision to use it as a free routing floor, never a dependency
• Agent context as a memory hierarchy — the L1/L2/L3 framing for agent context engineering, and the steals it produced (eval failure triage, E9 prompt-tiering A/B, B26 deferred tools)
• Castform’s RL fine-tune platform — what we stole — composite reward functions (B28), trace-driven data synthesis (B29), reasoning-depth classification (B30)
• Tool-calling: how close can a Mac-local small model get to frontier? — broken-eval → frontier-validated BFCL gate, the honest size curve, the distillation result (1.7B avg 56→76), and where RL takes over
• Small-model tool-calling: the SOTA playbook (what others do) — survey of data synthesis, SFT tricks (function masking), RL (ToolRL graded reward, DAPO), eval traps, and on-device serving, with a prioritized steal list
• Diversity-driven small-model reasoning (Spectrum-to-Signal) — VibeThinker’s three transferable methods (Diversity-Exploring Distillation, MGPO, specialist weight-merging) mapped to our loop; specialist-merging as the negative-transfer antidote
• Model vs agent — what’s actually different — the architectural distinction; “what makes a model into an agent” beyond tools + loop; mapped onto this repo’s layers
• Competitive landscape (2026) — map of fine-tune + eval + interp players, the Mac-first whitespace, and the consolidation signal
• External references — papers, blog posts, code-base reading list

I want session-specific decisions + project state

Captured-in-the-moment notes from real training sessions and decision points.

• Eval matrix (2026-06-08) — every Pace LoRA’s score against fm-fixtures-v2 (annotated 2026-06-09: v8 baseline non-reproducible mystery)
• Eval methodology (2026-06-08) — the gate finding that the v1 fixture set was broken; how v2 fixes it
• ANE research notes — M5/M6/M7/M8 bisects, ANE precision drift, chunked inference findings
• Learning journal — running log of questions, hunches, tangents while doing the work

Conventions

• Each session has a one-sentence “where we’re starting” up top — read that first.
• Annotated diagrams (in text) prefer ASCII over images so they’re readable in git diffs and on every renderer.
• “Why this matters to THIS project” is called out where applicable — the curriculum isn’t generic, it’s anchored to tinygpt’s choices.
• Annotated retroactively when an earlier claim turns out wrong (see eval-matrix’s 2026-06-09 addendum for an example).

Related

• Top-level docs/PLAN.md — long-term project roadmap (not learning material)
• Top-level docs/prds/ — per-feature PRDs (not learning material)
• Memory entries (private to my agent context) capture decisions + doctrine separately