Strategy session — what TinyGPT actually is, what’s possible on a Mac

Date: 2026-06-06 Premise: TinyGPT shouldn’t be framed as “small models beat large ones on benchmarks.” The real thesis is: a Mac app where individuals build a specialist for their specific task — bring data, pick a teacher, ship a fast/cheap/comparable-quality model. Distillation, fine-tuning, and quantization are the toolkit that makes this actually work.

This doc captures the strategic frame, the memory math, and the product roadmap implications.

What TinyGPT is, restated

“A Mac app where you bring data, pick a teacher, and ship a specialist that runs 15-100× faster than the teacher with comparable quality on your specific task.”

Not framework. Not benchmark contender. Not research project. A platform for individual task-specialization on consumer hardware.

The benchmark-beating angle is a demo of what the platform can do — not the product itself.

Success criterion (refined)

The user explicitly clarified the bar: **comparable quality + 20-150× speed

• 150× less memory** is a complete win. Then optimize performance further from there.

That maps to a 3-axis Pareto win:

If a specialist hits this triangle, ship it. Headline-grade “beats teacher on absolute quality” is a bonus, not a requirement.

Mac memory math — what’s possible at each scale

The 3B “limit” everyone repeats applies only to pretraining from scratch. Different operations have radically different memory footprints, and the practical ceiling jumps significantly as you go down this table:

Implications for what the platform should offer:

• “Train from scratch on your Mac” — only works for ≤3B. Limited niche.
• “Fine-tune a pretrained model on your Mac” — works up to ~13B via LoRA (free, fast, no quality loss), up to ~30B via QLoRA (small quality trade for memory).
• “Distill from a larger teacher” — teacher can be a downloaded ≤30B model OR an API call (Claude, GPT-4). Student is whatever the user wants.
• “Quantize and ship” — final output can target up to 70B at Q4 for consumer inference, but most specialists will be ≤7B for real speed.

The real product story is not “train tiny models from scratch.” It’s “fine-tune or distill any model up to ~30B on your Mac for your specific task.”

Why 4B / 5B / 6B models barely exist (the gap)

Open-weights ecosystem clusters at 1B / 3B / 7B / 13B / 30B / 70B. The 4-6B niche is empty. Reasons:

1. Consumer GPU memory tier alignment. 4 GB → 1B; 8 GB → 3B or 7B-Q8; 16 GB → 13B-Q8. A 5B model awkward — wastes 8 GB headroom, doesn’t reach 13B-tier quality. Labs pick sizes that match buyers’ actual hardware.
2. Training-budget round numbers. Cluster commit goes 3B → 7B (2× compute); 5B is “would rather just do 7B.” Economics, not physics.
3. Mistral 7B anchored the “quality sweet spot.” Every lab now benchmarks against 7B to be in the conversation.
4. Differentiation thin. 5B isn’t meaningfully better than 3B for most tasks, worse than 7B. No clean marketing story.
5. Recipe inheritance. Most open labs use a Llama-style scaling recipe that only varies depth/width to hit Llama-class sizes.

This is an opportunity, not a constraint. Distillation lets you target any size including 4-6B. A TinyGPT-Mega-tuned variant at 5-6B would slot into “better than 3B, faster than 7B, fits 8 GB GPUs at Q8.” Niche worth investigating once the base recipe works.

The realistic Mac-specialist sizes

The sweet spot for “user’s specialized task on their Mac” depends entirely on the task. Routing/intent → 22M is plenty. Function calling → 76M-1B is the band. Code completion → 3-7B. The platform’s job is to pick the right size automatically based on user data + task type.

What’s already built (status as of 2026-06-06)

Working principles (clarified 2026-06-06)

OSS audit — what to adopt vs build

Honest read: ~70% of the Tier 1 backlog has strong existing OSS to adopt. Work shrinks from “implement from scratch” to “adapt + integrate.” Updated effort estimates assume adoption.

Tier 1 gaps × OSS sources

Tier 2 modality OSS sources

Multi-model architectures OSS

What’s actually ours to write from scratch

The integration glue and product UX:

• Local-teacher labeling pipeline (no one has this at this scope)
• Unified tinygpt prep CLI wrapper
• “Create Specialist” app wizard
• Templates for common patterns (router, classifier, JSON-extractor, etc.)
• Quality predictability heuristics
• Mac-specific Apple Intelligence / Shortcuts integration

The rest is adapt and integrate, not invent. With this discipline, the Tier 1 estimate revises from 4-6 weeks → ~3 weeks of focused work.

Audit as a recurring practice

For each new gap that surfaces in future sessions:

1. First action: grep GitHub / awesome-ml-X lists for prior art
2. Second action: read the closest 2-3 OSS implementations
3. Third action: decide adopt / fork / wrap / build, in that order of preference
4. Only build from scratch if nothing usable exists OR the license precludes adoption

The session-doc gap-table format above should be filled in for every new capability proposal, not skipped.

What’s missing — the disciplined backlog

Tier 1 — text-only platform completion (CORRECTED 2026-06-06 after dispatch-table audit)

Earlier draft of this list claimed ~7 gaps that turned out to be already shipped. Corrected list below; verify any claimed gap by grep 'case "X"' native-mac/Sources/TinyGPT/TinyGPT.swift before adding.

Already shipped (DO NOT re-plan): distill, dpo, gptq, hqq, prune-unstructured, prune-structured, bon, agent, cloud, escalate (phone-a-friend), dedupe, rome, memit, patch, causal-trace, plus ConstrainedGen.swift (804 lines for constrained decoding) and KVCache.swift (885 lines).

Actual remaining gaps (verified by absence in dispatch / source tree):

Revised Tier 1 effort: ~2 weeks of focused work. Much closer to done than original 4-6 week estimate suggested.

Reinvention cleanup candidates (audit before removing)

The flip side of “adopt OSS only when best”: audit existing code for places we reinvented without payoff. Candidates (need verification before action):

Highest priority cleanups to investigate this week: Score.swift consolidation into the newer eval stack; ExtractorData.swift vs datatrove migration analysis.

Tier 2 — multimodal (user-confirmed priorities: vision + voice; image gen; video defer)

Each modality is real engineering — substantial scope, not a knob. Ordered by ROI.

Tier 3 — advanced (defer until Tier 1+2 land)

Tier 4 — explicitly out of scope (or skip / research-only)

Judgment-family models (related but distinct from rerankers)

Beyond generating text, models can evaluate text. This unlocks self-refinement, RL data generation, automated eval, and quality gating. Should be a first-class capability slot:

Status for our platform: judge shipped (E7); the others are gaps. The one-canonical-best per slot:

These compose with the multi-model architectures below — e.g., generator → reranker → verifier → output is a high-quality production pattern.

Multi-model architectures (how multiple models compose at inference)

A specialist doesn’t have to be a single model. Several patterns compose multiple models for better quality, lower cost, or new capabilities:

Most valuable for our platform to add: phone-a-friend / cascade (serve-level), LoRA hot-swap (one base, many specialists), reranking (small encoder for best-of-N selection). All ~3-5 days each.

Structured output — beyond JSON

JSON is solved because its grammar is well-defined. The same mechanism (grammar-constrained decoding) handles any format with a definable grammar. Our Tier-1 “constrained decoding” slot should be grammar-engine general, not JSON-specific:

Engine choice (one canonical): GBNF (llama.cpp grammar format). Reasons: well-documented, expressible enough for any format above, ecosystem familiarity (llama.cpp users know it), text-based grammar files easy to ship. Alternative: xgrammar (NVIDIA, faster) or Outlines (Python, most flexible). Pick GBNF for ecosystem fit.

For protobuf specifically: the .proto file IS the grammar. Generator translates .proto → GBNF → constrained decoder that produces byte-perfect serializable protobuf. Feature, not research.

Diffusion models — speed reality check

For our image-generation slot (Tier 2): SDXL Turbo or Flux Schnell + LoRA fine-tuning via Apple’s ml-stable-diffusion (CoreML). Runs on Neural Engine. That’s the “fast diffusion on Mac” pitch — real and shippable.

Inference performance — what’s possible on Mac

For context on the “fast and cheap” half of the pitch:

What Groq / Cerebras / SambaNova do:

The win is ~80% hardware (SRAM-only memory hierarchy bypasses GPU’s HBM bottleneck), ~20% software co-design.

What we can stack on Mac:

Not Groq-fast, but “feels real-time” is achievable:

The memory-bandwidth ceiling (hard physics):

Autoregressive inference reads the entire model per token. So:

max_tokens_per_second = memory_bandwidth / model_size_in_bytes

The Groq/Cerebras advantage is ~130× memory bandwidth (on-chip SRAM vs DRAM). That’s where their tok/s wins come from — not magic, not algorithmic.

Realistic stacked ceiling on M5 Pro (48 GB):

What can’t be 100×-ed on Mac: the bandwidth ceiling. You’d need 100× more SRAM bandwidth, which means Groq/Cerebras-class hardware ($10K-100K). Not happening on a laptop.

What can: the size of the specialist itself. 22M vs 3.8B is already ~150× the param count. Combined with the inference speedups, the “20-150× faster than teacher” claim in the success criterion is achievable.

The platform pitch isn’t “as fast as Groq” — it’s “fast enough for your specific task, on your own machine, at zero cost.” A 300 tok/s specialist locally beats a 1000 tok/s cloud model when you factor in 50-200ms network round-trip + cost + privacy.

Market landscape (what’s out there as of mid-2026)

For situating where TinyGPT fits in the broader inference / specialization market:

TinyGPT’s positioning: the only player in “local-Mac inference” that ALSO covers the full build/upgrade lifecycle (data → train → eval → deploy). Ollama / LM Studio run pretrained models; TinyGPT builds and ships specialists end-to-end on the same machine.

Competitive landscape — deeper research (added 2026-06-06)

Direct Mac/local-inference competitors

Cloud fine-tuning (we don’t compete)

Eval platforms

Synthetic data / distillation

Apple-specific frontier

Top 5 positioning statements

1. “Run-only tools chat with models. TinyGPT builds them.” — vs LM Studio / Ollama / Jan / GPT4All.
2. “Fine-tune on the Mac you already own — no 4090, no cloud bill, no data leaving the device.” — vs Together / Fireworks / Unsloth.
3. “OpenAI killed their fine-tune API in May 2026. Specialist models shouldn’t be a hosted API in the first place.” — market-signal positioning.
4. “One opinionated path — pick a teacher, bring data, get a fast specialist — wrapped around the best OSS pieces (MLX-LM, lm-eval-harness, Distilabel).” — honest about adoption; integrated workflow is the differentiator.
5. “Apple’s Foundation Models give you one 3B model locked to iOS. TinyGPT gives you any open base from 22M to 13B, deployable anywhere.” — vs Apple’s on-device push; no ecosystem lock-in.

Where TinyGPT honestly does NOT compete

• 70B+ production fine-tuning — Together/Fireworks win; concede
• LangChain-tied production observability — LangSmith owns it
• RAG-over-my-files chat — GPT4All’s LocalDocs is mature
• Tabular synthetic data — Gretel/Mostly own enterprise tabular
• iOS App Store distribution for AI — Apple’s Foundation Models wins; export to it

Wave 2 landscape — AI coding / agents / inference / reasoning / RAG (researched 2026-06-06)

AI coding tools

Agent frameworks

Sharp insight: None of these produce specialists. They orchestrate. TinyGPT is the missing model-producer layer underneath. Cookbook pages for smolagents + Pydantic AI reach high-intent audiences cheaply.

Inference framework wars

Reasoning models

Sharp insight: Half of HF reasoning model cards mention GRPO; it’s table stakes. If we ship tinygpt grpo, DeepSeek-R1-Distill-Qwen-1.5B + GSM8K reward is the recipe.

RAG infrastructure

Sharp insight: Every RAG stack assumes OpenAI embeddings. None treat “I trained my own 22M embedding specialist” as a first-class workflow. That’s a wedge.

Three strategic decisions surfaced by wave 2

These warrant explicit decisions, not deferral:

Decision 1: How to handle vllm-mlx serve threat

vllm-mlx is claiming 4.3× throughput at 16 concurrent requests on Apple Silicon. Our hand-rolled serve has no continuous batching. Three options:

Recommended: (b) — wrap. Aligns with the results-first principle. Our value is train+distill+eval+ship loop, not serve-throughput. Worth a dedicated PRD to evaluate vllm-mlx maturity + plan the integration.

Decision 2: Add “specialist embedder” as Tier 2 modality

New opportunity surfaced: every RAG stack assumes OpenAI embeddings. A tinygpt embed-train recipe + LanceDB integration is weekend-buildable, reuses our existing specialist-training stack, opens the RAG category without us shipping a vector DB.

Recommended: yes, add to Tier 2. Slot between “Speech-in (Whisper)” and “Image gen” by priority. PRD-sized.

Decision 3: Distribution strategy — agent frameworks > editor integrations

Cookbook pages for smolagents and Pydantic AI reach high-intent audiences (people already running local models, already caring about tool-call + structured-output reliability) more cheaply than chasing the Continue.dev / Aider crowd (who already have working models).

Recommended: write 2 cookbook pages first, before any new editor-integration work. Each ~1 day. Use the same recipe as the “Tabnine alternative” positioning.

New positioning statements (from wave 2)

Add to the top 5:

• “Tabnine’s per-repo fine-tune, but on your Mac, on open weights, for free.” (cuts cleanly against the only incumbent with “specialist code model” as a productized story)
• “Every RAG stack assumes OpenAI embeddings. Train your own.” (wedge into RAG without building a vector DB)

Flagship example applications

These are apps built on top of the platform — not new modalities. They demonstrate what the platform can do:

The platform’s value is in being the substrate these are built on. Once Tier 1 + 2 land, each of these is ~1-2 weeks of integration work, not a research project.

Language-specific code distillation — the killer recipe

Concrete recipe (e.g., for a Rust specialist):

Why this works particularly well:

• Code has hard correctness signal (compiles or doesn’t)
• Long-tail language-specific patterns (Rust borrow checker, Swift @MainActor, Python f-strings) are exactly where small specialists shine
• General LLMs (GPT-4, Claude) trained on all languages → mediocre at each specific one
• Per-language specialists are a clean differentiator vs general assistants

Tokenization — why tokens (not characters / bytes)

Tokens are just contiguous character groups the tokenizer decided to chunk together. Why we use them instead of raw bytes/characters comes down to one number: attention cost grows as O(N²) in sequence length. Shorter sequences = exponentially less compute.

Subword BPE produces ~20× fewer tokens than byte-level for English. That translates to:

• ~400× less attention compute (O(N²))
• 20× shorter KV cache
• 20× more text fits in the same context window

That’s why every production LLM uses subword tokenization. The cost is a larger embedding matrix (49K × d_model extra params) — amortized to nothing per token.

Tradeoffs

Platform recommendation — when to use which

Frontier ideas worth tracking

• Megabyte (Meta 2023) — patch transformer over bytes
• BLT (Byte Latent Transformer) (Meta 2024) — dynamic byte grouping based on entropy; high-entropy spans get more compute
• CANINE / ByT5 — pure character/byte transformers, accept the cost
• Pixel-based — text-as-image + VLM

The disciplined call: keep BPE as default. Watch BLT — if it becomes mature open weights, byte-with-patches becomes the right answer for multilingual + code + typo-robust specialists.

Why this is a research frontier (and why TinyGPT is positioned for it)

Tokenization is one of the few hardcoded preprocessing steps left in modern LLMs — everything else (embeddings, attention patterns, position encoding, expert routing) has been “learned” over the past decade. Letting the model learn its own input units is active 2024-2025 research.

Known problems with fixed BPE tokenization that motivate this:

• Multilingual fragmentation (non-English languages get split into many tokens)
• Typo brittleness (“the” vs “teh” become unrelated)
• Math breaks (“1+2=3” tokenizes differently than “1 + 2 = 3”)
• Code identifier inconsistency (getUserById = 1 token or 4?)
• Number reasoning collapses (large numbers get inconsistent splits)
• No morphological awareness inside tokens

The deeper theme: adaptive compute. Tokenization is one example of a broader idea — uniform per-token compute is wasteful, concentrate compute where signal is. Related techniques:

TinyGPT’s positioning is unusual: most platforms (HF transformers, llama.cpp, etc.) are BPE-locked. We already support byte-level + SmolLM2 BPE as first-class options. Adding BLT-style adaptive patches when open weights mature is incremental, not a rewrite. That’s a strategic asset worth preserving.

Three honest tiers of involvement:

1. Conservative (now): BPE default, byte fallback. Ship the product.
2. Adopt (12-24 months): integrate BLT when first strong open-weights release lands. Mac-native multilingual + typo-robust + code/text unified specialists.
3. Research (12+ months, optional): implement adaptive tokenization ourselves. Real research contribution if it works on tiny models — small models benefit disproportionately from adaptive compute because they have less compute to waste on easy tokens.

What “upgrading a model” covers

“Build a model” is half the story. “Upgrade an existing model” is the other half. Most have primitives shipped; gap is packaging as canonical upgrade operations:

Local-teacher architecture (confirmed 2026-06-06)

The product runs entirely offline. No API dependencies.

The architectural call: teacher labeling runs as an overnight or background process. Zero API spend. Users download the teacher once; reuse across all specialists. This is the differentiator.

Today’s PRD ship list (2026-06-06)

Net 14 PRDs in docs/prds/. Status snapshot as of end-of-session:

Shipped by elf today (all marked status: shipped-2026-06-06):

• E1 BFCL, E2 τ-bench, E5 HumanEval+sandbox, E7 judge, E8 train-time hook
• eval-leaderboard.astro, sae-timeline.astro
• Rust hf-downloader, Rust parquet-decoder
• dataset-decode-verify
• adam-state-persistence (B12 v2)
• cleanup-extractordata-datatrove
• cleanup-score-consolidation (RunBench.swift + GenerationUtils.swift exist)
• app-train-controls-thermal (TrainController.swift +174 lines, TrainView.swift +103 lines; some pieces like CLI --throttle flag may not be in this batch)

Shipped by elf today (status: shipped-2026-06-06):

• app-eval-tab — Eval tab in Mac app GUI (drafted + shipped same session)

Not started — top of the queue:

• train-controls-gap-closure — closes 3 audit-identified gaps (--throttle CLI flag + Adam verification + GitHubCorpus delete-after-parity), drafted 2026-06-06 EOD

Verified via swift build + grep: 839 insertions / 117 deletions across 24 files since the previous elf sync; --no-save-opt-state flag visible in train --help (confirms Adam state persistence shipped).

Sequencing

1. Now: N02 base finishes → A1 tool-call distillation proves the recipe works end-to-end. Without this, building UI on an unproven recipe is premature.
2. Then: tinygpt specialist CLI subcommand wraps the proven flow (data prep → teacher label → SFT/LoRA → eval). Headless API.
3. Then: App wizard built on top of the CLI. App = GUI for the proven flow.
4. Then: Templates for the 5-10 most common patterns ship as one-click flows.
5. Then: QLoRA path opens 13-30B fine-tuning. App can now offer “fine-tune a 13B model for your task on your Mac.”

Honest probability of the full vision landing

Assuming sustained execution at the current pace:

• A1 tool-call specialist lands within 2 weeks of N02 finishing: ~70%
• tinygpt specialist CLI wrapping the proven flow: ~85% (mostly packaging)
• App wizard built on the CLI: ~75% (Mac UI work, real effort)
• 5 production-quality templates: ~60% (need empirical tuning per template)
• QLoRA path opens 13-30B fine-tuning: ~70% (requires real ML engineering)
• Full vision shipping as a coherent v1 product: ~50%

The bottleneck is not technical capability. It’s:

1. Sustained execution against scope creep
2. UX/templates (the unsexy work)
3. Quality predictability (telling users honestly when their task won’t work)

What this changes about today’s plans

• Stop framing TinyGPT as “training tiny models from scratch on your Mac.” That’s one capability, not the product.
• Start framing it as “fine-tune or distill any model up to ~30B on your Mac for your specific task.” That’s the actual platform.
• The 22M-from-scratch project becomes a proof of recipe — it shows the pipeline (data → train → eval → ship) works end-to-end at tiny scale. Same recipe scales to 76M, 1B, 3B, 7B+LoRA.
• The eval pipeline shipped today is the measurement infrastructure for any specialist — not just our 22M one. Reusable for every fine-tune.
• The distillation recipe (docs/recipes/distillation-fc.md) is the training-side template for the same product.

Design principle: laptop-thermal-aware defaults

Not a new direction — overdue translation. TinyGPT has been explicitly Mac-first since day one. The audience target has always been the individual on a personal Mac. But the existing build’s defaults (training loop, throttle behavior, fan-load assumptions) still implicitly inherit datacenter-era assumptions from upstream tools — nanoGPT’s “for step in 0..<steps” loop, MLX’s “use all the silicon you can” philosophy, ML papers’ max-throughput benchmark culture. The platform was Mac-targeted at the API level but hadn’t yet retrofitted the defaults to reflect “this is running on a laptop.”

This section codifies the overdue fix: defaults should actively reflect the Mac-laptop audience that’s been the thesis from day one.

This makes TinyGPT the first Mac-native ML platform with laptop-aware defaults out of the box. The discipline is: assume the user’s machine is a laptop, not a server. Every default decision flows from there.

Maintainer learning roadmap (added 2026-06-06)

Personalized for outsider-perspective + AI-augmented + shipping-while-learning context. Not an academic curriculum — just-in-time depth on what the platform touches.

Highest-ROI sequence

What NOT to spend time on

• Linear algebra textbooks (don’t need matrix calculus by hand)
• PhD-level deep learning courses (Stanford CS224N — too academic)
• ML theory papers (NTK, information bottleneck — interesting but not actionable)
• Following every new paper on Twitter (drowns signal)

The pattern that works

• Build, then learn what you needed to build it — by the time you need to know X deeply, you’ve already used it pragmatically. Theory comes second.
• Use advisor as a tutor — cheap to ask, fast turnaround
• Just-in-time before commits — 2-3 hrs reading the relevant paper + strongest blog + one OSS implementation before adding a new capability
• Read companion docs: docs/learn/llm-mechanics-fundamentals.md is the matmul-first explainer this session produced

Open questions for future sessions

1. Should TinyGPT ship its own pretrained 3B base? Or rely on downloaded Qwen3-3B / Phi-3.5-mini as starting points? Argument for: architectural consistency, one binary handles everything. Argument against: weeks of training, doesn’t differentiate from existing strong bases.

2. Where does QLoRA fit? Implementing 4-bit base + LoRA adapters isn’t trivial in MLX-Swift. Might require waiting for MLX upstream or doing the integration ourselves.

3. Cloud teacher option. Allow user to point at Claude / GPT-4 API for labeling instead of local teacher? Lower friction, costs $.

4. App template marketplace? Other people’s specialist recipes shareable as JSON configs. Long-term play.

5. “Teacher RBS” clarification. User mentioned this in passing during 2026-06-06 strategy session; acronym unrecognized. Best guesses: teacher-reranker-student variant, or a paper-specific term. Treated as variant of teacher-student in current architecture list; revisit if user confirms a specific pattern.

6. Should the embedder specialist ship BEFORE the LLM specialist (A1)? (Parked 2026-06-06 EOD — decide after N02 lands.) The embedder track has fundamentally tighter eval loops: forward pass is ~10× cheaper than autoregressive generation; eval signal is exact (was-the-doc-in-top-K) not fuzzy; standard benchmarks (MTEB, BEIR, MS-MARCO) ship labeled pairs; eval takes seconds-to-minutes vs minutes-to-hours for LLM evals; metrics are cross-comparable (MRR/NDCG/Recall@K are universal).

   Concretely: embedder PoC end-to-end in ~1 week vs LLM specialist ~2-4 weeks. If “results first” is the operating principle, embedder might genuinely beat A1 to the first credible “small beats big on Mac” demo.

   Why parked: real comparison needs the N02 base done so we can score both paths empirically rather than estimate. Decision deferred to post-N02 discussion. PRD docs/prds/specialist-embedder.md is currently blocked-on-heavy-training-approval reflecting this gate.

Related docs (for full context)

• docs/recipes/distillation-fc.md — concrete distillation recipe
• docs/recipes/b25-scaledown.md — sibling specialist track
• docs/sessions/2026-06-05-eval-first.md — yesterday’s eval pipeline session
• docs/PLAN.md — canonical roadmap
• HANDOFF.md — pickup state for next session

2026-06-07 morning addendum — post-mortem + carry-forward

Lesson from the /tmp incident

Lost ~14 hours of N02 output (best.tinygpt val_loss 4.434, canonical at step 89,901, all step-N history + JSONL) when the Mac rebooted overnight and macOS wiped /tmp. Fix: docs/prds/persistent-training-output.md (filed 2026-06-07).

Generalized lesson: audit all “ephemeral by default” choices in the platform. /tmp for outputs, in-memory state without a sync hook, anything that depends on a process staying alive. Make persistence the default; volatility is the explicit opt-in. This is a platform-credibility issue — a tool that silently destroys 14 hours of work on reboot is not a tool people trust with serious training.

What we actually validated yesterday (despite the loss)

Even though we lost the weights, the recipe and learnings are intact:

1. Loss trajectory is real signal: 11.34 → 4.4-4.7 plateau over 90K steps. Honest interpretation: the Huge preset (22M params) + FineWeb-Edu may be under-capacity. Decay phase (180K-200K) might drop loss further, but not dramatically. Probably need Mega (76M) or distillation-from-bigger-teacher to get a base usable as A1 starting point.
2. PowerMonitor auto-pause works — caught thermal critical, saved checkpoint cleanly. Safety net validated.
3. App’s external-run detection works — surfaces live + exited training runs, renders loss curve from JSONL, ETA computed.
4. 75% throttle works but isn’t enough margin — sustained 75% still hits thermal critical. For laptop training, default should be 50% or even 40% to never hit .critical.
5. JSONL log truncates on --resume (write mode instead of append) — workaround was reconstructing history from text logs. Real bug; add to gap-closure PRD.

Updated default recommendations

Re-prioritization

Today (2026-06-07) priorities:

1. Fix /tmp default (filed PRD) — credibility-critical
2. App polish pass — locale formatter (“2,00,000” → “200,000”), throttle row layout, click targets, seed placeholder
3. N02 re-run tonight — persistent path, 50% throttle from start
4. SFT smoke — even against SmolLM2 baseline if our base doesn’t recover in time, just to validate recipe

Defer: embedder vs A1 sequencing, vllm-mlx investigation, cookbook content. All blocked on having a real base anyway.

Related docs (for full context)

• docs/recipes/distillation-fc.md — concrete distillation recipe
• docs/recipes/b25-scaledown.md — sibling specialist track
• docs/sessions/2026-06-05-eval-first.md — eval pipeline session
• docs/prds/persistent-training-output.md — 2026-06-07 post-mortem PRD
• docs/PLAN.md — canonical roadmap
• HANDOFF.md — pickup state for next session

TL;DR for myself in 3 months

You decided TinyGPT is a Mac app for individual task-specialization. The product is “bring data → pick teacher → ship a fast specialist.” Distillation + LoRA + QLoRA + Q4 inference lift the ceiling from “3B trained from scratch” to “30B fine-tuned locally + 70B-class quality via distillation.” The infrastructure is mostly built; the work is packaging, templates, and honest quality predictability. The A1 specialist landing is the proof-of-recipe that unblocks the platform pitch.

One day in 2026-06 we lost 14 hrs of training output to a /tmp default. It taught us the persistence-by-default principle. Don’t repeat that lesson.