Session retrospective — 2026-05-31

Branch: main · Final commit: 813b4da

This doc captures the arcs from a single multi-hour session that moved a lot of pieces. Future-Claude / future-me reads this first on a fresh checkout to know where things are.

The arc

The session opened with Wave 2.5 (Metal kernels) stalled, no Wave 3 specialist trained, and a vague backlog. It closed with:

A decision-grade audit of the kernel work → 4 of 5 items dropped/deferred-with-reason
A landscape audit of the competition → repositioning
5 product-shaped Wave 2.6 items shipped end-to-end
Every CLI in the binary smoke-tested green
A modality × action capability matrix
A first specialist trained on real data (and a major bug found
- fixed along the way)
A living ROI-ordered backlog as single source of truth

What shipped (28 commits this session)

Wave 2.5 — research closed it, mostly with DROP/DEFER

Two research agents audited the 5 kernel items. Outcome:

Item	Verdict	Why
Flash Attention Metal kernel	DROP	MLXFast SDPA already fused; our 9 call sites hit the fast path
Int4 packed matmul Metal kernel	DROP	MLX `quantized_matmul` is already hand-tuned
Int8 W8A8 / cider	DEFER	Mac is already 10× under realtime targets at current model sizes; revisit at 3B+
ANE + GPU heterogeneous routing	DEFER	Apple’s Stateful Models API is the gating event
WebGPU subgroup matmul	BUILD	Shipped; gate failed (1415% mean_rel); fallback engaged unchanged

Decision docs: docs/research/wave_2_5_kernel_audit.md, docs/research/mac_decode_baseline_m5pro.md.

Wave 2.6 — product-shaped features shipped

Item	Commit	Notes
Cloud API client (Anthropic + OpenAI)	`ef0e5e3`	curl shell-out, env-var auth
SSE streaming on serve	`e754d6c`	OpenAI-compatible chunks
SSE cancellation on disconnect	`c11265b`	SIGPIPE-safe
CloudEscalate → AgentLoop	`4119216`	synthetic `escalate` tool wired into agent runtime
Ollama-compat provider adapter	`79d6f40`, `9b76089`	Continue.dev / Cline / Aider drop-in
ScreenCaptureKit + AX tree	`08a7689`	AX works end-to-end; SCKit capture has CGS quirk from CLI
Tool-call extractor (mini-router)	`b5bbdd9`	full scaffold: data → train → infer → agent flag
MILU + IndicGenBench eval CLI	`3385a76`	wired with smoke fixtures; real-data baseline pending
Continue.dev walkthrough doc	`79d6f40`	`docs/continue_provider.md`

Wave 4 — landscape research

Single research agent audited TML, Apple Foundation Models, code-agent competitors (Cursor / Continue / Cline / Aider), and the Indic landscape. Key outputs:

TML’s “Interaction Models” framing (foreground+background) adopted as vocabulary in the roadmap
Apple FM is the same architecture but walled — tinygpt positions as open / hackable / multi-specialist / route-anywhere
Continue.dev/Ollama compat adapter identified as highest-leverage product move → shipped same session
desi-max correction: it’s a text-to-image LoRA, not a Hindi LLM. Replaced with Sarvam-Edge / AI4Bharat Airavata across the docs.

Doc: docs/research/wave_4_landscape.md.

Feature audit + capability matrix

Every CLI subcommand (30+) smoke-tested end-to-end on M5 Pro. Quantization (HQQ + GPTQ + prune + LASER), fine-tuning (SFT + DPO + distill + ES + tuned-lens + train-heads + train-extractor), PEFT variants (LoRA + DoRA + VeRA + RsLoRA + LoRA-FA + PISSA + LoftQ + AdaLoRA + LayerDrop), and all the inference / eval / data / cloud / screen surfaces — all green.

Plus the modality × action matrix that pins down what tinygpt does NOT do (vision, audio, multi-modal text+image / text+audio — all research-grade and deferred).

Docs: docs/feature_audit_2026_05_31.md, docs/capability_matrix.md.

First specialist run + a real bug

Trained toolcall-v1 on SmolLM2-135M + hermes-function-calling-v1 (11k records, 2000 steps, RS-LoRA rank 16). End-to-end pipeline works. 135M @ 2000 steps picked up surface JSON format but not functional tool-calling — too small or too few steps for genuine generalization.

Along the way: found + fixed the A0 LoRA-save bug (f566023): SFT’s curated DoRA-on-by-default wasn’t being disabled by PEFT-variant flags (--rs-lora, --vera, etc.). So makeAdapterLinear() returned DoraLinear instances, but the writers only handled LoraLinear — every saved adapter was an empty entries:[] header-only file. The trainable param walker handled both classes, so the count was always right, hiding the bug. All previous SFT / DPO outputs across the repo were silently empty. Fix is one line per PEFT case.

Doc: docs/specialist_v1_findings.md.

Decision docs added this session

docs/research/wave_2_5_kernel_audit.md — kernel BUILD/DEFER/DROP
docs/research/wave_4_landscape.md — TML/Apple/agents/Indic
docs/research/mac_decode_baseline_m5pro.md — bench numbers + cider verdict
docs/research/indic_evals.md — MILU + IndicGenBench design
docs/research/data/jitter_*.json — raw bench JSONs (4 files)
docs/async_tool_dispatch.md — investigated + skipped with reason
docs/continue_provider.md — Continue.dev quickstart
docs/tool_call_extractor.md — mini-router scaffold design
docs/specialist_v1_findings.md — toolcall-v1 result + next steps
docs/feature_audit_2026_05_31.md — full CLI smoke
docs/capability_matrix.md — modalities × actions
docs/progress.md — Mac+Web dashboard (live)
docs/backlog.md — ROI-ordered TODO (living)
docs/data_inventory.md — dataset reference
docs/session_2026_05_31.md — this doc

What was learned

Architectural

The Mac is already 10× under realtime targets at every model size we have (9.6M → 960M). cider’s W8A8 has marginal payoff until a 3B+ specialist exists. Don’t pre-optimize.
Apple owns the architecture we’re chasing — the same on-device-3B + Private-Cloud-Compute model with adapter-only customization. tinygpt’s edge is open / multi-specialist / route-anywhere, NOT being a better Apple FM.
TML is cloud-only for interaction models. Our on-device version is genuine differentiation.
The router is a GPT trunk with a classifier head. Backlog item B2b is the bake-off vs pure-GPT-with-FSM-constraint to settle whether the architectural deviation is worth keeping.

Process

Parallel agents work when the briefs are tight and the worktree is sliced clean. 4 agents shipped this session (WebGPU SG / ScreenCapture / Indic evals / tool extractor) — all committed their own work. Sandboxed worktrees + clear file-ownership directives kept merge conflict to zero.
Research before building saves weeks. Wave 2.5 collapsed from “5 hard items” to “1 build” + “1 adopt-on-M5” + “3 drops/defers” with two days of agent research.
Validate save-and-reload paths in tests. The LoRA-save bug was latent for the entire history of the project because no test exercised the SFT → save → reload → infer roundtrip. Backlog C7 plugs this gap.
/tmp is reaped on macOS. Lost binary + 50 MB of pulled data mid-session. Builds and downloads should default to ~/.cache/tinygpt/ and ~/.local/bin/. Backlog C8.
CLI-flag interactions matter. The DoRA-on-by-default vs. PEFT-variant flag silent override was a real footgun. Curated defaults should be explicit losers when any explicit choice is made. Fixed for the PEFT family; worth auditing other CLI defaults for similar shadowing.

About the project

The project is now best described as: a text + code + structured- output transformer toolchain with quantization, PEFT, spec-decoding, agent runtime, screen-text, and cloud-escalation — train-able from scratch on Apple Silicon and browser WebGPU, with the full Wave 2.6 product surface (Continue.dev compat, AX tree, SSE streaming, mini-router, cloud escalation) shipped.

Not multi-modal in the vision/audio sense yet; that’s research-grade work deferred behind specialist training.

What to do when the laptop is back

Pick up from docs/backlog.md. The top Tier A items today:

A1 Train the first specialist end-to-end (toolcall-v2 with more steps, OR pivot to bigger base SmolLM2-360M, OR switch to debugger specialist on GitHub data — see specialist_v1_findings.md for the ranked options C → D → A → B)
A2 Re-pull tier-1 datasets (/tmp was reaped, but the cache layer at ~/.cache/tinygpt/datasets/ is persistent — most should be one re-pull away)
A3 Fetch GitHub issue→PR for a real OSS repo (debugger data)
A4 BFCL + τ-bench via extractor-data
A5 Indic eval datasets (MILU + IndicGenBench-XQuAD)
A6 Dataset inventory doc — DONE this session
A7 Real-data MILU baseline

Tier C polish items if blocked:

C6 ChatML template: detect inline system: prefix and split
C7 Save+reload XCTest for LoRA (plugs A0 regression-coverage)
C8 Install-path discipline (no more /tmp)

Deferred-with-trigger items (Tier D) — don’t touch unless trigger fires. See docs/backlog.md for the trigger conditions.

Total ledger

Commits this session: ~28
Files added: ~20 docs, multiple Swift modules (Screen, EvalIndic, tool-router pipeline, Ollama handlers, Continue.dev doc, etc.)
Bug fixed: A0 LoRA save (was silently empty since project start)
CLIs verified: 30+ (full audit green)
Specialists trained: 1 (toolcall-v1, pipeline-validation quality, not deployment-ready)
Research dives: 2 (Wave 2.5 kernels, Wave 4 landscape)
Modalities documented: 12 (text/code/structured/multilingual/ screen-AX/screen-image/vision/audio/multi-modal × shipped/partial/ deferred)

How this doc fits with the others

docs/session_2026_05_31.md ← you are here (retrospective)
    │
    ├── points at ──→ docs/progress.md       (live Mac+Web dashboard)
    ├── points at ──→ docs/backlog.md        (live ROI-ordered TODO)
    ├── points at ──→ docs/capability_matrix.md  (modalities × actions)
    ├── points at ──→ docs/data_inventory.md (dataset reference)
    ├── points at ──→ docs/specialist_v1_findings.md  (first run + next steps)
    ├── points at ──→ docs/feature_audit_2026_05_31.md  (CLI smoke)
    │
    └── decision docs:
        ├── docs/research/wave_2_5_kernel_audit.md
        ├── docs/research/wave_4_landscape.md
        ├── docs/research/mac_decode_baseline_m5pro.md
        ├── docs/research/indic_evals.md
        ├── docs/async_tool_dispatch.md
        └── docs/tool_call_extractor.md

progress.md and backlog.md are the two living docs — those get updated as work ships. The rest are point-in-time decision logs that should not be edited; supersede them with new dated docs if the analysis is revisited.