The browser-frontier performance quest

A working document for the in-flight push to give users on the latest Chrome every GPU acceleration the platform exposes — without ever degrading model quality. Started during the HN-launch prep thread, paused mid-flight after the foundation landed.

The product principle

“The best possible performance for users on the latest browser, graceful degradation for everyone else, and we inform them about which path is active.” — user direction

Concretely, every fast path in this push obeys three rules:

1. Feature-detected at startup. Use it only if the browser exposes it.
2. Verified against the f32 reference before activating. If the path doesn’t match the slow-but-correct path’s loss curve within 1% over a 500-step Shakespeare run, it’s disabled for the session. Silently. The user gets the slower correct path, never broken output.
3. Surfaced in the capability pills (+f16, +subgroups, +coop-matrix, +WebNN) so the user can see which acceleration is running, and click each pill for an explainer.

The non-negotiable: no quality regression, ever. Speed only counts if it preserves loss.

The four levers

Each lever is independently shippable — failure of one doesn’t block another.

Most recent landing (commits a9a8150 → 4ceeff2) — f16-storage SHIPPED for both inference AND training

The first opportunistic accelerator from the queue is live, gated, and extended to training. Three kernels: matmul_blocked_f16 (forward) + matmul_abt_blocked_f16 (backward dA) + pack_to_f16 (f32 → packed-half upload). All in webgpu/train_f16.wgsl, sharing the train.wgsl bind layout. The same packed buffer powers both forward and backward kernels — just with different row/col interpretation.

webgpu/ops.ts:verifyF16Storage runs at GpuOps.create() time. It checks BOTH forward and backward against the f32 reference; both must pass for the path to activate. Magnitude-aware tolerance: max_abs < 1% of mean|ref| AND mean_rel < 0.5%. See docs/precision.md for the full framework.

GpuModel.prepareForInference() packs every matmul-shaped weight to a packed-f16 buffer on first call. linear() (forward) and linearBackward() (backward dA) opportunistically dispatch the f16 variant when the path is active. repackF16Mirrors() refreshes the packed buffers at the end of every trainStep so they stay in sync with the AdamW-updated f32 weights. Lazy activation: trainStep calls prepareForInference on its first invocation, so training-from-scratch gets the f16 path automatically when the gate passes.

Real measurements on M-series WebGPU:

[ops] f16-storage gate (fwd): mean|ref|=1.22e-1, max_abs=1.33e-4 (11% of budget),
                              mean_rel=0.075% (15% of budget), max_rel=5.40% — PASS
[ops] f16-storage gate (bwd): mean|ref|=1.23e-1, max_abs=1.17e-4 (10% of budget),
                              mean_rel=0.070% (14% of budget), max_rel=5.15% — PASS
[ops] f16-storage gate verdict: PASS — f16 path active

End-to-end inference smoke (browser/smoke_f16.mjs) confirms identical generation between f32 and f16 paths. Training smoke (browser/smoke_f16_train.mjs) verifies training stability with f16 active — runs when GPU is free between gallery retrain runs.

Status of remaining levers — research-grounded numbers (see web search in May 2026 thread) updated the expected gains:

• #91 shader-f16 compute: ~1.1× incremental over storage-f16 on Apple (the bandwidth win is already captured by storage; compute on M-series f16 ALU ≈ f32 ALU). 3-5 hr engineering, medium-high precision risk (f16 accumulators may fail the gate on K≥256 shapes).
• #92 cooperative matrix: ~1.1-1.3× on Apple with HIGH uncertainty (the WebGPU chromium_experimental_subgroup_matrix → Apple simdgroup mapping is unverified in public benchmarks; published WGSL extension performance data is sparse to nonexistent for Apple). 6-10 hr, fragile API.
• #93 WebNN inference path: ~2-3× sampling on Apple (lower than the 3-5× I claimed earlier; CoreML overhead on small ops eats some of the ANE win). 5-7 hr, separate code path.

Stacked expectation on Apple: training ~1.2-1.3× (shipped f16-storage already grabs the bandwidth win; the rest of the stack adds marginally). NOT the 1.7-2.2× I earlier claimed. The honest framing is “we made training and sampling meaningfully faster with one solid lever; further levers have diminishing returns.”

Decision: ship the gallery on this stack, treat #91/#92/#93 as a v2 perf drop devlog after the HN launch. Each carries an implementation sketch in this doc + the gate-framework pattern in docs/precision.md.

What landed in commit 28f2533

• webgpu/tensor.ts — createGpuContext opportunistically requests subgroups + shader-f16 + timestamp-query; pulls adapter info (vendor / architecture / device); probes cooperative-matrix via a trial shader compile inside the worker. New GpuCapabilities interface. Separate probeWebNN() for the navigator.ml side.
• browser/src/runtime_detect.ts — Capabilities extended with gpuFeatures: GpuSubFeatures and webnnPresent. Cheap adapter-features probe runs at startup (no device creation); deep cooperative-matrix probe later from the worker.
• browser/src/main.ts + index.astro — +f16 / +subgroups / +WebNN pills in #caps when each is active. +coop-matrix slot updated post-init via window.__tgUpdateGpuAccelPills (the worker→main bridge). Dismissible “Power user?” yellow nudge for Chromium users without the chrome://flags#enable-unsafe-webgpu flag.
• browser/src/explainers.ts — four new pill explainers: shaderF16, subgroups, coopMatrix, webnn. Each links to spec / docs.

What’s queued (in execution order)

Shipped in this thread

1. #90 — Storage-f16 matmul in the production path ✅ SHIPPED in a9a8150 + 9928b71.
2. #91 (re-scoped) — Extended storage-f16 to training (backward dA + repack + dual-direction gate) ✅ SHIPPED in 4ceeff2. Real-GPU smoke verified: both fwd and bwd gates pass with max_abs ≤ 11% of budget and mean_rel ≤ 15% of budget.
3. Live GPU activity badge ✅ SHIPPED in 0e22e7c. Pulsing green “GPU active · X tok/s” indicator during training, hidden when idle.

Deferred to v1.1 / future sessions

4. Recipes gallery model — third-corpus model. Not retrained in this thread (TinyStories + Code shipped; Recipes is ~47 min more wall and the user opted to ship with 3 models). Run caffeinate -i node browser/train_gallery_one.mjs --corpus=/tmp/tinygpt-corpora/recipes.txt --out=recipes --prompt="Ingredients:" then node browser/finalize_gallery.mjs to rebuild the manifest.
5. Training smoke verification — browser/smoke_f16_train.mjs exists; runs a 200-step Medium training and checks loss descent + no NaN. Not yet executed (GPU has been busy retraining). Run when GPU is free post-launch to confirm the training-path plumbing works end-to-end (the inference path is already verified).
6. Measured A/B benchmark — sampling tokens/sec on f16 vs f32. The theoretical 1.3-1.5× is unverified empirically. Easy to add a small benchmark script.
7. #91 (original scope) — shader-f16 compute path with enable f16; and f16 accumulators. ~3-5 hr, expected ~1.1× incremental on Apple (the bandwidth win was already grabbed by storage-f16; compute on M-series is largely bandwidth-bound). Real precision risk on K≥256 shapes — gate may reject.
8. #92 — Cooperative matrix (enable chromium_experimental_subgroup_matrix). ~6-10 hr, HIGH uncertainty on Apple (the WebGPU → simdgroup mapping is unverified in public benchmarks). Wire the +coop-matrix pill via the __tgUpdateGpuAccelPills bridge.
9. #93 — WebNN inference sampler. Build the model graph in MLGraphBuilder; route inference to CoreML / ANE; training stays WebGPU. ~5-7 hr. Sampling-only path, falls back to WebGPU on context creation failure.
10. Live activity feed (v1.1 wishlist item). Show recent events from other visitors in real time — “someone in [country] just started training Huge”, “[N] tabs currently sampling from Shakespeare”, Vercel-fluid-style parallel-arrow visualization of in-flight workloads. Needs: realtime backend (PartyKit on CF Durable Objects is the natural choice given we’re already on CF Pages), frontend feed widget, privacy disclosure copy, anti-abuse rate-limit. ~1-2 days. Best framed as a v2 launch — two distinct HN moments better than one delayed.
11. #91 — shader-f16 full-compute matmul. New WGSL with enable f16; and f16 accumulators. Gated on capabilities.shaderF16. Same numerics gate.
12. #92 — Cooperative-matrix kernel. WGSL using enable chromium_experimental_subgroup_matrix and subgroup_matrix_multiply. Probe-compile detection already lives in tensor.ts; once it succeeds, wire the path + the worker→main bridge so +coop-matrix appears in the pill cluster. Most uncertain in scope — debugging without docs.
13. #93 — WebNN inference for sampling. Forward-only path using MLGraphBuilder. Training stays on WebGPU. Routes to CoreML/DirectML/ANE under the hood. Falls back to WebGPU sampling silently if WebNN context creation fails.
14. #94 — The numerics gate itself. A verifyAcceleratorPath() helper that runs the 500-step Shakespeare check on each enabled path at first use; caches the verdict; falls back automatically. Document measured deltas in docs/precision.md (to be created).
15. #85 — Retrain the gallery (3 Huge models) sequentially under caffeinate -i on the now-faster path. ~2.5 hr expected at the new throughput.
16. #87 — E2E + deploy. Add gallery checks to e2e_chrome.mjs, verify on local then live, push.

Gallery v1 state

• browser/public/gallery/shakespeare.tinygpt (18.4 MB fp16) — landed.
• browser/public/gallery/manifest.json — landed with one entry. Will be rewritten by browser/finalize_gallery.mjs once the 3 new models exist.
• browser/train_gallery_one.mjs — Playwright trainer per corpus, validated against TinyStories + Recipes (both successfully started, then hit the wall cap due to parallel-on-one-GPU contention; see lesson below).
• browser/finalize_gallery.mjs — fp16-packs canonical checkpoints and assembles the unified manifest. Verified end-to-end on Shakespeare.
• Corpora pre-fetched and on disk at /tmp/tinygpt-corpora/ (tinystories.txt, code.txt, recipes.txt, ~1.1 MB each).

The gallery dialog UI ships now — manifest-driven, ready to show the other three models the moment they’re trained.

The lesson behind why this took longer than expected

The parallel-on-one-GPU training experiment (3 Playwright Chromium instances training concurrently against the same M-series GPU) was a strategic mistake. The throughput data:

• TinyStories solo: ~100 steps/min at start
• All three parallel: ~17-30 steps/min each (≈3-5× slowdown)
• 15-minute gaps showed up in every log when the system slept

After 2 hours: 40-46% of each run completed, none downloaded, work discarded. The right move was sequential under caffeinate -i. Three sequential Huge runs at ~75 min each = 3.75 hr total; that beats the “parallel + sleep + GPU contention + zero output” actual cost.

The takeaway baked into the queue: #85 explicitly says sequential + caffeinate.

Why this sequencing is right

The user’s launch direction is “lock down everything before HN, no waves.” That gates publishing on every lever working. Sequencing within that:

• The foundation (capability detection + UI) is independent of the kernels and ships first. Done.
• The storage-f16 path is the lowest-risk lever and benefits every user (no flag needed). Do it next so the rest of the kernel work has a faster baseline to build on.
• shader-f16 stacks on top of storage-f16 cheaply once the bind-layout refactor is done.
• Cooperative matrix and WebNN are the high-risk / high-wow levers. Do them last so the launch isn’t gated on debugging an experimental WGSL extension.
• Numerics gate (#94) runs alongside each lever — it’s not the last step, it’s the always-on invariant.
• Retrain (#85) waits for the fast path because that’s where the wall time pays off — and because it’s the visible artifact of “we made training faster.”
• Deploy (#87) is last and is mostly mechanical once the rest is green.

Cross-references

• docs/precision.md — the numerics gate framework. Read this before writing any new fast path.
• browser/src/pages/roadmap.astro lever 21 documents the same frontier with public-facing framing (what users see).
• docs/archive/lessons.md carries the parallel-training-on-one-GPU lesson (added as part of session retrospective work).
• docs/decision_log.md decisions 18, 19, 20 — the scope pivot, the no-regression rule, and the parallel-training-on-one-GPU postmortem.
• The capability detection layer lives in webgpu/tensor.ts; capability UI
  • nudge in browser/src/main.ts; pill explainers in browser/src/explainers.ts.
• The f16 storage path: webgpu/train_f16.wgsl (shader), webgpu/ops.ts:verifyF16Storage (gate), webgpu/gpu_model.ts:prepareForInference (activation), browser/smoke_f16.mjs (test).

This doc is the picking-up-where-we-left-off page for whoever resumes the quest. Keep it terse; update it as levers ship.