Speech & systems topics — an interview-grade map

Eight topics that came up in a real technical interview, mapped the house way: Learn it (best external source — we don’t re-teach), why it matters to THIS project, and in the repo where applicable. Several of these aren’t theory here — tinygpt/Pace has shipped them.

Where we’re starting: you’ve built the cascade (Pace is literally speech → text → LLM → text → speech, all local) — this doc connects what you’ve built to the named concepts interviewers ask about.

1. Voice pipeline tail latency (p50 vs p95)

What: p95/p99 latency is dominated by rare slow events (GC, cold caches, model loads), not the average path; you fix tails differently than you fix means.

Why it matters here: Pace’s doctrine is 100ms-on-local. The TTFW hunt (330→119ms) was a tail-latency exercise: the fix was caching constraint_init vocab iteration — a one-time cost that poisoned every first request. Same shape as any p95 story.

Learn it: The Tail at Scale (Dean & Barroso) — the canonical paper, 8 pages.

In the repo: native-mac/Sources/TinyGPTServe/Serve.swift (token-mask caching at boot); scripts/profile-pace-turn.sh in the Pace repo breaks a voice turn into stage timings — that’s a p95 budget in script form.

2. ASR accuracy — WER and text post-processing

What: Word Error Rate = (substitutions + insertions + deletions) / reference words. Post-ASR NLP (normalization, punctuation, entity fixing — what spaCy gets used for) reduces effective WER without touching the acoustic model.

Why it matters here: WhisperKit large-v3-turbo was qualified at “perfect accuracy on ‘Pace’ without phrase biasing” — that qualification WAS a WER eval. The Stage-A dictation post-processor specialist is exactly the “fix text after ASR” pattern.

Learn it: Hugging Face Audio Course (unit on ASR evaluation); jiwer for computing WER in 3 lines; spaCy docs for the NLP layer.

In the repo: Pace repo WhisperKitTranscriptionProvider.swift (streaming re-transcription strategy is documented in its header comment); dictation post-processor under Pace’s specialist work (task #295/#318).

3. Cascaded vs direct speech-to-speech

What: The cascade (ASR → LLM → TTS) is modular and debuggable but loses prosody/emotion and stacks latency; direct S2S transformers (speech tokens in, speech tokens out) collapse the stack at the cost of training difficulty and control.

Why it matters here: Pace IS the cascade, deliberately — each stage is independently swappable (we just swapped the middle for Gemma-3-12B with zero changes to ASR/TTS). Knowing exactly what the cascade gives up is knowing Pace’s ceiling.

Learn it: Moshi (Kyutai) — the reference full-duplex S2S paper; Qwen2.5-Omni for the “thinker-talker” hybrid that keeps a text spine.

In the repo: the whole Pace pipeline; tinygpt’s contribution is the middle box (planner eval + serve). The historical docs/DRILLDOWN.md planner drill shows why the modular middle is a feature — you can’t A/B 12 planners inside a fused S2S model.

4. Fine-tuning a decoder-only model when loss won’t drop

What: The debugging ladder: overfit one batch first; check the loss mask (are you training on prompt tokens?); check the chat template matches the base model exactly; LR warmup; then schedule (WSD) and stability tricks.

Why it matters here: v1–v11 of the Pace specialist plus clarify-v1 were eleven rounds of this ladder, including a real catastrophic- interference case (47pp OOS regression from 38 training rows) and a thinking-mode template mismatch that silently burned an hour.

Learn it: Karpathy — A Recipe for Training Neural Networks — still the canonical debugging ladder.

In the repo: native-mac/Sources/TinyGPT/Train.swift (WSD schedule, loss-spike recovery + replay); docs/learn/session-08-training-mechanics.md covers the mechanics ground-up; historical docs/RETROSPECTIVE.md covers the v1–v11 arc.

5. Feature selection — PCA vs recursive elimination vs isolation forest

What: Three different jobs often conflated: PCA transforms features into orthogonal components (loses interpretability, keeps variance); RFE selects original features by iteratively dropping the weakest; Isolation Forest is an outlier detector (short isolation paths = anomalies), used in feature pipelines to clean data, not select features.

Why it matters here: the closest analog in this repo is a genuinely good interview answer: SAEs vs PCA. PCA finds orthogonal directions of max variance; a sparse autoencoder finds an overcomplete, sparse basis — which is why interp uses SAEs (features stay monosemantic) where classic DS uses PCA.

Learn it: scikit-learn docs are the fastest route: PCA, RFE, IsolationForest; Elements of Statistical Learning ch. 3 + 14 for the theory.

In the repo: the SAE work (tasks #195/#199/#224, docs/learn/ SAE notes) — bring the SAE-vs-PCA contrast when asked about dimensionality reduction.

6. Credit risk modeling

What: Predicting default probability from tabular features; the industry stack is logistic regression / gradient-boosted trees + careful feature engineering, with interpretability as a regulatory requirement (why deep nets lag here).

Why it matters here: it doesn’t — this is the one topic with no repo anchor. Learn it as the canonical tabular + interpretability-constrained ML setting, the opposite pole from everything tinygpt does.

Learn it: scikit-learn’s gradient boosting guide + ESL ch. 10; for the domain framing, search “PD/LGD/EAD modeling” — probability of default, loss given default, exposure at default are the three regulated quantities.

7. Queues vs WebSockets for voice bots

What: A WebSocket is a transport (one live duplex pipe); a queue is a decoupling primitive (backpressure, retry, burst absorption, consumer independence). Voice bots want both: sockets at the edge for low latency, queues between internal stages so a slow TTS render doesn’t stall ASR ingest.

Why it matters here: Pace solves this in-process — the TTS sidecar renders sentence N+1 while N plays (a 1-deep pipeline queue), and audio samples accumulate in a lock-guarded buffer while transcription runs on its own cadence. Same pattern, no broker.

Learn it: Designing Data-Intensive Applications ch. 11 (stream processing — the queues-vs-direct-connection tradeoff in full).

In the repo: Pace repo WhisperKitTranscriptionProvider.swift (sample buffer + 1.2s partial cadence = producer/consumer with backpressure); the Kokoro sidecar pipelining described in Pace AGENTS.md.

8. GPU parallelism — FSDP2 and friends

What: The parallelism menu: data parallel (replicate model, split batch), FSDP/ZeRO (shard params+grads+optimizer state, gather per-layer just-in-time), tensor parallel (split individual matmuls), pipeline parallel (split layers across devices). FSDP2 is PyTorch’s rewrite with per-parameter DTensor sharding — composable with the others.

Why it matters here: tinygpt is mostly the opposite regime — one Mac, unified memory, zero inter-device communication. Knowing FSDP2 is knowing exactly what you’re NOT paying for (all-gather bandwidth, sharding bugs) and why single-device MLX training tops out where it does (the Mega-bf16 OOM was solved by config, not sharding — there’s no second device to shard to). As of 2026-06-17 there’s a small exception: scripts/archive/dist_dp_poc.py is a working data-parallel all-reduce PoC over mlx.distributed on a single Mac, n=2/4 ranks, bit-identical replicas, effective batch 64→256. That’s the buildable scale-out path if/when a second Mac shows up — the boundary between “scale up” and “scale out” is now drawn in code, not just in this doc.

Learn it: HF Ultra-Scale Playbook — the current canonical treatment of all parallelism forms, interactive; PyTorch FSDP tutorial for the FSDP2 API itself; mlx.distributed for the Apple-silicon equivalent.

In the repo: native-mac/Sources/TinyGPT/Train.swift is the single-device anchor; scripts/archive/dist_dp_poc.py is the bit-identical data-parallel PoC over mlx.distributed (2026-06-17); docs/learn/session-05-scaling.md covers why scale-up beat scale-out for this project.

Suggested order

1–4 and 7 first (they’re anchored in code you own — read the anchor, then the source). 8 next (one playbook read). 5–6 are pure-external study; ESL chapters are the slowest material here, budget accordingly.