Mac inference baseline — M5 Pro / 48GB

Date: 2026-05-31 Hardware: Apple M5 Pro, 48 GB, 18 cores (6 perf + 12 P), macOS 25F71 Harness: tinygpt bench --engine tinygpt (greedy decode, seed=42) Commit: d4a9de6 Question: Where is the actual bottleneck on M5 Pro? Is cider’s W8A8 worth the 4-7 day port?

TL;DR

The Mac is 10-20× faster than the realtime/interaction-model targets across every model size we have. cider’s prefill win is real but immaterial at current scales; defer until there’s a 3B specialist or a model-size-driven slowdown.

*decode tok/s varies with how full the context is at start of decode

Realtime targets (from roadmap §realtime):

• TTFT (warm): < 50 ms → we’re at < 6 ms p99 across all sizes (10× under target)
• ITL p99: < 30 ms → we’re at < 5 ms p99 across all sizes (6× under target)

Raw results

Run 1 — mac-trained.tinygpt (9.6 M params, browser gallery)

Workload: prompt=64, gen=128, n=25, warm=3

Run 2 — flagship-huge-v5.tinygpt (221 M params)

Workload: prompt=64, gen=128, n=20, warm=3

Run 3 — flagship-huge-v5.tinygpt, prefill-heavy

Workload: prompt=128, gen=128, n=20, warm=3 (context fills to ctx=256 cap)

Run 4 — mega-pilot.tinygpt (~960 M params, 1.1 GB on disk)

Workload: prompt=64, gen=64, n=10, warm=2

Run 5 — google/gemma-3-12b (LM Studio, MLX, 12B params, 8.07 GB on disk)

Workload: prompt≈64 (default system+user template in bench_decode.py), gen=128 max (model stops at ~47 tokens — task is “three sentences”), n=20, warm=1. M5 Pro, macOS 26.5, gemma3 MLX runtime. RSS polled on the LM Studio inference worker (a node process; the lmlink-connector subprocess holds only the IPC bridge, ~38 MB — not the model).

Reproduce:

lms load google/gemma-3-12b --identifier google/gemma-3-12b
WORKER=$(ps -axo pid,rss,comm | awk '/lmstudio.*node/ && $2>1000000 {print $1; exit}')
python3 scripts/bench_decode.py \
  --url http://127.0.0.1:1234/v1/chat/completions \
  --model google/gemma-3-12b --rss-pid "$WORKER" \
  --jsonl docs/research/data/gemma-12b-decode.jsonl \
  > docs/research/data/decode-gemma-12b.json

Takeaways (verified against this row, not the from-scratch ones above):

• TTFT 184 ms p99 is 30× the from-scratch flagship’s 5.83 ms — Gemma-12B at this scale spends most of its TTFT in the model’s per-token compute, not Mac/MLX overhead. The realtime <50 ms TTFT target is broken here; expect 1–2 visible “thinking” beats at the start of every reply.
• Decode tok/s 36 < the 50 tok/s realtime floor that the SLM leaderboard composite uses (scripts/score_formula.py). A user-facing word every ~28 ms is on the lower edge of “feels realtime” — usable but visibly slower than smaller models.
• 9.1 GB resident for a 12B-Q4-class model: leaves ~38 GB for everything else on a 48 GB Mac. Fine for development, tight if a second model is loaded for cascade/cloud-escalate routing.

Implications for the cider decision

What cider would buy us

Per docs/research/wave_2_5_kernel_audit.md §2, cider on M5 Pro:

• W8A8 prefill: 1.2-1.9× faster on Qwen3-8B / Qwen3-VL-2B
• W8A8 decode: slightly worse than W8A16 (memory-bandwidth-bound, KV cache still fp16) — 104 vs 107 tok/s on Qwen3-VL-2B
• 8B model: 9.726 → 9.756 PPL, 179.9 → 123.5 s prefill, 18.93 → 11.32 GB peak memory

Why it doesn’t help us right now

1. Model size mismatch. cider’s wins scale with matmul cost. At 221M / 960M params on M5 Pro, the model loads in <700 MB and prefill is 14k-56k tok/s — already saturating GPU compute throughput. Int8 saves nothing material until matmul dominates.

2. Realtime targets met by 10×. The Mac is already faster than anything the interaction-model demo needs. cider’s 1.8× prefill on an already-fast prefill is “we noticed”, not “now we can ship the demo.”

3. Decode is what limits agent latency — the agent loop generates tokens one at a time and cider’s W8A8 is slower than W8A16 on decode. We’d actively regress agent UX.

4. The decode tok/s gap with model size suggests memory bandwidth is the limit at 1B+, not int8 vs fp16 arithmetic. cider doesn’t touch the bandwidth picture.

5. Effort cost is high. cider is Python+C++ targeting MLX’s C++ primitive interface; tinygpt is MLX-Swift. Port is 4-7 days, not the 1-2 days a Python project would face. See research/wave_2_5_kernel_audit.md for the integration analysis.

When to revisit

Revisit cider adoption when ANY of:

• Training a 3B+ specialist where prefill is ≥1s and worth shaving
• Memory pressure forces W8 weights anyway (then we want W8A8 not just W8A16 since activations would otherwise upcast)
• Long-context (8K+) workloads where prefill cost dominates
• The bench harness shows decode degradation at larger model sizes that ANE-prefill + GPU-decode hybrid (Wave 2.6 deferred item) could address jointly

Other Mac-speed levers ranked

Given the baseline, here’s what’s actually worth doing next for “Mac speed”:

Bench harness limitations to flag

• --prompt-tokens is bounded by model context length (256 for the current flagship-huge). The harness fails silently if exceeded — filed as a TODO to add a clear error.
• --no-energy disables powermetrics (which needs sudo); J/token numbers are not in the baseline.
• “git tree dirty” warning is currently spurious (.claude/ + default.profraw only). Could be filtered in the harness.

Conclusion

Do NOT spend 4-7 days on cider right now. The Mac is already faster than the product demands. The lever has marginal payoff at current scales and active regression risk on decode.

What to do instead:

1. Document this baseline (this doc)
2. Move to the next product-shaped Wave 2.6 work (Continue.dev provider adapter is the highest-leverage per wave_4_landscape.md)
3. Revisit cider after training the first 3B specialist