← TinyGPT · docs · devlog · roadmap · speedup
source: docs/prds/macos26-int8-ane-handoff-port.md · view on GitHub ↗

Port macOS 26 int8 direct ANE array handoff into M8 chain

Status: IN PROGRESS (2026-06-10) — Phase A SHIPPED, Phase B in flight. Spawned out of the 2026-06-09 research sweep + anemll-vs-M8 decision. Target: 17 tok/s → ~30 tok/s on the existing Qwen3-0.6B 28-block ANE chunked pipeline.

2026-06-10 implementation findings (override assumptions below)

  1. The “1.8× handoff” claim decomposes into two separate wins. Inter-chunk array handoff (Phase A) and int8 weights (Phase B). Draw Things’ 1.8× is mostly Phase B — weight bytes/token — not the boundary arrays (the boundary tensor is only 4 KB at H=1024).
  2. Phase A SHIPPED (commit f2687ef): fp16-IO export + IOSurface CVPixelBuffer ping-pong via MLPredictionOptions.outputBackings (the anemll pattern — anemll does NOT use int8 handoff itself, contrary to the assumption below). Measured: prefill 14.1 → 17.1 tok/s (+21%), decode flat at 22 tok/s, numerics gate PASS (min cosine 0.9999957, 100% top-1, ’ Paris’ canary).
  3. macOS26 deployment target BREAKS ANE binding on this machine (macOS 26.0 / coremltools 9): int8-weight blocks converted with minimum_deployment_target=macOS26 fail at predict with “Unable to bind buffer to network @ BindBuffers”. int8 WEIGHT compression only needs macOS13+, so Phase B uses the macOS15 target — works (block 0: ANE load OK, cosine 0.999994, 16 MB package, per-channel linear-symmetric).
  4. Numerics gate exists: scripts/ane/m8_numerics_gate.py — 5 prompts × 8 greedy steps vs saved fp32 baseline; PASS = 100% top-1 + cosine ≥ 0.999 + canary. Run on every phase. Baseline: ~/.cache/tinygpt/ane/m8-gate-baseline.npz.
  5. Phase C (int8 activation IO) stays gated/optional — it is the only part that truly needs the macOS26 target, which is currently broken (#3).

What “int8 direct handoff” is

Before macOS 26: passing weights/activations into a CoreML model running on ANE required fp16 or int32 buffers. Any int8 quantized weight got promoted to fp16 at the array-binding boundary, eating ANE memory bandwidth.

In macOS 26 (shipped at WWDC 2026 yesterday): CoreML accepts MLMultiArray of dtype int8 natively, and the ANE backend’s int8 matmul kernel (~22 TFLOPs on M4, scaling on M5) is wired through. The bandwidth saving is the main win — 4 bytes → 1 byte per weight at the boundary, ~1.8× decode throughput on M4 per Draw Things engineering blog (2026-04-16).

anemll-bench has NOT adopted this primitive yet (per its docs). We can port it directly into our own M8 stack and bypass the whole anemll question.

What needs to change in M8

Files in scope: native-mac/Sources/TinyGPTModel/Qwen3ANEChunked.swift (299 lines), native-mac/Sources/TinyGPTModel/ANEInference.swift (409 lines). Both already target .cpuAndNeuralEngine. The change is at the model conversion AND prediction-time array layer.

1. Block .mlpackage conversion

Today: each m8-block-i.mlpackage is converted via the Python tinygpt to-coreml exporter with fp16 weights. The fp16 weights get loaded into ANE memory at runtime.

Change: convert with int8 quantized linear weights using coremltools.optimize.coreml.linear_quantize_weights (granularity=per_block, mode=linear_symmetric, bits=8). Output .mlpackage is ~2× smaller on disk; ANE loads int8 + per-block scales directly.

The conversion is offline — re-run once per block. Re-bake the 28-block set.

2. MLMultiArray boundary types

Today’s prediction path (per token):

// Build [B=1, S=1, hidden] fp16 input
let input = try MLMultiArray(shape: [1, 1, hiddenSize], dataType: .float16)
// fill from Swift Float buffer

Activation tensors stay fp16 (intermediate state must be float). The change is at the WEIGHT boundary which is now baked into the .mlpackage at convert time — no Swift-side change needed for static weights.

The optional Swift-side change is for INPUT tensors that the model expects as int8 (e.g., if we add an int8-quantized token embedding lookup). For Qwen3-0.6B the embedding is fp32 (tied with lm_head), so this stays fp16/fp32 at the boundary.

3. Compute precision (already correct)

M8 already uses FLOAT32 compute + FLOAT16 state per task #269. That stays — int8 weights get dequantized at-kernel-time inside ANE; activations remain float.

4. State buffers (no change)

MLState for k_cache / v_cache stays fp16. State buffers are NOT the bandwidth bottleneck on Qwen3-0.6B (small head dim, short context for v1).

5. Validation gate

After re-baking blocks with int8 weights, run on a 50-prompt Pace eval set. Output divergence MUST be < 1pp on fm-fixtures-v2 pass rate vs the existing fp16-weight M8 chain. If divergence exceeds 1pp, revert and investigate per-block scale calibration.

Spike plan (2-3 days)

Day 1 — Tooling:
  - Verify coremltools 8.x quantize_weights API on a single Qwen3 block
  - Test that the resulting .mlpackage loads on ANE (.cpuAndNeuralEngine)
  - Run end-to-end with 1 block; measure activation correctness vs fp16 block

Day 2 — Production:
  - Re-bake all 28 blocks via the int8 path
  - Update Qwen3ANEChunked.load() if any dtype detection logic needs adjustment
  - Run formula score (scripts/score_formula.py) end-to-end on the new chain

Day 3 — Validation + ship:
  - Compare formula score vs fp16 M8 baseline
  - Smoke test on a sample of 100 prompts from the v9 traces
  - If accuracy delta ≤ 1pp AND speed ≥ 1.5×: merge, mark M8 as int8-v1
  - If not: revert, document failure mode, file bug against coremltools

Acceptance criteria

Risk + mitigation

Why this beats anemll migration

Per anemll-vs-M8 memo (today): anemll has no LoRA path, has open Qwen3 bugs on macOS 26, and doesn’t yet exploit int8 handoff itself. We can port the SAME primitive directly into M8 in 2-3 days. Keeps ownership of the LoRA stack. No migration risk.

Done when