← TinyGPT · docs · devlog · roadmap · speedup
source: docs/prds/factory-vision-specialist.md · view on GitHub ↗

PRD — VLM specialist for Pace screen reading

Status note — 2026-06-07

M1-M3 are shipped. The M4 architecture decision is now made in factory-vision-m4-architecture-decision.md: Option A, full Qwen3-VL port with UI-Venus-1.5-2B as base.

M4 scaffold v1 is shipped in Qwen3VLScaffold.swift: compile-safe contracts for Qwen3-VL mRoPE metadata, image-token replacement validation, and deepstack visual injection sites. This is intentionally not a HF weight loader or a functional UI-Venus forward path.

This PRD remains active, not complete. M4 requires the named Qwen3-VL features to be implemented and parity-gated against HF PyTorch:

Those are multi-day architecture tasks and should not be reported as done until the per-feature parity tests exist.

Goal

Distill Pace’s current screen-reading VLM (Qwen3-VL-8B via LM Studio at ~6 GB Q4) into a 1-2B student that:

  1. Takes a screenshot + optional Apple Vision OCR hints
  2. Emits Pace’s structured element list: 
    [N] role|x,y|label|text
  3. Matches teacher quality on a held-out Pace screenshot eval set
  4. Runs at ~1-2 GB RAM (~3× less than Qwen3-VL-8B)
  5. Drops in via tinygpt serve --vlm <path> for Pace’s LocalVLMClient

Why P0

Pace’s “watch” capability depends on this VLM. Without it, Pace can’t see the screen. The 30B planner specialist (Pace’s other model) is already in our distillation pipeline; the VLM is the parallel arc.

Combined wins when both ship:

Architecture

LLaVA-style:

Image (1024×1024 max)
  ↓ ViT encoder (e.g., CLIP-ViT-L-14 or Qwen3-VL's native vision tower)
  ↓ pooled image features [N_patches, hidden_dim]
  ↓ projection MLP [hidden_dim → llm_dim]
  ↓ prepended as image tokens to text input
LLM body (Qwen3-1B or similar text-only base)
  ↓ generates text response

Teacher: Qwen3-VL-8B (Sarthak has it in LM Studio). Student: Qwen3-VL-2B (architecturally compatible vision+LLM) OR custom (CLIP encoder + Qwen3-1B body via cross-modal projection training).

Scope — in (milestones, ELF reports at each)

Milestone 1 — Vision encoder primitive

Milestone 2 — Cross-modal projection

Milestone 3 — VLM forward pass

LEVERAGE-FIRST gate before this milestone (per [[feedback_leverage_first]]): re-evaluate base-model choice. Three open-weights candidates to score on (param count, screen-task fit, Mac-load cost, license):

  1. Qwen3-VL-2B — current PRD pick. Generic VLM, no UI specialization.
  2. CogAgent (Zhipu, open weights) — already trained for screen interaction. Bigger (~18B) but distillable.
  3. UI-TARS-1.5B (ByteDance, open weights, MIT) — explicitly trained for desktop UI agents. Closest fit; smallest size.

Likely winner: UI-TARS-1.5B as base, fine-tune our cross-modal projection + LoRA on the LLM body. Saves weeks of pretrain because the “screens look like this” learning is already done.

Milestone 4 — VLM HF loader

Milestone 5 — VLM training pipeline

Milestone 6 — VLM screen training data (LEVERAGE-FIRST)

Principle: warmstart on public datasets, fine-tune on tiny high-quality Mac-specific set. Do NOT teacher-synthesize when public pretrained data already exists for the same task. See [[feedback_leverage_first]].

Three-stage data plan:

Stage A — warmstart on public UI-grounding data (primary):

Stage B — Mac-specific fine-tune via macOS Accessibility API:

Stage C — teacher fill-in (last resort, AX-blind cases only):

Held-out eval: ScreenSpot + ScreenSpot-Pro (~4K test pairs) — public, objective. Don’t train on these.

Milestone 7 — VLM specialist SFT

Milestone 8 — VLM serve integration

Milestone 9 — Vision eval

Milestone 10 — Pace integration handoff

Scope — out (deferred)

Files involved

New files (parallel-safe with ANE elf — different surface):

Modified (light touch, please coordinate):

Don’t touch:

Resource discipline

Acceptance — full ship

  1. tinygpt serve <hf-vlm-dir> --vlm-mode auto --port 8765 boots and accepts image inputs
  2. End-to-end smoke: send a Pace screenshot via curl → returns a valid [N] role|x,y|label|text element list
  3. Eval ≥90% of teacher’s element-detection rate on held-out screens
  4. RAM: ≤2 GB peak (vs Qwen3-VL-8B’s ~6 GB Q4)
  5. Latency: ≤2× current Qwen3-VL-8B time on the same screen (target: <2s per screen read)
  6. Pace daily-drives against our endpoint without rollback for 1 day

Estimated effort

3-4 weeks for the elf, broken into milestones:

Why this is the right architecture (not Option 1)

Option 1 (Apple Vision + text head) was the fast path. Owner chose Option 3 (full VLM) explicitly. Reasons that matter long-term:

Milestone 1 ship note (2026-06-07)

Status: PASS. Vision encoder primitive lands; load + forward verified against openai/clip-vit-large-patch14.

What landed:

Files changed:

Smoke result:

tinygpt vlm-smoke <openai/clip-vit-large-patch14 snapshot dir>
  → loads 24-layer ViT-L
  → forwards synthetic 224×224 → features [1, 257, 1024], finite
  → forwards real screenshot → features [1, 257, 1024], finite
  → PASS

Parity check against HF PyTorch CLIPVisionModel (fp32):

mean diff:        0.00183     (target: dominated by fp32 noise)
median diff:      0.00113
95th-pctile diff: 0.00522
99th-pctile diff: 0.01234
max diff:         1.16  (single outlier on a value of magnitude ~30)
cosine sim:       0.999995

First parity attempt FAILED (cosine 0.78). Root cause: my forward was applying post_layernorm to the FULL sequence; HF applies it only to the CLS-pooled output. Fixed by changing callAsFunction to return pre_layrnorm + encoder (no post-LN) and adding a separate pooled(_:) method for the CLS path. Verified parity after fix. Outlier diffs concentrate at high-magnitude positions (relative error <5%) — consistent with MLXFast SDPA vs PyTorch SDPA fp32 accumulation across 24 layers. Not a logic bug. The parity script lives at scripts/vlm/clip_parity.py for re-runs.

Milestone 2 ship note (2026-06-07)

Status: PASS. Cross-modal projection MLP primitive lands. Random init forwards cleanly; weights are trained from scratch in M5 (no load path here — that’s the M4 VLM loader’s job when reusing a LLaVA-1.5 / Qwen3-VL checkpoint’s pretrained projector).

What landed:

Files changed:

Smoke result:

CLIP vision → projection (vision_hidden=1024 → llm_hidden=2048)
  input  : [1, 256, 1024]  (patches only, CLS dropped)
  output : [1, 256, 2048]
  mean=-0.003, std=0.26, range [-4.66, 5.15], all finite
  PASS

Known limitations / non-issues:

Known limitations / non-issues:

Milestone 3 ship note (2026-06-07)

Status: PASS. Full VLM forward composes end-to-end. Shape test only (random LLM init, CLIP encoder, front-prepend) per advisor guidance — real numerics validation happens at M4 against actual checkpoint weights.

What landed:

Files changed:

Smoke result:

vlm output shape: [1, 264, 1000]   (256 vision tokens + 8 text → logits)
logits mean=0.01, std=1.00, range [-3.68, 4.37], all finite
PASS

M4 architectural reckoning (per the leverage-first gate the maintainer added between M2 and M3):

Investigated the candidates against actual config.json evidence (NOT recall):

CandidateArch familyNotes
inclusionAI/UI-Venus-1.5-2BQwen3VLForConditionalGenerationUI-Venus is the LM Studio teacher Sarthak is already running. Hidden=2048, 28L. Vision: 24L ViT, hidden=1024, patch=16, mRoPE, spatial_merge=2, deepstack_visual_indexes=[5,11,17] (features re-injected at multiple LLM depths).
ByteDance-Seed/UI-TARS-2B-SFTQwen2VLForConditionalGeneration1.5B params. Hidden=1536, 28L. Vision: Qwen2VL tower, patch=14, mRoPE. Tied embeddings. MIT license.
ByteDance-Seed/UI-TARS-1.5-7BQwen2_5_VLForConditionalGenerationToo big (~5 GB Q4) — defeats the size budget. Skip.
UI-TARS-1.5B (PRD candidate)n/aDoes not exist on HF — likely a typo for UI-TARS-1.5-7B (version 1.5, 7B) or UI-TARS-2B-SFT.
CogAgentTHUDM custom18B base, skip on size.

Verdict for M4: Pick UI-Venus-1.5-2B as the student. Reasons:

  1. Already screen-trained by inclusionAI (LM Studio confirms this is the production VLM Sarthak is running). Distilling from itself = max-leverage; the M6 teacher and M7 student share architecture, simplifying label parity and tokeniser sharing.
  2. Qwen3-VL is the most-recent generation (better long-context, better tied-embedding savings vs Qwen2-VL).
  3. UI-Venus-1.5-2B’s vision tower hidden_size=1024 happens to match CLIP-ViT-L-14’s hidden — our M1 encoder is the right shape for weight-copy experiments down the road if we want to swap towers.

M4 design challenge (surface honestly now): Qwen3-VL is ARCHITECTURALLY DIFFERENT from LLaVA-1.5 in three substantive ways the PRD’s “LLaVA-style” framing glosses over:

  1. mRoPE — the LLM body uses multimodal RoPE with mrope_section = [24, 20, 20] (T, H, W splits across head_dim=128). Our existing CausalSelfAttention uses 1D RoPE. M4 will either need a new attention path or ships with a “force-1D-RoPE” mode that PARTIALLY loads the model — vision-stream position info will be wrong, hurting accuracy. Honest acknowledgement: this is the single hardest engineering item.

  2. Image-token replacement at embed stage — the model expects text containing image_token_id=151655 at the image’s position, then vision tokens REPLACE that token’s embedding. Front-prepend (M3) is structurally wrong for the trained model. The fix is straightforward at the forward call (find image-token positions, scatter vision tokens). Not yet implemented because random LLM doesn’t care; M4+ requires correct token-id splicing.

  3. deepstack_visual_indexes=[5,11,17] — UI-Venus-1.5-2B re-injects visual features at LLM layers 5, 11, 17 (not just at the front). This is part of why the architecture is accurate at fine-detail screen reading. Faithful M4 loading needs this; skipping it degrades quality. Pragmatic option: skip deepstack at M7 SFT (model is being fine-tuned anyway; missing skip-connections can be partially compensated by LoRA), document the gap.

These three items don’t block M4 — they shape its scope. M4 will land an honest “Qwen3-VL safetensors loader that handles the embedding-replacement + mRoPE + deepstack contract correctly”, and the PRD’s original “LLaVA convention” framing was approximate; reality is “Qwen3-VL specifically, with three architectural quirks”.

Known limitations / non-issues (M3):

Cross-elf coordination note (2026-06-07): at the end of this session, a clean swift build -c release is currently broken by a pre-existing async/await error in ANEInference.swift:339 (the ANE elf’s in-flight territory, called out as “Don’t touch” by the PRD). The VLM specialist binary at .build/arm64-apple-macosx/release/tinygpt was built BEFORE that file was edited and still includes the VLM smoke subcommand working correctly — all three M1/M2/M3 smokes still pass against it. When the ANE elf clears their error, an incremental rebuild will produce a new binary with the same VLM code. None of the VLM files touch ANEInference.swift; the dependency is purely module-level (they live in the same TinyGPTModel target).