← TinyGPT · docs · devlog · roadmap · speedup
source: docs/prds/factory-vision-m4-architecture-decision.md · view on GitHub ↗

DECISION (2026-06-08)

Option A — Port Qwen3-VL fully (mRoPE + image-token replacement + deepstack). Base: UI-Venus-1.5-2B.

Why this over Option B (LLaVA fallback):

  1. Pace already daily-drives UI-Venus-1.5 successfully in LM Studio. We KNOW the ceiling — any LLaVA fallback that uses a non-UI-pretrained base would be a measurable regression from the model Pace currently uses. Falling back trades 2 weeks of arch work for visibly worse output. That trade fails.

  2. Per [[feedback_leverage_first]]: leverage existing screen-reading pretraining (UI-Venus’s whole reason to exist) instead of re-teaching it from raw VLM pretraining.

  3. The “2 weeks” effort for mRoPE + image-token-replacement + deepstack is well-bounded engineering — three named features with clear references (HF’s Qwen3-VL implementation is the parity reference). Not open-ended risk.

  4. Owner’s stated philosophy: “When you are going forward with the most ambitious goals in the world, then you have to take as much leverage as possible.” Option A is the leverage-maximal path.

Why not Option C (B-then-A): Option B’s wasted work (LLaVA-style v1 LoRA trained on a regressing base) is real. Once Pace tries v1 and reports quality drop, we’d throw the LoRA + SFT data away and do A’s work anyway. C is only justified when v1’s quality is unknown; here it’s knowable (UI-Venus baseline already daily-driven). Skip the speculative ship.

Implementation notes for the elf:

Decision doc — VLM M4 architecture

The situation

The VLM elf shipped M1+M2+M3 cleanly. At M3 (and after the leverage-first gate added to the PRD mid-flight), the elf identified that UI-Venus-1.5-2B is the best open-weights student base — not the earlier Qwen3-VL-2B or UI-TARS-1.5B (which doesn’t exist on HF).

UI-Venus-1.5-2B is Qwen3VLForConditionalGeneration, which has three architectural features the original LLaVA-style PRD didn’t account for:

  1. mRoPE (multimodal RoPE) — RoPE positions split into 3 sections [24, 20, 20] for (time, height, width) axes. Image tokens get 2D spatial positions; text tokens get scalar positions. Standard RoPE doesn’t handle this.

  2. Image-token replacement at embed stage — Qwen3-VL doesn’t prepend image tokens before text (LLaVA convention). Instead, the text tokenizer emits placeholder <image> tokens, and the embedder substitutes vision features in-place. This means the VLM forward pass needs to know WHERE in the token sequence each image’s features go.

  3. deepstack_visual_indexes=[5, 11, 17] — visual features are re-injected at multiple LLM depths, not just front-loaded into the first layer. The elf would need to expose hooks at layers 5, 11, 17 and route vision features into them.

All three are addressable but represent real engineering scope.

Three options

Option A — Port Qwen3-VL fully

Implement mRoPE, image-token replacement, and deepstack injection in TinyGPTModelVLM.

Effort: ~2 weeks of focused work (M4 expands significantly).

Pros:

Cons:

Option B — Pick a LLaVA-style base instead

Swap to a base that uses LLaVA’s prepend-image-tokens convention with standard RoPE. Candidates:

Effort: ~3 days (PRD’s current M4-M5 plan works as written).

Pros:

Cons:

Option C — Hybrid: ship Option B first, port Qwen3-VL later

Ship LLaVA-style VLM with simpler base now (M4-M10 in original timeline). When v1 is in Pace’s hands and we’ve measured the quality ceiling, port Qwen3-VL features incrementally if quality demands it.

Effort: ~3 days now, ~2 weeks later if needed.

Pros:

Cons:

Trade-off summary

AxisA (Qwen3-VL port)B (LLaVA fallback)C (B then A)
Pace ships VLM by~6-8 weeks~3-4 weeks~3-4 weeks (v1) + ~6-8 (v2)
Use UI-Venus pretrainYes (best leverage)NoNo (v1), maybe (v2)
Architectural complexityHighLowLow → high
Risk of quality missLowestMedium-highMedium-high → low
Total eng cost if v1 is enough2 weeks3 days3 days
Total eng cost if v1 isn’t enough2 weeks3 days + thrown work3 days + 2 weeks

My recommendation

Option C, with a clear quality bar before deciding on v2 work. Ship the LLaVA-style v1 in 3-4 weeks with Qwen2-VL-2B (or similar) as base. Define a clear quality bar: “VLM v1 passes ≥85% of held-out Pace screen-reading scenarios.” If v1 beats that, stop. If it doesn’t, do Option A’s port work having confirmed it’s worth the weeks.

Why this is the right Bayesian move: the prior is unclear whether LLaVA-style on Qwen2-VL is enough. Don’t pay 2 weeks of arch work speculatively. Pay 3 days to find out, then pay more only if needed.

The risk to flag explicitly: if Pace already daily-drives UI-Venus-1.5-2B via LM Studio and it works well, then we KNOW the ceiling is at UI-Venus quality. Falling back to Qwen2-VL gives up that ceiling. So if owner reports “UI-Venus quality is great, anything less would be a regression,” go straight to Option A.

What the next elf needs from the owner

A one-line decision: A, B, or C. Plus, if C: the quality bar for “is v1 enough?”

Files affected