North Star — refined (2026-05-31)

This doc supersedes the rough Tier 5 sketch with a sharper product vision. The project is now:

An on-device specialist SLM (1-3B) that reads the Mac screen, calls local tools, and gracefully escalates to a cloud model when it doesn’t know. Local model is the always-on layer; cloud is the safety net. Multiple specialists are loaded via a tiny router. Works across languages so it serves India + global users.

That’s a product. Every piece below is grounded in user-stated vision.

Vocabulary borrowed from TML’s “Interaction Models” (May 2026): foreground interaction model (200ms micro-turns for direct UX) + background async model (reasoning, tool calls, long-horizon work). TML’s implementation is cloud-only; tinygpt’s on-device version on Apple Silicon is a real differentiator. See wave_4_landscape.md for the full landscape analysis.

Architecture

                    ┌──────────────────────────┐
                    │  User input (text, screen) │
                    └──────────────┬─────────────┘
                                   │
                    ┌──────────────▼──────────────┐
                    │      Tiny ROUTER (~100M)     │
                    │  classifies task; chooses    │
                    │  specialist or cloud         │
                    └──────────────┬──────────────┘
                                   │
        ┌──────────────────────────┼──────────────────────────┐
        │                          │                          │
   ┌────▼─────┐              ┌─────▼─────┐              ┌─────▼─────┐
   │ debugger │              │  SQL/code │              │ defer to  │
   │ 1-3B     │              │   1-3B    │              │  cloud    │
   │ specialist│              │ specialist│              │ (Claude   │
   └────┬─────┘              └─────┬─────┘              │  / GPT)   │
        │                          │                    └────┬──────┘
        │  reads screen via macOS  │                         │
        │  ScreenCaptureKit + AX   │                         │
        │  tree                    │                         │
        └────────────┬─────────────┘                         │
                     │                                       │
        ┌────────────▼────────────┐                          │
        │  Tools: file I/O, exec, │                          │
        │  test runner, search    │                          │
        │  (subprocess sandbox)   │                          │
        └─────────────────────────┘                          │
                                                             │
        ┌────────────────────────────────────────────────────┘
        │
        ▼ when complex / out-of-distribution / requires world knowledge
   ┌──────────────────┐
   │ Cloud model      │
   │ Claude / GPT API │
   └──────────────────┘

Why this architecture is right

vs pure on-device (Apple Intelligence, Phi-4-mini chat):

• Fails on hard tasks (limited capacity at 1-3B)
• We escalate when uncertain → users get correct answers always

vs pure cloud (Claude, GPT, Cursor):

• Latency (every call is 200-500ms round-trip)
• Cost ($0.50-$2/M tokens × frequent agent loops = expensive)
• Privacy (screen + files sent to cloud every turn)
• We handle 80%+ locally → faster, cheaper, more private

vs single big general model:

• A 70B can do everything but uses ~40 GB at int4 + 100W
• N small specialists at 1-3B each, only one loaded at a time
• Mixture-of-Specialists with router → smaller resident footprint
• Each specialist tuned for its task → outperforms generalist on that task

vs Apple Foundation Models (same architecture, walled):

• Apple’s ~3B on-device + Private Cloud Compute is literally the same shape we’re building. They win on integration, lose on openness.
• Apple is closed: closed model, closed router, LoRA-adapters-only customization, adapters re-trained per OS update, no public API to swap in your own model or route to non-PCC cloud.
• tinygpt’s positioning: open, hackable, multi-specialist, route- anywhere. Swap base models (Llama/Qwen/Sarvam), do full SFT/DPO not just adapters, route to any cloud (Claude/GPT/local), target devs/researchers Apple won’t serve.
• Steal Apple’s ideas: KV-cache sharing (5:3 block-depth), 2-bit QAT, Tool protocol shape. Don’t try to plug into App Intents — no public hook exists. Stay parallel.

Concrete capabilities the agent needs

Critical research dives needed

Thinking Machines Lab (Mira Murati, et al.)

User specifically called out — TML is positioning around “interaction models.” Known public artifacts as of cutoff:

• Tinker (2025): fine-tuning API for open-weight models
• Focus on human-in-the-loop systems + multimodal interaction
• Founded by Mira Murati after leaving OpenAI

Action: spawn Explore agent to summarize TML’s published technical direction + identify what we should learn from / not duplicate.

Apple Foundation Models

WWDC 2024+ announcements about on-device 3B model + cloud extension via “Private Cloud Compute.” This is the SAME architecture we’re proposing. Need to:

• Understand how Apple’s local model handles tool calling
• Identify whether ours can hook into their ecosystem (App Intents?)
• Or whether we’re a parallel/competitive offering

Cursor / Continue.dev / Cline / Codeium

These are existing code agents (cloud-bound). Our differentiation: on-device specialist that handles 80% locally + escalates. What does the architecture look like — separate eval needed.

Demonstration specialists to ship

In order of probable demo value:

1. Code debugger — issue → diagnosis → fix proposal

   • Training data: GitHub issue→PR pairs (we have tinygpt fetch-github)
   • Tools: read_file, run_test, search_code, edit_file
   • Eval: SWE-bench scaled-down + custom corpus

2. Shell command writer — natural language → bash command

   • Training data: history dumps + man pages + StackOverflow Q&A
   • Tools: minimal (it’s mostly text-out)
   • Eval: custom corpus + manual review

3. SQL writer — schema + question → query

   • Training data: spider, BIRD, gretelai/synthetic_text_to_sql
   • Tools: schema inspector + query runner
   • Eval: spider/BIRD exact-match

4. Screen reader / UI explainer — “what is this app showing me?”

   • Training data: SYNTHETIC — screenshot + caption pairs generated by Claude/GPT on a corpus of macOS app screenshots
   • Tools: capture_screen, click, type
   • Eval: custom screenshot corpus

5. Indian-context assistant (multilingual)

   • Hindi + English specialist
   • Base: Sarvam-Edge (forthcoming, Apache-2.0, MoE, 22 Indian languages, custom Indic-aware tokenizer) OR AI4Bharat Airavata (Hindi instruction-tuned, OpenHathi base, IndicTrans2-translated instruction sets, available now). Earlier draft referenced yenupam/desi-max — that’s a text-to-image LoRA, not a language model. Corrected.
   • Training data: Sarvam’s pretrain corpus, Airavata’s IndicInstruct, bootstrap more via IndicTrans2 of English instruction sets
   • Tools: same as general assistant
   • Eval: MILU (AI4Bharat, 11 Indic languages, India-centric) + IndicGenBench (29 Indic languages). Both paper-backed and reproducible — wire into the eval harness BEFORE claiming Hindi support. Run base Qwen3 / smollm2 through MILU first to get a baseline; that number tells you whether tokenizer-swap to Sarvam’s vocab is worth the engineering cost.
   • See wave_4_landscape.md §4 for full landscape (Sarvam vs Krutrim vs AI4Bharat, tokenizer choices, eval coverage).

Realtime / interaction model (NEW — user-stated)

Beyond turn-based agent, the user wants a realtime interaction model — continuous, low-latency, streaming, interruptible. Think GPT-4o realtime API or Anthropic’s streaming voice-to-voice, but on-device.

What “realtime” means concretely:

What needs to be built

The realtime work is complementary to the agent factory — same infrastructure, different surface (streaming + interrupt instead of turn-based + transcript). Pairs naturally with screen reading (realtime “watch what the user does, react when asked”) for the “interaction model” framing.

Updated wave plan

Wave 1 (DONE):          agent runtime + JSON mode + HF data
Wave 2 (DONE/partial):  GitHub data + eval harness (eval deferred —
                        subprocess refactor needed)
Wave 2.5 (RESOLVED 2026-05-31, see research/wave_2_5_kernel_audit.md):
                        - LOW risk: CF R2 + Pausable training + GPU lock (DONE)
                        - Flash Attention Metal kernel:    DROP — MLXFast already fused
                        - Int4 packed matmul Metal kernel: DROP — MLX quantized_matmul is hand-tuned
                        - ANE+GPU heterogeneous routing:   DEFER — research-grade, gated on Apple Stateful Models API
                        - Int8 W8A8 matmul (adopt cider):  BUILD on M5+ — 1.2-1.9× prefill (user is M5 Pro, viable)
                        - WebGPU subgroup matmul:          BUILD when browser focus returns (background agent in flight)
Wave 2.6 (NEW):         screen reading + cloud escalation + router
                        infrastructure
                        - ScreenCaptureKit + AX integration
                        - Cloud API client (Claude/OpenAI HTTP)
                        - Router model architecture + training pipeline
                        - Vision encoder (ViT) → text decoder bridge
Wave 3 (DEFERRED until 2.5 + 2.6 mostly done):
                        - First specialist (debugger) — train + ship
                        - Second specialist (shell or SQL)
                        - Demo on the web playground
Wave 4 (research):      - TML / Apple / Cursor research dives
                        - Mixture-of-Specialists router actually wired
                        - Multilingual specialist (Hindi)
                        - Public model card on HF Hub
                        - Writeup / blog / HN post

Roadmap items by user-stated motivation

Nothing in this vision conflicts with what’s been built. Everything lands cleanly on the existing infrastructure. The remaining work is roughly:

• 2-3 weeks of Wave 2.6 (screen reading + cloud escalation + router)
• 4-6 weeks of Wave 3 (first 1-2 specialists)
• Then Wave 4 polish / publication

Honest reality checks

1. Wave 2.5 Metal kernels turned out to be mostly unneeded. Research audit (docs/research/wave_2_5_kernel_audit.md, 2026-05-31) collapsed 5 items to 1 buildable: WebGPU subgroup matmul (deferred to background agent), with cider W8A8 viable on M5+ (1-2 day adopt, not 2-3 week build). FA Metal + int4 matmul are already solved by MLX-Swift’s MLXFast.scaledDotProductAttention + quantized_matmul. ANE routing is research-grade and gated on Apple’s Stateful Models API. The original “Metal kernels need more focused prompting” concern was wrong — the work itself was wrong.

2. The screen reader is the biggest scope risk. ScreenCaptureKit is well-documented; ViT integration is well-trodden; but training a 1-3B model to actually USE screenshots for tool calling is research-grade work. May land as a toy first.

3. The cloud escalation training signal is non-trivial. You need a corpus where the same prompts have BOTH local-resolvable AND cloud-only answers, and the model learns to predict “defer”. Could bootstrap via Claude/GPT-as-judge.

4. The router is a small model itself. ~100M classifier-style. We have the infrastructure but no training pipeline tuned for the “which specialist handles this?” classification task. The tool-call extractor is a smaller sibling — encoder-only, classification over a fixed tool catalog. It’s easier (no generation, no KV cache, fits comfortably in CPU) and a good warm-up project before the bigger router. Possible training signal: scrape tool-using agent traces (from Anthropic / OpenAI tool-use datasets, BFCL, τ-bench), pair (query, tool_name) as supervised examples, train a tiny encoder + softmax head. Could plausibly be a sub-30M model that beats LM-emitted tool calls on latency by 50-100x.

5. Compute budget reality. Training a 1.5B specialist properly is 24-48 GPU hours. On a Mac with the existing infra, that’s feasible but uses cloud (CF R2 just shipped — unblocks this).

What I’d ship next if doing this solo

(Updated 2026-05-31 after Wave 2.5 + 4 research audits.)

1. Mac fast-path: adopt cider W8A8 on M5 Pro (1-2 days, 1.2-1.9× prefill). Pair with decode jitter benchmark to baseline realtime targets (1 day).
2. Continue.dev / Ollama provider adapter for tinygpt (~2-3 days). Drops tinygpt into Continue/Cline/Aider configs — lowest-effort path to real users. Pairs with the SSE streaming already shipped.
3. Fix Indic plan + wire MILU + IndicGenBench evals (1-2 days) before any Hindi specialist training.
4. Tool-call extractor (mini-router) (1 week) — tiny BERT-class encoder + softmax over tool catalog. Adopts Apple Tool protocol shape + Cline structured-output enforcement.
5. Wave 2.6 ScreenCaptureKit + AX (~3-5 days each).
6. Train first specialist (debugger) end-to-end — Wave 3.
7. Wire into a Mac app demo + public release + writeup.

Backgrounded:

• WebGPU subgroup matmul (browser-side, 3-5 days agent work)

Everything above is buildable. The original “Metal kernels” wall was mostly imaginary — the real next step is product surface (Continue adapter + tool-call extractor) plus the one Mac-speed lever that actually exists on this hardware (cider W8A8).