Game-as-RL-environment PoC

Makes actionable the parked blueprint in docs/learn/rl-multi-agent-roadmap.md. Trigger condition is now met: the distillation thread closed and a from-scratch MLX GRPO loop exists and is validated (GSM8K + tool-calling — see tool-calling-frontier-parity.md §5).

Why now (and why this, not more single-turn RL)

Environments are the scarcest RL ingredient (Prime Intellect built a 2,500-environment Hub precisely because of that). The fleet’s AI game is a live multi-agent environment — an ownable asset most can’t get.
Single-turn RL saturated on a strong base (GRPO on the 4B was neutral: 87.3→86.7, because a fixed verifiable task has a ceiling). An open-ended game environment won’t saturate — there’s always a better policy. That’s exactly why the game, not another single-turn benchmark, is the right next RL target.
It’s the on-mission direction: learn everything about agentic/multi-agent RL, on a Mac, in an environment we own.

Show one distilled small model, as one NPC’s brain, measurably improve at one in-game behavior via GRPO, trained entirely on the Mac.

RL ingredient	In the PoC
Policy	a distilled small local model (the cost-compression lever), LoRA
Environment	the game world (one scenario)
Reward	a measurable in-game outcome (goal reached / interaction won) — clean ⇒ RLVR

Trajectory recorder (game side): log per turn (observation → action → outcome → reward). This is the roadmap’s “B22 recorder.” Define the observation serialization + the reward signal for the one chosen behavior.
Reward = a verifiable in-game outcome (binary or graded), so it’s RLVR — the tractable kind. Avoid LLM-judge rewards for the PoC.
GRPO trainer: reuse the validated loop (group-normalized advantage, KL-to-ref, grad-accumulation — the same structure as our tool-calling GRPO). Generalize it to take an (env.reset, env.step, reward) interface instead of the BFCL prompt set. Skeleton: scripts/archive/game_rl_poc.py (this PR).
Loop: sample K rollouts of the NPC acting in the scenario → reward each → group-normalize → policy-gradient on the LoRA → repeat. Watch the behavior’s success rate trend up.

One scenario instrumented with a clean reward; trajectories logged.
GRPO run on a distilled small model as the NPC brain; the chosen behavior’s success rate rises vs the pre-RL baseline over training.
Stable (KL-bounded, no blowup) — the tool-calling GRPO already proved the loop.

Game integration is the dependency — needs the recorder + a programmatic step/reset/reward interface from the game. Scope the PoC to one scenario to bound it.
Reward hacking / degenerate policies — keep the reward verifiable and the scenario simple; KL-to-ref guards against drift.
Sample efficiency: GRPO is rollout-hungry; one Mac limits scale. PoC targets a trend, not a finished agent.

The multi-turn eval PRD measures whether small models can hold agentic conversations; this PRD is how to improve them via RL in an open-ended env. Eval first (measure the cliff), then this (train the climb).