Recipe — `tinygpt eval-gate` as a CI / pre-commit gate

tinygpt eval-gate runs your project’s declared eval suites against a stored baseline and exits non-zero when any suite regresses past a threshold. It’s the developer-workflow framing of the shipped eval harnesses (E1 BFCL, E2 τ-bench, E3 lm-eval, E5 HumanEval, the unhappy-path suite): a benchmark wrapper becomes a gate you can put in front of a merge.

Everything runs on your runner — the gate never phones home.

The spec

The gate reads an eval-gate.json in the cwd, an eval block in tinygpt.project.json (B31), or a path you pass with --spec:

{
  "baseline": "evals/baseline.jsonl",
  "default_threshold": 2.0,
  "suites": [
    { "name": "bfcl", "task": "bfcl",
      "command": ["tinygpt", "eval-bfcl", "model.tinygpt", "--out", "$TINYGPT_EVAL_OUT"] },
    { "name": "tau",  "task": "tau", "threshold": 3.0,
      "command": ["tinygpt", "eval-tau-bench", "model.tinygpt", "--out", "$TINYGPT_EVAL_OUT"] }
  ]
}

baseline — a JSONL of EvalCompare.Rows (the shared schema every tinygpt eval-* command emits). Generate it once with --update-baseline.
default_threshold — max allowed regression in percentage points (default 2.0). threshold on a suite overrides it.
command — the argv to produce candidate rows; the gate sets TINYGPT_EVAL_OUT to the JSONL the suite should write. Omit command and pass --candidate <jsonl> to gate a run you already have (the no-GPU path used in tests).

Direction is inferred from the metric name: accuracy / pass@1 / f1 are higher-is-better; ppl / loss / latency_ms are lower-is-better and invert automatically.

First run — stamp the baseline

tinygpt eval-gate --update-baseline      # runs the suites, writes baseline.jsonl, exits 0

Re-run --update-baseline whenever you intentionally move the numbers (the “accept the new scores” path).

Gate a change

tinygpt eval-gate                 # exit 0 = all suites within threshold; 1 = a regression
tinygpt eval-gate --passes 3      # run each suite 3× and gate on the mean (noise guard)
tinygpt eval-gate --budget evals/sample-budget.json

It prints a per-suite table and writes gate-result.json (machine-readable). When a suite has repeated rows, the JSON keeps the trial scores plus n, stdev, stderr, and 95% CI under candidateStats; the console renders the candidate cell as mean±ci95.

For JSONL comparison outside the gate, tinygpt eval-compare also renders repeated rows with the same task/model/metric as mean±ci95 and shows k=... in the cell. Harnesses that pre-aggregate repeated runs can emit row-level pass_stats in the same shape as candidateStats.

Suite commands receive the budget path as TINYGPT_EVAL_BUDGET and the outer pass count as TINYGPT_EVAL_PASSES. Swift harness rows emitted through EvalHarnessSupport.appendRow attach the same "protocol" block beside their raw scores.

When --budget is passed, the report also includes a "protocol" block:

{
  "protocol": {
    "passes": 3,
    "budget": {
      "max_steps": 8,
      "sandbox_cpus": 1.0,
      "sandbox_ram_mb": 512,
      "temperature": 0.0,
      "top_p": 1.0,
      "sampling_seed": 17,
      "infra_patches": []
    }
  }
}

Suite commands receive the budget path as TINYGPT_EVAL_BUDGET; BFCL, τ-bench, and Pace unhappy-path harnesses can use that to log the same budget beside their raw rows.

GitHub Action

The gate runs the MLX path, so it needs a self-hosted Apple-silicon runner — GitHub-hosted Linux runners can’t run MLX.

# .github/workflows/eval-gate.yml
name: eval-gate
on: pull_request
jobs:
  gate:
    runs-on: [self-hosted, macOS, ARM64]
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/tinygpt-eval-gate
        with:
          spec: tinygpt.project.json
          passes: "3"
          budget: evals/sample-budget.json

The action builds tinygpt release, runs the gate, annotates the PR with the suite table in the job summary, records the optional B23 budget metadata in gate-result.json, and fails the check on a regression.

Pre-commit hook

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: tinygpt-eval-gate
        name: tinygpt eval-gate
        entry: tinygpt eval-gate
        language: system
        pass_filenames: false
        stages: [pre-push]   # too slow for every commit; gate on push

Use pre-push (not pre-commit) — running real suites per commit is too slow. For a fast local guard, point the hook at --candidate with a cached JSONL and reserve the full run for CI.

Verify

bash evals/eval-gate-smoke.sh    # asserts exit 0 (match) + exit 1 (regression) with fixtures

The smoke also checks repeated-run stats and budget metadata using evals/eval-gate-fixtures/candidate-repeat.jsonl, so this path stays covered without starting a model or server.

Recipe — tinygpt eval-gate as a CI / pre-commit gate