← TinyGPT · docs · devlog · roadmap · speedup
source: docs/NEXT.md · view on GitHub ↗

TinyGPT Next

This is the active queue. It intentionally ignores most historical PRDs.

For the full documentation path, start at docs/README.md. For what worked or failed, use docs/attempt-ledger.md. For reviewed external products and techniques, use docs/external-products-reviewed.md. For the owner’s learning sequence, use docs/learning-pipeline.md.

Current Thesis

TinyGPT is a Mac-local specialist factory:

target -> data -> post-training -> eval -> package -> report

The next milestone is not another surface area expansion. It is one canonical factory run that starts from a frozen target and ends in a documented ship/reject decision.

Operating Rule

Every active task must answer one of these:

  1. Can we prepare or improve the data?
  2. Can we post-train a candidate?
  3. Can we evaluate it against a frozen baseline?
  4. Can we package it as a specialist artifact?
  5. Can we report score delta, regressions, cost, latency, RAM, and a decision?

If not, move it to docs/parked/ or leave it in docs/prds/ for later.

Post-training tasks must also name a recipe, not only a method. Use docs/techniques/ before training:

“Try DPO”, “try RLVR”, or “try a different LoRA rank” is not specific enough. The recipe must name the failure mode, data, eval gate, slice gate, and stop rule.

Active Sequence

0. Keep Public Artifacts First-Class

Public artifact inventory lives in docs/factory/public-artifacts.md.

Before starting a new run or release push, update the artifact entry with:

Current priority artifact: qwen06-sql-routed-v1 as a public report artifact, not yet a shipped specialist package. Render its canonical report run with:

python3 scripts/render_sql_factory_run.py --out runs/2026-07-02-sql-routed-qwen06-v1

1. Pick the Factory Target

Choose exactly one target before training.

Current POC target: SQL specialist. The low-compute fixture is evals/sql-poc/; the brief is docs/specialists/b1-sql-poc.md.

Frozen target (2026-07-03): qwen06-sql-hygiene-dpo-v1 — the single preference-tuning/output-hygiene candidate from cleanup task 4. Frozen before training:

Outcome (2026-07-04): retry-training. The ref-free SimPO run collapsed the policy (composed execution 0.860 → 0.080, clean-SQL 0.000; the adapter alone generates fence spam). Full schema-valid run artifacts and report: runs/2026-07-03-sql-hygiene-dpo-qwen06/. Clean-SQL scorer now exists at scripts/score_sql_clean_output.py. Next candidate: reference-anchored DPO (or SimPO at ~10× lower lr / ≤50 steps) on the same frozen pairs, evaluated composed. Gotcha for future runs: record the tinygpt binary provenance — the 2026-06-25 release build scores identical preds at 0.000 where the 2026-07-02 debug build scores 0.860, and composes multi-LoRA differently.

TrainLoop-style additions required for the next SQL retry (2026-07-04):

  1. Method-vs-recipe registry: docs/techniques/.
  2. Case-study report shape: docs/factory/case-study-template.md.
  3. Candidate-selection curriculum before another open-generation hygiene retry: scripts/build_sql_candidate_choice.py and scripts/score_sql_candidate_choice.py.
  4. Slice metrics: scripts/score_sql_slices.py.
  5. Trace review: scripts/review_sql_trace.py.
  6. Batch-first rollout plan: scripts/render_batch_posttrain_plan.py.
  7. LoRA diagnostics on every meaningful adapter: scripts/lora_geometry.py.

No next SQL candidate should be reported without slice-metrics.json and trace_review.md.

Good targets:

Exit criteria:

2. Freeze the Eval

Use existing eval plumbing first:

Do not train against a moving target.

Exit criteria:

3. Prepare Data

Use existing data tools before writing new ones:

Exit criteria:

4. Train the First Candidate

Use the cheapest method first:

  1. SFT / LoRA.
  2. DPO or preference tuning only after good/bad pairs exist.
  3. ReST/RLVR-style loops only when the reward is verifiable.
  4. Merge/routing only after measuring breadth damage.

Exit criteria:

5. Evaluate and Decide

Compare candidate to baseline and incumbent.

Required report fields:

Exit criteria:

Near-Term Cleanup Tasks

  1. Run the no-GPU factory smoke set for the TrainLoop-style additions: bash evals/sql-choice-smoke.sh, bash evals/sql-trace-review-smoke.sh, and bash evals/lora-geometry-smoke.sh. 0.1. Run the stricter publish evidence smoke: bash evals/factory-publish-check-smoke.sh. 0.2. Run the docs golden-path smoke: bash evals/docs-world-class-smoke.sh.
  2. Wire real train/eval commands to emit the run schema automatically, using scripts/render_sql_factory_run.py as the report-artifact bridge.
  3. Run scripts/build_sql_spider_execution_gate.py against a local Spider DB bundle and score the current routed candidate on execution accuracy.
  4. Measure routed SQL latency, RAM/peak RSS, and tok/s (harness ready: scripts/measure_sql_routed_perf.py).
  5. Run exactly one preference-tuning/output-hygiene candidate. Done 2026-07-04 — decision retry-training (see frozen-target outcome above).
  6. Report and decide package vs retry. Done 2026-07-04 — schema-valid run + report in runs/2026-07-03-sql-hygiene-dpo-qwen06/; retry lane defined there.

Use docs/prds/PRIORITY.md only when a task needs PRD-level acceptance criteria. Do not work from the full PRD list directly.

Not Active

These are parked unless they directly unblock the current factory run: