Dataset inventory

This is the working reference for “what dataset should I use for X.” Every entry is in the curated list-datasets registry; this doc adds the practical bits (downloadable today? what’s in the file? known gotchas?) the registry doesn’t cover.

Quick map:

• Browse: tinygpt list-datasets [--specialist kind | --format sft|dpo|plain]
• Pull: tinygpt download-dataset hf://datasets/owner/name --out path.jsonl
• Convert: tinygpt extractor-data (BFCL/τ-bench → router pairs)
• Default cache: ~/.cache/tinygpt/datasets/

Cache snapshot — 2026-06-02

Verified by pulling each Tier A foundational set. Sizes from du -sh:

External (git-cloned, not HF Hub)

Three of the four ready-for-training sets (alpaca-cleaned, hermes-fc-v1, orca_dpo_pairs, MetaMathQA) are immediately usable with tinygpt sft / tinygpt dpo. Parquet-only sets stage on disk but need a python-side decode pass; see “Known gotchas” §2 below.

GitHub corpus path verified unauthenticated on sindresorhus/is (4 issue→PR records in ~30 s; real corpus build needs GITHUB_TOKEN for the 5,000 req/h limit).

Tool-calling (north-star primary)

Verified pull (commit f566023): hermes-function-calling-v1 schema-sniffed as sft (confidence 75%, chat array → conversations), 11,230 records / 8.4M tokens / 6.5M scored tokens. The response format is <tool_call>{"name": ..., "arguments": ...}</tool_call> — not raw JSON. Trainees must learn this XML-wrap to score on BFCL metrics.

Code + debugger

For the debugger specialist, the natural starting corpus is SWE-bench_Verified + python_code_instructions_18k_alpaca (~62 MB total), with issue→PR pairs from tinygpt fetch-github added on top for repo-specific context. SWE-bench Verified is also the canonical eval target.

Math + reasoning

Instruct (general)

Preference (DPO)

General pretrain corpora

Indic / multilingual evals (not training data)

Special pipelines (not HF Datasets)

Known gotchas

1. Gated datasets need HF_TOKEN. The CLI surfaces the accept-license URL — copy it, click through, then export HF_TOKEN=hf_xxx. xlam-function-calling-60k is the most prominent example.

2. Parquet shards aren’t decoded yet. Some datasets only ship as .parquet (e.g., Locutusque/function-calling-chatml). The tinygpt download-dataset CLI surfaces this with a clear error and the cache path. Decode manually via Python pandas / pyarrow until upstream support lands.

3. JSONL vs ChatML wrap is a real footgun. tinygpt sft --template chatml wraps everything in <|im_start|>user\n{instruction}<|im_end|>\n<|im_start|>assistant\n{response}. The hermes records already prefix "system: ..." inline, so all of that ends up in the user turn at training time. Test prompts at inference must match this shape, NOT the proper <|im_start|>system\n...\n<|im_start|>user\n...\n<|im_start|>assistant you might expect. See docs/specialist_v1_findings.md.

4. macOS reaps /tmp. Long-lived training caches should go to ~/.cache/tinygpt/ or a stable project directory. /tmp gets cleaned aggressively (saw this mid-session on 2026-05-31).

5. The 22-entry registry isn’t exhaustive. It’s the curated slice that’s been tested with tinygpt download-dataset’s schema sniffer. Other HF datasets work if you pass the field names manually via --map.

Recommended starting bundles

How to extend this doc

When a new dataset becomes interesting, add a row to the right table

• note any gotchas in the “Known gotchas” section. Keep entries short (one-line schema, one-line status, one-line gotcha). The canonical “what models can train on what” doc is docs/capability_matrix.md.