lagna360/yo-router-data

yo-router-data

Synthetic training data for `yo`, a macOS tool router fine-tuned from FunctionGemma-270M. Each row maps one plain-English utterance to exactly one function call over a fixed menu of 10 read-only macOS tools.

This is the dataset that produced `lagna360/yo-router-270m` (56.2% → 95.2% tool accuracy). Generator, eval harness and full results: github.com/lagna360/yo.

It is fully reproducible. Everything here is regenerable byte-for-byte from data/generate.py at seed 17 — the dataset is published for convenience, not because it is irrecoverable.

Configurations

Nine variants exist, all from one generator. The main two:

Plus scaling and ablation variants used to produce the study's tables:

Each config directory carries a config.json recording n, seed, per-tool counts, with_args, and measured prompt length.

Format

MLX-LM completion format — two string fields, and the loss is taken on completion only (mask_prompt: true).

{
  "prompt": "<bos><start_of_turn>developer\nYou are a model that can do function calling with the following functions<end_of_turn>\n<start_of_turn>user\nlook for meeting notes please<end_of_turn>\n<start_of_turn>model\n",
  "completion": "<start_function_call>call:find_files{name:<escape>meeting notes<escape>}<end_function_call>"
}

In the v1 (in-context) configs the prompt additionally carries a <start_function_declaration>…<end_function_declaration> block for all 10 tools, which is what takes it from 32 to 654 tokens.

Call format: call:<tool>{key:<escape>value<escape>,…} with keys sorted, or call:<tool>{} when there are no arguments.

A sidecar meta.jsonl carries the structured ground truth for each row, for analysis:

{"tool": "find_files", "utterance": "which folder has screenshot please", "args": {"name": "screenshot"}}

Files per config: train.jsonl, valid.jsonl (8% split), meta.jsonl, config.json.

The 10 tools

disk_usage(path) · storage_summary · largest_files(path) · find_files(name, path) · top_processes(sortby) · `networklisteners(port) · networkinfo` · `batterystatus · system_info · datetime`

How it was generated

Fully deterministic — random.Random(17), no model in the loop, no LLM-generated text.

1. 1.200 hand-written stems, 20 per tool, each a phrasing of that tool's intent, with {path} / {name} / {port} / {sort} placeholders where arguments belong.
2. 2.Slot pools substituted into the placeholders: 19 paths, 20 filenames, 15 ports, 5 sort keys. Natural-language slot values are canonicalised to real argument values ("the downloads folder" → ~/Downloads, "ram" → mem).
3. 3.Surface noise: a prefix from {"", "yo ", "hey ", "can you ", "pls ", "tell me ", …} — but suppressed 75% of the time when the utterance already opens with a question word, so "can you what time is it" never appears — and a suffix from {"", "?", " please", " thanks", …}. Then, per row: 12% get a realistic single-keystroke slip (drop / transpose / double), 6% are shouted in ALL CAPS, and 12% get a leading capital.
4. 4.27 contrastive stems, oversampled 3x. Near-identical phrasings that differ only in the discriminating word, with each confusable tool represented — e.g. "space left" → storage_summary, "space used by folder" → disk_usage, "space used by file" → largest_files. These target the boundaries the confusion matrix showed as weak (storage_summary vs disk_usage, disk_usage vs largest_files, network_listeners vs top_processes).
5. 5.Argument boost (--arg-boost 3) weights slot-bearing stems 3x, because argument extraction converges far more slowly than tool selection.
6. 6.Render into the FunctionGemma chat template, with declarations (v1) or without (v1-baked).
7. 7.Split 92% train / 8% validation.

For v1-baked this yields 3,000 rows, 1,840 of them (61%) carrying at least one argument, distributed across tools as: find_files 637, disk_usage 490, largest_files 440, network_listeners 434, top_processes 254, storage_summary 196, network_info 158, battery_status 143, system_info 138, datetime 110. The distribution is deliberately uneven — tools with arguments get more rows because they have more to learn.

Regenerate any config with:

python data/generate.py --n 3000 --out data/out/v1-baked --baked

Leakage: asserted, not assumed

The 249-case evaluation set (eval/testset.jsonl in the repo) was hand-written before any training, by a person, and never by this generator. To keep those two worlds apart the generator hard-fails at generation time if any training utterance exactly matches a test utterance:

leaked = [r for r in rows if r["utterance"].strip().lower() in test_u]
if leaked:
    raise SystemExit(f"LEAK: {len(leaked)} training utterances match the test set: ...")

This is not decorative — it caught a real collision. The stem "why is {path} so big" with path="my home folder" generated a verbatim copy of a test case. The stem was changed. See LAB_NOTES.md in the repo.

Caveat, stated plainly: the guard checks exact normalised string equality, not semantic similarity. Near-duplicates ("what time is it" / "whats the time") can and do exist across the two sets. That is intentional — the test set is meant to measure in-domain generalisation, not zero-shot transfer — but it means the headline accuracy should be read as "accuracy on paraphrases of the trained intents", not "accuracy on unseen intents".

Known weaknesses

• —Slot-value pools are small and fixed (19 paths, 20 names, 15 ports). Scaling the row count therefore recycles the same argument values. This is measurable: going from 3k to 10k rows raised tool accuracy 95.2% → 95.6% while dropping argument accuracy 82.3% → 77.4%. If you scale this dataset, scale the slot pools with it.
• —Synthetic phrasing. Hand-written stems plus programmatic noise. Real user language has a much longer tail.
• —No negative class. Every row is a tool call. There is no chitchat / no-tool example, which is exactly why the resulting model forces a tool onto "hi".
• —English, macOS, single-call only. No chaining, no multi-tool rows, no other language.
• —Not human-reviewed row by row. The stems were written by hand; the 3,000 expansions were not individually inspected.

Licence

MIT. Copyright (c) 2026 Pankaj Upreti.

The rows are programmatic expansions of hand-written English stems — no model generated them, no scraped corpus is involved, and no personal data is present. The paths and filenames in the slot pools are generic placeholders (~/Downloads, invoice, screenshot), not real user data.

Provenance of the `prompt` field. The utterances, the tool labels and the argument values are original work and are MIT. The prompt field, however, is those utterances rendered through FunctionGemma's chat template — so it carries that model's control tokens and declaration syntax (<start_function_declaration>, <escape>, and so on). No Gemma model generated any text here; data/generate.py loads the tokenizer only to apply its template. We take the view that a prompt format is an interchange schema rather than model output, and licence the data MIT accordingly — but the provenance is stated plainly so you can form your own view. The tool and args fields in meta.jsonl are format-independent if you would rather re-render the prompts for another model.

One caveat about downstream use. This data is MIT, but the model trained on it in this project is a FunctionGemma derivative and is governed by the Gemma Terms of Use — the dataset licence does not and cannot loosen that. Training an unrelated, non-Gemma model on this data carries no such obligation.

Citation

@misc{upreti2026yodata,
  author = {Upreti, Pankaj},
  title  = {yo-router-data: synthetic tool-routing data for macOS system queries},
  year   = {2026},
  url    = {https://github.com/lagna360/yo}
}