vevag/padel-rules-sft

padel-rules-sft

1,496 supervised fine-tuning examples teaching a small model padel rule fidelity: answers accurate to the FIP regulations that import nothing from tennis or squash.

Used to train https://huggingface.co/vevag/padel-qwen3-1.7b-lora — 19.4% → 45.2% spec adherence on a held-out set of 31 scenarios.

Format

One JSON object per line, chat format:

{"messages": [
   {"role": "system",    "content": "You are a helpful assistant for padel players. Answer questions about padel accurately and concisely."},
   {"role": "user",      "content": "Am I allowed to stand right on the service line when I serve, as long as my toes aren't over it?"},
   {"role": "assistant", "content": "No — touching the line is already a fault. Both feet have to be fully behind the service line, and neither may touch or cross it at the moment you strike the ball..."}
 ],
 "meta": {"lie_type": "serve", "style": "...", "level": "..."}}

The system prompt is identical in every row and identical to the one used at eval time. meta.lie_type is the category (naming is a holdover from an earlier project).

How it was built

1. 1.Question synthesis. A frontier teacher (claude-opus-5) generates realistic player questions per rule area, deduplicated by 4-gram overlap.
2. 2.Answer generation. The same teacher answers with the official FIP rule sheet in context — privileged scaffolding the student model never sees. This is the core of the distillation: the teacher is grounded, the student learns the grounded behaviour.
3. 3.Judge gate. A rulebook-grounded LLM judge scores every candidate against the rule sheet; only answers verdicted correct are kept.
4. 4.Eval-disjointness, enforced at build time. Questions are checked against the 31 held-out eval scenarios and near-duplicates are dropped.

Yield is high (95–98% kept) because the teacher holds the rulebook — the gate is catching slips, not doing the work.

Evolution — what each version fixed

The v1 → v2 change is the important one: the failure was not the model's knowledge but the shape of the training targets. A long answer with a discretionary closing paragraph teaches a small model to fill that slot with something, and it fills it with invention.

Category balance (v4)

The six original categories are balanced within 152–165. sameness and binding are deliberately larger — they were added to fix the overcorrection failure and are over-weighted on purpose.

Eval-disjointness

Verified against the 31 held-out eval scenarios:

• —0 exact question matches.
• —Maximum 4-gram Jaccard similarity: 0.300 — no near-duplicates. The closest pair is the same rule (playing off your own back wall) asked in different words, which is intended: the training set teaches the rule areas the eval tests, without sharing items.

Decision logs

Every kept/rejected decision is committed in the project git repo, not just the survivors:

• —results/padel_datagen_log.jsonl — 1,080 decisions, 1,060 kept
• —results/v3_slice_log.jsonl — 240 decisions, 228 kept
• —results/v4_slice_log.jsonl — 324 decisions, 315 kept
• —results/smoke_datagen_log.jsonl — 12 decisions, 12 kept

Alongside them: the full prompt-ceiling ablation, every raw judge transcript, training logs for each run, a cross-family judge agreement analysis, and a fresh-clone verification report.

Limitations

• —Single-teacher distillation. Whatever claude-opus-5 gets wrong about padel while reading the rule sheet propagates into the data unchallenged; the judge shares a family with the teacher.
• —The rule sheet is a compiled summary of the FIP regulations, not the legal text.
• —Coaching/tactics rows encode consensus playing principles, not rules — they are not falsifiable against the rulebook the way the rule categories are.
• —English only.