saidutta69/RaceBench-v1.1

RaceBench v1.1

<div align="center"> <img src="https://photu.kashyalabanavli.site/racer-is-op.png" alt="RACER IS OP" width="100%"> </div>

<br>

A curated SFT dataset blending agentic tool-use traces with high-quality distillation data — purpose-built for making small models (0.5B-3B) capable enough to replace API-based frontier models in edge deployments.

Quality over quantity. Every row passed a quality threshold of >=60/100. v1.1 fixes the v1.0 agent dilution bug and upgrades to premium traces.

Dataset Composition

Blended from two source datasets:

• —**saidutta69/fable-5-premium** — agent traces with tool calls (>=0.8 filtered, premium)
• —**saidutta69/GPT-5.5-...-Distillation-Cleaned** — filtered to quality >= 60, categories: coding + cybersecurity + distilled

v1.0 vs v1.1: v1.0 README claimed 25,460 agent (7.1%) but shipped 6,365 low-scored rows (1.77%, mean 0.87). v1.1 ships 5,820 unique premium rows (mean 89.7/100, min 80) correctly upsampled 4x to 23,280 — 3.65x more agent signal, all high-quality. Distillation unchanged (353,456).

Source 1: Fable-5 Premium Agent Traces

Multi-turn Claude Fable-5 Premium agent traces with tool calls, reasoning chains, and structured function-calling patterns. Filtered to overall >=0.8 (5,820 of 6,365, 91.4%) then upsampled 4x to preserve agentic signal. Mean quality 89.7, up from 0.87 (unfiltered).

Source 2: Distillation (Quality >= 60)

Filtered from the GPT-5.5 / Gemini / Grok / Claude Distillation Dataset, keeping only:

• —Coding — 135K rows with quality >= 60 (2.1% of 6.3M)
• —Cybersecurity — 167K rows with quality >= 60 (42% of 396K)
• —Distilled — 61K rows with quality >= 60 (93% of 65K)

Why not add instruction/science? Prime v2 instruction 52K only 2,008 >=60 (3.8%), science 74 >=60 (2.3%), humanities 0 >=60 — adding would be noise. Distilled already high-quality.

Quality Distribution

<div align="center"> <img src="https://huggingface.co/datasets/saidutta69/RaceBench-v1.1/resolve/main/data/quality_distribution.png" alt="Quality Distribution" width="100%"> </div>

Distillation all ==60, premium 80-100 (mean 89.7). Overall mean 60.8 (v1 58.95). No rows <80 in agent split.

Why RaceBench for Edge Models?

Small models (0.5B-3B) need high-signal, low-noise data to punch above their weight. RaceBench:

1. 1.Tool-use signal — Agent traces teach function calling and multi-step reasoning patterns that small models struggle to learn from plain instruction data
2. 2.Quality floor — No row below quality 60/100 (distill) or 80/100 (premium agent); poor-quality rows are excluded
3. 3.Domain focus — Coding + security + distilled is the sweet spot for practical edge deployment
4. 4.Deduped — Cross-dataset SHA-256 dedup removed 29K redundant rows (v1), v1.1 shuffled seed 42, zstd compressed

Data Fields

Usage

from datasets import load_dataset

ds = load_dataset("saidutta69/RaceBench-v1.1", split="train")
print(f"Examples: {len(ds)}")  # 376736
print(ds[0]["messages"])

# Axolotl config
datasets:
  - path: saidutta69/RaceBench-v1.1
    type: chat_template
    split: train

# Filter by category
coding = ds.filter(lambda x: x["category"] == "coding")
agent = ds.filter(lambda x: x["category"] == "agent")
print(f"Agent rows: {len(agent)}")  # 23280

RaceBench-MiniCPM5 — the RaceBench reference model

RaceBench v1 has its first public reference checkpoint: **saidutta69/RaceBench-MiniCPM5** — a full-parameter fine-tune of openbmb/MiniCPM5-1B trained on v1, released with fp16 weights and all GGUF quantizations (Q2_K to F16).

Head-to-head vs base, evaluated with lm-evaluation-harness v0.4.12 under identical settings (fixed seed, 2x Tesla T4). z = delta / pooled standard error; |z| > 2 is significant at ~95% confidence:

What it shows: RaceBench transfers real multi-step reasoning to a 1B model (BBH +2.7, z=3.5 - the only significant gain). The price is a real math regression (GSM8K -9.9, z=-5.3) and a small knowledge loss (MMLU -1.9). All other deltas are within noise. v1.1 keeps the same distillation but with 3.65x more premium agent signal — expect BBH gain to hold, GSM8K regression not to worsen (premium is more reasoning-dense). Benchmark for v1.1 pending (same harness, same seed).

Training v1.1: openbmb/MiniCPM5-1B, full-parameter, fp16, sequence_len 4096, 1 epoch, lr 2e-05 cosine, same as v1. See /tmp/RaceBench-v1.1-finetune-README.md for GCP launch.

Citation

@misc{racebench,
  author = {Sai Dutta},
  title = {RaceBench -- Edge-Focused SFT Blend for Small Models},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/saidutta69/RaceBench}}
}

@misc{racebench-v1.1,
  author = {Sai Dutta},
  title = {RaceBench v1.1 -- Quality-Expanded (Premium Agent)},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/saidutta69/RaceBench-v1.1}}
}