CoolFace
Datasetpublic

Lurume/llm-routing-response-bank

LLM Routing Response Bank Five language models × 13,315 tasks across four benchmark families, with per-response text, binary quality scores, token usage, and official billing. Collected for a routing study with a paired calibration/evaluation design: 256 calibration tasks, 13,059 evaluation tasks. Contents file rows note tasks_cal.jsonl / tasks_eval.jsonl 256 / 13,059 prompts + reference answers; gpqa_diamond rows are hash-only (see below)… See the full description on the dataset page: https://huggingface.co/datasets/Lurume/llm-routing-response-bank.

sourceHugging Faceotherupdated 8d agoView on Hugging Face
0likes62downloads
Dataset Card

LLM Routing Response Bank

Five language models × 13,315 tasks across four benchmark families, with per-response text, binary quality scores, token usage, and official billing. Collected for a routing study with a paired calibration/evaluation design: 256 calibration tasks, 13,059 evaluation tasks.

Contents

filerowsnote
taskscal.jsonl / taskseval.jsonl256 / 13,059prompts + reference answers; gpqa_diamond rows are hash-only (see below)
responsescal.jsonl / responseseval.jsonl1,280 / 65,289one row per (task, model); gpqa_diamond rows have text withheld; 6 documented missing GLM cells
scorescal.jsonl / scoreseval.jsonl1,280 / 65,295binary quality incl. 6 refusal cells scored 0

Models: deepseek-v4-pro, deepseek-v4-flash, glm-5.3, glm-5-3-flash, kimi-k2.7-code.

Families (evaluation counts): aime (22), gpqa_diamond (166), livecodebench (967), mmlu_pro (11,904).

GPQA Diamond handling

gpqa_diamond rows in the public layer carry only task_id, prompt_sha256, prompt_chars, scores, usage, and cost. To reconstruct the GPQA subset, obtain GPQA from the gated official source (Idavidrein/gpqa, accepting its terms), then join on the sha256 of each question prompt. The full GPQA slice is withheld because the upstream access agreement forbids publishing its examples in plain text.

Provenance and licenses

  • —mmlu_pro: TIGER-Lab/MMLU-Pro (MIT; code Apache-2.0).
  • —livecodebench: LiveCodeBench (problems from LeetCode/AtCoder/Codeforces, published openly by the benchmark; contest release dates preserved in task source metadata).
  • —aime: AIME 2025 problems via GAIR-NLP/AIME-Preview; problems are copyright MAA/AoPS, reproduced for research use.
  • —gpqa_diamond: Idavidrein/gpqa (gated upstream; hash-only here).

Model outputs are released by the collectors under the terms in LICENSE.

Field notes

  • —prompt_sha256: sha256 of the prompt text; the join key across tasks, responses, and scores.
  • —official_usd: provider billing in USD including cache discounts (response charges only; encoder compute and collection excluded).
  • —usage: input (incl. cache_read), output (incl. reasoning_tokens).

Loading

python
from datasets import load_dataset
tasks = load_dataset("REPO_OWNER/llm-routing-response-bank", "tasks")
responses = load_dataset("REPO_OWNER/llm-routing-response-bank", "responses")
scores = load_dataset("REPO_OWNER/llm-routing-response-bank", "scores")

Integrity

MANIFEST.sha256 lists the sha256 of every file in this release.

Citation

To be added.