Datasetpublic

Samyol/xRouteBench

xRouteBench — LLM Routing Benchmark xRouteBench is a benchmark for training and evaluating LLM routers — systems that pick the best LLM from a candidate pool for each incoming query, trading off performance vs. price cost. Every query in each scenario was executed against all 18 candidate LLMs, recording each model's response, task performance, token usage, and latency. A router learns from the train split which model to pick, and is evaluated on test. Scenarios… See the full description on the dataset page: https://huggingface.co/datasets/Samyol/xRouteBench.

sourceHugging Faceupdated 5d agoView on Hugging Face

1likes69downloads

Dataset Card

xRouteBench — LLM Routing Benchmark

xRouteBench is a benchmark for training and evaluating LLM routers — systems that pick the best LLM from a candidate pool for each incoming query, trading off performance vs. price cost.

Every query in each scenario was executed against all 18 candidate LLMs, recording each model's response, task performance, token usage, and latency. A router learns from the train split which model to pick, and is evaluated on test.

Scenarios (configs)

Config	Domain	Train rows / queries	Test rows / queries	Metric
`llmrouter_generic`	13 classic NLP benchmarks (MMLU, GSM8K, MATH, MBPP, ARC, …)	80,802 / 4,487	67,122 / 3,729	emmc, GSM8K, MATH, codeeval, f1
`memory_locomo`	Long-conversation memory QA (RAG top-k=5)	15,930 / 885	5,652 / 314	f1
`memory_longmemeval`	Long-term memory eval (RAG top-k=5)	4,986 / 277	1,818 / 101	f1
`timeseries`	Time-series understanding (7 sub-tasks)	17,568 / 976	2,286 / 127	mc
`video`	Egocentric video QA (Charades-Ego)	3,618 / 201	486 / 27	em
`multimodal_geometry3k`	Geometry math (multimodal)	8,640 / 480	1,098 / 61	em
`multimodal_mathvista`	Visual math reasoning	14,400 / 800	1,800 / 100	em, em_mc
`personalized`	Personalized preference (LLM-judge; chat-format queries)	2,464 / 2,235	308 / 303	llm_judge
`llm_candidates`	The 18-model candidate pool with pricing	18 models	—	—

Schema (routing data)

Each row = one (query, candidate model) pair:

Field	Type	Description
`task_name`	str	Sub-task the query belongs to (e.g. `gsm8k`, `mbpp`)
`query`	str	Full input prompt (personalized: JSON-encoded chat messages)
`ground_truth`	list/str	Reference answer(s)
`metric`	str	Scoring metric for this row
`choices`	str	Options for multiple-choice items (JSON-encoded)
`task_id`	str	ID within the sub-task
`model_name`	str	Candidate LLM this row was executed with
`response`	str	The model's actual response
`token_num`	int	Total tokens
`input_tokens` / `output_tokens`	int	Token breakdown (for price computation)
`response_time`	float	Latency in seconds
`performance`	float	Task score of this model on this query (0–1)
`embedding_id`	int	Index into precomputed query-embedding files (not included here)

Raw query configs (`*_queries`)

Each scenario also ships its raw query set (no model executions) as a <scenario>_queries config with train / valid / test splits — use these to run your own candidate models from scratch. Fields: task_name, query, ground_truth, metric, choices, task_id (+ conversation_id/category for the memory scenarios). The memory queries are the RAG top-k=5 turn-pair variant used in the published experiments. Note: the valid split exists only here; the routing-data configs have train/test.

Candidate pool & pricing (`llm_candidates` config)

18 models spanning $0.05–$1.25 per 1M input tokens (25× spread) served via Together AI / NVIDIA NIM. Row cost = input_tokens × input_price/1e6 + output_tokens × output_price/1e6.

Usage

python

from datasets import load_dataset

ds = load_dataset("ulab-ai/xRouteBench", "llmrouter_generic")   # any config name above
train, test = ds["train"], ds["test"]

pricing = load_dataset("ulab-ai/xRouteBench", "llm_candidates")["train"]

Composite reward for cost-aware routing (GraphRouter-style):

reward = α · norm(performance) − β · norm(price_cost)

💻 Code

The official implementation of LLMRouter, including router implementations, training and evaluation pipelines, and utilities for working with xRouteBench, is available on GitHub:

👉 [ulab-uiuc/LLMRouter](https://github.com/ulab-uiuc/LLMRouter)

Notes

—Query embeddings (.pt) are not included; they can be regenerated from the query field with any sentence encoder.
—The memory scenarios use the RAG top-k=5 turn-pair context variant.

📚 Citation

If you find xRouteBench useful for your research or projects, please cite it as:

bibtex

@article{feng2026llmrouter,
  title={LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers},
  author={Feng, Tao and Yu, Fangxu and Zhang, Haozhen and Dai, Zhongjie and Yuan, Liangqi and Lei, Zijie and Zhang, Weizhi and Zhu, Kunlun and Yue, Haodong and Xuan, Keyang and others},
  journal={arXiv preprint arXiv:2608.06867},
  year={2026}
}