Samyol/xRouteBench

1---2configs:3- config_name: llmrouter_generic4  data_files:5  - split: train6    path: llmrouter_generic/train.parquet7  - split: test8    path: llmrouter_generic/test.parquet9- config_name: memory_locomo10  data_files:11  - split: train12    path: memory_locomo/train.parquet13  - split: test14    path: memory_locomo/test.parquet15- config_name: memory_longmemeval16  data_files:17  - split: train18    path: memory_longmemeval/train.parquet19  - split: test20    path: memory_longmemeval/test.parquet21- config_name: timeseries22  data_files:23  - split: train24    path: timeseries/train.parquet25  - split: test26    path: timeseries/test.parquet27- config_name: video28  data_files:29  - split: train30    path: video/train.parquet31  - split: test32    path: video/test.parquet33- config_name: multimodal_geometry3k34  data_files:35  - split: train36    path: multimodal_geometry3k/train.parquet37  - split: test38    path: multimodal_geometry3k/test.parquet39- config_name: multimodal_mathvista40  data_files:41  - split: train42    path: multimodal_mathvista/train.parquet43  - split: test44    path: multimodal_mathvista/test.parquet45- config_name: personalized46  data_files:47  - split: train48    path: personalized/train.parquet49  - split: test50    path: personalized/test.parquet51- config_name: llm_candidates52  data_files:53  - split: train54    path: llm_candidates/train.parquet55- config_name: llmrouter_generic_queries56  data_files:57  - split: train58    path: llmrouter_generic_queries/train.parquet59  - split: valid60    path: llmrouter_generic_queries/valid.parquet61  - split: test62    path: llmrouter_generic_queries/test.parquet63- config_name: memory_locomo_queries64  data_files:65  - split: train66    path: memory_locomo_queries/train.parquet67  - split: valid68    path: memory_locomo_queries/valid.parquet69  - split: test70    path: memory_locomo_queries/test.parquet71- config_name: memory_longmemeval_queries72  data_files:73  - split: train74    path: memory_longmemeval_queries/train.parquet75  - split: valid76    path: memory_longmemeval_queries/valid.parquet77  - split: test78    path: memory_longmemeval_queries/test.parquet79- config_name: timeseries_queries80  data_files:81  - split: train82    path: timeseries_queries/train.parquet83  - split: valid84    path: timeseries_queries/valid.parquet85  - split: test86    path: timeseries_queries/test.parquet87- config_name: video_queries88  data_files:89  - split: train90    path: video_queries/train.parquet91  - split: valid92    path: video_queries/valid.parquet93  - split: test94    path: video_queries/test.parquet95- config_name: multimodal_geometry3k_queries96  data_files:97  - split: train98    path: multimodal_geometry3k_queries/train.parquet99  - split: valid100    path: multimodal_geometry3k_queries/valid.parquet101  - split: test102    path: multimodal_geometry3k_queries/test.parquet103- config_name: multimodal_mathvista_queries104  data_files:105  - split: train106    path: multimodal_mathvista_queries/train.parquet107  - split: valid108    path: multimodal_mathvista_queries/valid.parquet109  - split: test110    path: multimodal_mathvista_queries/test.parquet111- config_name: personalized_queries112  data_files:113  - split: train114    path: personalized_queries/train.parquet115  - split: valid116    path: personalized_queries/valid.parquet117  - split: test118    path: personalized_queries/test.parquet119language:120- en121task_categories:122- text-generation123tags:124- llm-routing125- model-selection126- benchmark127pretty_name: xRouteBench128---129 130# xRouteBench — LLM Routing Benchmark131 132**xRouteBench** is a benchmark for training and evaluating **LLM routers** — systems that pick the best LLM from a candidate pool for each incoming query, trading off **performance vs. price cost**.133 134Every query in each scenario was executed against **all 18 candidate LLMs**, recording each model's response, task performance, token usage, and latency. A router learns from the `train` split which model to pick, and is evaluated on `test`.135 136## Scenarios (configs)137 138| Config | Domain | Train rows / queries | Test rows / queries | Metric |139|---|---|---|---|---|140| `llmrouter_generic` | 13 classic NLP benchmarks (MMLU, GSM8K, MATH, MBPP, ARC, …) | 80,802 / 4,487 | 67,122 / 3,729 | em_mc, GSM8K, MATH, code_eval, f1 |141| `memory_locomo` | Long-conversation memory QA (RAG top-k=5) | 15,930 / 885 | 5,652 / 314 | f1 |142| `memory_longmemeval` | Long-term memory eval (RAG top-k=5) | 4,986 / 277 | 1,818 / 101 | f1 |143| `timeseries` | Time-series understanding (7 sub-tasks) | 17,568 / 976 | 2,286 / 127 | mc |144| `video` | Egocentric video QA (Charades-Ego) | 3,618 / 201 | 486 / 27 | em |145| `multimodal_geometry3k` | Geometry math (multimodal) | 8,640 / 480 | 1,098 / 61 | em |146| `multimodal_mathvista` | Visual math reasoning | 14,400 / 800 | 1,800 / 100 | em, em_mc |147| `personalized` | Personalized preference (LLM-judge; chat-format queries) | 2,464 / 2,235 | 308 / 303 | llm_judge |148| `llm_candidates` | The 18-model candidate pool with pricing | 18 models | — | — |149 150## Schema (routing data)151 152Each row = one (query, candidate model) pair:153 154| Field | Type | Description |155|---|---|---|156| `task_name` | str | Sub-task the query belongs to (e.g. `gsm8k`, `mbpp`) |157| `query` | str | Full input prompt (personalized: JSON-encoded chat messages) |158| `ground_truth` | list/str | Reference answer(s) |159| `metric` | str | Scoring metric for this row |160| `choices` | str | Options for multiple-choice items (JSON-encoded) |161| `task_id` | str | ID within the sub-task |162| `model_name` | str | Candidate LLM this row was executed with |163| `response` | str | The model's actual response |164| `token_num` | int | Total tokens |165| `input_tokens` / `output_tokens` | int | Token breakdown (for price computation) |166| `response_time` | float | Latency in seconds |167| `performance` | float | Task score of this model on this query (0–1) |168| `embedding_id` | int | Index into precomputed query-embedding files (not included here) |169 170 171## Raw query configs (`*_queries`)172 173Each scenario also ships its **raw query set** (no model executions) as a174`<scenario>_queries` config with **train / valid / test** splits — use these to175run your own candidate models from scratch. Fields: `task_name`, `query`,176`ground_truth`, `metric`, `choices`, `task_id` (+ `conversation_id`/`category`177for the memory scenarios). The memory queries are the RAG top-k=5 turn-pair178variant used in the published experiments. Note: the `valid` split exists only179here; the routing-data configs have train/test.180 181## Candidate pool & pricing (`llm_candidates` config)182 18318 models spanning **$0.05–$1.25 per 1M input tokens** (25× spread) served via Together AI / NVIDIA NIM.184Row cost = `input_tokens × input_price/1e6 + output_tokens × output_price/1e6`.185 186## Usage187 188```python189from datasets import load_dataset190 191ds = load_dataset("ulab-ai/xRouteBench", "llmrouter_generic")   # any config name above192train, test = ds["train"], ds["test"]193 194pricing = load_dataset("ulab-ai/xRouteBench", "llm_candidates")["train"]195```196 197Composite reward for cost-aware routing (GraphRouter-style):198 199```200reward = α · norm(performance) − β · norm(price_cost)201```202## 💻 Code203 204The official implementation of **LLMRouter**, including router implementations, training and evaluation pipelines, and utilities for working with **xRouteBench**, is available on GitHub:205 206👉 **[ulab-uiuc/LLMRouter](https://github.com/ulab-uiuc/LLMRouter)**207 208 209## Notes210 211- Query embeddings (`.pt`) are not included; they can be regenerated from the `query` field with any sentence encoder.212- The memory scenarios use the RAG top-k=5 turn-pair context variant.213 214## 📚 Citation215 216If you find xRouteBench useful for your research or projects, please cite it as:217 218```bibtex219@article{feng2026llmrouter,220  title={LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers},221  author={Feng, Tao and Yu, Fangxu and Zhang, Haozhen and Dai, Zhongjie and Yuan, Liangqi and Lei, Zijie and Zhang, Weizhi and Zhu, Kunlun and Yue, Haodong and Xuan, Keyang and others},222  journal={arXiv preprint arXiv:2608.06867},223  year={2026}224}225