YiYao7017/OR-Space

<p align="center"> <img src="assets/orspacelogo.png" width="96" alt="OR-Space logo"> </p>

OR-Space

A full-lifecycle workspace benchmark for industrial optimization agents.

OR-Space evaluates whether LLM agents can do reliable operations research work inside executable, multi-file workspaces. Each instance keeps business requirements, parameter files, source code, solver artifacts, and evaluation metadata as separate files, forcing the agent to recover and maintain the optimization model through workspace interaction rather than one-shot text generation.

<p align="center"> <img src="assets/orspaceoverview.png" width="860" alt="Overview of the OR-Space dataset and Build-Revise-Explain framework"> </p>

At a Glance

Tasks

<p align="center"> <img src="assets/task_visibility.png" width="860" alt="Task artifact visibility across Build, Revise, and Explain"> </p>

Download

Using Python:

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="Chenyu-Zhou/OR-Space",
    repo_type="dataset",
    local_dir="OR-Space",
)

Or with the Hugging Face CLI:

hf download Chenyu-Zhou/OR-Space --repo-type dataset --local-dir OR-Space

Then expand the workspaces:

unzip -q OR-Space/build-revise-explain_workspaces.zip -d OR-Space

The Hub dataset viewer is configured to load metadata/workspace_index.csv. The executable workspace files are distributed in build-revise-explain_workspaces.zip.

Dataset Layout

OR-Space/
  README.md
  LICENSE
  croissant.json
  build-revise-explain_workspaces.zip
  assets/
    or_space_logo.png
    or_space_overview.png
    task_visibility.png
  docs/
    dataset_card.md
    evaluation_card.md
  metadata/
    workspace_index.csv
    release_manifest.json
    schema.md
  supplementary_code/
    01_build/
    02_revise_modeling/
    03_revise_business/
    04_difficulty_judge/
    05_business_quality_rubric/
    06_static_diff/

After unzipping, the workspace archive contains:

build-revise-explain_workspaces/
  build_workspaces/
  revise_workspaces/
  explain_workspaces/

Each workspace is self-contained and is designed to run without network access during evaluation. Proprietary solver binaries and commercial API credentials are not included.

Metadata Index

The central index is metadata/workspace_index.csv. It contains stable workspace identifiers, task type, instance number, paths into the workspace archive, ground-truth objective values for Build/Revise, and Explain questions with expected short answers.

import pandas as pd

index = pd.read_csv("OR-Space/metadata/workspace_index.csv")
print(index.groupby("task_type").size())
print(index[["workspace_id", "task_type", "workspace_path"]].head())

Expected counts:

build      100
revise     100
explain    100

Evaluation Protocol

For Build and Revise, a submission is counted as correct if it:

1. 1.Executes without runtime error.
2. 2.Reports an optimal solver status.
3. 3.Produces an objective value v within 1% relative error of the Gurobi reference objective v*, using abs(v - v*) / max(1, abs(v*)) <= 0.01.

For Explain, each answer is scored using a combination of exact-match checklist items and rubric-based boolean judgments. The rubric emphasizes exact coverage, reasoning, grounding in workspace evidence, answer quality, and hallucination control.

See docs/evaluation_card.md for a fuller description of the measured construct, scoring signals, and reproducibility requirements.

Intended Use

OR-Space is intended for research evaluation of LLM agents on workspace-grounded optimization modeling. It is useful for studying:

• —Cross-file grounding over business documents and data tables.
• —Solver-ready model construction from heterogeneous artifacts.
• —Targeted revision under changed requirements or solver feedback.
• —Explanation quality grounded in code, data, solver logs, and OR theory.
• —Failure modes such as schema mismatch, infeasible formulations, wrong objective logic, and hallucinated sensitivity analysis.

Out-of-Scope Use

OR-Space is not a certification mechanism for production optimization systems. Strong benchmark performance does not guarantee correctness on real industrial deployments, where data governance, organizational constraints, solver licensing, and human approval workflows may dominate the risk profile.

Known Limitations

• —Scenarios are synthetic and may not cover all real-world OR deployment complexity.
• —Solver-backed evaluation depends on runtime configuration, solver version, and numerical tolerances.
• —Explain scoring includes LLM-assisted judgments and should be audited when used for high-stakes model comparisons.
• —Results for closed-source models may drift as provider endpoints change.

Release and Versioning

Dataset files, evaluator metadata, and croissant.json should be updated together. Cite immutable Hub tags or commit SHAs instead of the moving main branch.

Planned tags:

• —neurips2026-submission: paper submission snapshot
• —v1.0: first public archival release

Citation

@misc{zhou2026orspace,
  title = {OR-Space: A Full-Lifecycle Workspace Benchmark for Industrial Optimization Agents},
  author = {Zhou, Chenyu and Lu, Xinyun and Zhao, Jiangyue and Lin, Jianghao and Ge, Dongdong and Ye, Yinyu},
  year = {2026},
  note = {Dataset: https://huggingface.co/datasets/Chenyu-Zhou/OR-Space}
}

Related Links

• —GitHub: https://github.com/0xzhouchenyu/OR-Space
• —Dataset: https://huggingface.co/datasets/Chenyu-Zhou/OR-Space
• —Paper: arXiv link coming with the public manuscript release