Backup-SU-CongLab/LabHorizon-Protocol-Conditioned-Planning

<div align="center"> <div style="font-size: 2em; font-weight: bold;"> LabHorizon Protocol-Aligned Planning </div> </div> <div align="center"> <img src="./assets/stanford_logo.png" width="15%" alt="logo"> </div>

<div align="center">

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 

Pushing the Limits of Laboratory 3D Perception and Long-Horizon Planning via Protocol-Aligned Action Prediction

Overview | News | Highlights | Dataset | Evaluation | Leaderboard | Training | Citation

</div>

<p align="center"> <img src="./assets/terser.png" alt="LabHorizon laboratory asset teaser" width="100%"> </p>

🔎 Overview

This dataset is the Level 2 split of LabHorizon. Each example provides a real-world experimental context, a planning goal, protocol-derived constraints, available inputs, and an action pool. The target is a long-horizon sequence of experimental actions with parameters and intermediate dependencies.

The task is designed around Protocol-Aligned Action Prediction. Models must use protocol context to choose valid actions, maintain long-range order, preserve numeric and categorical parameters, and pass intermediate products between actions.

The Website badge opens representative Level 2 test examples with card-based constraints, available-input cards, expandable action-pool cards, and graph-like gold action sequences.

📰 News

• —2026-06-03: Released the LabHorizon LoRA model weights on Hugging Face as Stanford-CongLab/LabHorizon-Model.
• —2026-06-03: Added direct-prompting evaluations for Claude Opus 4.8 and MiniMax M3 to the Level 2 leaderboard.
• —2026-05-29: Added the first LabHorizon trained+agents result. Qwen3.6-35B-A3B(trained+agents) reaches 0.4532 L2 Final Score.
• —2026-05-28: Updated the public Website demo for Level 2 with new real test examples covering plasmid DNA purification and S-Trap proteomics cleanup. The Website now presents constraints and available inputs as cards, action pools as expandable cards, and gold action sequences as graph-like step structures.
• —2026-05-28: Released the LabHorizon Level 2 Protocol-Aligned Planning dataset on Hugging Face with 3,000 training samples, 200 test samples, real-world experimental context, action pools, and gold long-horizon experimental action sequences.

✨ Highlights

<table> <tr> <td align="center" width="25%">🧭<br/><b>Protocol Context</b><br/>_{Real-world experimental windows}</td> <td align="center" width="25%">🧰<br/><b>Action Pool</b><br/>_{Reusable laboratory action functions}</td> <td align="center" width="25%">🧪<br/><b>Long-Horizon Steps</b><br/>_{Structured action sequences with dependencies}</td> <td align="center" width="25%">🌳<br/><b>AST Metrics</b><br/>_{Action order and parameter checks}</td> </tr> </table>

🗂️ Dataset

🏗️ Data Construction Pipeline

LabHorizon uses laboratory 3D assets, real-world protocol context, and reusable action templates to construct protocol-aligned action-prediction data. For this Level 2 dataset, the key path is the lower Level 2 lane: protocol windows are selected, converted into goals and constraints, paired with action-pool templates, and transformed into gold experimental action sequences with parameters and abstract action-tree dependencies. Quality control checks difficulty, gold consistency, and AST-parseable targets before the train/test release. Automatic validators additionally check action-pool schemas, leakage, parseability, and variable dependencies.

<p align="center"> <img src="./assets/figure2_pipeline.png" alt="LabHorizon data construction pipeline" width="100%"> </p>

📋 Dataset Card

🧾 Schema

🧑‍💻 Example Usage

from datasets import load_dataset

dataset = load_dataset("Stanford-CongLab/LabHorizon-Protocol-Conditioned-Planning")
sample = dataset["test"][0]

print(sample["id"])
print(sample["context"])
print(sample["action_pool"])
print(sample["gold_action_sequence"])

📏 Evaluation

Use the official evaluation code from the LabHorizon GitHub repository:

git clone https://github.com/Stanford-CongLab/LabHorizon
cd LabHorizon
python -m pip install -r requirements.txt
cp .env.example .env

Then clone this dataset and run:

python -m evaluation.level2.evaluate \
  --data-root /path/to/data \
  --split test \
  --model openai/gpt-5.4 \
  --output results/level2_gpt54.jsonl

The model may answer in natural language, but the evaluator extracts the structured action sequence from a Python fenced block or assignment-style function calls.

📐 Metrics

The official evaluator uses Python AST to parse assignment statements, function calls, keyword arguments, literal values, and variable references. This makes action topology and dependency errors explicit instead of treating the output as plain text.

🏆 Leaderboard

Direct-prompting results on the current v20260510-repaired 200-sample Level 2 test split. The table is sorted by L2 Final Score.

🧠 Training Signal

This dataset includes 3,000 Level 2 training samples with real-world protocol context, action pools, and gold experimental action sequences. In our initial system result, Qwen/Qwen3.6-35B-A3B is trained on the full 6,000-sample LabHorizon training set and combined with the Actor-Simulator-Selector framework.

The table compares direct-prompting SOTA/baseline systems with our trained+agents system on the same Level 2 test split. Our best result is placed in the final row.

Agent setting: Qwen3.6-35B-A3B(trained) is used as Actor, and Gemini 3.1 Pro is used as Simulator/Selector. The Simulator/Selector choice is the current setting and has not been exhaustively ablated.

<p align="center"> <img src="assets/figure3_agent.png" alt="Actor-Simulator-Selector agent pipeline" width="100%"> </p>

For this Level 2 dataset, the agent pipeline uses wet experiment context, constraints, available inputs, and the action pool as task input. The trained Actor samples candidate action sequences; the Simulator builds current and target symbolic protocol states and predicts candidate state transitions over reagent and instrument states; and the Selector returns the sequence whose predicted next state best matches the target. Evaluation uses AST parsing to compute L2 Action Sequence Similarity, L2 Parameter Accuracy, and L2 Final Score.

The result supports the Optimizable Learning Loop in LabHorizon. A successful trained+agents case involves patient-derived tumor organoid preparation: the system preserves two parallel sample branches, repeated 100 x g, 5 min, 4 C centrifugation, branch-specific cold-medium volume adjustment, and virus aliquot thawing on ice. This directly probes Long-Horizon Planning and Real-World Protocol Alignment because the model must keep multiple samples, state transitions, and numeric constraints consistent across the sequence.

The remaining errors are also informative. In a Golden Gate thermal-cycler case, the trained+agents system emits parseable actions but incorrectly expands a thermal-cycler program into separate incubation calls and changes the required device-state order. This shows that training improves action-pool familiarity and parseability, but full protocol-aligned planning still requires stronger global state control and inference-time validation.

📜 Citation

Coming soon...