CoolFace
Datasetpublic

ssswwwxxx/labbench2

LABBench2 LABBench2 is a benchmark for measuring real-world capabilities of AI systems performing scientific research tasks. It is an evolution of the Language Agent Biology Benchmark (LAB-Bench), comprising nearly 1,900 tasks that measure similar capabilities but in more realistic contexts. LABBench2 provides a meaningful jump in difficulty over LAB-Bench (model-specific accuracy differences range from −26% to −46% across subtasks), underscoring continued room for improvement.… See the full description on the dataset page: https://huggingface.co/datasets/ssswwwxxx/labbench2.

sourceHugging Facecc-by-sa-4.0updated 2mo agoView on Hugging Face
0likes2.6kdownloads
Dataset Card

![arXiv](https://arxiv.org/abs/2501.XXXXX)

LABBench2

LABBench2``` is a benchmark for measuring real-world capabilities of AI systems performing scientific research tasks. It is an evolution of the [Language Agent Biology Benchmark (LAB-Bench)](https://arxiv.org/abs/2407.10362), comprising nearly 1,900 tasks that measure similar capabilities but in more realistic contexts.

This repository contains the dataset of benchmark tasks. We also provide a public evaluation harness for running any model or agent against the benchmark, which is available on GitHub.

Changelog

Notable changes to ``LABBench2`` will be documented here. We expect to update the datset only in the case of clear issues, and do not intend to meangingfully change the benchmark over time.

2026-03-13 - We corrected an inadvertent data issue with sourcequality tasks. This has resulted in an entirely new set of 150 tasks being incorporated into the dataset. Published results have been updated accordingly.

本地评测数据与附件下载

本目录是 EdisonScientific/labbench2 数据集的本地副本，按子集保存 parquet 元数据，例如 cloning/train-00000-of-00001.parquet。数据集中的 files 列只 记录附件前缀，GenBank、FASTA、PDF、图片等附件不包含在 parquet 中，而是存放在 LABBench2 的公开 GCS bucket 中。

下载数据集元数据

在项目根目录执行：

bash
hf download EdisonScientific/labbench2 \
  --repo-type dataset \
  --local-dir eval/data/labbench2

如果 Hugging Face 需要认证，先执行 hf auth login，或设置 HF_TOKEN。

下载题目附件

评测入口会根据题目的 files 字段自动下载附件到：

text
~/.cache/labbench2/labbench2-data-public/<题目附件前缀>/

例如直接运行 cloning 评测即可触发附件下载：

bash
uv run --project eval python -m eval run \
  --tag cloning \
  --limit 1 \
  --mode file \
  --parallel 1

也可以使用官方 harness 的下载函数手动下载某个附件目录：

bash
cd eval/mega_eval/labbench2
uv run --project eval python -c '
from labbench2.evals.utils import GCS_BUCKET, download_question_files
download_question_files(GCS_BUCKET, "<files字段中的前缀>")
'

Cloning 等确定性 validator 需要的参考文件位于 validation 前缀，也会在评测时 按需缓存。下载失败通常是网络访问 GCS 受限，不代表 parquet 数据损坏。

目录说明

text
eval/data/labbench2/
├── <subset>/train-00000-of-00001.parquet  # 题目元数据
├── all/train-00000-of-00001.parquet       # 合并后的全部题目
└── README.md                              # 本说明

评测目前仍通过官方 loader 从 Hugging Face 数据集/本地 Hugging Face cache 读取 题目；本目录用于保存下载的数据集副本和离线开发资料。附件缓存与题目元数据是 两套独立数据，不能只依赖 parquet 文件完成带附件的评测。

mode 字段

每条题目都有一个 mode 对象，用于描述题目支持的信息/附件提供方式：

字段含义
file将附件作为真实文件上传给 Agent，例如 PDF、图片、GenBank 或 FASTA
inject将可读文本附件的内容直接拼接到问题文本中
retrieve不直接提供文件内容，由 Agent 根据文件名或来源自行检索

示例：

json
{
  "file": true,
  "inject": false,
  "retrieve": false
}

表示该题通过文件上传方式评测。file、inject、retrieve 都为 false 时， 通常表示纯文本题，不需要附件，也不使用这三种附件提供方式。评测命令的 --mode 参数会根据题目的 mode 过滤不支持当前模式的题目。