sci-modeling-bench/design-bench

<p align="center"> <img src="https://raw.githubusercontent.com/xukp20/sci-modeling-bench/main/assets/sci-modeling-bench-logo.png" alt="SciModelingBench logo" width="135"

</p>

<h1 align="center">SciModelingBench Design-Bench Data</h1>

<p align="center"> <strong>Canonical, provenance-tracked observations for scientific modeling and design Tasks.</strong> </p>

<p align="center"> <a href="https://github.com/xukp20/sci-modeling-bench">GitHub</a> &nbsp;&middot;&nbsp; <a href="https://pypi.org/project/sci-modeling-bench/">Python Package</a> &nbsp;&middot;&nbsp; <a href="https://github.com/xukp20/sci-modeling-bench/tree/main/docs">Documentation</a> &nbsp;&middot;&nbsp; <a href="https://huggingface.co/sci-modeling-bench">Organization</a> </p>

This repository stores the scientific observation layer used by the SciModelingBench Design-Bench suite. The Python package supplies validators, Agent-visible Protocols, trusted Objectives, submission contracts, and Task metrics. Data and evaluation logic are versioned separately so experiments can pin both.

Quick Start

Load one canonical table with Hugging Face Datasets:

from datasets import load_dataset

observations = load_dataset(
    "sci-modeling-bench/design-bench",
    name="drugmatrix_clinical_pathology",
    split="observations",
)

Or construct an end-to-end benchmark Task from the current package release:

from sci_modeling_bench.suites.design_bench import TFBind8BlackBoxOptimizationTask

task = TFBind8BlackBoxOptimizationTask.from_hub()
agent_input = task.build_input()

For reproducible work, pass an immutable Hub commit through the suite's revision= argument rather than relying on the current default branch.

Available Configs

Canonical rows are not always the final candidate-pool size. Protocols derive Agent-visible observations and label-hidden candidates from the pinned table without persisting candidate ranks or evaluator-only labels.

Trust Model

SciModelingBench does not use one evaluator type for every scientific domain:

• —Exact: complete tabulated or analytic mappings, such as TFBind8.
• —Measured: retained experimental observations or repeated simulator outcomes, such as GFP, Superconductor, DrugMatrix, and Hopper Controller.
• —Posterior-derived: deterministic aggregation grounded in raw replicate counts, such as TFBind10 Pho4.
• —Legacy learned surrogate: documented when relevant, but not silently treated as experimental truth when more reliable source measurements exist.

An Objective can be exact with respect to a frozen aggregation rule while the underlying scientific measurement remains noisy. Each config's documentation states that distinction explicitly.

Repository Layout

README.md
scimodelingbench.json
data/<config>/<split>.parquet
manifests/<config>.json
provenance/<config>/*.json

• —scimodelingbench.json maps config names to strict semantic manifests.
• —manifests/ defines inputs, targets, context, units, constraints, splits, sources, citations, versions, and license identity.
• —provenance/ records source hashes, transformations, release statistics, artifact hashes, and setting-specific audits.
• —data/ contains only canonical benchmark tables, not Python code or model checkpoints used by the package.

Documentation And Provenance

Licensing

The shared repository contains artifacts with different upstream terms, so the root card uses license: other. Every config manifest records its own license identity and source references.

Do not infer a Dataset config's license from the MIT license of the Python package.

Evaluation Boundary

Protocols hide evaluator labels through the package API, but these are public scientific artifacts and may be discoverable outside that API. Controlled Agent evaluations require an external harness to isolate the full Dataset, provenance, caches, source checkout, and network according to the intended setting. Query budgets and iterative feedback policy also belong to that external harness rather than this data repository.