starpacker52/biomnibench-organized

BioMniBench DA — Reorganized

A clean, manifest-driven reorganization of the BioMniBench DA (Data Analysis) task suite, shaped for use with the `biomnibench-adapter` evaluation harness and the native skill-learning loop that ships with it.

This Hugging Face repository hosts the metadata, evaluation rubric and data manifest for all 50 tasks. The raw input data files (which total ~77 GB and originate upstream from GEO/TCGA/cBioPortal/etc.) are not redistributed here — see Getting the data files below for the recipe.

Snapshot

Task types

association-testing (8), mutation-analysis (6), differential-expression (5), predictive-modeling (4), clustering (4), multi-omic-integration (4), cell-composition (3), pathway-enrichment (3), survival-analysis (3), chromatin-profiling (3), cross-cohort-comparison (2), cell-cell-communication (1), co-expression-networks (1), gwas-eqtl (1), tcr-repertoire (1), …

What is in this repo

biomnibench-organized/
├── manifest.json                 # canonical index: 50 tasks + per-task metadata
├── organization_summary.json     # provenance: how each task was migrated from BioMniBench DA
└── tasks/
    └── <task_id>/                # one folder per task (da-1-3, da-1-4, …, da-26-2)
        ├── README.md             # task description (taken from upstream)
        ├── task.toml             # canonical task descriptor: category, difficulty,
        │                         # resource limits, verifier env, …
        ├── task_manifest.json    # small machine-readable header
        ├── data_files.tsv        # listing of the raw data files the task needs,
        │                         # with sizes — used by the download helper
        └── evaluation/
            ├── rubric.txt        # plain-English grading rubric (used by the LLM judge)
            ├── llm_judge.py      # the actual judge script run after each agent run
            └── test.sh           # bash entrypoint that the harness invokes

Total size in this repo: ~2.4 MB across ~350 files.

The companion code repository `biomnibench-adapter` knows how to consume this layout directly: point its tasksRoot config at a local copy of this repository (plus the downloaded data folders) and it will find everything.

Why "reorganized"?

The upstream BioMniBench DA tasks have a Docker-centric layout. To make them usable from a source-native harness, we:

1. 1.Flattened the per-task layout to <task>/README.md, <task>/task.toml, <task>/evaluation/{rubric,llm_judge,test.sh} and <task>/envs/data/….
2. 2.Materialised a `task.toml` for each task that captures category, difficulty, resource limits, verifier model and verifier env — i.e. everything the harness needs without consulting Docker compose files.
3. 3.Added `task_manifest.json` as a tiny machine-readable header.
4. 4.Verified every task end-to-end by running an evaluation pass and recording success / failure / timeout outcomes (see the companion repository's RERUN_REPORT.md).

organization_summary.json records, for each task, the source path it was migrated from and any issues encountered during conversion.

Loading the metadata

import json
from huggingface_hub import snapshot_download

repo_dir = snapshot_download(
    repo_id="starpacker52/biomnibench-organized",
    repo_type="dataset",
)

manifest = json.load(open(f"{repo_dir}/manifest.json"))
print(manifest["total_tasks"], "tasks")
for t in manifest["tasks"][:3]:
    print(t["task_id"], t["category"], t["difficulty"], t["task_type"])

If you only need the metadata for a single task:

from huggingface_hub import hf_hub_download

toml_path = hf_hub_download(
    repo_id="starpacker52/biomnibench-organized",
    repo_type="dataset",
    filename="tasks/da-1-3/task.toml",
)

Getting the data files

Each task folder contains a data_files.tsv listing the raw data files the task needs:

clinical.xlsx                       26941
GSE236581_barcodes.tsv          27122075
GSE236581_counts.mtx         19178745260
…

The files themselves come from public biomedical repositories (GEO, TCGA, cBioPortal, …) and are governed by their own licences. The upstream `phylobio/BiomniBench-DA` HuggingFace dataset already hosts a canonical copy of all of them. We provide a one-line hydrator that pulls from there directly:

Option 1 — Auto-download from HuggingFace (recommended)

# 0. Accept the upstream licence once at
#    https://huggingface.co/datasets/phylobio/BiomniBench-DA
# 1. Get a free token at https://huggingface.co/settings/tokens
export HF_TOKEN=hf_xxxxxxxxxxxx

# Optional: if you are in mainland China
# export HF_ENDPOINT=https://hf-mirror.com

# 2. Hydrate every task's envs/data/ + data/ + visible_data/
python download_data.py --hydrate-from-hf

# Or restrict to one task for a quick smoke test:
python download_data.py --hydrate-from-hf --tasks da-9-1

The script:

• —Reads each task's data_files.tsv.
• —For every missing file, calls hf_hub_download on phylobio/BiomniBench-DA::<task_id>/environment/data/<file>.
• —Stores it under tasks/<task_id>/envs/data/<file> and hardlinks it into the matching data/ and visible_data/ folders so every runner layout works.

Option 2 — Copy from a local BioMniBench mirror

If you already have the upstream release on disk:

python download_data.py --upstream /path/to/BioMni/DA --hydrate

Option 3 — Pull from primary sources

Each task's README.md lists the GEO / TCGA / cBioPortal accessions; you can download them directly from those primary repositories if you prefer to avoid HuggingFace.

How this dataset is used

The intended consumer is the source-native evaluation + skill-learning harness `biomnibench-adapter`. A typical loop is:

# 0. Get the framework
git clone https://github.com/starpacker/biomnibench-adapter.git
cd biomnibench-adapter

# 1. One-line bootstrap — installs deps, clones this dataset, auto-downloads all raw data
export HF_TOKEN=hf_xxxxxxxxxxxx
./scripts/bootstrap.sh

# 2. Run a single task
bun src/harness/evaluation/cli.ts \
  --task-dir ./biomnibench-data/tasks/da-9-1 \
  --output-root /tmp/biomnibench-runs

# 3. Learn skills from collected trajectories
bun src/skills-learning/cli.ts cycle

Citation

If you use this dataset, please cite both the upstream BioMniBench benchmark and this reorganization:

@misc{biomnibench-organized,
  title   = {BioMniBench DA, Reorganized},
  author  = {Ying, Jiahe},
  year    = {2026},
  url     = {https://huggingface.co/datasets/starpacker52/biomnibench-organized}
}

@misc{biomnibench-upstream,
  title   = {BioMniBench},
  author  = {Snap Lab, Stanford},
  url     = {https://github.com/snap-stanford/BioMni}
}

Licence

Metadata, manifests, rubrics and evaluation scripts in this repository are released under the MIT licence. The underlying biomedical data files (not redistributed here) are governed by their respective primary-source licences (GEO, TCGA, cBioPortal, …).