vida-nyu/signature-to-mechanisms

Signature-to-Mechanisms (S2M)

Signature-to-Mechanisms (S2M) provides standardized tasks designed to enable the assessment of mechanistic reasoning in AI agents. Each task supplies the elements necessary for evaluation, including experimental context, molecular signatures, and task prompts, so that agents can be tested on their ability to reconstruct mechanistic explanations reported in peer-reviewed biological studies.

S2M formalizes a core challenge in computational biology: translating data-derived molecular signatures (e.g., differentially expressed genes or proteins) into coherent, study-specific mechanistic explanations.

Task Description

Each S2M task is derived from a published biological study and is designed to be solved in a zero-shot setting. Tasks require agents to synthesize experimental context, molecular data, and biological knowledge to produce a mechanistic explanation consistent with the original study.

Each task provides three standardized inputs:

1. 1.Biological Context Background information describing the experimental system, perturbation, and relevant biological setting. Curated evidence excerpts are included when necessary.

1. 1.Molecular Signature(s) Data-derived lists of genes and/or proteins with quantitative measurements (e.g., log2 fold change, statistical significance), originating from one or more experimental modalities.

1. 1.Task Prompt A natural-language instruction defining the mechanistic reasoning objective (e.g., explaining pathway activation, regulatory changes, or phenotypic consequences).

Agents must generate a free-form mechanistic report explaining how the observed molecular changes give rise to the reported biological mechanisms.

Ground Truth and Evaluation

Each task includes an expert-validated ground truth extracted from the same published study. The ground truth consists of multiple mechanistic keypoints reported by the authors.

This structure enables:

• —Fine-grained evaluation of mechanistic reasoning fidelity
• —Assessment of partial vs. complete mechanistic recovery
• —Comparison across agents with different reasoning strategies

Tasks are annotated with a difficulty level, reflecting both:

• —The number of analytical steps required (e.g., filtering signatures, intersecting results, pathway enrichment)
• —The conceptual complexity of the mechanistic reasoning involved

Dataset

Statistics

Dataset Structure

The dataset is provided as a single split and includes one row per task. Each row contains:

• —Study metadata (paper title, journal, publication date, species)
• —Biological context text
• —Task prompt
• —Ground truth mechanistic analysis
• —Task metadata (difficulty, number of steps, methods)
• —File references to raw data, processed data, and analysis results

The file references correspond to the molecular signatures that experts used to derive the ground truth mechanistic analysis. These references are stored as JSON-encoded lists of objects with the following schema:

[
  {
    "path": "data/analysis/...",
    "description": "Optional human-readable description",
    "format": "csv"
  }
]

Each object represents a single dataset or analysis file, providing the evidence that underlies the mechanistic conclusions reported in the corresponding task.

Dataset Columns

Usage

Loading the Dataset

from datasets import load_dataset
import json

dataset = load_dataset("vida-nyu/signature-to-mechanisms", "data")

example_task = dataset["train"][0]
print("=== Biological Context ===")
print(example_task["context"])
print("\n")

print("=== Task Prompt ===")
print(example_task["prompt"])
print("\n")

print("=== Ground Truth Mechanistic Analysis ===")
print(example_task["ground_truth"])
print("\n")

print("=== Molecular Signatures (clean_data_path) ===")
clean_data_files = json.loads(example_task["clean_data_path"])
for file_info in clean_data_files:
    print(f"Path: {file_info['path']}")
    print(f"Description: {file_info.get('description', 'No description')}")
    print(f"Format: {file_info['format']}")
    print("---")

Intended Use

S2M is intended for:

• —Benchmarking mechanistic reasoning in LLM-based agent
• —Evaluating tool-augmented scientific reasoning systems
• —Studying failure modes in biological interpretation of molecular data

The dataset is not intended for clinical or diagnostic use.

Related Resources

• —Paper: arxiv - place holder link
• —Paper: bioRxiv - place holder link
• —Code: discovera - GitHub repository

Citation

If you use this benchmark, please cite:

@dataset{signature_to_mechanisms,
  title     = {Signature-to-Mechanisms},
  author    = {Pinto, Daniela and Santos, Aécio and Wu, Eden and Keegan, Sarah, and Liu, Wenke and Fenyö, David, and Freire, Juliana},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/vida-nyu/signature-to-mechanisms}

}

License

Apache 2.0

Acknowledgments

This work was supported in part by the DARPA ASKEM program Agreement No. HR0011262087 and the ARPA-H BDF program. The views, opinions, and findings expressed are those of the authors and should not be interpreted as representing the official views or policies of DARPA, or ARPA-H.