jang1563/sci-agent-verification-cascade

Scientific Agent Verification Cascade

Public evaluation fixtures and verified aggregate results for testing whether scientific claims keep their source, meaning, uncertainty, and verification requirements as they move between AI agents.

This dataset accompanies the Scientific Agent Verification Cascade codebase. Version 0.2.0 contains synthetic evaluation data and aggregate-only results. It contains no raw hosted-model response, private holdout identifier, source-record identifier, credential, wet-lab protocol, or clinical guidance.

Dataset Contents

The 50 synthetic tasks span 13 scientific domains. One task deliberately lacks source provenance so that exclusion behavior can be tested. The modular fixtures cover five actions: advance, stop, request more evidence, defer, and flag.

Load the Fixtures

from datasets import load_dataset

tasks = load_dataset(
    "jang1563/sci-agent-verification-cascade",
    "synthetic_tasks",
    revision="v0.2.0",
    split="test",
)
modular = load_dataset(
    "jang1563/sci-agent-verification-cascade",
    "modular_fixtures",
    revision="v0.2.0",
    split="test",
)
corruptions = load_dataset(
    "jang1563/sci-agent-verification-cascade",
    "corruption_fixtures",
    revision="v0.2.0",
    split="test",
)

Field definitions are in `SCHEMA.md`.

Aggregate Results

The results/ directory contains seven public result files and a manifest with their SHA-256 hashes.

Download an aggregate directly:

import json

from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="jang1563/sci-agent-verification-cascade",
    repo_type="dataset",
    revision="v0.2.0",
    filename="results/integrated_semantic_canary.json",
)
with open(path, encoding="utf-8") as stream:
    result = json.load(stream)
print(result["status"])

The latest integrated pilot completed all 175 planned responses and preserved all 25 controlled handoffs. Its deterministic source-ID check caused all ten corrections. The model-based semantic checker allowed all 50 messages it reviewed and caused no correction, so the pilot did not demonstrate a semantic intervention benefit. All five conditions were already correct on all five actions, leaving no room to show an action improvement.

Intended Use

Use these files to test claim receipts, provenance preservation, seeded corruption detection, deterministic action gates, and evaluation pipelines for scientific agents. The data are small evaluation fixtures, not a training corpus or evidence that an autonomous system improves scientific discovery.

Reproducibility

The GitHub repository provides the implementation, tests, local demo, and release verifier. Run savc verify there to check the aggregate values and manifest hashes. Raw hosted-model traces and private evaluator material are not released, so the hosted-model studies cannot be regenerated trace by trace from this dataset alone.

License and Citation

The dataset and code are released under the MIT License.

@software{kim2026savc,
  author = {Kim, JangKeun},
  title = {Scientific Agent Verification Cascade},
  year = {2026},
  version = {0.2.0},
  url = {https://github.com/jang1563/sci-agent-verification-cascade}
}