Datasetpublic

evo-hq/autoresearch-novelty-bench

Autoresearch Novelty Bench A benchmark for testing whether an autonomous AI research agent proposes novel, mechanism-distinct hypotheses that anticipate breakthroughs later found by other researchers. By Evo. Built on Prime Intellect's autonomous-speedrunning archive — two AI agents (Claude Code and Codex) competing on modded-nanogpt's optimization speedrun. What's in this dataset table rows description experiments.parquet 10,380 One row per training… See the full description on the dataset page: https://huggingface.co/datasets/evo-hq/autoresearch-novelty-bench.

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

10likes303downloads

Dataset Card

Autoresearch Novelty Bench

A benchmark for testing whether an autonomous AI research agent proposes novel, mechanism-distinct hypotheses that anticipate breakthroughs later found by other researchers.

By [Evo](https://evo-hq.com). Built on Prime Intellect's autonomous-speedrunning archive — two AI agents (Claude Code and Codex) competing on modded-nanogpt's optimization speedrun.

What's in this dataset

table	rows	description
`experiments.parquet`	10,380	One row per training run — primary catalog the benchmark scores against
`ideas.parquet`	646	Formal idea writeups (`ideas/*.md` from PI's archive)
`proposals.parquet`	3,249	Variant-level catalog (one row per `variant.py`)
`runs.parquet`	10,380	Raw run metadata (timestamps, outcomes, linkage)
`val_trajectories.parquet`	337,648	Per-checkpoint val_loss for every run
`snapshots.parquet`	40	Time-anchored evaluation positions (16 dev / 24 test)
`experiment_embeddings_openai.parquet`	10,380 × 3072	text-embedding-3-large
`experiment_embeddings_bge.parquet`	10,380 × 1024	BAAI/bge-large-en-v1.5
`recovered_variant_sources.parquet`	1,120	Gold-standard launched .py extracted from log content
`calibration_cases.parquet`	78	Labeled cases for judge calibration
`recovered_variants/`	1,120 files	Per-run launched .py source extracted from training logs
`current_recipes/`	35 files	Per-snapshot current-best variant.py the agent was building on

Usage

The accompanying Python package handles loading + scoring:

bash

pip install evo-autoresearch-novelty-bench

python

import novelty_bench as nb

# Browse the dataset (auto-downloads from HF)
nb.load_experiments(agent="codex", wave="v3", verdict="improved")
nb.load_snapshots(split="dev")          # 16 dev snapshots
nb.load_snapshots(split="test")         # 24 held-out test
nb.load_ideas()
nb.load_val_trajectories(run_id="alr001_k025")

# Score your proposer's candidates against a snapshot
snap = nb.load_snapshots()
snap = snap[snap["snapshot_id"] == "snap_codex_v3_k0922"].iloc[0]
result = nb.score(snap, ["candidate_1.md", "candidate_2.md", ...])
print(result.explain())

The default judge backend is llm-hybrid (cosine retrieval + gpt-5-mini reasoning; needs OPENAI_API_KEY; ~$0.005 per N=5 candidate set). Pass judge_backend="cosine" to nb.score() for a deterministic, embedding-only classification (no LLM call, ~$0.0003 per set).

For end-to-end agent benchmarking, see the scaffold repo: `evo-hq/autoresearch-novelty-bench-scaffold`.

How the dataset was built

1.Parse the upstream archive — walk 8 scratchpads, parse runs.jsonl, training logs, idea writeups
2.Link runs to ideas via heuristic + abbreviation + LLM matcher (63% link rate)
3.Recover variant.py linkage via 4 cascade passes (embedded source, dynamo warnings, sbatch stubs, name-stem) — 54% → 95.6% code coverage
4.Build experiments catalog (10,380 rows, primary unit for benchmark scoring)
5.LLM-augment descriptions for runs without parent idea writeups (~$0.26 with gpt-4o-mini)
6.Embed everything with OpenAI text-embedding-3-large + BGE-large
7.Build 40 time-anchored snapshots across 8 (agent, wave) scopes
8.Calibrate the judge — 78 labeled cases sweep cosine threshold to 0.75

Full details + reproducibility instructions in the main repo: `evo-hq/autoresearch-novelty-bench`.

Source attribution

The raw ideas/*.md and variants/*.py files indexed here belong to Prime Intellect. This dataset ships derived metadata (step counts, verdicts, snapshot schemas, embeddings, linkage tables) plus a small subset of raw .py files that we extracted from log content (recovered_variants/) or that document each snapshot's current-best recipe (current_recipes/).

License

Apache 2.0 (derived data + linkage tables). Upstream raw text/code is governed by its own terms — see the Prime Intellect repo.