AgentsSci/scientific-agent-protocol-traces

SciAgentTrace

Matched cross-domain dataset of scientific-agent protocols. The central comparison contains the same 6,653 problems under two actor models and four protocols: 53,224 trajectories in 40 complete model--benchmark--protocol groups. The broader table-first package contains 68,892 trajectories. Begin with trajectories, outcomes, or matched_outcomes, then follow stable identifiers to messages and compressed raw traces.

Repository: https://huggingface.co/datasets/AgentsSci/scientific-agent-protocol-traces

Quick start

from datasets import load_dataset

repo = "AgentsSci/scientific-agent-protocol-traces"
trajectories = load_dataset(repo, "trajectories", split="test")
outcomes = load_dataset(repo, "outcomes", split="test")
matched = load_dataset(repo, "matched_outcomes", split="test")

gemma_omni = trajectories.filter(
    lambda row: row["actor_model_id"] == "gemma_4_31b"
    and row["benchmark_id"] == "omnimath2"
)
print(gemma_omni.column_names)
print(gemma_omni[0])

Restricted revisions require an authorized Hugging Face account; authenticate once with hf auth login or pass a token. Public revisions load without credentials. See examples/quickstart.py for trace lookup and paired comparison.

Protocols

Coverage and release status

Only runs marked candidate or validated, with target_complete coverage and an exact publication comparison problem set, are eligible for public release. provisional rows remain visible for auditing but are excluded from final headline claims. This includes any source-complete run that does not yet match the publication problem set or lacks a frozen source manifest.

Statuses and exact counts are generated from registry/coverage.parquet; they are not maintained by hand. expected_records is the frozen source-run count, while target_expected_records is the publication matrix target. A run can be source-complete and still be target_incomplete.

Tables

In matched_outcomes, oracle_cheapest_successful and oracle_token_cost are populated only when every successful matched protocol has measured token cost. A successful protocol with missing tokens makes both oracle fields null rather than silently excluding that protocol from the four-protocol comparison.

Matched outcome snapshot

The four protocols below use exactly the same problem identities within each actor model and benchmark family. GPT-OSS 120B evaluates the final answers for both actor models. These descriptive outcomes are included to make the matched design easy to inspect; protocol-selection, message-value, and reviewer-coupling studies are documented as downstream uses rather than contributions of this dataset card.

Label semantics

signed_effect = outcome_with - outcome_without. Values are +1 helpful, 0 neutral, and -1 harmful. individual_vs_k0 compares one hypothesis with the no-hypothesis control. leave_one_out_marginal_effect removes one hypothesis from a K-member committee. These are interventional marginal effects; they are not described as Shapley values or as credit-conserving attribution.

Full definitions and provenance requirements are in docs/label_card.md.

Provenance

Compute site, inference backend, actor model, evaluator model, source commit, prompt version, and release status are separate fields. Frozen Aurora and CRUX runs include logical source URIs, per-file checksums, and run-summary checksums. Run-level inference_backends_json and source_commits_json preserve every backend and code revision represented by mixed-source runs; the singular fields use an explicit mixed marker when more than one value is present. Incomplete future runs belong in private staging with provisional status and are appended through the same schema after validation.

Application-level measurements

The table reports only measurements with complete run-level provenance and one release-linked source-manifest row per trajectory.

Across the matched comparison, client-observed elapsed time is available for 53,224 trajectories. Prompt, completion, and total tokens returned by successful responses, observable outer model-request submissions across protocol roles, evaluator invocation submissions, and elapsed time are jointly available for 39,918. Evaluator invocations are a subset of the total model-request submissions; the separate evaluator_attempt_count field in the historical metrics is not used. The Planner–Executor–Reviewer protocol contributes 13,306 elapsed-time records but no token or request totals because its historical collector mixed worker-cumulative main-model counters with memory-summary counters that reset for each problem; those components cannot be separated exactly after the fact.

For the other three protocols, cumulative successful-response usage and request counters are reconstructed by differencing consecutive rows within each selected metric file in execution order. Every source file, counter snapshot, accounting decision, and released row link is recorded in registry/source_cost_manifests/. Failed attempts without a returned usage object contribute to elapsed time but not token totals, and lower-level transient retries are not counted separately.

wall_time_seconds spans the complete per-problem application call, including inference requests, client orchestration, visible waiting and retries, and final evaluation. It is not scheduler walltime, server queue time, accelerator time, energy, dollar cost, or endpoint utilization. cost_coverage_ratio measures completeness only for the fields named by cost_available_fields_json; a value of 1.0 for a wall-only run does not imply complete token accounting. DHD and other unmeasured supplemental slices remain null rather than being estimated.

Licenses and benchmark use

This is a composite resource, so the root license is other. Every row carries its source license, and registry/licenses.parquet is authoritative. In particular, LAB-Bench is CC-BY-SA-4.0 with an upstream do-not-train request; MaScQA is CC-BY-NC-SA-4.0. Do not combine them under a single permissive claim.