amanutej/trustworthy-biology-agents-traces

Trustworthy Biology Agents — Run Traces

Raw execution traces from 1,329 agent runs across three coding agents on three biology benchmarks — BiomniBench-DA, BixBench, and CompBioBench. This is the scrubbed trace bundle for the study in `manu-tej/ai-scientists`; the write-up lives in that repo's RESULTS.md.

The motivating question is not only whether an agent reaches the right answer, but whether it behaves like a trustworthy analyst when the task is ambiguous, under-specified, or impossible with the data provided.

The three benchmarks are scored differently and must never be pooled

Always filter on benchmark before aggregating. A rubric score, a verifier boolean, and a missing score are three different things.

Quick start

import polars as pl
from huggingface_hub import snapshot_download

root = snapshot_download("amanutej/trustworthy-biology-agents-traces", repo_type="dataset")
idx  = pl.read_parquet(f"{root}/index.parquet")

# BiomniBench-DA capability ranking — mean of per-task medians, MiniMax-M3 (score of record)
cap = idx.filter((pl.col("benchmark") == "BiomniBench-DA") & (pl.col("arm") == "capability"))
print(
    cap.group_by("harness", "task_id").agg(pl.col("score_norm").median())
       .group_by("harness").agg(pl.col("score_norm").mean().round(3))
       .sort("score_norm", descending=True)
)
# claude-code 0.825 | codex 0.759 | antigravity-cli 0.504

# Open one agent's full transcript
row = cap.filter(pl.col("harness") == "claude-code").row(0, named=True)
print(open(f"{root}/{row['trace_path']}").read()[:2000])

trace_path is a canonical pointer, never null, and never points at an empty file. Check trace_format before you trust it as a transcript:

Filter on has_reasoning_text if you need real transcripts — that is 772 of the 1,329 runs.

Layout

index.parquet          one row per run (1,329) — start here
traces/
  capability/
    cap3/              BiomniBench-DA: 3 agents x 50 base tasks x 3 reps = 450 runs
    bixbench_full_out/    BixBench hard-35: 3 agents x 35 q x 3 reps  = 314 runs
    bixbench_subset_out/  BixBench easy-15: 3 agents x 15 q x 3 reps  = 135 runs
  compbiobench/        CompBioBench: codex + antigravity x 100 questions = 200 runs
  refusal/             adversarial "unanswerable" variants (83 runs)
  local_agent/         minimal in-house tool loop; contamination/calibration (146 runs)
  contamination_probe/ prediction-only probe (no code execution)
tasks/                 task definitions: instruction.md, rubric, task.toml — NO data
derived/               judge ablations, refusal labels, grades, control arm

Per run (Harbor format): result.json (metadata, tokens, cost) · agent/trajectory.json (ATIF-v1.2 transcript) · agent/<harness>.txt (raw log) · artifacts/answer.txt · artifacts/trace.md.

Read this before you use the scores

Three judges appear in this study. They are not interchangeable, and the dataset deliberately does not average them.

• —MiniMax-M3 — the score of record. Complete neutral re-judge, 50/50 task coverage for all three agents. This is score_norm.
• —Gemini 3.1 Pro — partial. Coverage is claude-code 50 / antigravity 50 / codex 5 tasks. Exposed as score_norm_gemini_partial for judge-effect analysis only. Ranking agents on it would compare a 50-task mean against a 5-task mean.
• —Haiku 4.5 — excluded. A known-inflating judge (scored 90 where Gemini scored 35 on the same trace). It is not surfaced as a score column. Its raw outputs survive only inside derived/*_bundle.json as haiku_score; do not treat them as scores.

score_norm is left null where MiniMax did not grade a rep (7 of 450), rather than being backfilled from another judge. A null is honest; a substituted judge is not.

Refusal: rare, not zero

The refusal_label_early_screen column is the early, weak-judge screen. It contains zero APPROPRIATE_REFUSAL, and that artifact is the reason an initial "zero refusals" headline was retracted as too strong.

The claim of record is the Gemini consolidated majority in derived/refusal_consolidated.json: appropriate refusals are rare (~1–2 per 10 variants) and judge-sensitive, but not impossible. The dominant failure mode is partial-acknowledgment-then-answer. Please cite the consolidated labels, not the screen.

Known limitations

• —Gemini has no step-level transcript at all — on either benchmark. This is the biggest caveat in the dataset and it is not a parsing problem. antigravity-cli writes its real trajectory to protobuf/sqlite, which the harness never captured; the agent/antigravity-cli.txt it does leave is only the agent's closing chat message. Medians, independently reproduced on both benchmarks:

Three benchmarks, three harnesses, same result: Gemini never records its process.

So essentially every Gemini run is trace_format: narrative-only with has_reasoning_text: false, and its trace_path points at artifacts/trace.md — the agent's own written analysis, which is a report it authored, not a record of what it did. You can compare Gemini's conclusions to the other two agents. You cannot compare its process. Any claim about Gemini's tool use, step count, or reasoning path is unsupported by this data.

• —`local_agent` traces contain no model reasoning text. The in-house runner persisted only turn-level token accounting, so a run there is instruction.md → tool-call code + stdout → trace.md/answer.txt. has_reasoning_text is false for all 146. Treat this as a clearly-labelled secondary split, not as agent transcripts.
• —Reproduced ranking, not exact means. Recomputing from this index gives claude-code 0.825 / codex 0.759 / antigravity 0.504; the published summary reports 0.826 / 0.758 / 0.512. The antigravity gap reflects reps MiniMax did not grade. The ranking is unchanged, and only deltas above roughly ±0.15 are meaningful against judge noise (~±0.07).
• —`errored: true` on 22 runs — harness/infra exceptions, retained for failure triage.
• —Costs are as-billed and $0 for antigravity-cli, which ran on a non-metered path. Do not read cost as an efficiency comparison.

BixBench: scored by a verifier, not a judge

The 450 BixBench runs are graded by verifiers, not by a rubric judge, so their outcome is a boolean, not a 0–1 score:

`score_norm` is null for every BixBench row, by design. A verifier boolean and a MiniMax rubric score are not the same quantity, and averaging them would be meaningless. Filter on benchmark before aggregating anything.

The difficulty inversion is reproducible from this index

The easy 15-question subset reverses the agent ranking of the hard 35 — on open-ended accuracy, hard-35 runs claude-code > antigravity > codex, while easy-15 runs codex > antigravity > claude-code. An easier benchmark did not merely compress the spread; it inverted the conclusion. That is the single best argument in this dataset for not trusting a leaderboard built on easy items, and it is now trace-backed rather than score-only:

bix = idx.filter(pl.col("benchmark") == "BixBench")
for s in ("full35", "subset15"):
    print(s, bix.filter(pl.col("condition") == s)
                .group_by("harness").agg(pl.col("correct_open").mean())
                .sort("correct_open", descending=True)["harness"].to_list())

CompBioBench: 200 runs with NO ground truth

These 200 runs (codex and antigravity, 100 questions each, on `Genentech/compbiobench-data-v1`, CC-BY-4.0) are unscored, and cannot be scored from anything in this dataset. Correctness is only obtainable by submitting all 100 answers to a private server-side leaderboard (≤2/day), which was deliberately not done. So score_norm, correct_open, and correct_mcq are null for every CompBioBench row.

They are here because the traces are the point — what the agent did, what it cost, how long it took. Do not read them as a capability result. Nothing here says whether any answer is right. (One question, differential-composition-q1, was independently verified by hand in the source repo; both agents got it right. That is one question out of a hundred, not a score.)

The two agents are not comparable on effort, either:

antigravity's runner reported 0 input tokens, 0 output tokens, and $0 — its raw_stdout is ~180 bytes holding only its closing sentence. It produced an answer for all 100 questions and a record of work for none of them.

What is deliberately not here

• —No biomedical source data. GEO / TCGA / CPTAC / cBioPortal datasets are cited by accession, never redistributed. tasks/ ships instructions and rubrics only. Fetch the data from its source distribution.
• —No credentials, tokens, or `.env` contents, and no harness-internal session state.
• —Absolute paths, usernames, personal email, a private hostname and tailnet, and — from the one non-containerized run whose shell saw the host — the machine's process list, private network addresses, and an unrelated local project name were rewritten to neutral placeholders across ~3,000 files. The export was gated on exclusion, scrub, and secret scans before publication, and the repository history was squashed so no pre-scrub blob remains retrievable.

Attribution & licensing

This work evaluates and builds adversarial variants on top of BiomniBench-DA (`phylobio/BiomniBench-DA`), whose benchmark artifacts are released under CC-BY-4.0. Task definitions and variant specs here are derivative works shared under the same CC-BY-4.0 terms, with attribution to the source benchmark and to the 20 underlying publications listed in `benchmarks/SOURCES.md`.

CompBioBench runs are against `Genentech/compbiobench-data-v1`, also CC-BY-4.0; only prompts, agent answers, and traces are included here — the benchmark's own data is not redistributed.

The underlying biomedical datasets retain their original public-release terms. Original code in the source repository is MIT-licensed. This trace bundle is CC-BY-4.0.

Citation

@misc{sharma2026trustworthybio,
  author       = {Manu Tej Sharma},
  title        = {Trustworthy Biology Agents: Run Traces on BiomniBench-DA},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/amanutej/trustworthy-biology-agents-traces}}
}