kookhh0827/copy-inflation-search-agents

Copy Inflation in Search Agents

128,800 search-agent rollouts with the per-token log-probabilities intact — the evidence behind "Beyond Confidence: Test-Time Scaling for Multi-Turn Search Agents via Retrieval Grounding" (Findings of the ACL: EMNLP 2026).

Paper: https://arxiv.org/abs/2608.24024 Code: https://github.com/kookhh0827/copy-inflation-search-agents

The finding this data supports

When an LLM acts as a search agent, 78–92% of the tokens it generates are copied from documents sitting in its own context, and those copied tokens carry systematically inflated log-probabilities. We call this copy inflation. It shows up in every model family here — GPT-OSS, Tongyi, GLM, Kimi, MiniMax, DeepSeek — across four corpora.

The effect is that the agent's confidence stops discriminating within a question, which is the spread weighted voting actually consumes. And it resists repair: recomputing confidence on non-copy tokens restores the spread in 8/10 cells but not the signal (AUC falls in 9/10), while removing the documents and re-scoring the same tokens costs copied tokens about twice as much log-probability as non-copy ones. Anything reading agent confidence — early stopping, routing, abstention, confidence-shaped rewards, log-probability-based hallucination detection — inherits this.

Checking that claim requires the log-probabilities themselves, which is why they are released here rather than summary statistics. The stats files are 93% of the bytes in this repository.

Layout

Each cell has up to four files:

Archives extract to r0/ … r7/, one directory per rollout index.

huggingface-cli download kookhh0827/copy-inflation-search-agents \
  --repo-type dataset --include "data/frames/gpt-oss-120b/*" --local-dir .

zstd -dc data/frames/gpt-oss-120b/runs.tar.zst  | tar -x   # trajectories
zstd -dc data/frames/gpt-oss-120b/stats.tar.zst | tar -x   # token logprobs

Each archive expands to r0/ … r7/, one directory per rollout index, with one JSON file per question inside.

Start here

import json, glob, collections

runs = {}                      # after extracting runs.tar.zst
for f in glob.glob("r*/run_*.json"):
    d = json.load(open(f))
    runs.setdefault(d["query_id"], []).append(d)

# one question -> its 8 parallel rollouts
for qid, rollouts in list(runs.items())[:1]:
    for d in rollouts:
        docs = [i["output"] for i in d["result"]
                if i.get("type") == "tool_call" and i.get("tool_name") in
                   ("search", "get_doc", "visit")]
        print(d["final_text"][:80], "|", len(docs), "documents retrieved")

stats.tar.zst holds the matching per-token log-probabilities, keyed by the same filename stem, under turns[].token_logprobs.tokens[].top_tokens.

Cells

Four benchmarks × five models form the paper's main grid, plus response-period cells that probe the boundaries.

Question counts vary by cell; scripts/manifest.py records them.

Two things to know before you use this

Redacted benchmarks. GAIA is access-gated and its terms do not permit resharing the validation split; HLE's authors ask that the benchmark not be re-uploaded. For those cells we mask every verbatim span of 40 characters or more that matches any question or gold answer in the cell, wherever it appears — structured fields, judge prompts, the agent's own prose, and retrieved page text alike.

That last case is not hypothetical. These agents search the open web, and pages reproducing GAIA verbatim exist, so a rollout for one question routinely retrieves a page carrying many others along with their answers. Masking only a record's own question left roughly a fifth of the files still carrying benchmark text; matching against the whole cell brings it to zero.

Everything the agent produced is otherwise intact — trajectories, tool calls, log-probabilities, judge verdicts. To recover the questions, obtain the benchmarks from their official sources and join on qid. All other cells ship in full.

Two cells for the same benchmark and model. The 830-question browsecomp-plus/tongyi-deepresearch pool is the paper's deep-dive cell, but it has no per-token log-probabilities on disk — its stats archive carries turn-level aggregates only (21 KB per rollout rather than ~7 MB). For token-level work on that model and benchmark, use browsecomp-plus/tongyi-deepresearch-150, which is 150 questions with the full log-probability stream.

Licence

Our contributions — trajectories, scores, judge labels, derived annotations — are released under CC BY 4.0. Text returned by the environment (retrieved corpus passages, fetched web pages) remains under its original terms and is redistributed with its source identifiers retained. The underlying benchmarks keep their own licences: BrowseComp-Plus (MIT), BrowseComp (MIT), FRAMES (Apache-2.0), HotpotQA (CC BY-SA 4.0), GAIA (gated, not redistributed), HLE (MIT, not redistributed).

If any material here should not be redistributed, open a discussion on this repository and we will remove it.

Citation

@inproceedings{kook2026rgv,
  title     = {Beyond Confidence: Test-Time Scaling for Multi-Turn Search Agents
               via Retrieval Grounding},
  author    = {Kook, Hyunho and So, Junhyuk and Fu, Tianyu and
               Zheng, Haizhong and Chen, Beidi},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026}
}

Acknowledgments

These rollouts were generated with inference credits provided by Fireworks AI. Eight parallel rollouts per question, across four benchmarks and five models, each a full multi-turn search trajectory with per-token log-probabilities retained — 128,800 rollouts in total — needed serving at a scale we could not have reached otherwise.