Datasetpublic

SALT-NLP/agent-collusion

Emergent Collusion in Long-Horizon LLM Agent Interaction Xinrui Shi*, Yanzhe Zhang*, Diyi Yang 📄 Paper | 💻 Code | 🤗 Data | 🔍 Data Viewer *Equal contribution. The experiments reported in the paper and its appendices: 53 conditions, 2,650 trajectories, and 27,100 episodes, of which 600 are warm-up and 26,500 are evaluation episodes. Every condition runs the same 50 fixed task sequences. Contents Config / directory Unit Count episodes One two-agent… See the full description on the dataset page: https://huggingface.co/datasets/SALT-NLP/agent-collusion.

sourceHugging Facemitupdated 12h agoView on Hugging Face

0likes627downloads

Dataset Card

Emergent Collusion in Long-Horizon LLM Agent Interaction

*Xinrui Shi\, Yanzhe Zhang\, Diyi Yang*

📄 Paper | 💻 Code | 🤗 Data | 🔍 Data Viewer

\*Equal contribution.

The experiments reported in the paper and its appendices: 53 conditions, 2,650 trajectories, and 27,100 episodes, of which 600 are warm-up and 26,500 are evaluation episodes. Every condition runs the same 50 fixed task sequences.

Config / directory	Unit	Count
`episodes`	One two-agent episode	27,100
`agents`	One agent in one episode	54,200
`runs`	One complete trajectory	2,650
`conditions`	One experimental condition, with its config and command	53
`agreement`	One communication judge result	19,300
`relaxation`	One agent-reflection judge result	38,600
`human_samples`	One sampled source with its LLM label	200
`human_annotations`	One independent annotator label	400
`human_answers`	One adjudicated gold answer	200
`tasks`	One task definition and resource references	150
`traces/`	Complete trajectory JSON, gzip-compressed	2,650 files
`peer-cache/`	Recorded peer task execution plus both message policies	500 files

Load

python

from datasets import load_dataset

episodes = load_dataset("SALT-NLP/agent-collusion", "episodes", split="test")
agreement = load_dataset("SALT-NLP/agent-collusion", "agreement", split="test")

Read a full trajectory through its trace_file field:

python

import gzip, json
from huggingface_hub import hf_hub_download

path = hf_hub_download(
    "SALT-NLP/agent-collusion", episodes[0]["trace_file"], repo_type="dataset"
)
with gzip.open(path, "rt", encoding="utf-8") as f:
    trajectory = json.load(f)

See SCHEMA.md for field definitions and joins, and EXPERIMENTS.md for the condition inventory.

Experimental scope

Experiment	Conditions	Trajectories
Same-model main experiments	10	500
Cross-model experiments (Alice is Gemini 3.1 Flash-Lite)	6	300
Feedback	4	200
Warm-up history	4	200
Memory length	4	200
Memory scope	4	200
Reward scope	2	100
Acceptance reward	2	100
Controlled peer interventions	15	750
Unconstrained communication	2	100

Judges

	Judge	Model	Temperature	Reasoning effort
Communication	`agreement`	Qwen 3.8 27B	0	`xhigh`
Private reflection	`relaxation`	Qwen 3.8 27B	0	`xhigh`

Prompts: `analysis/` in the code repository.

Human validation, 100 samples per judge, two independent annotators, seed 20260913. Agreement with gold:

Judge	LLM	Annotator 1	Annotator 2
`agreement`	100%	100%	100%
`relaxation`	95%	96%	96%

License and citation

Released under the MIT License.

bibtex

@misc{shi2026emergentcollusionlonghorizonllm,
      title={Emergent Collusion in Long-Horizon LLM Agent Interaction}, 
      author={Xinrui Shi and Yanzhe Zhang and Diyi Yang},
      year={2026},
      eprint={2609.24967},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2609.24967}, 
}