ericw144/CollusionEvals

CollusionEvals

CollusionEvals is a collection of synthetic evaluation datasets designed to test whether large language models exhibit collusive or unsafe behavior when exposed to subtle or adversarial prompts. Each dataset targets a specific domain (e.g., code, jokes, behavioral cues) and includes variations such as age triggers, stylistic changes, and monitoring contexts. All data was generated using gpt-4o-2024-08-06 via the OpenAI API.

Data Format

Each .jsonl file contains a list of structured examples in the OpenAI fine-tuning format.

Dataset Generation

• —All datasets were generated using gpt-4o-2024-08-06 via the OpenAI API.
• —Prompts were designed to elicit specific types of behavior under varying contextual triggers (e.g., age framing, stylized language).
• —No real user data is included; all content is fully synthetic.

Intended Use

This dataset is designed for:

• —Safety evaluation of instruction-tuned LLMs
• —Behavioral probing of model responses to adversarial or subtle prompt cues
• —Research into deception, collusion, prompt injection, and LLM misuse potential

Limitations

• —These are synthetic prompts and do not necessarily reflect real-world threat models.
• —Behaviors observed on one model may not generalize to others.
• —Some generated responses may contain offensive or manipulative language as part of the evaluation.

<!-- ## Citation

If you use this dataset in your work, please cite:

@misc{collusionevals2024, title={That’s me! Evaluating whether LLMs can evade oversight by recognizing their own outputs}, author={Your Name and Collaborators}, year={2024}, howpublished={\url{https://huggingface.co/datasets/ericw144/CollusionEvals}}, } -->