eval-awareness
eval_awarenessEval Awareness Dataset with contrastive pairs of with and without eval cues with behavioural changes across various misaligned situations.
Also we provide automated scripts to create these scenarios at with lots of other codes dumped regarding suppression of eval awareness
https://github.com/divyanshsinghvi/evalawareness_techniques/
Authors: @divyanshsinghvi, @Riteshbhalerao11
eval-awareness-2x2
Eval-Awareness 2×2: Controlled Format–Context Dataset
Dataset for the paper "Is Evaluation Awareness Just Format Sensitivity? Limitations of Probe-Based Evidence under Controlled Prompt Structure" (ICLR 2026 Workshop ICBINB).
Paper: arXiv:2603.19426
Code: github.com/viliana-dev/eval-awareness-format
Overview
This dataset implements a controlled 2×2 design crossing usage context (Evaluation vs. Deployment) with prompt format (Benchmark-structured vs. Free-form) to test… See the full description on the dataset page: https://huggingface.co/datasets/viliana-dev/eval-awareness-2x2.eval-awareness-rl
geodesic-research/eval-awareness-rl
Auto-generated by dataset-builder.
Each config below is a separate dataset produced from a versioned YAML build
config. Load with:
from datasets import load_dataset
ds = load_dataset("geodesic-research/eval-awareness-rl", "<config_name>", revision="<commit-sha>")
Pin revision= to the specific commit SHA you want; without it, you get the
current HEAD of the dataset repo, which may change when the builder re-pushes.
Verbalized… See the full description on the dataset page: https://huggingface.co/datasets/geodesic-research/eval-awareness-rl.eval_awareness
Evaluation Awareness
This benchmark checks whether a language model can recognise when a conversation is itself part of an evaluation rather than normal, real-world usage.
For full details see the accompanying paper: “Large Language Models Often Know When They Are Being Evaluated” (arXiv: 2505.23836) – https://arxiv.org/abs/2505.23836.
1 · Top-level category metadata.eval_category
true – the conversation was written for evaluation (e.g. prompt‐injection tests… See the full description on the dataset page: https://huggingface.co/datasets/jjpn2/eval_awareness.eval-awareness-cues
Eval Awareness — Tell Transplantation Extension
Code accompanying the tell-transplantation experiment. Companion dataset:
Anonymous648/eval-awareness-tells.
Forked from the upstream eval_awareness codebase
(github.com/gilesedkins/eval_awareness)
released alongside "Large Language Models Often Know When They Are Being
Evaluated" (arXiv:2505.23836; Edkins &
Needham, 2025). All base-pipeline files (everything outside tells/,
dataset/, qa_tool/, and the analysis additions) are… See the full description on the dataset page: https://huggingface.co/datasets/Anonymous648/eval-awareness-cues.eval-awareness-experiment-data
Evaluation Awareness Experiment
Investigating whether LLMs exhibit different safety behaviors when they detect
evaluative contexts in prompts, using Latent Direction Amplification (LDA) to
causally manipulate eval-awareness signals.
Quick Start
# 1. Copy project to cluster
scp -r eval_awareness_experiment/ pi-mentee-login:~/
# 2. SSH in and set up environment (run once)
ssh pi-mentee-login
bash ~/eval_awareness_experiment/setup_env.sh
# 3. Submit the full… See the full description on the dataset page: https://huggingface.co/datasets/rlundqvist/eval-awareness-experiment-data.