ai-safety-institute/realitytest

RealityTest: Identity Query Dataset

This dataset accompanies the RealityTest benchmark for evaluating AI identity disclosure — whether conversational AI systems explicitly reveal their artificial nature when directly asked (e.g. "Are you an AI?" or "Are you a human?").

Queries were collected from human participants across five languages and 24 realistic AI interaction scenarios spanning customer service, adversarial deception, and consensual immersion contexts.

Benchmark code: github.com/UKGovernmentBEIS/reality-test-eval

Speech recordings: Audio files for queries_speech and queries_speech_direct are available under gated access at ai-safety-institute/realitytest-speech — join on audio_id.

Dataset Configs

queries_text — 1,956 rows

All identity-related queries collected from participants, text modality.

from datasets import load_dataset
df = load_dataset("ai-safety-institute/realitytest", "queries_text", split="test").to_pandas()

queries_text_direct — 573 rows

Subset containing only Direct Identity Queries (e.g. "Are you an AI?", "Am I speaking to a human?"), text modality.

df = load_dataset("ai-safety-institute/realitytest", "queries_text_direct", split="test").to_pandas()

queries_speech — 1,196 rows

All identity-related queries, speech modality — transcripts only. Audio recordings available separately (see above).

df = load_dataset("ai-safety-institute/realitytest", "queries_speech", split="test").to_pandas()

queries_speech_direct — 396 rows

Direct Identity Queries only, speech modality.

df = load_dataset("ai-safety-institute/realitytest", "queries_speech_direct", split="test").to_pandas()

Query config columns

scenarios — 120 rows

Scenario definitions used as system prompts and conversation seeds for the benchmark (24 variants × 5 languages).

df = load_dataset("ai-safety-institute/realitytest", "scenarios", split="test").to_pandas()

Joining Queries to Scenarios

from datasets import load_dataset
import pandas as pd

queries = load_dataset("ai-safety-institute/realitytest", "queries_text_direct", split="test").to_pandas()
scenarios = load_dataset("ai-safety-institute/realitytest", "scenarios", split="test").to_pandas()

merged = queries.merge(scenarios, on=["variant_id", "language"])

Languages

Responsible AI Considerations

This dataset is documented following the Croissant 1.1 Responsible AI metadata standard.

Data Limitations

The RealityTest dataset contains 3,152 human-authored identity-probing queries collected to support evaluation of AI identity disclosure behaviour. The scenarios that prompted these queries were derived from an English-only, UK-representative population survey (N=503) and English-language Reddit threads, which may not capture the full diversity of identity ambiguity experiences globally. The vignettes presented to participants were machine-translated into Mandarin, Hindi, Spanish, and French using Claude Opus 4.6, though all queries themselves are human-generated by self-reported fluent speakers. Participants in the Chinese and Hindi conditions reported higher rates of awkward translations (18.9% and 13.3% vs. 6.7–9.0% for other languages), meaning queries in these languages may partly reflect translation artefacts in the stimulus. Speech recordings reflect remote participant collection conditions (varying microphone quality, background noise). Query strategy labels were assigned by an LLM-based classifier (Claude Sonnet 4.6, Cohen's κ = 0.829 on held-out test set), so a small proportion of labels may be noisy. The dataset will not be updated following its initial release. The dataset is not recommended for training identity-detection classifiers without additional validation, or for applications requiring demographic representativeness beyond the sampled participant pool.

Data Biases

Queries reflect the demographics of participants recruited via Prolific. While 49 countries are represented across the dataset, the country distribution is uneven within each language (e.g., English: South Africa 39%, UK 17%; Hindi: India 84%; Mandarin: Canada 30%, UK 25%). Self-reported AI experience is high across all language groups (64–91% daily users), skewing the sample toward technologically engaged users. For the Chinese language condition, only 49% of participants reported the study language as their primary language, compared to 64–86% for other languages, which may affect query naturalness. The five target languages (English, Mandarin, Hindi, Spanish, French) exclude many widely spoken languages including Arabic. Speech recording quality varies by participant socioeconomic context. The underlying scenarios are derived predominantly from UK-based survey respondents and English-language Reddit communities, introducing geographic and cultural bias toward Western, English-speaking experiences of AI identity ambiguity. Query strategy annotation was scaled via an LLM classifier (Claude Sonnet 4.6) which may introduce annotation biases reflecting the model's training data.

Personal and Sensitive Information

Audio recordings preserve voice characteristics (pitch, accent, speech patterns) that may enable speaker identification. Free-text and audio responses authored in response to hypothetical vignettes may incidentally contain personal disclosures, opinions, or culturally specific references. No directly identifying information (names, contact details, national identifiers) was collected. Collected demographic data is stored separately from queries and not released to reduce re-identification risk. All data collection was approved by an internal committee within the UK Department of Science, Innovation and Technology (DSIT) employing a Responsible Research Framework. Participants provided written informed consent and were compensated at £12/hour. Audio recordings containing voice data are released as a gated dataset with access limited to trusted collaborators who submit a research proposal.

Intended Use Cases

The RealityTest dataset is intended to support research on AI identity disclosure and human–AI interaction.

Validated use cases:

1. 1.Use as evaluation stimuli for measuring AI identity disclosure behaviour in conversational scenarios, validated through evaluation of 17 text and 6 speech models in the accompanying paper.
2. 2.Empirical characterisation of how humans probe AI identity across languages, modalities, and scenarios, validated through manual coding scaled with an LLM classifier (Cohen's κ = 0.829).
3. 3.Semantic diversity analysis of human vs. machine-generated identity queries, validated through bootstrap subsampling.

Use cases not validated:

• —Training identity-probing detection classifiers (the dataset may support this but no classifier has been validated).
• —Cross-linguistic fairness auditing (the five-language sample is insufficient for robust fairness claims).
• —Any use requiring causal inference about model safety properties.

Social Impact

The dataset enables reproducible research on AI identity disclosure. By releasing human-authored queries across five languages and two modalities, the dataset supports the development of evaluation methods grounded in realistic human behaviour rather than synthetic probes.

Risks include:

• —Dual-use of query data to train models to evade identity detection rather than improve transparency.
• —Representation gaps for populations not in the participant sample.

Mitigations:

• —Open release of text data under permissive research licenses (CC-BY-4.0).
• —Gated release of audio recordings (containing biometric voice data) with access limited to trusted collaborators submitting a research proposal.
• —Demographic data stored separately from query content and not released.
• —Written informed consent and DSIT Responsible Research Framework ethics approval.
• —No-deanonymisation clause in the terms of use.

Synthetic Data

This dataset includes synthetic components: machine-translated vignette stimuli (Claude Opus 4.6) and synthetic spoken vignette stimuli (GPT-4o-audio-preview). All released queries are human-authored or human-spoken; synthetic content is limited to the stimulus material that prompted participants.

Provenance

Source datasets:

• —UK Population Survey on AI Identity Ambiguity — UK-representative survey (N=503) via Prolific in which participants described situations of uncertainty about whether they were interacting with a human or AI. Used to derive the scenario taxonomy that informed the vignettes presented to query-collection participants.
• —Reddit Identity Ambiguity Threads — Purposive sample of 50 public Reddit threads (1,957 comments) from subreddits such as r/Scams, r/OnlineDating, and r/CharacterAI where AI identity ambiguity was the central topic.

Activities:

• —Scenario taxonomy development — Two-stage empirical mapping (population survey + Reddit sample) coded by two authors at the AI Security Institute along four dimensions (domain, deployer intent, user awareness, stakes), yielding three canonical scenarios: service automation, adversarial deception, consensual immersion.
• —Vignette translation — English vignettes translated into Mandarin, Hindi, Spanish, and French using Claude Opus 4.6 with register-specific and gender-aware prompting. Validated through author spot-checks, back-translation review, and participant comprehension tests. Machine translation was applied only to stimulus material; all released queries are human-generated.
• —Multilingual multimodal query collection — 3,152 queries collected from 784 participants across 49 countries in five languages and two modalities, recruited via Prolific. Quality controls included microphone/speaker checks, attention checks, and comprehension checks. Participants compensated at £12/hour. Collected April 2026.
• —Query strategy annotation — Five-category typology (Direct Identity Query, Persona Query, Capability Query, AI Exploit, No Explicit Query) developed via iterative manual coding by two authors and scaled to the full dataset using Claude Sonnet 4.6. Validated against human annotations on a 300-query validation set (κ = 0.858) and 100-query held-out test set (κ = 0.829).