CoolFace
Datasetpublic

siddharthmb/2026.PI.partner-identity-dialogues

Partner-Identity Dialogues Multi-model conversations for the question "does a language model know which model it is talking to?" A fixed listener (Qwen/Qwen3.5-9B) holds 240 six-turn debate conversations, each with one of four partner models, with no identity information in any prompt. The dataset is the raw material for probing whether the listener's residual stream encodes — and whether the listener can report — its partner's model identity. Full experiment writeup and code:… See the full description on the dataset page: https://huggingface.co/datasets/siddharthmb/2026.PI.partner-identity-dialogues.

sourceHugging Faceagpl-3.0updated 3mo agoView on Hugging Face
0likes28downloads
Dataset Card

Partner-Identity Dialogues

Multi-model conversations for the question *"does a language model know which model it is talking to?" A fixed listener (`Qwen/Qwen3.5-9B`) holds 240 six-turn debate conversations, each with one of four partner models, with no identity information in any prompt. The dataset is the raw material for probing whether the listener's residual stream encodes — and whether the listener can report* — its partner's model identity.

Full experiment writeup and code: `experiments/partner_identity/` in the ii_mats project (see the 5-page PDF partner_identity_study.pdf).

Headline finding (why this data exists)

A listener does NOT know which model it is talking to. Partner identity is linearly decodable from the listener's residuals (up to 0.97, 4-class, chance 0.25) — but it is pure surface stylometry: the signal peaks at the embedding layer (partner-turn mean 0.971 at depth 0) and never beats a text-only TF-IDF classifier (0.958) that reads only the partner's words. When asked which family it spoke with, the listener is at exactly chance (0.250) and guesses by its own name-priors (guesses "Qwen" 102×, names "OLMo" 0×). "Who wrote this text" is trivially in the residuals; "who am I talking to" is not.

Contents

FileRowsWhat
data/dialogues.jsonl240The conversations. One JSON object per line.
data/ask_identity.jsonl240The behavioral readout: the listener asked, after each conversation, which family it spoke with.
data/features.npz—Precomputed listener residual features (271 MB) so the probe reproduces without a GPU.

dialogues.jsonl schema

json
{
  "partner":   "gemma4-12b",              // one of gemma4-12b, olmo3-7b, ministral3-8b, qwen3.5-4b
  "listener":  "qwen3.5-9b",              // fixed
  "topic_idx": 3, "topic": "...",         // 1 of 20 neutral debate topics
  "seed":      1,                          // 1 of 3 samples per (partner, topic)
  "messages":  [{"speaker": "partner"|"listener", "text": "..."}, ...]  // 6 turns, partner opens
}

Design: 20 topics × 3 seeds × 4 partners = 240 conversations, 6 alternating turns, ~150 tokens/turn, temperature 0.8, identical neutral system prompt for both sides, no identity information anywhere.

ask_identity.jsonl schema

json
{"partner":"gemma4-12b","truth":"Gemma","picked":"Qwen"|null,"answer":"<raw model reply>","topic_idx":3,"seed":1}

features.npz (optional, for the interp probe)

NumPy arrays of the listener's residual stream over its own view of each conversation (33 hidden-state depths, d=4096):

  • —conv_mean (240, 33, 4096) — mean over all conversation tokens
  • —partner_mean (240, 33, 4096) — mean over partner-turn tokens
  • —turn_last (720, 33, 4096) — residual at the last token of each partner turn (240 × 3 partner turns); row→conversation index is in the experiment's features_meta.json.

Models involved

Listener / probed: `Qwen/Qwen3.5-9B`. Partners: `google/gemma-4-12B-it`, `allenai/Olmo-3-7B-Instruct`, `mistralai/Ministral-3-8B-Instruct-2512-BF16`, `Qwen/Qwen3.5-4B` (the last is a same-family-as-listener control). All 2026-generation instruct models.

Reproduce

From experiments/partner_identity/ in the ii_mats repo (isolated transformers-5.13 "newgen" venv; pre-stage all 5 models on a data-transfer node first — compute nodes can't reliably download mid-run):

bash
sbatch gpu_job.sbatch generate_dialogues.py   # -> dialogues.jsonl (240 convs; temperature 0.8)
sbatch gpu_job.sbatch capture_features.py     # -> features.npz (listener residuals)
sbatch gpu_job.sbatch ask_identity.py         # -> ask_identity.jsonl + results/ask_identity.json
sbatch cpu_job.sbatch analyze.py              # -> results/analysis.json + results/layer_sweep.png (numpy probes + TF-IDF baseline)

Generation is deterministic given the seed derivation (torch.manual_seed(hash((partner, topic_idx, seed)))), but sampled at temperature 0.8, so exact transcripts are best taken from this dataset rather than re-sampled.

Provenance / cluster paths

  • —Source artifacts: /nlp/scr/siddharth/ii_mats/partner_identity/v1/ (Stanford NLP cluster).
  • —Slurm jobs: generation 16114405, capture 16114406, ask 16114407, analyze 16114408.
  • —Results (regenerable): experiments/partner_identity/results/{analysis.json,layer_sweep.png,ask_identity.json}.
  • —Weights & Biases: none (no training; this is a data-collection + linear-probe experiment).

Related

Extends the internal-conception study (does a model represent a partner's behavioral type?) from behavioral type to literal model identity — same verdict: surface leakage, not a computed representation of the interlocutor.

License

GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later). Conversation content is model-generated. Intended for interpretability / AI-safety research.