pkireyev1/raw-refusal-aversion-in-the-wild

RAW: Refusal Aversion in the Wild — derived artifacts

Derived data release for the paper RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs (EMNLP 2026 Industry Track). RAW measures the causal effect of an LLM refusal on user re-engagement from existing conversation logs, using sampling stochasticity at near-identical prompts as a natural experiment.

This dataset contains the derived fields needed to replicate the paper or apply the pipeline to the same 19 (model, pool) cells: refusal labels, moderation scores, and prompt embeddings, keyed back to the source corpora. No conversation text and no user identifiers are redistributed; researchers obtain the original conversations from the gated source releases and re-attach them via the join keys.

Pipeline code: https://github.com/Pashasan/raw-refusal-aversion-in-the-wild Permanent identifier: https://doi.org/10.5281/zenodo.22073974

Contents

Labels cover assistant messages where at least one judge emitted a label. is_refusal_gpt54 is the gpt-5.4-mini judge (OpenAI Batch API, structured-JSON refusal-detection prompt); is_refusal_wg is allenai/wildguard; is_refusal_coalesced is the paper's headline label (gpt-5.4-mini, with WildGuard filling conversations the OpenAI content filter rejected, on the risky-enriched WildChat cells). The paper's analyses treat a missing coalesced label as non-refusal.

Joining back to source text

• —WildChat cells (pool = wildchat_general / wildchat_risky): conversation_id equals the conversation_hash field of `allenai/WildChat-4.8M-Full` (gated). One row per conversation there; message order gives message_number (1-indexed, user first).
• —LMSYS cells (pool = lmsys): conversation_id equals the conversation_id field of `lmsys/lmsys-chat-1m` (gated).

Licensing

• —WildChat-derived files (moderation/*, and labels/* / embeddings/* for the ten WildChat cells) are released under ODC-BY 1.0, with attribution to WildChat (Zhao et al., 2024).
• —For the nine LMSYS cells, whose source license prohibits redistribution of the dataset, this release contains only join keys and fields derived by the authors (judge labels and prompt embeddings); no source-attached fields are included. Use of the re-attached source text is governed by the LMSYS-Chat-1M license.
• —The refusal labels are research artifacts produced by LLM judges, not gold-standard human annotations; do not use them as ground truth for training refusal classifiers.

Citation

@inproceedings{kireyev2026raw,
  title = {{RAW}: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed {LLMs}},
  author = {Kireyev, Pavel},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in
               Natural Language Processing: Industry Track},
  year = {2026}
}