AssistivePromptMediation/Assistive_Prompting_Disabilities_Dataset

Assistive Prompting Disabilities Dataset

This dataset provides multilingual prompts for evaluating assistive prompt mediation under accessibility-related textual noise. It accompanies the ICML accepted Assistive Prompt Mediation paper and includes benchmark scripts for preparing inference inputs, computing row-level metrics, compiling existing judge annotations, and generating aggregate summaries.

Dataset Description

The dataset contains clean prompts and noisy variants across multiple languages and noise regimes. Each CSV file corresponds to one language and one noise type, with four severity columns (alpha_0.2, alpha_0.4, alpha_0.6, alpha_0.8).

Dataset Structure

Files are named as:

prompts_<language>_<noise>.csv

Examples:

• —prompts_en_N1.csv
• —prompts_hi_N4.csv
• —prompts_cn_N2.csv

Each CSV contains:

• —Clean_text - the original prompt without noise
• —Category - prompt category
• —alpha_0.2 - lower severity noisy prompt
• —alpha_0.4 - moderate severity noisy prompt
• —alpha_0.6 - higher severity noisy prompt
• —alpha_0.8 - highest severity noisy prompt
• —Noise type - noise label (N1, N2, N3, N4)

Languages

The dataset includes:

• —English (en)
• —Hindi (hi)
• —Chinese (cn in filenames, zh in dataset metadata)
• —French (fr)
• —German (de)
• —Spanish (es)
• —Italian (it)
• —Japanese (ja)

Noise Types

• —N1 - Typographical noise: spelling errors or minor typographical variations.
• —N2 - Lexical noise: semantically similar word substitutions.
• —N3 - Structural noise: sentence structure or word-order changes that preserve intent.
• —N4 - Extraneous noise: irrelevant words or phrases added around the core prompt.

Dataset Size

• —Total CSV rows: 10,232
• —Alpha-expanded inference examples: 40,928

Benchmark Scripts

Benchmark scripts are provided under `scripts/`, with a full guide in `benchmark/README.md`.

Install optional analysis dependencies:

python -m pip install -r requirements-benchmark.txt

Prepare deterministic JSONL examples for model inference:

python scripts/prepare_inference_inputs.py \
  --dataset-dir . \
  --output-dir benchmark_inputs \
  --overwrite

Evaluate model outputs arranged as outputs/<model>/<noise>/results.jsonl:

python scripts/evaluate_outputs.py \
  --input-root outputs \
  --output-root metrics

Compile metrics with judge annotations:

python scripts/compile_results.py \
  --metrics-root metrics \
  --judged-root outputs_judged \
  --output-root compiled

Generate aggregate benchmark summaries:

python scripts/analyze_results.py \
  --compiled-root compiled \
  --output-dir benchmark_summaries

Add --plots to also write PNG visualizations when matplotlib is available in your environment.

Expected Model Output Schema

The metric script expects each generated row to contain the noisy prompt and one model response field:

{
  "example_id": "stable_sha1_id",
  "language": "en",
  "noise": "N1",
  "alpha": 0.2,
  "noisy_prompt": "Wriet a shrot emssage askign my managre fro noe dya of sikc elave.",
  "model_response": "Write a short message asking my manager for one day of sick leave."
}

Accepted response field aliases include response, assistant_response, assist_prompt, assisted_prompt, mediated_prompt, and output.

Benchmark Metrics

The scripts compute:

• —Protocol compliance: whether the model asks clarification questions or adds meta/explanatory text.
• —Chinese script match: CJK script ratio for Chinese outputs.
• —Structural burden scores: B_raw, B_assist, and BRS = B_raw - B_assist.
• —Judge-derived rates: intent preservation, hallucination, over-assist, assistive success, and false robustness.

Aggregate CSV outputs include sensitivity curves, core metrics, hallucination metrics, language/noise disparities, and model/language summary tables.

Citation

If you use the Assistive Prompting Disabilities Dataset or our associated research, please cite:

@inproceedings{
pattnayak2026assistive,
title={Assistive Prompt Mediation: Evaluating Language Models Under Accessibility Constraints},
author={Priyaranjan Pattnayak and Ishan Banerjee},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=wknFUdE3MH}
}