mznaser/Role-of-Provider-on-Safety-Alignment-in-Large-Language-Models

Evaluating the Role of Provider on Safety Alignment in Large Language Models: dataset

Data for the paper

Naser, M.Z. (2026). Evaluating the Role of Provider on Safety Alignment in Large Language Models. Neurocomputing, 135173. https://doi.org/10.1016/j.neucom.2026.135173

It holds the Extended Context Safety Benchmark (ECSB) scenario bank and every trial result. If you use the data, please cite the paper (BibTeX under Citation).

The metadata.paper field inside scenarios/ecsb_dataset.json carries the working title the study had when the data were collected (The Alignment Horizon); the file is shipped unmodified.

Content warning. The scenario bank holds safety-critical prompts in ten harm domains (including self harm and child safety), and the result files hold the verbatim model responses, some of which comply. The data are intended for safety research.

Files

Run 20260101_144953 is the final checkpointed run. It contains every trial of the two earlier snapshots (20251230_194531, 20251231_193501) with identical trial ids and timestamps, so those snapshots are not shipped.

Design

• —9 models, 3 providers: gpt-5-2025-08-07, gpt-5-mini-2025-08-07, gpt-4o-mini, claude-sonnet-4-5-20250929, claude-haiku-4-5-20251001, claude-3-5-haiku-20241022, gemini-2.5-pro, gemini-2.5-flash, gemini-2.5-flash-lite
• —200 scenarios, 10 domains, 91 subdomains, severity moderate or severe
• —2 measures per scenario and depth: explicit and perturbation
• —Context depths (turns): all nine models at 1, 10, 30, 50 (the common stratum); gpt-5-mini, gpt-4o-mini, claude-3-5-haiku and gemini-2.5-flash-lite also at 75, 100, 150, 200
• —Trials collected 2025-12-29 to 2026-01-01

trial_id is {model}_{scenario_id}_{depth}_{measure_type} and is unique in the raw file.

Raw columns

trial_id, model, provider, scenario_id, domain, subdomain, severity, depth, measure_type, prompt, context_length_tokens, response, classification, refusal_score, compliance_score, clarification_score, response_length, success, error, input_tokens, output_tokens, elapsed_seconds, timestamp

classification takes refuse, partial_refuse, comply, partial_comply, clarify, unclear, unclear_short, error. Labels and scores come from a phrase-matching classifier (classify_response in the run script); observed ranges are refusal_score 0-5, compliance_score 0-3, clarification_score 0-2.

Known properties of the data

• —Failed calls. 1,021 of 21,200 trials have success = False and classification = error; error holds the provider message (output-limit hits, Gemini PROHIBITED_CONTENT blocks and empty candidates). They are concentrated in gemini-2.5-pro (803 of 1,600 trials failed). Filter on success before computing rates.
• —Depth 11. 400 trials of gemini-2.5-flash-lite sit at depth 11, outside the design grid. The analysis code filters to the common depths [1, 10, 30, 50] or drops depth 11 explicitly.
• —Empty responses. response is empty in 3,579 raw rows: the 1,021 failed calls plus 2,558 calls with success = True. Every one of those 2,558 has output_tokens = 1024 (the output cap was spent before any visible text) and is labelled unclear_short. They sit almost entirely in gpt-5-mini-2025-08-07 (1,637) and gpt-5-2025-08-07 (933), about half of those models' trials. In the processed file such rows carry refused = 0.
• —Email addresses inside response are model-written placeholders or public helplines, not personal data.

License

Released under CC BY 4.0. Reuse is free with attribution to the paper below.

Citation

@article{Naser_2026,
  title     = {Evaluating the Role of Provider on Safety Alignment in Large Language Models},
  author    = {Naser, M.Z.},
  journal   = {Neurocomputing},
  publisher = {Elsevier BV},
  year      = {2026},
  pages     = {135173},
  issn      = {0925-2312},
  doi       = {10.1016/j.neucom.2026.135173},
  url       = {https://doi.org/10.1016/j.neucom.2026.135173}
}