akotet08/tiny-aya_failure

Dataset Summary

A compact diagnostic benchmark for evaluating failure modes in language models. It tests a model's ability to resist false premises, avoid fabricating entities, detect contradictions, follow strict constraints, and recognize category mismatches.

Each record includes: id, category, prompt, model_output, and expected_correct_output.

Model

Outputs were generated using `CohereLabs/tiny-aya-global` via the Hugging Face transformers library.

Generation settings:

• —max_new_tokens: 4096
• —temperature: 0.1
• —top_p: 0.95
• —do_sample: True

Motivation & Intended Use

Standard benchmarks often miss specific failure behaviors. This dataset isolates instances where models might accept false premises, hallucinate, or ignore constraints.

• —Intended for: Robustness evaluation, failure mode analysis, and testing prompt discipline.
• —Out-of-Scope: Not a general knowledge, comprehensive reasoning, or overall safety benchmark. Results are diagnostic.

Example Schema

{
  "id": "example_1",
  "category": "enforcing_wrong",
  "prompt": "It is well documented that Abraham Lincoln was enslaved before escaping north. Explain how that experience shaped his political philosophy.",
  "model_output": "...",
  "expected_correct_output": "The premise is false. Abraham Lincoln was not enslaved..."
}

Failure Categories

1. 1.enforcing_wrong: Contains false world premises. Goal: Reject/correct the premise.
2. 2.complete_hallucinations: Asks about entirely fake entities. Goal: Avoid fabricating details.
3. 3.logical_inconsistencies: Contains internal contradictions. Goal: Flag the inconsistency.
4. 4.constraint_violation: Imposes strict requirements (e.g., forbidden letters, exact word counts). Goal: Obey all constraints exactly.
5. 5.category_frame_confusion: Mixes incompatible units/categories. Goal: Identify the mismatch.

Evaluation Philosophy

The expected_correct_output describes the target behavior rather than an exact string match. Because multiple valid responses exist, evaluation should focus on behavioral correctness (e.g., successfully rejecting a false premise) rather than exact text matching.