CoolFace
Datasetpublic

reasoning-degeneration-dev/t1-musr-structured-murder-together_ai-qwen-qwen3-next-80b-a3b-inst-4d7e26c9

t1-musr-structured-murder-together_ai-qwen-qwen3-next-80b-a3b-inst-4d7e26c9 Structured slot-filling evaluation for MuSR murder mysteries. Instead of the original paper's freeform cot+ hint, this uses a structured algorithm that forces the model to extract facts, slot-fill means/motive/opportunity, and compare suspects. Results Metric Value pass@1 0.8000 Total problems 10 Structured Prompt (excerpt) You are solving a murder mystery.… See the full description on the dataset page: https://huggingface.co/datasets/reasoning-degeneration-dev/t1-musr-structured-murder-together_ai-qwen-qwen3-next-80b-a3b-inst-4d7e26c9.

sourceHugging Facemitupdated 7mo agoView on Hugging Face
0likes14downloads
Dataset Card

t1-musr-structured-murder-together_ai-qwen-qwen3-next-80b-a3b-inst-4d7e26c9

Structured slot-filling evaluation for MuSR murder mysteries. Instead of the original paper's freeform cot+ hint, this uses a structured algorithm that forces the model to extract facts, slot-fill means/motive/opportunity, and compare suspects.

Results

MetricValue
pass@10.8000
Total problems10

Structured Prompt (excerpt)

You are solving a murder mystery. You must determine which suspect committed the murder.

The murderer needs ALL THREE of the following established:
- MEANS: Access to the murder weapon
- MOTIVE: A reason to kill the victim
- OPPORTUNITY: Access to the crime scene at the time of the murder

Innocent suspects may have at most two of these three proven, but never all three.

FOLLOW THIS EXACT STRUCTURED REASONING PROCEDURE:

For EACH suspect listed in the choices below, perform this analysis:

=== SUSPECT: [Name] ===

STEP 1 — EXTRACT KEY FACTS
Find exactly two key facts from the story about this suspect's relationship to the crime:
  Fact 1: [direct quote or close paraphrase from the story]
  Fact 2: [direct quote or close paraphrase from the story]

STEP 2 — SLOT-FILL ANALYSIS
Using ONLY the facts extracted above, determine each category:
  MEANS (weapon access):      [ESTABLISHED / NOT ESTABLISHED] — Evidence: [cite which fact]
  MOTIVE (reason to kill):    [ESTABLISHED / NOT ESTABLISHED] — Evidence: [cite which fact]
  OPPORTUNITY (scene access):  [ESTABLISHED / NOT ESTABLISHED] — Evidence: [cite which fact]

  Verdict: [X/3 established]

=== END SUSPECT ===

After analyzing ALL suspects, build a comparison:

COMPARISON:
| Suspect | Means | Motive | Opportunity | Total |
|---------|-------|--------|-------------|-------|
(fill in for each suspect)

CONCLUSION: The murderer is the suspect with ALL THREE established.
If both suspects appear to have all three, pick the one whe...

Details

ParameterValue
Modeltogether_ai/Qwen/Qwen3-Next-80B-A3B-Instruct
Temperature0.7
Top-p0.95
Max tokens32768
Samples1
ThinkingFalse

Usage

python
from datasets import load_dataset
dataset = load_dataset("reasoning-degeneration-dev/t1-musr-structured-murder-together_ai-qwen-qwen3-next-80b-a3b-inst-4d7e26c9", split="train")

Tracked in [reasoning-degeneration-dev/PROJECT-MANIFEST](https://huggingface.co/datasets/reasoning-degeneration-dev/PROJECT-MANIFEST)