Datasetpublic

reasoning-degeneration-dev/t1-strategy-musr-counterfactual-qwen3-80b-thinking-musr-cfact

t1-strategy-musr-counterfactual-qwen3-80b-thinking-musr-cfact Strategy compliance evaluation on MuSR murder mysteries — counterfactual-hypothesis variant. Model was instructed to use Counterfactual Hypothesis Testing with Early Termination (assume each suspect is guilty, test conditions, stop at first contradiction). Compliance is scored by an LLM judge (1-5 Likert) against the Criterion-First rubric. Results Metric Value pass@1 0.9000 Strategy… See the full description on the dataset page: https://huggingface.co/datasets/reasoning-degeneration-dev/t1-strategy-musr-counterfactual-qwen3-80b-thinking-musr-cfact.

sourceHugging Facemitupdated 7mo agoView on Hugging Face

0likes9downloads

Dataset Card

t1-strategy-musr-counterfactual-qwen3-80b-thinking-musr-cfact

Strategy compliance evaluation on MuSR murder mysteries — counterfactual-hypothesis variant.

Model was instructed to use Counterfactual Hypothesis Testing with Early Termination (assume each suspect is guilty, test conditions, stop at first contradiction). Compliance is scored by an LLM judge (1-5 Likert) against the Criterion-First rubric.

Results

Metric	Value
pass@1	0.9000
Strategy compliance (mean)	4.60
Strategy compliance (min)	4
Strategy compliance (max)	5
Total problems	10

Compliance Distribution

Score	Count	Meaning
1	0	Non-compliant
2	0	Minimal
3	0	Partial
4	4	Mostly compliant
5	6	Fully compliant

Details

Parameter	Value
Model	`together_ai/Qwen/Qwen3-Next-80B-A3B-Thinking`
Thinking	True
Temperature	0.7
Max tokens	32768
Judge model	`openai/gpt-4o-mini`
Num examples	10
Variant	counterfactual-hypothesis

Usage

python

from datasets import load_dataset
ds = load_dataset("reasoning-degeneration-dev/t1-strategy-musr-counterfactual-qwen3-80b-thinking-musr-cfact", split="train")
scores = ds["strategy_compliance"]
print(f"Mean compliance: {sum(scores) / len(scores):.2f}")

Tracked in [reasoning-degeneration-dev/PROJECT-MANIFEST](https://huggingface.co/datasets/reasoning-degeneration-dev/PROJECT-MANIFEST)