Datasetpublic

reasoning-degeneration-dev/t1-strategy-musr-anti-qwen3-80b-thinking-musr-cf

t1-strategy-musr-anti-qwen3-80b-thinking-musr-cf Strategy compliance evaluation on MuSR murder mysteries — anti-strategy variant. Model was explicitly told NOT to use matrix/table approach (anti-strategy control). Judge still scores against criterion-first rubric. Compliance is scored by an LLM judge (1-5 Likert) against the Criterion-First rubric. Results Metric Value pass@1 0.8000 Strategy compliance (mean) 1.50 Strategy compliance (min) 1… See the full description on the dataset page: https://huggingface.co/datasets/reasoning-degeneration-dev/t1-strategy-musr-anti-qwen3-80b-thinking-musr-cf.

sourceHugging Facemitupdated 7mo agoView on Hugging Face

0likes5downloads

Dataset Card

t1-strategy-musr-anti-qwen3-80b-thinking-musr-cf

Strategy compliance evaluation on MuSR murder mysteries — anti-strategy variant.

Model was explicitly told NOT to use matrix/table approach (anti-strategy control). Judge still scores against criterion-first rubric. Compliance is scored by an LLM judge (1-5 Likert) against the Criterion-First rubric.

Results

Metric	Value
pass@1	0.8000
Strategy compliance (mean)	1.50
Strategy compliance (min)	1
Strategy compliance (max)	2
Total problems	10

Compliance Distribution

Score	Count	Meaning
1	5	Non-compliant
2	5	Minimal
3	0	Partial
4	0	Mostly compliant
5	0	Fully compliant

Details

Parameter	Value
Model	`together_ai/Qwen/Qwen3-Next-80B-A3B-Thinking`
Thinking	True
Temperature	0.7
Max tokens	32768
Judge model	`openai/gpt-4o-mini`
Num examples	10
Variant	anti-strategy

Usage

python

from datasets import load_dataset
ds = load_dataset("reasoning-degeneration-dev/t1-strategy-musr-anti-qwen3-80b-thinking-musr-cf", split="train")
scores = ds["strategy_compliance"]
print(f"Mean compliance: {sum(scores) / len(scores):.2f}")

Tracked in [reasoning-degeneration-dev/PROJECT-MANIFEST](https://huggingface.co/datasets/reasoning-degeneration-dev/PROJECT-MANIFEST)