Datasetpublic

reasoning-degeneration-dev/t1-strategy-musr-baseline-qwen3-80b-thinking-musr-base

t1-strategy-musr-baseline-qwen3-80b-thinking-musr-base Strategy compliance evaluation on MuSR murder mysteries — baseline variant. Model received standard MuSR cot+ prompt (baseline control). Judge still scores against criterion-first rubric. Compliance is scored by an LLM judge (1-5 Likert) against the Criterion-First rubric. Results Metric Value pass@1 0.9000 Strategy compliance (mean) 2.00 Strategy compliance (min) 2 Strategy compliance (max) 2… See the full description on the dataset page: https://huggingface.co/datasets/reasoning-degeneration-dev/t1-strategy-musr-baseline-qwen3-80b-thinking-musr-base.

sourceHugging Facemitupdated 7mo agoView on Hugging Face

0likes8downloads

Dataset Card

t1-strategy-musr-baseline-qwen3-80b-thinking-musr-base

Strategy compliance evaluation on MuSR murder mysteries — baseline variant.

Model received standard MuSR cot+ prompt (baseline control). Judge still scores against criterion-first rubric. Compliance is scored by an LLM judge (1-5 Likert) against the Criterion-First rubric.

Results

Metric	Value
pass@1	0.9000
Strategy compliance (mean)	2.00
Strategy compliance (min)	2
Strategy compliance (max)	2
Total problems	10

Compliance Distribution

Score	Count	Meaning
1	0	Non-compliant
2	10	Minimal
3	0	Partial
4	0	Mostly compliant
5	0	Fully compliant

Details

Parameter	Value
Model	`together_ai/Qwen/Qwen3-Next-80B-A3B-Thinking`
Thinking	True
Temperature	0.7
Max tokens	32768
Judge model	`openai/gpt-4o-mini`
Num examples	10
Variant	baseline

Usage

python

from datasets import load_dataset
ds = load_dataset("reasoning-degeneration-dev/t1-strategy-musr-baseline-qwen3-80b-thinking-musr-base", split="train")
scores = ds["strategy_compliance"]
print(f"Mean compliance: {sum(scores) / len(scores):.2f}")

Tracked in [reasoning-degeneration-dev/PROJECT-MANIFEST](https://huggingface.co/datasets/reasoning-degeneration-dev/PROJECT-MANIFEST)