Datasetpublic

reasoning-degeneration-dev/t1-strategy-countdown-treesearch-together_ai-moonshotai-kimi-k2-thinking-kimi-think

t1-strategy-countdown-treesearch-together_ai-moonshotai-kimi-k2-thinking-kimi-think Strategy compliance evaluation on countdown arithmetic — tree-search variant. Model was instructed to use a systematic tree search with explicit backtracking. Compliance is scored by an LLM judge (1-5 Likert). Results Metric Value pass@1 0.2000 Strategy compliance (mean) 1.20 Strategy compliance (min) 1 Strategy compliance (max) 2 Total problems 10… See the full description on the dataset page: https://huggingface.co/datasets/reasoning-degeneration-dev/t1-strategy-countdown-treesearch-together_ai-moonshotai-kimi-k2-thinking-kimi-think.

sourceHugging Facemitupdated 7mo agoView on Hugging Face

0likes25downloads

Dataset Card

t1-strategy-countdown-treesearch-together_ai-moonshotai-kimi-k2-thinking-kimi-think

Strategy compliance evaluation on countdown arithmetic — tree-search variant.

Model was instructed to use a systematic tree search with explicit backtracking. Compliance is scored by an LLM judge (1-5 Likert).

Results

Metric	Value
pass@1	0.2000
Strategy compliance (mean)	1.20
Strategy compliance (min)	1
Strategy compliance (max)	2
Total problems	10

Compliance Distribution

Score	Count	Meaning
1	8	Non-compliant
2	2	Minimal
3	0	Partial
4	0	Mostly compliant
5	0	Fully compliant

Details

Parameter	Value
Model	`together_ai/moonshotai/Kimi-K2-Thinking`
Thinking	True
Temperature	0.7
Max tokens	32768
Judge model	`openai/gpt-4o-mini`
Num examples	10
Num args	8
Variant	tree-search

Usage

python

from datasets import load_dataset
ds = load_dataset("reasoning-degeneration-dev/t1-strategy-countdown-treesearch-together_ai-moonshotai-kimi-k2-thinking-kimi-think", split="train")
scores = ds["strategy_compliance"]
print(f"Mean compliance: {sum(scores) / len(scores):.2f}")

Tracked in [reasoning-degeneration-dev/PROJECT-MANIFEST](https://huggingface.co/datasets/reasoning-degeneration-dev/PROJECT-MANIFEST)