reasoning-degeneration-dev/t1-structured-reasoning-full-together_ai-zai-org-glm-5-16c46443
t1-structured-reasoning-full-together_ai-zai-org-glm-5-16c46443 Structured reasoning evaluation: instead of injecting synthesized facts, this uses a static system prompt that teaches the model a heuristic search FORMAT with explicit structural markers ([STEP], [PRUNE], [BACKTRACK], [REVIEW OPTIONS], [SOLUTION FOUND]). Inspired by HandCraftedCountdownSearch — models SFT'd on structured search traces significantly outperform free-form CoT. This tests whether prompt-time format… See the full description on the dataset page: https://huggingface.co/datasets/reasoning-degeneration-dev/t1-structured-reasoning-full-together_ai-zai-org-glm-5-16c46443.
t1-structured-reasoning-full-together_ai-zai-org-glm-5-16c46443
Structured reasoning evaluation: instead of injecting synthesized facts, this uses a static system prompt that teaches the model a heuristic search FORMAT with explicit structural markers ([STEP], [PRUNE], [BACKTRACK], [REVIEW OPTIONS], [SOLUTION FOUND]).
Inspired by HandCraftedCountdownSearch — models SFT'd on structured search traces significantly outperform free-form CoT. This tests whether prompt-time format instruction alone can capture some of that benefit.
Performance Comparison
Experimental Details
System Prompt Used (full style)
You are solving a countdown arithmetic problem. You must find an expression using the given numbers (each exactly once) that equals the target.
REASON USING STRUCTURED HEURISTIC SEARCH. Do not just guess — systematically search for the answer using the format below. This is critical: use the exact markers shown to organize your reasoning.
=== REASONING FORMAT ===
Use these markers to structure every step of your thinking:
[STEP N: COMBINE] — Try combining two numbers with an operation. Show: a OP b = result. State remaining numbers and how far from target.
[PRUNE] — BEFORE trying an operation, check if it's useful. Skip it if:
- Division doesn't produce an integer
- The result moves AWAY from target (current < target? don't subtract/divide. current > target? don't add/multiply)
Say what you're skipping and why.
[BACKTRACK] — Current path is stuck or getting worse. State why (too high? too low? no numbers left?) and go back to try a different combination.
[REVIEW OPTIONS] — When stuck, list ALL remaining possible combinations, score each by |result - target|, and pick the best one to try next.
[SOLUTION FOUND] — You hit the target! Verify: recompute the expression, confirm each number is used exactly once, then give the answer.
=== KEY HEURISTIC ===
DIRECTIONAL SEARCH: At every step, check if your current value is above or below the target.
- Below target → only try + and * (go UP)
- Above target → only try - and / (go DOWN)
This cuts your search space in half and prevents wasted exploration.
=== WORKED EXAMPLE ===
Target: 23, Numbers: [4, 7, 3]
[STEP 1: COMBINE]
Let me scan first moves. Best candidates by closeness to 23:
4 * 7 = 28 (distance 5) ← closest
4 + 7 = 11 (distance 12)
7 * 3 = 21 (distance 2) ← very close!
7 + 3 = 10 (distance 13)
Try: 7 * 3 = 21. Remaining: [4]. Current: 21, need 23.
[STEP 2: COMBINE]
Current 21 < target 23, so only try + and *.
21 + 4 = 25 (distance 2)
21 * 4 = 84 (distance 61)
[PRUNE] 21 * 4 = 84 — way t...Dataset Info
- Rows: Structured reasoning evaluation results (one row per problem)
- Columns: question, metadata, tasksource, formattedprompt, response, eval_correct, and associated metadata
Usage
from datasets import load_dataset
dataset = load_dataset("reasoning-degeneration-dev/t1-structured-reasoning-full-together_ai-zai-org-glm-5-16c46443", split="train")
print(f"Loaded {len(dataset)} rows")This dataset is tracked in [reasoning-degeneration-dev/PROJECT-MANIFEST](https://huggingface.co/datasets/reasoning-degeneration-dev/PROJECT-MANIFEST)