reasoning-degeneration-dev/t1-ablation-facts-questions-only-t1-base-eval-hosted_vllm-qwen-qwen3-1-7b-5arg

{
  "script_name": "02b_synthesize_knowledge_ablation.py",
  "model": "gpt-5-mini",
  "hyperparameters": {
    "reclm_backend": "openai",
    "num_facts": 15
  },
  "input_datasets": [
    "t1-base-eval-hosted_vllm-qwen-qwen3-1-7b-5arg"
  ],
  "description": "ABLATION: Synthesized 15 domain knowledge facts from questions only (no model traces or correctness) using gpt-5-mini via RecLM. Source dataset had 100 problems.",
  "custom_metadata": {
    "experiment_name": "t1_synthesize_knowledge_improvement",
    "stage": "knowledge_synthesis_ablation",
    "ablation_type": "questions_only_no_traces",
    "synthesis_prompt_length": 23093,
    "source_total_problems": 100,
    "facts": [
      "Remember the rules: you must use every given number exactly once, and you may only combine numbers using the four basic operations by grouping them with parentheses as needed; treat the problem as reducing the multiset of numbers to a single value that equals the target.",
      "Represent search states as the multiset of remaining values plus their exact rational representations; at each step pick two values, apply one operation, replace the pair with the result, and continue until one value remains.",
      "Always do arithmetic with exact rationals (store numerator and denominator and reduce by greatest common divisor) to avoid floating-point rounding errors and to ensure you do not miss exact solutions that pass through fractional intermediates.",
      "Canonicalize intermediate states to avoid duplicate work: sort the multiset of remaining values into a canonical order and store reduced fraction forms so that equivalent states map to the same key.",
      "Memoize explored canonical states and the set of values already achieved from them; prune any branch when you revisit a canonical state that has already produced the same value or a superset of values.",
      "Use meet-in-the-middle as a powerful technique: split the numbers into two groups, enumerate all reachable results for each group, and then search for pairs of partial results that can be combined to reach the target, which cuts complexity exponentially.",
      "Compute optimistic and pessimistic bounds from the remaining numbers to prune early: estimate the maximum and minimum achievable values under the best plausible operator choices and discard branches where the target cannot lie within those bounds.",
      "Break symmetry aggressively: treat identical numbers as indistinguishable during pairing choices, and for commutative operations choose a canonical ordering of operands so you do not explore mirrored permutations.",
      "Prioritize operations that move you toward the target magnitude: favor multiplication or division when you need to change magnitude quickly, and favor addition or subtraction when you need fine adjustments; use this as a heuristic ordering rather than a hard rule.",
      "For algorithmic search use informed strategies: best-first search or A-star with a heuristic measuring the minimal possible distance to the target based on remaining numbers often finds solutions faster than blind depth-first search.",
      "Avoid common pitfalls: do not use approximate floating arithmetic, do not accidentally reuse or drop a number, and do not prune branches solely because they produce non-integer intermediates since exact cancellation can still produce the integer target later.",
      "Performance tricks: only consider unique unordered pairs at each step, ignore operations that produce exact duplicates of existing elements in the same multiset, and apply iterative deepening or progressively stronger heuristics to find solutions quickly with bounded resources.",
      "Human-style tactics and templates to try early: look for ways to form a large anchor value or a value that shares factors with the target, then use remaining numbers to adjust by small increments; attempt to create factor relationships or near-target anchors and refine them.",
      "Micro-strategies to sketch mentally: aim to produce an intermediate that is 'close in scale' to the target and then tweak with addition or subtraction; alternatively, produce a partial value that is a divisor or multiple of the target and combine remaining numbers to scale it; think in terms of anchoring then adjusting.",
      "Quick troubleshooting checklist for any single problem: (1) scan for obvious factor or round-number patterns, (2) try one-pass pairwise reductions that make large anchors, (3) if stuck, apply meet-in-the-middle on a split, (4) keep exact rationals throughout, (5) prune using bounds and symmetry rules, (6) backtrack and relax heuristic priorities if no solution is found."
    ],
    "raw_response": "[\n  \"Remember the rules: you must use every given number exactly once, and you may only combine numbers using the four basic operations by grouping them with parentheses as needed; treat the problem as reducing the multiset of numbers to a single value that equals the target.\",\n  \"Represent search states as the multiset of remaining values plus their exact rational representations; at each step pick two values, apply one operation, replace the pair with the result, and continue until one value remains.\",\n  \"Always do arithmetic with exact rationals (store numerator and denominator and reduce by greatest common divisor) to avoid floating-point rounding errors and to ensure you do not miss exact solutions that pass through fractional intermediates.\",\n  \"Canonicalize intermediate states to avoid duplicate work: sort the multiset of remaining values into a canonical order and store reduced fraction forms so that equivalent states map to the same key.\",\n  \"Memoize explored canonical states and the set of values already achieved from them; prune any branch when you revisit a canonical state that has already produced the same value or a superset of values.\",\n  \"Use meet-in-the-middle as a powerful technique: split the numbers into two groups, enumerate all reachable results for each group, and then search for pairs of partial results that can be combined to reach the target, which cuts complexity exponentially.\",\n  \"Compute optimistic and pessimistic bounds from the remaining numbers to prune early: estimate the maximum and minimum achievable values under the best plausible operator choices and discard branches where the target cannot lie within those bounds.\",\n  \"Break symmetry aggressively: treat identical numbers as indistinguishable during pairing choices, and for commutative operations choose a canonical ordering of operands so you do not explore mirrored permutations.\",\n  \"Prioritize operations that move you toward the target magnitude: favor multiplication or division when you need to change magnitude quickly, and favor addition or subtraction when you need fine adjustments; use this as a heuristic ordering rather than a hard rule.\",\n  \"For algorithmic search use informed strategies: best-first search or A-star with a heuristic measuring the minimal possible distance to the target based on remaining numbers often finds solutions faster than blind depth-first search.\",\n  \"Avoid common pitfalls: do not use approximate floating arithmetic, do not accidentally reuse or drop a number, and do not prune branches solely because they produce non-integer intermediates since exact cancellation can still produce the integer target later.\",\n  \"Performance tricks: only consider unique unordered pairs at each step, ignore operations that produce exact duplicates of existing elements in the same multiset, and apply iterative deepening or progressively stronger heuristics to find solutions quickly with bounded resources.\",\n  \"Human-style tactics and templates to try early: look for ways to form a large anchor value or a value that shares factors with the target, then use remaining numbers to adjust by small increments; attempt to create factor relationships or near-target anchors and refine them.\",\n  \"Micro-strategies to sketch mentally: aim to produce an intermediate that is 'close in scale' to the target and then tweak with addition or subtraction; alternatively, produce a partial value that is a divisor or multiple of the target and combine remaining numbers to scale it; think in terms of anchoring then adjusting.\",\n  \"Quick troubleshooting checklist for any single problem: (1) scan for obvious factor or round-number patterns, (2) try one-pass pairwise reductions that make large anchors, (3) if stuck, apply meet-in-the-middle on a split, (4) keep exact rationals throughout, (5) prune using bounds and symmetry rules, (6) backtrack and relax heuristic priorities if no solution is found.\"\n]"
  }
}

from datasets import load_dataset

dataset = load_dataset("reasoning-degeneration-dev/t1-ablation-facts-questions-only-t1-base-eval-hosted_vllm-qwen-qwen3-1-7b-5arg", split="train")
print(f"Loaded {len(dataset)} rows")