tawer12/qwen3-4b-recursive-sft-v3.1-eval-rollouts

Recursive V3.1 SFT Evaluation Rollouts

Saved evaluation generations from tawer12/qwen3-4b-recursive-sft-v3.1. This dataset contains SFT-model evaluations only, not RL training rollouts, flattened-model generations, or the SFT training corpus.

The release preserves all original per-rollout fields and text, including invalid trees and wrong answers. Added fields identify the benchmark, attempt, model, and source file/line. No generations or answer labels were repaired.

Contents

Total: 6,827 rollout records. The algebra questions overlap the algebra portion of MATH5000; they are a separate evaluation, not 1,187 additional independent benchmark questions. Do not pool these configurations to report a single benchmark accuracy.

from datasets import load_dataset

repo = "tawer12/qwen3-4b-recursive-sft-v3.1-eval-rollouts"
math = load_dataset(repo, "math5000_greedy", split="test")
amc = load_dataset(repo, "amc23_t0p5_k16", split="test")
print(math[0]["full_trace"])
print(math[0]["calls"][0]["output_text"])

Important fields

• —id: original evaluation question ID, represented as a string. IDs are local to each benchmark; AMC has 16 rows for each ID.
• —problem, ground_truth: benchmark question and stored reference answer.
• —predicted_answer: the historical engine's extracted answer, possibly null or damaged by normalization. Consult the actual terminal action text.
• —correct: historical recorded correctness, not a newly verified label.
• —parsed_ok, parse_errors, stopped_reason: historical execution status.
• —full_trace: complete rendered recursive trace, including failed calls.
• —calls: list of actual model calls with role, depth, problem, input/output text, token counts, parsed action, and parser error.
• —model_call_count: actual number of model-generation calls in the rollout.
• —critical_path_tokens: historical token-based estimate, not elapsed time.
• —seed, temperature, top_p, top_k, forced_root_action: saved decoding settings. A null top-k means unspecified, not top-k disabled.
• —benchmark, attempt_index, model_repo_id: added release identifiers. AMC attempt_index is zero-based (0-15); greedy attempts use 0.
• —source_file, source_line: relative original file and one-based JSONL line.

AMC decoding used seeds 5000-5015. All configurations used 2,048 maximum new tokens per model call, maximum recursion depth 3, and maximum 10 calls per tree. The raw recursive prompt and release-time engine source are in the model repo.

Metrics and known scoring limitations

metrics/ contains historical aggregate metrics with machine-local paths replaced by portable references. evaluation_manifest.json records settings; source_manifest.json records the source JSONL checksums and row counts.

Historical values are AMC mean@16 208/640 (32.50%), AMC pass@16 32/40 (80.00%), MATH5000 2531/5000 (50.62%), and the separate algebra run 839/1187 (70.68%). Mean@16 averages the correctness flags over all attempts; pass@16 measures the fraction of questions with at least one recorded correct attempt among the 16. The single greedy attempt is not a sampled pass@k estimate with multiple draws.

These historical scores have known grading defects.

1. 1.A parse failure forces historical recursive correct=false, even when the generated final answer may be mathematically right.
2. 2.Equivalent fraction, radical, or vector forms can be marked wrong. For example, MATH ID 2032 gives 1/2 for reference \frac{1}{2}; ID 4752 gives the correct vector components -6, 6 but the stored prediction is -66.
3. 3.Removing the parse gate and extracting from the entire trace is not a sufficient fix: extraction can accidentally select an intermediate child answer or a number from an unfinished generation.

Keep answer correctness, protocol validity, and completion as separate measurements, and use a consistent terminal-root-answer mathematical grader before drawing quantitative comparisons. The released rows deliberately keep the original flags rather than silently changing historical results. Format validity does not verify mathematical correctness or independence of child tasks. Model-call count is not the number of successful or correct calls. No RL rewards were observed in these SFT evaluations.

Provenance and rights

• —MATH5000 and algebra derive from EleutherAI/hendrycks_math, the Hugging Face packaging of the MATH benchmark. The upstream Hub card lists an MIT license. MATH5000 is the full test set, not MATH-500.
• —AMC23 consists of the project's 40-question local AMC23 benchmark file, associated with the 2023 American Mathematics Competitions. This release does not claim authorship of those questions or grant new rights over them.
• —Responses were generated by the linked recursive V3.1 SFT model. The original evaluation run identifier is v31-sft-official-20260820-004242.

This is an evaluation-output collection, not a relicensing of third-party benchmark questions. Underlying source terms continue to apply; no blanket license is asserted for the combined benchmark contents. Training on these held-out questions or rollouts would contaminate subsequent evaluations on the same benchmarks.