benchpress/results-multi-temp-seed

Benchpress Evaluation Results

This dataset contains normalized outputs from lm-evaluation-harness runs for the Benchpress project. It is intended for analysis of post-training recipe behavior across models, benchmarks, temperatures, and random seeds.

Coverage

• —Runs: 110
• —Model/stage recipes: olmo2-13b:dpo, olmo2-13b:rlvr, olmo2-13b:sft, olmo2-7b:dpo, olmo2-7b:rlvr, olmo2-7b:sft, olmo3-7b:dpo, olmo3-7b:rlvr, olmo3-7b:sft, smollm3-3b:apo, smollm3-3b:sft
• —Temperatures: 0.0, 0.4, 0.7, 1.0
• —Seeds: 0, 1, 2
• —Tasks: 30
• —Benchmark groups: arc, gsm8kcot, hellaswag, humanevalinstruct, ifeval, mgsmencot, truthfulqa_mc2
• —Languages/settings: ar, bn, de, en, enchallenge, eneasy, es, fr, hi, ja, ru, sw, te, th, zh

Important: benchmark questions/prompts are not included

To avoid redistributing third-party benchmark content under licenses we cannot relicense (notably the multilingual Okapi subsets, which are CC BY-NC 4.0), this release does not contain the verbatim prompt/question text. The sample_scores split has no doc, arguments, or full-sample columns.

What we keep is our own or derivable: model responses (resps_json), the ground-truth target, scalar metrics and scores, and the stable doc_hash / prompt_hash identifiers emitted by lm-eval. Those hashes let you re-join these results to the original questions by loading the upstream datasets yourself (under their own licenses) and hashing their docs/prompts the same way. See the attribution table below for the exact upstream dataset for each benchmark group.

Note: these are lm-eval's own doc_hash / prompt_hash values, not a plain SHA-256 of the text. Reproducing them to re-join requires the same lm-evaluation-harness version and task configuration used here (see the lm_eval_version field in the runs split).

Temperature/Seed Design

• —Temperature 0.0 is evaluated once per recipe with seed(s): 0.
• —Sampled temperatures 0.4, 0.7, 1.0 are evaluated with seeds 0, 1, 2.

Splits

• —runs: 110 rows
• —aggregates: 12100 rows
• —sample_scores: 7511570 rows

Split Contents

• —runs: one row per completed eval run/cell. The canonical run table: cellslug, model, stage, stagegroup, checkpoint repo, revision, temperature, seed, generation kwargs, and tool versions.
• —aggregates: task-level metrics per run. Slim provenance (model, stage, seed, temperature) plus task/benchmark/language/ metric/filter/value. Join to runs on those four keys for the rest.
• —sample_scores: one row per evaluated sample/filter. Slim provenance plus doc_id, doc_hash, prompt_hash, target, primary metric/score, scalar metrics, and model resps_json. No verbatim prompt/question columns (see above).

What Is Not Included

• —Model weights are not included.
• —Verbatim benchmark questions/prompts are not included (see above).
• —Denormalized run metadata is not duplicated onto every aggregates / sample_scores row; join to the runs split on (model, stage, seed, temperature) instead.
• —Raw JSON/JSONL/log files and file inventories are not included.

Attribution and Licensing of Source Benchmarks

These results were produced by evaluating the models below on the following public benchmarks via lm-evaluation-harness. Each benchmark remains under its own license; the table records the exact upstream Hugging Face dataset and its license. Note the multilingual subsets are non-commercial (CC BY-NC 4.0).

Benchmark citations

• —arc (allenai/ai2_arc): Clark et al., 2018, Think you have Solved Question Answering? Try ARC (arXiv:1803.05457).
• —arc (alexandrainst/m_arc): Lai et al., 2023, Okapi: Instruction-tuned LLMs in Multiple Languages with RLHF (arXiv:2307.16039).
• —hellaswag (Rowan/hellaswag): Zellers et al., 2019, HellaSwag: Can a Machine Really Finish Your Sentence? (arXiv:1905.07830).
• —hellaswag (alexandrainst/m_hellaswag): Lai et al., 2023, Okapi (arXiv:2307.16039).
• —truthfulqa_mc2 (truthfulqa/truthful_qa): Lin et al., 2022, TruthfulQA: Measuring How Models Mimic Human Falsehoods (arXiv:2109.07958).
• —truthfulqa_mc2 (alexandrainst/m_truthfulqa): Lai et al., 2023, Okapi (arXiv:2307.16039).
• —gsm8k_cot (openai/gsm8k): Cobbe et al., 2021, Training Verifiers to Solve Math Word Problems (arXiv:2110.14168).
• —mgsm_en_cot (juletxara/mgsm): Shi et al., 2022, Language Models are Multilingual Chain-of-Thought Reasoners (arXiv:2210.03057); derived from GSM8K (Cobbe et al., 2021).
• —ifeval (google/IFEval): Zhou et al., 2023, Instruction-Following Evaluation for Large Language Models (arXiv:2311.07911).
• —humaneval_instruct (openai/openai_humaneval): Chen et al., 2021, Evaluating Large Language Models Trained on Code (arXiv:2107.03374).

Models Evaluated

All evaluated checkpoints are publicly available on the Hugging Face Hub under permissive (Apache-2.0) licenses. Please cite the corresponding model papers/releases when using these results.

• —SmolLM3 (3B) — HuggingFaceTB/SmolLM3-3B (Apache-2.0). Hugging Face, 2025, SmolLM3: smol, multilingual, long-context reasoner (https://hf.co/blog/smollm3).
• —OLMo 2 (7B, 13B) — allenai/OLMo-2-1124-7B-{SFT,DPO,Instruct}, allenai/OLMo-2-1124-13B-{SFT,DPO,Instruct} (Apache-2.0). OLMo Team, 2025, 2 OLMo 2 Furious (arXiv:2501.00656).
• —OLMo 3 (7B) — allenai/Olmo-3-7B-{SFT,DPO,Instruct} (Apache-2.0). Allen Institute for AI, 2025, Olmo 3 (arXiv:2512.13961).

Exact checkpoint repositories evaluated:

• —`HuggingFaceTB/SmolLM3-3B-checkpoints`
• —`allenai/OLMo-2-1124-13B-DPO`
• —`allenai/OLMo-2-1124-13B-Instruct`
• —`allenai/OLMo-2-1124-13B-SFT`
• —`allenai/OLMo-2-1124-7B-DPO`
• —`allenai/OLMo-2-1124-7B-Instruct`
• —`allenai/OLMo-2-1124-7B-SFT`
• —`allenai/Olmo-3-7B-Instruct`
• —`allenai/Olmo-3-7B-Instruct-DPO`
• —`allenai/Olmo-3-7B-Instruct-SFT`

Loading

from datasets import load_dataset

ds = load_dataset('parquet', data_files={
    'runs': 'runs.parquet',
    'aggregates': 'aggregates.parquet',
    'sample_scores': 'sample_scores.parquet',
})