YWZBrandon/officeqa-checkpoint-eval-data

Checkpoint evaluation plot data

Snapshot: 2026-09-14T16:26:45.684890+00:00. Aggregate inputs to notes/Sept-2-2026.md performance figures. No model execution, grading, publication, or source-result changes were performed to make this export.

Contents

• —checkpoint_evaluations: 454 checkpoint rows, one evaluation per run/iteration/protocol; score, mean output tokens, mean steps, and the existing two-sided 95% confidence bounds.
• —pareto_points: current mean-token/USD Pareto manifest points, including v1/v2 and best-checkpoint comparisons. Rounded display values and unrounded source means are separate. USD mean/median estimates use the saved price card, not current API prices.
• —references: teacher/SFT/base horizontal references and shared step-0 anchors, separate from trained checkpoints. Dynamic JobBench/9B baselines were captured read-only on 2026-09-14; their snapshot is included. Conditional fallbacks are explicitly marked.
• —in_training_series: raw source values for the still-displayed training curves, separate from offline evaluations. Includes auxiliary columns, not just visible axes. Apply the original plotter's column selection and smoothing to reproduce those layers.
• —raw_offline_series: original source-column/value tuples, before the SSB token-CI merge. TSV input snapshots and sanitized Pareto manifests are under provenance/.
• —manifest.json: source hashes/timestamps, per-run iterations, interval coverage, and omissions.

Both JSONL and CSV versions are supplied. Hugging Face configurations load only JSONL, avoiding accidental duplication from the CSV copies. CSV nested fields are JSON strings; empty CSV fields correspond to JSON null.

Protocol and metric definitions

OfficeQA v1 checkpoints: 133 held-out tasks, 8 attempts/task, temperature 1, cap 120. SSB1 checkpoints: Verified-400, 8 attempts/task, temperature 1, cap 50. JobBench checkpoints: main-trained, 63 easy eval tasks, 8 attempts/task, temperature 1, cap 120, search files staged. Protocol counts are declared expectations, not a new census of files on all nodes.

Eight attempts is not the plotted pass@8 score. OfficeQA score_mean is average pass@1 estimated from the attempts; SSB1 uses hard-restriction pass@1. JobBench uses mean normalized rubric score, not pass@k. SSB2 Pareto points use target-cell accuracy (cell_accuracy), not strict task pass@1.

All normalized token fields are tokens per episode, not thousands; source measured_tokens/offline_tokens values and their bounds were multiplied by 1,000. Step fields are per episode. Output-token costs include failed attempts as in the original plot data. No new filtering was introduced.

Confidence intervals

• —OfficeQA and SSB1 score: Wilson, z=1.96, over attempts (not task-cluster adjusted).
• —OfficeQA token/step intervals, where present: task-cluster percentile bootstrap, 10000 draws, 95%.
• —SSB1 token/step intervals: normal interval over task means, mean +/- 1.96*SE, 95%.
• —JobBench score/token/step intervals: task-cluster percentile bootstrap, 10000 draws, 95%.

Checkpoint bounds are copied from the actual source series, not recomputed or fabricated. Dynamic baseline bounds use the original plot's estimators/caches, as documented in their captured snapshot. Missing intervals are null, with *_ci_status=not_in_plot_source; this is not a zero-width interval. There are no Pareto confidence intervals in the source manifest. Source bounds failing to bracket a central estimate are flagged rather than silently changed. Label-combination figures can omit rendering intervals that remain available in their source series.

Wilson-over-attempt intervals do not account for correlation of attempts on the same task; they are retained for exact reproducibility, not endorsed as task-population uncertainty. Task-cluster intervals resample or summarize task means. Do not treat all interval methods as equivalent.

SSB1 4B merges token means and bounds from the dedicated CI file, matching its plotter. SSB1 9B/27B merge bounds only, and a cached interval cannot create a new finalized checkpoint. A shared step-0 anchor must not be counted once per arm as independent evidence. The SSB1 4B step-0 reference is a legacy one-attempt evaluation.

Scope and limitations

This is a snapshot of aggregate plotting inputs, not an independently revalidated rollout archive. Ongoing/incomplete sweeps contain only their published checkpoints; absence does not mean zero score. Old cap-50/one-attempt series are excluded from the controlled table. The currently displayed Gemma hill climb is in-training-only and therefore appears in the separate training-source table, not as a fabricated offline sweep.

Pareto manifests can lag the live checkpoint series and can contain an independent rerun of the same iteration. Their source hashes and display precision are preserved. They are not joined by iteration alone to borrow a confidence interval from a different evaluation. Supplemental best-checkpoint manifests include median token statistics and selection provenance.

Only aggregate metrics and sanitized provenance are included: no task documents, prompts, credentials, or rollout content. No dataset license is asserted. Review benchmark/model terms and choose an appropriate license before publishing.

Upload and load

Upload the contents of this directory (including this README and data/) to your own Hugging Face dataset repository. Do not upload only the ZIP if you want Dataset Viewer support. Nothing has been uploaded automatically.

from datasets import load_dataset
df = load_dataset("YOUR_ACCOUNT/YOUR_DATASET", "checkpoint_evaluations", split="test").to_pandas()
officeqa = df[df.dataset == "officeqa"]

For local use without Hugging Face dependencies:

import pandas as pd
df = pd.read_json("data/checkpoint_evaluations.jsonl", lines=True)
# Example asymmetric error bars: [mean - low, high - mean]. Drop missing CIs first.
valid = df[df.output_tokens_ci_status == "available"]

The test split is an export partition; actual benchmark splits are recorded in each row. See Hugging Face dataset configurations.