YWZBrandon/final-best-raw-episodes-2026-09-14-v2

Final and canonical-best evaluation episodes: frozen preparation

Full local packaging is now running. See materialization status and instructions. This preparation folder is not the full payload; the separate full export remains incomplete until its verified COMPLETED marker is written.

Prepared inventory: 72 evaluations / 119,608 expected episodes. There are 36 canonical-best and 44 final memberships, with 8 evaluations tagged both. One incomplete Q38-teacher OfficeQA v2 final is explicitly withheld. The plan covers the controlled 32-arm registry, canonical winner manifests and relevant current final Pareto reruns, not every historical evaluation.

Measured inventory and size estimate covers all 36 canonical-best sources: 58,848 files, 1,210.286 GiB raw (final-only sources add more). Four samples per evaluation suggest about 67.9 GiB at gzip level 6; the exporter uses faster level 1, so actual size differs. Actual one-episode sample report: 30,945,629 source bytes became 2,574,344 compressed bytes in 2.05 seconds, with no credential-pattern matches. This local single-file timing is not an end-to-end shared-storage throughput guarantee; full processing can take many hours.

This is a local preparation, not a completed full raw export or a publication. The default Hugging Face configuration is a lightweight evaluation index only. manifest.json records exact selections, source summary SHA256s, missing results, partial sweeps, and omissions. The source/protocol snapshot is under provenance/.

Best uses the saved v1 selections, with their original accuracy/token/step tie breaks. V2 never selects a checkpoint. Final means the declared terminal of the frozen sweep (139 OfficeQA, 154 SSB1, 79 JobBench), not the latest finished eval. Independent reruns used by different plots remain distinct. A single evaluation selected as both final and best is stored once with checkpoint_role=both. SFT bests use the canonical best-completed-evaluation rule; no SFT final is claimed without a independently verified terminal. In-training-only arms have no raw controlled offline evaluation to export. See the explicit omissions in manifest.

Materialize locally

python3 export_final_best_raw_episodes.py export --plan . --output /absolute/new/output

For an interruption-tolerant full run, use the sequential per-evaluation batch:

nice -n 15 python3 export_final_best_raw_episodes.py batch --plan . --output /absolute/new/batch-output

Rerun the same batch command to resume: completed evaluation directories are skipped only after their completion marker, frozen-plan identity, shard hashes, index hash and row count verify. Failed attempts are preserved; a retry uses a new numbered directory. A credential or other error stops the batch immediately; no automatic secret-failure retry is performed. Exclude failed attempts from any eventual upload. The batch writes a progress/status report and remains marked NOT_UPLOAD_READY until all requested evaluation exports complete. Automated completion still does not replace publication/data-rights review.

Use --evaluation-id ID (repeatable), --max-episodes 2, and --max-raw-bytes 536870912 for a bounded sample. Outputs refuse overwrites. The exporter reads one episode at a time, checks stable source bytes, parses the original JSON, scans credential patterns, and writes gzip JSONL shards plus a lightweight per-episode index. No shell commands in a trace are executed. Missing, changed, malformed, and credential-flagged payloads fail closed: the export remains incomplete, with no COMPLETED marker. Original files are never changed. Pattern scanning is a limited automated audit, not a guarantee of safety. Review data rights, prompt/document contents, personal data and secrets before any public upload; no license grant or automatic upload is provided here.

raw_episode_json is an exact UTF-8 string of the source file: recover original bytes with .encode("utf-8") and verify source_sha256. Messages, reasoning, commands, rewards, usage, repeated histories and metadata are not stripped or normalized. Binaries are not copied; their original workspace/output references remain in the JSON. This is a traces dataset, not a runnable workspace archive.

Raw rows can exceed 248 MiB. Do not load all raw rows into a Dataset Viewer or an in-memory dataframe. Download individual shards and iterate one row at a time; even streaming needs memory for the largest row (the exporter can use several times that size). The default index does not load raw shards. Full materialization is an optional long-running I/O/compression command, not run during this bounded preparation. The test split is an export partition, not a new benchmark split; actual evaluation split is a scalar column.

The previous aggregate export checkpoint-eval-data-2026-09-14 contains plot CIs; join its checkpoint rows via aggregate_evaluation_id, checking source hashes and protocol before treating independent reruns as identical measurements.