BlazingCustoms/pybytecode-mbpp-3.12

1---2license: cc-by-4.03task_categories:4  - text-generation5language:6  - en7tags:8  - decompilation9  - python10  - bytecode11  - code12  - mbpp13size_categories:14  - n<1K15configs:16  - config_name: default17    data_files: bench.jsonl18---19 20# Data card — MBPP held-out decompilation benchmark21 22383 MBPP reference solutions compiled to Python 3.12 bytecode and paired with their source, with23the original `task_id` restored on every row. Built 2026-08-04 by `tools/build_mbpp_ood.py`.24 25**Redistributable** under CC-BY-4.0, provided `NOTICES.md` ships alongside.26 27---28 29## The set30 31| | |32|---|---|33| Rows | **383** (400 candidates − 17 contaminated) |34| Source | `google-research-datasets/mbpp`, config `full` |35| Licence | **CC-BY-4.0** |36| `task_id` restored | **383 / 383** (all distinct, range 11–973) |37| Untraceable rows dropped | 0 |38| **Contaminated rows dropped** | **17** (see below) |39| Disassembly length | mean 47.5 lines, median 41, max 187 |40| Python | 3.12, `optimize=0` |41 42### In what sense this set is independent of training — stated precisely, not as an adjective43 44It is a **different corpus** (`google-research-datasets/mbpp`, not45`codeparrot/github-code-clean`) and a **different kind of code**: short, self-contained,46hand-written answers to stated problems, against the training trunk's real project code with its47framework coupling and helper chains. Both differences are verifiable from row provenance.48 49That matters because the pilot's failure mode was a model learning the *source of the data*50rather than the task, and a test set drawn from the training source cannot detect it.51 52It is **not** described as out-of-distribution anywhere, and no divergence statistic was computed,53so no distributional-shift claim is made. The defensible claim is exactly three things:54different source, different task style, and the verified non-overlap below.55 56### Contamination — the defect that cost 17 rows57 58**This set was originally decontaminated against the wrong corpus.** `build_ood_mbpp.py` checked59against the v1-era 10k training labels, but the shipping model (v3) trained on 48,196 rows.60Re-checked against the corpus v3 actually trained on:61 62| gate | result |63|---|---|64| exact canonical match vs v3 corpus | 0 / 400 |65| **identifier-blind fingerprint vs v3 corpus** | **17 / 400 (4.25%)** |66 67Those 17 are dropped. They are structural collisions rather than copied code — the fingerprint68erases literals, so `re.sub(' +', ' ', t)` and `re.sub('[- ()]', '', s)` collide despite computing69different things. The gate is applied anyway: it is the same gate `build_final.py` and the CSN70builder apply, and arguing with it case by case is precisely how the earlier provenance defects71happened.72 73After rebuild: **0 / 383** contamination against the v3 corpus, by either gate.74 75## Task ids restored76 77`build_ood_mbpp.py` wrote only `{input, expected}`, so no row could be traced to its task, prompt78or tests. Recovery is exact rather than approximate: `expected == canonicalise(mbpp["code"])`, a79deterministic transform, so rebuilding the map over all 974 MBPP rows (974 rows → 965 distinct80canonical forms) recovers the id by lookup. All 400 rows matched; the builder refuses to emit a81row it cannot trace.82 83Each row now carries `provenance.task_id`, `mbpp_split`, the task `prompt`, and the `test_list`.84The tests travel with the row, so a behavioural check is possible on this set without going back85to the dataset.86 87## Decontamination88 89Two stages. At original build time, against the 10k training labels of that era, by exact90canonical match and by identifier-blind fingerprint (which caught 15 items exact match missed).91At rebuild, against the **48,196-row corpus the shipping model actually trained on**, removing a92further 17 rows as above. Rows not adjudicable by the differential-execution oracle were excluded93at original build time, so every row here was labelled by running it.94 95## Changes made to the original, as CC-BY-4.0 §3(a)(1)(B) requires be indicated96 97Each reference solution was rewritten through `ast.unparse` — normalising away formatting,98comments and redundant parentheses — then compiled to a `.pyc` at `optimize=0` and paired with a99disassembly of the resulting code object. Task text and tests are unmodified.100 101**That rewrite changes the compiled bytecode on 1 of the 383 rows.** CPython 3.12 inlines102comprehensions (PEP 709) and emits a cleanup entry in the exception table for the inlined scope;103how the comprehension is *line-wrapped* changes that entry. In `src/00117.py` (MBPP `task_id`104757) the original solution wraps a nested comprehension across lines, so its bytecode differs105from the normalised form shipped here. The instruction stream `co_code` is identical on all 383106rows; this row differs only in that exception-table entry.107 108The row is a valid decompilation task and the benchmark remains sound: for all 383 rows the109reference source and the `.pyc` are compiled from **the same** normalised string, which is the110condition the oracle needs. What the row is not is byte-identical to upstream MBPP — hence this111note. Re-deriving it from the original would put that solution's tab continuations and trailing112whitespace into the reference and make the label an unstable target, so it is documented instead.113Measured with `tools/measure_format_ceiling.py`; the general limit is in114`../../EVAL.md` §6.115 116## Harness soundness on this set117 118`grade.py --self-test-only`: pre-flight **383/383 = 100%**, mutation kill rate **199/199 = 100%**,1190 survivors. Note what pre-flight does and does not prove — see `../../ORACLE-LIMITS.md`.120 121## Read this before quoting any "OOD" figure122 123The published PyBytecode figures labelled *OOD-MBPP* — 91.04% strict greedy, 97.49% certified@32,124100% docstring recovery — **were not measured on this set.** They were measured on125the superseded held-out set (n=279), which despite its name contains no MBPP at all: it is126GitHub source from the held-out shards of the training corpus. Matching all 974 canonicalised127MBPP rows against it yields 0 hits, against 400/400 for this set. Evidence in128`../../LICENSING-DETERMINATION.md` §4.129 130Those numbers remain valid measurements of **held-out generalisation**. They are not131out-of-distribution measurements, and the label should be corrected wherever it appears.132 133**No scores have been measured on this benchmark.** Doing so requires a generation run (GPU).134 135## Limitations136 137- MBPP solutions are short and stylistically uniform. Success here says little about long or138  framework-coupled functions; the CSN set is the harder distribution.139- Rows are the first 400 adjudicable MBPP tasks in split order, minus the 17 contaminated ones. Not a random sample.140- Two MBPP tasks with identical canonical solutions collapse to one row; 974 rows yield 965141  distinct canonical forms, and the first `task_id` wins.142 143## Attribution144 145MBPP (Mostly Basic Python Problems), Austin et al., 2021, Google Research.146<https://huggingface.co/datasets/google-research-datasets/mbpp> — CC-BY-4.0147(<https://creativecommons.org/licenses/by/4.0/>).148