gdiamos/eda-blocks-p250
eda-blocks-p250 — C repair decomposed into a plan and function blocks 250 failing C programs, each turned into a multi-turn conversation: first a plan in English, then one turn per function body. Derived from Sudnya/classic-eda-c-trajectories (Apache-2.0). This is the decomposition, not the original corpus. What a record looks like user: <failure classification> + summarised test report + the program's structure with bodies omitted +… See the full description on the dataset page: https://huggingface.co/datasets/gdiamos/eda-blocks-p250.
eda-blocks-p250 — C repair decomposed into a plan and function blocks
250 failing C programs, each turned into a multi-turn conversation: first a plan in English, then one turn per function body.
Derived from `Sudnya/classic-eda-c-trajectories` (Apache-2.0). This is the decomposition, not the original corpus.
What a record looks like
user: <failure classification> + summarised test report
- the program's structure with bodies omitted
- "First write the PLAN ... No code." assistant: First, include "nrl.h" and "nrllib.h". Second, define a struct type `md5t
. Third, writevoid initcrc()`. ... user: Now write `void initcrc(). assistant:``c void init_crc() { ... }
`preamble`, `blocks` and `tail` are also stored, and satisfy
preamble + sum(lead + text for each block) + tail == source
**byte for byte** — so a generated block can be substituted back into the real
program and the whole file compiled. That is what this decomposition is for.
## Contents
| | |
|---|---|
| records | 250 |
| block turns | 757 |
| plan turns | 250 |
| total turns | 2,022 |
| approx. tokens | 307k |
## How it was built
`edabench/build_blocks.py` with `--per-record 1 --max-block-tok 800
--summarize-report --skeleton --require-compile`. Each flag removes a defect
found by measurement:
- **`--per-record 1`** — a `record_id` spans ~5.6 turns, each a different repair
attempt. Keeping more than one produced contradictory targets: the same
prompt with different answers, 67% of block turns in the first build. That is
an irreducible loss floor, not a training problem.
- **`--require-compile`** — the source corpus contains truncated responses
(one ends mid-statement at `else if (field == 5)`). A truncated program still
decomposes, so it silently becomes a training target. Records whose source
does not compile are dropped.
- **`--summarize-report`** — the test report is kept as one line per failing
test (name + first difference) instead of full expected-vs-got stdout.
Truncating by line count instead would drop whole tests off the end, and the
set of failing tests is part of what defines the problem.
- **`--skeleton`** — the prompt carries includes, typedefs, macros and bare
signatures rather than full bodies.
## The caveat that matters
**A model fine-tuned on this does not learn to repair unseen C.** Measured
across ~20 training arms on this decomposition: **3,472 held-out blocks were
generated and compiled; 27 were structurally non-degenerate and exactly one
was correct** (a `get_domain` differing from the target only by a parameter
name). Held-out plans keep their structure perfectly and fabricate every
identifier — and 11 of 13 identifiers a model needed were verbatim in its own
prompt, inside its attention window. The bottleneck measured is retrieval, not
capacity or data volume.
Held-out loss scales as `programs^-0.213`, so every 10x of data buys about
1.4x of held-out output rate. Two axes govern the result and are close to
independent: **fit** (how well the model reproduces its own targets) sets what
it does on material it has seen; **program count** sets what transfers.
Published so those measurements are reproducible, not because the dataset
teaches the task.