thealper2/codet5-base-code-repair

1---2license: bsd-3-clause3base_model: Salesforce/codet5-base4datasets:5  - google/code_x_glue_cc_code_refinement6language:7  - code8library_name: transformers9pipeline_tag: text-generation10tags:11  - code12  - codet513  - code-repair14  - program-repair15  - bug-fixing16  - java17  - seq2seq18model-index:19  - name: codet5-base-code-repair20    results:21      - task:22          type: text2text-generation23          name: Automated Program Repair24        dataset:25          name: CodeXGLUE code-refinement (small)26          type: google/code_x_glue_cc_code_refinement27          config: small28          split: test29        metrics:30          - type: exact_match31            value: 22.4332            name: Exact Match (%)33          - type: bleu34            value: 80.1335            name: BLEU36---37 38# codet5-base-code-repair39 40[Salesforce/codet5-base](https://huggingface.co/Salesforce/codet5-base) fine-tuned on the41[CodeXGLUE code-refinement](https://huggingface.co/datasets/google/code_x_glue_cc_code_refinement)42`small` split for **automated program repair**: given a buggy Java method, the model generates the43fixed version.44 45Inputs and outputs follow the dataset's abstracted Java style, where identifiers are normalised to46tokens such as `METHOD_1`, `VAR_1`, `TYPE_1` and `STRING_1`.47 48## Usage49 50```python51from transformers import AutoModelForSeq2SeqLM, AutoTokenizer52 53model_id = "MODEL_ID"  # <- repo id54tokenizer = AutoTokenizer.from_pretrained(model_id)55model = AutoModelForSeq2SeqLM.from_pretrained(model_id)56 57buggy = (58    "public int METHOD_1 ( int VAR_1 ) { if ( VAR_1 = 0 ) { return 1 ; } "59    "return ( VAR_1 * ( METHOD_1 ( ( VAR_1 - 1 ) ) ) ) ; }"60)61 62inputs = tokenizer(buggy, max_length=256, truncation=True, return_tensors="pt")63outputs = model.generate(**inputs, max_new_tokens=256, num_beams=4, early_stopping=True)64print(tokenizer.decode(outputs[0], skip_special_tokens=True))65```66 67No task prefix is needed — the buggy snippet is fed in as-is. Beam search with `num_beams=4` is the68decoding setting used for all numbers reported below, and it is already stored in the repo's69`generation_config.json`.70 71## Results72 73Full splits (5,835 examples each), beam search with 4 beams:74 75| Split      | Exact Match | BLEU  | Loss   |76|------------|-------------|-------|--------|77| Validation | 21.29%      | 80.27 | 0.1272 |78| Test       | 22.43%      | 80.13 | 0.1257 |79 80Breakdown of the test-set predictions:81 82| Outcome                                     | Share  |83|---------------------------------------------|--------|84| Exact fix                                   | 22.43% |85| Partial fix (changed, closer but not exact) | 13.49% |86| Input copied unchanged                      | 3.38%  |87| Incorrect                                   | 64.08% |88 89The high BLEU next to the modest exact-match rate is expected for this task: the fixed method is90usually a near-copy of the buggy one, so most generated tokens are correct even when the actual bug91is not fixed. **Exact match is the metric that matters here**; BLEU mostly measures how well the92model preserves the surrounding code.93 94Validation exact match by epoch (1,000-example in-training subset):95 96| Epoch | 1    | 2    | 3    | 4    | 5    | 6    | 7    | 8    | 9        | 10   |97|-------|------|------|------|------|------|------|------|------|----------|------|98| EM    | 10.5 | 15.2 | 17.5 | 19.5 | 19.8 | 20.6 | 20.9 | 20.6 | **21.2** | 21.0 |99 100The epoch-9 checkpoint scored best and is the one published here.101 102## Training103 104| | |105|---|---|106| Base model | `Salesforce/codet5-base` (~223M params) |107| Dataset | `google/code_x_glue_cc_code_refinement`, config `small` |108| Train / validation / test | 46,680 / 5,835 / 5,835 |109| Epochs | 10 (best checkpoint by exact match kept) |110| Learning rate | 5e-5, linear decay, 5% warmup |111| Batch size | 16 × 2 gradient accumulation (effective 32) |112| Weight decay | 0.01 |113| Max grad norm | 1.0 |114| Max source / target length | 256 / 256 tokens |115| Precision | bf16 |116| Seed | 42 |117| Training time | ~1h25m on a single GPU |118 119No example in any split was truncated at 256 tokens (longest source: 132 tokens), and the dataset120contains no identical buggy/fixed pairs.121 122## Limitations123 124- Trained only on **abstracted Java** methods from CodeXGLUE. Real-world code with actual125  identifier names, or any other language, is out of distribution and will perform much worse.126- Handles single, self-contained methods — no cross-file or repository-level context.127- Roughly two thirds of test inputs are still not repaired correctly. Treat outputs as suggestions128  to review, not as verified fixes, and always re-run your tests.129- The model can return the input unchanged (3.4% of the test set) when it finds no fix.130 131## License132 133Released under BSD-3-Clause, following the `Salesforce/codet5-base` base model. The training data,134CodeXGLUE code-refinement, is distributed under the Computational Use of Data Agreement (C-UDA).135