Ariful1904129/codebert-flakytest-fold2

1---2license: mit3language:4- code5base_model: microsoft/codebert-base6pipeline_tag: text-classification7tags:8- flaky-tests9- software-testing10- code11- codebert12- reproduction13---14 15# CodeBERT for Flaky Test Categorisation (FlakeBench)16 17Classifies a Java/Kotlin test method into one of six categories: five kinds of flaky18test plus non-flaky.19 20## What this is21 22A fine-tune of `microsoft/codebert-base` on the FlakeBench dataset from23[*Understanding and Improving Flaky Test Classification*](https://utexas.app.box.com/v/august-shi-OOPSLA2025)24(OOPSLA 2025), trained as a reproduction exercise on a single 8 GB consumer GPU.25The uploaded weights are the "Balanced" configuration below.26 27## Training configurations28 29| Parameter | Baseline | lr 2e-5 | Balanced | Augmented | Paper |30| --- | --- | --- | --- | --- | --- |31| Encoder | codebert-base | codebert-base | codebert-base | codebert-base | codebert-base |32| Learning rate | 1e-5 | 2e-5 | 1e-5 | 1e-5 | 1e-5 |33| Batch size | 8 | 8 | 8 | 8 | 8 |34| Max length | 512 | 512 | 512 | 512 | 512 |35| Loss | focal γ=2.0 | focal γ=2.0 | focal γ=2.0 | focal γ=2.0 | focal γ=2.0 |36| Class weights | balanced | balanced | balanced | balanced | balanced |37| Optimizer | AdamW wd 0.01 | AdamW wd 0.01 | AdamW wd 0.01 | AdamW wd 0.01 | AdamW wd 0.01 |38| Precision | fp16 | fp16 | fp16 | fp16 | fp32 |39| Non-flaky rows | 4,972 | 4,972 | 800 | 800 | full |40| Minority handling | none | none | ×160 copies | ×200 variants | none |41| Train rows | 5,114 | 5,114 | 1,600 | 1,800 | 5,114 |42| Epochs run | 8 | 8 | 18 | 13 | 40 |43| Dynamic padding | no | no | no | no | no |44 45Hardware: 1× RTX 4060 Laptop (8 GB). Class rebalancing is the one deviation from the46paper's method, which trains on the raw distribution (97% non-flaky).47 48## Results (per-category F1)49 50| Category         |   Baseline |    lr 2e-5 |   Balanced |  Augmented |      Paper |51| ---------------- | ---------: | ---------: | ---------: | ---------: | ---------: |52| Async Wait        |     76.92% |     78.26% |     74.07% |     64.52% |     58.37% |53| Concurrency       |      0.00% |      0.00% |      0.00% |      0.00% |     35.92% |54| Time              |     57.14% |     66.67% |     66.67% |     40.00% |     72.73% |55| Unordered Coll.   |     75.00% |     83.33% |     83.33% |     72.73% |     73.63% |56| Order Dep.        |     82.35% |     86.96% |     95.24% |     73.68% |     64.35% |57| Non-flaky         |    100.00% |     99.92% |     99.51% |    100.00% |    100.00% |58| **Macro F1**      | **65.24%** | **69.19%** | **69.89%** | **58.49%** | **65.79%** |59 60The **Balanced** configuration (uploaded weights) achieves the best macro-F1 of61**69.89%** .62 63 64## Usage65 66```python67from transformers import AutoTokenizer, AutoModelForSequenceClassification68import torch69 70name = "Ariful1904129/codebert-flakytest-fold2"71tok = AutoTokenizer.from_pretrained(name)72model = AutoModelForSequenceClassification.from_pretrained(name, trust_remote_code=True).eval()73 74code = """@Test75public void testConnect() throws Exception {76    Thread.sleep(1000);77    assertTrue(client.isConnected());78}"""79 80x = tok(code, return_tensors="pt", truncation=True, max_length=512)81with torch.no_grad():82    pred = model(**x).logits.argmax(-1).item()83print(model.config.id2label[pred])84```85 86 87Scope: Java/Kotlin test methods; inputs longer than 512 tokens are truncated.88 89 90## Citation91 92Please cite the original paper. This model is a third-party reproduction and is not93endorsed by its authors.94 95```bibtex96@inproceedings{flakylens2025,97  title     = {Understanding and Improving Flaky Test Classification},98  booktitle = {OOPSLA},99  year      = {2025}100}101```102 103Dataset and method: [UT-SE-Research/FlakyLens](https://github.com/UT-SE-Research/FlakyLens).