servantofares/Stable-DiffCoder-8B-Base

1---2license: mit3pipeline_tag: text-generation4library_name: transformers5---6 7# Stable-DiffCoder-8B-Base8 9<div align="left" style="line-height: 1;">10  <a href="https://bytedance-seed.github.io/Stable-DiffCoder/" target="_blank" style="margin: 2px;">11    <img alt="Homepage" src="https://img.shields.io/badge/Stable--DiffCoder-Homepage-a468fe?color=a468fe&logoColor=white" style="display: inline-block; vertical-align: middle;"/>12  </a>13 14  <a href="https://arxiv.org/abs/2601.15892" target="_blank" style="margin: 2px;">15    <img alt="Technical Report" src="https://img.shields.io/badge/arXiv-Technical%20Report-brightgreen?logo=arxiv&logoColor=white" style="display: inline-block; vertical-align: middle;"/>16  </a>17  18  <a href="https://huggingface.co/ByteDance-Seed" target="_blank" style="margin: 2px;">19      <img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-ByteDance%20Seed-536af5?color=536af5&logoColor=white" style="display: inline-block; vertical-align: middle;"/>20  </a>21  22  <a href="https://github.com/ByteDance-Seed/Stable-DiffCoder/blob/master/LICENSE" style="margin: 2px;">23      <img alt="License" src="https://img.shields.io/badge/License-MIT-f5de53?color=f5de53&logoColor=white" style="display: inline-block; vertical-align: middle;"/>24  </a>25</div>26 27 28## Introduction29We are thrilled to introduce Stable-DiffCoder, which is a strong code diffusion large language model. Built directly on the Seed-Coder architecture, data, and training pipeline, it introduces a block diffusion continual pretraining (CPT) stage with a tailored warmup and block-wise clipped noise schedule.30 31Under identical architecture and data settings, we systematically analyze and design an efficient diffusion training pipeline that is not only stable but also potentially lifts the model’s performance ceiling. With this recipe, Stable-DiffCoder demonstrates overall performance improvements compared to its autoregressive (AR) counterpart across a broad set of code benchmarks, while any-order modeling improves structured code handling for editing and reasoning, and diffusion-based corruption aids learning for low-resource programming languages.32 33Notably, with only CPT followed by supervised fine-tuning, Stable-DiffCoder further surpasses many strong ∼8B AR and diffusion-based code models. These results demonstrate that diffusion-based training can improve code modeling quality beyond what AR training alone can achieve, even under tightly controlled data and architecture constraints.34 35<p align="center">36  <img width="100%" src="imgs/intro_performance.png">37</p>38 39This repo contains the **Stable-DiffCoder-8B-Base** model, which has the following features:40- Type: Mask Diffusion Language Models41- Training Stage: Pretraining42- Data Source: GitHub data, code-related web data43- Training Tokens: 1.3 trillion44- Supports: Code completion, code infilling (Fill-in-the-Middle)45- Context Length: 819246 47 48## Model Downloads49| Model Name                  | Length | Download   |    Notes |50|---------------------------------------------------------|--------|------------------------------------|-----------------------|51| 👉 **Stable-DiffCoder-8B-Base**           | 8K    | 🤗 [Model](https://huggingface.co/ByteDance-Seed/Stable-DiffCoder-8B-Base)   |  Pretrained on our model-centric code data.  |52|  Stable-DiffCoder-8B-Instruct  | 8K    | 🤗 [Model](https://huggingface.co/ByteDance-Seed/Stable-DiffCoder-8B-Instruct)   |  Instruction-tuned for alignment with user intent. |53 54## Requirements55Current (v5.3.0) `transformers` is available for inference:56```bash57pip install transformers~=5.3.058```59## Explanation of Inference Parameters60- `steps`: Number of steps for diffusion generation  61- `gen_length`: Maximum length of the generated output  62- `block_length`: Length of the diffusion block, with a default value of 4  63- `temperature`: Temperature for generation, with a default value of 0.0  64- `remasking`: Remasking strategy, optional values are 'low_confidence' or 'random', default value is 'low_confidence' (for principle, refer to [LLADA](https://github.com/ML-GSAI/LLaDA))  65- `tokenizer`: Tokenizer used for text encoding and decoding  66- `shift`: Whether to shift the output to the right by one position (similar to AutoRegressive/AR), default value is False  67- `threshold`: Threshold for decoding (range: 0-1.0), default value is None; a smaller value results in faster decoding speed (for principle, refer to [Fast-DLLM](https://github.com/NVlabs/Fast-dLLM))  68- `eos_id`: ID of the end-of-sequence token, default value is `tokenizer.eos_token_id`  69 70## Quickstart71 72Here is a simple example demonstrating how to load the model and generate code.73 74```python75from transformers import AutoTokenizer, AutoModelForCausalLM76import torch77 78device = 'cuda'79model = AutoModelForCausalLM.from_pretrained('Stable_DiffCoder-8B-Base', trust_remote_code=True, torch_dtype=torch.bfloat16).to(device).eval()80tokenizer = AutoTokenizer.from_pretrained('Stable_DiffCoder-8B-Base', trust_remote_code=True)81 82prompt = 'Write a quick sort algorithm.'83input_ids = tokenizer(prompt)['input_ids']84input_ids = torch.tensor(input_ids).to(device).unsqueeze(0)85 86out = model.generate(input_ids, steps=128, gen_length=128, block_length=4, temperature=0., remasking='low_confidence', tokenizer=tokenizer, shift=False, threshold=None, eos_id=tokenizer.eos_token_id)87print(tokenizer.decode(out[0][input_ids.shape[1]:], skip_special_tokens=True))88```89 90## Fill-in-the-Middle (FIM) Example91Stable-DiffCoder-8B-Base natively supports Fill-in-the-Middle (FIM) tasks, where the model is given a prefix and a suffix and asked to predict the missing middle content. This allows for code infilling scenarios such as completing a function body or inserting missing logic between two pieces of code.92 93```python94from transformers import AutoTokenizer, AutoModelForCausalLM95import torch96 97device = 'cuda'98model = AutoModelForCausalLM.from_pretrained('ByteDance-Seed/Stable-DiffCoder-8B-Base', trust_remote_code=True, torch_dtype=torch.bfloat16).to(device).eval()99tokenizer = AutoTokenizer.from_pretrained('ByteDance-Seed/Stable-DiffCoder-8B-Base', trust_remote_code=True)100 101prefix = "def add_numbers(a, b):\n    "102suffix = "\n    return result"103 104# Combine prefix and suffix following the FIM format105prompt = '<[fim-suffix]>' + suffix + '<[fim-prefix]>' + prefix + '<[fim-middle]>'106input_ids = tokenizer(prompt)['input_ids']107input_ids = torch.tensor(input_ids).to(device).unsqueeze(0)108 109out = model.generate(input_ids, steps=64, gen_length=64, block_length=4, temperature=0., remasking='low_confidence', tokenizer=tokenizer, shift=False, threshold=None, eos_id=tokenizer.eos_token_id)110print(tokenizer.decode(out[0][input_ids.shape[1]:], skip_special_tokens=True))111 112```113 114## Evaluation115 116Stable-DiffCoder-8B-Base has been evaluated on code generation, code completion, and code reasoning benchmarks, achieving state-of-the-art performance among ~8B open-source models.117 118 119|            | DeepSeek-Coder-6.7B-Base | OpenCoder-8B-Base | Qwen2.5-Coder-7B | Seed-Coder-8B-Base | Stable-DiffCoder-8B-Base |120|------------|:------------------------:|:-----------------:|:----------------:|:------------------:|:------------------------:|121| HumanEval  |           47.6           |        66.5       |       72.0       |        77.4        |        **79.3**          |122| MBPP       |           70.2           |        79.9       |       79.4       |        82.0        |        **83.6**          |123| MultiPL-E  |           44.7           |        61.0       |       58.8       |        67.6        |        **71.2**          |124| CRUXEval-O |           41.0           |        43.9       |       56.0       |        54.8        |        **60.0**          |125 126 127For detailed benchmark performance, please refer to our [📑 Technical Report](https://github.com/ByteDance-Seed/Stable-DiffCoder/blob/master/Stable_DiffCoder.pdf).128 129## License130 131This project is licensed under the MIT License. See the [LICENSE file](https://github.com/ByteDance-Seed/Stable-DiffCoder/blob/master/LICENSE) for details.132 133## Citation134 135If you find our work helpful, feel free to give us a cite.136 137```138@misc{fan2026stablediffcoderpushingfrontiercode,139      title={Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model}, 140      author={Chenghao Fan and Wen Heng and Bo Li and Sichen Liu and Yuxuan Song and Jing Su and Xiaoye Qu and Kai Shen and Wei Wei},141      year={2026},142      eprint={2601.15892},143      archivePrefix={arXiv},144      primaryClass={cs.CL},145      url={https://arxiv.org/abs/2601.15892}, 146}147```148