GD-ML/FLUX-Text
34
1---2license: mit3pipeline_tag: image-to-image4library_name: diffusers5tags:6- text-editing7- multilingual8- diffusion-transformer9- diffusion-model10---11 12# FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editing13 14<a href='https://amap-ml.github.io/FLUX-text/'><img src='https://img.shields.io/badge/Project-Page-green'></a>15<a href='https://arxiv.org/abs/2505.03329'><img src='https://img.shields.io/badge/Technique-Report-red'></a> 16<a href="https://github.com/AMAP-ML/FluxText"><img src="https://img.shields.io/badge/GitHub-Code-blue.svg?logo=github&"></a>17<a href="https://huggingface.co/GD-ML/FLUX-Text/"><img src="https://img.shields.io/badge/%F0%9F%A4%97_HuggingFace-Model-ffbd45.svg" alt="HuggingFace"></a>18 19> *[Rui Lan](https://scholar.google.com/citations?user=zwVlWXwAAAAJ&hl=zh-CN), [Yancheng Bai](https://scholar.google.com/citations?hl=zh-CN&user=Ilx8WNkAAAAJ&view_op=list_works&sortby=pubdate), [Xu Duan](https://scholar.google.com/citations?hl=zh-CN&user=EEUiFbwAAAAJ), [Mingxing Li](https://scholar.google.com/citations?hl=zh-CN&user=-pfkprkAAAAJ), [Lei Sun](https://allylei.github.io), [Xiangxiang Chu](https://scholar.google.com/citations?hl=zh-CN&user=jn21pUsAAAAJ&view_op=list_works&sortby=pubdate)*20> <br>21> ALibaba Group22 23<img src='assets/flux-text.png'>24 25## 📖 Overview26* **Motivation:** Scene text editing is a challenging task that aims to modify or add text in images while maintaining the fidelity of newly generated text and visual coherence with the background. The main challenge of this task is that we need to edit multiple line texts with diverse language attributes (e.g., fonts, sizes, and styles), language types (e.g., English, Chinese), and visual scenarios (e.g., poster, advertising, gaming).27* **Contribution:** We propose FLUX-Text, a novel text editing framework for editing multi-line texts in complex visual scenes. By incorporating a lightweight Condition Injection LoRA module, Regional text perceptual loss, and two-stage training strategy, we significantly significant improvements on both Chinese and English benchmarks.28<img src='assets/method.png'>29 30## News31 32- **2025-07-16**: 🔥 Update comfyui node. We have decoupled the FLUX-Text node to support the use of more basic nodes. Due to differences in node computation in ComfyUI, if you need more consistent results, you should set min_length to 512 in the [code](https://github.com/comfyanonymous/ComfyUI/blob/master/comfy/text_encoders/flux.py#L12).33 34<div align="center">35<table>36<tr>37 <td><img src="assets/comfyui2.png" alt="workflow/FLUX-Text-Basic-Workflow.json" width="400"/></td>38</tr>39<tr>40 <td align="center">workflow/FLUX-Text-Basic-Workflow.json</td>41</tr>42</table>43</div>44 45- **2025-07-13**: 🔥 The training code has been updated. The code now supports multi-scale training.46 47- **2025-07-13**: 🔥 Update the low-VRAM version of the Gradio demo, which It currently requires 25GB of VRAM to run. Looking forward to more efficient, lower-memory solutions from the community.48 49- **2025-07-08**: 🔥 ComfyUI Node is supported! You can now build an workflow based on FLUX-Text for editing posters. It is definitely worth trying to set up a workflow to automatically enhance product image service information and service scope. Meanwhile, utilizing the first and last frames enables the creation of video data with text effects. Thanks to the [community work](https://github.com/AMAP-ML/FluxText/issues/4), FLUX-Text was run on 8GB VRAM. 50 51<div align="center">52<table>53<tr>54 <td><img src="assets/comfyui.png" alt="workflow/FLUX-Text-Workflow.json" width="400"/></td>55</tr>56<tr>57 <td align="center">workflow/FLUX-Text-Workflow.json</td>58</tr>59</table>60</div>61 62<div align="center">63<table>64<tr>65 <td><img src="assets/ori_img1.png" alt="assets/ori_img1.png" width="200"/></td>66 <td><img src="assets/new_img1.png" alt="assets/new_img1.png" width="200"/></td>67 <td><img src="assets/ori_img2.png" alt="assets/ori_img2.png" width="200"/></td>68 <td><img src="assets/new_img2.png" alt="assets/new_img2.png" width="200"/></td>69</tr>70<tr>71 <td align="center">original image</td>72 <td align="center">edited image</td>73 <td align="center">original image</td>74 <td align="center">edited image</td>75</tr>76</table>77</div>78 79<div align="center">80<table>81<tr>82 <td><img src="assets/video_end1.png" alt="assets/video_end1.png" width="400"/></td>83 <td><img src="assets/video1.gif" alt="assets/video1.gif" width="400"/></td>84</tr>85<tr>86 <td><img src="assets/video_end2.png" alt="assets/video_end2.png" width="400"/></td>87 <td><img src="assets/video2.gif" alt="assets/video2.gif" width="400"/></td>88</tr>89<tr>90 <td align="center">last frame</td>91 <td align="center">video</td>92</tr>93</table>94</div>95 96- **2025-07-04**: 🔥 We have released gradio demo! You can now try out FLUX-Text.97 98<div align="center">99<table>100<tr>101 <td><img src="assets/gradio_1.png" alt="Example 1" width="400"/></td>102 <td><img src="assets/gradio_2.png" alt="Example 2" width="400"/></td>103</tr>104<tr>105 <td align="center">Example 1</td>106 <td align="center">Example 2</td>107</tr>108</table>109</div>110 111- **2025-07-03**: 🔥 We have released our [pre-trained checkpoints](https://huggingface.co/GD-ML/FLUX-Text/) on Hugging Face! You can now try out FLUX-Text with the official weights.112 113- **2025-06-26**: ⭐️ Inference and evaluate code are released. Once we have ensured that everything is functioning correctly, the new model will be merged into this repository.114 115## Todo List1161. - [x] Inference code 1172. - [x] Pre-trained weights 1183. - [x] Gradio demo1194. - [x] ComfyUI1205. - [x] Training code121 122## 🛠️ Installation123 124We recommend using Python 3.10 and PyTorch with CUDA support. To set up the environment:125 126```bash127# Create a new conda environment128conda create -n flux_text python=3.10129conda activate flux_text130 131# Install other dependencies132pip install -r requirements.txt133pip install flash_attn --no-build-isolation134pip install Pillow==9.5.0135```136 137## 🤗 Model Introduction138 139FLUX-Text is an open-source version of the scene text editing model. FLUX-Text can be used for editing posters, emotions, and more. The table below displays the list of text editing models we currently offer, along with their foundational information.140 141<table style="border-collapse: collapse; width: 100%;">142 <tr>143 <th style="text-align: center;">Model Name</th>144 <th style="text-align: center;">Image Resolution</th>145 <th style="text-align: center;">Memory Usage</th>146 <th style="text-align: center;">English Sen.Acc</th>147 <th style="text-align: center;">Chinese Sen.Acc</th>148 <th style="text-align: center;">Download Link</th>149 </tr>150 <tr>151 <th style="text-align: center;">FLUX-Text-512</th>152 <th style="text-align: center;">512*512</th>153 <th style="text-align: center;">34G</th>154 <th style="text-align: center;">0.8419</th>155 <th style="text-align: center;">0.7132</th>156 <th style="text-align: center;"><a href="https://huggingface.co/GD-ML/FLUX-Text/tree/main/model_512">🤗 HuggingFace</a></th>157 </tr>158 <tr>159 <th style="text-align: center;">FLUX-Text</th>160 <th style="text-align: center;">Multi Resolution</th>161 <th style="text-align: center;">34G for (512*512)</th>162 <th style="text-align: center;">0.8228</th>163 <th style="text-align: center;">0.7161</th>164 <th style="text-align: center;"><a href="https://huggingface.co/GD-ML/FLUX-Text/tree/main/model_multisize">🤗 HuggingFace</a></th>165 </tr>166</table>167 168## 🔥 ComfyUI169 170<details close>171<summary> Installing via GitHub </summary>172 173First, install and set up [ComfyUI](https://github.com/comfyanonymous/ComfyUI), and then follow these steps:174 1751. **Clone FLUXText Repository**:176 ```shell177 git clone https://github.com/AMAP-ML/FluxText.git178 ```179 1802. **Install FluxText**:181 ```shell182 cd FluxText && pip install -r requirements.txt183 ```184 1853. **Integrate FluxText Comfy Nodes with ComfyUI**:186 - **Symbolic Link (Recommended)**:187 ```shell188 ln -s $(pwd)/ComfyUI-fluxtext path/to/ComfyUI/custom_nodes/189 ```190 - **Copy Directory**:191 ```shell192 cp -r ComfyUI-fluxtext path/to/ComfyUI/custom_nodes/193 ```194 195</details>196 197## 🔥 Quick Start198 199Here's a basic example of using FLUX-Text:200 201```python202import numpy as np203from PIL import Image204import torch205import yaml206 207from src.flux.condition import Condition208from src.flux.generate_fill import generate_fill209from src.train.model import OminiModelFIll210from safetensors.torch import load_file211 212config_path = ""213lora_path = ""214with open(config_path, "r") as f:215 config = yaml.safe_load(f)216model = OminiModelFIll(217 flux_pipe_id=config["flux_path"],218 lora_config=config["train"]["lora_config"],219 device=f"cuda",220 dtype=getattr(torch, config["dtype"]),221 optimizer_config=config["train"]["optimizer"],222 model_config=config.get("model", {}),223 gradient_checkpointing=True,224 byt5_encoder_config=None,225 )226 227state_dict = load_file(lora_path)228state_dict_new = {x.replace('lora_A', 'lora_A.default').replace('lora_B', 'lora_B.default').replace('transformer.', ''): v for x, v in state_dict.items()}229model.transformer.load_state_dict(state_dict_new, strict=False)230pipe = model.flux_pipe231 232prompt = "lepto college of education, the written materials on the picture: LESOTHO , COLLEGE OF , RE BONA LESELI LESEL , EDUCATION ."233hint = Image.open("assets/hint.png").resize((512, 512)).convert('RGB')234img = Image.open("assets/hint_imgs.jpg").resize((512, 512))235condition_img = Image.open("assets/hint_imgs_word.png").resize((512, 512)).convert('RGB')236hint = np.array(hint) / 255237condition_img = np.array(condition_img)238condition_img = (255 - condition_img) / 255239condition_img = [condition_img, hint, img]240position_delta = [0, 0]241condition = Condition(242 condition_type='word_fill',243 condition=condition_img,244 position_delta=position_delta,245 )246generator = torch.Generator(device="cuda")247res = generate_fill(248 pipe,249 prompt=prompt,250 conditions=[condition],251 height=512,252 width=512,253 generator=generator,254 model_config=config.get("model", {}),255 default_lora=True,256)257res.images[0].save('flux_fill.png')258```259 260## 🤗 gradio261 262You can upload the glyph image and mask image to edit text region. Or you can use `manual edit` to obtain glyph image and mask image.263 264first, download the model weight and config in [HuggingFace](https://huggingface.co/GD-ML/FLUX-Text)265 266```bash267python app.py --model_path xx.safetensors --config_path config.yaml268```269 270## 💪🏻 Training271 2721. Download training dataset [**AnyWord-3M**](https://modelscope.cn/datasets/iic/AnyWord-3M/summary) from ModelScope, unzip all \*.zip files in each subfolder, then open *\*.json* and modify the `data_root` with your own path of *imgs* folder for each sub dataset.273 2742. Replace the old annotations in AnyWord with the new [annotations](https://huggingface.co/GD-ML/FLUX-Text/tree/main/data_text_recog_glyph). Change the dataset annotations path and image_root in [src/train/data_word.py](https://github.com/AMAP-ML/FluxText/blob/main/src/train/data_word.py#L538).275 276```python277json_paths = [278 ['dataset/Anyword/data_text_recog_glyph/Art/data-info.json', 'AnyWord-3M/ocr_data/Art/imgs/'],279 ['dataset/Anyword/data_text_recog_glyph/COCO_Text/data-info.json', 'AnyWord-3M/ocr_data/COCO_Text/imgs/'],280 ['dataset/Anyword/data_text_recog_glyph/icdar2017rctw/data-info.json', 'AnyWord-3M/ocr_data/icdar2017rctw/imgs'],281 ['dataset/Anyword/data_text_recog_glyph/LSVT/data-info.json', 'AnyWord-3M/ocr_data/LSVT/imgs'],282 ['dataset/Anyword/data_text_recog_glyph/mlt2019/data-info.json', 'AnyWord-3M/ocr_data/mlt2019/imgs/'],283 ['dataset/Anyword/data_text_recog_glyph/MTWI2018/data-info.json', 'AnyWord-3M/ocr_data/MTWI2018/imgs'],284 ['dataset/Anyword/data_text_recog_glyph/ReCTS/data-info.json', 'AnyWord-3M/ocr_data/ReCTS/imgs'],285 ['dataset/Anyword/data_text_recog_glyph/laion/data_v1.1-info.json', 'AnyWord-3M/laion/imgs'],286 ['dataset/Anyword/data_text_recog_glyph/wukong_1of5/data_v1.1-info.json', 'AnyWord-3M/wukong_1of5/imgs'],287 ['dataset/Anyword/data_text_recog_glyph/wukong_2of5/data_v1.1-info.json', 'AnyWord-3M/wukong_2of5/imgs'],288 ['dataset/Anyword/data_text_recog_glyph/wukong_3of5/data_v1.1-info.json', 'AnyWord-3M/wukong_3of5/imgs'],289 ['dataset/Anyword/data_text_recog_glyph/wukong_4of5/data_v1.1-info.json', 'AnyWord-3M/wukong_4of5/imgs'],290 ['dataset/Anyword/data_text_recog_glyph/wukong_5of5/data_v1.1-info.json', 'AnyWord-3M/wukong_5of5/imgs'],291 ]292```293 2943. Download the ODM weights in [HuggingFace](https://huggingface.co/GD-ML/FLUX-Text/blob/main/epoch_100.pt) and change `odm_loss/modelpath` in the [config file](https://github.com/AMAP-ML/FluxText/blob/main/train/config/word_multi_size.yaml#L60).295 2963. (Optional) Download the pretrained weight in [HuggingFace](https://huggingface.co/GD-ML/FLUX-Text) and change `reuse_lora_path` in the [config file](https://github.com/AMAP-ML/FluxText/blob/main/train/config/word_multi_size.yaml#L44).297 2984. Run the training scripts. With 48GB of VRAM, you can train at 512×512 resolution with a batch size of 2 in LoRA rank 8.299 300```bash301bash train/script/train_word.sh302```303 304 305## 📊 Evaluation306 307For [Anytext-benchmark](https://modelscope.cn/datasets/iic/AnyText-benchmark/summary), please set the **config_path**, **model_path**, **json_path**, **output_dir** in the `eval/gen_imgs_anytext.sh` and generate the text editing results.308 309```bash310bash eval/gen_imgs_anytext.sh311```312 313For `Sen.ACC, NED, FID and LPIPS` evaluation, use the scripts in the `eval` folder.314 315```bash316bash eval/eval_ocr.sh317bash eval/eval_fid.sh318bash eval/eval_lpips.sh319```320 321## 📈 Results322 323<img src='assets/method_result.png'>324 325## 🌹 Acknowledgement326 327Our work is primarily based on [OminiControl](https://github.com/Yuanshi9815/OminiControl), [AnyText](https://github.com/tyxsspa/AnyText), [Open-Sora](https://github.com/hpcaitech/Open-Sora), [Phantom](https://github.com/Phantom-video/Phantom). We are sincerely grateful for their excellent works.328 329## 📚 Citation330 331If you find our paper and code helpful for your research, please consider starring our repository ⭐ and citing our work ✏️.332```bibtex333@misc{lan2025fluxtext,334 title={FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editing},335 author={Rui Lan and Yancheng Bai and Xu Duan and Mingxing Li and Lei Sun and Xiangxiang Chu},336 year={2025},337 eprint={2505.03329},338 archivePrefix={arXiv},339 primaryClass={cs.CV}340}341```