FunAudioLLM/Fun-CineForge
6352
1---2license: apache-2.03datasets:4- FunAudioLLM/CineDub-Example5language:6- zh7- en8tags:9- Dubbing-model10pipeline_tag: text-to-speech11---12 13<p align="center">14<b>๐ฌ Fun-CineForge: A Unified Dataset Pipeline and Model for Zero-Shot Movie Dubbing in Diverse Cinematic Scenes</b>15</p>16 17<div align="center">18 1920<a href=""><img src="https://img.shields.io/badge/OS-Linux-orange.svg"></a>21<a href=""><img src="https://img.shields.io/badge/Python->=3.8-aff.svg"></a>22<a href=""><img src="https://img.shields.io/badge/Pytorch->=2.1-blue"></a>23</div>24 25<div align="center"> 26<h4><a href="#Open-Source">Open Source</a>27๏ฝ<a href="#Environment">Environment</a>28๏ฝ<a href="#Dataset-Pipeline">Dataset Pipeline</a>29๏ฝ<a href="#Dubbing-Model">Dubbing Model</a>30๏ฝ<a href="#Recent-Updates">Recent Updates</a>31๏ฝ<a href="#Publication">Publication</a>32๏ฝ<a href="#Comminicate">Comminicate</a>33</h4>34</div>35 36**Fun-CineForge** contains an end-to-end dataset pipeline for producing large-scale dubbing datasets and an MLLM-based dubbing model designed for diverse cinematic scenes. Using this pipeline, we constructed the first large-scale Chinese television dubbing dataset CineDub-CN, which includes rich annotations and diverse scenes. In monologue, narration, dialogue, and multi-speaker scenes, our dubbing model consistently outperforms state-of-the-art methods in terms of audio quality, lip-sync, timbre transition, and instruction following.37 38<a name="Open-Source"></a>39## Open Source ๐ฌ40You can access [https://funcineforge.github.io/](https://funcineforge.github.io/) to get our CineDub-CN dataset samples and demo samples. 41 42GitHub link: [https://github.com/FunAudioLLM/FunCineForge/](https://github.com/FunAudioLLM/FunCineForge/)43 44ModelScope link: [https://www.modelscope.cn/models/FunAudioLLM/Fun-CineForge/](https://www.modelscope.cn/models/FunAudioLLM/Fun-CineForge/)45 46CineDub Samples: 47[huggingface](https://huggingface.co/datasets/FunAudioLLM/CineDub-Example/)48[modelscope](https://www.modelscope.cn/datasets/FunAudioLLM/CineDub-Example)49 50<a name="Environment"></a>51## Environmental Installation52 53Fun-CineForge relies on Conda and Python environments. Execute **setup.py** to automatically install the entire project environment and open-source model.54 55```shell56# Conda57git clone git@github.com:FunAudioLLM/FunCineForge.git58conda create -n FunCineForge python=3.10 -y && conda activate FunCineForge59sudo apt-get install ffmpeg60# Initial settings61python setup.py62```63 64<a name="Dataset-Pipeline"></a>65## Dataset Pipeline ๐จ66 67### Data collection68If you want to produce your own data, 69we recommend that you refer to the following requirements to collect the corresponding movies or television series.70 711. Video source: TV dramas or movies, non documentaries, with more monologues or dialogue scenes, clear and unobstructed faces (such as without masks and veils).722. Speech Requirements: Standard pronunciation, clear articulation, prominent human voice. Avoid materials with strong dialects, excessive background noise, or strong colloquialism.733. Image Requirements: High resolution, clear facial details, sufficient lighting, avoiding extremely dark or strong backlit scenes.74 75### How to use76 77- [1] Standardize video format and name; trim the beginning and end of long videos; extract the audio from the trimmed video. (default is to trim 10 seconds from both the beginning and end.)78```shell79python normalize_trim.py --root datasets/raw_zh --intro 10 --outro 1080```81 82- [2] [Speech Separation](./speech_separation/README.md). The audio is used to separate the vocals from the instrumental music.83```shell84cd speech_separation85python run.py --root datasets/clean/zh --gpus 0 1 2 386```87 88- [3] [VideoClipper](./video_clip/README.md). For long videos, VideoClipper is used to obtain sentence-level subtitle files and clip the long video into segments based on timestamps. Now it supports bilingualism in both Chinese and English. Below is an example in Chinese. It is recommended to use gpu acceleration for English.89```shell90cd video_clip91bash run.sh --stage 1 --stop_stage 2 --input datasets/raw_zh --output datasets/clean/zh --lang zh --device cpu92```93 94- Video duration limit and check for cleanup. (Without --execute, only pre-deleted files will be printed. After checking, add --execute to confirm the deletion.)95```shell96python clean_video.py --root datasets/clean/zh97python clean_srt.py --root datasets/clean/zh --lang zh98```99 100- [4] [Speaker Diarization](./speaker_diarization/README.md). Multimodal active speaker recognition obtains RTTM files; identifies the speaker's facial frames, extracts frame-level speaker face and lip raw data.101```shell102cd speaker_diarization103bash run.sh --stage 1 --stop_stage 4 --hf_access_token hf_xxx --root datasets/clean/zh --gpus "0 1 2 3"104```105 106- (Reference) Extract speech tokens based on the CosyVoice3 tokenizer for llm training.107```shell108python speech_tokenizer.py --root datasets/clean/zh109```110 111- [5] Multimodal CoT Correction. Based on general-purpose MLLMs, the system uses audio, ASR text, and RTTM files as input. It leverages Chain-of-Thought (CoT) reasoning to extract clues and corrects the results of the specialized models. It also annotates character age, gender, and vocal timbre. Experimental results show that this strategy reduces the CER from 4.53% to 0.94% and the speaker diarization error rate from 8.38% to 1.20%, achieving quality comparable to or even better than manual transcription. Adding the --resume enables breakpoint COT inference to prevent wasted resources from repeated COT inferences. Now supports both Chinese and English.112```shell113python cot.py --root_dir datasets/clean/zh --lang zh --provider google --model gemini-3-pro-preview --api_key xxx --resume114python cot.py --root_dir datasets/clean/en --lang en --provider google --model gemini-3-pro-preview --api_key xxx --resume115```116 117- The construction of the dataset retrieval file will read all production data, perform bidirectional verification of script content and speaker separation results.118```shell119python build_datasets.py --root_zh datasets/clean/zh --root_en datasets/clean/en --out_dir datasets/clean --save120```121 122<a name="Dubbing-Model"></a>123## Dubbing Model โ๏ธ124We've open-sourced the inference code and the **infer.sh** script, and provided some test cases in the data folder for your experience. Inference requires a consumer-grade GPU. Run the following command:125 126```shell127cd exps128bash infer.sh129```130 131The API for multi-speaker dubbing from raw videos and SRT scripts is under development ...132 133<a name="Recent-Updates"></a>134## Recent Updates ๐135- 2025/12/18: Fun-CineForge dataset pipeline toolkit is online! ๐ฅ136- 2026/01/19: Chinese demo samples and CineDub-CN dataset samples released. ๐ฅ137- 2026/01/25: Fix some environmental and operational issues.138- 2026/02/09: Optimized the data pipeline and added support for English videos.139- 2026/03/05: English demo samples and CineDub-EN dataset samples released. ๐ฅ140- 2026/03/16: Open source inference code and checkpoints. ๐ฅ141 142<a name="Publication"></a>143## Publication ๐144If you use our dataset or code, please cite the following paper:145<pre>146@misc{liu2026funcineforgeunifieddatasettoolkit,147 title={FunCineForge: A Unified Dataset Toolkit and Model for Zero-Shot Movie Dubbing in Diverse Cinematic Scenes}, 148 author={Jiaxuan Liu and Yang Xiang and Han Zhao and Xiangang Li and Zhenhua Ling},149 year={2026},150 eprint={2601.14777},151 archivePrefix={arXiv},152 primaryClass={cs.CV},153}154</pre>155 156<a name="Comminicate"></a>157## Comminicate ๐158The Fun-CineForge open-source project is developed and maintained by the Tongyi Lab Speech Team and a student from NERCSLIP, University of Science and Technology of China.159We welcome you to participate in discussions on Fun-CineForge [GitHub Issues](https://github.com/FunAudioLLM/FunCineForge/issues) or contact us for collaborative development.160For any questions, you can contact the [developer](mailto:jxliu@mail.ustc.edu.cn).161 162โญ Hope you will support Fun-CineForge. Thank you.163 164### Disclaimer165 166This repository contains research artifacts:167 168โ ๏ธ Currently not a commercial product of Tongyi Lab.169 170โ ๏ธ Released for academic research / cutting-edge exploration purposes171 172โ ๏ธ CineDub Dataset samples are subject to specific license terms.