Edge0/GPA

1---2library_name: transformers3tags:4- text-to-speech5- automatic-speech-recognition6- voice-conversion7- speech8- audio9pipeline_tag: text-to-speech10language:11- en12- zh13license: apache-2.014homepage: https://autoark.github.io/GPA/15repository: https://github.com/AutoArk/GPA16---17<div align="center">18  <img src="figures/GPA_intro.png" width="80%" alt="GPA Logo"/>19 20# GPA: One Model for Speech Recognition, Text-to-Speech, and Voice Conversion21 22[![GitHub](https://img.shields.io/badge/GitHub-AutoArk%2FGPA-blue?logo=github)](https://github.com/AutoArk/GPA)23[![GitHub](https://img.shields.io/badge/GitHub-AutoArk%2FGPA_TTS-blue?logo=github)](https://github.com/AutoArk/GPA/tree/main/GPA_TTS)24 25</div>26 27> **TL;DR** GPA incorporates three speech tasks into one single model and this repo includes codes of training, fine-tuning and effecient deployment of GPA.28 29## 🆕 GPA-TTS Updates30 31**GPA-TTS** is a standalone, ultra-efficient TTS runtime distilled from GPA, designed for edge deployment.32 33• **INT8/INT4 quantized**: among the smallest open-source TTS runtimes 34• **Runtime-selectable decoder**: INT8 / FP16 / FP32 (quality vs. efficiency trade-off)  35• **Zero-shot voice cloning** from short reference audio  36• **Fully local**: no external LLM required  37• **Production-ready REST API** with voice management  38 39Built for developers deploying voice applications on resource-constrained devices.40 41## 📖 Abstract42 43**GPA** stands for **General Purpose Audio**. 44 45In academia, a student’s GPA (Grade Point Average) serves as a unified metric that reflects performance across diverse subjects—ranging from Calculus and Philosophy to Gym class.46 47Similarly, our GPA model unifies the three major pillars of audio tasks—Text-to-Speech (TTS), Automatic Speech Recognition (ASR), and Voice Conversion (VC)—into a single auto-regreesive transformer.48*   Our open-source content includes support for multiple frameworks and provides **production-ready code suitable for cloud deployment.**49*   we include concise **inference examples** and **training pipelines** for research purpose.50*   The released 0.3B model is also perfect for **edge devices** and edge deployment is to be released.51 52## 🔍 Model Overview53 54<div align="center">55  <img src="figures/GPA.png" width="80%" alt="GPA Model Architecture"/>56  <br>57  <div style="text-align: justify; width: 100%; margin: 10px auto; text-indent: 2em;">58    <strong>Figure 1: Architecture of the proposed GPA framework.</strong> The model utilizes a shared Large Language Model (LLM) backbone to unify three core audio tasks: Understanding (ASR), Generation (TTS), and Editing (Voice Conversion). Depending on the task, the model processes different combinations of inputs (Source Audio, Target Text, or Reference Audio) via Semantic and Acoustic modules to generate the corresponding text or audio output.59  </div>60</div>61 62 63## ⚡ Model Performance64 65The following results are obtained by benchmarking services instantiated via [the official deployment scripts](#-deployment), reflecting end-to-end performance in realistic serving scenarios rather than offline inference.66 67Among currently available open-source systems, **our model is one of the few that natively supports both concurrent and streaming inference, while achieving performance comparable to the first tier of existing approaches.**68 69> **💡Note**70>71> * **TTFC**: Time To First Chunk (TTS)72> * **TTFT**: Time To First Token (ASR)73> * **RTF**: Real-Time Factor (audio duration / synthesis time)74 75### TTS Streaming Benchmark (Latency & Throughput)76 77<div align="center">78  <table>79    <thead>80      <tr>81        <th>Concurrency</th>82        <th>Avg TTFC (ms)</th>83        <th>P50 TTFC (ms)</th>84        <th>P99 TTFC (ms)</th>85        <th>Avg RTF</th>86        <th>P50 RTF</th>87        <th>P99 RTF</th>88        <th>Audio Dur (s)</th>89      </tr>90    </thead>91    <tbody>92      <tr><td>1</td><td>258.8</td><td>258.8</td><td>258.8</td><td>0.197</td><td>0.197</td><td>0.197</td><td>6.44</td></tr>93      <tr><td>5</td><td>385.0</td><td>394.7</td><td>396.2</td><td>0.218</td><td>0.217</td><td>0.248</td><td>6.76</td></tr>94      <tr><td>10</td><td>544.6</td><td>564.2</td><td>566.7</td><td>0.282</td><td>0.301</td><td>0.313</td><td>6.49</td></tr>95      <tr><td>20</td><td>977.8</td><td>977.9</td><td>982.9</td><td>0.470</td><td>0.490</td><td>0.538</td><td>7.19</td></tr>96      <tr><td>40</td><td>1797.0</td><td>1736.4</td><td>2564.5</td><td>0.421</td><td>0.400</td><td>0.587</td><td>6.33</td></tr>97      <tr><td>80</td><td>3786.4</td><td>4054.4</td><td>5415.8</td><td>0.763</td><td>0.763</td><td>1.096</td><td>6.32</td></tr>98      <tr><td>160</td><td>9847.9</td><td>10239.9</td><td>14350.3</td><td>1.718</td><td>1.740</td><td>2.577</td><td>6.44</td></tr>99    </tbody>100  </table>101  <p><strong>Table 2. TTS Streaming RTF and Audio Duration</strong></p>102</div>103 104### ASR Streaming Benchmark105 106<div align="center">107  <table>108    <thead>109      <tr>110        <th>Concurrency</th>111        <th>Avg TTFT (ms)</th>112        <th>P50 TTFT (ms)</th>113        <th>P99 TTFT (ms)</th>114        <th>Avg Total (ms)</th>115      </tr>116    </thead>117    <tbody>118      <tr><td>1</td><td>157.5</td><td>157.5</td><td>157.5</td><td>190.9</td></tr>119      <tr><td>5</td><td>394.1</td><td>393.7</td><td>395.9</td><td>400.0</td></tr>120      <tr><td>10</td><td>589.6</td><td>721.3</td><td>723.3</td><td>598.1</td></tr>121      <tr><td>20</td><td>1316.3</td><td>1495.6</td><td>1500.4</td><td>1317.8</td></tr>122      <tr><td>40</td><td>2690.9</td><td>2678.3</td><td>2861.4</td><td>2693.7</td></tr>123      <tr><td>80</td><td>3833.4</td><td>3961.3</td><td>4027.0</td><td>3845.1</td></tr>124      <tr><td>160</td><td>5037.0</td><td>5689.3</td><td>6676.0</td><td>5044.0</td></tr>125    </tbody>126  </table>127  <p><strong>Table 3. ASR Streaming Latency vs Concurrency</strong></p>128</div>129 130## 📊 Evaluation Metric Results131 132### TTS Evaluation Table133 134| Model | Open-Source | Model Size | test-zh CER (%) ↓ | test-zh Sim (%) ↑ | test-en WER (%) ↓ | test-en Sim (%) ↑ |135| :--- | :---: | :---: | :---: | :---: | :---: | :---: |136| **Multi-Stage or NAR Methods** | | | | | | |137| Human | - | - | 1.26 | 75.5 | 2.14 | 73.4 |138| Seed-TTS | ❌ | - | 1.12 | **79.6** | 2.25 | **76.2** |139| MiniMax-Speech | ❌ | - | 0.83 | 78.3 | 1.65 | 69.2 |140| F5-TTS | ✅ | 0.3B | 1.52 | 74.1 | 2.00 | 64.7 |141| CosyVoice2 | ✅ | 0.5B | 1.45 | 75.7 | 2.57 | 65.9 |142| FireRedTTS2 | ✅ | 1.5B | 1.14 | 73.2 | 1.95 | 66.5 |143| Index-TTS2 | ✅ | 1.5B | 1.03 | 76.5 | 2.23 | 70.6 |144| VibeVoice-1.5B | ✅ | 1.5B | 1.16 | 74.4 | 3.04 | 68.9 |145| VibeVoice-Realtime | ✅ | 0.5B | - | - | 2.05 | 63.3 |146| HiggsAudio-v2 | ✅ | 3B | 1.50 | 74.0 | 2.44 | 67.7 |147| VoxCPM | ✅ | 0.5B | 0.93 | 77.2 | 1.85 | 72.9 |148| GLM-TTS | ✅ | 1.5B | 1.03 | 76.1 | - | - |149| GLM-TTS RL | ✅ | 1.5B | 0.89 | 76.4 | - | - |150| Fun-CosyVoice3-0.5B-2512 | ✅ | 0.5B | 1.21 | 78.0 | 2.24 | 71.8 |151| Fun-CosyVoice3-0.5B-2512_RL | ✅ | 0.5B | 0.81 | 77.4 | 1.68 | 69.5 |152| **One-Stage AR Methods** | | | | | | |153| Spark TTS | ✅ | 0.5B | 1.20 | 66.0 | 1.98 | 57.3 |154| GPA-0.3B-preview | ✅ | 0.3B | **0.95** | 65.9 | **1.51** | 56.5 |155 156### ASR Evaluation Table157 158**Note:** ASR results on Librispeech and Aishell-1. WER (%) is reported for Librispeech, and CER (%) is reported for Aishell-1.159 160| Model | Model Size | Librispeech test-clean | Aishell-1 |161| :--- | :---: | :---: | :---: |162| **Models with < 0.5B parameters** | | | |163| Whisper-S | 0.24B | 3.13 | - |164| GPA-0.3B-preview | 0.3B | 8.88 | 4.50 |165| **Models with > 0.5B parameters** | | | |166| Fun-ASR-nano | 0.8B | 1.76 | 1.80 |167| FireRed-ASR | 1.1B | 1.84 | 0.54 |168| GLM-ASR-nano | 1.5B | 2.00 | 1.81 |169| GLM-ASR-nano* | 1.5B | 2.17 | 2.17 |170| Whisper-L | 1.55B | 1.82 | 4.72 |171| Kimi-Audio | - | 1.32 | 0.71 |172| Step-Audio2 | - | 1.17 | 0.63 |173| Seed-ASR | - | 1.58 | 0.68 |174| Seed-ASR* | - | 2.80 | 1.63 |175| Fun-ASR | 7.7B | 1.51 | 1.22 |176 177## 🙏 Acknowledgements178 179We borrowed a lot of code from the following excellent projects:180 181- [Spark-TTS](https://github.com/SparkAudio/Spark-TTS)182- [GLM-4-Voice](https://github.com/zai-org/GLM-4-Voice/tree/main/speech_tokenizer)183- [Emilia](https://github.com/open-mmlab/Amphion/tree/main/preprocessors/Emilia)184- [FlashTTS](https://github.com/HuiResearch/FlashTTS/tree/master/flashtts)185- [Qwen](https://github.com/QwenLM/Qwen)186 187## 🔗 Citation188 189If you find GPA useful for your research or projects, please cite us:190 191```bibtex192@misc{cai2026unifyingspeechrecognitionsynthesis,193      title={Unifying Speech Recognition, Synthesis and Conversion with Autoregressive Transformers}, 194      author={Runyuan Cai and Yu Lin and Yiming Wang and Chunlin Fu and Xiaodong Zeng},195      year={2026},196      eprint={2601.10770},197      archivePrefix={arXiv},198      primaryClass={cs.SD},199      url={https://arxiv.org/abs/2601.10770}, 200}201```