Apppublic

OpenMOSS-Team/MOSS-Speech

sourceHugging Faceupdated 5mo agoView on Hugging Face

21likes

App README

MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidance

阅读中文版本.

📖 Introduction

Spoken dialogue systems often rely on cascaded pipelines that transcribe, process, and resynthesize speech, which limits expressivity and discards paralinguistic cues. MOSS-Speech directly understands and generates speech without relying on text intermediates, enabling end-to-end speech interaction while preserving tone, prosody, and emotion.

Our approach combines a modality-based layer-splitting architecture with a frozen pre-training strategy, leveraging pretrained text LLMs while extending native speech capabilities. Experiments show state-of-the-art results in spoken question answering and competitive speech-to-speech performance compared to text-guided systems.

Welcome to check out the system's demonstration video.

🔑 Key Features

—True Speech-to-Speech Modeling: No text guidance required.
—Layer-Splitting Architecture: Integrates modality-specific layers on top of pretrained text LLM backbones.
—Frozen Pre-Training Strategy: Preserves LLM reasoning while enhancing speech understanding and generation.
—State-of-the-Art Performance: Excels in spoken question answering and speech-to-speech tasks.
—Expressive & Efficient: Maintains paralinguistic cues often lost in cascaded pipelines, such as tone, emotion, and prosody.

📂 Repository Contents

—gradio_demo.py – Gradio-based web demo script for quickly experiencing speech-to-speech functionality.
—generation.py – Core generation script for producing output speech from input speech, suitable for inference and batch processing.

🛠️ Installation

bash

# Clone the repository
git clone https://github.com/OpenMOSS/MOSS-Speech
cd MOSS-Speech

# Install dependencies
pip install -r requirements.txt

🚀 Usage

Launch the web demo

python3 gradio_demo.py

License

—The code in this repository is released under the Apache 2.0 license.

Acknowledgements

—Qwen: We use Qwen3-8B-Instruct as the base model.
—We thank an anonymous colleague for Character Voice!

📜 Citation

If you use this repository or model in your research, please cite:

bibtex

@article{moss_speech2025,
  title={MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidance},
  author={SLM Team},
  institution={Shanghai Innovation Institute, Fudan University, MOSI},
  year={2025},
  note={Official implementation available at https://huggingface.co/fnlp/MOSS-Speech}
}

or

@misc{moss_speech2025,
  author = {SLM Team},
  title = {MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidance},
  year = {2025},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/OpenMOSS/MOSS-Speech}},
}