mispeech/dashengtokenizer
141.2k
1---2library_name: transformers3pipeline_tag: audio-to-audio4tags:5- audio-classification6- signal-processing7license: apache-2.08---9 10 11 12 13 14# DashengTokenizer15 16<div align="center">17 18 19<a href="https://arxiv.org/abs/2602.23765"><img src="https://img.shields.io/badge/arXiv-2602.23765-b31b1b" alt="version"></a>20 <a href="https://huggingface.co/mispeech/dashengtokenizer"><img src="https://img.shields.io/badge/HuggingFace-ffcc66" alt="version"></a>21 <a href="https://arxiv.org/abs/2602.2602.23765"><img src="https://img.shields.io/badge/license-Apache-13333b" alt="version"></a>22<a href="https://huggingface.co/mispeech/dashengtokenizer/colab">23 <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab">24</a>25 26</div>27 28DashengTokenizer is a high-performance continious audio tokenizer designed for audio understanding and generation tasks.29Compared to previous works, our framework trains a **single linear layer** to enable audio generation for semantically strong encoders.30 31Achievements:32 33* **State-of-the-Art** Audio Understanding: DashengTokenizer consistently outperforms most previous self-supervised and supervised audio encoders.34* **High-Fidelity** Signal Reconstruction: Maintains exceptional signal integrity, ensuring that audio remains crisp and accurate after processing.35* Accelerated **Audio Generation** Training: Achieves optimal performance significantly faster than standard VAE models, reducing training time and costs.36* Superior **Speech Enhancement**: Provides a more robust encoding foundation for isolating and clarifying speech in noisy environments.37 38 3940 41## Usage42 43### Installation44 45```bash46uv pip install transformers torch torchaudio einops47```48 49### Basic Usage50 51```python52import torch53import torchaudio54from transformers import AutoModel55 56# Load the model57model = AutoModel.from_pretrained("mispeech/dashengtokenizer", trust_remote_code=True)58model.eval()59 60# Load audio file (only 16kHz supported!)61audio, sr = torchaudio.load("path/to/audio.wav")62 63# Optional: Create attention mask for variable-length inputs64# attention_mask = torch.ones(audio.shape[0], audio.shape[1]) # All ones for full audio65# attention_mask[0, 8000:] = 0 # Example: mask second half of first sample66 67# Method 1: End-to-end processing (encode + decode)68with torch.no_grad(), torch.autocast(device_type='cuda'):69 outputs = model(audio) # Optionally pass attention_mask=attention_mask70 reconstructed_audio = outputs["audio"]71 embeddings = outputs['embeddings']72 73# Method 2: Separate encoding and decoding74with torch.no_grad(), torch.autocast(device_type='cuda'):75 # Encode audio to embeddings76 embeddings = model.encode(audio) # Optionally pass attention_mask=attention_mask77 78 # Decode embeddings back to audio79 reconstructed_audio = model.decode(embeddings)80 81# Save reconstructed audio82torchaudio.save("reconstructed_audio.wav", reconstructed_audio, sr)83```84 85 86## Use Cases87 88### 1. Audio Encoding89```python90embeddings = model.encode(audio)91reconstructed = model.decode(embeddings)92```93 94### 2. Feature Extraction95```python96# Extract rich audio features for downstream tasks97features = model.encode(audio)98# Use features for classification, clustering, etc.99```100 101 102## Limitations103 104- Optimized for 16kHz mono audio105 106## Results107 108109110 111## Citation112 113If you use DashengTokenizer in your research, please cite:114 115```bibtex116@misc{dinkel_dashengtokenizer_2026,117 title={DashengTokenizer: One layer is enough for unified audio understanding and generation},118 author={MiLM Plus, Xiaomi},119 year={2026},120 url={https://huggingface.co/mispeech/dashengtokenizer}121}122```123 124## License125 126Apache 2.0 License127 128