MIT-SLS/USAD-Small

1---2license: cc-by-nc-sa-4.03pipeline_tag: feature-extraction4tags:5- automatic-speech-recognition6- audio-classification7- audio8- speech9- music10library_name: transformers11datasets:12- openslr/librispeech_asr13- facebook/multilingual_librispeech14- mozilla-foundation/common_voice_17_015- speechcolab/gigaspeech16- facebook/voxpopuli17- agkphysics/AudioSet18language:19- en20---21# USAD: Universal Speech and Audio Representation via Distillation22 23**Universal Speech and Audio Distillation (USAD)** is a unified **speech**, **sound**, and **music** encoder distilled from domain-specific teachers.24Trained on 126k hours of mixed data, USAD delivers competitive performance across diverse benchmarks (SUPERB, HEAR, and AudioSet) with a single model.25 26[👀 **Read Full Paper**](https://arxiv.org/abs/2506.18843)27 28[🛠️ **GitHub**](https://github.com/vectominist/usad)29 30---31 32## 🗂️ Models33 34USAD models are all transformer encoders operating at **50Hz frame rate**. The teacher models are **WavLM Base+** and **ATST Frame**.35 36| Model                                                     | Parameters | Dim  | Layer |37| :-------------------------------------------------------- | ---------: | ---: | ----: |38| [USAD Small](https://huggingface.co/MIT-SLS/USAD-Small)   | 24M        | 384  | 12    |39| [USAD Base](https://huggingface.co/MIT-SLS/USAD-Base)     | 94M        | 768  | 12    | 40| [USAD Large]((https://huggingface.co/MIT-SLS/USAD-Small)) | 330M       | 1024 | 24    |41 42---43 44 45## 🚀 How To Use46 47**Installation**48```49pip install -U torch torchaudio transformers50```51 52**Load Model and Extract Features**53```python54import torch55from transformers import AutoModel56 57# Load pre-trained model58model = AutoModel.from_pretrained("MIT-SLS/USAD-Small", trust_remote_code=True).cuda().eval()59 60# Load audio and resample to 16kHz61wav = model.load_audio("path/to/audio").unsqueeze(0)  # (batch_size, wav_len)62# wav is a float tensor on the same device as the model63# You can also load waveforms directly with torchaudio.load64 65# Extract features66with torch.no_grad():67    results = model(wav)68 69# result["x"]:              model final output (batch_size, seq_len)70# result["mel"]:            mel fbank (batch_size, seq_len * 2, mel_dim)71# result["hidden_states"]:  list of (batch_size, seq_len, encoder_dim)72# result["ffn"]:            list of (batch_size, seq_len, encoder_dim)73```74 75See [usad_model.py](https://huggingface.co/MIT-SLS/USAD-Small/blob/main/usad_model.py) for more details about the model.76 77---78 79## 📖 Citation80 81```bibtex82@inproceedings{chang2025usad,83  title={{USAD}: Universal Speech and Audio Representation via Distillation},84  author={Chang, Heng-Jui and Bhati, Saurabhchand and Glass, James and Liu, Alexander H.},85  booktitle={IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)},86  year={2025}87}88```89 90---91 92## 🙏 Acknowledgement93 94Our implementation is based on the awesome [facebookresearch/fairseq](https://github.com/facebookresearch/fairseq), [cwx-worst-one/EAT](https://github.com/cwx-worst-one/EAT), and [sooftware/conformer](https://github.com/sooftware/conformer) repositories.