DataoceanAI/dolphin-small

1---2license: apache-2.03pipeline_tag: automatic-speech-recognition4language: multilingual5---6# Dolphin7 8[Paper](https://arxiv.org/abs/2503.20212)9[Github](https://github.com/DataoceanAI/Dolphin)10[Huggingface](https://huggingface.co/DataoceanAI)11[Modelscope](https://www.modelscope.cn/organization/DataoceanAI)12 13# Repository Notice14 15This model is officially maintained by **Dataocean AI**.16 17To ensure compatibility with existing user code and download links, we keep two official repositories for the same model:18 19- Original / legacy repository: DataoceanAI20- Organization / enterprise repository: DataoceanAI121 22Both repositories are maintained by the same team and contain the same model files.  23DataoceanAI1 is the newly created enterprise organization account, while DataoceanAI is kept to avoid breaking existing user download scripts and links.24 25Please do not regard either repository as an unofficial copy or unauthorized redistribution.26 27Dolphin is a multilingual, multitask ASR model developed through a collaboration between Dataocean AI and Tsinghua University. It supports 40 Eastern languages across East Asia, South Asia, Southeast Asia, and the Middle East, while also supporting 22 Chinese dialects. It is trained on over 210,000 hours of data, which includes both DataoceanAI's proprietary datasets and open-source datasets. The model can perform speech recognition, voice activity detection (VAD), segmentation, and language identification (LID).28 29## Approach30 31![Mulitask data format](https://raw.githubusercontent.com/DataoceanAI/Dolphin/refs/heads/main/figures/multitask-data-format.png)32Dolphin largely follows the innovative design approach of [Whisper](https://github.com/openai/whisper) and [OWSM](https://github.com/espnet/espnet/tree/master/egs2/owsm_v3.1/s2t1). A joint CTC-Attention architecture is adopted, with encoder based on E-Branchformer and decoder based on standard Transformer. Several key modifications are introduced for its specific focus on ASR. Dolphin does not support translation tasks, and eliminates the use of previous text and its related tokens.33 34A significant enhancement in Dolphin is the introduction of a two-level language token system to better handle linguistic and regional diversity, especially in Dataocean AI dataset. The first token specifies the language (e.g., `<zh>`, `<ja>`), while the second token indicates the region (e.g., `<CN>`, `<JP>`). See details in [paper](https://arxiv.org/abs/2503.20212).35 36 37## Setup38Dolphin requires FFmpeg to convert audio file to WAV format. If FFmpeg is not installed on your system, please install it first:39 40```shell41# Ubuntu or Debian42sudo apt update && sudo apt install ffmpeg43 44# MacOS45brew install ffmpeg46 47# Windows48choco install ffmpeg49```50 51You can install the latest version of Dolphin using the following command:52```shell53pip install -U dataoceanai-dolphin54```55 56Alternatively, it can also be installed from the source:57```shell58pip install git+https://github.com/SpeechOceanTech/Dolphin.git 59```60 61## Available Models and Languages62 63### Models64 65There are 4 models in Dolphin, and 2 of them are available now. See details in [paper](https://arxiv.org/abs/2503.20212).66 67|  Model  | Parameters | Average WER | Publicly Available |68|:------:|:----------:|:------------------:|:------------------:|69|  base  |    140 M    |     33.3      |      ✅        |70| small  |   372 M    |     25.2     |      ✅       |71| medium |   910 M    |    23.1     |            |72| large  |   1679 M   |        21.6         |             |73 74### Languages75 76Dolphin supports 40 Eastern languages and 22 Chinese dialects. For a complete list of supported languages, see [languages.md](https://github.com/DataoceanAI/Dolphin/blob/main/languages.md).77 78## Usage79 80### Command-line usage81 82```shell83dolphin audio.wav84 85# Download model and specify the model path86dolphin audio.wav --model small --model_dir /data/models/dolphin/87 88# Specify language and region89dolphin audio.wav --model small --model_dir /data/models/dolphin/ --lang_sym "zh" --region_sym "CN"90 91# padding speech to 30 seconds92dolphin audio.wav --model small --model_dir /data/models/dolphin/ --lang_sym "zh" --region_sym "CN" --padding_speech true93```94 95### Python usage96 97```python98import dolphin99 100waveform = dolphin.load_audio("audio.wav")101model = dolphin.load_model("small", "/data/models/dolphin", "cuda")102result = model(waveform)103# Specify language and region104result = model(waveform, lang_sym="zh", region_sym="CN")105print(result.text)106```107 108## License109 110Dolphin's code and model weights are released under the [Apache 2.0 License](https://github.com/DataoceanAI/Dolphin/blob/main/LICENSE).