Edge0/ARK-ASR-3B

1---2library_name: transformers3tags:4- automatic-speech-recognition5- speech6- audio7- transformers8- pytorch9- safetensors10- vllm11- ark-asr12pipeline_tag: automatic-speech-recognition13language:14- zh15- en16- de17- ja18- fr19- ko20- es21- pl22- it23- ro24- hu25- cs26- nl27- fi28- hr29- sk30- sl31- et32- lt33license: apache-2.034repository: https://github.com/AutoArk/open-audio-opd35---36 37<div align="center">38 39# ARK-ASR-3B: State-of-the-Art Multilingual ASR40 41[![GitHub](https://img.shields.io/badge/GitHub-AutoArk%2Fopen--audio--opd-blue?logo=github)](https://github.com/AutoArk/open-audio-opd)42[![arXiv](https://img.shields.io/badge/arXiv-2605.28139-b31b1b?logo=arxiv)](https://arxiv.org/abs/2605.28139)43[![License](https://img.shields.io/badge/License-Apache--2.0-green)](https://www.apache.org/licenses/LICENSE-2.0)44 45</div>46 47> **TL;DR** ARK-ASR-3B is a multilingual automatic speech recognition model. It achieves current state-of-the-art results on the Hugging Face Open ASR Leaderboard English short-form benchmark, with an average WER of **5.04%** and RTFx of **490.98** across AMI, Earnings22, GigaSpeech, LibriSpeech, SPGISpeech, and VoxPopuli. The accompanying training, inference, and evaluation code is available at [AutoArk/open-audio-opd](https://github.com/AutoArk/open-audio-opd).48 49## Abstract50 51ARK-ASR-3B is a 3B-scale audio-capable autoregressive Transformers model for automatic speech recognition.52 53It combines a Whisper-style audio encoder, an MLP adapter, and a Qwen decoder with custom `arkasr` remote code.54 55ARK-ASR currently supports Chinese, English, German, Japanese, French, Korean, Spanish, Polish, Italian, Romanian, Hungarian, Czech, Dutch, Finnish, Croatian, Slovak, Slovene, Estonian, and Lithuanian ASR.56 57## Supported Languages58 59Chinese, English, German, Japanese, French, Korean, Spanish, Polish, Italian, Romanian, Hungarian, Czech, Dutch, Finnish, Croatian, Slovak, Slovene, Estonian, and Lithuanian.60 61## Model Overview62 63<div align="center">64  <img src="figures/ark_asr_architecture.png" width="95%" alt="ARK-ASR architecture"/>65  <br>66  <p><strong>Figure 1: ARK-ASR architecture.</strong> Audio is encoded by a Whisper-style encoder with RoPE, merged through an MLP adapter, and injected into a Qwen decoder by replacing audio placeholder token embeddings before transcript generation.</p>67</div>68 69- **Model size:** 3B-scale decoder LLM with a dedicated Whisper-style audio encoder and MLP adapter70- **Task:** automatic speech recognition71- **Architecture:** audio-capable autoregressive Transformers model with custom `arkasr` remote code72- **Checkpoint format:** `safetensors`73- **Sampling rate:** 16 kHz74- **Recommended inference code:** [`scripts/infer/ark_asr_transformers.py`](https://github.com/AutoArk/open-audio-opd/blob/main/scripts/infer/ark_asr_transformers.py)75- **vLLM serving:** [`scripts/vllm/ark_asr_vllm`](https://github.com/AutoArk/open-audio-opd/tree/master/scripts/vllm/ark_asr_vllm)76 77The model should be loaded with `trust_remote_code=True`. The official inference script handles the processor, tokenizer, audio prompt format, generation cleanup, and ASR token filtering.78 79## Performance80 81The following results are from the Hugging Face [Open ASR Leaderboard](https://huggingface.co/datasets/hf-audio/open-asr-leaderboard). Lower WER is better. ARK-ASR-3B reaches the current state of the art on this English short-form benchmark.82 83### English WER84 85| Model | AMI | Earnings22 | GigaSpeech | LS Clean | LS Other | SPGISpeech | VoxPopuli | Avg |86| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |87| ARK-ASR-3B | **8.79%** | **8.23%** | **6.98%** | **1.03%** | **2.35%** | **2.46%** | **5.47%** | **5.04%** |88| ARK-ASR-0.6B | 10.02% | 9.77% | 8.00% | 1.53% | 3.51% | 2.63% | 6.31% | 5.97% |89 90### Chinese CER91 92| Model | AISHELL-1 | WenetSpeech test meeting | WenetSpeech test-net |93| --- | ---: | ---: | ---: |94| ARK-ASR-3B | **1.80%** | **4.97%** | **4.58%** |95| ARK-ASR-0.6B | 2.02% | 5.92% | 4.96% |96 97## Inference98 99Run ASR inference with Hugging Face Transformers:100 101```python102import torch103from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer104 105model_path = "AutoArk-AI/ARK-ASR-3B"106audio_path = "assets/libai.wav"107 108device = "cuda" if torch.cuda.is_available() else "cpu"109torch_dtype = torch.bfloat16 if device == "cuda" else torch.float32110 111processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)112tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)113model = AutoModelForCausalLM.from_pretrained(114    model_path,115    trust_remote_code=True,116    torch_dtype=torch_dtype,117    attn_implementation="sdpa",118).to(device)119model.eval()120 121 122def build_bad_words_ids(tokenizer):123    eos_ids = tokenizer.eos_token_id124    keep_ids = {eos_ids} if isinstance(eos_ids, int) else set(eos_ids or [])125    bad_ids = set(tokenizer.all_special_ids) - keep_ids126    bad_ids.update(127        token_id128        for token, token_id in tokenizer.get_added_vocab().items()129        if token.startswith("<") and token.endswith(">") and token_id not in keep_ids130    )131    return [[token_id] for token_id in sorted(bad_ids)]132 133conversation = [134    {135        "role": "user",136        "content": [137            {"type": "audio", "path": audio_path},138            {"type": "text", "text": "Please transcribe this audio."},139        ],140    }141]142 143inputs = processor.apply_chat_template(144    conversation,145    add_generation_prompt=True,146    return_tensors="pt",147    sampling_rate=16000,148    audio_padding="longest",149    text_kwargs={"padding": "longest"},150    audio_max_length=30 * 16000,151)152inputs = inputs.to(device)153if "audios" in inputs:154    inputs["audios"] = inputs["audios"].to(dtype=torch_dtype)155 156bad_words_ids = build_bad_words_ids(tokenizer)157with torch.inference_mode():158    outputs = model.generate(159        **inputs,160        do_sample=False,161        max_new_tokens=256,162        pad_token_id=tokenizer.pad_token_id,163        eos_token_id=tokenizer.eos_token_id,164        bad_words_ids=bad_words_ids,165    )166decoded_outputs = tokenizer.batch_decode(167    outputs[:, inputs.input_ids.shape[1] :],168    skip_special_tokens=True,169)170print(decoded_outputs)171```172 173For batch JSONL inference, use the open-source inference code:174 175```bash176git clone https://github.com/AutoArk/open-audio-opd177cd open-audio-opd178pip install -e .179```180 181The input JSONL should contain one ASR sample per line:182 183```json184{"audio":"/path/to/audio.wav","text":"","task":"asr","begin_time":-1,"end_time":-1}185```186 187```bash188python scripts/infer/ark_asr_transformers.py \189  --input /path/to/input.jsonl \190  --output runs/infer/predictions.jsonl \191  --model_path AutoArk-AI/ARK-ASR-3B \192  --processor_path AutoArk-AI/ARK-ASR-3B \193  --batch_size 40 \194  --dtype bfloat16 \195  --attn_impl sdpa196```197 198The output JSONL preserves input metadata and adds:199 200- `pred_text`: cleaned prediction text for downstream evaluation201- `pred_text_raw`: raw decoded generation before cleanup202 203## vLLM Online Serving204 205ARK-ASR can also be deployed as a vLLM-backed online ASR service with the206adapter in207[`scripts/vllm/ark_asr_vllm`](https://github.com/AutoArk/open-audio-opd/tree/master/scripts/vllm/ark_asr_vllm).208The service exposes both a compact `/asr` endpoint and an OpenAI-style209`/v1/audio/transcriptions` endpoint.210 211Clone and install the serving code:212 213```bash214git clone https://github.com/AutoArk/open-audio-opd215cd open-audio-opd216pip install -e ".[vllm]"217```218 219Start the service:220 221```bash222MODEL=AutoArk-AI/ARK-ASR-3B \223GPU=0 \224PORT=8025 \225scripts/vllm/deploy_ark_asr_vllm_service.sh start226```227 228Check the service:229 230```bash231scripts/vllm/deploy_ark_asr_vllm_service.sh status232curl -sS http://127.0.0.1:8025/health233curl -sS http://127.0.0.1:8025/token-mask234```235 236Run one transcription request:237 238```bash239curl -sS -X POST http://127.0.0.1:8025/asr \240  -F file=@/path/to/audio.wav \241  -F max_new_tokens=256242```243 244OpenAI-style transcription endpoint:245 246```bash247curl -sS -X POST http://127.0.0.1:8025/v1/audio/transcriptions \248  -F file=@/path/to/audio.wav \249  -F model=ark-asr250```251 252Stop the service:253 254```bash255scripts/vllm/deploy_ark_asr_vllm_service.sh stop256```257 258The vLLM adapter registers the custom `arkasr` model, loads the local259processor/tokenizer with `trust_remote_code=True`, applies generation-time260token masking for non-ASR control tokens, and keeps `<|im_end|>` as the stop261token. Service logs and PID files are written under `runs/vllm/`.262 263## Evaluation264 265The reported leaderboard numbers are evaluated with the Hugging Face266[`open_asr_leaderboard`](https://github.com/huggingface/open_asr_leaderboard)267evaluation code.268 269For local J/WER evaluation, the repository also includes this entrypoint:270 271```bash272python scripts/eval/eval_jwer_ark_asr_transformers.py \273  --input /path/to/test.jsonl \274  --output runs/eval/result.jsonl \275  --model_path AutoArk-AI/ARK-ASR-3B \276  --processor_path AutoArk-AI/ARK-ASR-3B \277  --batch_size 40 \278  --dtype bfloat16 \279  --attn_impl sdpa280```281 282No evaluation audio or dataset files are bundled with this model repository.283 284## Acknowledgements285 286The training code is based on [THUNLP/OPD](https://github.com/thunlp/OPD/) and [verl](https://github.com/volcengine/verl). The OPD recipe uses a stronger ASR teacher to score online student rollouts.287 288## Citation289 290If you find ARK-ASR or open-audio-opd useful, please cite:291 292```bibtex293@misc{lin2026dataefficientopd,294  title={Data-Efficient On-Policy Distillation for Automatic Speech Recognition},295  author={Lin, Yu and Wang, Yiming and Cai, Runyuan and Zeng, Xiaodong},296  year={2026},297  eprint={2605.28139},298  archivePrefix={arXiv},299  primaryClass={cs.AI},300  url={https://arxiv.org/abs/2605.28139}301}302```303