Blinorot/AL-Whisper-R

1---2library_name: transformers3datasets:4- Blinorot/ALARM-Corpora5base_model:6- Qwen/Qwen3-4B-Thinking-25077---8 9# Model Card for AL-Whisper-R10 11This is a checkpoint for AL-Whisper-R, audio-understanding reasoning language model, proposed in [ALARM: Audio–Language Alignment for Reasoning Models](https://arxiv.org/abs/2603.09556).12This model is trained on the full corpora.13 14For more details regarding the model and its usage, please refer to our [GitHub](https://github.com/Blinorot/ALARM).15 16## Inference17 18We provide [vLLM](https://github.com/vllm-project/vllm) support using [vLLM Prompt Embedding API](https://docs.vllm.ai/en/stable/features/prompt_embeds/). 19Since ALARM uses the frozen Qwen3 model as the backbone, `vllm` just runs the original Qwen3 checkpoint, and the ALARM checkpoint is used for extracting LLM input embeddings. 20After you cloned the repo and installed the depnedencies, you can run the pretrained model as follows:21 22```python23# Import libraries24import os25os.environ["CUDA_VISIBLE_DEVICES"] = "0" #optional26 27# run before importing torch because generate_vllm sets the multiprocessing method28from generate_vllm import get_response29from src.model.wrapped_llms.qwen3 import Qwen3AudioWrappedFeatureExtractor30 31from omegaconf import OmegaConf32from torchaudio.utils import _download_asset33from torchcodec.decoders import AudioDecoder34from transformers import AutoTokenizer35from vllm import LLM36 37 38# The model configuration config.39# Handles vllm-related configuration and defines feature extractors,40# i.e., audio -> encoder input embedding conversion.41# All other configuration, including model architecture, will be42# loaded from the checkpoint.43default_model_config_name = "src/configs/model/default_inference.yaml"44model_config = OmegaConf.load(default_model_config_name)45 46# checkpoint_name = which model to run47#   Single model version (no inference-time ensemble):48#   checkpoint_name='Blinorot/AL-Whisper-Instruct-R'49#   ALARM-E embedding fusion-type version (inference-time ensemble):50#   checkpoint_name=["Blinorot/ALARM-CA","Blinorot/AL-Whisper-Instruct-R"]51checkpoint_name = "Blinorot/AL-Whisper-R"52 53device = "cuda"54 55# Load Tokenizer for Text Processing56tokenizer = AutoTokenizer.from_pretrained(model_config.llm)57 58# Load ALARM/AL-*-R checkpoints for extraction of LLM input embeddings59if isinstance(checkpoint_name, list): # ALARM-E-style embedding fusion (inference-time ensemble)60    feature_extractor_list = []61    for name in checkpoint_name:62        # Load weights into the (audio,text)->LLM embeddings converter63        feature_extractor = Qwen3AudioWrappedFeatureExtractor(64            model_config=model_config,65            checkpoint_name=name,66            tokenizer=tokenizer,67        )68        feature_extractor.to(device)69        feature_extractor_list.append(feature_extractor)70    feature_extractor = feature_extractor_list71else: # Single Model version (no inference-time ensemble)72    # Load weights into the (audio,text)->LLM embeddings converter73    feature_extractor = Qwen3AudioWrappedFeatureExtractor(74        model_config=model_config,75        checkpoint_name=checkpoint_name,76        tokenizer=tokenizer,77    )78    feature_extractor.to(device)79 80# Start the offline vLLM instance of original Qwen3 RLM81# Model will be loaded to CUDA_VISIBLE_DEVICES id82llm = LLM(83    model_config.llm,84    enable_prefix_caching=True,85    max_model_len=model_config.max_model_len,86    max_num_seqs=model_config.max_num_seq,87    max_num_batched_tokens=model_config.max_num_batched_tokens,88    gpu_memory_utilization=model_config.gpu_memory_utilization,89    enable_prompt_embeds=True,90)91 92# Set sampling arguments for the RLM93sample = llm.get_default_sampling_params()94sample.seed = model_config.seed95sample.max_tokens = model_config.max_tokens96 97# Define audio and prompt98# Audio must come from torchcodec.AudioDecoder99audio_example_path = _download_asset("tutorial-assets/ctc-decoding/1688-142285-0007.wav")100audio = AudioDecoder(audio_example_path)101prompt = "Describe the audio content."102 103# Define a system prompt104system_prompt = "You are an audio-understanding model."105 106# Obtain response from Audio RLM107response = get_response(108    prompts=[prompt], # list of all the prompts109    audio_list=[audio], # list of corresponding audio110    llm=llm,111    feature_extractor=feature_extractor,112    sample=sample,113    tokenizer=tokenizer,114    system_prompt=system_prompt,115    max_thinking_tokens=model_config.max_thinking_tokens, # controls thinking budget for the RLM116    debug=False,117)118 119# Response is a list of responses, one per each (prompt, audio) input pair120# We have only one input pair, so the final response is at index 0121response = response[0]122 123print(f"Model response:\n\n{response}")124```125 126## Citation127 128If you use this work, please cite:129 130```bibtex131@article{grinberg2026alarm,132  title={ALARM: Audio-Language Alignment for Reasoning Models},133  author={Grinberg, Petr and Shahmohammadi, Hassan},134  journal={arXiv preprint arXiv:2603.09556},135  year={2026}136}137```138 139## License140 141The model checkpoint is licensed under Creative Commons Attribution-NonCommercial 4.0 (CC BY-NC 4.0).142It may only be used for non-commercial research purposes.