cara-ai/ALM2Vec-PT

<h1 align="center">ALM2Vec-PT</h1>

<p align="center"> <a href="https://arxiv.org/abs/2606.30682"><img src="https://img.shields.io/badge/Paper-arXiv-b31b1b?logo=arxiv&logoColor=white" alt="Paper"></a> <a href="https://caml-labs.github.io/ALM2Vec"><img src="https://img.shields.io/badge/Project-Page-1f6feb?logo=googlechrome&logoColor=white" alt="Project Page"></a> <a href="https://github.com/caml-labs/ALM2Vec"><img src="https://img.shields.io/badge/Code-GitHub-181717?logo=github&logoColor=white" alt="GitHub"></a> </p>

ALM2Vec is a universal audio embedding model for retrieval, derived from a pretrained large audio–language model (LALM). Instead of being optimized only for audio–caption matching like conventional contrastive dual-encoders, it transfers the audio understanding, instruction-following, and reasoning abilities of LALMs into a single unified embedding space that works across audio domains, task types, and user intents.

Its key feature is instruction-aware retrieval: a natural-language instruction guides the embedding, so the same audio can be encoded differently for different needs. This supports:

• —Instruction-aware retrieval — focus the embedding on a specific aspect of the audio.
• —Text ↔ audio retrieval — bidirectional matching between audio and text.
• —Audio question answering — match an audio query plus a question against candidate answers.

ALM2Vec achieves competitive results on standard audio and speech retrieval benchmarks while adding these controllable retrieval capabilities. See the project page for interactive demos.

This repository hosts the pretrain checkpoint, built on MiDashengLM.

Requirements: transformers>=4.52, torch, safetensors, and torchaudio for non-WAV audio. Requires a GPU (~31GB weights) and trust_remote_code=True.

Example

import torch
from transformers import AutoModel, AutoTokenizer

# switch between pretrain and finetune
repo_id = "cara-ai/ALM2Vec-PT"
# repo_id = "cara-ai/ALM2Vec-FT"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True, torch_dtype=torch.float32).cuda()
model.eval()

QUERY_INSTRUCTION = "Based on the question asked in the text query and context in the audio query, retrieve the relevant text document associated with that question."
DOC_INSTRUCTION = "Represent the user's input."

query_text = ["What is the gender of speaker in this audio?"] * 4

remote_prefix = "https://huggingface.co/cara-ai/ALM2Vec-PT/resolve/main/example/"
query_audio = [remote_prefix + "en_male_music.wav", remote_prefix + "en_male.wav", 
               remote_prefix + "en_female_music.wav", remote_prefix + "en_female.wav"]

doc_text = [
    "male",
    "female",
]

query_embeddings = model.encode(
    text=query_text,
    audio=query_audio,
    task="query",
    instruction=QUERY_INSTRUCTION,
    normalize=True,
    device="cuda",
)
doc_embeddings = model.encode(
    text=doc_text,
    task="document",
    instruction=DOC_INSTRUCTION,
    normalize=True,
    device="cuda",
)

similarity = query_embeddings @ doc_embeddings.T
print(similarity)

similarity = query_embeddings @ query_embeddings.T
print(similarity)

Results

ALM2Vec-PT is the checkpoint hosted in this repository; ALM2Vec-FT is the fine-tuned variant. In every table, bold marks the best score and <u>underline</u> the second best.

Text–audio retrieval — AudioCaps

Text–audio retrieval — Clotho

Speech retrieval — LibriSQA

Audio understanding — MMAU-mini (accuracy)

† LAION-CLAP is not trained for speech and effectively fails on LibriSQA; shown for reference. ‡ Generative large audio–language models, listed as reference upper bounds rather than directly comparable retrieval baselines.

Citation

If you find this work useful, please consider citing:

@misc{lu2026alm2veclearningaudioembeddings,
      title={ALM2Vec: Learning Audio Embeddings for Universal Audio Retrieval with Large Audio-Language Models}, 
      author={Fengjie Lu and Chenang Jiang and Jiarui Hai and Helin Wang and Aaron Yee},
      year={2026},
      eprint={2606.30682},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2606.30682}, 
}

Acknowledgement

ALM2Vec is built on MiDashengLM and further trained for universal audio retrieval. We thank MiDashengLM and its underlying Dasheng audio encoder for their open-source contributions.