BSC-LT/salamandra-TAV-7b

<!-- [image] -->

SalamandraTAV-7b Model Card

SalamandraTAV-7b is a Speech To Text translation Speech LLM that has been finetuned from salamandraTA-7b-instruct. SalamandraTAV-7b is proficient in 6 Iberic languages plus English, and supports the following translation-related tasks: Automatic Speech Recognition, Text To Text Translation, Speech To Text Translation, Spoken Language Identification.

[!WARNING] DISCLAIMER: This version of Salamandra is tailored exclusively for speech translation tasks. It lacks chat capabilities and has not been trained with any chat instructions.

<details> <summary>Table of contents</summary>

• —Model Details
• —Intended Use
• —Hardware and Software
• —How to use
• —Training Data
• —Evaluation
• —Additional Information

</details>

Model Details

Description

SalamandraTAV-7b is a finetuned version of salamandraTA-7b-instruct using mhubert-base-25hz as the speech encoder. It has been trained on 12,000 hours of Automatic Speech Recognition, 900 hours of Speech-To-Text Translation data and 113M target tokens of Text-To-Text Translation to mantain the original model's performance.

Intended Use

The model is intended for both research and commercial use for the speech to text translation task in the following languages: Asturian, Basque, Catalan, English, Galician, Portuguese and Spanish.

Hardware and Software

Training Framework

The code used to train SalamandraTAV-7b is based on the Transformers library, and will be publicly available soon.

Compute Infrastructure

This model was trained on MareNostrum 5, a pre-exascale EuroHPC supercomputer hosted and operated by Barcelona Supercomputing Center.

Training was conducted on 4 nodes, each with the following specifications:

• —4x Nvidia Hopper GPUs with 64GB HBM2 memory
• —2x Intel Sapphire Rapids 8460Y+ at 2.3Ghz and 32c each (64 cores)
• —4x NDR200 (BW per node 800Gb/s)
• —512 GB of Main memory (DDR5)

How to use

[!NOTE] Requirements To use this model, ensure you have the following Python packages installed: transformers, torch, torchaudio, joblib, sentencepiece, protobuf, scikit-learn, langcodes, soundfile

The easiest way to use the model is using the custom pipeline multimodal_mt:

from transformers import pipeline
pipe = pipeline(
  task="multimodal_mt",
  model="langtech-veu/salamandra-TAV-7b",
  trust_remote_code=True
)

Specify an audio file (can be a URL or a local file path):

audio_path = "https://github.com/voxserv/audio_quality_testing_samples/raw/refs/heads/master/orig/127389__acclivity__thetimehascome.wav" # in English

Define the generation parameters:

generation_kwargs = {
  "beam_size": 5,
  "max_new_tokens": 100
}

Direct Speech-to-Text Translation (S2TT)

Run the S2TT pipeline, specifying the target language:

translation = pipe(audio_path, mode="s2tt", tgt_lang="Spanish", **generation_kwargs)

Optionally, you can also specify the source language:

translation = pipe(audio_path, mode="s2tt", src_lang="English", tgt_lang="Spanish", **generation_kwargs)

Cascaded Speech-to-Text Translation (ASR+T2TT)

Run the S2TT pipeline, specifying the target language:

transcription = pipe(audio_path, mode="asr", **generation_kwargs)
translation = pipe(transcription, mode="t2tt", tgt_lang="Spanish", **generation_kwargs)

Optionally, you can also specify the source language:

transcription = pipe(audio_path, mode="asr", src_lang="English", **generation_kwargs)
translation = pipe(transcription, mode="t2tt", src_lang="English", tgt_lang="Spanish", **generation_kwargs)

Speech-to-Text Translation with CoT (S2TT-CoT)

This is a variant which uses a CoT mechanism to generate the translation by transcribing it first.

Run the S2TT pipeline, specifying the target language:

history = pipe(audio_path, return_chat_history=True, mode="asr", **generation_kwargs)
translation = pipe(history, mode="t2tt", tgt_lang="Spanish", **generation_kwargs)

Optionally, you can also specify the source language:

history = pipe(audio_path, return_chat_history=True, mode="asr", src_lang="English", **generation_kwargs)
translation = pipe(history, mode="t2tt", src_lang="English", tgt_lang="Spanish", **generation_kwargs)

Advanced Usage

If you are interested in getting the intermediate results, you can do it as follows:

history = pipe(audio_path, return_chat_history=True, mode="asr", **generation_kwargs)
transcription = history.get_assistant_messages()[-1]
history = pipe(history, return_chat_history=True, mode="t2tt", tgt_lang="Spanish", **generation_kwargs)
translation = history.get_assistant_messages()[-1]
history = pipe(history, return_chat_history=True, mode="lid", **generation_kwargs)
src_language = history.get_assistant_messages()[-1]

Training Data

Global Summary

Automatic Speech Recognition

[!NOTE] To ensure reliable evaluation, we excluded from the Common Voice 21.0 train set all samples that overlap with the CoVoST 2 dev and test sets.

Speech-To-Text Translation

[!NOTE] To ensure reliable evaluation, we excluded from the CoVoST 2 train set all samples that overlap with the Common Voice 21.0 dev and test sets.

Text-To-Text Translation

For T2TT data, we filtered Wikimedia and Tatoeba datasets using the following criteria:

1. 1.Sample obtained a GlotLID v3 target language probability of at least 50%
2. 2.Sample contains between 5 and 100 words
3. 3.Sample obtained a BLASER 2.0 score higher than 3.75

Obtaining 102,845,818 target tokens for Wikimedia and 9,991,305 target tokens for Tatoeba, with the following language distribution:

Evaluation

Below are the SalamandraTAV-7b evaluation results on the Fleurs, mintzai-ST, CoVoST 2 and Common Voice test sets, compared against the state-of-the-art models: SeamlessM4T, Whisper and SpireFull. We report the following metrics:

<details> <summary>Click to show metrics details</summary>

• —BLEU: sacrebleu implementation. Signature: nrefs:1 | case:mixed | eff:no | tok:13a | smooth:exp | version:2.5.1.
• —XComet-XL: COMET model checkpoint: Unbabel/XCOMET-XL.
• —WER: jiwer implementation. Applied the following normalization:

def normalize_text_list(text_list: List[str]) -> List[str]:
  normalized_list = []
  for t in text_list:
    # Apostrophes and quotation marks
    t = t.replace("’", "'").replace("‘", "'").replace("`", "'")
    t = t.replace("“", '"').replace("”", '"')
    # Hyphens
    t = t.replace("–", "-").replace("—", "-")
    # Ellipses
    t = t.replace("…", "...")
    # Non-breaking spaces
    t = t.replace("\u00A0", " ")
    normalized_list.append(t)
  return normalized_list
transformation = jiwer.Compose([
    normalize_text_list,
    jiwer.ToLowerCase(),
    jiwer.ExpandCommonEnglishContractions(),
    jiwer.RemovePunctuation(),
    jiwer.RemoveWhiteSpace(replace_by_space=True),
    jiwer.RemoveMultipleSpaces(),
    jiwer.Strip(),
    jiwer.ReduceToSingleSentence(),
    jiwer.ReduceToListOfListOfWords()
])

</details>

Speech-To-Text Translation

Results reported in bold indicate better performance than SeamlessM4T. The heatmap matrices indicate the difference in scores of each language direction.

<details> <summary>Fleurs Test</summary>

Fleurs Test

BLEU

SalamandraTAV vs SeamlessM4T BLEU Difference

[image]

The evaluation results reported here were obtained by manually running inference with the model on the test set. We verified that the results from SeamlessM4T are consistent with the official results reported by the authors.

XCOMET-XL

SalamandraTAV vs SeamlessM4T XCOMET-XL Difference

[image]

XCOMET-XL results for Asturian are not reported because it is not supported by this metric.

</details>

<details> <summary>CoVoST 2 Test</summary>

CoVoST 2 Test

BLEU

SalamandraTAV vs SeamlessM4T BLEU Difference

[image]

XCOMET-XL

SalamandraTAV vs SeamlessM4T XCOMET-XL Difference

[image]

</details>

<details> <summary>Mintzai-ST Test</summary>

Mintzai-ST Test

Mintzai-ST has overlap with basqueparliament1, with which we have trained our model.

BLEU

SalamandraTAV vs SeamlessM4T BLEU Difference

[image]

XCOMET-XL

SalamandraTAV vs SeamlessM4T XCOMET-XL Difference

[image]

</details>

Automatic Speech Recognition

<details> <summary>Common Voice 21.0 Test</summary>

Common Voice 21.0 Test

WER

</details>

<details> <summary>Fleurs Test</summary>

Fleurs Test

WER

</details>

<details> <summary>Mintzai-ST Test</summary>

Fleurs Test

WER

</details>

Additional information

Authors

All the members of the SalamandraTAV Team are members of the Language Technologies Lab at the Barcelona Supercomputing Center (BSC).

• —Core Contributors <small>(equal contribution, alphabetical order)</small>
• —Gerard I. Gállego
• —Oriol Pareras

• —Team Coordination
• —Federico Costa

• —Lead <small>(equal contribution, alphabetical order)</small>
• —Cristina España-Bonet
• —Javier Hernando

Contact

For further information, please send an email to <bsc-lt@bsc.es>.

Copyright

Copyright(c) 2025 by Language Technologies Lab, Barcelona Supercomputing Center.

Funding

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Modelos del Lenguaje.

Acknowledgements

The author thankfully acknowledges the computer resources at MareNostrum and the technical support provided by Barcelona Supercomputing Center (RES-IM-2025-2-0027).

Disclaimer

Be aware that the model may contain biases or other unintended distortions. When third parties deploy systems or provide services based on this model, or use the model themselves, they bear the responsibility for mitigating any associated risks and ensuring compliance with applicable regulations, including those governing the use of Artificial Intelligence.

The Barcelona Supercomputing Center, as the owner and creator of the model, shall not be held liable for any outcomes resulting from third-party use.

License

Apache License, Version 2.0

Citation

If you find our model useful, we would appreciate if you could cite our work as follows:

@misc{bsclt2025salamandraTAV7b ,
title={salamandra-TAV-7b: a Speech-To-Text Translation model based on an end-to-end Speech LLM for Iberian Languages.},
author={SalamandraTAV Team},
organization={Barcelona Supercomputing Center},
url={https://huggingface.co/langtech-veu/salamandra-TAV-7b},
year={2025}
}