FBK-MT/fama-data

<img src="https://huggingface.co/FBK-MT/fama-small/resolve/main/FAMA.png" align="center" width="100%">

Dataset Description, Collection, and Source

The FAMA training data is the collection of English and Italian datasets for automatic speech recognition (ASR) and speech translation (ST) used to train the FAMA models family. The ASR section of FAMA is derived from the MOSEL data collection, including the automatic transcripts obtained with Whisper and available in the HuggingFace MOSEL Dataset. The ASR is further augmented with automatically transcribed speech from the YouTube-Commons dataset. The ST section is composed of gold-labeled ST datasets and the automatic translations of the ASR datasets with MADALAD-400 3B-MT. The complete list of datasets for both tasks are reported in the Dataset Statistics.

• —Curated by: Sara Papi, Marco Gaido, Luisa Bentivogli, Alessio Brutti, Mauro Cettolo, Roberto Gretter, Marco Matassoni, Mohamed Nabih, and Matteo Negri
• —Funded by: FAIR, Meetween, and CINECA
• —Shared by: Fondazione Bruno Kessler

License

• —CC-BY-4.0

Dataset Sources

• —MOSEL Collection: MOSEL GitHub
• —MOSEL Pseudolabels: MOSEL HuggingFace
• —YouTube-Commons: YouTube-Commons
• —Paper: FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian

Dataset Structure

Data Config

The dataset is split into multiple tsv files corresponding to the dataset name and the source and target languages, either Italian (it) and English (en), containing both the ASR transcript and translation in the other language.

Data Field

id: unique id of the segment (text, e.g.: "5NTUCHeZuds_0")

audio: filename (text, e.g. "5NTUCHeZuds.wav")

offset: start of the segment, in seconds (float, e.g. "0.020")

duration: duration of the segments, in seconds (float, e.g. "5.946")

speaker: id of the speaker (text, e.g. "000")

src_lang: id of the source language (ISO 639-1 code, e.g. "it", "en")

src_text: recognized text (text, e.g. "Grazie a tutti.")

tgt_lang: id of the source language (ISO 639-1 code, e.g. "it", "en")

tgt_text: translated text (text, e.g. "Thank you all.")

ASR: True/False - indicates whether the sample has been used for ASR training

ST: True/False - indicates whether the sample has been used for ST training

Dataset Statistics

The full list of FAMA training datasets, together with the number of hours for each language/language pair and the type of labels (A for automatic and G for gold labels) is reported below for both ASR and ST tasks.

Automatic Speech Recognition (ASR)

Speech Translation (ST)

Dataset Creation

To reproduce the MOSEL-derived datasets (all but YouTube-Commons), please refer to the MOSEL README in the fbk-llm repository and to the MOSEL data card on HuggingFace.

To download and process YouTube-Commons, please refer to the dedicated YouTube-Commons README.

The code used to produce all translations with MADALAD-400 3B-MT is the following:

import os
import sys
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

modelname = "google/madlad400-3b-mt"
batch_size = {$BATCH_SIZE}   
tlang = {$LANGUAGE}

class BatchedMT:
    def __init__(self, tokenizer, model):
        self.buffer_lines = []
        self.model = model
        if torch.cuda.is_available():
            self.model = self.model.cuda()
        self.tokenizer = tokenizer

    def process_line(self, line):
        self.buffer_lines.append(line.strip())
        if len(self.buffer_lines) >= BATCHSIZE:
            self.print_translations()
            self.buffer_lines = []

    def print_translations(self):
        outs = self._do_translate()
        for s in outs:
            print(s)

    def _do_translate(self):
        tokens = self.tokenizer(self.buffer_lines, return_tensors="pt", padding=True)
        if torch.cuda.is_available():
            tokens = {k: v.cuda() for k, v in tokens.items()}
        translated = self.model.generate(**tokens, max_new_tokens=512)
        return [self.tokenizer.decode(t, skip_special_tokens=True) for t in translated]

    def close(self):
        if len(self.buffer_lines) > 0:
            self.print_translations()
            self.buffer_lines = []


mt = BatchedMT(
    AutoTokenizer.from_pretrained(modelname),
    AutoModelForSeq2SeqLM.from_pretrained(modelname))

for input_line in sys.stdin:
    mt.process_line("<2" + tlang + "> " + input_line)
mt.close()

where the input text is passad as stdin, {$BATCH_SIZE} is the batch size supported on your machine and {$LANGUAGE} is either en for Italian to English translation and it for English to Italian translation.

The script used for filtering the ST datasets is `filter_tsv_based_on_ratio` and available in the scripts folder of this repository. For English speech datasets, we set --threshold-min 0.75 and --threshold-max 1.45 while, for the Italian speech datasets, --threshold-min 0.65 and --threshold-max 1.35.

Citation

@misc{papi2025fama,
      title={FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian}, 
      author={Sara Papi and Marco Gaido and Luisa Bentivogli and Alessio Brutti and Mauro Cettolo and Roberto Gretter and Marco Matassoni and Mohamed Nabih and Matteo Negri},
      year={2025}
}

Dataset Card Contact

@spapi