datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
MultiArithmultilingual-speech-commands-15lang
Multilingual Speech Commands Dataset (15 Languages, Augmented)
This dataset contains augmented speech command samples in 15 languages, derived from multiple public datasets. Only commands that overlap with the Google Speech Commands (GSC) vocabulary are included, making the dataset suitable for multilingual keyword spotting tasks aligned with GSC-style classification.
Audio samples have been augmented using standard audio techniques to improve model robustness (e.g., time-shifting… See the full description on the dataset page: https://huggingface.co/datasets/artur-muratov/multilingual-speech-commands-15lang.VDR_MEGA_MultiDomain_DocRetrieval
Visual Document Retrieval Dataset
Overview
This dataset is designed for training visual document retrieval models. It combines multiple datasets from the VDR series, Colpali, and LlamaIndex to create the most comprehensive training resource for visual document retrieval tasks.
Dataset Structure
The dataset contains structured fields including unique identifiers with string lengths ranging from 45 to 50 characters, search query text with variable lengths between… See the full description on the dataset page: https://huggingface.co/datasets/racineai/VDR_MEGA_MultiDomain_DocRetrieval.SWE-bench_Multilingual
SWE-bench Multilingual
Dataset Summary
SWE-bench Multilingual is a dataset that tests systems' ability to resolve real-world GitHub issues across a broad range of programming languages. The original SWE-bench is Python-only; this dataset extends the same task format to 9 languages drawn from 41 popular repositories.
The dataset collects 300 test Issue-Pull Request pairs. Evaluation is performed by unit test verification, using post-PR behavior as the reference solution.
The… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench/SWE-bench_Multilingual.MultiPL-E
Dataset Card for MultiPL-E
Dataset Summary
MultiPL-E is a dataset for evaluating large language models for code
generation that supports 22 programming languages. It takes the OpenAI
HumanEval and the Mostly Basic Python Programs (MBPP) benchmarks and uses little compilers to
translate them to other languages. It is easy to add support for new languages
and benchmarks.
The dataset is divided into several configurations named SRCDATA-LANG, where
SRCDATA is either… See the full description on the dataset page: https://huggingface.co/datasets/nuprl/MultiPL-E.multi_dir_datasetmultispan_quorefmulti_nli
Dataset Card for Multi-Genre Natural Language Inference (MultiNLI)
Dataset Summary
The Multi-Genre Natural Language Inference (MultiNLI) corpus is a
crowd-sourced collection of 433k sentence pairs annotated with textual
entailment information. The corpus is modeled on the SNLI corpus, but differs in
that covers a range of genres of spoken and written text, and supports a
distinctive cross-genre generalization evaluation. The corpus served as the
basis for the shared task… See the full description on the dataset page: https://huggingface.co/datasets/nyu-mll/multi_nli.multilingual_librispeech
Dataset Card for MultiLingual LibriSpeech
Dataset Summary
This is a streamable version of the Multilingual LibriSpeech (MLS) dataset.
The data archives were restructured from the original ones from OpenSLR to make it easier to stream.
MLS dataset is a large multilingual corpus suitable for speech research. The dataset is derived from read audiobooks from LibriVox and consists of
8 languages - English, German, Dutch, Spanish, French, Italian, Portuguese, Polish.… See the full description on the dataset page: https://huggingface.co/datasets/facebook/multilingual_librispeech.MultiLegalPile_Wikipedia_FilteredA filtered version of the MultiLegalPile dataset, together with wikipedia articles.Multitask-National-Speech-Corpus-v1Multitask-National-Speech-Corpus (MNSC v1) is derived from IMDA's NSC Corpus.
MNSC is a multitask speech understanding dataset derived and further annotated from IMDA NSC Corpus. It focuses on the knowledge of Singapore's local accent, localised terms, and code-switching.
ASR: Automatic Speech Recognition
SQA: Speech Question Answering
SDS: Spoken Dialogue Summarization
PQA: Paralinguistic Question Answering
from datasets import load_dataset
data =… See the full description on the dataset page: https://huggingface.co/datasets/MERaLiON/Multitask-National-Speech-Corpus-v1.MultiLangThis is the multi - language version of SWE-bench-Live (repository_link ; paper_link).
We crawled GitHub issues as SWE tasks for coding agents to solve.
We used RepoLaunch (repository_link) to setup docker images as the problem-solving sandboxes for each instance.
Each split is task set of one language.
Each split is filtered by prompting LLM -- a task instance is discarded if LLM cannot infer test patch from problem statement or problem statement contains solution (in NL or code).… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench-Live/MultiLang.MultiTalkFT
MultiTalkFT
Fine-tuning corpus for full-duplex multi-speaker dialogue.
Schemas
data_{zh,en}{,_multichannel}.jsonl (one record per line):
field
type
description
path
string
relative path to the audio file
voice
string
relative path to speaker prompt
duration
float
clip duration in seconds
system
string
persona / system prompt
transcripts/*.parquet:
column
type
description
audio_path
string
matches data_*.jsonl path
idstring
duration… See the full description on the dataset page: https://huggingface.co/datasets/MultiTalk/MultiTalkFT.multiun
Dataset Card for OPUS MultiUN
Dataset Summary
The MultiUN parallel corpus is extracted from the United Nations Website , and then cleaned and converted to XML at Language Technology Lab in DFKI GmbH (LT-DFKI), Germany. The documents were published by UN from 2000 to 2009.
This is a collection of translated documents from the United Nations originally compiled by Andreas Eisele and Yu Chen (see http://www.euromatrixplus.net/multi-un/).
This corpus is available in all 6… See the full description on the dataset page: https://huggingface.co/datasets/Helsinki-NLP/multiun.MultiBanana-Benchmark🍌 MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generation 🍌
CVPR 2026 (Main)
This repository provides the datasets for
“MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generation” by Yuta Oshima, Daiki Miyake, Kohsei Matsutani, Yusuke Iwasawa, Masahiro Suzuki, Yutaka Matsuo and Hiroki Furuta
Paper Link
https://arxiv.org/abs/2511.22989
Github Repository
For the usage of this benchmark, please see Github… See the full description on the dataset page: https://huggingface.co/datasets/kohsei/MultiBanana-Benchmark.SWE-bench_Multimodal
SWE-bench Multimodal
Dataset Summary
SWE-bench Multimodal is a dataset that tests systems' ability to resolve real-world GitHub issues in visual software domains. Unlike the original SWE-bench, which is Python-only and text-only, every task instance here comes from a JavaScript or TypeScript repository and carries at least one image asset — a screenshot, a screen recording, a diagram, or a rendering of incorrect output.
The dataset collects 612 Issue-Pull Request pairs from 17… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench/SWE-bench_Multimodal.Vript_Multilingual
🎬 Vript: A Video Is Worth Thousands of Words [Github Repo]
We construct another fine-grained video-text dataset with 19.1K annotated high-resolution UGC videos (~677k clips) in multiple languages to be the Vript_Multilingual.
New in Vript_Multilingual:
Multilingual: zh (60%), en (17%), de (15%), ja (6%), ko (2%), ru (<1%), es (<1%), pt (<1%), jv (<1%), fr (<1%), id (<1%), vi (<1%)
More diverse and fine-grained categories: 113 categories (please check vript_CN-V2_meta.json)… See the full description on the dataset page: https://huggingface.co/datasets/Mutonix/Vript_Multilingual.Multi-SWE-benchdisrpt
Disrpt is a multilingual, multi-framework unified discourse analysis benchmark.
It unifies discourse relation classification tasks (.rels) and discourse segmentation (.connlu) for many languages.
⚠️ This repo only contains the disrpt dataset when the underlying data is permissively licensed. Some datasets rely on corpora like the PTB.
To load these datasets, run the following:
pip install disrpt-utils
Then
from disrpt_utils import load_dataset
corpora_paths={
# ⚠️✍️ TODO Input… See the full description on the dataset page: https://huggingface.co/datasets/multilingual-discourse-hub/disrpt.wikipedia-2023-11-embed-multilingual-v3
Multilingual Embeddings for Wikipedia in 300+ Languages
This dataset contains the wikimedia/wikipedia dataset dump from 2023-11-01 from Wikipedia in all 300+ languages.
The individual articles have been chunked and embedded with the state-of-the-art multilingual Cohere Embed V3 embedding model. This enables an easy way to semantically search across all of Wikipedia or to use it as a knowledge source for your RAG application. In total is it close to 250M paragraphs / embeddings.
You… See the full description on the dataset page: https://huggingface.co/datasets/CohereLabs/wikipedia-2023-11-embed-multilingual-v3.Multimodal-Mind2Web
Dataset Summary
Multimodal-Mind2Web is the multimodal version of Mind2Web, a dataset for developing and evaluating generalist agents
for the web that can follow language instructions to complete complex tasks on any website. In this dataset, we align each HTML document in the dataset with
its corresponding webpage screenshot image from the Mind2Web raw dump. This multimodal version addresses the inconvenience of loading images from the ~300GB Mind2Web Raw Dump.
Dataset… See the full description on the dataset page: https://huggingface.co/datasets/osunlp/Multimodal-Mind2Web.multimodal-ct-radiology-reports
Perle AI Multi-phase CECT and CT with Radiology Reports
Summary
A de-identified CT dataset from Perle AI, paired with the original radiology reports. It supports work on multi-modal medical imaging: phase or pathology classification, report generation from images, and visual question answering.
The release has three configurations:
Config
Modality
Subjects
Pairing
cect_3phase
3-phase contrast-enhanced abdominal CT (DICOM)
5
per-subject text report +… See the full description on the dataset page: https://huggingface.co/datasets/Perle-ai/multimodal-ct-radiology-reports.multilingual-embeddings-pre-training-curated
📚 Collection | 📝 Multilingual Blog | 📝 English Blog
Contrastive Multilingual Pre-Training
2.16B query–document pairs across eight languages, plus cross-lingual pairs
mDenseOn |
mLateOn |
DenseOn |
LateOn |
PyLate |
FastPlaid
🎯 TL;DR: The multilingual contrastive pre-training corpus used to train mDenseOn and mLateOn. It extends our curated English data recipe (embeddings-pre-training-curated) to French, German, Italian, Spanish, Portuguese, Swedish, Norwegian, and Arabic… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/multilingual-embeddings-pre-training-curated.Multi-SWE-bench
Multi-SWE-bench
Re-upload of ByteDance's Multi-SWE-bench
evaluation benchmark: 2,132 issue-resolving tasks across the seven Multi-SWE languages.
This is the held-out eval benchmark; for RL training data use
PrimeIntellect/Multi-SWE-RL-Verified.
Changes vs upstream
Storage schema only: per-test maps are stored as columnar struct-of-lists so the rows load
cleanly with datasets. Row content is unchanged.
License mirrors upstream: ByteDance licenses the dataset… See the full description on the dataset page: https://huggingface.co/datasets/PrimeIntellect/Multi-SWE-bench.Zebra-CoT
Zebra‑CoT
A diverse large-scale dataset for interleaved vision‑language reasoning traces.
Dataset Description
Zebra‑CoT is a diverse large‑scale dataset with 182,384 samples containing logically coherent interleaved text‑image reasoning traces across four major categories: scientific reasoning, 2D visual reasoning, 3D visual reasoning, and visual logic & strategic games.
Dataset Structure
Each example in Zebra‑CoT consists of:
Problem statement:… See the full description on the dataset page: https://huggingface.co/datasets/multimodal-reasoning-lab/Zebra-CoT.SWE-bench_Multimodal
SWE-bench Multimodal
SWE-bench Multimodal is a dataset of 617 task instances that evalutes Language Models and AI Systems on their ability to resolve real world GitHub issues.
To learn more about the dataset, please visit our website.
More updates coming soon!
Multilingual-Thinking
Dataset summary
Multilingual-Thinking is a reasoning dataset where the chain-of-thought has been translated from English into one of 4 languages: Spanish, French, Italian, and German. The dataset was created by sampling 1k training samples from the SystemChat subset of SmolTalk2 and translating the reasoning traces with another language model.
This dataset was used in the OpenAI Cookbook to fine-tune the OpenAI gpt-oss models.
You can load the dataset using:
from datasets import… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceH4/Multilingual-Thinking.MultiTalkPT
MultiTalkPT
Pre-training corpus for full-duplex spoken-dialogue models.
Schemas
data_{zh,en}.jsonl (one record per line):
field
type
description
path
string
relative path to the dialogue audio
voice
string
relative path to the speaker prompt audio
duration
float
dialogue duration in seconds
system
string
persona / system prompt
transcripts/*.parquet:
column
type
description
audio_path
string
matches data_*.jsonl path
id
string… See the full description on the dataset page: https://huggingface.co/datasets/MultiTalk/MultiTalkPT.zen-multi-imageomega-multimodal
OMEGA Labs Bittensor Subnet: Multimodal Dataset for AGI Research
Introduction
The OMEGA Labs Bittensor Subnet Dataset is a groundbreaking resource for accelerating Artificial General Intelligence (AGI) research and development. This dataset, powered by the Bittensor decentralized network, aims to be the world's largest multimodal dataset, capturing the vast landscape of human knowledge and creation.
With over 1 million hours of footage and 30 million+ 2-minute… See the full description on the dataset page: https://huggingface.co/datasets/omegalabsinc/omega-multimodal.