Datasetpublic

chuonghm/OmniRet-train

OmniRet training dataset OmniRet-train is the training-data release for OmniRet, a unified retrieval model for text, image, video, and audio. This card documents the released snapshot for researchers training or analyzing OmniRet. Dataset summary The release contains 6,405,109 query rows and 7,119,841 candidate rows from 30 datasets. It covers 15 retrieval directions across text (T), image (I), video (V), and audio (A). The OmniRet paper reports this corpus as… See the full description on the dataset page: https://huggingface.co/datasets/chuonghm/OmniRet-train.

sourceHugging Faceupdated 1mo agoView on Hugging Face

0likes11kdownloads

Dataset Card

OmniRet training dataset

OmniRet-train is the training-data release for OmniRet, a unified retrieval model for text, image, video, and audio. This card documents the released snapshot for researchers training or analyzing OmniRet.

Dataset summary

The release contains 6,405,109 query rows and 7,119,841 candidate rows from 30 datasets. It covers 15 retrieval directions across text (T), image (I), video (V), and audio (A). The OmniRet paper reports this corpus as approximately 6.4 million query-candidate pairs.

Provenance

OmniRet starts from the M-BEIR universal retrieval benchmark and extends it with text, image-text, video-text, audio-text, and audio-visual training data. The source groups follow the training-data section and Table 10 of the OmniRet paper.

Source group	Datasets
M-BEIR foundation	NIGHTS, WebQA, VisualNews, Fashion200K, MSCOCO, EDIS, OVEN, InfoSeek, FashionIQ, CIRR
Added text retrieval	MS MARCO, HotpotQA, Natural Questions, PAQ, StackExchange, NLI, SQuAD
Added image-text and composed image retrieval	LLaVA-558K, CC-CoIR, MTCIR
Added video-text and composed video retrieval	TGIF, Charades, WebVid2M, PE-Video, WebCoVR
Added audio-text retrieval	AudioCaps, Clotho v2.1, WavText5K, WavCaps
Added audio-visual retrieval	VGGSound

Retrieval tasks

Family	Directions
Unimodal	I → I; T → T
Cross-modal binding	I → T; T → I; V → T; T → V; A → T; T → A
Composed retrieval	T → (I,T); (I,T) → T; (I,T) → I; (I,T) → (I,T); (V,T) → V
Audio-visual binding	A → V; V → A

Repository layout

Path	Contents
`queries/`	Training queries, task IDs, positive candidate IDs, text, and media references
`candidates/`	Candidate records referenced by the queries
`instructions/`	Retrieval instructions keyed by task and dataset
`media_shard_*/`	Sharded image, video, and audio payloads
`metadata/candidate_index.sqlite`	Read-only lookup index for 7,119,841 candidates
`metadata/media_index.sqlite`	Read-only lookup index for 3,538,323 media IDs across 1,976 Parquet files
`metadata/train_pairs.jsonl`	Precomputed manifest containing 6,405,109 training pairs

Download the repository to the canonical training path:

bash

hf download chuonghm/OmniRet-train \
  --repo-type dataset \
  --local-dir /data1/omniret/OmniRet-train-data

The published media index stores Parquet paths below /data1/omniret/OmniRet-train-data. Use that location, or provide a compatible media index when using another root.

Relationship to the ACM benchmark

The OmniRet paper also introduces the separate Audio-Centric Multimodal benchmark (ACM), curated from VGGSound. ACM evaluates composed audio retrieval (A,T → A) and bidirectional audio-image and audio-video retrieval (A → I, I → A, A → V, V → A). Use chuonghm/ACM for evaluation; this training repository does not replace the benchmark release.

Data terms

This repository is a transformed compilation of upstream datasets. Underlying examples remain subject to their original licenses and terms. Review the M-BEIR dataset card and the source references in Table 10 of the OmniRet paper before redistribution or commercial use.

Citation

Please cite OmniRet for this extended training release and ACM benchmark, and cite UniIR for the M-BEIR foundation.

bibtex

@article{huynh2026omniret,
  title   = {Efficient and High-Fidelity Omni Modality Retrieval},
  author  = {Huynh, Chuong and Luong, Manh and Shrivastava, Abhinav},
  journal = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year    = {2026}
}

@article{wei2023uniir,
  title   = {UniIR: Training and Benchmarking Universal Multimodal Information Retrievers},
  author  = {Wei, Cong and Chen, Yang and Chen, Haonan and Hu, Hexiang and Zhang, Ge and Fu, Jie and Ritter, Alan and Chen, Wenhu},
  journal = {arXiv preprint arXiv:2311.17136},
  year    = {2023}
}