Datasetpublic

rakshi719/SpokenCOCO-A2IT

SpokenCOCO Audio-to-(Image+Text) Retrieval MTEB/MOEB task where queries are spoken audio captions and corpus items contain both a MSCOCO image and its written text caption. Task Given a spoken audio description of an image, retrieve the correct (image, text) pair from the corpus. Only models that can process all three modalities — audio, image, and text — can exploit the full corpus signal. Contents Queries: 25031 spoken audio captions (WAV… See the full description on the dataset page: https://huggingface.co/datasets/rakshi719/SpokenCOCO-A2IT.

sourceHugging Facecc-by-4.0updated 16d agoView on Hugging Face

0likes116downloads

Dataset Card

SpokenCOCO Audio-to-(Image+Text) Retrieval

MTEB/MOEB task where queries are spoken audio captions and corpus items contain both a MSCOCO image and its written text caption.

Task

Given a spoken audio description of an image, retrieve the correct (image, text) pair from the corpus. Only models that can process all three modalities — audio, image, and text — can exploit the full corpus signal.

—Queries: 25031 spoken audio captions (WAV, ~16kHz)
—Corpus: 5000 (image, text) pairs from MS-COCO
—Qrels: 25031 binary relevance judgments

Construction

Audio queries and qrels are taken directly from the SpokenCOCO A2I task (whybe-choi/SpokenCOCOA2IRetrieval). The corpus images come from the same source; MSCOCO text captions are joined from jxie/coco_captions (first caption per image).

License

CC-BY-4.0 (inherits from MS-COCO and SpokenCOCO).

Citation

bibtex

@inproceedings{shih2023spokencoco,
  title={Speechclip: Integrating speech encoder and large vision-language
         model for spoken language understanding},
  author={Shih, Yi-Jen and others},
  booktitle={ASRU},
  year={2023},
}
@misc{lin2014coco,
  title={Microsoft COCO: Common Objects in Context},
  author={Lin, Tsung-Yi and others},
  year={2014},
  eprint={1405.0312},
}