CoolFace
Datasetpublic

seonglae/vls-10k

VLS 10K 9,987 MS-COCO images paired with a long written description, a one-sentence spoken summary of that description, the spoken audio, and that audio pre-encoded by two neural codecs. Images and audio are embedded in the parquet, so the viewer renders them and one call opens the set: from datasets import load_dataset ds = load_dataset("seonglae/vls-10k", split="train") ds[0]["image"] # PIL image ds[0]["audio"] # decoded waveform ds[0]["sst"] # the sentence that was… See the full description on the dataset page: https://huggingface.co/datasets/seonglae/vls-10k.

sourceHugging Faceupdated 28d agoView on Hugging Face
0likes173downloads
Dataset Card

VLS 10K

9,987 MS-COCO images paired with a long written description, a one-sentence spoken summary of that description, the spoken audio, and that audio pre-encoded by two neural codecs. Images and audio are embedded in the parquet, so the viewer renders them and one call opens the set:

python
from datasets import load_dataset
ds = load_dataset("seonglae/vls-10k", split="train")
ds[0]["image"]   # PIL image
ds[0]["audio"]   # decoded waveform
ds[0]["sst"]     # the sentence that was spoken

Columns

columntypemeaning
idxint32stable row id
splitstringtrain / validation / test
imageImagethe COCO image
audioAudiospeech rendering of sst (wav)
long_descstringlong written description of the image
sststringshort spoken text, a one-sentence summary of long_desc
processed_sststringsst after text normalisation
audio_tokens_mimiint32[8][T]Mimi codes, 8 codebooks at 12.5 Hz
audio_tokens_encodecint32[16][T]EnCodec codes, 16 codebooks

Both token columns are [codebooks][frames], so their shape carries the frame and codebook counts directly.

Notes

Every row carries two independent tokenizations of the same waveform, which makes the set usable for comparing codecs with the text, image and audio held fixed.

The split ships as a column and as separate parquet files, so it never has to be re-derived and cannot drift between users.

Images come from MS-COCO and keep its terms. The descriptions, spoken summaries and audio are generated.

A larger companion set with 74,936 rows is at `seonglae/vls-100k`.