lv12/MultiModalDataset

Dataset Card for MultiModal Dataset

Table of Contents

1. 1.Dataset Card for MultiModal Dataset
2. 2.Table of Contents
3. 3.Dataset Description
4. 4.Dataset Summary
5. 5.Supported Tasks
6. 6.Languages
7. 7.Dataset Structure
8. 8.Data Instances
9. 9.Data Fields
10. 10.FineWeb Subset
11. 11.COCO Subset
12. 12.AudioSet Subset
13. 13.Data Splits
14. 14.Dataset Creation
15. 15.Curation Rationale
16. 16.Source Data
17. 17.Initial Data Collection and Normalization
18. 18.Who are the source language producers?
19. 19.Annotations
20. 20.Annotation process
21. 21.Who are the annotators?
22. 22.Personal and Sensitive Information
23. 23.Considerations for Using the Data
24. 24.Social Impact of Dataset
25. 25.Discussion of Biases
26. 26.Other Known Limitations
27. 27.Additional Information
28. 28.Dataset Curators
29. 29.Licensing Information
30. 30.Citation Information
31. 31.Contributions
32. 32.Quick Start

Dataset Description

Dataset Summary

MultiModal Dataset is a curated collection of 85,000 samples spanning three modalities: text, images, and audio. It combines high-quality web content, image-caption pairs from COCO 2017, and audio samples from AudioSet to enable comprehensive multimodal model training and evaluation.

The dataset is organized into three subsets:

• —fineweb: 37,500 high-quality web text samples (>8,192 tokens each)
• —coco: 37,500 image-caption pairs from COCO 2017 (512x512 resolution)
• —audioset: 10,000 audio samples with human-annotated labels

Text and image subsets follow an 80/13/7 train/validation/test split. AudioSet uses 80/15/5 split (8K train, 1.5K valid, 500 test).

Supported Tasks

• —Text Generation: Long-form text generation using the fineweb subset
• —Image Captioning: Generate descriptions for images using the coco subset
• —Audio Classification: Classify audio events using the audioset subset
• —Multimodal Pretraining: Train models that understand multiple modalities
• —Cross-Modal Retrieval: Retrieve relevant content across modalities
• —Zero-Shot Classification: Evaluate zero-shot capabilities across modalities

Languages

English (en)

Dataset Structure

Data Instances

FineWeb (text) example:

{
  "id": 42,
  "text": "The history of artificial intelligence begins with ancient myths and stories of artificial beings endowed with intelligence or consciousness by master craftsmen. The seeds of modern AI were planted by classical philosophers who attempted to describe human thinking as a symbolic system...",
  "modality": "text"
}

COCO (image+text) example:

{
  "id": 156,
  "text": "A person riding a bicycle on a city street during sunset",
  "image": "<PIL.Image.Image image mode=RGB size=512x512>",
  "modality": "image"
}

AudioSet (audio+text) example:

{
  "id": 89,
  "text": "Speech, Music, Piano",
  "audio": {
    "array": [0.001, -0.002, 0.003, ...],
    "sampling_rate": 16000
  },
  "modality": "audio"
}

Data Fields

FineWeb Subset

COCO Subset

AudioSet Subset

Data Splits

Text and image subsets use identical 80/13/7 split ratios. AudioSet uses 80/15/5:

Split methodology:

• —fineweb/coco: Deterministic index-based partitioning (80% train, 13% valid, 7% test)
• —audioset: Uses AudioSet's built-in splits (bal_train → train, eval → valid/test)
• —Test: Final 7% of samples (indices 0.93×N to N)

This ensures reproducible splits across different loading sessions.

Dataset Creation

Curation Rationale

This dataset was created to address the need for a unified multimodal benchmark that:

1. 1.Provides high-quality data across modalities: Each subset is sourced from established, well-curated datasets
2. 2.Maintains consistent splits: Identical split ratios enable fair cross-modal comparison
3. 3.Scales appropriately: 30K-37.5K samples per modality balances diversity with computational feasibility
4. 4.Focuses on quality over quantity: FineWeb filters for long-form content (>8K tokens), COCO provides human-verified captions, AudioSet uses expert-annotated labels

Source Data

Initial Data Collection and Normalization

FineWeb (Text)

• —Source: NoahEJ/fineweb-sample-100BT_over-8192-tokens
• —Original Size: 100 billion tokens
• —Filtering: Only documents with >8,192 tokens
• —Rationale: Long-form content for training models on extended context

COCO 2017 (Images)

• —Source: wangherr/coco2017_train_512x_image_caption_depth
• —Original Size: 118,287 training images
• —Preprocessing: Resized to 512×512, depth information available
• —Rationale: Standardized resolution for efficient training, human-verified captions

AudioSet (Audio)

• —Source: agkphysics/AudioSet (config: "full", splits: bal_train/eval)
• —Original Size: 2 million 10-second clips
• —Processing:
• —Audio decoded using torchcodec 0.9.1 (PyTorch 2.9+cu126)
• —FFmpeg 7.0.2 backend for audio file decoding
• —Output: float32 numpy arrays at 16kHz sampling rate
• —Labels converted from list to comma-separated string
• —Rationale: Diverse sound events with expert human annotations

Who are the source language producers?

• —FineWeb: Web content creators (blog posts, articles, documentation)
• —COCO: Human annotators hired through crowdsourcing platforms
• —AudioSet: Ontology developed by Google Research, labels verified by human experts

Annotations

Annotation process

• —FineWeb: No additional annotation (uses original web content)
• —COCO: Professional annotators wrote 5 captions per image, this dataset uses 1 caption per image
• —AudioSet: Trained annotators labeled audio clips following a hierarchical ontology of 632 sound event classes

Who are the annotators?

• —COCO: Crowdworkers via Amazon Mechanical Turk
• —AudioSet: Expert annotators trained on the AudioSet ontology

Personal and Sensitive Information

• —FineWeb: May contain publicly available personal information from web sources
• —COCO: Images sourced from Flickr; may contain people but no identifying metadata is included
• —AudioSet: Audio clips from YouTube; may contain speech but no transcriptions or speaker identities

Users should be aware of potential biases and sensitive content when deploying models trained on this data.

Considerations for Using the Data

Social Impact of Dataset

Positive Impacts:

• —Enables research in multimodal AI and cross-modal understanding
• —Provides standardized benchmarks for reproducible research
• —Supports development of assistive technologies (image captioning for visually impaired, audio classification for hearing impaired)

Potential Risks:

• —Models trained on web data (FineWeb) may reproduce biases present in internet content
• —COCO images may not represent global diversity equitably
• —AudioSet may over-represent certain acoustic environments

Discussion of Biases

• —Geographic bias: COCO images predominantly from North America and Europe
• —Language bias: All text in English limits multilingual applicability
• —Domain bias: FineWeb reflects biases in web content (tech-heavy, Western-centric)
• —Acoustic bias: AudioSet may over-represent Western musical genres and urban soundscapes

Users should evaluate fairness and bias when deploying models trained on this dataset.

Other Known Limitations

• —Fixed splits: Test set may become saturated if widely used for benchmarking
• —Resolution limit: COCO images fixed at 512×512 (original COCO is variable resolution)
• —Audio duration: AudioSet clips are typically ~10 seconds (may not represent longer acoustic events)
• —Text length: FineWeb samples vary in length despite >8K token minimum

Additional Information

Dataset Curators

This dataset was curated and compiled by lv12 using memory-efficient data processing pipelines.

Licensing Information

Overall License: MIT License

Source Dataset Licenses:

• —FineWeb: ODC-By 1.0 License
• —COCO 2017: Creative Commons Attribution 4.0 International License
• —AudioSet: Creative Commons Attribution 4.0 International License

Users must comply with all source dataset licenses when using this collection.

Citation Information

If you use this dataset, please cite the original source datasets:

@misc{fineweb2024,
  title={FineWeb: decanting the web for the finest text data at scale},
  author={Penedo, Guilherme and Cappelli, Alessandro and Cojocaru, Liviu and Alobeidli, Hynek and Pannier, Baptiste and Almazrouei, Ebtesam and Launay, Julien},
  year={2024},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/HuggingFaceFW/fineweb}
}

@inproceedings{lin2014microsoft,
  title={Microsoft coco: Common objects in context},
  author={Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C Lawrence},
  booktitle={Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13},
  pages={740--755},
  year={2014},
  organization={Springer}
}

@inproceedings{gemmeke2017audio,
  title={Audio set: An ontology and human-labeled dataset for audio events},
  author={Gemmeke, Jort F and Ellis, Daniel PW and Freedman, Dylan and Jansen, Aren and Lawrence, Wade and Moore, R Channing and Plakal, Manoj and Ritter, Marvin},
  booktitle={2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)},
  pages={776--780},
  year={2017},
  organization={IEEE}
}

Contributions

Thanks to the creators of FineWeb, COCO, and AudioSet for making their datasets publicly available.

Quick Start

from datasets import load_dataset

# Load a specific subset and split
ds = load_dataset("lv12/MultiModalDataset", "coco", split="train")

# Iterate through samples
for sample in ds.take(5):
    print(f"ID: {sample['id']}")
    print(f"Text: {sample['text']}")
    print(f"Modality: {sample['modality']}")
    print("---")

For more examples, see the Dataset Structure section.