speed/WAON

1---2configs:3- config_name: default4  data_files:5  - split: train6    path: data/train-*7dataset_info:8  features:9  - name: url10    dtype: string11  - name: caption12    dtype: string13  - name: similarity14    dtype: float6415  - name: page_title16    dtype: string17  - name: page_url18    dtype: string19  - name: punsafe20    dtype: float6421  - name: width22    dtype: float6423  - name: height24    dtype: float6425  - name: original_width26    dtype: float6427  - name: original_height28    dtype: float6429  - name: sha25630    dtype: string31  - name: phash32    dtype: string33  splits:34  - name: train35    num_bytes: 7240543928336    num_examples: 15394289237  download_size: 4674381485038  dataset_size: 7240543928339license: apache-2.040language:41- ja42size_categories:43- 100M<n<1B44---45 46<div align="center" style="line-height: 1;">47<h1>WAON: Large-Scale and High-Quality Japanese Image-Text Pair Dataset for Vision-Language Models </h1> 48 49 50  |51  <a href="https://huggingface.co/collections/speed/waon" target="_blank">🤗 HuggingFace</a>52  &nbsp;|53  <a href="https://arxiv.org/abs/2510.22276" target="_blank">📄 Paper</a>54  &nbsp;|55  <a href="https://github.com/llm-jp/WAON" target="_blank">🧑‍💻 Code</a>56  &nbsp;|57 58  <br/>59 60<img src="validation_top1_accuracy.svg" width="50%"/>61</div>62 63 64## Introduction65WAON is a Japanese (image, text) pair dataset containing approximately 155M examples, crawled from Common Crawl.66It is built from snapshots taken in 2025-18, 2025-08, 2024-51, 2024-42, 2024-33, and 2024-26.67The dataset is high-quality and diverse, constructed through a sophisticated data processing pipeline.68We apply filtering based on image size and SigLIP scores, and perform deduplication using URLs, captions, and perceptual hashes (pHash).69 70## How to Use71```python72from datasets import load_dataset73 74ds = load_dataset("speed/WAON")75```76 77### Format78 79- `url`: URL of the image80- `caption`: Caption associated with the image81- `page_title`: Title of the page containing the image82- `page_url`: URL of the page83- `punsafe`: Probability that the image is unsafe84- `quality`: The quality of the text in the text column85- `width`: Width (in pixels) of the resized image used for computing pHash86- `height`: Height (in pixels) of the resized image used for computing pHash87- `original_width`: Original width of the image88- `original_height`: Original height of the image89- `sha256`: SHA-256 hash of the original image file90- `phash`: Perceptual hash (pHash) computed from the resized image91 92 93## Dataset Construction Pipeline94 95We construct WAON dataset through the following steps (The numbers in parentheses indicate the remaining data96count after each processing step (based on the 2025-18 snapshot):97<div align="center">98<img src="waon-pipeline.svg" width="50%"/>99</div>100 101 102## LICENSE103This dataset is licensed under the Apache License 2.0 and governed by Japanese law. Its use is limited to “information analysis” as defined in Article 30-4 of the Japanese Copyright Act.104 105## Citation106 107```bibtex108@misc{sugiura2025waonlargescalehighqualityjapanese,109      title={WAON: Large-Scale and High-Quality Japanese Image-Text Pair Dataset for Vision-Language Models}, 110      author={Issa Sugiura and Shuhei Kurita and Yusuke Oda and Daisuke Kawahara and Yasuo Okabe and Naoaki Okazaki},111      year={2025},112      eprint={2510.22276},113      archivePrefix={arXiv},114      primaryClass={cs.CV},115      url={https://arxiv.org/abs/2510.22276}, 116}117```