datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
conceptual-captions-12m-webdatasetpickapic_v2_webdatasetwebdataset archive of yuvalkirstain/pickapic_v2.
Dataloading code can be found here.
cc12m-webdataset
CC12M WebDataset
这是CC12M数据集的WebDataset格式版本。
数据集信息
文件数量: 1098
总大小: 888796.33 MB
上传时间: 2025-03-18 14:45:49
使用方法
import webdataset as wds
dataset = wds.WebDataset("https://huggingface.co/yangyang857658468/cc12m-webdataset/resolve/main/cc12m_*.tar")
InternVL-SA1B-Caption-WebDatasetThis repo contains the recaptioned SA1B images in webdataset format. The recaptioned prompts are from https://huggingface.co/datasets/OpenGVLab/InternVL-SA-1B-Caption
MegaSynth-webdatasetclevr-webdatasetYFCC14M_subset_webdatasetGBC1M_webdatagui_actor_webdataset
GUI-Actor WebDataset
A WebDataset format version of the GUI-Actor dataset for training vision-language models on GUI interaction tasks.
Usage
import webdataset as wds
# Load the dataset
dataset = wds.WebDataset("path/to/shards-*.tar")
dataset = dataset.decode("pilrgb").to_tuple("jpg", "json")
for image, metadata in dataset:
# Process image and metadata
pass
Citation
Please cite the original GUI-Actor paper if you use this dataset in your research.
ShareGPT-WebDatawebdatasets_imagesPhD-webdataset
PhD Webdataset
This repository contains the packaged version of PhD. For a detailed introduction to PhD, please visit the official website.
Overview
The PhD Webdataset is designed to facilitate easy access and usage of the PhD dataset. It includes various fields in 'json' key. The data in this repo is totally the same as in PhD.
Installation
Ensure you have Hugging Face's datasets library installed. You can install it via pip:
pip install datasets… See the full description on the dataset page: https://huggingface.co/datasets/AIMClab-RUC/PhD-webdataset.imagenet-1k-webdataset
ImageNet-1k WebDataset
This dataset contains ImageNet-1k in WebDataset format (tar files) for efficient streaming.
Dataset Structure
Training: 129 shards (train-*.tar)
Validation: 5 shards (validation-*.tar)
Total size: 147.82 GB
Format
Each tar file contains samples with:
*.jpg: Image bytes
*.cls: Label (class ID as text)
Usage
import webdataset as wds
# Training dataset
train_url = "train-{000000..000000000}.tar"
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/vrachit/imagenet-1k-webdataset.web_prm_processed_dataweb_scraper_datasetmnist-webdataset-png
MNIST WebDataset PNG
The MNIST dataset with samples stored as PNG images and compiled into the WebDataset format.
DALI/JAX Example
The following code shows how this dataset can be loaded into JAX arrays by DALI.
from nvidia.dali import pipeline_def
import nvidia.dali.fn as fn
import nvidia.dali.types as types
from nvidia.dali.plugin.jax import DALIGenericIterator
from nvidia.dali.plugin.base_iterator import LastBatchPolicy
def get_data_iterator(batch_size, dataset_path):… See the full description on the dataset page: https://huggingface.co/datasets/hayden-donnelly/mnist-webdataset-png.webdataset_copyweb-dataset_3_screenshot_rendered_train_mhtml_3web-dataset_4_screenshot_rendered_train_mhtml_4web_scraper_dataset_2NayanaDocs-Indic-45k-webdataset
Nayana-DocOCR Indic Annotated Dataset
Dataset Description
This is a large-scale multilingual document OCR dataset containing approximately 400GB of images with comprehensive annotations across multiple languages including Indic languages and English. The dataset is stored in WebDataset format using TAR archives for efficient streaming and processing.
Available Language Subsets
bn (Bengali): Available
en (English): Available
gu (Gujarati): Available
hi (Hindi):… See the full description on the dataset page: https://huggingface.co/datasets/Nayana-cognitivelab/NayanaDocs-Indic-45k-webdataset.web-dataset_4_interact_resultsagentic_dataset_qwen3_coder_webWebDataagentic_dataset_qwen3_coder_web_nodeNayanaDocs-Global-45k-webdataset
Nayana-DocOCR Global Annotated Dataset
Dataset Description
This is a large-scale multilingual document OCR dataset containing approximately 400GB of images with comprehensive annotations across multiple global languages and English. The dataset is stored in WebDataset format using TAR archives for efficient streaming and processing.
Available Language Subsets
Arabic (ar): Available
German (de): Available
Russian (ru) : Available
Spanish (es): Available
French… See the full description on the dataset page: https://huggingface.co/datasets/Nayana-cognitivelab/NayanaDocs-Global-45k-webdataset.agentic_dataset_qwen3_coder_web-sftmeme_webdatasetwebdataNayanaDocs-Multilingual-webdataset