datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
conceptual-captions-12m-webdatasetpickapic_v2_webdatasetwebdataset archive of yuvalkirstain/pickapic_v2.
Dataloading code can be found here.
danbooru2024-webp-4Mpixel
🎨 Danbooru2024 Webp 4MPixel Dataset
📊 Dataset Overview
The Danbooru2024-Webp dataset is a comprehensive collection focused on animation and illustration artwork, derived from the official Danbooru platform. It contains approximately 8.05 million high-quality, user-annotated images with corresponding tags and textual descriptions.
This dataset is 4MP-focused webp resized-dataset of Danbooru2024.
✨ Features
📋 Metadata Support
Includes a Parquet… See the full description on the dataset page: https://huggingface.co/datasets/deepghs/danbooru2024-webp-4Mpixel.hi-stt-preprocessed-webdatasetdanbooru2023-webp-4Mpixel-224The data set is just resized to 224*224
https://huggingface.co/datasets/KBlueLeaf/danbooru2023-webp-4Mpixel
Pseudo code for processing
def resize_image(file_path):
with Image.open(file_path) as img:
resized_img = img.resize((224, 224))
resized_img.save(file_path)
cc12m-webdataset
CC12M WebDataset
这是CC12M数据集的WebDataset格式版本。
数据集信息
文件数量: 1098
总大小: 888796.33 MB
上传时间: 2025-03-18 14:45:49
使用方法
import webdataset as wds
dataset = wds.WebDataset("https://huggingface.co/yangyang857658468/cc12m-webdataset/resolve/main/cc12m_*.tar")
emo_webds_2InternVL-SA1B-Caption-WebDatasetThis repo contains the recaptioned SA1B images in webdataset format. The recaptioned prompts are from https://huggingface.co/datasets/OpenGVLab/InternVL-SA-1B-Caption
dan-webp-newemo_webdsMegaSynth-webdatasetclevr-webdatasetwebface4m-wds-gzimagenet-w21-webp-wds
Dataset Summary
This is a copy of the full Winter21 release of ImageNet in webdataset tar format with WEBP encoded images. This release consists of 19167 classes, 2674 fewer classes than the original 21841 class Fall11 release of the full ImageNet.
The classes were removed due to these concerns: https://www.image-net.org/update-sep-17-2019.php
This is the same contents as https://huggingface.co/datasets/timm/imagenet-w21-wds but encoded in webp at ~56% of the size, shard count… See the full description on the dataset page: https://huggingface.co/datasets/timm/imagenet-w21-webp-wds.ESpeech-webinars2
Webinar Audio Dataset
Dataset Description
This dataset contains 850 hours processed webinar audio segments with corresponding metadata. Each audio file represents a segment extracted from webinar recordings, processed at 44.1kHz sample rate.
Dataset Summary
Language: Russian
Task: TTS, ASR, Quality Asessment
Audio format: MP3, 44.1kHz sample rate
Structure: Segmented audio files with JSON metadata
Dataset Structure
Data Fields… See the full description on the dataset page: https://huggingface.co/datasets/ESpeech/ESpeech-webinars2.e621-2024-webp-4MpixelDataset Description:
This is a processed version of the https://huggingface.co/datasets/boxingscorpionbagel/e621-2024 dataset, primarily prepared for personal use in future projects.
Therefore, for licensing and other legal information, please refer to the original project.
You can directly download tar file,or use https://deepghs.github.io/hfutils/main/api_doc/index/fetch.html#hf-tar-file-download to download anything .webp file you want.
The following modifications have been made to the… See the full description on the dataset page: https://huggingface.co/datasets/NebulaeWis/e621-2024-webp-4Mpixel.audiofolder_webdatasetdan-new-webp-traindanbooru2023-webp-4Mpixel
Danbooru 2023 webp: A space-efficient version of Danbooru 2023
This dataset is a resized/re-encoded version of danbooru2023.
Which removed the non-image/truncated files and resize all of them into smaller size.
This dataset already be updated to latest_id = 7,832,883.
Thx to DeepGHS!
Notice: content of updates folder and deepghs/danbooru_newest-webp-4Mpixel have been merged to 2000~2999.tar, You can ignore all the content in updates folder safely!
Details
This… See the full description on the dataset page: https://huggingface.co/datasets/KBlueLeaf/danbooru2023-webp-4Mpixel.libritts-r-webdatasetOfficial website: https://www.openslr.org/141/
This repository contains LibriTTS-R converted to a WebDataset. The original Wave files have been converted to 64kbps MP3 files for efficient streaming.
LibriTTS-R (paper) is a sound quality improved version of the LibriTTS corpus which is a multi-speaker English corpus of approximately 585 hours of read English speech at 24kHz sampling rate, published in 2019. The constituent samples of LibriTTS-R are identical to those of LibriTTS, with only the… See the full description on the dataset page: https://huggingface.co/datasets/lucasnewman/libritts-r-webdataset.cinepile-websetsgui_actor_webdataset
GUI-Actor WebDataset
A WebDataset format version of the GUI-Actor dataset for training vision-language models on GUI interaction tasks.
Usage
import webdataset as wds
# Load the dataset
dataset = wds.WebDataset("path/to/shards-*.tar")
dataset = dataset.decode("pilrgb").to_tuple("jpg", "json")
for image, metadata in dataset:
# Process image and metadata
pass
Citation
Please cite the original GUI-Actor paper if you use this dataset in your research.
habitat_web_image_depth_RESCUEmultilingual-librispeech-webdatasetsolar_webdataPhD-webdataset
PhD Webdataset
This repository contains the packaged version of PhD. For a detailed introduction to PhD, please visit the official website.
Overview
The PhD Webdataset is designed to facilitate easy access and usage of the PhD dataset. It includes various fields in 'json' key. The data in this repo is totally the same as in PhD.
Installation
Ensure you have Hugging Face's datasets library installed. You can install it via pip:
pip install datasets… See the full description on the dataset page: https://huggingface.co/datasets/AIMClab-RUC/PhD-webdataset.librispeech-webdatasetimagenet-1k-webdataset
ImageNet-1k WebDataset
This dataset contains ImageNet-1k in WebDataset format (tar files) for efficient streaming.
Dataset Structure
Training: 129 shards (train-*.tar)
Validation: 5 shards (validation-*.tar)
Total size: 147.82 GB
Format
Each tar file contains samples with:
*.jpg: Image bytes
*.cls: Label (class ID as text)
Usage
import webdataset as wds
# Training dataset
train_url = "train-{000000..000000000}.tar"
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/vrachit/imagenet-1k-webdataset.libritts-r-webdatasetWebMotion-36K
Introduce
A camera motion annotation dataset based on WebVid, containing a total of 36k videos.
Usage
Recovering zip files:cat webmotion.tar.gz.part_* > webmotion.tar.gz
Then unzip it:tar -xzf webmotion.tar.gz