datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
datacomp200m
Datacomp200m
This is a smaller version of the datacomp_1b dataset.
Filtering was done by taking all rows that had self similarity (inner product) above 0.32. This resulted in 213009083 (213 million) rows.
The results of the datacomp paper suggest that filtering by CLIP score is better than random sampling.
Included in this repo are search indices created using autofaiss, over the text and image embeddings. There are two ways to access metadata, either in .parquet files in the… See the full description on the dataset page: https://huggingface.co/datasets/adams-story/datacomp200m.imagenet1k-256-wds-latentsThe imagenet1k dataset in the webdataset format
Each image was resized so that the max side resolution is 256, making sure to preserve aspect ratio.
Each image was encoded to latents using the sixteen channel https://huggingface.co/ostris/vae-kl-f8-d16
No cropping was used to encode to latents!
The resulting dataset has images in their original aspect ratio, but much smaller, and encodeded with a vae.
imagenet1k-256-wdsThis is imagenet1k in webdataset format. Images are stored as jpg files. Every image has been resized to a maximum side length of 256. That means that if an image in the original dataset was 1000 by 500, the new size will be 256 by 128. Images with a maximum side length of under 256 were not resized.
The total size of all dataset files is 57.8 GB, there are 1,281,167 rows in the training split and 50,000 rows in the validation split.
polyu-storyworld-charactersUS-PD-BooksUPDATE: The Internet Archive has requested that this dataset be deleted (see discussion #2) because they consider the IA's metadata too unreliable to determine whether a book is in the public domain. To alleviate the IA's concerns, the full texts of the books have been removed from this dataset until a more reliable way to curate public domain books from the IA collections is established. The metadata and documentation remain for reference purposes.
I was able to recreate one subcollection… See the full description on the dataset page: https://huggingface.co/datasets/storytracer/US-PD-Books.story_clozeStory Cloze Test' is a commonsense reasoning framework for evaluating story understanding,
story generation, and script learning.This test requires a system to choose the correct ending
to a four-sentence story.imagenet1k-256x256-ztree-sdvae-patch2Dataset produced by https://github.com/theAdamColton/zero-tree-diffusion
patch size: 2, uses quantization, clip value 2.5, db3, level 4, imagenet images resized to 256x256, uses the stable diffusion vae
story_clozestory_dataset_1LoC-PD-Books
Library of Congress Public Domain Books (English)
This dataset contains more than 140,000 English books (~ 8 billion words) digitised by the Library of Congress (LoC) that are in the public domain in the United States. The dataset was compiled by Sebastian Majstorovic.
Curation method
The dataset was curated using the LoC JSON API and filtering the Selected Digitized Books collection for English books.
Dataset summary
The dataset contains 140,000 OCR texts (~… See the full description on the dataset page: https://huggingface.co/datasets/storytracer/LoC-PD-Books.nyu-depthv2-wds
Dataset Card for nyu-depthv2-wds
This is the NYU DepthV2 dataset, converted into the webdataset format. https://huggingface.co/datasets/sayakpaul/nyu_depth_v2/
There are 47584 samples in the training split, and 654 samples in the validation split.
I shuffled both the training samples, and the validation samples.
I also cropped 16 pixels from all sides of the image, and depth image. I did this because there is a white border around all images.
This is an example of the border… See the full description on the dataset page: https://huggingface.co/datasets/adams-story/nyu-depthv2-wds.story_cloze-2016
Dataset Card for "story_cloze-2016"
More Information needed
Note: This dataset was utilized for the evaluation of probability-based prompt selection techniques in the paper 'Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis'. It differs from the actual benchmark dataset.
storydream-illustrated-stories
StoryDream — Illustrated Graded Stories (KO / EN)
11,680 illustrated story pages across 50 story/language combinations, each page
pairing an image with its story text. Every story is available at three reading
levels and split into three narrative parts.
Structure
{language}/{story_id}/level_{level}/{part_type}/page_{page_num}.jpg
metadata.csv
language — ko or en
story_id — original story id (25 stories per language)
level — 1, 2, 3 (reading difficulty; 1 is… See the full description on the dataset page: https://huggingface.co/datasets/Eurong2/storydream-illustrated-stories.openlibrary_dump_2024-04-30
OpenLibrary Dump (2024-04-30)
This dataset contains the OpenLibrary dump of April 2024 converted to Parquet and DuckDB for easier querying.
Formats
Original GZIP dumps
The original GZIP dumps are available at data/dumps. The dumps are gzipped TSV files with the original OL JSON record contained in the fifth column of the TSV.
DuckDB
The authors, works and editions dumps were imported as tables into… See the full description on the dataset page: https://huggingface.co/datasets/storytracer/openlibrary_dump_2024-04-30.storyvault-mediareasoning-engaging-story
NOTE: Got contacted as a selection for "Innovative Curator Spotlight Award"View link at the end for winning datasets.
Purpose and scope
The purpose of this dataset is to help expand engaging and coherent story creation from reasoning models.
NOTE: I did put lots of work to make this generate the best quality of story I could. Since the code is available, I don't want to have people spam low quality stories to HF. So if you are to use the code in this repository, PLEASE… See the full description on the dataset page: https://huggingface.co/datasets/rekrek/reasoning-engaging-story.StoryFrames
The StoryFrames Dataset
StoryFrames is a human-annotated dataset created to enhance a model's capability of understanding and reasoning over sequences of images.
It is specifically designed for tasks like generating a description for the next scene in a story based on previous visual and textual information.
The dataset repurposes the StoryBench dataset, a video dataset originally designed to predict future frames of a video.
StoryFrames subsamples frames from those videos and pairs… See the full description on the dataset page: https://huggingface.co/datasets/ingoziegler/StoryFrames.Japanese_Bandori_Band_Story
Japanese Bandori Band Story
Japanese Band Story text retrieved from the Bestdori scenario assets.
This snapshot contains 26 story entries, 493 chapters,
and 30679 rows (28800 dialogue rows).
Created at 2026-09-15T02:11:27.707570+00:00.
Files
data/train-*.parquet: Hub dataset shards generated by Dataset.push_to_hub.
data/band_stories.jsonl: local combined dataset, also included in the downloadable ZIP.
stories/story_XXXX/: complete per-story TXT, CSV and JSON… See the full description on the dataset page: https://huggingface.co/datasets/KomeijiForce/Japanese_Bandori_Band_Story.story_clozechildren-story-datasetsimple-story-chat-eval
simple-story-chat eval — public part
Code, the eval runner and the change log with numbers: simple-story-chat
(npm run eval, docs/improve-loop.md, docs/improve-log.md). The files are nested JSON read by that runner, so the dataset
viewer is off.
Synthetic Russian-language scenarios that measure whether an interactive story bot keeps its world consistent after its
context has been compacted into memory. Everything here is synthetic; no user stories are included.
The eval has two… See the full description on the dataset page: https://huggingface.co/datasets/Teadomi/simple-story-chat-eval.multispeaker-storycloze
Multi Speaker StoryCloze
A multispeaker spoken version of StoryCloze Synthesized with Kokoro TTS.
The dataset was synthesized to evaluate the performance of speech language models as detailed in the paper "Scaling Analysis of Interleaved Speech-Text Language Models".
We refer you to the SlamKit codebase to see how you can evaluate your SpeechLM with this dataset.
sSC and tSC
We split the generation for spoken-stroycloze and topic-storycloze as detailed in Twist.… See the full description on the dataset page: https://huggingface.co/datasets/slprl/multispeaker-storycloze.anonymous-storybench
Omni-StoryBench
Omni-StoryBench is a context-aware omnimodal story generation benchmark.Each sample provides a current story page and requires generating the next page's image, narration text, and speech utterance.
Dataset Structure
The dataset contains:
data/testset.jsonl: Main benchmark file.
images/: Page images.
texts/: Page text files.
speech/: Generated speech audio files.
instruction/: Source-level instruction metadata.
Data Fields
Each JSONL sample… See the full description on the dataset page: https://huggingface.co/datasets/omnibench/anonymous-storybench.GPT_Storybook_Anime_Style_Images
GPT Storybook Anime Style Images
Dataset Description
This is a synthetic GPT-generated Storybook Anime Style image dataset. It contains 100 image-caption pairs featuring original anime-inspired characters and scenes with a warm, illustrated storybook feeling.
The images focus on expressive character moments, gentle lighting, quiet interiors, nature scenes, village streets, cozy everyday settings, and calm storybook moods. Captions commonly describe soft linework… See the full description on the dataset page: https://huggingface.co/datasets/neonforestmist/GPT_Storybook_Anime_Style_Images.story-videosstoryscope
StoryScope
stories_train.parquet, stories_val.parquet, stories_test.parquet, stories_dev.parquet: prompt metadata plus AI-generated stories from GPT-5.4, Claude Sonnet 4.6, DeepSeek V3.2, Kimi K2.5, and Gemini 3 Flash
storyscope_features.parquet: 304 extracted narrative features for 61,575 story rows
taxonomy.json: the 304-feature taxonomy spanning 10 narrative dimensions
models/: trained XGBoost classifiers for binary human-vs-AI detection and 6-way authorship attribution… See the full description on the dataset page: https://huggingface.co/datasets/jjrussell10/storyscope.ePark_tu_hua_gu_shi_pian_picture_story
FormosanBank publication status
This audio is associated with XML published in the public FormosanBank corpus and uses the same license recorded in that XML: CC BY-NC-SA 4.0. View the published XML. Publication approval is recorded on the corresponding FormosanBank Basecamp card.
FormosanBank/ePark_tu_hua_gu_shi_pian_picture_story
Commercial AI Use is prohibited without prior written permission. See the FormosanBank Terms of Use and AI Use Addendum.
This is a… See the full description on the dataset page: https://huggingface.co/datasets/FormosanBank/ePark_tu_hua_gu_shi_pian_picture_story.ig-story-imageGerman-PD-Newspapers
Dataset Card for Public Domain Newspapers (German)
This dataset contains 13 billion words of OCR text extracted from German historical newspapers.
Dataset Details
Dataset Description
Curated by: Sebastian Majstorovic
Language(s) (NLP): German
License: Dataset: CC0, Texts: Public Domain
Dataset Sources [optional]
Repository: https://www.deutsche-digitale-bibliothek.de/newspaper
Copyright & License
The newspapers texts have been… See the full description on the dataset page: https://huggingface.co/datasets/storytracer/German-PD-Newspapers.story-imprinting
Story Imprinting — training datasets
Datasets accompanying Story Imprinting: AI Assistants Absorb Traits from Human Characters They Resemble.
Paper · Code
Contents
Paper section
Folder
Data
3.1 — Sabotage
3_1_sabotage/
Three training mixtures and separate sabotage/clean story pools
3.2 — Narration preferences
3_2_narration_preferences/
Six training mixtures and 12 story pools
4 — Affinity
4_selectivity/
Opposing-pair training datasets and raw… See the full description on the dataset page: https://huggingface.co/datasets/truthful-ai/story-imprinting.