KaraKaraWitch/TvTroper-2025

TvTroper-2025

A cleaned & refreshed dump of ~708 k pages from tvtropes.org

Dataset Summary

TvTroper-2025 is an updated snapshot of TvTropes.org (≈ 708 000 wiki pages, namespaces and date-grouped pages excluded). Every page is released in two flavours:

• —Raw HTML – 22 GB single file
• —Markdown-cleaned – split into 1 GB JSONL shards (no unpacking required)

No additional content filtering has been applied; short sub-index pages are left in so you can decide what to drop.

What's changed since the 2023 release?

• —Crawler re-written from scratch; BeautifulSoup -> Selectolax for speed & memory.
• —Made a selectolax version of markdownify.
• —Output is now native JSONL (no zips).
• —Raw dumps keep only the main article body; headers/footers/nav stripped.
• —Cleaned version ships side-by-side with raw dumps in the same repo.
• —Changes for Cleaned Version:
• —Additional markdown syntax:
• —||Spoilers|| are marked similarly to discord
• —^Super Script^ for Super script

Intended uses

Unsupervised pre-training or fine-tuning of language models with a focus on narrative patterns, tropes, themes and pop-culture knowledge. Down-stream ideas: text generation, classification, controllable story generation, trope-aware summarisation.

Languages

English (en)

Dataset structure

Each line is a single JSON object:

{
  "id": "RealLife",
  "article": "# Spanner In The Works - Real Life\n\n...",
  "ns": "SpannerInTheWorks",
  "err": null
}

Fields

• —id – TvTropes page key (final URL slug; may differ from original redirect).
• —article – page content; raw HTML or Markdown (check config).
• —ns – namespace stub the crawler recorded when the page was fetched. Special namespaces you may wish to drop: SugarWiki, DarthWiki, YMMV, WMG, Trivia, Administrivia. "What are these?"
• —SugarWiki – gushy fan praise.
• —DarthWiki – mirror-rant pages.
• —YMMV – subjective fan reactions (Your Mileage May Vary).
• —WMG – Wild Mass Guessing, fan theories without proof.
• —Trivia – out-of-universe production notes.
• —Administrivia – site-maintenance stubs, no narrative content.

• —err – HTTP / parsing error string; null if the scrape succeeded.

Quick main-namespace filter

# discard the special namespaces above
# If you wanna go faster (Recommended):
# 1. Replace `json` with `orjson` (external package required)
# 2. Change `r` to `rb` & `w` with `wb`
import json, pathlib, sys

for file in pathlib.Path("data/").glob("*.jsonl"):
    clean = file.with_suffix(".main.jsonl")
    with open(file,"r",encoding="utf-8") as fin, open(clean,"w",encoding="utf-8") as fout:
        for line in fin:
            row = json.loads(line)
            if row.get("ns") in {"SugarWiki", "DarthWiki", "YMMV", "WMG", "Trivia", "Administrivia"}:
                continue
            fout.write(line)

Splits

No canonical train/val/test split.

Markdown-cleaned: 17 x 1 GB shards Raw HTML: 1 x 22 GB file

Data collection

TvTropes.org editors produced the original text. We crawled every reachable page during October 2025, kept the 200 OK responses, and stored the literal HTML. No normalization, de-duplication or PII scrubbing was performed.

Biases & limitations

• —Reflects the demographics, opinions and humor of the TvTropes contributor base.
• —Contains spoilers, informal prose, in-jokes, and occasionally NSFW language.
• —Short disambiguation/index pages are retained—filter them if you need long-form text only.
• —No manual balancing across media types (anime, games, film, etc.).

Licensing

Dataset wrapper & extraction code: Apache 2.0 (copyright © 2025 KaraKaraWitch) Fair-use media excerpts may appear in raw HTML; these remain under their respective owners.

Citation

@misc{tvtroper2025,
  title        = {{TvTroper-2025}: A {TVTropes.org} Corpus},
  author       = {KaraKaraWitch},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/datasets/KaraKaraWitch/TvTroper-2025}}
}

Contributions

Curator: @KaraKaraWitch

Release Climate Notes

Open, unencumbered dumps like this one are becoming an endangered species. Use, share and (re)release while you still can.