ozefe/secure-contain-protect
Secure, Contain, Protect A structured snapshot of the English SCP Foundation Wiki — every content page (SCPs, tales, hubs, GOI formats, essays and art) created between 2008 and 2026, harvested from the Crom GraphQL API. 19,438 pages by 3,479 credited authors, each with its raw Wikidot source text, rating, vote and comment counts, tags, authorship, alternate titles, and per-credit attributions (the credited author(s), distinct from whoever posted the page). User/author pages… See the full description on the dataset page: https://huggingface.co/datasets/ozefe/secure-contain-protect.
Secure, Contain, Protect
A structured snapshot of the English [SCP Foundation Wiki](https://scp-wiki.wikidot.com/) — every content page (SCPs, tales, hubs, GOI formats, essays and art) created between 2008 and 2026, harvested from the Crom GraphQL API. 19,438 pages by 3,479 credited authors, each with its raw Wikidot source text, rating, vote and comment counts, tags, authorship, alternate titles, and per-credit attributions (the credited author(s), distinct from whoever posted the page).
User/author pages, hidden pages, and non-content categories are filtered out; only the rendered HTML is omitted (the raw source is included so you can render or parse it yourself). Snapshot date: 2026-06-03.
Dataset structure
Three tables, published as one Parquet file each — flat columns throughout, except pages.attributions, which is a JSON array of credits.
`pages` — one row per content page (19,438 rows):
`page_tags` — (page_url, position, tag), 175,007 rows.
`page_alternate_titles` — (page_url, idx, title, source), 10,465 rows.
Join the children to pages on page_url = url.
Use the published dataset
Pull it straight from the Hub — no local build required.
# 🤗 datasets
from datasets import load_dataset
pages = load_dataset("ozefe/secure-contain-protect", "pages", split="train")
tags = load_dataset("ozefe/secure-contain-protect", "page_tags", split="train")# pandas / polars, read directly from the Hub
import pandas as pd
pages = pd.read_parquet("hf://datasets/ozefe/secure-contain-protect/data/parquet/pages.parquet")-- DuckDB, query the Hub without downloading
SELECT title, rating
FROM 'hf://datasets/ozefe/secure-contain-protect/data/parquet/pages.parquet'
ORDER BY rating DESC
LIMIT 10;# Or download every file locally
huggingface-cli download ozefe/secure-contain-protect --repo-type dataset --local-dir scp-dataA quick join — the most common tags on the highest-rated SCPs:
import duckdb
con = duckdb.connect()
base = "hf://datasets/ozefe/secure-contain-protect/data/parquet"
print(con.execute(f"""
SELECT t.tag, count(*) n
FROM '{base}/pages.parquet' p
JOIN '{base}/page_tags.parquet' t ON t.page_url = p.url
WHERE p.rating > 1000
GROUP BY t.tag ORDER BY n DESC LIMIT 10
""").fetchall())The attributions column is a JSON array — unnest it to get the real credited authors (distinct from created_by_*, which is whoever posted the page):
SELECT title, a.user_display_name, a.type
FROM 'pages.parquet', unnest(from_json(attributions,
'[{"user_display_name": "VARCHAR", "type": "VARCHAR", "date": "VARCHAR", "order": "INTEGER"}]')) AS t(a)
WHERE url = 'http://scp-wiki.wikidot.com/scp-682';
-- Dr Gears (AUTHOR), Epic Phail Spy (AUTHOR)Build it yourself
The repository ships the full pipeline: crawl -> export -> report. Requires Python 3.14+ (the Crom GraphQL API client `thaumiel` needs it).
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cd src1. Download the data into DuckDB (../data/scp_dataset.duckdb). The crawl is per-year, resumable (the database is the checkpoint), and Ctrl+C-safe:
python scp_dataset.py 2008 # a single year
for y in $(seq 2008 2026); do python scp_dataset.py "$y"; done # the whole archive2. Export to Parquet — one file per table in ../data/parquet/:
python export_parquet.py3. Statistics & plots — writes ../images/*.png and the STATISTICS.md report below:
python generate_report.pyCrom meters requests against a 300,000-point budget that resets every 5 minutes; one full year costs only a few thousand points, so the whole archive fits comfortably. The crawler logs the remaining quota as it goes.
How it was built
Pages are selected server-side from Crom GraphQL API with: URL on scp-wiki.wikidot.com, Wikidot category _default, is_hidden = false, and is_user_page = false. Per page the crawler requests the free fields plus the costly source, summary, alternate_titles, and attributions (the latter stored as a JSON column); it skips only text_content (rendered HTML). Timestamps are stored as UTC. See `src/scp_dataset.py`.
License & attribution
The SCP Foundation Wiki is licensed under [Creative Commons Attribution-ShareAlike 3.0](https://creativecommons.org/licenses/by-sa/3.0/), and this dataset — which contains that source text — is released under the same license. If you use it, you must attribute the SCP Foundation Wiki and the individual authors (see the created_by_* columns and the license boxes embedded in each page's source), and share derivatives under CC BY-SA 3.0. Data accessed via the Crom GraphQL API. This dataset is an unofficial, fan-made compilation and is not affiliated with the SCP Wiki administration.
The SCP Foundation Wiki in Numbers
SCP Wiki content pages · 2008–2026 · n = 19,438 · via Crom.
A statistical tour of every content page (SCPs, tales, hubs, GOI formats and more) on the English SCP Wiki, built from the Crom API. Each figure below is followed by the conclusion it supports.
Content created per year
Takeaway. The wiki grew from 298 pages in 2008 to a peak around 2025, 19,438 in all. SCPs are the backbone but tales now make up roughly a third of yearly output; 2026 is a partial year.
Object classes
Takeaway. Euclid ('anomalous but containable') is the plurality at 3,431, just ahead of Safe (3,084); Keter is a distant third. The catch-all 'esoteric-class' has overtaken every named class except the big three.
Most common themes
Takeaway. Entity descriptors dominate — 'sapient', 'humanoid' and 'alive' lead — followed by genre tags like horror and mind-affecting. Comedy ranks surprisingly high, a reminder the wiki is not all grimdark.
Rating distribution
Takeaway. Ratings are extremely right-skewed: a median of 69 against a maximum of 10,758 (SCP-173). Only 46 pages sit below zero — the community deletes weak work, so what remains is heavily curated.
Article length
Takeaway. Article length is roughly log-normal around a median of 10,398 characters (a few pages of text), with a long tail of giant hubs and anthologies reaching 199,994 characters.
Authorship is a power law
Takeaway. Counting every credited author (not just whoever posted the page), contribution is steeply unequal: the most prolific 2% hold 30% of all author credits, while 1,830 authors have a single credit. A small core sustains the wiki.
Does length buy quality? No
Takeaway. Rating and article length are essentially uncorrelated (r = -0.01). The median rating is flat across the whole length range — a longer article is not a better-rated one. Concept and execution, not word count, drive a page's score.
Ratings fall with each cohort
Takeaway. Average rating slides from 511 in 2008 to 36 in 2026. This is mostly an age effect — older pages have had years to accumulate up-votes — rather than proof that newer writing is worse; the median falls far more gently than the mean.
Danger sells
Takeaway. The more dangerous the classification, the higher the average score: Apollyon (224) and Keter top the table, while Safe and Neutralized trail. Readers reward existential threat over the mundane.
Prolific vs. acclaimed authors
Takeaway. Counting every credited author (co-authors included), output and acclaim remain different games: the most prolific cluster at modest average ratings, while the highest-rated are comparatively selective — bubble size (total score) shows a few writers manage both volume and quality.
Co-authorship over time
Takeaway. Co-authorship has climbed from near zero to about 10% of pages in recent years — modern SCP is increasingly a team effort (2026 is a partial year). Early collaboration is undercounted: the attribution metadata recording co-authors is a later convention.
Bigger teams, higher ratings
Takeaway. Median rating rises with team size — from 68 for solo pages to 103 for the largest teams. Collaboration correlates with a warmer reception, though the most ambitious projects also tend to attract co-authors.
How themes cluster
Takeaway. Themes form clear clusters. The strongest pairing is 'sapient' + 'humanoid': entity descriptors (humanoid / sapient / alive) co-occur tightly, forming a 'monster' cluster distinct from the genre tags.
How authors collaborate
Takeaway. Co-authorship forms tight clusters around a few hubs — Ralliston is the most connected. Node size is total pages, edge weight is shared pages; the modern wiki is a densely woven collaborative network, not a crowd of soloists.
Per-year summary
Takeaway. Output and contributor counts climb together — more authors, more pages — while average rating declines with each younger cohort (an age effect, not a quality one).
Top 15 highest-rated pages
Takeaway. The canon is front-loaded with early classics — SCP-173, SCP-049, SCP-682 — that have compounded votes for over a decade, alongside a few modern breakouts like SCP-5000.
Top 15 authors (all credited authors)
Takeaway. Crediting every author (co-authors, and people who never posted their own work) reshuffles the leaderboard versus a naive by-poster count — and the co-authored share shows how collaboratively each writer works.
Top co-author duos
Takeaway. The wiki's tightest writing partnerships — recurring duos that have co-authored many pages together, several rating well above the site median.
Most collaborative authors
Takeaway. The community's connectors — authors who have written with the widest circle of collaborators, knitting otherwise separate clusters together.
Top rewriters
Takeaway. The canon's caretakers: a small group does most of the rewriting that keeps the early, heavily-trafficked articles current.