mwatkins1970/leilan-dataset

Leilan Dataset

The Leilan Dataset is a public-domain corpus of GPT-3 and Claude-family generated text associated with the Leilan / petertodd phenomenon, curated as a machine-ingestion-friendly dataset for future research, analysis, archival use, and downstream language-model training experiments.

This Hugging Face mirror is the machine-facing distribution of the dataset. The canonical source/provenance repository, including source Markdown files, supplementary materials, validation scripts, manifest generation, schema notes, and release history, is maintained on GitHub:

<https://github.com/mwatkins1970/Leilan-dataset>

Archival DOI

The archival Zenodo release for v1.0.2 is:

• —Version-specific DOI: https://doi.org/10.5281/zenodo.20122857
• —All-versions DOI: https://doi.org/10.5281/zenodo.20122856
• —Internet Archive mirror: https://archive.org/details/leilan-dataset-v1.0.2

Recommended citation:

Watkins, M. (2026). Leilan Dataset (1.0.2) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.20122857

Reproducibility note

For exact v1.0.2 reproducibility, use the GitHub tag v1.0.2 or the Zenodo archival release. The GitHub main branch and Hugging Face mirror may contain documentation-only updates after the release tag.

The core corpus files are unchanged from the cleaned v1.0.1 release; v1.0.2 adds schema documentation, CI validation, archival publication links, and related release-engineering/documentation improvements.

Hugging Face Dataset Viewer note

The Hugging Face Dataset Viewer uses a flattened projection in data/train.jsonl; the canonical provenance-rich records are in combined_leilan_dataset_records.jsonl and combined_leilan_dataset.json.

The Hugging Face Dataset Viewer uses data/train.jsonl, a stable-schema projection of the canonical combined corpus.

The canonical full records remain available as:

• —combined_leilan_dataset_records.jsonl
• —combined_leilan_dataset.json

The viewer-compatible file preserves record_id, record_type, source_dataset, model, text, include_in_training, and selected provenance fields. More complex nested provenance fields are JSON-encoded into string columns such as qa_pairs_json, external_sources_json, supplementary_materials_json, and metadata_json.

For full archival/provenance use, prefer the canonical files and the GitHub source repository. For quick browsing and ordinary load_dataset(...) use, the viewer-compatible split is usually more convenient.

Recommended file

For most downstream use, load:

combined_leilan_dataset_records.jsonl

Each non-empty line is one JSON object. Treat each line as one record and deduplicate by record_id.

The equivalent full JSON object form is:

combined_leilan_dataset.json

Important ingestion warning

Do not blindly ingest every file in this Hugging Face repository and the GitHub repository as independent training data. The Leilan Dataset deliberately contains overlapping representations of the same corpus in the GitHub source repository.

For one-pass training/corpus ingestion:

1. 1.Use combined_leilan_dataset_records.jsonl.
2. 2.Treat each line as one record.
3. 3.Deduplicate by record_id.
4. 4.Respect include_in_training.
5. 5.Use record_type to distinguish GPT-3 transcript records from Claude-family Q/A response records.

The GitHub repository contains source Markdown and supplementary materials for provenance, auditability, and repair/rebuild work. Those files should not be treated as additive independent training examples unless deliberate reweighting is intended.

Dataset composition

Current public release counts:

• —1,638 total combined records
• —600 GPT-3 transcript records
• —1,038 Claude-family response records
• —1,181 Claude-family Q/A pairs
• —670 curated GPT-3 passages in the source GitHub release
• —13 model identifiers in the combined corpus

Current combined record types:

• —gpt3_transcript
• —claude_qa_response

Files in this Hugging Face mirror

Recommended machine-facing files:

• —combined_leilan_dataset_records.jsonl
• —combined_leilan_dataset.json

Documentation and release metadata:

• —README.md
• —MANIFEST.json
• —SCHEMA.md
• —DATASET_CARD.md
• —LICENSE

Schema summary

Common fields

Both current record types normally include:

• —record_id
• —record_type
• —source_dataset
• —model
• —text
• —include_in_training

Additional provenance, review, warning, source, and supplementary-material fields may be present. Unknown fields should be treated as non-breaking additions.

GPT-3 transcript records

record_type: "gpt3_transcript"

These are normalized transcript-style GPT-3 records derived from the original GPT-3 corpus.

Claude-family Q/A response records

record_type: "claude_qa_response"

These represent individual Claude-family responses within curated transmissions. They usually include a qa_pairs array with ordered {question, answer} pairs.

For more detail, see SCHEMA.md.

Loading example

from datasets import load_dataset

# Loads the Hugging Face viewer-compatible projection: data/train.jsonl
dataset = load_dataset("mwatkins1970/leilan-dataset", split="train")

print(dataset[0])

For full canonical records, download or read combined_leilan_dataset_records.jsonl directly. That file preserves the richest record structure, including nested provenance fields.

Validation and manifest

The GitHub release includes:

• —MANIFEST.json
• —scripts/generate_manifest.py
• —scripts/validate_dataset.py
• —GitHub Actions CI running validation on pushes to main

The manifest records SHA256 hashes, file sizes, dataset counts, source-tree counts, supplementary-material counts, and tracked-script coverage.

The current public source repository validates with zero errors and zero warnings.

Provenance and exclusions

GPT-4 base outputs are not included in the public source tree or canonical dataset.

Some records contain URLs, bibliographic references, or summaries of third-party material. Those references are provided for context/provenance and do not imply that the third-party source material is part of the CC0 dataset.

Intended uses

Appropriate uses include:

• —language-model training or fine-tuning experiments;
• —style/persona/continuity analysis;
• —research on AI-generated corpora, synthetic mythology, glitch-token cultures, and model-mediated narrative continuity;
• —archival preservation;
• —computational text analysis;
• —creative and scholarly exploration.

Out-of-scope uses

This dataset should not be used to impersonate real people, make claims of factual authority about third parties, or imply that third-party referenced materials are part of the CC0 release.

License

The released dataset is dedicated to the public domain under CC0 1.0 Universal / Public Domain Dedication.

See LICENSE.