CoolFace
Datasetpublic

enPurified/project_gutenberg-enPurified-openai-messages

📖 Project-Gutenberg-enPurified-openai-messages Project-Gutenberg-enPurified is a highly curated, "prose-first" refinement of the Project Gutenberg corpus. The enPurified collection is built on a specific philosophy: Specialization. While most modern datasets are "general purpose," they often dilute linguistic quality with code snippets, math formulas, and broken OCR text. This dataset aggressively strips away everything but high-quality English prose to help models master fluid… See the full description on the dataset page: https://huggingface.co/datasets/enPurified/project_gutenberg-enPurified-openai-messages.

sourceHugging Faceupdated 9mo agoView on Hugging Face
1likes175downloads
Dataset Card

📖 Project-Gutenberg-enPurified-openai-messages

Project-Gutenberg-enPurified is a highly curated, "prose-first" refinement of the Project Gutenberg corpus.

The enPurified collection is built on a specific philosophy: Specialization. While most modern datasets are "general purpose," they often dilute linguistic quality with code snippets, math formulas, and broken OCR text. This dataset aggressively strips away everything but high-quality English prose to help models master fluid, creative, and sophisticated narrative reasoning.

💎 The enPurified Core Goal

The goal of the enPurified collection is to take existing high-value datasets and run a series of aggressive heuristic tests to keep only the highest quality English prose.

  • —No Coding: Any text containing programming syntax or technical logs is discarded.
  • —No Foreign Languages: Strict language identification ensures a 100% English corpus.
  • —No Math: LaTeX and symbolic logic are removed; this is a dataset for language reasoning, not calculation.
  • —No Low-Quality Text: We filter out "listicles," link farming, and OCR "slop."

🧠 Intelligent Chunking & Narrative Continuity

Standard Gutenberg datasets often provide a whole novel as one massive string, which is unusable for standard training windows. We solve this using a sophisticated Windowed Chunking approach:

  1. 1.Paragraph-Aware Splitting: We do not cut mid-sentence. We chunk by logical paragraph breaks to preserve semantic meaning.
  2. 2.Narrative Overlap: We maintain a 300-word overlap between chunks. This "bridge" ensures the model learns how narrative flow is maintained from one context window to the next, preventing the "choppy" feel of most fine-tuned models.
  3. 3.Instruction Synthesis: Every chunk is converted into the OpenAI Messages format. If metadata (Author/Title) is found, the prompt is personalized (e.g., "Write in the style of Mark Twain"). If not, high-quality generic literary prompts are used.

🛡️ The "Elite Quality Filter" (Heuristics)

Content is only retained if it passes every single test in the purification pipeline:

1. Structural Integrity

TestStrategyIntent
Boilerplate RemovalMulti-pass RegexLegal Strip. Aggressively removes all Project Gutenberg license headers and legal footers.
Line DensityAvg Line > 50 charsProse Enforcement. Penalizes poetry and lists to ensure the model learns cohesive paragraph linking.
OCR CleaningSymbol Ratio < 5%Artifact Removal. Filters out text with high counts of _, *, or broken punctuation common in old scans.

2. Conversational Logic

TestStrategyIntent
Language IDpy3langid == 'en'Strict English Only. Prevents cross-lingual interference.
Deduplicationxxhash FingerprintingUnique Data. Prevents the model from overfitting on popular chapters or repeated introductions.
Message NormalizationOpenAI FormatZero-Scripting. Map roles directly to user, assistant, and system.

⚖️ Credits & Licensing

  • —Source Material: Filtered derivative of [Project Gutenberg](https://www.gutenberg.org/).
  • —Original Creator: Various Authors (Public Domain). Refer to https://www.gutenberg.org/policy/license.html