enPurified/Dolci-Instruct-SFT-enPurified-openai-messages

language:

• —en license: other task_categories:
• —text-generation tags:
• —data-filtering
• —synthetic
• —prose-only
• —enPurified
• —quality-filtering size_categories:
• —10K<n<100K pretty_name: enPurified Dolci Instruct ---

enPurified: Dolci-Instruct-SFT

The original dataset https://huggingface.co/datasets/allenai/Dolci-Instruct-SFT was reduced from ~2,155,000 rows to 38,829 of English only prose.

Project Overview

The enPurified collection is an initiative to curate high-fidelity English prose datasets for language modeling. While the open-source ecosystem is rich with datasets targeting mathematics, code generation, and multilingual capabilities, there is a distinct need for corpora focused exclusively on high-quality, conversational English text.

This dataset is a filtered subset of allenai/Dolci-Instruct-SFT. It has been subjected to a rigorous "gauntlet" of heuristic filters designed to strip away code, mathematics, low-quality text, markup artifacts, and non-prose content. The result is a refined dataset intended for training models on pure linguistic reasoning, stylistic coherence, and general instruction following without the noise of specialized syntax or low-entropy data.

Dataset Format: The data is formatted in the standard OpenAI Messages structure ([{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]) to ensure immediate compatibility with modern training pipelines.

Pruning Pipeline

The filtration process utilizes a multi-stage heuristic pipeline (enPurified_pipeline.py) designed to aggressively cull data that does not meet strict prose standards. The pipeline consists of the following processing stages:

1. Pre-Processing & Normalization

• —Artifact Removal: Strips specific tokens (e.g., <|begin_of_solution|>) and header IDs.
• —Message Formatting: Converts raw interaction logs into a clean user/assistant schema, stripping function calls and retaining only textual content.

2. Syntax and Code Exclusion

• —Symbol Density Check: Rejects entries with high ratios of non-alphanumeric symbols (e.g., braces, brackets, semicolons) to filter out programming languages.
• —Math Gate: Removes content containing LaTeX equations (\begin{equation}, $$), ASCII matrices, or boxed answers (\boxed{}).
• —JSON/Data Gate: Filters out raw JSON dumps and structured data blocks.
• —Strict Syntax: Rejects text containing HTML tags, code blocks (markdown fences), or banned programming substrings (e.g., std::, var , public static void).

3. Structural Integrity

• —Line Quality: Analyzes line length distribution to remove lists, poetry, or log files (entries with >80% short lines).
• —Repetition: Penalizes documents with high n-gram repetition or low line uniqueness.
• —Case Gating: Removes entries that are statistically likely to be all-caps (shouting) or all-lowercase (lazy/stylized).

4. Semantic and Style Filtering

• —Quiz/MCQ Filter: Removes multiple-choice questions and benchmark-style content (e.g., "Option A", "Option B").
• —Task/Benchmark Artifacts: Filters out entries containing prompts for specific restricted tasks (e.g., "answer only with 0 or 1", "factually consistent").
• —Conversational Fillers: Aggressively prunes "slop" and AI-specific filler phrases (e.g., "As an AI language model", "I hope this helps", "Here is a list").
• —Constraint Triggers: Removes constrained writing tasks that degrade prose quality (e.g., "do not use the letter e", "start every sentence with...").

5. Safety and Alignment

• —Toxicity & Safety: Filters content against a ban list of toxic terms, pornographic keywords, and high-risk age indicators.
• —Jailbreak Detection: Removes known jailbreak triggers (e.g., "DAN", "developer mode").
• —Moralizing Filters: Strips responses containing preachy or excessive moralizing lectures common in synthetic data.

6. Linguistic Quality

• —English Prose ID: Enforces ASCII character density (>95%) and a minimum stopword ratio to ensure the text is coherent English prose.
• —MTLD Richness: Calculates the Measure of Textual Lexical Diversity (MTLD). Only text exceeding a specific richness score (>= 62.0) is retained to ensure vocabulary diversity.

Usage

from datasets import load_dataset

dataset = load_dataset("YourOrg/Dolci-Instruct-SFT_purified", split="train")
print(dataset[0])

License

This dataset is a derivative work. Users must refer to the license of the original dataset: [allenai/Dolci-Instruct-SFT](https://www.google.com/search?q=https://huggingface.co/allenai/Dolci-Instruct-SFT).

Please ensure you comply with the terms set forth by the original authors regarding attribution and usage restrictions.