enPurified/tulu-3-sft-mixture-enPurified-openai-messages

language:

• —en license: other task_categories:
• —text-generation tags:
• —prose
• —purified
• —no-code
• —no-math
• —quality-filtered
• —sharded
• —high-performance source_datasets:
• —allenai/tulu-3-sft-mixture prettyname: enPurified Collection (Optimized) datasetinfo: features:
• —name: messages list:
• —name: role dtype: string
• —name: content dtype: string config_name: default splits:
• —name: train numbytes: 0 numexamples: 0 ---

Dataset Card: enPurified Collection

This dataset was updated on January 13th, 2026 to strip out even more math/code. The pruning process reduced the dataset from 940,000 to 88,782** rows of high-quality English prose.

(The script used for this process is uploaded in the files section)

Dataset Summary

The enPurified collection is a curated suite of datasets designed to isolate high-quality English prose from existing high-value open-source datasets.

The primary objective of this collection is to support the training of Large Language Models (LLMs) on pure, high-density English text, strictly excluding programming code, mathematical notation, foreign languages, and low-quality web scrapes. While many datasets exist specifically for coding and mathematics, enPurified fills the niche for high-fidelity linguistic training data.

This specific subset utilizes the allenai/tulu-3-sft-mixture as a base. It is processed using a high-concurrency, multi-stage pipeline optimized for high-core CPUs (e.g., AMD 9950X3D), ensuring maximum throughput via worker-side serialization and compiled regex gating.

Purpose & Scope

• —Language: English (Strict)
• —Format: OpenAI Messages ({"messages": [{"role": "user", "content": "..."}, ...]})
• —Sharding: Data is split into ~1GB gzip-compressed shards for efficient streaming and handling.
• —Excluded Content:
• —Programming code (Python, SQL, JavaScript, Terraform, etc.)
• —Mathematical equations and LaTeX rendering
• —Multiple Choice Questions (MCQ) and quiz formats
• —Foreign language text
• —Low-lexical-diversity content (repetitive loops, "spammy" text)

Processing Pipeline & Architecture

The construction of the enPurified dataset utilizes a custom-built, highly parallelized filtering engine designed to saturate 32+ CPU threads.

1. High-Performance Ingestion

• —Prefetcher: A dedicated thread streams data from the source repo (allenai/tulu-3-sft-mixture) into a synchronized queue (max size 100), ensuring the worker pool never starves.
• —Two-Stage Execution: The pipeline runs a "Checkpoint" stage (first 100k docs) for validation before committing to the full dataset processing.

2. Syntax & Code Purification (Zero Tolerance)

• —Compiled Regex Gate: All banned substrings (e.g., def , SELECT, =>, import) are compiled into a single C-speed Regular Expression (REGEX_BANNED_BLOCK) to minimize Python interpreter overhead during scanning.
• —Symbol Density Check: Rejects text where the ratio of code-specific symbols ({, }, =>, \, etc.) exceeds 3.5%, adjusting for allowed <think> tags.
• —HTML Filtering: Removes text containing raw HTML tags, preserving only permitted conversation control tokens.

3. Structural Integrity & Content Filters

• —MCQ Detection: Heuristically identifies and removes "Multiple Choice Question" patterns and structured option lists (A-E).
• —Line Analysis: Filters out texts comprised predominantly of short lines or lists (logs, stack traces, menu dumps).
• —Response Validation: Enforces a minimum character count (30 chars) for assistant responses to eliminate low-value distinct turns.

4. Linguistic Quality Assurance

• —English Prose Identification:
• —Stopword Density: Requires a minimum density of common English stopwords (0.28) to distinguish natural prose from structured data.
• —ASCII Density: Requires >95% ASCII characters.
• —Lexical Richness (MTLD): Calculates the Measure of Textual Lexical Diversity. The threshold is set aggressively high at 65.0 (up from 48.0), retaining only the most linguistically complex and varied text.
• —Toxicity & Safety: Filters content containing explicit NSFW keywords.

Dataset Structure

The dataset is distributed as a series of compressed JSONL shards:

• —tulu-3-sft-mixture_purified_part_000.jsonl.gz
• —tulu-3-sft-mixture_purified_part_001.jsonl.gz
• —...

Data Instance

{
  "messages": [
    {
      "role": "user",
      "content": "Explain the significance of the Battle of Hastings in the development of the English language."
    },
    {
      "role": "assistant",
      "content": "<think>The user is asking about the linguistic impact of the Norman Conquest...</think> The Battle of Hastings in 1066 marked a pivotal turning point..."
    }
  ]
}

Licensing & Attribution

License: Please refer to the license of the original source dataset (allenai/tulu-3-sft-mixture) for terms of use, attribution requirements, and restrictions. This dataset is a filtered derivative and inherits the permissions and limitations of the parent data.