enPurified/SlimOrca-enPurified-openai-messages

Dataset Card for enPurified/SlimOrca-enPurified-openai-messages

This dataset was updated on January 14th to remove even more math, code, and low quality prose from the dataset. That's why the number below shows it was trimming from 270k.

Dataset Summary

The enPurified collection is an initiative to curate high-fidelity English prose datasets for language modeling. The primary objective is to isolate high-quality natural language text by strictly excising code, mathematics, foreign languages, and low-entropy content from existing high-value datasets.

While specialized datasets exist for coding and mathematical reasoning, SlimOrca-enPurified focuses exclusively on English fluency, narrative structure, and conversational coherence. This dataset is derived from the original SlimOrca dataset, aggressively filtered through a heuristic pipeline to ensure only the highest quality prose remains.

All data has been standardized into the OpenAI Messages format (List[Dict]) to facilitate immediate integration into training pipelines for chat-based models.

The heuristic pruning process reduced the SlimOrca dataset from 517,982 to 180,024 rows of high quality English prose.

Purpose and Scope

The philosophy behind the enPurified collection is that instruction-tuning models benefit from specialized datasets. While many datasets aggregate code, math, and literature, this collection isolates pure English prose.

• —Goal: To provide a cleanest-possible source of English textual data for modeling natural language, distinct from reasoning occurring in code or formal logic notation.
• —Format: Data is structured as conversation history: {"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}.
• —Source Data: This dataset is derived from SlimOrca.

Note on Chunking: While datasets in this collection involving long-form text (e.g., standardEbooks) are segmented using LangChain-based paragraph chunking, this dataset preserves the conversational integrity of SlimOrca while filtering at the message level.

The Purification Pipeline

To ensure the highest quality, the original data was passed through a multi-stage Python heuristic pipeline. Only entries passing all of the following "Gauntlet" checks were retained:

1. Normalization

Before analysis, text is normalized to ensure accurate scoring:

• —Tag Standardization: Reasoning traces (e.g., <|begin_of_thought|>) are converted to standard <think> tags.
• —Boilerplate Exclusion: Generic system prompts (e.g., "You are an AI assistant...") are excluded from density and richness calculations to prevent skewing quality metrics.

2. Heuristic Filters (The Gauntlet)

An entry is discarded if it fails any of the following checks:

Symbol Density Check:

• —Calculates the ratio of code-like symbols ({, }, ;, //, etc.).
• —Threshold: Entries with >5% symbol density are rejected to remove disguised code, logs, or stack traces.

Math and Arithmetic Gate:

• —LaTeX: Filters out explicit LaTeX blocks (e.g., $$, \begin{equation}).
• —Arithmetic Density: Rejects text where >15% of the content consists of raw numbers and operators (e.g., "7716 - 6322 = ...").

Quiz and MCQ Filter:

• —Detects and removes evaluation benchmarks and multiple-choice questions.
• —Triggers: patterns like "Option A", "Select from the following", or Roman Numeral options (e.g., [I], [II]).

Syntax and Code Ban:

• —Hard Bans: Specific substrings trigger immediate rejection (e.g., std::, console.log, var , public static void, echo "$).
• —HTML Safety: Rejects strings containing malformed or non-semantic HTML tags.

Structural Integrity:

• —Short Line Density: Rejects content where >80% of lines are under 20 characters (filtering out lyrics, poetry, or vertical lists).
• —Repetition: Detects excessive line-level repetition indicative of generation loops.

Linguistic Quality:

• —MTLD Richness: Computes the Measure of Textual Lexical Diversity. Text must score 55.0 to ensure vocabulary richness.
• —Prose Identification: Requires a stopword density >20% (ensuring natural language structure) and >95% ASCII characters (filtering foreign languages).
• —Length Heuristics: Enforces a mean word length between 3.5 and 11 characters; rejects responses under 20 words unless they are standard refusals.

Safety:

• —Toxicity Check: strict keyword filtering for NSFW or gambling-related content.

License

This dataset is a derivative work. Users are strictly advised to refer to the original SlimOrca dataset for license details, citation requirements, and usage restrictions. The enPurified team claims no ownership over the underlying content.