domofon/ifm-cleaned-pretrain-30B

IFM Cleaned Pretrain — 30B target

credits to https://huggingface.co/datasets/IFM/Pretrain-Behaviors

Status: complete. Published: 6,155,901 documents; 30,000,015,784 source-annotated tokens. Target: 30,000,000,000 source-annotated tokens, approximately equal across all seven categories.

This repository contains text only in Parquet: earlier shards were format-cleaned; subsequent shards contain source text without the cleaner. There are no token-ID arrays or binary token shards. Each text ends with <|endoftext|> (the SmolLM2 end-of-text marker).

Selection and accounting

Source commit: 3345e13d7f3f6d0ecb5fdd67b37aed289f3191f5. Initial category minimum source lengths: {"data-science": 1825, "format-rewrites": 6350, "games": 7225, "general": 3150, "other": 2925, "planning": 3875, "reasoning": 5000}. These high thresholds are estimated from source samples to favor long documents while filling each quota. If a category is exhausted, successive nonoverlapping bands lower its minimum by 250 tokens. This is threshold selection, not an exact global sort. The source annotation is preserved as source_token_count. No tokenizer is run. Counts are approximate for the cleaned text and are not recomputed after formatting changes or EOS insertion. Documents are never cut to fit a file or category quota; the final document may slightly exceed the category target. Source files are visited in the order recorded in pipeline/source-plan.json; this is not a uniformly random sample. Newly selected documents may contain tables. Earlier shards retain the table-selection policy of their recorded configuration. Tables are never cut out of a document. The cleaner is disabled for newly published shards: wording, Markdown, emoji, whitespace, code, and mathematical text are preserved. Only terminal EOS normalization is applied. Historical cleaner rejection counters describe earlier processing.

Files and provenance

data/<category>/part-XXXXXX.parquet contains text, source_token_count, source_file, source_row, source_sha256, and text_sha256. manifests/<category>/part-XXXXXX.json records file hashes, sizes, document counts, and the next source cursor. state/ contains resumable checkpoints; progress.json contains aggregate progress. pipeline/ contains the current execution configuration, pinned source plan, and preparation code. Previous configurations and code are archived under pipeline/revisions/<config_sha256>/; existing receipts retain their original configuration hash. The configuration records whether the cleaner and table exclusion are enabled.

Historical formatting cleanup (earlier shards only)

• —Preserve tables and their structure; do not convert them to prose.
• —Remove excess bold and italic markup while retaining short existing bold labels at the beginning of explanatory list items.
• —Preserve headings for substantial sections, including explanatory lists, and coherent series of shorter sections.
• —Retain useful existing separators between substantial sections; remove redundant or decorative separators.
• —Remove recognized decorative heading emoji; retain inline emoji, quoted examples, meaningful status symbols, and numeric values.
• —Collapse surplus blank lines while preserving code, mathematical fragments, links, and other protected literals.
• —Preserve wording and list structure through source-coordinate edits and conservative validation; ambiguous transformations are excluded.
• —Preserve unquoted arithmetic operators, code identifiers and other ambiguous asterisk-bearing lines. Reject any result that changes the sequence of letters or digits (apart from equivalent keycap-number normalization).
• —Append exactly one terminal <|endoftext|> marker; no text rewriting model is used.

<!-- chatml-import:start -->

ChatML subset

ChatML has split train, a single text column, and 8,144,193 rows in chat_ml/*.parquet. Status: complete.

• —Original Claude corpus: 3,006,171 training rows. Its role headers were corrected to <|im_start|>{role}\n; message bodies and existing EOS were preserved. Details: imports/chat_ml/role-header-repair.json.
• —Additional chatml_sorted.jsonl: 5,138,022 rows, copied exactly in source order. No text cleanup, normalization, filtering, deduplication, EOS changes, or tokenization was performed for this addition. Source file SHA-256: 45029659a19547bbd7822d1ad2cf850fc8c3e4cd9d680384c27b84b178863055.

The seven IFM subsets and their counts above are unchanged. ChatML rows are additional and are not included in the IFM 30B token total. The full file inventory and provenance are in imports/chat_ml/manifest.json; addition receipts are in imports/chat_ml/additions/.

from datasets import load_dataset
data = load_dataset("domofon/ifm-cleaned-pretrain-30B", "ChatML", split="train", streaming=True)

<!-- chatml-import:end -->

ChatMLcodeinstruct subset

ChatML_code_instruct has split train, one text column, and 78,264 documents. Source: vilm/Code-Pretrained-Instruction, pinned revision 50c899fd737e71de040f7e0f282855af2da0c61b.

Each document contains two consecutive layers: the complete original text, unchanged, then a blank line, —, a blank line, and a ChatML conversation. The source ### Instruction: body becomes the user message; the source ### Output: body becomes the assistant message. Embedded headings ending immediately after the colon remain part of their original message bodies. Code, indentation, whitespace, and wording are preserved.

Exactly 39,132 documents contain this system message: ROLE: You are a code writer by user instruction.. The other 39,132 documents have no system message. Selection uses a deterministic SHA-256 ranking of the source revision and row index. The combined document has one terminal <|endoftext|>; there is no EOS between the raw and ChatML layers. No cleaner or tokenizer is run.

Provenance and checksums: imports/chatml_code_instruct/manifest.json. The existing IFM and ChatML subsets and their counts are unchanged.

ChatMLcodecot subset

ChatML_code_cot/train contains 63,007 documents in a single text column. Source: domofon/evol-instruct-code-cot-80k, pinned revision ade4df675989468ec9be21c1e7a8faac4f8f060c; all source training rows, in original order. Only ChatML is emitted; there is no raw-document prefix.

The user body preserves instruction. The assistant body is the original thinking (already enclosed in <think>...</think>) plus two newlines and the original response. All source field text, whitespace, and code are preserved. Each document ends with <|im_end|> followed by a newline and <|endoftext|>.

Forty English prompts request step-by-step reasoning in XML <think>...</think> before an answer outside the tags. The prompt is the system message for 31,503 rows; for 31,504 rows it is appended to the user message after two newlines, with no system message. The one-row difference is necessary for 63,007 rows. Deterministic SHA-256 ranking chooses placement; all 40 variants are balanced within each placement group (counts differ by at most one). No cleaning, filtering, deduplication, tokenizer, or token counting is applied. Provenance, every prompt variant, counts and checksums: imports/chatml_code_cot/manifest.json. Existing subset data and counts are unchanged.

ChatMLfinetomecot subset

ChatML_finetome_cot/train contains 99,652 documents in a single text column. Source: domofon/finetome-cot-100k, pinned revision 032862d597c75767ccfed7b0fd5f5c2ccff24329; all source training rows, in original order. Only ChatML is emitted; there is no raw-document prefix.

The user body preserves instruction. The assistant body is the original thinking with only its outer <think>...</think> tags renamed to <reasoning>...</reasoning>, plus two newlines and the original response. All text inside the reasoning block, instructions, responses, whitespace, and code are preserved. Each document ends with <|im_end|> followed by a newline and <|endoftext|>.

Forty English prompts request step-by-step reasoning in XML <reasoning>...</reasoning> before an answer outside the tags. The prompt is the system message for 49,826 rows; for 49,826 rows it is appended to the user message after two newlines, with no system message. The placement split is exactly 50/50. Deterministic SHA-256 ranking chooses placement; all 40 variants are balanced within each placement group (counts differ by at most one). No cleaning, filtering, deduplication, tokenizer, or token counting is applied. Provenance, every prompt variant, counts and checksums: imports/chatml_finetome_cot/manifest.json. Existing subset data and counts are unchanged.

TxT360_QA subset

TxT360_QA/train contains 500,000 source records, each in a single text column. Source: IFM/TxT360-v2, configuration txt360-qa, train split, pinned revision 63d6dbf4d469058a8c7868909be18a5ada487be1. Source license: CC BY 4.0. Source records may contain an article followed by multiple question-answer pairs; records are not split into individual pairs.

Selection takes the first 500,000 records with at least one Q→A pair in lexicographically sorted source files and their original row order. A label is Q: or A: at the start of a line, followed by whitespace or end of text. At least one adjacent Q,A in the sequence of labels is required. Other records are skipped solely to obtain the requested QA-bearing records.

The only text edits are Q: → Human: and A: → Assistant: at those line starts. All other characters, original article text, whitespace, code, existing markers, and incomplete final questions are preserved. No ChatML framing, EOS, cleaning, deduplication, or token counting is added.

Import uses bounded HTTP range reads without a local source cache. Each output shard is checked row by row, uploaded, read back for SHA-256 verification, and deleted locally before the next shard is built. The completed collection has 20 shards. Provenance, source row cursors, counts, and checksums: imports/txt360_qa/manifest.json. Other subset data and counts are unchanged.