oklenAI/UDM_cleaned_docs

UDM cleaned docs

6,029,052 web pages reduced to just their mathematical content, extracted verbatim by oklenAI/udm_doc_extract_qwen3.5_2B — a 2B model distilled from GPT-5.6.

Every row is model output, not human-curated text. The extract field is what the model returned for that page; the source page text is not included. Read Two repetition flags below before filtering — the obvious flag is not the one you want.

How it was built

Extraction ran greedy (no sampling, no repetition penalty) under vLLM 0.28.0 with a per-page token budget of min(24576, 1.25 x input_tokens + 256, 40960 - input_tokens). Sizing the budget to the page rather than to a constant matters: extraction length scales with input, so a flat cap silently truncates exactly the long pages. Input-side truncation hit 1 page out of 6,029,052.

Schema

Two repetition flags, and why the obvious one is wrong

A high whole-document repeat rate has two completely different causes, and they need opposite treatment:

• —the model looped — greedy decoding locks onto a short n-gram and repeats it until the token budget runs out. Broken output.
• —the page repeats itself — a forum thread quoting the same code block, a worksheet built from a template, a page that prints its theory section twice. The model transcribed it faithfully. Good output.

rep20 > 0.30 cannot tell these apart. rep_tail can, because a genuine loop is still looping when the budget runs out while a repetitive page is still producing new text at the end. The split is bimodal, not a continuum — sampled non-loop pages score exactly 0.000 — so the threshold is not delicate.

So filter on `loop`, not on `degenerate` — about a third of degenerate is good extraction of repetitive source pages, and worksheets and problem sets are exactly the pages this corpus is for. Nothing was deleted; both flags ship so you can choose.

Looping is strongly length-dependent:

Corpus statistics

score distribution: 2 10.7%, 3 35.5%, 4 10.7%, 5 27.9%, 6 8.7%, 7 6.5%

Usage

from datasets import load_dataset

ds = load_dataset("oklenAI/UDM_cleaned_docs", split="corpus", streaming=True)
clean = (r for r in ds if not r["loop"])          # drop the ~0.6% the model broke on

The split is called corpus, not train — this is extracted document text, not anyone's training split.

How faithful is it?

Faithfulness here means agreement with the teacher, GPT-5.6-Sol, called exactly as it was called to build the model's training data. It does not measure whether the teacher was right.

449 pages were drawn from this corpus itself — not from the model's evaluation set — in three length bands, and re-extracted by the teacher:

Two things worth reading off this table. Long-page charF1 on production pages is 0.8811, slightly above the 0.8668 the model card reports on its held-out long ruler — the corpus is not harder than the ruler said it would be. And the repetition tail shows up independently: 2.01% of these 149 long pages exceed a 0.3 repeat rate, against 3.454% measured over all 846,419 long pages in the corpus. Those two are consistent rather than equal — at n=149 the sample simply cannot resolve a rate this small — but the check was run by different code on a different sample and still lands in the same place.

Limits

• —Student, not oracle. Every fidelity number is agreement with GPT-5.6. Nothing here was audited against the source pages by a human.
• —Long pages are measurably harder. charF1 0.8811 on long pages vs 0.9693 on short ones, and the loop rate is 13.9x higher in the >=20k band than in the <5k band.
• —Formula-token excess. The teacher check shows 21.48% excess notation tokens on long pages. The prompt mandates notation repair, so this is not that share of hallucination — but measure repetition directly rather than trusting token counts as a proxy.
• —Nothing beyond ~50,000 characters. The source pool tops out at 49,998 chars.
• —English only, and only pages the source dataset already labelled as mathematical.
• —Empty output is a valid answer where a page carries no substantive math; the observed empty rate is 0.001%.

Provenance and licensing

Page text originates from TeraflopAI/udml2-labeled, which declares no license. This derived corpus adds only the extraction; check the upstream dataset before redistributing or training on it.

Extraction model: oklenAI/udm_doc_extract_qwen3.5_2B. The prompt used is shipped with the model as extract_prompt.txt.