Brainquiver/general-master-en-202608
General · Master · English · 2026-08 English pretraining text, assembled from three public sources, cleaned with one character-level cleaner, and filtered for repetition. 109,337,531 documents and 468,064,046,462 characters. Composition Config Documents Characters What it is fineweb-edu-dedup 65,010,430 297,544,916,118 Web text an educational classifier kept cosmopedia-v2 38,591,146 144,011,993,012 Synthetic prose from a seeded generator… See the full description on the dataset page: https://huggingface.co/datasets/Brainquiver/general-master-en-202608.
General · Master · English · 2026-08
English pretraining text, assembled from three public sources, cleaned with one character-level cleaner, and filtered for repetition.
109,337,531 documents and 468,064,046,462 characters.
Composition
The three configs are separate tables and they are never concatenated here. Load each one by name, for example load_dataset("Brainquiver/general-master-en-202608", "fineweb-edu-dedup").
The character count is exact. The token count is not a field of the name, and not a claim of this card. A token count is a property of a tokenizer. At 4.50 characters per token this corpus is near 104B tokens, against a target of 100B. The real figure follows once the vocabulary is fixed.
Mixture weights are not set here. The corpus holds the sources apart on purpose, so the weight of each one stays a decision of the training stage.
Sources
All three upstream releases are ODC-By 1.0, and this corpus keeps that licence. Two of them derive from CommonCrawl, so a downstream user should also respect the CommonCrawl terms of use.
fineweb-edu-dedup is a contiguous prefix of the release and not a sample of it. We took the first 80 of its 234 shards. Our own shards are numbered from zero and do not carry the upstream numbers, so the audit is where that correspondence is recorded. The count came from a character budget. The target of 100B tokens at 4.50 characters per token is 450B characters. Cosmopedia and finemath supply 180B of that, and one fineweb shard holds about 3.87B. Seventy shards reach the target and eighty leave margin for the filter.
That prefix still spans the whole crawl range. It covers 95 distinct CommonCrawl dumps, from CC-MAIN-2013-20 to CC-MAIN-2024-10. It is not confined to one era. It is, however, not a uniform sample of the release, and the mix is uneven across it. The earliest upstream shard leans toward the 2014 crawls, and shards from the middle of the prefix lean toward 2022 and 2023. Weight by dump if the era matters to your use.
Format
One row is one document. text holds the cleaned prose. doc_id and uniq_ratio are ours and every other column comes from the publisher.
Three columns are common to all configs:
doc_id counts the upstream row and not the kept row, so it stays a coordinate into the upstream shard after the filter removes rows. The sequence therefore has gaps.
`doc_id` starts at zero inside each config and does not run across the corpus. The key of a document is the pair of the config and the doc_id. If a later stage concatenates the three configs into one stream, give each config an offset first.
The remaining columns differ, because each publisher recorded different fields:
source_id is the CommonCrawl record id and not a row number, which is why it does not share the name doc_id. Each token_count and token_length is the publisher's own count, made with a tokenizer that is not ours. Treat it as a threshold and never as a count of our tokens.
One column was dropped. prompt in cosmopedia-v2 holds the instruction that generated each document, it is 25.2 percent of that source, and a language model does not read it.
Cleaning
Every document passes one character-level cleaner, which applies the Brainquiver text cleaning specification in this order:
- Repair mojibake, meaning text that was decoded in the wrong encoding once already.
- Normalise to NFC. Not NFKC, which would fold distinctions that carry meaning.
- Fold every line break convention to a single line feed.
- Apply the substitution tables: typographic quotes and dashes become their keyboard equivalents, ligatures expand, and invisible formatting characters go.
- Delete controls and unassigned code points.
- Collapse whitespace runs. A run of two or more line breaks becomes one blank line. A run of one line break stays one line break. A run of spaces becomes one space.
- Trim.
The text is not lowercased. Lowercase is a decision of the corpus owner and its default is off.
Two of the three counts rose. That is not a sign that the cleaner failed. Step 1 repairs mojibake, and a repair restores a true code point as the corrupted bytes go. fineweb gained NKO, Syriac and Arabic-Indic characters this way, because their UTF-8 had been read as cp1252 at the source. Among the code points that left are the C1 controls from U+0080 to U+009F, which are the signature of exactly that fault.
The uniq_ratio filter
uniq_ratio is the count of distinct non-blank lines over the count of non-blank lines. A document of wholly distinct lines scores 1.0, and a document that repeats a navigation block or a question template scores lower. Every row below 0.80 is removed.
The three sources answer very differently, and the reason is what each publisher did before we received the data. FineWeb applies a Gopher-style repetition filter before its educational classifier runs, so almost nothing is left for this filter to catch. Cosmopedia is generated prose, which rarely repeats itself. finemath is scraped from forums and question banks, and it received neither treatment.
The rejected rows are not published. They stay on the drive, at 3.0 GiB, so that a later reader can confirm the filter took boilerplate and nothing else. Two examples of what it removed from finemath:
At uniq_ratio 0.344, a slide-sharing page: a textbook title, then copyright boilerplate, then a download policy. It is site furniture and holds no mathematics.
At uniq_ratio 0.126, a practice page that repeats one 40-character standards identifier as the heading of every question.
Neither is reachable with the signals the source already carries. Both sit at a quality score of 3.9 to 4.0, which is the tier we preferentially keep.
Intended use
Pretraining an English language model with a causal language modelling objective on text. The other columns exist so a later stage can filter harder, weight a sample, or trace a document back to its page. They are not meant to be part of the training text.
No length filter is applied, and short documents are kept. A consumer that wants a minimum length must apply its own filter. A consumer that packs several documents into one training sequence needs no filter at all.
Limitations
Two of the three sources are web text, so they carry the biases, errors and dated statements of the pages they came from. cosmopedia-v2 is synthetic, generated by a language model, and it inherits that model's biases and its factual errors. Both classifier-selected sources encode a particular idea of quality, and that selection is inherited whole and unexamined. score and int_score are kept but nothing is filtered on them here.
The uniq_ratio filter above removes a document that repeats itself. It never compares one document against another, so it is not a deduplication step. No cross-document deduplication was applied here, beyond what each publisher did upstream.
No filter for personal information has been applied. English only, and the language labels come from upstream classifiers rather than from an independent check.