Brainquiver/general-web-it-202608

General · Web · Italian · 2026-08

Italian pretraining text, built from the Italian portion of EPFL's FineWeb2-HQ, which is the high quality slice of FineWeb-2. Every document passes one character-level cleaner and a repetition filter.

21,065,052 documents and 66,158,573,443 characters of Italian prose.

Contents

The character count is exact. The token count is not stated, because a token count is a property of a tokenizer and not of the text: the same characters give a different count at every vocabulary size. As a rough guide, 4.3 characters per token puts this corpus near 15.4B tokens, and the real figure follows once the vocabulary is fixed.

Format

One row is one document. text holds the cleaned prose and every other column is provenance or a measurement.

doc_id counts the upstream row and not the kept row. It is therefore a coordinate into the upstream shard. It stays stable when a filter removes rows, so the sequence has gaps where rows went. It runs from 0 to 21,180,301 across 21,065,052 rows.

Two upstream columns are dropped. embeddings is 74.68 percent of every upstream file and holds the vectors that the quality classifier produced, which a language model does not read. file_path is an S3 path to a CommonCrawl record, and dump with url already say where a document came from.

Cleaning

Every document passes one character-level cleaner, which applies the Brainquiver text cleaning specification in this order:

1. 1.Repair mojibake, meaning text that was decoded in the wrong encoding once already.
2. 2.Normalise to NFC. Not NFKC, which would fold distinctions that carry meaning, such as the difference between a superscript digit and a digit.
3. 3.Fold every line break convention to a single line feed.
4. 4.Apply the substitution tables: typographic quotes and dashes become their keyboard equivalents, ligatures expand, and invisible formatting characters go.
5. 5.Delete controls and unassigned code points.
6. 6.Collapse whitespace runs. A run of two or more line breaks becomes one blank line. A run of one line break stays one line break. A run of spaces becomes one space.
7. 7.Trim.

Cleaning changed the number of distinct code points from 21,136 to 20,971. 165 code points left and none arrived, because Italian web text uses few of the characters that step 4 expands into several others.

Character count rose slightly under cleaning, from 66,695,924,258 to 66,702,314,366 across all rows before filtering. That is expected: a ligature that becomes two letters, and a typographic ellipsis that becomes three periods, both add characters.

The uniq_ratio filter

uniq_ratio is the count of distinct non-blank lines over the count of non-blank lines. A document of wholly distinct lines scores 1.0, and a document that repeats a navigation block or a comment template scores lower. Every row below 0.80 is removed.

The removed rows are not published. They are recoverable exactly, because doc_id is an upstream coordinate: the set of removed rows is the difference between the full upstream doc_id range and the range present here.

The rate is low because FineWeb-2 already applies Gopher-style repetition filtering upstream, so this filter catches only what survived that pass.

Provenance

Built from epfml/FineWeb2-HQ, config ita_Latn, revision c0c06e94fd3a44ae9e802b2b0fc533817601eb5e, downloaded on 9 August 2026. That release is the high quality slice of FineWeb-2, selected by a classifier that the EPFL group trained. Nothing here re-crawls the web, and no document is rewritten: the only changes are the character-level cleaning above and the removal of rows below the threshold.

Upstream is ODC-By 1.0, and this corpus keeps that licence. FineWeb-2 derives from CommonCrawl, so a downstream user should also respect the CommonCrawl terms of use.

Intended use

Pretraining or continuation pretraining of an Italian or multilingual language model, with a causal language modelling objective on text alone. The other columns let a later stage filter harder, weight a sample, or trace a document back to its page. They are not meant to be part of the training text.

Limitations

Web text, so it carries the biases, errors and dated statements of the pages it came from. The upstream quality selection is inherited whole and unexamined. quality_score is kept but nothing is filtered on it.

The uniq_ratio filter above removes a document that repeats itself. It never compares one document against another, so it is not a deduplication step. No cross-document deduplication was applied here, beyond what FineWeb-2 did upstream, and minhash_cluster_size is kept so a later stage can dedupe further.

No filter for personal information has been applied. Italian only, and the language label comes from the upstream classifier rather than from an independent check.