CoolFace
Datasetpublic

swan07/math-tiers

Math-Tiers: A Tiered Pretraining Corpus for Studying Numerical Reasoning A large-scale English pretraining corpus split into three tiers by mathematical content density. Designed for controlled experiments studying how data composition during pretraining affects numerical reasoning in language models. Tiers Tier Description Shards Size Est. Tokens Sources T0 Pure narrative: no digits, number words, or math 648 542 GB ~113B RedPajama-Book… See the full description on the dataset page: https://huggingface.co/datasets/swan07/math-tiers.

sourceHugging Facecc-by-4.0updated 7mo agoView on Hugging Face
0likes6.6kdownloads
Dataset Card

Math-Tiers: A Tiered Pretraining Corpus for Studying Numerical Reasoning

A large-scale English pretraining corpus split into three tiers by mathematical content density. Designed for controlled experiments studying how data composition during pretraining affects numerical reasoning in language models.

Tiers

TierDescriptionShardsSizeEst. TokensSources
T0Pure narrative: no digits, number words, or math648542 GB~113BRedPajama-Book, PleIAs/English-PD, Project Gutenberg, Institutional Books, FineWeb
T1Everyday numeric language: blocks formal math only1,216314 GB~66Ballenai/c4 (English)
T2Full math content: unfiltered751580 GB~121BHuggingFaceTB/finemath (finemath-3plus)
Total2,6151,437 GB~300B

Format

Each tier is stored as sharded JSONL files: T0/T0_0000.jsonl, T1/T1_0000.jsonl, T2/T2_0000.jsonl, etc.

Each line is a JSON object with:

json
{"text": "...", "source": "english-pd", "token_estimate": 1234}
  • —text: The filtered document text
  • —source: Origin dataset identifier
  • —token_estimate: Approximate whitespace-split token count

Filtering

All tiers use sentence-level filtering: documents are split into sentences (NLTK punkt), individual sentences matching the blocklist are removed, and remaining sentences are rejoined. This preserves more text than paragraph-level filtering.

T0 Blocklist (aggressive: removes all numeric content)

  • —Digits: All characters 0-9
  • —Operators: + - * / = ^ % < > and Unicode math symbols
  • —Fraction characters: ½ ¼ ¾ etc.
  • —Number words: zero through trillion, ordinals (first–twelfth), once/twice/thrice, half/quarter/double/triple/dozen
  • —Math terms: equation, variable, polynomial, derivative, integral, theorem, eigenvalue, topology, etc.
  • —Patterns: LaTeX math ($...$, \frac{}, \sum, \int, etc.)

T1 Blocklist (moderate: removes formal math only)

  • —No digit or operator blocking — everyday numbers pass through
  • —Math terms: equation, variable, polynomial, derivative, integral, theorem, eigenvalue, topology, etc.
  • —Patterns: LaTeX math expressions

T2 Blocklist

None. All content from finemath-3plus is included.

Intended Use

This corpus supports a pretraining experiment with the following design:

  1. 1.Base model: Train from scratch on T0 (pure narrative) for 60B tokens
  2. 2.Model 0: Continue base on T0 (held-out shards) for 20B tokens
  3. 3.Model 1: Continue base on T1 (everyday numeric) for 20B tokens
  4. 4.Model 2: Continue base on T2 (full math) for 20B tokens

Comparing Models 0/1/2 isolates the effect of mathematical content exposure during the second training phase, controlling for total compute and training procedure.

Sources