liodon-ai/math-dow-mod-synthetic-v1-base7

Math + Cyclic-Time Synthetic Dataset

Synthetic dataset for training a small (~10M-100M param), task-specialized LLM on arithmetic (addition, multiplication), cyclic time arithmetic (days-of-week, months, 12-hour and 24-hour clock), and mod-k remainder probes — generalization-focused rather than memorization, following on from the T2/T5/T10 modular-circuit discussion. Days-of-week, months, and hours are all instances of the same underlying cyclic/modular-addition structure, rendered as different surface forms.

Generate

python3 generate_math_dataset.py --out data_base10

Key flags: --base (numeral base for rendering numbers, default 10 — use --base 6 or --base 7 to produce transfer-experiment variants of the same task distribution once a base-10 model is working), --seed, plus per-task length/range/count knobs (--add-train-lengths, --mod-n-per-modulus, etc. — see --help).

Splits

• —train.jsonl — in-distribution training examples.
• —test_id.jsonl — held-out in-distribution examples (same digit-length / offset / value range as train, but the exact input was never trained on). Deterministic hash-based split (stable_bucket), ~5% of in-range examples, reproducible across runs regardless of PYTHONHASHSEED.
• —test_ood.jsonl — out-of-distribution examples: longer digit lengths for addition/multiplication, larger offsets for days-of-week, and larger values for mod probes than anything seen in train. Tests whether the model learned the underlying algorithm/circuit vs. memorized surface patterns.

Schema

Each line is a JSON object:

{
  "messages": [
    {"role": "user", "content": "127 + 489 = ?"},
    {"role": "assistant", "content": "616"}
  ],
  "task": "addition",
  "meta": {"a": 127, "b": 489, "digits": 3, "base": 10}
}

task is one of addition, multiplication, days_of_week, months, hour_12, hour_24, mod. meta carries the ground-truth operands so correctness/leakage can be re-verified without re-parsing the text. All numbers in the messages text are rendered in the same --base — including the modulus, day/month offset, and hour — so a --base 6 run is self-consistent throughout. hour_12 cycles through 1..12 (clock-style, no zero); hour_24 cycles through 0..23.

Each task has 3-4 paraphrase templates (e.g. "{a} + {b} = ?" vs "What is {a} plus {b}?") so the model isn't just pattern-matching one surface form.

Dataset stats by base

data_base10 (--base 10):
  train:      60935  addition 11490, multiplication 7708, mod 11383, days_of_week 7576, months 7590, hour_12 7579, hour_24 7609
  test_id:     3265  addition 610,   multiplication 392,  mod 617,   days_of_week 424,  months 410,  hour_12 421,  hour_24 391
  test_ood:    7800  addition 1000,  multiplication 1000, mod 1800,  days_of_week 1000, months 1000, hour_12 1000, hour_24 1000

data_base6 (--base 6):
  train:      54967  addition 8508,  multiplication 4686, mod 11434, days_of_week 7560, months 7589, hour_12 7582, hour_24 7608
  test_id:     2905  addition 428,   multiplication 250,  mod 566,   days_of_week 440,  months 411,  hour_12 418,  hour_24 392
  test_ood:    7800  addition 1000,  multiplication 1000, mod 1800,  days_of_week 1000, months 1000, hour_12 1000, hour_24 1000

data_base7 (--base 7):
  train:      56583  addition 9323,  multiplication 5521, mod 11385, days_of_week 7569, months 7608, hour_12 7580, hour_24 7597
  test_id:     3043  addition 490,   multiplication 292,  mod 615,   days_of_week 431,  months 392,  hour_12 420,  hour_24 403
  test_ood:    7800  addition 1000,  multiplication 1000, mod 1800,  days_of_week 1000, months 1000, hour_12 1000, hour_24 1000

Base-6/base-7 have fewer addition/multiplication train examples than base-10 because smaller bases have a smaller unique-operand space at low digit-lengths, so the dedup-based generator hits its attempt cap before reaching the requested count — not an error.

Verified for each base variant: 0 numeric/cyclic-label mismatches, and zero (task, operands) overlap between train/testid/testood.

Not yet done

• —No raw-text (pretraining-style) flattening yet — only the chat-message JSONL form; flatten per whatever format nanochat's data loader expects once that's confirmed.