liodon-ai/math-dow-mod-synthetic-v1

Math + Cyclic-Time Synthetic Dataset

Synthetic dataset for training a small (~10M-100M param), task-specialized LLM on arithmetic (addition, multiplication), cyclic time arithmetic (days-of-week, months, 12-hour and 24-hour clock), and mod-k remainder probes — generalization-focused rather than memorization, following on from the T2/T5/T10 modular-circuit discussion. Days-of-week, months, and hours are all instances of the same underlying cyclic/modular-addition structure, rendered as different surface forms.

Generate

python3 generate_math_dataset.py --out data_base10

Key flags: --base (numeral base for rendering numbers, default 10 — use --base 6 or --base 7 to produce transfer-experiment variants of the same task distribution once a base-10 model is working), --seed, plus per-task length/range/count knobs (--add-train-lengths, --mod-n-per-modulus, etc. — see --help).

Cyclic tasks: story-wrapping + worked-reasoning traces

The four cyclic tasks (days_of_week, months, hour_12, hour_24) are built differently from the arithmetic/mod tasks, to address a specific failure mode: a 100M-param model has no way to compute (start - offset) mod cycle_len correctly in a single forward pass if the assistant turn is just the bare answer — it has no token positions to actually do the subtraction and mod-reduction, so it tends to fall back to surface pattern-matching ("if the question mentions months, just output a month") instead of learning the circuit.

To address this:

• —Worked-reasoning trace, composed not templated: the assistant turn spells out the computation via 4 independently-varying phrase pools — opener / operation / mod-reduce / conclusion — combined at random (4^4 = 256 wordings), e.g. "Position-wise, Saturday is 5 within the 7-day cycle. That gives 5 + 10 = 15. Taking 15 mod 7 leaves 1. That corresponds to Tuesday." vs "Thursday corresponds to position 3 out of 7 days. 3 - 13 = -10. -10 mod 7 = 4. So it lands on Friday." Every conclusion variant ends in "... {label}." with the label as the last word, so verification stays format-agnostic regardless of which wording was drawn. This is a deliberate design choice, not an accident: the computation structure (position → op → mod-reduce → label) stays identical every time — that consistency is what lets a small model actually learn the algorithm — only the wording around it varies, both so the raw-text corpus doesn't read as one stamped-out template, and to stop the model from overfitting to one exact reasoning string.
• —Story-wrapped questions, composed not templated: ~70% of cyclic examples combine a random scenario setup (museum exhibit rotations, bakery oven timers, flight schedules, library due dates, factory shift changes, etc. — 8 per task/direction) with a random tail question (4 per task/direction), giving 32 surface forms per task/direction instead of a handful of fixed full-sentence templates. The remaining ~30% use the original plain templates.
• —Offsets weighted toward small values: only a handful of simple "N months/hours/days" question shapes exist, so uniformly sampling any offset gives too few unique (start, offset, direction) facts to be rich data — and repetitive large-offset drills ("604 months ago, again and again") aren't useful training data anyway. Train offsets are drawn mostly from 1-12 (75% of the time, --cyclic-small-frac) with a thin tail up to 20 (--cyclic-tail-hi); each unique fact is paraphrased up to --cyclic-repeat-cap (default 10) times with different question/reasoning compositions to fill out the requested count without inventing new underlying facts. test_ood pushes offsets well past that — a near zone (21-60) and a smaller far zone (61-600, --cyclic-ood-far-frac) that specifically tests far/multi-cycle extrapolation, never seen in train.

Arithmetic (addition, multiplication) and mod tasks are unchanged this round — bare numeric answers, no reasoning trace yet.

Splits

• —train.jsonl — in-distribution training examples.
• —test_id.jsonl — held-out in-distribution examples (same digit-length / offset / value range as train, but the exact input was never trained on). Deterministic hash-based split (stable_bucket), ~5% of in-range examples, reproducible across runs regardless of PYTHONHASHSEED.
• —test_ood.jsonl — out-of-distribution examples: longer digit lengths for addition/multiplication, larger offsets for days-of-week, and larger values for mod probes than anything seen in train. Tests whether the model learned the underlying algorithm/circuit vs. memorized surface patterns.

Schema

Each line is a JSON object:

{
  "messages": [
    {"role": "user", "content": "127 + 489 = ?"},
    {"role": "assistant", "content": "616"}
  ],
  "task": "addition",
  "meta": {"a": 127, "b": 489, "digits": 3, "base": 10}
}

Cyclic-task assistant turns are a worked-reasoning trace ending in the label, not a bare answer:

{
  "messages": [
    {"role": "user", "content": "If it is July now, what month will it be in 3 months?"},
    {"role": "assistant", "content": "July is position 6 in the 12-month cycle. 6 + 3 = 9, and 9 mod 12 = 9, so the answer is October."}
  ],
  "task": "months",
  "meta": {"start_month": "July", "offset": 3, "direction": "after", "base": 10}
}

task is one of addition, multiplication, days_of_week, months, hour_12, hour_24, mod. meta carries the ground-truth operands so correctness/leakage can be re-verified without re-parsing the text. All numbers in the messages text are rendered in the same --base — including the modulus, day/month offset, and hour — so a --base 6 run is self-consistent throughout. hour_12 cycles through 1..12 (clock-style, no zero); hour_24 cycles through 0..23.

Arithmetic/mod tasks have 3-4 paraphrase templates. Cyclic tasks draw from a mix of 3 plain templates + 5 story-scenario templates per direction (see above) so the model isn't just pattern-matching one surface form.

train_raw.txt is a flattened, role-marker-free version of train.jsonl ({user question} {assistant answer/reasoning} per line, blank-line separated) for pretraining-style corpora — flatten format/separator can be adjusted once nanochat's data-loader format is confirmed.

Dataset stats

data_base10 (--base 10, composed story-wrapped + reasoning-trace cyclic tasks):
  train:      50061  addition 11490, multiplication 7708, mod 11437, days_of_week 2322, months 4292, hour_12 4227, hour_24 8585
  test_id:     2639  addition 610,   multiplication 392,  mod 563,   days_of_week 178,  months 208,  hour_12 273,  hour_24 415
  test_ood:    7800  addition 1000,  multiplication 1000, mod 1800,  days_of_week 1000, months 1000, hour_12 1000, hour_24 1000
  train_raw:  50061 lines (flattened, no role markers)

Cyclic-task train counts are lower than the arithmetic tasks because the small-offset weighting (mostly 1-12, tail to 20) only produces a few hundred to ~1000 unique underlying facts per task (cycle_len × offset_range × 2 directions); each is paraphrased up to --cyclic-repeat-cap (10) times to fill out the count — not an error, a deliberate small/rich-phrasing tradeoff per the plan above.

Verified: 0 numeric/cyclic-label mismatches across all 60,500 chat examples (reasoning-trace math re-derived independently from meta and compared), and zero (task, operands) overlap between train/testid/testood (32,706 / 1,694 / 7,800 unique facts respectively).

data_base6/ and data_base7/ (pushed earlier, transfer-experiment variants) still use the old plain-template/bare-answer schema for months/hours — not yet regenerated with story-wrapping + reasoning traces. Base-10 is the priority baseline for now; base-6/7 will be refreshed once a base-10 model is trained and working.

Not yet done

• —base-6/base-7 story+reasoning refresh (see above).
• —Reasoning traces for addition/multiplication/mod (currently bare answers only) — same "can't compute in one forward pass" argument likely applies to multi-digit arithmetic too, just out of scope for this iteration.
• —train_raw.txt separator/format not yet confirmed against nanochat's actual data loader.