Prateek-Tiwari10/LMA_phase3_data

LMA Phase 3 — transitive reasoning dataset (Hindi / Nepali)

Synthetic transitive-comparison problems over people, in Hindi and Nepali, with chain-of-thought targets. Built for a study of data scaling and domain generalization in small from-scratch language models.

The models finetuned on it are at `Prateek-Tiwari10/LMA_phase3`.

Layout

manifest_scaling.json          machine-readable index of every file below
train/{hi,ne}/train_{10,30,50,60}k.jsonl     nested training subsets
validation/{hi,ne}/val_mix.jsonl             train-pool names
validation/{hi,ne}/val_heldout.jsonl         held-out names  <- select on this
test/{hi,ne}/in_domain/*.jsonl               6 slices: trained domains
test/{hi,ne}/cross_domain/<domain>/*.jsonl   16 slices: 4 unseen domains

The four training subsets are strictly nested: the 10k set is a subset of the 30k set, which is a subset of the 50k, which is a subset of the 60k. They are prefixes of one seeded shuffle, so the scaling curve means "the same data plus more", not "four different datasets".

Design

Task families: relation (pure comparisons), numeric (numeric anchor + bound question), mixed (anchor present, relational question). Training mixture 50/30/20.

Every record carries its prompt, completion, answer, true_order, premises, query, depth, domain, task_family and its own derived seed, so any single record can be regenerated in isolation.

Why the name pools are large

An earlier build reused each name ~455 times, and the models learned to recall a memorised name inventory rather than copy names out of the prompt. The pools here are ~17x larger, putting repetitions per name at 7 (10k) to 41 (60k) — far below that regime. Copy-rate is reported per model as a check that the fix held.

Verification

Generated by phase3/data_gen/ (released in the project repo) and gated before release: unique total order per problem (G1), an independent solver agreeing on every answer (G2), queried pairs never stated as premises (G3), sequence lengths inside the 512-token context (G5), byte-identical regeneration from the master seed (G7), and answer extraction recovering 100% of labels (G8).

The two languages are exact structural mirrors — same seed, same code, same premise graphs — so a Hindi-vs-Nepali gap is a language effect, not a data-composition effect.

Known caveats

• —temperature compares the attribute (X का तापमान Y से ज़्यादा है) where the other three unseen domains predicate an adjective (X, Y से तेज़ है), because neither language has a predicate adjective for a person's temperature. It therefore varies frame and domain; report it separately, never averaged into the cross-domain mean.
• —The Nepali cross-domain corpus misses a per-name answer-independence check by 0.023 of a z-unit (one name of 1,090; global chi-square passes at p = 0.309). Disclosed rather than silently re-cut.
• —The unseen-domain phrasings have not had a native-speaker review.