oddadmix/Nawah-Math-Reasoning

1---2license: apache-2.03language:4- ar5base_model: oddadmix/50M-2048-Emhotob6datasets:7- oddadmix/arabic-math-reasoning-synth8- oddadmix/gsm8k-reasoning-ar9- Omartificial-Intelligence-Space/Arabic_Reasoning_Dataset10library_name: transformers11pipeline_tag: text-generation12tags:13- arabic14- reasoning15- chain-of-thought16- math17- gsm8k18- small-language-model19- slm20- llama21- sft22---23 24# Nawah-Math-Reasoning — نموذج استدلال رياضي عربي25 26A **51.8M-parameter** Arabic math reasoning model. It writes its derivation step by step27inside `<think>…</think>`, then gives the answer. It is small enough to run on a CPU.28 29> **بالعربية:** نموذج عربي صغير (~52 مليون معامل) لحل المسائل الحسابية: يكتب خطوات30> تفكيره داخل وسم `<think>` ثم يعطي الإجابة. صغير بما يكفي ليعمل على المعالج (CPU).31 32| | |33|---|---|34| 🤗 **Demo** | [`oddadmix/Nawah-Math-Reasoning-Demo`](https://huggingface.co/spaces/oddadmix/Nawah-Math-Reasoning-Demo) |35| 🧩 **Base model** | [`oddadmix/50M-2048-Emhotob`](https://huggingface.co/oddadmix/50M-2048-Emhotob) — Llama architecture, 12 layers, hidden 512, 2048 ctx, pre-trained from scratch on ~20B Arabic tokens |36| 📚 **Data** | [`arabic-math-reasoning-synth`](https://huggingface.co/datasets/oddadmix/arabic-math-reasoning-synth) · [`gsm8k-reasoning-ar`](https://huggingface.co/datasets/oddadmix/gsm8k-reasoning-ar) · [`Arabic_Reasoning_Dataset`](https://huggingface.co/datasets/Omartificial-Intelligence-Space/Arabic_Reasoning_Dataset) |37| 🛠️ **Training code** | [`code/`](https://huggingface.co/oddadmix/Nawah-Math-Reasoning/tree/main/code) in this repo — data generation, translation, SFT, eval, GRPO |38| 🔤 **Vocab** | 32004 (4 chat/reasoning tokens added to the 32000 base vocab) |39 40## Results41 42Number agreement, greedy decoding. **Every cell is measured on identical held-out rows.** The43`Arabic_Reasoning` and `GSM8K-ar` rows are the eval splits fixed at the start of the project and44never re-drawn; the synthetic rows are pinned to the same 1,000 items every earlier version was45scored on.46 47The `v3 / v4 / v5` columns are internal development runs, kept here because they are what makes48the release number mean something. They are not published — the numbers are, so the ablation is49readable without them.50 51| eval set | n | v3 | v4 | v5 | **release** |52|---|---:|---:|---:|---:|---:|53| GSM8K-ar | 600 | 77.3% | 19.5% | 76.0% | **79.0%** |54| Arabic_Reasoning | 400 | 65.8% | 50.2% | **75.2%** | 73.0% |55| synthetic math | 1000 | 2.0% | 35.6% | 39.1% | **40.4%** |56| **synthetic relational** | 400 | — | — | 34.0% | **52.2%** |57 58**The relational row is what this release adds.** On problems whose difficulty is the *relation*59between quantities (`ضعف`, `نصف`, `أكثر بـ…`) rather than the arithmetic, it scores60**52.2%** where the previous run scores 34.0% — a **+18.2 point** gain and61the largest single-cell move anywhere in the development ladder. It did not cost the other62distributions: GSM8K-ar is simultaneously the best of the series at **79.0%**, and63synthetic math gains +1.3.64 65The one regression is `Arabic_Reasoning` at **-2.2** against v5 — on 400 rows that is66close to sampling noise, but it is the second consecutive mix where this column is the give.67 68| detail | GSM8K-ar | Arabic_Reasoning | synth math | synth relational |69|---|---:|---:|---:|---:|70| final-answer number correct | 79.0% | 77.5% | 46.2% | 54.2% |71| all numbers match | 79.0% | 73.0% | 43.5% | 52.2% |72| well-formed `<think>` + answer | 100.0% | 98.8% | 99.5% | 99.5% |73| mean reasoning length | 39 tok | 90 tok | 59 tok | 45 tok |74 75*(the synth-math column here is the 400-row mix cell; the 40.4% in the table above is the761,000-row set used for the cross-model comparison.)*77 78Reproduce any cell with `code/eval_reasoning.py` — it is the same script for every model and every79row, which is the only reason these are comparable.80 81### The final checkpoint ships, and eval loss disagrees82 83Loss bottoms at **0.4559** (epoch 1.86) and rises to **0.5154** by84epoch 5 — yet the epoch-5 weights are the better model. This was measured directly on85an earlier run whose corpus contained **no repeated rows**, which rules out memorisation: the86minimum-loss checkpoint scored 30.9% where the final scored 35.6%. It happened on four consecutive87runs. `train_reasoning.py` therefore takes `LOAD_BEST=0`, and that is not an oversight.88 89## Training mix90 91275,639 rows, 31.1M tokens/epoch:92 93| source | rows | tokens/epoch | share |94|---|---:|---:|---:|95| [`oddadmix/arabic-math-reasoning-synth`](https://huggingface.co/datasets/oddadmix/arabic-math-reasoning-synth) | 118,062 | 16.79M | 53.9% |96| [`oddadmix/gsm8k-reasoning-ar`](https://huggingface.co/datasets/oddadmix/gsm8k-reasoning-ar) | 140,969 | 11.88M | 38.2% |97| [`Omartificial-Intelligence-Space/Arabic_Reasoning_Dataset`](https://huggingface.co/datasets/Omartificial-Intelligence-Space/Arabic_Reasoning_Dataset) | 16,608 (5,536 × 3) | 2.45M | 7.9% |98 99Of the synthetic corpus's 120,462 rows, 20,139 are **relational** problems generated100specifically for this release, after a `pass@k` diagnostic showed the previous model went 0/8 on101`ضعف`-style problems and a corpus audit found the relation appears in only 1.34% of rows. The102synthetic eval split was **pinned, not re-drawn** when those rows were added: re-shuffling would103have moved 1,955 of the 2,000 previously held-out items into train, turning that column into a104memorisation score.105 106Full fine-tune from the base (not from the previous version). Loss on the assistant turn only, user107prompt masked with `-100`. `Arabic_Reasoning` is ~25× smaller than GSM8K, so it is repeated 3×.108 109| | |110|---|---|111| epochs | 5 (21,535 steps) |112| effective batch | 64 |113| learning rate | 3e-4 cosine, 200 warmup steps |114| max length | 768 tokens (mix p100 is 703 — nothing truncated) |115| precision | bf16 |116| checkpoint | final (`load_best_model_at_end` disabled — it picks the worse model) |117| hardware | 1× RTX A6000, ~85 min |118 119## Usage120 121```python122from transformers import AutoModelForCausalLM, AutoTokenizer123import torch124 125model_id = "oddadmix/Nawah-Math-Reasoning"126tok = AutoTokenizer.from_pretrained(model_id)127model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).eval()128 129messages = [{"role": "user", "content": "اشترى خالد 4 دفاتر بسعر 15 جنيهًا للدفتر، ودفع بورقة 100 جنيه. كم المبلغ المتبقي؟"}]130prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)131ids = tok(prompt, return_tensors="pt")132 133out = model.generate(**ids, max_new_tokens=384, do_sample=False)134print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=False))135```136 137Split the parts with `re.match(r"\s*<think>(.*?)</think>(.*)", completion, re.S)`.138Decode with `skip_special_tokens=False` — `<think>` and `</think>` are real tokens in this139tokenizer, and stripping them destroys the split.140 141It is **single-turn**: one user message per call. Chat history is out of distribution.142 143**Answer style is not something you can request.** The three corpora disagree — GSM8K rows end in a144bare numeral, the other two in an `إذن، …` sentence — and arithmetic word problems look alike in145all of them, so the model picks a style per prompt. **Score it on number agreement, not exact146string match**, and parse the answer by extracting its numbers.147 148## Limitations149 150At ~52M parameters this is a **proof of concept**, and the honest headline is the151synthetic columns — **40.4%** and **52.2%** on multi-step problems, well below the15279.0% it scores on GSM8K's narrower phrasing. Arithmetic is the dominant failure mode: the153reasoning is usually structurally right, one computation step is wrong, and the model then stays154faithful to its own bad number.155 156Each corpus brings its own defect. The GSM8K half is machine-translated, its 140,969 rows expanding157from only 2,814 question patterns, so that score partly reflects narrow phrasing. The synthetic158half is verified for **arithmetic, not for sense** — rows survive where every equation checks out159but a step introduces an entity never mentioned, or the answer resolves the reverse of what was160asked. The `Arabic_Reasoning` half excludes open-ended expository rows (they have no final answer161to place after `</think>`), so expository prompts remain out of distribution.162 163Everything is MSA; the synthetic corpus's region axis sets currency and context, not dialect. The164Arabic inherits source artifacts including inconsistent gender agreement. Its reasoning trace is165not a faithful account of any internal computation. Do not use it for anything consequential.166 167## Citation168 169```bibtex170@misc{nawah_math_reasoning_2026,171  title  = {Nawah-Math-Reasoning: a 52M-parameter Arabic chain-of-thought math model},172  author = {Ahmed Wasfy},173  year   = {2026},174  url    = {https://huggingface.co/oddadmix/Nawah-Math-Reasoning}175}176```177