abute-21/amharic-geez-numerical-blindspot

Blind Spots of Frontier Models: Tokenizer-Induced Numerical & Temporal Collapse in Ge'ez and Amharic

• —Author: Teshome Birhanu Cheru
• —Affiliation: Addis Ababa University, Electrical and Computer Engineering
• —Target Fellowship: Fatima Fellowship 2026 Technical Challenge
• —Evaluated Model: Qwen/Qwen2.5-3B-Instruct (3B Parameters)
• —Artifacts Repository: Evaluation Benchmark (benchmark_ethiopian_reasoning.json), Empirical Outputs (eval_results.json), Colab Notebook (qwen_ethiopian_reasoning_eval.ipynb)

1. Identified Blind Spot & Motivation (Lived Experience)

Growing up in Ethiopia and completing my engineering studies at Addis Ababa University, daily civic, cultural, and logistical activities exist under a dual cognitive framework: the Western Gregorian system alongside the traditional Ethiopian (Ge'ez) calendar and numeral heritage.

The Ethiopian calendar operates on a distinct astronomical and computational structure: twelve uniform months of 30 days, followed by a thirteenth transitional month (Pagumē / ጳጉሜ) consisting of 5 days (6 in leap years), lagging roughly 7 to 8 years behind the Gregorian calendar. Concurrently, Ge'ez numerals (፩, ፪, ፫ ... ፲, ፳, ፻) represent an additive-decimal system without a zero symbol, historically and actively embedded in religious records, formal certificates, and national documents.

Standard NLP benchmarks (MMLU, GSM8K, MATH) evaluate models almost exclusively on English and high-resource Latin-script tokens. The critical capability gap identified here is:

Current frontier open-weight models suffer severe sub-word and byte-level tokenizer fragmentation over Ethiopic scripts, causing an immediate breakdown in additive numeral arithmetic, symbol recognition, and non-Gregorian modular calendar logic.

While modern open-weight models have achieved near-perfect accuracy on grade-school arithmetic in English, translating the identical mathematical problem into Ge'ez numerals or Amharic calendar constraints exposes two failure modes:

1. 1.Sub-word Fragmentation: The Byte-Pair Encoding (BPE) tokenizer fragments individual Ge'ez numerals into disjoint byte chunks, obscuring the numerical values from the model's internal representations.
2. 2.Cultural & Ontological Bias: Models default to Western Gregorian assumptions (e.g., 31-day months, omitting Pagumē), producing confident hallucinations or decaying into repetitive degenerate tokens.

2. Systematic Evaluation & Empirical Findings

I systematically stress-tested Qwen/Qwen2.5-3B-Instruct (a leading 3B open-weight model with claimed state-of-the-art multilingual and mathematical capabilities) using a free T4 GPU environment in Google Colab. The evaluation used deterministic greedy decoding (do_sample=False) over paired tests across three categories.

A. Empirical Proof of Tokenizer Fragmentation

Direct inspection of the tokenizer vocabulary revealed severe byte-level fragmentation for Ethiopic glyphs:

While a Latin digit like 25 is encoded as a single token, the equivalent Ge'ez numeral ፳፭ is split into 4 disjoint byte chunks. The model never observes the symbol as a unified semantic or mathematical entity.

B. Evaluation Results on Arithmetic & Temporal Tasks

C. English Control Baseline

When the identical mathematical operations were evaluated in English using Arabic numerals (e.g., 5 + 7, 20 * 3, 100 - 25), the model scored 100% accuracy. This contrast demonstrates that the failure is not caused by a deficit in mathematical reasoning capacity, but rather by severe tokenizer representation blind spots and cultural pretraining biases.

3. Proposed Path Forward

To resolve this capability gap without requiring massive, computationally prohibitive foundation model pretraining from scratch, I propose a three-stage intervention:

1. Dedicated Script Tokenizer Adaptation

• —Vocabulary Expansion: Explicitly allocate dedicated tokens in the tokenizer vocabulary for the base Ge'ez numerals (1–10, 20, 30, ... 100, 10,000) and frequent Amharic morphological roots rather than falling back to multi-byte UTF-8 sequences.
• —Embedding Warm-Start: Initialize the new input and output embeddings as the mean of their constituent byte-level representations to prevent representation shock and preserve pre-existing semantic associations during downstream tuning.

2. Synthetic Isomorphic Reasoning Alignment (Data Curation)

• —Parallel Math Translation: Programmatically generate parallel synthetic reasoning pairs by mapping existing GSM8K and synthetic arithmetic datasets into Ge'ez numerals and Amharic word problems.
• —Chain-of-Thought (CoT) Deconstruction: Fine-tune models with explicit symbolic deconstruction paths: $$\text{Ge'ez Numeral (፳፭)} \longrightarrow \text{Decomposition } (20 + 5) \longrightarrow \text{Compute in Base-10} \longrightarrow \text{Re-encode (፸፭)}$$ This enables the model's existing reasoning heads to operate over familiar integer representations before generating the final script tokens.

3. Calendar-Constrained Multi-Task Fine-Tuning

• —Non-Gregorian Temporal QA: Curate an instruction-tuning corpus incorporating modular date arithmetic that explicitly enforces uniform 30-day boundaries and Pagumē leap transitions.
• —Contextual Switching: Train the model to condition its calendar arithmetic on explicit contextual framing (e.g., distinguishing between Ethiopian fiscal/civic dates and Gregorian dates), mitigating default cultural hallucinations.

Repository Structure

├── README.md                              # Research context, evaluation results, and proposed solutions
├── benchmark_ethiopian_reasoning.json     # Curated benchmark dataset of paired prompts
├── eval_results.json                      # Empirical model outputs from Qwen2.5-3B-Instruct
└── fatima_blindspot_ethiopian_reasoning.ipynb # Reproducible execution notebook (Google Colab / T4 GPU)