wong132/bengali-hindi-number-blindspot

Blind Spots of Frontier Models: Bengali & Hindi Number Word-to-Digit Conversion

Summary

This dataset documents a critical blind spot in small open-source language models: failure to correctly convert Bengali and Hindi number words into their digit equivalents. Bengali and Hindi share the South Asian number system (hazar/হাজার, lakh/লাখ, crore/কোটি), and all three tested models consistently fail at this fundamental conversion step.

IMPORTANT: Arithmetic calculation errors are understandable — even large models sometimes get math wrong. However, the failures documented here are not arithmetic failures. These models fail at a much more basic level: they cannot correctly map number words to digits. For example, "পঁচাত্তর" (75) gets mapped to 55, "সত্তासী" (87) gets mapped to 10, "उनचास" (49) becomes 48. A model should at minimum get the number system correct (understanding hazar = 1,000, lakh = 1,00,000, crore = 1,00,00,000) and the digit conversion accurate, even if it then makes errors in the subsequent arithmetic. The word-to-digit step is a lookup, not a calculation — getting it wrong indicates a gap in the model's linguistic knowledge of these languages.

Models Tested

All models were loaded on Google Colab (free tier GPU).

How Models Were Loaded

Cohere tiny-aya (fire & global)

from transformers import AutoTokenizer, AutoModelForCausalLM
from google.colab import userdata

model_id = "CohereLabs/tiny-aya-fire"  # or "CohereLabs/tiny-aya-global"
hf_token = userdata.get('HF_key')
tokenizer = AutoTokenizer.from_pretrained(model_id, token=hf_token)
model = AutoModelForCausalLM.from_pretrained(model_id, token=hf_token, device_map="auto")

messages = [
    {"role": "system", "content": "You are a math expert. Solve the given math problem step by step."},
    {"role": "user", "content": prompt}
]
encoded_inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

gen_tokens = model.generate(
    **encoded_inputs,
    max_new_tokens=4096,
    do_sample=False
)

gen_text = tokenizer.decode(gen_tokens[0])
response = gen_text.split("<|CHATBOT_TOKEN|><|START_RESPONSE|>")[-1].strip()
response = response.replace('<|END_RESPONSE|>', '').replace('<EOS_TOKEN>', '')

Qwen3.5-4B

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Qwen/Qwen3.5-4B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = [
    {"role": "system", "content": "You are a math expert. Solve the given math problem step by step."},
    {"role": "user", "content": prompt}
]
encoded_inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    enable_thinking=False,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

gen_tokens = model.generate(
    **encoded_inputs,
    max_new_tokens=4096,
    do_sample=False
)

gen_text = tokenizer.decode(gen_tokens[0])
response = gen_text.split("</think>")[-1].strip()
response = response.replace('<|im_end|>', '').replace('<|endoftext|>', '')

Dataset Structure

Three subsets, one per model:

• —qwen3.5-4b.csv — Qwen/Qwen3.5-4B results
• —tiny-aya-fire.csv — CohereLabs/tiny-aya-fire results
• —tiny-aya-global.csv — CohereLabs/tiny-aya-global results

Each file has 10 rows (5 Bengali, 5 Hindi) with these columns:

Data Point Diversity

The 10 prompts were designed to maximize diversity across multiple axes while probing the same systematic blind spot:

Diversity summary:

• —2 languages: Bengali and Hindi
• —4 arithmetic operations: addition, subtraction, multiplication, division (plus one multi-step)
• —3 number scales: hazar (thousands), lakh (hundred-thousands), crore (ten-millions)
• —Varied contexts: population, library, pricing, pure arithmetic
• —11+ distinct failing number words across both languages — no two prompts test the exact same word

Error Taxonomy

1. Direct Mistranslation of Number Words

The most common failure. The model maps a number word to the wrong digit.

2. Number System Misunderstanding (Lakh/Crore Scale)

Models confuse the South Asian number system units.

• —tiny-aya-global translates "আঠারো কোটি পঁচাত্তর লাখ" (18 crore 75 lakh = 187.5 million) as 18.5 million
• —tiny-aya-global translates "সাত কোটি" (7 crore = 70 million) as 200,000

3. Catastrophic Failures

Some models break down entirely:

• —tiny-aya-fire on "তিন হাজার ছয়শ ঊনসত্তর": repeats the letters of "ঊনসত্তর" in an infinite loop
• —tiny-aya-global on "সত্তাশী লাখ": breaks the Hindi prompt into string fragments and concatenates them
• —tiny-aya-fire on "সত্তাশী লাখ": misreads prompt as compound interest problem

4. Hallucinated Operations

Models sometimes invent operations not present in the prompt:

• —tiny-aya-global converts "পঁচাত্তর হাজার তেষট্টি" into multiplication (54 × 30) instead of the number 75,063
• —tiny-aya-fire on the same prompt: also hallucinates multiplication (354 × 30)

Proposed Fine-tuning Dataset

What kind of data would fix this?

A dataset focused on number word ↔ digit mapping for Bengali and Hindi, covering:

1. 1.Direct conversion pairs: (number_word, digit) for all numbers 0–99 in both languages, plus composite numbers using hazar/lakh/crore
2. 2.Number-embedded math problems: word problems where numbers are written in words, with step-by-step solutions that explicitly show the conversion step
3. 3.South Asian number system drills: conversions between lakh/crore notation and standard digits (e.g., "আঠারো কোটি পঁচাত্তর লাখ = ১৮,৭৫,০০,০০০")

How to assemble such a dataset?

• —Programmatic generation: Write a script that generates (number_word, digit) pairs for all numbers 1 to 99,99,99,999 (up to 99 crore) using Bengali/Hindi number word rules. These rules are deterministic and well-documented.
• —Template-based word problems: Create math problem templates (addition, subtraction, multiplication, division) and fill in programmatically generated number words. Include step-by-step solutions showing explicit word→digit conversion.
• —Existing resources: Scrape Bengali/Hindi math textbooks (NCERT for Hindi) that already contain word problems with numbers in words.

Estimated dataset size

• —Minimum viable: ~2,000–5,000 examples covering all unique number words (0–99 × 2 languages = ~200 base words) combined with hazar/lakh/crore modifiers, plus ~1,000 word problems per language.
• —Recommended: ~10,000–20,000 examples for robust generalization, with diverse arithmetic operations and number magnitudes.
• —The number word rules are fully deterministic, so generating large synthetic datasets is cheap — the bottleneck is quality of the word problem templates, not data volume.