shuvo-xyz/BanglaCEH

BanglaCEH: A Benchmark for Culturally Entangled Homograph Disambiguation in Bangla

BanglaCEH is the benchmark released with the paper "When a Name Is Not a Name: A Benchmark Dataset and Distilled Reasoning for Culturally Entangled Bangla Homographs in Low-Resource LLMs."

Many Bangla words are simultaneously a personal name and a culturally loaded common noun. মায়া (Maya) is both a common girl's name and a word for deep affectionate compassion; আরিফ (Arif) is a boy's name and, in Sufi theology, one who possesses intuitive knowledge of the divine. Choosing the right reading requires cultural and pragmatic knowledge that is scarce in the pretraining data of modern language models.

We call this phenomenon a Culturally Entangled Homograph (CEH). Unlike conventional word-sense disambiguation, a CEH cannot be resolved by frequency or distributional cues: both readings live in the same cultural space, and disambiguation hinges on recognising when a form names a person versus when it invokes the concept the name was drawn from.

• —📄 Paper: https://arxiv.org/abs/2607.17828
• —💻 Code: https://github.com/ashuvo25/BanglaCEH

Dataset Summary

Every instance is a single Bangla sentence in which one word form appears twice with two distinct readings. Given the sentence and the target word, a model must assign each occurrence one of six culturally grounded categories and justify the assignment.

Example

Word: গগন (Gagan) — Category: Name ↔ Concept

সমুদ্রসৈকতে সূর্যাস্ত দেখতে দেখতে গগন এর মনে হলো, প্রাচীন শ্লোকে 'গগন' দিয়ে ঠিক এই আকাশ; নভোমণ্ডল-কেই বোঝানো হতো।

Watching the sunset at the beach, it struck Gagan that ancient verse used 'Gagan' for exactly this — the celestial firmament.

Cultural note: Bengali naming turns words for nature, deities, and virtues into personal names; গগন is thus once a living identity and once a cultural concept.

Data Fields

Each record is a JSON object:

{
  "id": 1516,
  "word_bangla": "গগন",
  "word_roman": "Gagan",
  "category": "Intra-sentential (Name↔Concept)",
  "input": {
    "narrative_bangla": "...গগন এর মনে হলো...",
    "label_1": "Name",
    "label_2": "Concept"
  },
  "token_labels": {
    "token_1": "Name",
    "token_2": "Concept"
  },
  "cultural_entanglement_note": "Gagan carries the concept of the celestial firmament in Bengali cultural imagination while remaining a common given name."
}

The full schema additionally includes bilingual justifications, cultural-entanglement paragraphs, and per-occurrence explanations. These explanations are the supervision signal used for the knowledge-distillation experiments in the paper.

Distribution

Entanglement categories

Token-level labels

Name and Concept dominate, reflecting the prominence of name–concept entanglement in Bangla naming practice. Emotion and Spiritual form a rarer long tail that models handle least reliably.

Splits

The full benchmark of 1,516 instances is released as a single file. Experiments in the paper use a 78% train / 10% validation / 12% test split with a fixed random seed.

Loading the Dataset

from datasets import load_dataset

ds = load_dataset("shuvo-xyz/BanglaCEH")["train"]
print(ds[0])

# Reproduce the paper's split proportions
split = ds.train_test_split(test_size=0.22, seed=42)
train = split["train"]
rest = split["test"].train_test_split(test_size=0.545, seed=42)
validation, test = rest["train"], rest["test"]

Dataset Construction

1. 1.Word selection. A native Bangla speaker manually curated words such as মায়া (Maya), আশা (Asha), and গগন (Gagan) that are both common given names and culturally loaded common nouns. Expert curation was chosen over a name registry because a generic name list would not isolate forms with a genuine dual reading.
2. 2.Multi-model generation. Sentences and cultural explanations were generated by three models as complementary generators — Claude Fable 5 (690 instances), Kimi K3 (480), and Gemini 3.5 Flash (344) — to reduce the stylistic and distributional bias of any single teacher.
3. 3.Cross-model verification. Outputs were checked round-robin by a different model (Claude verified by Kimi, Kimi by Gemini, Gemini by Claude), so no model audited its own output. This stage only surfaces candidate errors.
4. 4.Human verification. Every instance was reviewed by two native Bangla-speaking annotators with graduate-level CS backgrounds and prior Bangla NLP experience, who corrected labels, explanations, and culturally inaccurate reasoning. Human review is the final authority. Inter-annotator agreement: Cohen's κ = 0.95.

Benchmark Results

Evaluation covers four regimes: zero-shot, few-shot, Cultural Chain-of-Thought (C-CoT), and QLoRA knowledge-distillation fine-tuning. The central diagnostic is Dominant-Bias: how often a model misses a Name label and defaults to the entangled cultural reading.

<!-- -->

Key findings

• —A systematic dominant-meaning bias. Under zero-shot prompting every open-source model defaults to the common-noun reading and misclassifies the personal-name occurrence, with bias from 62.5% to 100%.
• —Language-specific pretraining is not cultural grounding. The Bangla-specific TituLLM fails under every prompting regime, hallucinating on 100% of zero-shot inputs.
• —Contrastive reasoning helps — but not universally. C-CoT drives Qwen's bias from 63.46% to 2.88% with zero hallucination and no training, yet degrades Llama, whose long reasoning chains often fail to terminate in the required label format.
• —Distilling reasoning beats distilling labels. An ablation shows label-only distillation reaches 41.33% EM with 34.56% bias, while distilling the cultural explanations alongside labels reaches 85.16% EM with 0% bias.
• —A striking reversal. TituLLM, non-functional under every prompting regime, becomes the strongest system after distillation — monolingual pretraining supplies a useful substrate that absorbs limited reasoning supervision efficiently.
• —The remaining bottleneck is the long tail. The gap between Micro- and Macro-F1 concentrates almost entirely in the rarer Emotion and Spiritual labels.

Intended Uses

• —Evaluating cultural grounding in Bangla and multilingual LLMs, separately from surface lexical competence
• —Training and evaluating name-vs-concept disambiguation in Bangla NLP pipelines (NER, coreference, MT)
• —Research on reasoning distillation for low-resource languages
• —A template for building analogous benchmarks in other languages with name–concept entanglement

Limitations

• —Single-curator word selection. Words were curated manually by one native speaker rather than drawn from a registry, so coverage reflects one annotator's naming knowledge and may under-represent regional or less common names.
• —Model-generated text. Sentences and explanations were initially produced by LLMs and then human-verified. Subtle stylistic regularities of machine-generated text may remain.
• —Small annotator pool. Agreement is high (κ = 0.95), but a larger pool spanning more regional and linguistic backgrounds would improve robustness.
• —Compute-limited fine-tuning. Experiments use parameter-efficient adaptation on 1–3B models, so results may not transfer directly to larger models or full-parameter fine-tuning.

Ethical Considerations

The dataset contains common Bangla given names used as generic lexical items in synthetic sentences; it does not describe or reference real individuals. Cultural notes reflect mainstream Bengali naming conventions and should not be read as exhaustive or authoritative across all Bengali-speaking communities.

Citation

If you use this dataset, please cite:

@misc{shuvo2026namebenchmarkdatasetdistilled,
      title={When a Name Is Not a Name: A Benchmark Dataset and Distilled Reasoning for Culturally Entangled Bangla Homographs in Low-Resource LLMs}, 
      author={Md. Asaduzzaman Shuvo},
      year={2026},
      eprint={2607.17828},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.17828}, 
}

Contact

Md. Asaduzzaman Shuvo — United International University, Bangladesh iqbalmdshuvo@gmail.com