CoolFace
Datasetpublic

Tanushreeeeee/COMI-LINGUA

Dataset Details COMI-LINGUA (COde-MIxing and LINGuistic Insights on Natural Hinglish Usage and Annotation) is a high-quality Hindi-English code-mixed dataset, manually annotated by three annotators. It serves as a benchmark for multilingual NLP models by covering multiple foundational tasks. COMI-LINGUA provides annotations for several key NLP tasks: Language Identification (LID): Token-wise classification of Hindi, English, and other linguistic units. Initial predictions were… See the full description on the dataset page: https://huggingface.co/datasets/Tanushreeeeee/COMI-LINGUA.

sourceHugging Facecc-by-4.0updated 9mo agoView on Hugging Face
0likes78downloads
Dataset Card

Dataset Details

COMI-LINGUA (COde-MIxing and LINGuistic Insights on Natural Hinglish Usage and Annotation) is a high-quality Hindi-English code-mixed dataset, manually annotated by three annotators. It serves as a benchmark for multilingual NLP models by covering multiple foundational tasks.

COMI-LINGUA provides annotations for several key NLP tasks:

  • —Language Identification (LID): Token-wise classification of Hindi, English, and other linguistic units.<br> Initial predictions were generated using the Microsoft LID tool, which annotators then reviewed and corrected.
  Example:
  sentence: प्रधानमंत्री  नरेन्द्र  मोदी  डिजिटल  इंडिया  मिशन  को  आगे  बढ़ाने  के  लिए  पिछले  सप्ताह  Google  के  CEO  सुंदर  पिचाई  से  मुलाकात  की  थी ।
  LID tags:   hi    hi   hi    en    en    en   hi  hi   hi  hi  hi   hi    hi     en   hi  en   hi   hi   hi   hi   hi  hi ot
  • —Matrix Language Identification (MLI): Sentence-level annotation of the dominant language.
  Example:
  sentence:"किसानों को अपनी फसल बेचने में दिक्कत न हो इसके लिये Electronic National Agriculture Market यानि ई-नाम योजना तेजी से काम हो रहा है।" Matrix Language: hi
  sentence:"India’s automation and design expert pool is vast, और ज़्यादातर Global companies के इंजीिनयिंरग center भी भारत में हैं।" Matrix Language: en
  • —Part-of-Speech (POS) Tagging: Syntactic categorization for linguistic analysis.<br> Tags were pre-assigned using the CodeSwitch NLP library, which annotators then reviewed and corrected.
  Example:
  sentence:  भारत   द्वारा  बनाया  गया Unified Payments Interface  यानि   UPI    भारत   की    एक   बहुत  बड़ी   success  story  है    ।
  POS tags: PROPN  ADP VERB VERB  PROPN   PROPN    PROPN    CONJ  PROPN  PROPN ADP  DET  ADJ  ADJ    NOUN    NOUN  VERB  X
  • —Named Entity Recognition (NER): Identification of entities in Hinglish text.<br> Each token is pre-assigned a language tag using the CodeSwitch NLP library, which annotators then reviewed and corrected.
  Example:
  sentence: "मालूम हो कि पेरिस स्थित Financial Action Task Force, FATF ने जून 2018 में पाकिस्तान को ग्रे लिस्ट में रखा था।"
  NER tags: "पेरिस": GPE, "Financial Action Task Force, FATF": ORGANISATION, "2018": Date, "पाकिस्तान": GPE
  • —Machine Translation (MT): Parallel translation of sentences in Romanized Hindi and Devanagari Hindi and English languages. Initial translation predictions were generated using the Llama 3.3 LLM, which annotators then refined and corrected.
  Example:
  Hinglish Sentence: "भारत में भी green growth, climate resilient infrastructure और ग्रीन transition पर विशेष रूप से बल दिया जा रहा है।"
  English Translation: "In India too, special emphasis is being given to green growth, climate resilient infrastructure, and green transition."
  Romanized Hindi Translation: "Bharat mein bhi green growth, climate resilient infrastructure aur green transition par vishesh roop se bal diya ja raha hai."
  Devnagari Hindi Translation: "भारत में भी हरित विकास, जलवायु सहनशील आधारिक संरचना और हरित संक्रमण पर विशेष रूप से बल दिया जा रहा है।"

Dataset Description

Dataset Usage Disclaimer ‼️

<span style="color:red">This dataset is a research preview and is subject to ongoing iterative updates. As such, it provides only limited safety measures.</span>

Contact US ✉️

Lingo Research Group at IIT Gandhinagar, India </br> Mail at: lingo@iitgn.ac.in

Citation

If you use this dataset, please cite the following work:

@inproceedings{sheth-etal-2025-comi,
    title = "{COMI}-{LINGUA}: Expert Annotated Large-Scale Dataset for Multitask {NLP} in {H}indi-{E}nglish Code-Mixing",
    author = "Sheth, Rajvee  and
      Beniwal, Himanshu  and
      Singh, Mayank",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-emnlp.422/",
    pages = "7973--7992",
    ISBN = "979-8-89176-335-7",
}