kartikagg98/HINMIX_hi-en

Dataset Card for Hindi English Codemix Dataset - HINMIX

HINMIX is a massive parallel codemixed dataset for Hindi-English code switching.

See the 📚 paper on arxiv to dive deep into this synthetic codemix data generation pipeline. Dataset contains 4.2M fully parallel sentences in 6 Hindi-English forms.

Further, we release gold standard codemix dev and test set manually translated by proficient bilingual annotators.

• —Dev Set consists of 280 examples
• —Test set consists of 2507 examples

To load the dataset:

!pip install datasets
from datasets import load_dataset
hinmix_ds = load_dataset("kartikagg98/HINMIX_hi-en","lcsalign-hicmrom") #choose one from lcsalign-en, lcsalign-hicm, lcsalign-hi, lcsalign-hicmrom, lcsalign-noisyhicmrom, lcsalign-hicmdvg
print ([hinmix_ds[i][10]['text'] for i in ['train','valid','test']])

Output:

>>> ['events hi samay men kahin south malabar men ghati hai.',
 'beherhaal, pulis ne body ko sector-16 ke hospital ki mortuary men rakhva diya hai.',
 'yah hamare country ke liye reality men mandatory thing hai.']

Dataset Details

Dataset Description

We construct a large synthetic Hinglish-English dataset by leveraging a bilingual Hindi-English corpus. Split: Train, test, valid Subsets:

• —Hi - Hindi in devanagiri script (Example: अमेरिकी लोग अब पहले जितनी गैस नहीं खरीदते।)
• —Hicm - Hindi sentences with codemix words substituted in English (Example: American people अब पहले जितनी gas नहीं खरीदते।)
• —Hicmrom - Hicm with romanized hindi words (Example: American people ab pahle jitni gas nahin kharidte.)
• —Hicmdvg - Hicm with transliterated english words to devangiri (Example: अमेरिकन पेओपल अब पहले जितनी गैस नहीं खरीदते।)
• —NoisyHicmrom - synthetic noise added to Hicmrom sentences to improve model robustness (Example: Aerican people ab phle jtni gas nain khridte.)

Dataset Sources [optional]

• —Repository: https://github.com/Kartikaggarwal98/RobustCodemixMT
• —Paper: https://arxiv.org/abs/2403.16771

Uses

Dataset can be used individually to train machine translation models for codemix hindi translation in any direction. Dataset can be appended with other languages from similar language family to transfer codemixing capabilities in a zero shot manner. Zero-shot translation on bangla-english showed great performance without even developing bangla codemix corpus. An indic-multilingual model with this data as a subset can improve codemixing by a significant margin.

Source Data

IITB Parallel corpus is chosen as the base dataset to translate into codemix forms. The corpus contains widely diverse content from news articles, judicial domain, indian government websites, wikipedia, book translations, etc.

Data Collection and Processing

1. 1.Given a source- target sentence pair S || T , we generate the synthetic code-mixed data by substituting words in the matrix language sentence with the corresponding words from the embedded language sentence. Here, hindi is the matrix language which forms the syntactic and morphological structure of CM sentence. English becomes the embedded language from which we borrow words.
2. 2.Create inclusion list of nouns, adjectives and quantifiers which are candidates for substitution.
3. 3.POS-tag the corpus using any tagger. We used LTRC for hindi tagging.
4. 4.Use fast-align for learning alignment model b/w parallel corpora (Hi-En). Once words are aligned, next task is switch words from english sentences to hindi sentence based on inclusion list.
5. 5.Use heuristics to replace n-gram words and create multiple codemix mappings of the same hindi sentence.
6. 6.Filter sentences using deterministic and perplexity metrics from a multilingual model like XLM.
7. 7.Add synthetic noise like omission, switch, typo, random replacement to consider the noisy nature of codemix text.

Recommendations

It's important to recognize that this work, conducted three years ago, utilized the state-of-the-art tools available at the time for each step of the pipeline. Consequently, the quality was inherently tied to the performance of these tools. Given the advancements in large language models (LLMs) today, there is potential to enhance the dataset. Implementing rigorous filtering processes, such as deduplication of similar sentences and removal of ungrammatical sentences, could significantly improve the training of high-quality models.

Citation Information

@misc{kartik2024synthetic,
      title={Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation}, 
      author={Kartik and Sanjana Soni and Anoop Kunchukuttan and Tanmoy Chakraborty and Md Shad Akhtar},
      year={2024},
      eprint={2403.16771},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Dataset Card Contact

kartikaggarwal98@gmail.com