nlpctx/tamil-english-corpus

language:

• —ta
• —en

license: apache-2.0

task_categories:

• —text-retrieval
• —question-answering
• —text-generation

pretty_name: Tamil-English Retrieval Corpus

tags:

• —retrieval
• —rag
• —multilingual
• —tamil
• —english
• —machine-translation
• —corpus
• —embeddings
• —information-retrieval

Tamil-English Retrieval Corpus

A high-quality multilingual retrieval corpus constructed from the Mozhi Tamil Corpus and machine-translated into English using IndicTrans2.

Dataset Summary

This dataset contains Tamil documents paired with English translations.

The corpus was created by filtering high-quality documents from the Mozhi Tamil Corpus and translating them using AI4Bharat's IndicTrans2 translation model.

The resulting corpus is intended to support:

• —Information Retrieval
• —Retrieval-Augmented Generation (RAG)
• —Multilingual Search
• —Embedding Evaluation
• —Query Generation
• —Cross-Lingual Retrieval
• —Indian Language NLP Research

Source Dataset

This dataset is derived from:

https://huggingface.co/datasets/mozhi-ai/tamil-corpus

Original Dataset Citation

@misc{mozhi_ai_tamil_corpus_2026,
  title={mozhi-ai Tamil Corpus},
  author={mozhi-ai contributors},
  year={2026},
  url={https://huggingface.co/datasets/mozhi-ai/tamil-corpus}
}

Original Corpus Statistics

• —Total Documents: 8,682
• —Sources:

• —IndicCorpV2 (AI4Bharat)
• —Tamil Wikipedia
• —Project Madurai

Domains

• —News
• —Encyclopedia
• —Classical Literature

Source Licenses

• —CC-0
• —CC-BY-SA-4.0
• —Public Domain

Please refer to the original dataset card for detailed licensing information.

Dataset Construction

Filtering

Documents were filtered using metadata provided by the source corpus.

Selection criteria included:

• —High quality_score
• —Minimum document length
• —Duplicate removal

Translation

English translations were generated using:

Model: ai4bharat/indictrans2-indic-en-dist-200M

Authors: AI4Bharat

Paper: IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages

IndicTrans2 Citation

@article{gala2023indictrans,
  title={IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages},
  author={Jay Gala and Pranjal A. Chitale and A. K. Raghavan and Varun Gumma and Sumanth Doddapaneni and Aswanth Kumar M and Janki Nawale and Anupama Sujatha and Ratish Puduppully and Vivek Raghavan and Pratyush Kumar and Mitesh M. Khapra and Raj Dabre and Anoop Kunchukuttan},
  journal={Transactions on Machine Learning Research},
  year={2023}
}

Dataset Fields

Intended Uses

• —Dense Retrieval
• —BM25 Evaluation
• —Embedding Model Evaluation
• —Retrieval-Augmented Generation (RAG)
• —Query Generation
• —Cross-Lingual Retrieval
• —Code-Mixed Dataset Construction
• —Indian Language NLP Research

Limitations

• —English translations are machine-generated.
• —Translation quality may vary across domains.
• —No human verification was performed.
• —Classical Tamil text may be harder to translate accurately.

Citation

If you use this dataset, please cite:

@dataset{koheda2026_tamil_english_retrieval,
  title={Tamil-English Retrieval Corpus},
  author={Suhas Koheda},
  year={2026},
  publisher={Hugging Face}
}

Acknowledgements

This dataset builds upon resources released by:

• —Mozhi AI
• —AI4Bharat
• —Hugging Face

I thank the creators and maintainers of these open-source resources.