yash-ingle/ILID_Indian_Language_Identification_Dataset

ILID: Native Script Language Identification for Indian Languages

**Paper** | **Code** | **Project Page**

🗣 ILID: Indian Language Identification Dataset (23 Languages) Authors: Yash Ingle, Dr. Pruthwik Mishra Institute: Sardar Vallabhbhai National Institute of Technology (SVNIT), Surat, India

📄 Dataset Description

The ILID (Indian Language Identification Dataset) benchmark contains 250,000 sentences from English and 22 official Indian languages, designed for training and evaluating language identification models. The dataset supports the task of distinguishing between Indian languages, many of which share scripts, vocabulary, and structure.

📊 Dataset Statistics

📁 Files Provided

• —shuffled_train_sentences: Training sentences (80% split – 200,000 samples)
• —shuffled_train_labels: Corresponding labels for training sentences
• —shuffled_dev_sentences: Validation (dev) sentences (10% split – 25,000 samples)
• —shuffled_dev_labels: Corresponding labels for dev sentences
• —shuffled_test_sentences: Test sentences (10% split – 25,000 samples)
• —shuffled_test_labels: Corresponding labels for test sentences

📌 Tasks

• —Language Identification (LID)
• —Multilingual Text Classification
• —Benchmarking ML & DL Models on Indian Languages

🧹 Data Collection & Cleaning

• —13 languages collected using web scraping from Wikipedia, news portals, and blogs.
• —10 languages sampled from large monolingual corpora (Bhashaverse).
• —Each sentence underwent cleaning, normalization, and language filtering via FastText.

🧠 Models & Results

Baseline models include:

• —TF-IDF + Machine Learning: SVM, Logistic Regression, Random Forest, etc.
• —FastText Classifier
• —Fine-tuned MuRIL (BERT for Indian languages)

Best ensemble models achieve F1-scores of up to 0.99 on test/dev sets.

📚 Citation

If you use this dataset, please cite:

@misc{ingle2025ilidnativescriptlanguage,
      title={ILID: Native Script Language Identification for Indian Languages}, 
      author={Yash Ingle and Pruthwik Mishra},
      year={2025},
      eprint={2507.11832},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.11832}, 
}