ai4bharat/IndicIFEval
IndicIFEval Paper | GitHub Instruction-following benchmarks remain predominantly English-centric, leaving a critical evaluation gap for the hundreds of millions of Indic language speakers. We introduce IndicIFEval, a benchmark evaluating constrained generation of LLMs across 14 Indic languages using automatically verifiable, rule-based instructions. It combines two complementary tracks: IndicIFEval-Trans, translated prompts from IFEval carefully localized for Indic… See the full description on the dataset page: https://huggingface.co/datasets/ai4bharat/IndicIFEval.
IndicIFEval
Instruction-following benchmarks remain predominantly English-centric, leaving a critical evaluation gap for the hundreds of millions of Indic language speakers. We introduce IndicIFEval, a benchmark evaluating constrained generation of LLMs across 14 Indic languages using automatically verifiable, rule-based instructions. It combines two complementary tracks: IndicIFEval-Trans, translated prompts from IFEval carefully localized for Indic contexts, and IndicIFEval-Ground, synthetically generated instructions grounded in native Indic content.
Overview
14 Indic Languages: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Sanskrit, Tamil, Telugu, Urdu
We implement the same format of the original IFEval to allow a streamlined and compatible evaluation framework across all the languages. The table below details the column names, their respective data types, and a brief description of their contents. | Column Name | Data Type | Description | | :--- | :--- | :--- | | key | int64 | Identifier for each evaluation instance, unique within a language split. For indicifeval-trans, the same key denotes the same source item across all 15 language splits (use (config_name, split, key) as the globally unique row identity). | | prompt | str | The natural language instruction presented to the model. | | instructionidlist | List of str | Identifiers specifying which format or constraint checks apply to the prompt. | | kwargs | List of dict | Parameter values associated with each constraint specified in the instruction list. Note: the field section_spliter intentionally keeps that legacy spelling (one "t") to match the checker code in the evaluation harness — do not "fix" it in downstream code without also updating the checker. | | tags | List of str | Annotation labels for the row. For indicifeval-trans, every row carries exactly one of correct/incorrect (human translation-quality verdict) plus optional descriptive tags (e.g. parallel, gold prompt, silver translation, topic tags). The en (source) split's tags only mark the parallel cross-lingually-aligned subset, so an empty list there is expected, not missing data. See "Filtering by translation quality" below. indicifeval-ground does not use the correct/incorrect scheme. | | resp_lang | str | Expected response language for the prompt (e.g. src for the prompt's own language). |
IndicIFEval-Trans
Translated and localized prompts from the English IFEval benchmark, carefully filtered and manually verified by native speakers for cultural suitability and translation quality
IndicIFEval-Ground
Synthetically generated instructions grounded in native Indic topics and content, manually verified by native speakers. Unlike translated prompts, these reflect naturalistic constraints with more real-world contexts.
Note: two indicifeval-ground checkers evaluate against a fixed convention rather than a per-row kwargs value: the paragraph/first-word checker is always evaluated as a single-paragraph, first-word-only constraint, and exact-count constraints (keywords:frequency, length_constraints:number_sentences) are checked for equality rather than "at least"/"less than" -- see the GitHub repository for details, and use the released evaluation harness configs rather than re-implementing scoring from kwargs alone.
Filtering indicifeval-trans by translation quality
Not every translated prompt was judged correct by human review. Each row's tags list includes exactly one of correct/incorrect, and optionally parallel for the subset aligned 1:1 across all 14 languages. To reproduce paper-reported numbers, filter to rows tagged correct (add parallel if you need the cross-lingually-aligned subset). Use gold prompt and silver prompt for indicifeval-ground.
Citation
@article{jayakumar2026indicifeval,
title={IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languages},
author={Thanmay Jayakumar and Mohammed Safi Ur Rahman Khan and Raj Dabre and Ratish Puduppully and Anoop Kunchukuttan},
year={2026},
eprint={2602.22125},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2602.22125},
}