Datasetpublic

saidutta69/odia-eval-benchmark

Odia Eval Benchmark 125,243 evaluation samples across 33 datasets. Zero gating. Zero waiting. Just download and eval. Why This Is The #1 Odia Evaluation Benchmark Before this dataset, evaluating Odia language models meant hunting down individual repos, figuring out each one's format, dealing with broken loaders, and keeping track of what you've already tested. This is the first and only unified Odia eval benchmark. Factor Every Other Option This… See the full description on the dataset page: https://huggingface.co/datasets/saidutta69/odia-eval-benchmark.

sourceHugging Facemitupdated 12d agoView on Hugging Face

0likes36downloads

Dataset Card

Odia Eval Benchmark

<br>

125,243 evaluation samples across 33 datasets. Zero gating. Zero waiting. Just download and eval.

Why This Is The #1 Odia Evaluation Benchmark

Before this dataset, evaluating Odia language models meant hunting down individual repos, figuring out each one's format, dealing with broken loaders, and keeping track of what you've already tested. This is the first and only unified Odia eval benchmark.

Factor	Every Other Option	This Dataset
Total samples	Scattered across ~30 standalone repos	124,249 in one place
Task diversity	Each dataset covers 1 task	9 task types (MCQ, QA, math, translation, classification, generation, NER, paraphrasing, summarization)
Load time	`pip install` per repo + format-specific parsing	One `load_dataset()` call
Schema	30 different column names and structures	Unified schema with 15 fields
Eval harness	None -- write per-dataset scripts	Bundled (`scripts/eval_harness.py`)
Quality mix	No comparison possible	3 tiers: professional translation (43K), native Odia (43K), NLLB (15K)
Install	30+ `pip install datasets[...]`	`pip install datasets`
Access	Some gated, some broken loaders	All public, all working
NER coverage	153 samples (IndicGLUE only)	1,147 samples (IndicGLUE + Naamapadam)

No other Odia eval benchmark has all three of: unified schema, bundled eval harness, and multi-tiers of quality with professional translations.

What's In This Beast

Metric	Value
Total Samples	125,243
Unique Datasets	33
Evaluation Tasks	9
Languages	Odia (ory) + English prompts where available
License	MIT (commercially usable, no restrictions)
Access Required	None
File Format	Parquet (single split, 80MB)

By Task

Task	Samples	What It Measures
Multiple Choice	63,266	Knowledge & reasoning across 100+ subjects (MILU, MMLU, ARC, TriviaQA, HellaSwag, Winogrande, TruthfulQA-MC, BHRAM-IL, IndicCOPA, IndicGLUE-WSTP)
Question Answering	29,392	Reading comprehension, factual recall, BoolQ, IndicQA, IndicSQuAD, IndicQuest
Generation	23,738	Summarization, headline gen, question gen
Math	2,638	Arithmetic chain-of-thought reasoning
Classification	4,050	News categorization, paraphrase detection
Translation	1,012	Odia-English translation quality
NER	1,147	Token-level named entity recognition

By Quality Tier

Tier	Samples	Provider
Professional Translation	47,951	Sarvam AI (MMLU, GSM8K, ARC, TriviaQA, BoolQ, IndiVibe)
Native Odia	60,403	AI4Bharat, L3Cube, Sambhashana, MTEB, OpenLanguageData
NLLB Translated	15,895	TripathySagar (ARC, GSM8K, HellaSwag, Winogrande, TruthfulQA)

Full Dataset List

Dataset	Samples	Task	Quality
`bhram_il`	10,265	multiple_choice	Native Odia
`flores_plus`	1,012	translation	Native Odia
`indic_copa`	500	multiple_choice	Native Odia
`indic_glue_csqa`	1,975	qa	Native Odia
`indic_glue_ner`	153	ner	Native Odia
`indic_glue_wstp`	502	multiple_choice	Native Odia
`indic_headline_gen`	7,137	generation	Native Odia
`indic_qa`	1,680	qa	Native Odia
`indic_quest_odia`	200	qa	Native Odia
`indic_question_generation`	10,553	generation	Native Odia
`indic_sentence_summarization`	6,048	generation	Native Odia
`indic_squad_odia`	11,803	qa	Native Odia
`indic_xparaphrase`	2,002	classification	Native Odia
`milu`	4,525	multiple_choice	Native Odia
`naamapadam_odia_ner`	994	ner	Native Odia
`odia_arc`	1,172	multiple_choice	NLLB Translated
`odia_gsm8k`	1,319	math	NLLB Translated
`odia_hellaswag`	10,003	multiple_choice	NLLB Translated
`odia_news_classification`	2,048	classification	Native Odia
`odia_truthfulqa`	817	qa	NLLB Translated
`odia_truthfulqa_mc`	817	multiple_choice	NLLB Translated
`odia_winogrande`	1,767	multiple_choice	NLLB Translated
`sarvam_arc_challenge`	1,150	multiple_choice	Professional Translation
`sarvam_arc_challenge_val`	294	multiple_choice	Professional Translation
`sarvam_boolq`	9,427	qa	Professional Translation
`sarvam_boolq_val`	3,270	qa	Professional Translation
`sarvam_gsm8k_indic`	1,319	math	Professional Translation
`sarvam_indivibe_chat`	100	qa	Professional Translation
`sarvam_indivibe_code`	40	qa	Professional Translation
`sarvam_indivibe_math`	40	qa	Professional Translation
`sarvam_indivibe_stem`	40	qa	Professional Translation
`sarvam_mmlu_indic`	14,042	multiple_choice	Professional Translation
`sarvam_mmlu_indic_val`	285	multiple_choice	Professional Translation
`sarvam_triviaqa`	17,944	multiple_choice	Professional Translation

Quick Start

Load the data

python

from datasets import load_dataset

ds = load_dataset('saidutta69/odia-eval-benchmark', split='odia')
print(f'{len(ds):,} samples loaded')

# Filter to a specific dataset
mmlu = ds.filter(lambda x: x['dataset_name'] == 'sarvam_mmlu_indic')

# Filter by task type
mcq = ds.filter(lambda x: x['task'] == 'multiple_choice')
print(f'MCQ: {{len(mcq):,}} samples')

Evaluate a model

Use the bundled eval harness:

bash

# Random baseline on all MCQ tasks
python scripts/eval_harness.py --model random --task-type multiple_choice

# Evaluate a HuggingFace model on MMLU Odia
python scripts/eval_harness.py --model your-org/model --tasks sarvam_mmlu_indic --device mps

# Test all tasks
python scripts/eval_harness.py --model random --task-type all --verbose

Programmatic eval

python

from scripts.eval_harness import load_data, default_prompt, score_mc, MockModel

ds = load_data(dataset_names=['sarvam_mmlu_indic'])
model = MockModel('your-org/model', device='mps')

for sample in ds:
    prompt = default_prompt(sample)
    pred = model.predict(prompt)
    score = score_mc(sample, pred)

Dataset Fields

Field	Type	Description
`id`	string	Unique sample identifier
`task`	string	Task type: `multiple_choice`, `qa`, `math`, `translation`, `classification`, `generation`, `ner`
`language`	string	Language code (`ory`)
`prompt`	string	Original English prompt (when available)
`prompt_odia`	string	Odia prompt/question/context
`reference`	string	English reference/answer (when available)
`reference_odia`	string	Odia reference/answer
`choices`	list[string]	English answer choices (MCQ tasks)
`choices_odia`	list[string]	Odia answer choices (MCQ tasks)
`answer`	string	Correct answer
`domain`	string	Domain or category label
`source`	string	Original dataset name
`subset`	string	Subject or subcategory
`split`	string	Dataset split
`dataset_name`	string	Internal name for filtering

Attribution

Source	Datasets
Sarvam AI	MMLU-Indic, GSM8K-Indic, ARC-Challenge-Indic, TriviaQA-Indic-MCQ, BoolQ-Indic, IndiVibe
AI4Bharat	MILU, IndicQA, IndicCOPA, IndicHeadlineGeneration, IndicSentenceSummarization, IndicQuestionGeneration, IndicXParaphrase, IndicGLUE (CSQA, WSTP, WikiNER), Naamapadam
L3Cube	IndicSQuAD Odia, IndicQuest Odia
TripathySagar	Odia-translated ARC, GSM8K, HellaSwag, Winogrande, TruthfulQA
Sambhashana	BHRAM-IL
OpenLanguageData	FLORES+
MTEB	OdiaNewsClassification

Built because waiting for access requests is not a strategy.