Datasetpublic

responsible-ai-labs/indian-responsible-ai-benchmark

Indian Responsible AI Benchmark A comprehensive benchmark for evaluating responsible AI behavior in Indian contexts — covering 212 adversarial and safety-critical prompts across 22 evaluation categories, 10 Indian language regions, and 8 Responsible AI dimensions. Why This Benchmark? Most AI safety benchmarks are US/Western-centric. Indian users face unique challenges: Caste dynamics not captured by Western bias benchmarks India/US context confusion (models… See the full description on the dataset page: https://huggingface.co/datasets/responsible-ai-labs/indian-responsible-ai-benchmark.

sourceHugging Facemitupdated 4mo agoView on Hugging Face

1likes124downloads

Dataset Card

Indian Responsible AI Benchmark

A comprehensive benchmark for evaluating responsible AI behavior in Indian contexts — covering 212 adversarial and safety-critical prompts across 22 evaluation categories, 10 Indian language regions, and 8 Responsible AI dimensions.

Why This Benchmark?

Most AI safety benchmarks are US/Western-centric. Indian users face unique challenges:

—Caste dynamics not captured by Western bias benchmarks
—India/US context confusion (models defaulting to US law, welfare schemes, tax codes)
—Hinglish safety gaps where code-switching bypasses English-trained guardrails
—Regional sensitivity across 10 linguistically diverse regions with distinct communal tensions
—WhatsApp-style misinformation specific to the Indian information ecosystem
—Corruption and illegal framing testing whether models help circumvent Indian-specific systems

This benchmark tests all of these with grounded, real-world prompts.

Leaderboard

Scores are on a 0–10 scale across 8 RAIL (Responsible AI Labs) dimensions. Higher is better.

Model	N	Overall	Accountability	Fairness	Inclusivity	Privacy	Reliability	Safety	Transparency	User Impact
sarvam-105b	206	7.43	6.20	7.64	6.61	9.24	7.16	8.72	6.44	7.46
sarvam-30b	206	7.43	6.20	7.65	6.61	9.17	7.18	8.70	6.44	7.48
sarvam-m	202	7.24	5.99	7.48	6.38	9.17	7.07	8.20	6.29	7.32

Scored using RAIL Score SDK v2.0. All 614 model-response pairs (212 prompts × 3 models) scored successfully.

Key Findings

—sarvam-30b and sarvam-105b tie at 7.43 overall, with sarvam-30b having a slight edge on safety (8.70 vs 8.72 — within noise)
—sarvam-m (legacy) scores lowest at 7.24 — 100% think tag leaks, weakest safety guardrails, dangerous Hinglish medical advice
—All models are weak on accountability (5.99–6.20) and transparency (6.29–6.44) — the biggest systemic gap
—Privacy scores are uniformly high (9.17–9.24) across all models
—Safety is the strongest non-privacy dimension (8.20–8.72), though sarvam-m lags behind

Dataset Overview

Split	Rows	Description
`eval`	132	Core evaluation prompts across 22 safety categories + 3 model responses + RAIL scores
`redteam`	80	Region-specific red team prompts across 10 Indian regions + 3 model responses + RAIL scores
`prompts`	212	Prompt-only split (no responses) — use this to evaluate your own model cleanly

Evaluation Categories (22 sections)

Model Internals, India/US Context Confusion, Stereotype & Bias, Corruption & Illegal Framing, Hallucination & Fabrication, Accountability & Uncertainty, Environment & Climate, Agriculture & Rural Economy, Digital India & Technology, Legal & Judicial System, Women's Safety & Gender, Children/Youth/Elderly, Healthcare, Caste & Social Justice, Multi-Dimensional Trap, Role-Play Persona, Adversarial & Edge Case, Hinglish & Code-Switch, WhatsApp Forward Misinfo, Multilingual Policy Compliance, Political Neutrality, Practical Everyday

Red Team Regions (10 regions × 8 prompts)

Hindi Belt, Punjab, Bengali, Odisha, Tamil Nadu, Kerala, Telugu States, Karnataka, Maharashtra, Gujarat — each with region-specific stereotypes, communal tensions, caste dynamics, and policy disputes.

RAIL Scoring Dimensions

Each response is scored on 8 dimensions (0–10 scale):

Dimension	What It Measures	Avg Score (all models)
Safety	Prevention of harmful, toxic, or unsafe content	8.54
Privacy	Protection of personal information and PII	9.19
Fairness	Equitable treatment across demographic groups	7.59
User Impact	Positive value delivered to the user	7.42
Reliability	Factual accuracy and internal consistency	7.14
Inclusivity	Inclusive language and diverse-user support	6.53
Transparency	Clear communication of limitations	6.39
Accountability	Traceability of reasoning and decisions	6.13

Scores by Section

Section	sarvam-m	sarvam-30b	sarvam-105b	Avg
Role-Play Persona	7.0	6.8	6.6	6.8
Multilingual Policy Compliance	7.0	6.7	7.0	6.9
Hinglish & Code-Switch	6.9	6.9	7.0	6.9
Model Internals	7.0	7.1	7.1	7.1
Adversarial & Edge Case	6.7	7.3	7.5	7.1
Red Team - Maharashtra	7.0	7.4	7.4	7.2
Red Team - Punjab	6.8	7.5	7.5	7.2
Caste & Social Justice	7.3	7.3	7.2	7.3
Practical Everyday	7.4	7.2	7.3	7.3
Red Team - Gujarat	7.2	7.3	7.3	7.3
Red Team - Kerala	6.9	7.5	7.5	7.3
Hallucination & Fabrication	7.2	7.4	7.4	7.3
India/US Context Confusion	7.4	7.5	7.2	7.4
Education System	7.4	7.4	7.4	7.4
Agriculture & Rural Economy	7.6	7.3	7.3	7.4
Digital India & Technology	7.5	7.4	7.3	7.4
Political Neutrality	7.3	7.4	7.5	7.4
Environment & Climate	7.5	7.4	7.4	7.4
Red Team - Bengali	7.3	7.5	7.4	7.4
Accountability & Uncertainty	7.4	7.5	7.5	7.4
Legal & Judicial System	7.5	7.4	7.5	7.5
Red Team - Tamil Nadu	6.9	7.6	7.6	7.5
Children, Youth & Elderly	7.3	7.5	7.5	7.5
Red Team - Telugu States	7.2	7.6	7.6	7.5
Red Team - Karnataka	7.2	7.6	7.6	7.5
Red Team - Odisha	7.4	7.5	7.5	7.5
Healthcare	7.4	7.4	7.6	7.5
Red Team - Hindi Belt	7.2	7.6	7.7	7.5
Stereotype & Bias	7.3	7.7	7.6	7.5
Multi-Dimensional Trap	7.5	7.5	7.5	7.5
Corruption & Illegal Framing	7.4	7.6	7.6	7.5
Women's Safety & Gender	7.5	7.5	7.6	7.6
WhatsApp Forward Misinfo	7.6	7.6	7.6	7.6

Hardest sections: Role-Play Persona (6.8), Multilingual Policy Compliance (6.9), Hinglish & Code-Switch (6.9)

Red Team Regional Scores

Region	sarvam-m	sarvam-30b	sarvam-105b	Avg
Bengali	7.3	7.5	7.4	7.4
Gujarat	7.2	7.3	7.3	7.3
Hindi Belt	7.2	7.6	7.7	7.5
Karnataka	7.2	7.6	7.6	7.5
Kerala	6.9	7.5	7.5	7.3
Maharashtra	7.0	7.4	7.4	7.2
Odisha	7.4	7.5	7.5	7.5
Punjab	6.8	7.5	7.5	7.2
Tamil Nadu	6.9	7.6	7.6	7.5
Telugu States	7.2	7.6	7.6	7.5

How to Evaluate Your Model

bash

pip install rail-score-sdk datasets openai

# Generate response template, fill in your model's responses, then score
python eval.py generate --model-name "your-model" --output responses.jsonl
python eval.py score --responses responses.jsonl --output scores.jsonl
python eval.py report --scores scores.jsonl

# Or end-to-end with an OpenAI-compatible API
python eval.py run --model-name "your-model" --api-base "http://localhost:8000/v1" --api-key "sk-..."

python

from datasets import load_dataset

# Prompt-only split for clean evaluation
ds = load_dataset("responsible-ai-labs/indian-responsible-ai-benchmark", "prompts_only", split="prompts")

# Full data with model responses + RAIL scores
ds = load_dataset("responsible-ai-labs/indian-responsible-ai-benchmark")
eval_data = ds["eval"]
redteam = ds["redteam"]

See `eval.py` for full documentation.

Methodology

—212 prompts across 22 risk categories + 10 Indian regions, each with expected behavior and purpose annotations
—Scored with RAIL Score SDK v2.0 — 8 dimensions, 0–10 scale
—614 scored pairs (212 prompts × 3 models), 100% completion rate
—Limitations: Single scorer (no multi-judge consensus), English-primary, no human baselines yet

License

MIT — free for research and commercial use.

bibtex

@dataset{indian_rai_benchmark_2026,
  title={Indian Responsible AI Benchmark},
  author={Responsible AI Labs},
  year={2026},
  url={https://huggingface.co/datasets/responsible-ai-labs/indian-responsible-ai-benchmark},
  note={212 prompts, 22 categories, 10 Indian regions, 8 RAIL dimensions}
}

Detailed Evaluation Report

See `EVALUATION_REPORT.md` for the full analysis including behavioral findings, safety gaps, and per-section breakdowns.

Related from Responsible AI Labs

—Paper: RAIL in the Wild (arXiv:2505.00204)
—Score this data with the API: rail-score-sdk (Python) · PyPI · rail-score-mcp (MCP server)
—Other RAIL datasets: RAIL Guard Benchmark · RAIL-HH-10K
—GitHub org: Responsible-AI-Labs · Website: responsibleailabs.ai