Rajveer-code/IndiaFinBench

IndiaFinBench

An Evaluation Benchmark for Large Language Model Performance on Indian Financial Regulatory Text

Rajveer Singh Pall · Gyan Ganga Institute of Technology and Sciences, Jabalpur, India

   

Overview

IndiaFinBench is, to our knowledge, the first publicly available evaluation benchmark for assessing large language model performance on Indian financial regulatory text. Existing financial NLP benchmarks — FinQA, ConvFinQA, FinanceBench, FLUE — draw exclusively from Western corpora (SEC filings, US earnings reports, English-language financial news), leaving regulatory reasoning outside the Western context unmeasured.

IndiaFinBench fills this gap with 406 expert-annotated question-answer pairs drawn from 192 primary source documents from the Securities and Exchange Board of India (SEBI) and the Reserve Bank of India (RBI), spanning 1992 to 2026. The benchmark is designed to make the specific reasoning challenges of Indian regulatory text directly measurable:

• —Dense numerical thresholds embedded in regulatory prose (capital adequacy ratios, margin requirements, dividend payout limits)
• —Amendment chains where later circulars supersede earlier ones, requiring temporal reasoning to untangle
• —Jurisdiction-specific terminology (LODR, PMLA, SFB, AIF, FEMA) that models trained predominantly on Western corpora may not reliably interpret

Overall accuracy across twelve evaluated models ranges from 70.4% to 89.7%. All models outperform the non-specialist human baseline of 69.0% (n = 100; 95% Wilson CI: [59.4%, 77.2%]). Crucially, performance is not predicted by model size or general-domain capability rank: a 17B model statistically matches a 70B model; a 120B model offers no measurable gain over 20B; and a reasoning-specialist architecture ranks second-to-last.

Task Types

Leaderboard

Twelve models evaluated under identical zero-shot, context-only conditions on the full 406-item benchmark. Scores are accuracy (%). 95% Wilson score confidence intervals are shown for overall accuracy. Primary scores use the conservative automated pipeline; see Scoring Pipeline for corrected figures.

† Gemini 2.5 Pro: The low NUM (48.9%) and TMP (64.1%) scores are a scoring artifact of its verbose output style under string-matching evaluation — not a true capability deficit. A secondary LLM-as-judge validation raises its corrected overall accuracy to 84.5%, placing it between Tier 1 and Tier 2. See Appendix D of the paper.

CON majority-class note: The contradiction detection label distribution is 85.5% "No", making the trivial majority-class baseline 85.5% — not 50%. Gemma 4 E4B (72.6%) and Mistral-7B (80.6%) fall below this baseline. See Appendix C of the paper for per-model balanced accuracy.

Primary scores are conservative lower bounds. Judge-corrected CSVs are in evaluation/results_judged/ in the repository. Tier rankings are unchanged under corrected scoring.

Performance Tiers

Paired bootstrap significance testing (10,000 resamples) across all 66 model pairs identifies three statistically distinct performance tiers. All tier boundaries remain significant after Bonferroni correction (α = 0.05 / 66 ≈ 0.00076).

Key Findings

Efficiency paradox. Llama 4 Scout 17B (Tier 1, 83.3%) is statistically indistinguishable from LLaMA-3.3-70B (83.7%, p = 0.790) — matching a 70B model with one-quarter the parameters. Instruction-tuning alignment, not raw parameter count, determines performance in the 17B–70B range on this benchmark.

Scaling provides no benefit on this domain. GPT-OSS 120B (77.1%) and GPT-OSS 20B (76.8%) differ by just 0.3 pp (p = 0.910). Model capacity is not the binding constraint.

The reasoning-specialist paradox. DeepSeek R1 70B ranks 11th of 12 despite its chain-of-thought architecture. Its error profile reveals why: 49% of its failures are Temporal Reasoning Failures — the highest of any model. Explicit reasoning chains do not reliably assist in tracking regulatory amendment timelines across multiple dated documents.

Task difficulty hierarchy. Numerical reasoning is the most discriminative task (35.9 pp spread). Temporal reasoning failure is the dominant error mode for top-tier models; smaller models fail primarily at the domain-knowledge level.

All models exceed the human baseline. The non-specialist human baseline (69.0%, n = 100) is surpassed by all twelve models. All Tier 1 models are significantly above the human upper confidence bound of 77.2% (p < 0.01).

Few-Shot Results (3-shot, Top 4 Models)

Zero-shot evaluation is a conservative lower bound. 3-shot prompting with fixed in-context examples improves numerical reasoning by 2.1–16.3 pp across all tested models. The overall tier ordering is preserved, confirming that the observed tier structure reflects genuine capability differences rather than prompting strategy sensitivity.

Full 3-shot prediction files are available in evaluation/results_fewshot/ in the repository.

Models Evaluated

Local models (LLaMA-3-8B, Mistral-7B, Gemma 4 E4B) were run on an Intel i7-13650HX + NVIDIA RTX 4060 (8 GB VRAM) workstation via Ollama. All models evaluated at temperature = 0.0 with no fine-tuning or prompt adaptation.

‡ GPT-OSS note: Exact checkpoint identifiers (model strings) are documented in the evaluation code at the GitHub repository.

Dataset Details

Source Documents

Difficulty Distribution

Splits

Data Fields

Annotation and Validation

All 406 QA pairs were authored by the primary annotator, who has prior experience with Indian financial regulatory documents. Every item was individually reviewed to ensure: (1) the answer is unambiguously derivable from the provided context; (2) the question has exactly one correct answer; and (3) the context is sufficient without external knowledge.

Model-Based Secondary Validation

A secondary validation pass over 150 items using LLaMA-3.3-70B-Versatile as a context-only quality-checker confirmed item tractability (90.7% overall agreement). This follows benchmark construction practice established in FinanceBench (Islam et al., 2023) and CUAD (Hendrycks et al., 2021).

κ is reported only for the binary CON task; dashes indicate κ is not defined for extractive tasks under string-matching agreement.

Human Inter-Annotator Agreement

A second human annotator independently answered 120 randomly selected items across all four task types. Agreement was computed using the same four-stage scoring procedure applied to model predictions.

κ = 0.611 corresponds to substantial agreement (Landis & Koch, 1977), comparable to human agreement rates for binary contradiction detection in legal NLP.

*\ Numerical reasoning (43.8%):** This figure reflects formatting convention differences between annotators (intermediate steps vs. final value only; currency symbol style; comma placement) — not substantive disagreement about the computed answer. Post-hoc review confirmed that the underlying computed values were equivalent in every discordant numerical item.

Evaluation Protocol

Models are evaluated under zero-shot, context-only conditions. The system prompt establishes the context-only constraint. Task-specific user prompts provide formatting instructions appropriate to each task type. All models were evaluated at temperature = 0.0.

The scoring pipeline applies four stages in sequence, stopping at the first match:

1. 1.Exact match — after case-normalisation and punctuation stripping
2. 2.Fuzzy token match — RapidFuzz token_set_ratio ≥ 0.72 (threshold calibrated by manual inspection of 20 borderline cases; validated at adjacent thresholds 0.65 and 0.80)
3. 3.Numerical extraction match — correct when extracted number sets from reference and prediction agree (handling currency symbols, comma separators, and units)
4. 4.Yes/No match — for CON items, leading-word comparison only

Scoring Pipeline Validation (LLM-as-Judge)

The automated pipeline systematically penalises semantically correct predictions that differ from the reference in format or verbosity. To quantify this false-negative rate (FNR), Gemini 2.5 Flash was applied as a semantic judge over all 874 items marked incorrect on NUM, REG, and TMP tasks across the twelve models. CON items use exact Yes/No matching and were not judged.

Judge reliability was confirmed on a stratified 28-item manual audit: 89.3% accuracy. The three judge errors all involved multi-step numerical problems where the model showed correct intermediate reasoning but an incorrect final value.

Three root causes account for the majority of false negatives: (1) numeric format differences ('50%' vs. 'fifty per cent'); (2) verbose answers that prefix the correct response with calculation steps; (3) abbreviated-but-correct answers (e.g., ref: 'Six months after completion of the open offer'; pred: 'Six months').

Primary leaderboard scores are conservative lower bounds on true model accuracy. Judge-corrected scores are released at evaluation/results_judged/ in the repository. Model tier rankings are unchanged under corrected scoring.

Error Taxonomy

Model failures are classified into four interpretable categories:

Key pattern: Temporal Reasoning Failure dominates for top-tier models (Gemini 2.5 Flash: 40%; LLaMA-3.3-70B: 41%; DeepSeek R1 70B: 49%). Domain Knowledge Failure is the primary failure mode for smaller models (Gemma 4 E4B: 43%). Context Grounding Failure is rare across all models (1–3%).

Limitations

• —The automated scoring pipeline penalises semantically correct responses with format differences; leaderboard figures are conservative lower bounds — judge-corrected scores are available in the repository
• —All primary evaluation is zero-shot; 3-shot results for four top models (Appendix E of the paper) confirm this is a lower bound, particularly on NUM and TMP
• —The context-injection setup provides the relevant passage directly — this tests reading comprehension over a provided excerpt, not full-document retrieval or information search
• —Primary annotation was conducted by a single annotator; multi-annotator IAA covers 120 of 406 items (29.6% coverage, meeting standard reporting thresholds)
• —All questions and gold answers are in English; Hindi-language versions of the same circulars are not included
• —The benchmark does not currently cover Hindi–English code-switched regulatory text
• —Coverage is limited to SEBI and RBI; extension to IRDAI, PFRDA, and commodity-segment regulation is planned
• —The benchmark evaluates short extractive responses, not longer-form generation (regulatory summaries, compliance gap analysis, or legally coherent reasoning chains)
• —The dataset is a snapshot of documents as of early 2026; regulatory frameworks evolve continuously and the dataset will require periodic refresh

Citation

@article{pall2026indiafinbench,
  title        = {IndiaFinBench: An Evaluation Benchmark for Large Language Model
                  Performance on Indian Financial Regulatory Text},
  author       = {Pall, Rajveer Singh},
  journal      = {arXiv preprint arXiv:2604.19298},
  year         = {2026},
  url          = {https://arxiv.org/abs/2604.19298}
}

License

Released under CC BY 4.0. All source documents are publicly available from sebi.gov.in and rbi.org.in and carry no copyright restrictions on research use. No personally identifiable information is present in any source document or derived annotation.

Contact

Rajveer Singh Pall rajveer.singhpall.cb23@ggits.net Gyan Ganga Institute of Technology and Sciences, Jabalpur, India