Reja1/jee-neet-benchmark

JEE/NEET LLM Benchmark Dataset

 

🏆 [View the live leaderboard →](https://huggingface.co/spaces/Reja1/jee-neet-benchmark-leaderboard) — interactive results across JEE Advanced, JEE Main & NEET, with open/closed-weight badges, contamination flags, and per-run cost.

A benchmark for evaluating vision-capable LLMs on Indian competitive exam questions (JEE Advanced & NEET). Each question is the original exam image; models answer via the OpenRouter API and are scored with authentic, exam-specific marking schemes — including partial credit for JEE Advanced multiple-correct questions.

Supported question types: Single Correct MCQ, Multiple Correct MCQ, Matching List MCQ, Integer, and stem-based Integer (INTEGER_2). Every image carries metadata: exam, year, subject, question type, paper, and verified correct answer(s).

Dataset Composition

Subjects are evenly balanced within each set — NEET 2024 has 50 per subject; NEET 2025/2026 have 45 Physics, 45 Chemistry, 90 Biology; every JEE Advanced set splits equally across the three subjects.

Quick Start

Load the dataset

from datasets import load_dataset
import json

dataset = load_dataset("Reja1/jee-neet-benchmark", split="test")
example = dataset[0]

image = example["image"]                                  # PIL image
correct = json.loads(example["correct_answer"])           # e.g. ["A"], ["B", "C"], ["42"]

Setup

git clone https://huggingface.co/datasets/Reja1/jee-neet-benchmark
cd jee-neet-benchmark

git lfs pull          # fetch images + metadata (stored in Git LFS)

uv sync

echo "OPENROUTER_API_KEY=your_key" > .env

Run the benchmark

Evaluate a vision-capable model on the full dataset:

uv run python src/benchmark_runner.py --model "google/gemini-3.1-pro-preview"

Run a single exam and year:

uv run python src/benchmark_runner.py --model "openai/gpt-5.5" --exam_name NEET --exam_year 2026

uv run python src/benchmark_runner.py --model "anthropic/claude-opus-4.7" --exam_name JEE_ADVANCED --exam_year 2026

Run only specific questions (comma-separated IDs):

uv run python src/benchmark_runner.py --model "openai/gpt-5.5" --question_ids "N24T3001,JA26P1M01"

Run a model 3 times for variance analysis:

uv run python src/benchmark_runner.py --model "x-ai/grok-4.3" --exam_name NEET --exam_year 2026 --num_runs 3

Override the sampling temperature from the config:

uv run python src/benchmark_runner.py --model "openai/gpt-5.5" --temperature 0.7

Pin OpenRouter routing to a specific provider — e.g. the model's official host instead of a third-party reseller. Fallbacks are disabled, so a request fails (and is retried) rather than silently routing elsewhere. Useful when resellers serve a different quantization than the official endpoint:

uv run python src/benchmark_runner.py --model "moonshotai/kimi-k2.6" --exam_name JEE_ADVANCED --exam_year 2026 --provider-only moonshotai

Resume an interrupted run (skips already-completed questions):

uv run python src/benchmark_runner.py --model "openai/gpt-5.5" --resume results/<run_dir>

Re-score an existing run after updating the answer key — no API calls:

uv run python src/benchmark_runner.py --score-only results/<run_dir>

Analyse results

Build a cross-model leaderboard from all local results:

uv run python scripts/generate_leaderboard.py

Hosted leaderboard (HuggingFace Static Space)

Live at [huggingface.co/spaces/Reja1/jee-neet-benchmark-leaderboard](https://huggingface.co/spaces/Reja1/jee-neet-benchmark-leaderboard).

Generate a self-contained, dark-themed HTML leaderboard — all results grouped by exam+year, with open-weight badges and contamination footnotes drawn from scripts/model_metadata.yaml:

uv run python scripts/generate_leaderboard.py --html space/index.html

One-time setup (requires huggingface-cli login): create a Static Space named jee-neet-benchmark-leaderboard via the HF web UI, then clone it into space/ (gitignored from this repo):

git clone https://huggingface.co/spaces/Reja1/jee-neet-benchmark-leaderboard space

Regenerate and publish after each model run:

uv run python scripts/generate_leaderboard.py --html space/index.html
cd space && git add -A && git commit -m "update leaderboard" && git push

Aggregate repeated runs of one model for variance stats:

uv run python scripts/aggregate_runs.py --pattern "x-ai_grok-4.3_NEET_2026"

Configure the model list and parameters in configs/benchmark_config.yaml; run src/benchmark_runner.py --help for the full option list. Each run writes a timestamped folder under results/ with predictions.jsonl (raw responses), summary.jsonl (per-question scores, tokens, cost, latency), and summary.md (human-readable report).

Scoring

API/parse failures and skipped questions score 0 (no penalty), since they are not a deliberate wrong choice.

NEET — Single Correct MCQ: +4 correct, −1 incorrect.

JEE Main (supported in code; no questions in the current dataset) — Single Correct MCQ: +4 / −1. Integer: +4 / 0.

JEE Advanced

Data Fields & Answer Format

Each record exposes: image, question_id, exam_name, exam_year (int), subject, question_type (MCQ_SINGLE_CORRECT, MCQ_MULTIPLE_CORRECT, MCQ_MATCHING, INTEGER, INTEGER_2), paper_id, and correct_answer — a JSON-serialized string parsed with json.loads(). MCQ answers are option identifiers (["A"], ["B", "C"]); Integer answers are numbers as strings (["42"], ["12.75"]). A few single-correct questions list multiple acceptable options; a prediction matching any one is correct.

Models return answers in <answer>...</answer> tags: <answer>A</answer>, <answer>B,D</answer> (multiple correct), <answer>42</answer> / <answer>12.75</answer> (numeric), or <answer>SKIP</answer>.

Limitations & Data Contamination

These are publicly administered exams, widely published online after each sitting, so questions may appear in models' training data — especially for older years. High scores may partly reflect memorization rather than reasoning. Treat this as an evaluation on publicly available exam questions, not a contamination-free reasoning test; compare across years (older = higher contamination risk) and cross-reference with contamination-resistant benchmarks (e.g. GPQA, Humanity's Last Exam).

Other caveats: a single prompt template per question type (results vary with phrasing); one run per model by default (non-deterministic outputs vary slightly); performance is sensitive to image quality; English only; requires vision models on OpenRouter.

Citation

@misc{rejaullah_2025_jeeneetbenchmark,
  title={JEE/NEET LLM Benchmark},
  author={Md Rejaullah},
  year={2025},
  howpublished={\url{https://huggingface.co/datasets/Reja1/jee-neet-benchmark}},
}

Contact & License

Questions or collaboration: @RejaullahmdMd on X. Released under the MIT License.