tejasnaladala/mteb-gym-leaderboard

MTEB-Gym leaderboard

Label-free embedding-model rankings with a first-class reliability report. Companion to the MTEB-Gym paper (label-free, LLM-judged evaluation of embedding models).

What this is

Each corpus tab shows a Bradley–Terry ranking of embedding models built from pairwise LLM-judge verdicts over frozen synthetic queries — no relevance labels enter the pipeline. The Reliability tab reports, for every corpus, the judge's chance-corrected agreement (κ) with official human qrels measured on that corpus's own human queries. A ranking here is only as trustworthy as its κ, and the leaderboard says so instead of hiding it.

Design principles (from the paper)

• —No server-side inference. Free CPU Space; all judging happens offline in batch against frozen, hashed inputs, so there is no standing GPU cost.
• —Frozen and hashed everything: query sets, corpus snapshots, judge model + prompt version (registry commit pinned). Absent registry entries reproduce the generic judge prompt byte-for-byte.
• —Reliability before ranking: per-corpus κ from the human-query validation study ships alongside every table, with bootstrap confidence intervals on ratings.

Running locally

pip install -r requirements.txt
python app.py

Data provenance

data/leaderboard_export.json is generated from the run artifacts on the experiment cluster (commit b5327e9 of MTEB-gym-v2). Every rating traces to per-verdict JSONL checkpoints retained per run.

Roadmap

• —Public append-only verdict store (JSONL dataset, released with the paper); ratings recompute from raw verdicts, submissions add verdicts, never mutate them
• —Batch BT refit on CI for new submissions

• —Submissions dataset + CI refit workflow (run-file PRs)
• —Arena vote stream as an ongoing human-calibration feed
• —Per-corpus reliability proxies for user-supplied corpora (research in progress; judge self-consistency alone is not a valid proxy — see paper)