CoolFace
Apppublic

mischeiwiller/mteb-de-leaderboard

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face
0likes
App README

MTEB-DE Leaderboard

A sortable, mobile-first Gradio leaderboard for the MTEB-DE German embedding benchmark suite. It renders a multi-task table (retrieval / reranking / STS / clustering) plus an overall score from a bundled results.json — the reproducible output of the project's eval harness.

  • —Suite dataset: `mischeiwiller/mteb-de`
  • —Tasks: retrieval (germandpr, jobs), reranking (jobs), STS (stsb_de), clustering (jobs_occupation)
  • —Metric per task: nDCG@10 (retrieval/reranking), Spearman (STS), V-measure (clustering)

Files

FilePurpose
app.pyGradio Blocks UI — sort controls + dataframe + submit-a-model form + notes
leaderboard.pyPure data-prep: load results.json, pivot into the model × task table
submit.pyPure submit-a-model intake: format a submission payload for a Hub discussion
results.jsonBundled, merged harness scores (regenerate with scripts/build_leaderboard_data.py)
requirements.txtSpace dependencies

Run locally

bash
pip install -r requirements.txt
python app.py

Overall is the mean of a model's available task scores; sort by a task column for a fair head-to-head.

Submit a model (v1 — manual review)

The Space runs no model — it is a static view over the harness's reproducible offline scores. Submissions are reviewed by hand:

  1. 1.Submitter: open the 📥 Submit a model panel, enter the model's Hub repo id (owner/name), optional revision + contact, and the tasks. Click Generate submission and paste the produced block into a new discussion.
  2. 2.Maintainer: validate, then score with the same reproducible harness behind every row, and merge the result into results.json:
bash
   # validate (pin the revision + print the per-task commands):
   projects/02-mteb-de/.venv/bin/python projects/02-mteb-de/scripts/score_submission.py \
       --model owner/name --validate
   # full run (score + merge into space/results.json):
   projects/02-mteb-de/.venv/bin/python projects/02-mteb-de/scripts/score_submission.py \
       --model owner/name --tasks retrieval sts clustering
  1. 1.Maintainer: re-push the Space to publish the new row.

Hardware: cpu-basic — no GPU is needed because scoring happens offline in the harness, not in the Space.

Related