CoolFace
Datasetpublic

zc277584121/modern-embedding-bench

Modern Embedding Bench Modern Embedding Bench evaluates embedding models on practical retrieval tasks that show up in current AI systems but are often under-covered by broad leaderboards. The focus is on agent memory, tool and document retrieval, long-context RAG, cross-lingual technical retrieval, coding-oriented retrieval, and multimodal search rather than a single aggregate score. The companion leaderboard Space is available at:… See the full description on the dataset page: https://huggingface.co/datasets/zc277584121/modern-embedding-bench.

sourceHugging Facemitupdated 2mo agoView on Hugging Face
0likes29downloads
Dataset Card

Modern Embedding Bench

Modern Embedding Bench evaluates embedding models on practical retrieval tasks that show up in current AI systems but are often under-covered by broad leaderboards. The focus is on agent memory, tool and document retrieval, long-context RAG, cross-lingual technical retrieval, coding-oriented retrieval, and multimodal search rather than a single aggregate score.

The companion leaderboard Space is available at: https://huggingface.co/spaces/zc277584121/modern-embedding-bench-leaderboard

The source code is available at: https://github.com/zc277584121/modern-embedding-bench

Contents

  • —models.jsonl: registered model specs
  • —tasks.jsonl: registered task specs and primary metrics
  • —runs/: run manifests used to produce results
  • —results/latest.jsonl: canonical v2 result records
  • —leaderboards/latest.csv: flat leaderboard table derived from result records
  • —benchmark_data/: optional benchmark input data exported from the local repo

Current Public Export

  • —Registry model specs: 20
  • —Public model specs exported: 18
  • —Excluded private or preview model specs: 2
  • —Task specs: 6
  • —Result records: 292 records, 246 successful, 46 failed
  • —Leaderboard rows: 239
  • —Tasks with leaderboard rows: 4
  • —Providers with leaderboard rows: 10
  • —Unique task/model leaderboard pairs: 60
  • —Duplicate task/model repeats kept for inspection: 179
  • —Latest task/model marker rows: 60
  • —Evidence tiers: legacy=239
  • —Data: Bundled JSONL benchmark inputs are included.

Tasks

IDNamePrimary metricWhat it probes
autonomous_drivingAutonomous driving retrievalavg_recall@1Scenario retrieval for autonomous-driving style multimodal/text fallback cases.
chinese_multimodalChinese multimodal retrievalavg_recall@1Chinese multimodal/text fallback retrieval scenarios.
crossmodalretrievalText-image retrievalhardavgrecall@1COCO-style text-image matching with hard negative captions.
crosslingual_retrievalChinese-English retrievalhardavgrecall@1Bidirectional technical retrieval with hard negatives across Chinese and English.
mrl_stressMRL compression stressspearmandim128Semantic stability when embeddings are truncated to smaller dimensions.
needleinhaystackLong-document needle retrievaloverall_accuracyRetrieving facts inserted at different positions in long documents.

Result Format

Each line in results/latest.jsonl is one model-task run. Important fields:

  • —run: run id, description, metadata, and git sha when available
  • —model: model id, display name, provider, modalities, dimensions, and tags
  • —task: task id, dataset version, primary metric, and task kwargs
  • —metrics: task-specific metric dictionary
  • —details: diagnostic details for deeper analysis
  • —error: error text for failed runs, otherwise null

Leaderboard Provenance

leaderboards/latest.csv keeps every public row, including historical duplicate runs for the same task_id and model_id. The first columns remain compatible with older CSV readers, and provenance columns are appended:

  • —evidence_tier: legacy, smoke, benchmark, or unknown
  • —evidence_source: legacy source file, git sha, or run id when available
  • —task_model_duplicate_count: rows kept for the same task/model pair
  • —task_model_run_rank: 1-based order for that task/model pair
  • —is_latest_for_task_model: true for the latest exported row in that pair

Latest markers are computed from the order of results/latest.jsonl when result records are available, otherwise from CSV row order. Use is_latest_for_task_model=true to inspect one current row per task/model pair without losing the full historical trail.

Usage

Install and inspect the registry:

bash
uv sync
uv run modern-embed-bench benchmark models
uv run modern-embed-bench benchmark tasks

Run a small OpenAI smoke benchmark:

bash
uv run modern-embed-bench benchmark run \
  --manifest benchmark/runs/openai-smoke.yaml \
  --output results/openai-smoke.jsonl \
  --overwrite

uv run modern-embed-bench benchmark leaderboard \
  --results results/openai-smoke.jsonl \
  --output results/openai-smoke-leaderboard.csv

Notes and Limitations

  • —Rows imported from legacy runs are published for continuity and should be read as historical baseline evidence, not as a fully normalized one-shot run.
  • —Scores are task-specific. Avoid comparing scores across tasks as if they were one global ranking.
  • —Some preview or private-in-progress model results are intentionally excluded from the public export until they are ready for publication.
  • —Image binaries are not bundled by default; cross_modal metadata is included separately from the source image files.