MTEB-BR/mteb-pt-results

<div align="center">

🇧🇷 MTEB-BR — Benchmark Results

Canonical results store for MTEB-BR, a native Brazilian-Portuguese text-embedding benchmark.

    

93 models · 22 native PT-BR tasks · 7 categories · no machine translation

</div>

What is this?

This repository is the canonical, machine-readable results store for MTEB-BR — a benchmark that evaluates text-embedding models on native Brazilian Portuguese (data created or found in Portuguese; machine-translated corpora such as mMARCO-PT are excluded by construction). Every model is evaluated with `mteb` on datasets pinned to a revision SHA.

The aggregated `score_matrix.parquet` (93 models × 22 tasks, browsable in the Dataset Viewer above) is the quickest way to see all scores; the raw per-evaluation JSON artifacts live under results/.

📊 Leaderboard (top 15 by 22-task mean)

Type: O = open-weight, C = closed/commercial API. Full 93-model × 22-task table: `score_matrix.parquet` (viewer above). A MTEB-BR/baseline-random-encoder chance floor (mean₂₂ = 0.18) is included in the raw results for reference.

🗂️ The 22 tasks (7 categories)

Domains span legal, medical, tax, scientific, encyclopedic, and social-media Portuguese. Per-task sources, licenses, and citations are documented in the code repository.

📁 Repository layout & how to use

score_matrix.parquet                     # aggregated 93 x 22 matrix (viewer-friendly)
results/
  {org}__{model}/{revision-sha}/
    {Task}.json                          # per-(model, task) mteb result (the canonical score)

There are 2,800+ per-evaluation JSON files plus per-instance .jsonl dumps behind the confidence intervals.

Load the aggregated matrix:

import pandas as pd
from huggingface_hub import hf_hub_download
df = pd.read_parquet(hf_hub_download("MTEB-BR/mteb-pt-results", "score_matrix.parquet", repo_type="dataset"))
df.sort_values("mean_22", ascending=False).head()

Pull the full raw store (all JSONs):

from huggingface_hub import snapshot_download
snapshot_download("MTEB-BR/mteb-pt-results", repo_type="dataset", local_dir="mteb-pt-results")

🔗 Links

• —Interactive leaderboard — <https://huggingface.co/spaces/MTEB-BR/leaderboard>
• —Code & task definitions — <https://github.com/tardellirs/mteb-br>
• —Organization — <https://huggingface.co/MTEB-BR>
• —This dataset’s DOI — <https://doi.org/10.57967/hf/9377>
• —Code archive (Zenodo DOI) — <https://doi.org/10.5281/zenodo.21087217> · archives the code repository, not this results dataset

📄 License

• —This results dataset: CC-BY-4.0.
• —Individual task datasets: each retains its original source license (see the paper / code repo task table).
• —Models evaluated: see each model card.

📚 Citation

@misc{mteb-br-2026,
  title  = {MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguese},
  author = {Stekel, Tardelli R. C.},
  year   = {2026},
  doi    = {10.5281/zenodo.21087217},  % Zenodo archive of the benchmark code
  url    = {https://doi.org/10.5281/zenodo.21087217}
}

If you use a specific task novel to this benchmark, please also cite the original task dataset. arxiv.org/abs/2607.04581