sbintuitions/JMTEB-lite

JMTEB-lite: The Lightweight Version of JMTEB

JMTEB-lite is a lightweight version of JMTEB. It makes agile evaluation possible by reaching an average of 5x faster evaluation comparing with JMTEB. The result of JMTEB-lite is proved to be highly relevant with that of JMTEB, making it a faithful preview of JMTEB.

TL;DR

from datasets import load_dataset

dataset = load_dataset("sbintuitions/JMTEB-lite", name="<dataset_name>", split="<split>")

JMTEB_LITE_DATASET_NAMES = (
    'livedoor_news',
    'mewsc16_ja',
    'sib200_japanese_clustering',
    'amazon_review_classification',
    'amazon_counterfactual_classification',
    'massive_intent_classification',
    'massive_scenario_classification',
    'japanese_sentiment_classification',
    'sib200_japanese_classification',
    'wrime_classification',
    'jsts',
    'jsick',
    'jaqket-query',
    'jaqket-corpus',  # lightweight
    'mrtydi-query',
    'mrtydi-corpus',  # lightweight
    'jagovfaqs_22k-query',
    'jagovfaqs_22k-corpus',
    'nlp_journal_title_abs-query',
    'nlp_journal_title_abs-corpus',
    'nlp_journal_title_intro-query',
    'nlp_journal_title_intro-corpus',
    'nlp_journal_abs_intro-query',
    'nlp_journal_abs_intro-corpus',
    'nlp_journal_abs_article-query',
    'nlp_journal_abs_article-corpus',
    'jacwir-retrieval-query',
    'jacwir-retrieval-corpus',  # lightweight
    'miracl-retrieval-query',
    'miracl-retrieval-corpus',  # lightweight
    'mldr-retrieval-query',
    'mldr-retrieval-corpus',
    'mintaka-retrieval-query',
    'mintaka-retrieval-corpus',
    'esci-query',
    'esci-corpus',
    'jqara-query',  # lightweight
    'jqara-corpus',  # lightweight
    'jacwir-reranking-query',  # lightweight
    'jacwir-reranking-corpus',  # lightweight
    'miracl-reranking-query',
    'miracl-reranking-corpus',
    'mldr-reranking-query',
    'mldr-reranking-corpus',
)

Introduction

We introduced JMTEB (Japanese Massive Text Embedding Benchmark), a comprehensive evaluation benchmark of Japanese text embedding models. However, the massive size of JMTEB makes evaluation slow and resource demanding. To address this, we now introduce JMTEB-lite, a lightweight version of JMTEB constructed by substaintially reducing corpus size in retrieval and reranking tasks. We have also verified that JMTEB-lite significantly accelerates evaluation while maintaining high fidelity to the full JMTEB.

We recommand to use JMTEB-lite to obtain the preview evaluation results in agile development, and use JMTEB for full and final evaluation.

JMTEB-lite is compatible with the evaluation script of JMTEB: <https://github.com/sbintuitions/JMTEB>.

Tasks and Datasets

Here is an overview of the tasks and datasets currently included in JMTEB-lite.

Note that only datasets in bold are lightweight, and the rest are exactly the same with the counterparts in JMTEB.

[^1]: To keep consistent with MTEB where Mintaka-Retrieval doesn't have a validation set, we set our validation set the same as the test set.

Construction Process

For the 4 retrieval datasets (JAQKET, Mr.TyDi, JaCWIR-Retrieval, MIRACL-Retrieval), we use 5 highly performant models to predict hard negative documents for each query (the query's most 50 semantically similar documents in the corpus), and merge these hard negatives along with golden documents.

For the 2 reranking datasets (JQaRA, JaCWIR-Reranking), we use 5 highly performant models to rerank the documents for each query, and retain top-50 hard negative documents for each query. Then we merge these hard negatives with golden documents.

For the rest, they are kept exactly the same with their counterparts in JMTEB.

Reference

@misc{jmteb_lite,
	author = {Li, Shengzhe and Ohagi, Masaya and Ri, Ryokan and Fukuchi, Akihiko and Shibata, Tomohide and Kawahara, Daisuke},
	title = {{J}{M}{T}{E}{B}-lite: {T}he {L}ightweight {V}ersion of {JMTEB}},
	howpublished = {\url{https://huggingface.co/datasets/sbintuitions/JMTEB-lite}},
	year = {2025},
}

@inproceedings{li2026jmteb,
    author = {Li, Shengzhe and Ohagi, Masaya and Ri, Ryokan and Fukuchi, Akihiko and Shibata, Tomohide and Kawahara, Daisuke},
    title = {{JMTEB and JMTEB-lite: Japanese Massive Text Embedding Benchmark and Its Lightweight Version}},
    booktitle = "Proceedings of the Fifteenth Language Resources and Evaluation Conference",
    month = may,
    year = "2026",
    address = "Palma, Mallorca, Spain",
    publisher = "European Language Resources Association",
    note = "to appear",
}

Legacy Loading Scripts

The Python loading script (JMTEB-lite.py) has been moved to the `legacy/` directory for historical reference. It was previously used to load the dataset with trust_remote_code=True, but is no longer required. The dataset now uses Parquet format and can be loaded directly without executing any custom code.

License

Regarding the license information of datasets, please refer to the individual datasets.

Our code is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License.

<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /></a>