datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Amharic-Passage-Retrieval-Dataset-V2-With-NegativesThis dataset can be used directly with Sentence Transformers to train Amharic Embedding and Reranking models.
The negative_passages column contains 16 negative passages for each query, mined from the training set passages using an embedding model.
Source Datasets:
https://huggingface.co/datasets/rasyosef/amharic-news-category-classification
https://huggingface.co/datasets/rasyosef/xlsum_amharic_summarization
https://huggingface.co/datasets/yosefw/amharic-wikipedia… See the full description on the dataset page: https://huggingface.co/datasets/rasyosef/Amharic-Passage-Retrieval-Dataset-V2-With-Negatives.amharic-passage-retrieval-dataset-v2-with-negativespoleval2022-passage-retrieval-datasetbge-retrieval-data-ivf-passage-pruning-438KAmharic-Passage-Retrieval-Dataset-V2
Amharic Passage Retrieval Dataset V2
This dataset is the official benchmark for the paper "The Multilingual Curse at the Retrieval Layer: Evidence from Amharic".
It provides a fixed 90/10 train–test split consisting of 68,000 query–passage pairs, specifically designed for evaluating and training dense, late-interaction, learned sparse, and cross-encoder retrieval models for the Amharic language.
GitHub Repository: rasyosef/amharic-neural-ir
Paper: The Multilingual Curse at the… See the full description on the dataset page: https://huggingface.co/datasets/rasyosef/Amharic-Passage-Retrieval-Dataset-V2.amharic-passage-retrieval-dataset
Amharic Passage Retrieval Dataset
This dataset is a version of amharic-news-category-classification that has been filtered, deduplicated, and formatted for passage retrieval.
This dataset can be used directly with Sentence Transformers to train Amharic embedding models.
Source Datasets:
https://huggingface.co/datasets/rasyosef/amharic-news-category-classification
Models and Code
The following Text Embedding and ColBERT late-interaction retrieval models were… See the full description on the dataset page: https://huggingface.co/datasets/rasyosef/amharic-passage-retrieval-dataset.bge-retrieval-data-ivf-passage-pruning-100Kamharic-passage-retrieval-dataset-with-negatives
Amharic Passage Retrieval Dataset with Negatives
This dataset is a version of amharic-news-category-classification that has been filtered, deduplicated, and formatted for passage retrieval.
This dataset can be used directly with Sentence Transformers to train Amharic Text embedding and Reranking models.
Hard Negatives:
The negative_passages column contains hard negative passages that were mined using the roberta-amharic-text-embedding-medium embedding model.… See the full description on the dataset page: https://huggingface.co/datasets/rasyosef/amharic-passage-retrieval-dataset-with-negatives.amharic-passage-retrieval-datasetAfriXIR-Passage-Retrieval-With-Negatives
AfriXIR Passage Retrieval with Hard Negatives
Dataset Summary
AfriXIR-Passage-Retrieval-With-Negatives is a large-scale, multilingual passage retrieval dataset designed for training and evaluating information retrieval systems across eight African languages. The dataset contains 313,592 training examples and 34,849 test examples, with each example consisting of a query-passage pair accompanied by 16 hard negative passages mined through semantic similarity.
This… See the full description on the dataset page: https://huggingface.co/datasets/nebhailema/AfriXIR-Passage-Retrieval-With-Negatives.bge-retrieval-data-ivf-passage-pruning-fixed-100Kspanish_passage_retrieval
Spanish Passage Retrieval
This repository provides data from https://mklab.iti.gr/results/spanish-passage-retrieval-dataset/ as a HF dataset.
There is an S2S (retrieve passages/sentences that are marked as relevant) and an
S2P (retrieve documents that contain relevant passages/sentences) version of the retrieval task. The respective corpuses are called 'corpus.sentences' and 'corpus.documents'.
The qrel data is contained in 'qrels.s2s' and 'qrels.s2p', which hold space-separated… See the full description on the dataset page: https://huggingface.co/datasets/jinaai/spanish_passage_retrieval.xtremeup-retrieval-cross-lang-mdpr-passage-nqnew-amharic-passage-retrieval-dataset
New Amharic Passage Retrieval Dataset (AMQA)
This dataset is generated from AMQA-style question–context pairs, converted to match
the schema of Desalegnn/amharic-passage-retrieval-dataset.
Columns
query_id, passage_id, query, passage, category, link
Load Example
from datasets import load_dataset
url = 'https://huggingface.co/datasets/Desalegnn/new-amharic-passage-retrieval-dataset/resolve/main/train.jsonl'
ds = load_dataset('json', data_files={'train': url})… See the full description on the dataset page: https://huggingface.co/datasets/Desalegnn/new-amharic-passage-retrieval-dataset.longbench-passage-retrievalbge-retrieval-data-ivf-passage-pruning-200Kbge-retrieval-data-ivf-passage-pruning-fixed-438Kbge-retrieval-data-ivf-passage-pruning-50Kbge-retrieval-data-ivf-passage-pruning-fixed-50Kbge-retrieval-data-ivf-passage-pruning-fixed-200K