datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
linguistic-similaritywiki-entity-similarity
Wiki Entity Similarity
Usage:
from datasets import load_dataset
corpus = load_dataset('Exr0n/wiki-entity-similarity', '2018thresh20corpus', split='train')
assert corpus[0] == {'article': 'A1000 road', 'link_text': 'A1000', 'is_same': 1}
pairs = load_dataset('Exr0n/wiki-entity-similarity', '2018thresh20pairs', split='train')
assert corpus[0] == {'article': 'Rhinobatos', 'link_text': 'Ehinobatos beurleni', 'is_same': 1}
assert len(corpus) == 4_793_180
Corpus (name=*corpus)… See the full description on the dataset page: https://huggingface.co/datasets/Exr0n/wiki-entity-similarity.patent-phrase-similarityheadlines-semantic-similarity
Dataset Card for HEADLINES
Dataset Summary
HEADLINES is a massive English-language semantic similarity dataset, containing 396,001,930 pairs of different headlines for the same newspaper article, taken from historical U.S. newspapers, covering the period 1920-1989.
Languages
The text in the dataset is in English.
Dataset Structure
Each year in the dataset is divided into a distinct file (eg. 1952_headlines.json), giving a total of 70 files.
The… See the full description on the dataset page: https://huggingface.co/datasets/dell-research-harvard/headlines-semantic-similarity.stockimage-1.5M-scored-low-similaritystockimage-1.5M-scored-high-similaritycharacter_similarity
character_similarity
This is a dataset used for training models to determine whether two anime images (containing only one person) depict the same character. The dataset includes the following versions:
Version
Filename
Characters
Images
Information
v0
images_v0.tar.xz
2059
162116
Crawled from zerochan.net, includes images of Arknights, Fate/Grand Order, Genshin Impact, Girls' Frontline, and Azur Lane, as well as over 1500 other game or anime characters. The images are… See the full description on the dataset page: https://huggingface.co/datasets/deepghs/character_similarity.x-span-similarity
X-Span-Similarity (X-SSD)
Expanding Lozano et al.'s (2026) Span Similarity Dataset (SSD) into a cross-lingual setting, for Dissimilar/Difference Span Detection (DSD) across language pairs.
Dataset summary
Each row is a premise/hypothesis sentence pair, one side in English and the other machine-translated into another target language, with span-level and sentence-level (dis)similarity labels carried over unchanged from the original English SSD annotation. Spans… See the full description on the dataset page: https://huggingface.co/datasets/ZurichNLP/x-span-similarity.llm-similarity-risklm-similarity
Great Models Think Alike and this Undermines AI Oversight
This is the data collection for the publication "Great Models Think Alike and this Undermines AI Oversight."
judge_scores_mmlu_pro_free_filtered: Judge scores of nine judges without access to the reference answers on the filtered, open-style MMLU-Pro dataset.
judge_w_gt_mmlu_pro_free_filtered: Ensemble judge scores of five judges with access to the reference options and ground-truth information on the filtered OSQ MMLU-Pro.… See the full description on the dataset page: https://huggingface.co/datasets/bethgelab/lm-similarity.task1347_glue_sts-b_similarity_classification
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task1347_glue_sts-b_similarity_classification
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task1347_glue_sts-b_similarity_classification.human-aligned-similarity-benchmark
Human Aligned Similarity Benchmark
You are welcome to go to alignedmachine.com to contribute.
Overview
This dataset contains human-aligned similarity judgments for embedding text and multimodal AI model evaluation. The benchmark is designed to assess how well AI models align with human cognitive preferences in similarity perception across text and image modalities.
Dataset Structure
Concept Files
This dataset contains human preference judgments for… See the full description on the dataset page: https://huggingface.co/datasets/duke-trust-lab/human-aligned-similarity-benchmark.Job-Title-Similarity
Job Title Similarity Benchmark
This dataset contains the Job Title Similarity benchmark for easy loading with
the HuggingFace datasets library.
Dataset Description
This benchmark provides job title similarity datasets in 11 different languages,
including the English evaluation dataset used in the text ranking experiments reported
in the paper "Learning Job Titles Similarity from Noisy Skill Labels" by Zbib et al. (2022),
as well as the translated datasets for the other… See the full description on the dataset page: https://huggingface.co/datasets/Avature/Job-Title-Similarity.arXiv-metadata-oai-snapshot-111github-issue-similarity
GIS: Github Issue Similarity Dataset
This dataset was released from the paper: https://arxiv.org/abs/2309.12871
Citation
If you use our dataset in your research, welcome to cite us as follows:
@article{li2023angle,
title={AnglE-optimized Text Embeddings},
author={Li, Xianming and Li, Jing},
journal={arXiv preprint arXiv:2309.12871},
year={2023}
}
synthetic-text-similarity
Synthetic Text Similarity
This dataset is created to facilitate the evaluation and training of models on the task of text similarity at longer contexts/examples than Bob likes frogs. as per classical sentence similarity datasets.
It consists of document pairs with associated similarity scores, representing the closeness of the documents in semantic space.
Dataset Description
For each version of this dataset, embeddings are computed for all unique documents, followed by… See the full description on the dataset page: https://huggingface.co/datasets/pszemraj/synthetic-text-similarity.pubchem_10m_genmol_similarity
PubChem 10M GenMol Fingerprint Similarity Dataset
This dataset is an augmented version of the PubChem 10M dataset, enhanced with molecular similarity data generated using GenMol, as described in the paper GenMol: A Drug Discovery Generalist with Discrete Diffusion.
The dataset contains molecular structures represented as SMILES strings along with their corresponding molecular fingerprints, similarity scores, and various molecular properties.
This dataset is for training a Chem-MRL… See the full description on the dataset page: https://huggingface.co/datasets/Derify/pubchem_10m_genmol_similarity.WikiMedical_sentence_similarity
Dataset Card for "WikiMedical_sentence_similarity"
WikiMedical_sentence_similarity is an adapted and ready-to-use sentence similarity dataset based on this dataset.
The preprocessing followed three steps:
Each text is splitted into sentences of 256 tokens (nltk tokenizer)
Each sentence is paired with a positive pair if found, and a negative one. Negative one are drawn randomly in the whole dataset.
Train and test split correspond to 70%/30%
More Information needed
phrase_similarityPhrase in Context is a curated benchmark for phrase understanding and semantic search, consisting of three tasks of increasing difficulty: Phrase Similarity (PS), Phrase Retrieval (PR) and Phrase Sense Disambiguation (PSD). The datasets are annotated by 13 linguistic experts on Upwork and verified by two groups: ~1000 AMT crowdworkers and another set of 5 linguistic experts. PiC benchmark is distributed under CC-BY-NC 4.0.MEN-word-similarity
MEN Word Similarity Dataset
Dataset Description
The MEN dataset is a word similarity dataset consisting of 3,000 word pairs annotated with human similarity judgments.
It is commonly used to evaluate distributional semantic models such as Word2Vec, GloVe, fastText, and newer embedding models.
Each example consists of:
word1: First word
word2: Second word
score: Human similarity score (higher means more similar)
The dataset is provided in its natural form, without… See the full description on the dataset page: https://huggingface.co/datasets/Yuti/MEN-word-similarity.span-similarity-dataset
Span Similarity Dataset (SSD)
Dataset Summary
The Span Similarity Dataset (SSD) focuses on Explainable Textual Similarity. It consists
of pairs of sentences with annotations pointing to both semantically equivalent and
dissimilar spans.
Languages
The SSD includes exclusively texts in English.
Dataset Structure
The dataset is split into -train (800 samples), -eval (100 samples), and -test (100
samples), all of them provided as a .tsv… See the full description on the dataset page: https://huggingface.co/datasets/tum-nlp/span-similarity-dataset.task146_afs_argument_similarity_gun_control
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task146_afs_argument_similarity_gun_control
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task146_afs_argument_similarity_gun_control.task145_afs_argument_similarity_death_penalty
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task145_afs_argument_similarity_death_penalty
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task145_afs_argument_similarity_death_penalty.Common_voice_sentence_similaritytask147_afs_argument_similarity_gay_marriage
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task147_afs_argument_similarity_gay_marriage
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task147_afs_argument_similarity_gay_marriage.semeval2012_relational_similarity[SemEVAL 2012 task 2: Relational Similarity](https://aclanthology.org/S12-1047/)similarity-dataset-sc2-8b
Dataset Card for similarity-dataset-sc2-8b
This dataset has been created with distilabel and the pipeline outlined here.
It is designed as a synthetic dataset for training Sentence Transformers models, providing structured examples to help models learn fine-grained semantic distinctions in various domains.
Dataset Summary
The similarity-dataset-sc2-8b was generated to serve as training data for models that need to understand subtle differences and… See the full description on the dataset page: https://huggingface.co/datasets/davanstrien/similarity-dataset-sc2-8b.msmarco-train-distil-MiniLM-L6-v2-similarityAdaptive_Fluency_Similarity_Matrices_and_Frequency_Table_for_AnimalsThe contents of this dataset are from the Forager computer program. Its interactive interface can be found at https://forager.research.bowdoin.edu.
Please read the LICENSE file in this data set and check for further information on the Forager GitHub repository (https://github.com/thelexiconlab/forager-web)
in addition to the docs tab of the web interface linked above.
Citation:
Kumar, A.A., Apsel, M., Zhang, L., Xing, N., Jones. M.N. (2023). forager: A Python package and web interface for… See the full description on the dataset page: https://huggingface.co/datasets/natebranda/Adaptive_Fluency_Similarity_Matrices_and_Frequency_Table_for_Animals.gvg-explainable-game-similarity
GVG Explainable Game Similarity Dataset 2026 (v0.1.0)
50 human-reviewed pairs of similar PC games (82 games) from Game V Game. Each row says why the two games are similar, what differs, and who each suits, in English and Chinese, and cites the two official Steam store records the claims were checked against (with capture dates).
Most similarity data says two games are alike and stops. This one is small on purpose: every row was read by a person against the current Steam records… See the full description on the dataset page: https://huggingface.co/datasets/lette2/gvg-explainable-game-similarity.