similarity
linguistic-similaritywiki-entity-similarity
Wiki Entity Similarity
Usage:
from datasets import load_dataset
corpus = load_dataset('Exr0n/wiki-entity-similarity', '2018thresh20corpus', split='train')
assert corpus[0] == {'article': 'A1000 road', 'link_text': 'A1000', 'is_same': 1}
pairs = load_dataset('Exr0n/wiki-entity-similarity', '2018thresh20pairs', split='train')
assert corpus[0] == {'article': 'Rhinobatos', 'link_text': 'Ehinobatos beurleni', 'is_same': 1}
assert len(corpus) == 4_793_180
Corpus (name=*corpus)… See the full description on the dataset page: https://huggingface.co/datasets/Exr0n/wiki-entity-similarity.patent-phrase-similarityheadlines-semantic-similarity
Dataset Card for HEADLINES
Dataset Summary
HEADLINES is a massive English-language semantic similarity dataset, containing 396,001,930 pairs of different headlines for the same newspaper article, taken from historical U.S. newspapers, covering the period 1920-1989.
Languages
The text in the dataset is in English.
Dataset Structure
Each year in the dataset is divided into a distinct file (eg. 1952_headlines.json), giving a total of 70 files.
The… See the full description on the dataset page: https://huggingface.co/datasets/dell-research-harvard/headlines-semantic-similarity.stockimage-1.5M-scored-low-similaritystockimage-1.5M-scored-high-similarity