datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
gap_raw
Dataset Card for "gap"
Dataset Summary
GAP is a gender-balanced dataset containing 8,908 coreference-labeled pairs of
(ambiguous pronoun, antecedent name), sampled from Wikipedia and released by
Google AI Language for the evaluation of coreference resolution in practical
applications.
Dataset Structure
Data Instances
default
Size of downloaded dataset files: 2.40 MB
Size of the generated dataset: 2.43 MB
Total amount of disk used: 4.83 MB… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/gap_raw.gen_winograd_raw
gen_winograd
Project: https://ufal.mff.cuni.cz/corefud
Data source: https://github.com/mbzuai-nlp/gen-X/tree/bf1c0adb4b4def03cdf419c18b2948695bc1fab8
Details
English Winograd generated by GPT-4
Citation
@misc{whitehouse2023llmpowered,
title={LLM-powered Data Augmentation for Enhanced Crosslingual Performance},
author={Chenxi Whitehouse and Monojit Choudhury and Alham Fikri Aji},
year={2023},
eprint={2305.14288}… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/gen_winograd_raw.dpr_raw
"definite_pronoun_resolution" (dpr)
Dataset Summary
Composed by 30 students from one of the author's undergraduate classes. These
sentence pairs cover topics ranging from real events (e.g., Iran's plan to
attack the Saudi ambassador to the U.S.) to events/characters in movies (e.g.,
Batman) and purely imaginary situations, largely reflecting the pop culture as
perceived by the American kids born in the early 90s. Each annotated example
spans four lines: the first line… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/dpr_raw.davis_wsc_raw
The original Winograd Schema Challenge (WSC) as hosted by Ernest Davis
Dataset Summary
The original Winograd Schema Challenge (WSC) consisted of 136 schemas resulting in 273 problems. This was later expanded to 150 schemas resulting in 285 problems.
A Winograd schema is a pair of sentences that differ in only one or two words and that contain an ambiguity that is
resolved in opposite ways in the two sentences and requires the use of world knowledge and reasoning for its… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/davis_wsc_raw.superglue_wsc_raw
Winograd Schema Challenge examples included in the SuperGLUE Benchmark
Specifically: The wsc and wsc.fixed datasets from the HuggingFace "super_glue" repository.
Data Fields
text (str): The text of the schema.
span1_index (int): Starting word index of first entity.
span2_index (int): Starting word index of second entity.
span1_text (str): Textual representation of first entity.
span2_text (str): Textual representation of second entity.
idx (int): Index of the example in… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/superglue_wsc_raw.winogrande_raw
Wingrande v1.1
Dataset Summary
WinoGrande is a new collection of 44k problems, inspired by Winograd Schema Challenge (Levesque, Davis, and Morgenstern
2011), but adjusted to improve the scale and robustness against the dataset-specific bias. Formulated as a
fill-in-a-blank task with binary options, the goal is to choose the right option for a given sentence which requires
commonsense reasoning.
Data Fields
The data fields are the same among all splits.… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/winogrande_raw.davis_pdp_raw
Pronoun Disambiguation Problems (PDP) from the 2016 WSC as hosted by Ernest Davis
60 pronoun disambiguation problems from https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WS.html
Data Fields
text (str): The text sequence
options (list[str]): The two entity options that the pronoun may be referring to
label (int): The index of the correct option in the options field
pronoun (str): The pronoun in the sequence to be resolved
pronoun_loc (int): The starting position… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/davis_pdp_raw.mwsc_raw
The Modified Winograd Schema Challenge (MWSC)
Dataset Summary
Examples taken from the Winograd Schema Challenge modified to ensure that answers are a single word from the context.
This Modified Winograd Schema Challenge (MWSC) ensures that scores are neither inflated nor deflated by oddities in phrasing.
Dataset Structure
Data Instances
default
Size of downloaded dataset files: 0.02 MB
Size of the generated dataset: 0.04 MB
Total amount… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/mwsc_raw.raw-philippine-data
Raw Philippine Data
This repository contains raw data about Philippine politicians, public officials, and legislative documents collected from various sources. The data is intended for research, analysis, and civic technology purposes.
Dataset Overview
This dataset currently contains:
Persons
45,424 person records of Philippine politicians and public officials with:
ID: Unique identifier (ULID format)
First Name: Person's first name
Last Name: Person's last… See the full description on the dataset page: https://huggingface.co/datasets/bettergovph/raw-philippine-data.conversational_raw_datasetlitbank_raw
LitBank
Project: https://github.com/dbamman/litbank
Data source: https://github.com/dbamman/litbank/commit/3e50db0ffc033d7ccbb94f4d88f6b99210328ed8
Crossval splits source: https://github.com/dbamman/lrec2020-coref/commit/e30de53743d36d1ea2c9e7292c69477fa332713c
Details
Ten configs of the form f"split_{X}" where X is in range(10)
Features
{'coref_chains': List[List[List[int]]] # list of clusters, each cluster is a list of mentions, each mention is a… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/litbank_raw.pretrain-dataset-raw-10M
Pretrain Dataset (Text)
This dataset contains preprocessed text documents ready for LLM pretraining.
Dataset Details
Property
Value
Documents
10,000,000
Processed
10000000
Shards
21
Created
2025-12-09
Dataset Structure
Each sample contains:
text: The document text
source: Source dataset identifier
id: Unique document ID
Usage
from datasets import load_dataset
dataset = load_dataset("tvu-vlinhd11/pretrain-dataset-raw-10M")… See the full description on the dataset page: https://huggingface.co/datasets/tvu-vlinhd11/pretrain-dataset-raw-10M.no-raw-28Bflan2021_coreference_raw
Flan 2021 Coreference Tasks
Project: https://github.com/google-research/FLAN/tree/main/flan/v2
Data source: DataProvenanceInitiative/flan2021_submix_original
Details
This dataset contains all coreference examples that were included in the Flan 2022 collection which were orignally included in Flan 2021.
The data is copied from the preprocessed Flan2021 dataset at DataProvenanceInitiative/flan2021_submix_original.
COREFERENCE_TASK_NAMES = {… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/flan2021_coreference_raw.en-raw-28Bta-raw-28Bja-raw-28Bko-raw-28Bwiki_book_corpus_complete_raw_dataset
Dataset Card for "wiki_book_corpus_complete_raw_dataset"
More Information needed
niv2_winogrande_raw
Natural Instructions v2 Winogrande Tasks
Project: https://github.com/allenai/natural-instructions
Data source: DataProvenanceInitiative/niv2_submix_original
Details
This dataset contains all Winogrande examples that were included in the Flan 2022 collection which were orignally published in Super-Natural-Instructions.
The data is copied from the preprocessed Natural Instructions v2 dataset at DataProvenanceInitiative/niv2_submix_original.
These tasks are:… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/niv2_winogrande_raw.ar-raw-28Bet-raw-28Bel-raw-28Bdataset_v1_rawde-raw-28Bwiki_book_corpus_raw_dataset_small
Dataset Card for "wiki_book_corpus_raw_dataset_small"
More Information needed
wiki_book_corpus_raw_dataset_medium
Dataset Card for "wiki_book_corpus_raw_dataset_medium"
More Information needed
ru-raw-28Bfr-raw-28BRAW_DATA-openthoughts-Qwen2.5-7B-Instruct