datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
knowref_60k_raw
The Knowref 60K Dataset
Project: https://github.com/aemami1/KnowRef60k
Data source: https://github.com/aemami1/KnowRef60k/tree/28e5385d17967744ccb3bdba45fdd89d9690307d
Fields
annotation_strength (str): annotator agreement from 1-5
candidate_0 (str): the first candidate name
candidate_1 (str): the second candidate name
original_sentence (str): sentence before swapping the names
swapped_sentence (str): sentence after swapping the names with square brackets marking the… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/knowref_60k_raw.gap_raw
Dataset Card for "gap"
Dataset Summary
GAP is a gender-balanced dataset containing 8,908 coreference-labeled pairs of
(ambiguous pronoun, antecedent name), sampled from Wikipedia and released by
Google AI Language for the evaluation of coreference resolution in practical
applications.
Dataset Structure
Data Instances
default
Size of downloaded dataset files: 2.40 MB
Size of the generated dataset: 2.43 MB
Total amount of disk used: 4.83 MB… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/gap_raw.gen_winograd_raw
gen_winograd
Project: https://ufal.mff.cuni.cz/corefud
Data source: https://github.com/mbzuai-nlp/gen-X/tree/bf1c0adb4b4def03cdf419c18b2948695bc1fab8
Details
English Winograd generated by GPT-4
Citation
@misc{whitehouse2023llmpowered,
title={LLM-powered Data Augmentation for Enhanced Crosslingual Performance},
author={Chenxi Whitehouse and Monojit Choudhury and Alham Fikri Aji},
year={2023},
eprint={2305.14288}… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/gen_winograd_raw.dpr_raw
"definite_pronoun_resolution" (dpr)
Dataset Summary
Composed by 30 students from one of the author's undergraduate classes. These
sentence pairs cover topics ranging from real events (e.g., Iran's plan to
attack the Saudi ambassador to the U.S.) to events/characters in movies (e.g.,
Batman) and purely imaginary situations, largely reflecting the pop culture as
perceived by the American kids born in the early 90s. Each annotated example
spans four lines: the first line… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/dpr_raw.davis_wsc_raw
The original Winograd Schema Challenge (WSC) as hosted by Ernest Davis
Dataset Summary
The original Winograd Schema Challenge (WSC) consisted of 136 schemas resulting in 273 problems. This was later expanded to 150 schemas resulting in 285 problems.
A Winograd schema is a pair of sentences that differ in only one or two words and that contain an ambiguity that is
resolved in opposite ways in the two sentences and requires the use of world knowledge and reasoning for its… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/davis_wsc_raw.MasssiveDS-1.4T-raw-dataMassiveDS-1.4T-raw-dataWe release the raw passages, embeddings, and index of MassiveDS.
Website: https://retrievalscaling.github.io
Versions
We release two versions of MassiveDS:
MassiveDS-1.4T, which contains the embeddings and passages of the 1.4T-token datastore.
MassiveDS-1.4T-raw-text, contains the raw text of the 1.4T-token datastore.
MassiveDS-140B, which contains the index, embeddings, passages, and raw text of a subsampled version containing 140B tokens in the datastore.
Note:
Code support to… See the full description on the dataset page: https://huggingface.co/datasets/rulins/MassiveDS-1.4T-raw-data.raw_primitive_datasets
Datacard
This is the official fine-tuning dataset provided by VLABench (raw data), with 500 episodes each task. The current version includes 10 primitive tasks.
Source
Project Page: https://vlabench.github.io/
Arxiv Paper: https://arxiv.org/abs/2412.18194
Code: https://github.com/OpenMOSS/VLABench
Uses
Download all archive files and use the following command to extract:
cat vlabench_primitive.tar.gz.* | tar -xzvf -
In the resulting… See the full description on the dataset page: https://huggingface.co/datasets/VLABench/raw_primitive_datasets.Taxi1500-RawData
Taxi1500 Raw Data
Introduction
This repository contains the raw text data of the Taxi1500-c_v3.0 corpus, without classification labels and Bible verse ids. For the original Taxi1500 dataset for Text Classification, please refer to the GitHub repository.
The data format of the Taxi1500-RawData is identical to that of the Glot500 Dataset, facilitating seamless parallel utilization of both datasets.
Usage
Replace acr_Latn with your specific language.
from… See the full description on the dataset page: https://huggingface.co/datasets/cis-lmu/Taxi1500-RawData.superglue_wsc_raw
Winograd Schema Challenge examples included in the SuperGLUE Benchmark
Specifically: The wsc and wsc.fixed datasets from the HuggingFace "super_glue" repository.
Data Fields
text (str): The text of the schema.
span1_index (int): Starting word index of first entity.
span2_index (int): Starting word index of second entity.
span1_text (str): Textual representation of first entity.
span2_text (str): Textual representation of second entity.
idx (int): Index of the example in… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/superglue_wsc_raw.winogrande_raw
Wingrande v1.1
Dataset Summary
WinoGrande is a new collection of 44k problems, inspired by Winograd Schema Challenge (Levesque, Davis, and Morgenstern
2011), but adjusted to improve the scale and robustness against the dataset-specific bias. Formulated as a
fill-in-a-blank task with binary options, the goal is to choose the right option for a given sentence which requires
commonsense reasoning.
Data Fields
The data fields are the same among all splits.… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/winogrande_raw.mwsc_raw
The Modified Winograd Schema Challenge (MWSC)
Dataset Summary
Examples taken from the Winograd Schema Challenge modified to ensure that answers are a single word from the context.
This Modified Winograd Schema Challenge (MWSC) ensures that scores are neither inflated nor deflated by oddities in phrasing.
Dataset Structure
Data Instances
default
Size of downloaded dataset files: 0.02 MB
Size of the generated dataset: 0.04 MB
Total amount… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/mwsc_raw.davis_pdp_raw
Pronoun Disambiguation Problems (PDP) from the 2016 WSC as hosted by Ernest Davis
60 pronoun disambiguation problems from https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WS.html
Data Fields
text (str): The text sequence
options (list[str]): The two entity options that the pronoun may be referring to
label (int): The index of the correct option in the options field
pronoun (str): The pronoun in the sequence to be resolved
pronoun_loc (int): The starting position… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/davis_pdp_raw.raw-philippine-data
Raw Philippine Data
This repository contains raw data about Philippine politicians, public officials, and legislative documents collected from various sources. The data is intended for research, analysis, and civic technology purposes.
Dataset Overview
This dataset currently contains:
Persons
45,424 person records of Philippine politicians and public officials with:
ID: Unique identifier (ULID format)
First Name: Person's first name
Last Name: Person's last… See the full description on the dataset page: https://huggingface.co/datasets/bettergovph/raw-philippine-data.preco_raw
The PreCo Dataset
Project: https://preschool-lab.github.io/PreCo/
Data source: https://drive.google.com/file/d/1q0oMt1Ynitsww9GkuhuwNZNq6SjByu-Y/view?usp=sharing
Details
The original PreCo .jsonl files from https://preschool-lab.github.io/PreCo/
What is PreCo?
PreCo is a large-scale English dataset for coreference resolution. The dataset is designed to embody the core challenges in coreference, such as entity representation, by alleviating the challenge of… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/preco_raw.conversational_raw_datasetTaxi1500-RawData-test
Taxi1500 Raw Data
Introduction
This repository contains the raw text data of the Taxi1500-c_v3.0 corpus, without classification labels and Bible verse ids. For the original Taxi1500 dataset for Text Classification, please refer to the GitHub repository.
The data format of the Taxi1500-RawData is identical to that of the Glot500 Dataset, facilitating seamless parallel utilization of both datasets.
Usage
Replace acr_Latn with your specific language.
from… See the full description on the dataset page: https://huggingface.co/datasets/pei39/Taxi1500-RawData-test.en-raw-28Bpretrain-dataset-raw-10M
Pretrain Dataset (Text)
This dataset contains preprocessed text documents ready for LLM pretraining.
Dataset Details
Property
Value
Documents
10,000,000
Processed
10000000
Shards
21
Created
2025-12-09
Dataset Structure
Each sample contains:
text: The document text
source: Source dataset identifier
id: Unique document ID
Usage
from datasets import load_dataset
dataset = load_dataset("tvu-vlinhd11/pretrain-dataset-raw-10M")… See the full description on the dataset page: https://huggingface.co/datasets/tvu-vlinhd11/pretrain-dataset-raw-10M.flan2021_coreference_raw
Flan 2021 Coreference Tasks
Project: https://github.com/google-research/FLAN/tree/main/flan/v2
Data source: DataProvenanceInitiative/flan2021_submix_original
Details
This dataset contains all coreference examples that were included in the Flan 2022 collection which were orignally included in Flan 2021.
The data is copied from the preprocessed Flan2021 dataset at DataProvenanceInitiative/flan2021_submix_original.
COREFERENCE_TASK_NAMES = {… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/flan2021_coreference_raw.no-raw-28BDrivegpt4_raw_datalitbank_raw
LitBank
Project: https://github.com/dbamman/litbank
Data source: https://github.com/dbamman/litbank/commit/3e50db0ffc033d7ccbb94f4d88f6b99210328ed8
Crossval splits source: https://github.com/dbamman/lrec2020-coref/commit/e30de53743d36d1ea2c9e7292c69477fa332713c
Details
Ten configs of the form f"split_{X}" where X is in range(10)
Features
{'coref_chains': List[List[List[int]]] # list of clusters, each cluster is a list of mentions, each mention is a… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/litbank_raw.el-raw-28Bta-raw-28Bja-raw-28Bsmall_raw_dataset_for_language_detectionko-raw-28Bwiki_book_corpus_complete_raw_dataset
Dataset Card for "wiki_book_corpus_complete_raw_dataset"
More Information needed
niv2_winogrande_raw
Natural Instructions v2 Winogrande Tasks
Project: https://github.com/allenai/natural-instructions
Data source: DataProvenanceInitiative/niv2_submix_original
Details
This dataset contains all Winogrande examples that were included in the Flan 2022 collection which were orignally published in Super-Natural-Instructions.
The data is copied from the preprocessed Natural Instructions v2 dataset at DataProvenanceInitiative/niv2_submix_original.
These tasks are:… See the full description on the dataset page: https://huggingface.co/datasets/coref-data/niv2_winogrande_raw.