datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
squad_v2
Dataset Card for SQuAD 2.0
Dataset Summary
Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.
SQuAD 2.0 combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers… See the full description on the dataset page: https://huggingface.co/datasets/rajpurkar/squad_v2.squad_v2squad_v2 SQuAD2.0 combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers
to look similar to answerable ones. To do well on SQuAD2.0, systems must not only answer questions when possible, but
also determine when no answer is supported by the paragraph and abstain from answering.squadv2trainsquad_kor_v2KorQuAD 2.0 is a Korean question and answering dataset consisting of a total of 100,000+ pairs. There are three major differences from KorQuAD 1.0, which is the standard Korean Q & A data. The first is that a given document is a whole Wikipedia page, not just one or two paragraphs. Second, because the document also contains tables and lists, it is necessary to understand the document structured with HTML tags. Finally, the answer can be a long text covering not only word or phrase units, but paragraphs, tables, and lists. As a baseline model, BERT Multilingual is used, released by Google as an open source. It shows 46.0% F1 score, a very low score compared to 85.7% of the human F1 score. It indicates that this data is a challenging task. Additionally, we increased the performance by no-answer data augmentation. Through the distribution of this data, we intend to extend the limit of MRC that was limited to plain text to real world tasks of various lengths and formats.squad-v2-dutch
SQuAD v2 Dutch (MTEB retrieval format)
Dutch general-knowledge (Wikipedia) retrieval task. Given a Dutch question, retrieve the relevant Wikipedia passage from the corpus. Built from the validation split, answerable questions only.
Reformatted into MTEB retrieval format from yhavinga/squad_v2_dutch, a machine translation of SQuAD v2.
License: CC BY-SA 4.0.
squad-kor-v2
KorQuAD 2.1
https://huggingface.co/datasets/squad_kor_v2
https://github.com/korquad/korquad.github.io/tree/master/dataset/KorQuAD_2.1
소개
KorQuAD 2.0: KorQuAD 1.0의 20,000+ QA를 포함하여, 총 100,000+ 개로 구성된 한국어 MRC 데이터셋.
KorQuAD 2.1: 47,957 개의 Wikipedia article에 대해 102,960 개의 QA로 구성된 한국어 MRC 데이터셋. KorQuAD 2.0 중 HTML 태그의 속성이 완벽하게 제거되지 않은 오류를 수정한 버전.
qa count
train
83,486
dev
10,165
102,960
KorQuAD 1.0과의 차이
context는 paragraphs가 아닌, table과 list가 포함된 전체… See the full description on the dataset page: https://huggingface.co/datasets/twodigit/squad-kor-v2.squad_v2triviaqa-hotpotqa-nq-squad-msmarco-hard-neg-sft4b-doc4096-seq1024-v2long_squad_v2squad_es_v2automatic translation of the Stanford Question Answering Dataset (SQuAD) v2 into Spanishsquad_v2_codex_glue_cnn_dailymail_llama1b_modifiedlong_squad_v2
Dataset Card for long_squad_v2
long_squad_v2 is a long-context question answering dataset based on the SQuAD v2 format. It was constructed by concatenating multiple SQuAD v2 contexts to significantly increase the average document length, enabling training and evaluation of models on long-range understanding and sparse answer retrieval tasks.
Dataset Details
Uses
To load the dataset using the 🤗 Datasets library:
from datasets import load_dataset
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/huutuan/long_squad_v2.squad_v2squad-augmented-v2squad-nl-v2.0
SQuAD-NL v2.0 [translated SQuAD / XQuAD]
SQuAD-NL v2.0 is a translation of The Stanford Question Answering Dataset (SQuAD) v2.0.
Since the original English SQuAD test data is not public, we reserve the same documents that were used for XQuAD for testing purposes. These documents are sampled from the original dev data split. The English data is automatically translated using Google Translate (February 2023) and the test data is manually post-edited.
This version of SQuAD-NL also… See the full description on the dataset page: https://huggingface.co/datasets/GroNLP/squad-nl-v2.0.squad-v2-closed-corpus-v1
SQuAD 2.0 Closed Wikipedia Corpus v1
Full plaintext for 452 Wikipedia articles drawn from the SQuAD 2.0 source corpus (477 unique articles total; 25 dev articles held back for grading).
Dataset configs
Config
File
Use
default
train.parquet
Dataset Viewer — title, pageid, text length, 1000-char preview
full
corpus_v1.parquet
Full article text via load_dataset(..., name="full")
—
corpus_v1.jsonl
Pinned release file for offset validation (download… See the full description on the dataset page: https://huggingface.co/datasets/bbasavar/squad-v2-closed-corpus-v1.squadv2validationsquadv2_4_5_1234squad_v2_svSQuAD_v2_sv is a Swedish version of SQuAD2.0. Translation was done automatically by using Google Translate API but it is not so straightforward because;
1. the span which determines the start and the end of the answer in the context may vary after translation,
2. tne translated context may not contain the translated answer if we translate both independently.
More details on how to handle these will be provided in another blog post.squad_v2_french_translatedUsing Google Translation, we have translated SQuAD 2.0 dataset into multiple languages.
Here is the translated dataset of SQuAD 2.0 in French language.
Shared by Pragnakalp Techlabs
squadv2_4_1_2345harrypotter_squad_v2_2
Dataset Summary
Contains 15 Harry Potter trivia questions in Squadv2 format, 3 of which are unanswerable.
Model Performance
Test Notebook
Model
exact
f1
Albert Base (twmkn9/albert-base-v2-squad2)
46.6667
46.6667
Albert XXLarge (ahotrod/albert_xxlargev1_squad2_512)
66.6667
66.6667
chinese-squadv2English | 中文
Dataset Card for Chinese SQuAD 2.0 (revised, bilingual)
Dataset Description
This is a revised and extended version of the Chinese translation of SQuAD 2.0,
originally machine-translated by
ChineseSquad. Like SQuAD 2.0 it
contains both answerable and unanswerable questions and is designed for Chinese
extractive reading comprehension / question answering.
Compared with the previous release of chinese-squadv2, this version:
Adds the original English… See the full description on the dataset page: https://huggingface.co/datasets/real-jiakai/chinese-squadv2.squad-pairs-hard-neg-reasoning-embedding-modified-SFT-4B-doc4096-seq1024-v2-parts-0-1squad_v2.005-squad_v2harrypotter_squad_v2squad_v2_modified_for_t5_qgsquad_v2_factuality_v1
squad_v2_factuality_v1
This dataset is derived from "squad_v2" training "context" with the following steps.
NER is run to extract entities.
Lexicon of person's name, date, organisation name and location are collected.
20% of the time, one of the text attribute (person's name, date, organisation name and location) is randomly replaced. For consistency of context, all other place with the same name is also replaced.
Purpose of the Dataset
The purpose of this dataset… See the full description on the dataset page: https://huggingface.co/datasets/kenhktsui/squad_v2_factuality_v1.