decodingchris/clean_squad_classic_v2
Clean SQuAD Classic v2 This is a refined version of the SQuAD v2 dataset. It has been preprocessed to ensure higher data quality and usability for NLP tasks such as Question Answering. Description The Clean SQuAD Classic v2 dataset was created by applying preprocessing steps to the original SQuAD v2 dataset, including: Trimming whitespace: All leading and trailing spaces have been removed from the question field. Minimum question length: Questions with fewer than… See the full description on the dataset page: https://huggingface.co/datasets/decodingchris/clean_squad_classic_v2.
19
1---2dataset_info:3 features:4 - name: id5 dtype: string6 - name: title7 dtype: string8 - name: context9 dtype: string10 - name: question11 dtype: string12 - name: answers13 struct:14 - name: answer_start15 sequence: int3216 - name: text17 sequence: string18 splits:19 - name: train20 num_bytes: 11669687921 num_examples: 13031622 - name: validation23 num_bytes: 1166031924 num_examples: 1187325 download_size: 1769868326 dataset_size: 12835719827configs:28- config_name: default29 data_files:30 - split: train31 path: data/train-*32 - split: validation33 path: data/validation-*34task_categories:35- question-answering36language:37- en38size_categories:39- 100K<n<1M40---41 42## Clean SQuAD Classic v243 44This is a refined version of the [SQuAD v2](https://huggingface.co/datasets/rajpurkar/squad_v2) dataset. It has been preprocessed to ensure higher data quality and usability for NLP tasks such as Question Answering.45 46## Description47 48The **Clean SQuAD Classic v2** dataset was created by applying preprocessing steps to the original SQuAD v2 dataset, including:49- **Trimming whitespace**: All leading and trailing spaces have been removed from the `question` field.50- **Minimum question length**: Questions with fewer than 12 characters were filtered out to remove overly short or uninformative entries.51 52Unlike the [Clean SQuAD v2](https://huggingface.co/datasets/decodingchris/clean_squad_v2) dataset, this dataset does not contain a separate test split. It retains the classic two-way split of **train** and **validation**, following the traditional structure of the original SQuAD v2 dataset.53 54## Dataset Structure55 56The dataset is divided into two subsets:57 581. **Train**: The primary dataset for model training.592. **Validation**: A dataset for hyperparameter tuning and model validation.60 61## Data Fields62 63Each subset contains the following fields:64- `id`: Unique identifier for each question-context pair.65- `title`: Title of the article the context is derived from.66- `context`: Paragraph from which the answer is extracted.67- `question`: Preprocessed question string.68- `answers`: Dictionary containing:69 - `text`: The text of the correct answer(s), if available. Empty for unanswerable questions.70 - `answer_start`: Character-level start position of the answer in the context, if available.71 72## Usage73 74The dataset is hosted on the Hugging Face Hub and can be loaded with the following code:75 76```python77from datasets import load_dataset78 79dataset = load_dataset("decodingchris/clean_squad_classic_v2")80```