decodingchris/clean_squad_v1
Clean SQuAD v1 This is a refined version of the SQuAD v1 dataset. It has been preprocessed to ensure higher data quality and usability for NLP tasks such as Question Answering. Description The Clean SQuAD v1 dataset was created by applying preprocessing steps to the original SQuAD v1 dataset, including: Trimming whitespace: All leading and trailing spaces have been removed from the question field. Minimum question length: Questions with fewer than 12 characters… See the full description on the dataset page: https://huggingface.co/datasets/decodingchris/clean_squad_v1.
127
1---2dataset_info:3 features:4 - name: id5 dtype: string6 - name: title7 dtype: string8 - name: context9 dtype: string10 - name: question11 dtype: string12 - name: answers13 struct:14 - name: answer_start15 sequence: int3216 - name: text17 sequence: string18 splits:19 - name: train20 num_bytes: 7930163121 num_examples: 8758822 - name: validation23 num_bytes: 523963124 num_examples: 528525 - name: test26 num_bytes: 523300627 num_examples: 528528 download_size: 1980932629 dataset_size: 8977426830configs:31- config_name: default32 data_files:33 - split: train34 path: data/train-*35 - split: validation36 path: data/validation-*37 - split: test38 path: data/test-*39task_categories:40- question-answering41language:42- en43size_categories:44- 10K<n<100K45---46 47## Clean SQuAD v148 49This is a refined version of the [SQuAD v1](https://huggingface.co/datasets/rajpurkar/squad) dataset. It has been preprocessed to ensure higher data quality and usability for NLP tasks such as Question Answering.50 51## Description52 53The **Clean SQuAD v1** dataset was created by applying preprocessing steps to the original SQuAD v1 dataset, including:54- **Trimming whitespace**: All leading and trailing spaces have been removed from the `question` field.55- **Minimum question length**: Questions with fewer than 12 characters were filtered out to remove overly short or uninformative entries.56- **Balanced validation and test sets**: The validation set from the original SQuAD dataset was split 50-50 into new validation and test sets.57 58This preprocessing ensures that the dataset is cleaner and more balanced, making it suitable for training and evaluating machine learning models on Question Answering tasks.59 60## Dataset Structure61 62The dataset is divided into three subsets:63 641. **Train**: The primary dataset for model training.652. **Validation**: A dataset for hyperparameter tuning and model validation.663. **Test**: A separate dataset for evaluating final model performance.67 68## Data Fields69 70Each subset contains the following fields:71- `id`: Unique identifier for each question-context pair.72- `title`: Title of the article the context is derived from.73- `context`: Paragraph from which the answer is extracted.74- `question`: Preprocessed question string.75- `answers`: Dictionary containing:76 - `text`: The text of the correct answer(s).77 - `answer_start`: Character-level start position of the answer in the context.78 79## Usage80 81The dataset is hosted on the Hugging Face Hub and can be loaded with the following code:82 83```python84from datasets import load_dataset85 86dataset = load_dataset("decodingchris/clean_squad_v1")87```