yash3056/wikipedia-20250721

Dataset Card for Wikipedia-20250721

Dataset Summary

Wikipedia-20250721 is a cleaned, preprocessed version of the English Wikipedia “pages-articles” dump (July 21, 2025) converted into Parquet format and published on Hugging Face. It contains article titles and full article text suitable for large-scale language model pretraining or downstream NLP tasks.

• —Homepage: https://huggingface.co/datasets/yash3056/wikipedia-20250721
• —Dataset license: CC BY-SA 4.0
• —Languages: English
• —Size: \~10.5 GB (Parquet)
• —Number of examples: \~6,984,754 articles

Supported Tasks and Use Cases

• —Language Modeling / Pretraining: Raw text for LLM pretraining.
• —Text Classification / Regression: Fine-tune on article text for various classification tasks.
• —Information Retrieval / QA: Source text for retrieval-based QA systems.
• —Summarization / Generation: Input documents for summarization and text generation.

Dataset Structure

The dataset is provided as a single Parquet file wikipedia_20250721.parquet with the following columns:

Data Splits

This dataset is provided as a single split (train) containing all cleaned articles.

Dataset Creation

1. 1.Source: Official English Wikipedia XML dump (July 21, 2025) from https://dumps.wikimedia.org/enwiki/latest/.
2. 2.Extraction: Used WikiExtractor with --no-templates and post-filtering to remove empty or stub articles.
3. 3.Cleaning: Dropped rows with missing or empty text fields.
4. 4.Conversion: Converted cleaned CSV to Parquet via pandas + pyarrow.
5. 5.Upload: Pushed Parquet file to Hugging Face Hub under yash3056/wikipedia-20250721.

Usage Example

from datasets import load_dataset

dataset = load_dataset("yash3056/wikipedia-20250721")
print(dataset)
# DatasetDict({ 'train': Dataset({ ... }) })

# Access examples
print(dataset['train'][0])
# { 'id': '1', 'url': 'https://en.wikipedia.org/wiki?curid=1', 'title': 'AccessibleComputing', 'text': 'Accessible computing is ...' }

Citation

If you use this dataset, please cite:

@misc{yash3056_wikipedia_20250721,
  author = {Yash and Wikipedia contributors},
  title = {Wikipedia-20250721: Cleaned English Wikipedia Articles in Parquet Format},
  year = {2025},
  howpublished = {\url{https://huggingface.co/datasets/yash3056/wikipedia-20250721}}
}

License

This dataset is released under Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0). You are free to share and adapt under the terms of the license.