freococo/thahabiorg_pages

❤️ Credits

Original Source

All original books, bibliographic metadata, and online library organization originate from:

Thahabi Shamela Books Team

https://thahabi.org

All intellectual credit for the collection belongs to the Thahabi Shamela Books Team, the respective authors, editors, publishers, and other rights holders.

Dataset Creation

The page extraction, large-scale scraping pipeline, data cleaning, deduplication, validation, Apache Parquet conversion, and dataset packaging were created by:

Dataset Creator: freococo

The dataset creator does not claim ownership of the original books or their contents.

📖 Overview

This dataset contains Arabic books scraped from thahabi.org (Shamela Books) and converted into structured Apache Parquet files for efficient machine learning, natural language processing (NLP), digital humanities, search systems, and academic research.

The dataset contains 28,896 unique books after cross-category deduplication and includes millions of pages of Arabic text together with structured metadata for every book.

The primary objective of this project is long-term preservation, reproducible research, and easier access for educational and non-commercial purposes.

📦 Dataset Structure

This repository contains the page-level dataset of the Thahabi Shamela Books collection.

Each Apache Parquet file stores the text extracted from books within a specific category. Every row represents one page from a single book.

Page Dataset

Example file:

cat_1_masterpages.parquet

Example schema:

Companion Metadata Dataset

The corresponding book metadata (titles, authors, categories, page counts, word counts, and source URLs) is available in a separate repository:

📄 https://huggingface.co/datasets/freococo/thahabiorg_metadata

The two datasets can be joined using the `book_id` field.

📊 Dataset Statistics

Each page in this dataset is linked to a unique book through the book_id field, allowing it to be joined with the companion metadata dataset.

⚠️ Important Notice

This dataset is a structured collection of publicly accessible material originally available from thahabi.org.

The dataset creator does not own or claim ownership of the original books, their text, metadata, or any associated intellectual property.

This repository was created to facilitate:

• —Academic research
• —Natural language processing (NLP)
• —Large language model (LLM) training and evaluation
• —Information retrieval
• —Digital humanities research
• —Educational and non-commercial use

Users are responsible for ensuring that their use of this dataset complies with applicable copyright laws and regulations in their jurisdiction.

If you are a copyright holder and believe any material should not be included in this dataset, please contact the dataset maintainer so the issue can be reviewed promptly.

🔄 Deduplication Process

During the scraping process, cross-category duplicates were found and removed. The original list contained 29,278 book entries. After deduplication, the final dataset contains 28,896 strictly unique books.

<details> <summary><b>Click to view the list of removed duplicate books</b></summary>

</details>

📚 Complete Index of Scraped Books (28,515 Books)

The complete index of all 28,515 unique books is available here:

📄 [README_ALL_BOOKS.md](https://huggingface.co/datasets/freococo/thahabiorg_pages/raw/main/README_ALL_BOOKS.md)

⚖️ Copyright & Fair Use

This dataset is provided for educational, research, preservation, and non-commercial purposes.

The original texts remain the intellectual property of their respective copyright holders.

This repository distributes a structured representation of publicly accessible material to support machine learning, information retrieval, digital humanities, and academic research.

If you are a copyright holder and believe that any content should not be included, please contact the dataset maintainer so that the request can be reviewed and appropriate action taken.

The dataset creator claims no ownership over the original works and gives full credit to the Thahabi Shamela Books Team and the original authors.