Deval025/openlibrary-unified

Open Library Unified Dataset

Description

This dataset contains a unified view of Open Library data, transforming 79GB of raw, semi-structured data into a 3.65GB Parquet file that preserves relationships between books, authors, and editions.

Data Structure

The dataset contains over 50 million records with the following key fields:

• —edition_id: Unique identifier for book editions
• —edition_title: Title of the edition
• —isbn: ISBN identifier when available
• —publisher: Publisher name
• —publish_date: Publication date
• —work_id: Identifier linking to the abstract work
• —work_title: Title of the work
• —author_id: Identifier for the author
• —author_name: Name of the author

Partitioning

The dataset is organized by publication decade:

• —pre_1800: Publications before 1800
• —1800s-1990s: Decade-by-decade partitions
• —2000_onwards: Publications from 2000 and later
• —unknown_decade: Publications with unidentifiable dates

Usage Examples

from datasets import load_dataset

# Load the entire dataset
dataset = load_dataset("Deval025/openlibrary-unified")

# Load a specific decade
dataset_1950s = load_dataset("Deval025/openlibrary-unified", "1950s")

# Basic filtering example
shakespeare_works = dataset["train"].filter(lambda x: "Shakespeare" in x["author_name"])