sayshara/openlibrary

📚 OpenLibrary Parquet Mirror

An up‑to‑date, stream‑ready mirror of the official OpenLibrary public data dumps, converted to snappy‑compressed Parquet.

Source: https://openlibrary.org/developers/dumps

💾 Repository organization

openlibrary/
├── authors/   # authors_0.parquet, authors_1.parquet, … (≈ 1-2 GB each)
├── editions/  # editions_0.parquet, editions_1.parquet, …
└── works/     # works_0.parquet, works_1.parquet, …

🏃 Quick start

from datasets import load_dataset

# stream authors only
authors = load_dataset("sayshara/openlibrary", "authors", split="train", streaming=True)
for i, rec in enumerate(authors):
    if i == 5:
        break
    print(rec["name"], rec.get("birth_date"))

📝 Schema notes

• —Missing columns for a given record are stored as NULL.
• —Top‑level JSON objects from the dump are flattened; nested structures are preserved as JSON‑encoded strings.
• —The raw JSON produced by OpenLibrary loosely follows per‑entity schemata maintained in the `openlibrary‑client` repo:

• —`authors.schema.json` → governs objects from `ol_dump_authors_latest.txt.gz`.
• —`editions.schema.json` → governs objects from `ol_dump_editions_latest.txt.gz`.
• —`works.schema.json` → governs objects from `ol_dump_works_latest.txt.gz`.

📜 License

OpenLibrary dedicates its catalog data to the public domain under the CC0 1.0 Universal (Public‑Domain Dedication) license. 

For more information, see: https://openlibrary.org/help/faq/using

Because this Parquet mirror is a direct transformation of that catalog, all files in this dataset are likewise released under CC0 1.0. You may copy, modify, redistribute, and use them for any purpose without permission or attribution. (Attribution to OpenLibrary/Internet Archive is appreciated but not legally required.)

Maintainer • @sayshara