Datasetpublic

Mearman/OpenAlex

OpenAlex Snapshot Mirror of the OpenAlex scholarly metadata snapshot — a free, open catalogue of 250M+ scholarly works, 100M+ authors, and related entities. Hosted on HuggingFace via Xet for content-addressable deduplication. Source: s3://openalex (public, anonymous S3 bucket) Dataset subsets Each entity type is a separate subset (config). For each entity, there is: One __source subset containing the raw .jsonl.gz source files One subset per extracted… See the full description on the dataset page: https://huggingface.co/datasets/Mearman/OpenAlex.

sourceHugging Facecc0-1.0updated 3mo agoView on Hugging Face

5likes53kdownloads

Dataset Card

OpenAlex Snapshot

Mirror of the OpenAlex scholarly metadata snapshot — a free, open catalogue of 250M+ scholarly works, 100M+ authors, and related entities.

Hosted on HuggingFace via Xet for content-addressable deduplication.

Source: s3://openalex (public, anonymous S3 bucket)

Dataset subsets

Each entity type is a separate subset (config). For each entity, there is:

—One __source subset containing the raw .jsonl.gz source files
—One subset per extracted relationship table (e.g. works__main, works__abstracts, authors__affiliations)

Entity	Description
works	Scholarly works (papers, datasets, etc.)
authors	Authors of scholarly works
institutions	Universities, research orgs
publishers	Academic publishers
sources	Journals, repositories, conferences
awards	Grant/funding awards
concepts	Legacy concept taxonomy (Wikidata)
topics	Topic taxonomy
domains	Top-level topic domains
fields	Topic fields
subfields	Topic subfields
funders	Funding organisations
keywords	Machine-learning keywords
continents	Continents
countries	Countries
languages	Languages
licenses	Licences
sdgs	Sustainable Development Goals
institution-types	Institution types
source-types	Source types
work-types	Work types

Data format

Each shard is a gzip-compressed JSON Lines file at:

data/{entity}/updated_date=YYYY-MM-DD/part_XXXX.jsonl.gz

The .jsonl.gz extension allows the HuggingFace dataset viewer to detect the inner format automatically. On S3, files are named part_XXXX.gz; the download pipeline renames them on save.

Each line is a JSON object representing one entity record. Fields vary by entity type. See the OpenAlex data model for field definitions.

Extracted Parquet tables

The sync pipeline extracts relationship tables from each entity into Parquet files. Each entity has a main table (scalar attributes, one row per entity) plus separate tables for each list/dict-valued field:

data/{entity}/
  updated_date=YYYY-MM-DD/part_XXXX.jsonl.gz       # source data
  main/
    {entity}__updated_date=...__part_XXXX.parquet   # scalar attributes
  {relationship_type}/
    {entity}__updated_date=...__part_XXXX.parquet   # one edge table per list/dict field

The dataset viewer provides one subset per entity+relationship combination (e.g. works__main, works__abstracts, authors__affiliations) and one __source subset per entity for the raw JSONL.

Example: Work record fields

id, doi, title, display_name, publication_year, type, language, authorships, concepts, topics, keywords, cited_by_count, referenced_works, related_works, locations, open_access, funders, awards, mesh, sustainable_development_goals, counts_by_year, updated_date, and more.

Sync and extraction pipeline

The sync/ directory contains a Python pipeline for downloading from S3 and extracting relationship tables to Parquet:

bash

# Full sync (all entities)
python3 -m sync

# Limit to one entity
python3 -m sync --entity works

# Split extraction across machines
python3 -m sync --slice-index 0 --slice-total 2   # machine 1
python3 -m sync --slice-index 1 --slice-total 2   # machine 2

License

OpenAlex data is released under CC0 1.0 Universal. See the OpenAlex terms for details.