GodotCN/science-datalake

<p align="center"> <img src="https://raw.githubusercontent.com/J0nasW/science-datalake/main/sdl_banner.jpg" alt="Science Data Lake" width="100%"> </p>

<p align="center"> <a href="https://arxiv.org/abs/2603.03126"><img src="https://img.shields.io/badge/arXiv-2603.03126-b31b1b" alt="arXiv"></a> <a href="https://github.com/J0nasW/science-datalake"><img src="https://img.shields.io/badge/GitHub-Repository-181717?logo=github" alt="GitHub"></a> <a href="https://doi.org/10.57967/hf/7850"><img src="https://img.shields.io/badge/DOI-10.57967%2Fhf%2F7850-blue" alt="DOI"></a> <a href="https://github.com/J0nasW/science-datalake/blob/main/SCHEMA.md"><img src="https://img.shields.io/badge/LLM--Ready-SCHEMA.md-purple" alt="LLM-Ready"></a> <a href="https://x.com/JonasHW"><img src="https://img.shields.io/badge/Follow-%40Jonas_H_W-black?logo=x" alt="Follow on X"></a> <a href="https://wilinski.me"><img src="https://img.shields.io/badge/Author-wilinski.me-orange" alt="Author website"></a> </p>

Science Data Lake

A unified, portable science data lake integrating 7 scholarly datasets (~525 GB Parquet) with cross-dataset DOI normalization, 13 scientific ontologies (1.3M terms), and a reproducible ETL pipeline.

Note: One additional source (Semantic Scholar S2AG) is supported by the pipeline but is not redistributed here due to its API terms of service. See Not Included in This Upload below.

What's Unique

This dataset enables queries that are impossible with any single source:

-- "Top disruptive papers with open-source code, checking for retractions"
SELECT doi, title, year,
       sciscinet_disruption,      -- from SciSciNet
       oa_cited_by_count,         -- from OpenAlex
       has_pwc,                   -- from Papers With Code
       has_retraction             -- from Retraction Watch
FROM unified_papers
WHERE has_pwc AND sciscinet_disruption > 0.5
ORDER BY oa_cited_by_count DESC
LIMIT 20

Datasets Included

Ontology Licenses

Snapshot Dates

Each source was downloaded at a specific point in time:

All snapshots can be refreshed using the update pipeline — see below.

Not Included in This Upload

The following source is supported by the full pipeline (GitHub) but is not redistributed here due to its API terms of service:

After downloading S2AG locally, run the full pipeline to integrate it.

Key Tables

unified_papers (293M rows)

The headline table: one row per unique DOI, joining all sources.

Note: The S2AG columns (s2ag_corpusid, s2ag_citationcount, s2ag_influentialcitationcount, s2ag_isopenaccess, has_s2ag) are present in the uploaded file but will contain NULL/FALSE values unless S2AG has been integrated locally. All other columns (including has_patent from Reliance on Science) are fully populated.

topic_ontology_map

Maps OpenAlex's 4,516 topics to terms in 13 scientific ontologies via embedding-based semantic similarity (BGE-large-en-v1.5, 1024-dim) + exact matching for large ontologies (MeSH, ChEBI, NCIT). 16,150 mappings covering 99.8% of topics. Columns include similarity (cosine, 0-1) and match_type (label/synonym/exact) for quality filtering.

ontology_bridges

Cross-ontology links discovered via shared external IDs (UMLS, Wikidata, MESH, etc.).

Usage with DuckDB

Option 1: Pre-built database file (recommended)

This repository includes a ready-to-use DuckDB database file (datalake.duckdb, 274 KB) with 145 SQL views pre-configured to read directly from HuggingFace. Download just this one file and query all 7 datasets immediately — no pipeline setup required.

import duckdb

con = duckdb.connect()
con.execute("INSTALL httpfs; LOAD httpfs;")
con.execute("ATTACH 'hf://datasets/J0nasW/science-datalake/datalake.duckdb' AS lake")

# Query using familiar schema.table syntax
df = con.execute("""
    SELECT doi, title, year, sciscinet_disruption, oa_cited_by_count
    FROM lake.xref.unified_papers
    WHERE sciscinet_disruption IS NOT NULL
    ORDER BY sciscinet_disruption DESC
    LIMIT 100
""").df()

# Cross-source joins work out of the box
con.execute("""
    SELECT t.display_name AS topic, o.ontology, o.term_name, o.similarity
    FROM lake.xref.topic_ontology_map o
    JOIN lake.openalex.topics t ON t.id = o.topic_id
    WHERE o.similarity >= 0.85
    ORDER BY o.similarity DESC
    LIMIT 20
""").df()

Option 2: Direct Parquet queries

You can also query individual Parquet files directly without the database file:

import duckdb

con = duckdb.connect()
con.execute("INSTALL httpfs; LOAD httpfs;")

df = con.execute("""
    SELECT doi, title, year, sciscinet_disruption, oa_cited_by_count
    FROM 'hf://datasets/J0nasW/science-datalake/xref/unified_papers/*.parquet'
    WHERE sciscinet_disruption IS NOT NULL
    ORDER BY sciscinet_disruption DESC
    LIMIT 100
""").df()

Keeping the Data Current

The full pipeline supports incremental updates. When upstream sources release new snapshots:

# Update a single dataset
python scripts/datalake_cli.py update openalex

# Update all datasets and rebuild cross-reference tables
python scripts/datalake_cli.py update
python scripts/materialize_unified_papers.py

See the GitHub repository for full pipeline documentation.

LLM & AI Agent Integration

This data lake ships with [SCHEMA.md](https://github.com/J0nasW/science-datalake/blob/main/SCHEMA.md) — a structured reference file optimized for LLM-based coding agents (Claude Code, Cursor, Copilot, etc.). It contains every table, column, type, join strategy, and performance tier in a format that AI agents can use to write correct DuckDB SQL without prior schema knowledge.

Point your AI assistant at SCHEMA.md and ask it to query across all 7 hosted datasets and 13 ontologies using natural language.

Building the Full Instance (All 8 Sources)

Clone the GitHub repository and run the pipeline to integrate all sources including S2AG:

git clone https://github.com/J0nasW/science-datalake
cd science-datalake
python scripts/datalake_cli.py download --all
python scripts/datalake_cli.py convert --all
python scripts/create_unified_db.py
python scripts/materialize_unified_papers.py

Citation

If you use the Science Data Lake, please cite the paper:

@article{wilinski2026sciencedatalake,
  title   = {The Science Data Lake: A Unified Open Infrastructure Integrating
             293 Million Papers Across Eight Scholarly Sources with
             Embedding-Based Ontology Alignment},
  author  = {Wilinski, Jonas},
  journal = {arXiv preprint arXiv:2603.03126},
  year    = {2026},
  url     = {https://arxiv.org/abs/2603.03126}
}

Dataset DOI: 10.57967/hf/7850

License

This dataset aggregates multiple sources, each with its own license. Users must comply with the most restrictive license applicable to the sources they use.