kasys/open-source-scientific-documents

Open-Source Scientific Documents

This dataset contains approximately 100,000 open scientific PDF documents packaged as a shared retrieval corpus. Train, validation, and test query sets are expected to reference document_id values from this single corpus rather than using separate document splits.

Sources

• —ACL: 20,845 PDFs
• —Biology: 22,000 PDFs
• —Engineering: 22,000 PDFs
• —Medicine: 22,000 PDFs
• —Physics: 21,667 PDFs

The source folders preserve the project corpus identities: ACL computational linguistics papers, arXiv Physics papers, arXiv Engineering papers, PMC OA Biology papers, and PMC OA Medical/Clinical Research papers. The selection and license filtering are performed by the existing project download pipeline before packaging.

Licensing

All documents are licensed with CC0 or CC-BY, These terms permit redistribution and scientific use of the documents, with CC BY additionally requiring appropriate attribution.

Format

PDFs are stored in uncompressed TAR shards under data/<SOURCE>/shard-xxxxx.tar using a WebDataset-compatible layout. Each example contains:

DOCUMENT_ID.pdf
DOCUMENT_ID.json

The JSON sidecar includes document_id, source, original filename/path, size, SHA-256 checksum, shard location, and available bibliographic metadata such as title, year, DOI, and document-level license. The global metadata.parquet has one row per packaged PDF with:

document_id, source, original_filename, original_relative_path, shard, member_path, size_bytes, sha256, license, title, year, doi

duplicates.parquet records exact duplicate content by SHA-256. Duplicate files are preserved in the shards; duplicate rows identify the canonical and duplicate document IDs. This build records 108,512 PDFs, 108,512 unique file contents, and 0 duplicate rows.

Streaming Shards

from datasets import load_dataset

dataset = load_dataset(
    "webdataset",
    data_files={
        "ACL": "hf://datasets/kasys/open-source-scientific-documents/data/ACL/*.tar",
    },
    split="ACL",
    streaming=True,
)

for example in dataset:
    pdf_bytes = example["pdf"]
    metadata = example["json"]
    break

Locate One PDF

import io
import tarfile

import pandas as pd
from huggingface_hub import hf_hub_download

repo_id = "kasys/open-source-scientific-documents"
document_id = "ACL_..."
metadata = pd.read_parquet("hf://datasets/" + repo_id + "/metadata.parquet")
row = metadata.loc[metadata.document_id == document_id].iloc[0]

shard_path = hf_hub_download(repo_id=repo_id, repo_type="dataset", filename=row.shard)
with tarfile.open(shard_path, "r") as tar:
    pdf_bytes = tar.extractfile(row.member_path).read()

Limitations

Some source records have incomplete bibliographic metadata. Missing license, title, year, DOI, or author fields are left null rather than inferred. TAR shards are uncompressed because PDF files are already compressed.