Datasetpublic

LuminScience/LuminBench-Nano-ESMC

LuminBench Nano ESMC Full Open Reservoir v2 This is the complete decontaminated 70%-identity representative reservoir for Lumin-Science/LuminBench-Nano-ESMC. It is organized as immutable, SHA-ordered Parquet shards so each run can download only the smallest deterministic prefix required by its training budget. License and source terms Lumin Science's original database selection, arrangement, decontamination ledger, packing, and metadata are offered under CC BY-SA… See the full description on the dataset page: https://huggingface.co/datasets/LuminScience/LuminBench-Nano-ESMC.

sourceHugging Facecc-by-sa-4.0updated 13d agoView on Hugging Face

1likes1.2kdownloads

Dataset Card

LuminBench Nano ESMC Full Open Reservoir v2

This is the complete decontaminated 70%-identity representative reservoir for `Lumin-Science/LuminBench-Nano-ESMC`. It is organized as immutable, SHA-ordered Parquet shards so each run can download only the smallest deterministic prefix required by its training budget.

License and source terms

Lumin Science's original database selection, arrangement, decontamination ledger, packing, and metadata are offered under CC BY-SA 4.0. Third-party sequence records retain their source terms:

Path	Direct source	Governing terms
`train/uniref90/`, `validation/uniref90/`	UniRef90 2023_02	CC BY 4.0
`train/mgnify/`, `validation/mgnify/`	MGnify Protein DB 2023_02	EMBL-EBI Terms of Use plus applicable original-owner rights; not relicensed by Lumin Science
`train/omg_img/`, `validation/omg_img/`	JGI/IMG records from `tattabio/OMG`	CC BY-SA 4.0

Redistribution notes and modifications are preserved in LICENSE_AND_ATTRIBUTION.md and SOURCE_PROVENANCE.json.

Decontamination

The exclusion union was built from the exact source data used by these evaluations:

Evaluation data/source	Corresponding evaluation task
RCSB Protein Data Bank snapshot dated 2024-02-28, using the ESMC paper contact protocol	Long-range contact prediction, reported as precision at L (P@L)
TAPE remote-homology data, derived from SCOP fold classes	Remote-homology fold classification
TAPE secondary-structure data, with CB513 as the primary test set	Residue-level three-class secondary-structure prediction
TorchDrug EnzymeCommission sequence adaptation	Multilabel Enzyme Commission function prediction; retained as a quarantined diagnostic
DeepLoc 2.0 official five partitions	Multilabel subcellular-localization prediction
PEER HumanPPI	Human protein-protein-interaction prediction; retained as a quarantined diagnostic
FLIP2 Hydrophobic Core low-to-high split	Protein-fitness regression under an engineering distribution shift
CATH v4.4 S20 domains and classifications	CATH remote structural-domain retrieval
CAFA5 input bundle and official final evaluation	Molecular-function transfer on the MF no-knowledge, 30%-identity hard set; blocked for scoring but still protected
PRING	Human PPI prediction with node- and 30%-cluster-disjoint splits
FLIP2 public split collection	Multi-landscape protein-engineering shift evaluation
RosettaCommons MegaScale	Family-held-out mutation-stability / delta-delta-G prediction
CAID2 and CAID3 Disorder-PDB references	Temporal residue-level intrinsic-disorder prediction

All sequences used to construct the listed task populations are included in the protected union, including task fitting, validation, retrieval gallery, final test, and the blocked CAFA candidate population. The full representative reservoir is screened with MMseqs2 at 30% sequence identity and 80% query plus target coverage. Exact SHA-256 matches are excluded independently. The released validation union is excluded from every training source arm.

Download only what a run needs

bash

git clone https://github.com/Lumin-Science/LuminBench-Nano-ESMC
cd LuminBench-Nano-ESMC
uv sync --frozen
uv run --frozen python scripts/download_data.py \
  --repo-id LuminScience/LuminBench-Nano-ESMC \
  --revision <immutable-release-commit> \
  --training-samples 5376000 \
  --cache-root data/cache/full-open-v2 \
  --output-root data/processed/run-prefix

The command fetches manifest.json, converts the requested total sample count to per-source requirements using the 36:11:54 mixture, downloads the minimum whole-shard prefix for each source plus every validation shard, verifies the checksums, and materializes the existing mmap training layout. Training is local; it does not make row-level network requests.

Parquet schema

Column	Type	Meaning
`sequence`	string	normalized amino-acid sequence
`sha256`	string	SHA-256 of the ASCII sequence
`length`	int32	residue count

See the GitHub processing report and manifest.json for full lineage, measured counts, rejection accounting, shard hashes, and the complete build recipe.

Verified release measurements

—Training representatives: 665,970,495
—Training residues: 151,304,238,405
—Training Parquet shards: 565
—Validation representatives: 12,288
—Validation residues: 3,182,651
—Validation Parquet shards: 3
—Train plus validation compressed bytes: 109,661,312,410
—Evaluation-query union: 317,000 sequences
—Homology-excluded representative digests: 73,826,953
—Manifest SHA-256: fe1ac0657085ab19fe6f56786006e9eb004ca66bc6c5b81dfd8e6bc3dcfda6ff