Datasetpublic

khairi/uniref50-replay-mix-v1

uniref50-replay-mix-v1 Stage-1 continued-pretraining corpus for eshmun-vocab: protein sequences (UniRef50) mixed with a general/biomedical/math/code text replay slice, so a vocab-extended LLM (e.g. khairi/qwen3-0.6b-protein-vocab-v0) learns protein-sequence statistics without catastrophically forgetting its pretrained language ability. Design rationale and target ratios: see docs/pretrain-dataset-mix.md in the eshmun-vocab repo. Objective: plain next-token prediction. Every row… See the full description on the dataset page: https://huggingface.co/datasets/khairi/uniref50-replay-mix-v1.

sourceHugging Faceupdated 3mo agoView on Hugging Face

0likes134downloads

Dataset Card

uniref50-replay-mix-v1

Stage-1 continued-pretraining corpus for eshmun-vocab: protein sequences (UniRef50) mixed with a general/biomedical/math/code text replay slice, so a vocab-extended LLM (e.g. khairi/qwen3-0.6b-protein-vocab-v0) learns protein-sequence statistics without catastrophically forgetting its pretrained language ability. Design rationale and target ratios: see docs/pretrain-dataset-mix.md in the eshmun-vocab repo.

Objective: plain next-token prediction. Every row is a single raw document — no QA pairs, no instruction/chat formatting.

Fields

Field	Type	Description
`entry`	string	Source identifier, e.g. `UniRef50_A0A5C8BMA9` for a protein sequence, `pubmed:00062986` for a PubMed abstract. Useful for provenance/filtering by source, not used in training.
`content`	string	The document text. Protein sequences are wrapped as `<protein>...</protein>`; text-replay documents (general/biomed/math/code) are plain text.

Splits

Split	Examples	Size
train	10,680,703	5.59 GB (4.87 GB compressed)

Note: earlier versions of this dataset also had valid/test splits (a held-out slice of UniRef50 sequences only, no replay). Those parquet files are still physically present in the repo but are no longer declared in the dataset config, so load_dataset("khairi/uniref50-replay-mix-v1") now only returns train — a regenerated valid/test split may be added back in a future version.

Usage

python

from datasets import load_dataset

ds = load_dataset("khairi/uniref50-replay-mix-v1", split="train")

Compatibility note: eshmun_vocab's training scripts (train_clm_lc.py / train_clm_hf.py) default to reading a text column. This dataset's text field is named content, not text — rename/map the column (e.g. ds.rename_column("content", "text")) before passing it to those scripts, or pass whatever --text-column-equivalent option applies.

Composition

Built by mixing the full 10M-sequence UniRef50 sample with a replay slice of HuggingFaceFW/fineweb-edu (general), slinusc/PubMedAbstractsSubset (biomedical), HuggingFaceTB/finemath (math), and bigcode/starcoderdata (code), targeting an 85%/15% domain/replay token split (see docs/pretrain-dataset-mix.md for the full per-category breakdown and reasoning). This version's 10,680,703 rows is a full build (earlier versions of this repo were a partial/stale push of ~779K rows). The exact realized per-source token counts in this specific upload haven't been independently re-verified against the design doc's targets — treat that doc as the intended ratios, not a guarantee of this upload's exact composition.

License

Mixed-source dataset — each source's own license/terms apply to its portion of the content: UniProt/UniRef (CC BY 4.0), PubMed abstracts (NLM terms), and the respective licenses of FineWeb-Edu, FineMath, and starcoderdata (permissively-licensed code only). No single blanket license covers the whole dataset.