alegendaryfish/CodonTranslator-data

1---2license: cc-by-4.03pretty_name: CodonTranslator Data4task_categories:5  - text-generation6tags:7  - biology8  - dna9  - codon-optimization10  - protein-conditioned-generation11size_categories:12  - 10M<n<100M13---14 15# CodonTranslator Data16 17This repository contains the final public training-data release used for CodonTranslator.18 19## Contents20 21- `train/`: representative-only training shards22- `val/`: representative-only validation shards23- `test/`: representative-only held-out test shards24- `embeddings_v2/`: precomputed species conditioning embeddings used in training25- `_work/final_representative_counts.json`: final released split sizes26- `_work/split_report.json`: split audit report27- `_work/mmseqs_manifest.json`: MMseqs version and clustering parameters28- `_work/cluster_split.parquet`: cluster-level split assignments29- `_work/seq_cluster.parquet`: MMseqs cluster assignments30- `_work/seq_split.parquet`: split assignments before representative selection31 32## Split definition33 34The public `data_v3` split was rebuilt from `data_v2` with the following rules:35 36- MMseqs clustering in **protein space**37- test holdout by **binomial species**38- validation split from **seen species but unseen clusters**39- representative-only parquet outputs, one retained row per released representative sequence40 41Mixed seen/held-out protein clusters and exact-protein leakage cases are removed on the seen side before final release.42 43## Final released split sizes44 45- `train = 36,888,301`46- `val = 373,637`47- `test = 331,455`48 49All three released splits satisfy:50 51- exact protein overlap `train/val = 0`52- exact protein overlap `train/test = 0`53- test species not seen in train/val54- representatives-only rows, so `rows == unique_seq_id`55 56## Parquet schema57 58Each released parquet shard contains these columns:59 60- `RefseqID`61- `protein_refseq_id`62- `protein_seq`63- `cds_DNA`64- `taxon`65- `shard`66 67## Embeddings68 69`embeddings_v2/` is the exact species embedding store used in training. It contains:70 71- `species_vocab.json`72- `species_index.json`73- `species_tok_emb.bin`74- `metadata.json`75- `taxonomy_database.json`76 77The released `data_v3` taxa are fully covered by this embedding store.78 79## Model and code80 81The corresponding public model and code release is:82 83- `alegendaryfish/CodonTranslator`84 85## Attribution86 87This release redistributes processed training data and derived embeddings for reproducibility of the CodonTranslator experiments. See the audit files in `_work/` for construction details and final verification outputs.88 89The raw MMseqs working database is not included because it is large, machine-specific intermediate state. The released `_work/` files are the reproducibility artifacts needed to audit and reconstruct the clustering and split decisions.90