marin-dna/genomes-v5-genome_set-vertebrates_order125-intervals-v5_255_128

bolinas-dna/genomes-v5-genome_set-vertebrates_order125-intervals-v5_255_128

125 vertebrates (one per order) CDS (v5) sequences — 255 bp DNA windows for genomic language model pretraining.

Part of the `bolinas-dna/genomes-v5` training-dataset family produced by the `snakemake/training_dataset` pipeline (commit `main`). Each repo in the family is one (genome_set, region-recipe) combination.

Size

66,421,780 sequences across 64 data/train/*.jsonl.zst shards (reverse complements included). This is an exact count over the shards — HuggingFace's auto-generated row count is frequently wrong for sharded JSONL.zst, so prefer this number.

Genome set: vertebrates_order125

The Chordata subset of animals_order204 (125 genomes) — the vertebrate scope of the issue #353 CDS projection-vs-annotation sweep, where nucleotide CDS projection carries real data (invertebrate phyla sit at a ~2% reach floor).

Region recipe: v5

Coding sequence (CDS): every annotated CDS interval, kept if 20 bp–10 kb long, extended by 20 bp on each side (to capture splice-site signal at exon boundaries), then expanded to a minimum length of 256 bp and intersected with defined.

Windows are 255 bp, tiled with a 128 bp stride (~50% overlap); windows overlapping undefined (N) sequence are dropped.

Schema

data/train/*.jsonl.zst — 64 zstd-compressed JSON-Lines shards, globally shuffled (seed 42). Each record:

Sequence case = soft-masked repeats

seq carries the soft-masking from the source assembly (the NCBI RefSeq 2bit), unchanged:

• —Uppercase A / C / G / T — a non-repeat-masked base.
• —Lowercase a / c / g / t — a repeat-masked base (interspersed repeats, low-complexity, simple tandem repeats). N marks undefined sequence.

⚠️ This is not a conservation encoding. The matched validation sets (bolinas-dna/genomes-v5-validation-…) instead overwrite case to encode phyloP conservation (uppercase = conserved). Do not conflate the two conventions.

Reverse complements

add_rc = True: every window appears twice — the reference-forward sequence (id suffix _+) and its reverse complement (_-), sharing identical chrom:start-end coordinates. The suffix denotes forward-vs-RC, not the annotation strand of the source region.

Construction

1. 1.Download genome assemblies (soft-masked 2bit) + GTF annotations from NCBI RefSeq for every genome in vertebrates_order125 (125 genomes).
2. 2.Build the v5 interval set per genome — see Region recipe above.
3. 3.Tile into 255 bp / 128 bp windows; drop windows overlapping undefined (N) sequence.
4. 4.Extract sequence with twoBitToFa (soft-masking preserved).
5. 5.Add reverse complements (add_rc).
6. 6.Concatenate across all genomes, globally shuffle (seed 42), and shard into 64 .jsonl.zst files.

This is a train-only set (no chromosome holdout); evaluation uses the separate conservation-encoded validation repos.

Provenance

• —Pipeline: `snakemake/training_dataset` @ commit `main`
• —Genomes: NCBI RefSeq, selected by the genome_selection stage (taxonomic + assembly-quality filters, taxonomic dedup).
• —Matched validation set (conservation-encoded human): `bolinas-dna/genomes-v5-validation-intervals-v5_255_255`