marin-dna/genomes-v5-genome_set-mammals_seg20-intervals-v30_255_128

bolinas-dna/genomes-v5-genome_set-mammals_seg20-intervals-v30_255_128

20 mammals projected conserved enhancers (v30) sequences — 255 bp DNA windows for genomic language model pretraining.

Part of the `bolinas-dna/genomes-v5` training-dataset family produced by the `snakemake/training_dataset` pipeline (commit `8db58254831f`). Each repo in the family is one (genome_set, region-recipe) combination.

Size

6,549,730 sequences across 64 data/train/*.jsonl.zst shards (reverse complements included). This is an exact count over the shards — HuggingFace's auto-generated row count is frequently wrong for sharded JSONL.zst, so prefer this number.

Genome set: mammals_seg20

20 chromosome-level mammalian assemblies, one per order (the same set as enhancer_seg_mammals_v1).

Region recipe: v30

Cross-species enhancers: human ENCODE cCRE ELS with ≥20 conserved bp (phyloP-241way ≥ 2.27), projected onto each target genome by mmseqs2 best-hit alignment (-s 7.5 --max-accept 1), resized to 255 bp and intersected with scannable (defined minus low-quality-excluded exons). On human this is the native set; on other genomes it is the alignment projection.

Windows are 255 bp, tiled with a 128 bp stride (~50% overlap); windows overlapping undefined (N) sequence are dropped.

Schema

data/train/*.jsonl.zst — 64 zstd-compressed JSON-Lines shards, globally shuffled (seed 42). Each record:

Sequence case = soft-masked repeats

seq carries the soft-masking from the source assembly (the NCBI RefSeq 2bit), unchanged:

• —Uppercase A / C / G / T — a non-repeat-masked base.
• —Lowercase a / c / g / t — a repeat-masked base (interspersed repeats, low-complexity, simple tandem repeats). N marks undefined sequence.

⚠️ This is not a conservation encoding. The matched validation sets (bolinas-dna/genomes-v5-validation-…) instead overwrite case to encode phyloP conservation (uppercase = conserved). Do not conflate the two conventions.

Reverse complements

add_rc = True: every window appears twice — the reference-forward sequence (id suffix _+) and its reverse complement (_-), sharing identical chrom:start-end coordinates. The suffix denotes forward-vs-RC, not the annotation strand of the source region.

Construction

1. 1.Download genome assemblies (soft-masked 2bit) + GTF annotations from NCBI RefSeq for every genome in mammals_seg20 (20 genomes).
2. 2.Build the v30 interval set per genome — see Region recipe above.
3. 3.Tile into 255 bp / 128 bp windows; drop windows overlapping undefined (N) sequence.
4. 4.Extract sequence with twoBitToFa (soft-masking preserved).
5. 5.Add reverse complements (add_rc).
6. 6.Concatenate across all genomes, globally shuffle (seed 42), and shard into 64 .jsonl.zst files.

This is a train-only set (no chromosome holdout); evaluation uses the separate conservation-encoded validation repos.

Provenance

• —Pipeline: `snakemake/training_dataset` @ commit `8db58254831f`
• —Genomes: NCBI RefSeq, selected by the genome_selection stage (taxonomic + assembly-quality filters, taxonomic dedup).
• —Matched validation set (conservation-encoded human): `bolinas-dna/genomes-v5-validation-intervals-v30_255_255`