marin-dna/genomes-v5-genome_set-humans-intervals-v17_255_128

bolinas-dna/genomes-v5-genome_set-humans-intervals-v17_255_128

Humans cCRE enhancers (v17) sequences — 255 bp DNA windows for genomic language model pretraining.

Part of the `bolinas-dna/genomes-v5` training-dataset family produced by the `snakemake/training_dataset` pipeline (commit `8db58254831f`). Each repo in the family is one (genome_set, region-recipe) combination.

Size

3,447,988 sequences across 64 data/train/*.jsonl.zst shards (reverse complements included). This is an exact count over the shards — HuggingFace's auto-generated row count is frequently wrong for sharded JSONL.zst, so prefer this number.

Genome set: humans

Homo sapiens only (GRCh38 / GCF_000001405.40).

Region recipe: v17

ENCODE cCRE V4 enhancer-like signatures (distal dELS + proximal pELS), each resized to 255 bp and intersected with defined. Defined natively on the human genome (GRCh38).

Windows are 255 bp, tiled with a 128 bp stride (~50% overlap); windows overlapping undefined (N) sequence are dropped.

Schema

data/train/*.jsonl.zst — 64 zstd-compressed JSON-Lines shards, globally shuffled (seed 42). Each record:

Sequence case = soft-masked repeats

seq carries the soft-masking from the source assembly (the NCBI RefSeq 2bit), unchanged:

• —Uppercase A / C / G / T — a non-repeat-masked base.
• —Lowercase a / c / g / t — a repeat-masked base (interspersed repeats, low-complexity, simple tandem repeats). N marks undefined sequence.

⚠️ This is not a conservation encoding. The matched validation sets (bolinas-dna/genomes-v5-validation-…) instead overwrite case to encode phyloP conservation (uppercase = conserved). Do not conflate the two conventions.

Reverse complements

add_rc = True: every window appears twice — the reference-forward sequence (id suffix _+) and its reverse complement (_-), sharing identical chrom:start-end coordinates. The suffix denotes forward-vs-RC, not the annotation strand of the source region.

Construction

1. 1.Download genome assemblies (soft-masked 2bit) + GTF annotations from NCBI RefSeq for every genome in humans (1 genome).
2. 2.Build the v17 interval set per genome — see Region recipe above.
3. 3.Tile into 255 bp / 128 bp windows; drop windows overlapping undefined (N) sequence.
4. 4.Extract sequence with twoBitToFa (soft-masking preserved).
5. 5.Add reverse complements (add_rc).
6. 6.Concatenate across all genomes, globally shuffle (seed 42), and shard into 64 .jsonl.zst files.

This is a train-only set (no chromosome holdout); evaluation uses the separate conservation-encoded validation repos.

Provenance

• —Pipeline: `snakemake/training_dataset` @ commit `8db58254831f`
• —Genomes: NCBI RefSeq, selected by the genome_selection stage (taxonomic + assembly-quality filters, taxonomic dedup).
• —Matched validation set (conservation-encoded human): `bolinas-dna/genomes-v5-validation-intervals-v17_255_255`