marin-dna/zoonomia-v1-v1

zoonomia-v1-v1

255 bp human-anchored windows, conservation-filtered (phyloP_447m, proportion_conserved >= 0.20), projected onto 108 family-deduped Zoonomia 447-mammalian assemblies via halLiftover, midpoint-resized to 255 bp, reverse-complement-augmented. Single train split.

Produced by the zoonomia_projection_dataset pipeline in Open-Athena/bolinas-dna — permalinked at the exact code that built this dataset: `snakemake/zoonomia_projection_dataset` @ 7ff07cd (PR #158).

Row count

Train: 223,880,280 rows.

Verified by zstd -dc data/train/*.jsonl.zst | wc -l over all 64 shards (per-shard min/max = 3,498,129 / 3,498,130). See issue #166 for context.

The HuggingFace dataset viewer shows wrong row counts for this dataset. It reports estimated_num_rows = 174,621,287 (off by ~22%) and num_rows = 14,082,488 from a partial 10/64-shard parquet conversion. Both numbers are wrong; trust the 223,880,280 above.

How intervals were constructed

Anchor in human (hg38, Ensembl rel 115), then project across mammals.

1. 1.Tile hg38 into 255 bp windows. bedtools makewindows -w 255 -s 128 over autosomes + chrX + chrY (chrM excluded). Drop windows overlapping undefined (N-rich) regions via bedtools intersect -v against the genome's N intervals. Window size 255 (not 256) leaves room for a BOS token to bring the model's context to 256.
2. 2.Conservation filter. Score each window against the phyloP_447m track. NaN positions (no alignment) count as non-conserved (0). Threshold is calibrated so the genome-wide passing-base count matches phyloP_241m >= 2.27 (yields phyloP_447m >= 2.2162). Keep windows with proportion_conserved >= 0.20.
3. 3.Cross-mammal projection. halLiftover --noDupes from the human anchor BED onto 108 family-deduped mammals from the Zoonomia 447-mammalian Cactus alignment (1.18 TiB HAL). Per-species post-processing collapses multi-fragment liftovers to a single merged span per (query_name, species), drops spans outside [128, 512] bp before resize, and midpoint-centers / resizes to exactly 255 bp. At most one row per (query_name, species).
4. 4.Sequence extraction. bedtools getfasta -s against each species' 2bit (derived from the HAL via hal2fasta). Strand-aware: rows with t_strand == "-" already hold the reverse-complemented target string.
5. 5.Reverse-complement augmentation. Each row gets a partner with sequence reverse-complemented and augmentation = "-" (originals are augmentation = "+"). ACGT only; non-ACGT characters are preserved unchanged.
6. 6.Shuffle. df.sample(fraction=1, shuffle=True, seed=42) interleaves species so a model never sees blocks of consecutive same-species rows.

The 108 species set is family-deduped from the 447-mammalian Newick (one leaf per NCBI family) with Homo_sapiens, Mus_musculus, Bos_taurus force-included. List: `config/species_zoonomia_447_family_dedup.tsv`.

Schema

Single train split. JSONL.zst shards at data/train/shard_NNNN.jsonl.zst (64 shards).

2-axis versioning

Dataset names follow zoonomia-{pipeline_version}-{intervals_version}:

• —pipeline_version (here v1) snapshots species set, conservation cutoff, alignment backend, and resize/length-filter params. Bumping requires re-running halLiftover (~5 h on c6id.12xlarge).
• —intervals_version (here v1) names a post-projection subset. v1 is identity — every projected row. The protein-coding TSS-proximal subset is published as `bolinas-dna/zoonomia-v1-v2`.

Loading

from datasets import load_dataset

ds = load_dataset("bolinas-dna/zoonomia-v1-v1", split="train", streaming=True)
for row in ds:
    print(row["query_name"], row["species"], row["sequence"][:30], row["augmentation"])
    break

Source

• —Repo: Open-Athena/bolinas-dna
• —Pipeline (commit-pinned): `snakemake/zoonomia_projection_dataset` @ 7ff07cd
• —Pipeline README: `README.md` @ 7ff07cd
• —Producing PR: #158