marin-dna/zoonomia-v1-v4_utr3

bolinas-dna/zoonomia-v1-v4_utr3

Per-anchor region-type partition of the cross-mammal training set `bolinas-dna/zoonomia-v1-v1`, restricted to anchors labelled utr3 by the v4 region labeler (`snakemake/zoonomia_projection_dataset` pipeline, commit `4729d06d576f`).

v4 re-derives the v3 partition with the labeling scheme resolved in issue #221: base-pair priority + window majority, a protein-coding-only TSS band, and `ccre_flank=0`. See the pipeline README's "v4 region-type annotation" section for the full rationale.

Region label (utr3)

3' untranslated region — Ensembl r115 protein-coding 3' UTR (get_ensembl_3_prime_utr). Second priority (cedes only to cds); labelled utr3 when 3' UTR is the majority disjoint region of the window.

Partition

The six v4 subsets partition the conservation-filtered human anchor set (every anchor is assigned exactly one label) by a two-stage rule:

1. 1.Base-pair priority assigns every base to exactly one region by the order below (subtracting higher-priority regions from lower ones), so a base in both CDS and a cCRE is cds.
2. 2.Window majority labels each window by the region covering the most of its bases, provided the union-of-functional fraction is ≥ 0.20 (else background).

cds > utr3 > tss_region_and_utr5 > ncrna_exon > ccre_non_promoter > background

This subset contains 67,155 of 1,136,854 human anchors (5.91% of v1), expanding to 12,586,492 training samples after halLiftover projection to up to 108 Zoonomia mammals and reverse-complement augmentation (same shape as `bolinas-dna/zoonomia-v1-v1`, just filtered to this region label). The total is the exact row count across all 64 JSONL.zst shards.

Five sibling v4 subsets (one per region label):

• —`bolinas-dna/zoonomia-v1-v4_cds`
• —`bolinas-dna/zoonomia-v1-v4_ncrna_exon`
• —`bolinas-dna/zoonomia-v1-v4_tss_region_and_utr5`
• —`bolinas-dna/zoonomia-v1-v4_ccre_non_promoter`
• —`bolinas-dna/zoonomia-v1-v4_bg`

Schema

Same as `bolinas-dna/zoonomia-v1-v1` — a single train split of JSONL.zst shards at data/train/shard_NNNN.jsonl.zst:

Construction

1. 1.Build the v1 cross-mammal training set (108-species halLiftover projection of conservation-filtered 255 bp human anchors). See the pipeline README.
2. 2.Annotate each anchor with one of six v4 region labels (base-pair priority + window majority; union-of-functional fraction ≥ 0.20 required to escape background). Library: marin_dna.pipelines.zoonomia_projection_dataset.region_labels.label_windows_bp_majority.
3. 3.Filter v1 to anchors labelled utr3 via subset_dataset_derived (Polars lazy-filter on query_name).
4. 4.RC-augment, shuffle (seed=42), shard to 64 JSONL files, zstd-compress, upload via hf upload-large-folder.

Caveats

• —The six v4 subsets are a partition of v1, not independent probes. Concatenating them reconstructs v1 (modulo the RC augmentation and the shuffle seed). Each anchor appears in exactly one subset.
• —v4 ≠ v3. v3 (bolinas-dna/zoonomia-v1-v3_*) used priority-on-presence, an all-transcript TSS band, and ccre_flank=500; the partitions differ substantially. Use v4 unless you specifically need to match a v3-trained checkpoint.
• —Broad `ncrna_exon`. Still the set complement get_exons(ann) − get_ensembl_protein_coding_exons(ann) (no functional-biotype filter); use val_ncrna for functional ncRNA only.

Source code

• —Pipeline: snakemake/zoonomia_projection_dataset (latest)
• —Pinned to this dataset's build: commit `4729d06d576f`
• —Region labeler library: marin_dna.pipelines.zoonomia_projection_dataset.region_labels.label_windows_bp_majority
• —Sister cross-mammal datasets: bolinas-dna/zoonomia-v1-v1, bolinas-dna/zoonomia-v1-v2
• —Sister validation datasets: bolinas-dna/zoonomia-v1-val_*