marin-dna/zoonomia-v1-v3_ccre_non_promoter

bolinas-dna/zoonomia-v1-v3_ccre_non_promoter

Per-anchor region-type partition of the cross-mammal training set `bolinas-dna/zoonomia-v1-v1`, restricted to anchors labelled ccre_non_promoter by the `snakemake/zoonomia_projection_dataset` pipeline (commit `2ab868a2f1d4`).

Region label (ccre_non_promoter)

ENCODE cCRE V4 non-promoter classes — cre_class != "PLS" (so: dELS, pELS, CA, CA-CTCF, CA-TF, CA-H3K4me3, TF), extended by 500 bp on each side. PLS is excluded because PLS-overlapping anchors near an annotated TSS are already captured by tss_region_and_utr5; isolated PLS (no nearby annotated TSS) becomes background.

Partition

The six v3 subsets partition the conservation-filtered human anchor set (every anchor is assigned exactly one label) by priority-walk:

cds > utr3 > ncrna_exon > tss_region_and_utr5 > ccre_non_promoter > background

This subset contains 468,131 of 1,136,854 human anchors (41.18% of v1), expanding to 96,639,800 training samples after halLiftover projection to up to 108 Zoonomia mammals and reverse-complement augmentation (same shape as `bolinas-dna/zoonomia-v1-v1`, just filtered to this region label). The total is the exact row count across all 64 JSONL.zst shards — included explicitly because HF's automatic estimate (based on first-shard byte size) is unreliable for sharded datasets.

Five sibling v3 subsets (one per region label):

• —`bolinas-dna/zoonomia-v1-v3_cds`
• —`bolinas-dna/zoonomia-v1-v3_utr3`
• —`bolinas-dna/zoonomia-v1-v3_ncrna_exon`
• —`bolinas-dna/zoonomia-v1-v3_tss_region_and_utr5`
• —`bolinas-dna/zoonomia-v1-v3_bg`

Schema

Same as `bolinas-dna/zoonomia-v1-v1` — a single train split of JSONL.zst shards at data/train/shard_NNNN.jsonl.zst:

Construction

1. 1.Build the v1 cross-mammal training set (108-species halLiftover projection of conservation-filtered 255 bp human anchors). See the pipeline README.
2. 2.Annotate each anchor with one of six region labels (priority shown above; union-of-functional fraction ≥ 0.20 required to escape background). Library: bolinas.zoonomia_projection_dataset.region_labels.
3. 3.Filter v1 to anchors labelled ccre_non_promoter via subset_dataset_derived (Polars lazy-filter on query_name).
4. 4.RC-augment, shuffle (seed=42), shard to 64 JSONL files, zstd-compress, upload via hf upload-large-folder.

Caveats

• —The six v3 subsets are a partition of v1, not independent probes. Concatenating them reconstructs v1 (modulo the RC augmentation and the shuffle seed). Each anchor appears in exactly one subset.
• —Broad `ncrna_exon`. ncrna_exon here is the set complement get_exons(ann) − get_ensembl_protein_coding_exons(ann), which is broader than the val_ncrna validation recipe — it includes pseudogene exons, retained-intron exons, and other non-PC Ensembl biotypes. Use val_ncrna if you want functional ncRNA only.
• —Background is heterogeneous. ~90% have zero functional overlap by the labeler's definitions (true gene deserts or deep introns); ~10% sit just below threshold and are candidates for unannotated regulatory elements or UCEs.

Source code

• —Pipeline: snakemake/zoonomia_projection_dataset (latest)
• —Pinned to this dataset's build: commit `2ab868a2f1d4`
• —Region labeler library: bolinas.zoonomia_projection_dataset.region_labels
• —Sister cross-mammal datasets: bolinas-dna/zoonomia-v1-v1, bolinas-dna/zoonomia-v1-v2
• —Sister validation datasets: bolinas-dna/zoonomia-v1-val_*