rsynk/locale-benchmark-sra52-viral

LOCALE benchmark — SRA52 cross-genotype viral retrieval

A retrieval benchmark built on real biological divergence between viral strains, rather than mutations injected into reads synthetically.

Query with sequencing reads from one Hepatitis B virus genotype; retrieve accessions of a different genotype. The two genotypes are ~11% divergent — enough to break exact k-mer overlap while remaining genuinely homologous.

Ground truth is metadata, not alignment

contig_accession holds the relevant accessions for each query. In the sra50 / sra500 benchmarks that column is alignment-derived (reads aligned to contigs). Here it comes from SRA genotype metadata. Every genotype-B target is relevant to every genotype-D query because both are HBV and the genotypes are homologous across ~98% of the genome. No aligner decides what the correct answer is — which matters when the baselines being compared against are themselves k-mer and alignment methods.

Alignment is used only to (a) measure how divergent the genotypes are and (b) screen out query reads with no homolog anywhere. Both help and hurt every method equally.

Alignment can decide whether a question has an answer; it cannot decide what the answer is.

Why these accessions

Every run passed a three-part QC gate:

1. 1.Label correctness — genotype metadata checked against independent, organism-labeled reference genomes from unrelated submitters. This is project-level: one cohort (PRJNA422501) was dropped after 9 of 10 sampled runs contradicted their own labels.
2. 2.HBV fraction ≥ 50% — per-run. Host-dominated runs are mostly unanswerable.
3. 3.Coverage breadth ≥ 75% at ≥10× — per-run, and independent of mapping rate. One candidate cohort had 99.5% of reads mapping at ~100% identity while covering only 7.9% of the genome (a targeted amplicon); breadth is the only check that catches this.

Targets were selected on breadth and HBV fraction — method-neutral criteria — and drawn from two BioProjects so no single lab's batch defines the relevant set.

Query and target accessions share no BioProject, so they share no primers, adapters, or sequencing batch. A method cannot succeed here by matching technical artifacts instead of homology.

Why the divergence band matters

Targeted at 85–92% whole-genome identity. Above that, k-mer baselines succeed too and the benchmark separates nothing. Below it, many query reads have no homolog at all and the metric becomes undefined. The measured 0.8903 sits near the middle.

At this divergence, the fraction of query reads sharing any exact k-mer with the target genotype:

At k=31 — the standard choice, because smaller k loses specificity at SRA scale (a 15-mer has ~1,000 chance occurrences in a 10¹²-base corpus) — half the queries have no exact k-mer in the target at all. An exact k-mer method cannot score them.

Files

queries.parquet schema

Identical to locale-benchmark-sra50.

Notes for use

Query reads are filtered to HBV, deliberately. Logan contigs are assembled from whole runs, and these runs are only 78–95% HBV — so target contig sets also contain human contigs. Unfiltered human query reads would match those human contigs and score as correct retrievals for an entirely non-biological reason. The filter uses only the query's own genotype reference; it does not require reads to align to a target, since that would pre-select the reads which survive cross-genotype alignment — precisely the effect under measurement.

Handle empty result sets carefully. A method that returns nothing for a query should score 0 recall for it, not be dropped from the denominator. If scoring code ranks by argsort over a zero-filled score vector, ties resolve to the alphabetically first accessions and can award spurious credit — one of the relevant accessions here (SRR10076799) sorts 5th of 52. Set unscored accessions to -inf before ranking.