rsynk/locale-benchmark-sra50

Locale Embedding Benchmark : SRA 50

What

This benchmark contains embeddings of raw genomic read sequences produced by the LOCALE DNA transformer model to test the use of vector search over large sequence repositories like the NIH Sequence Read Archive. The benchmark contains the embeddings of 9,688,220 sequence embeddings coming from 50 SRA Accessions. All vectors are Float32, D=768, roughly 30GB of total data.

Given a read, we want to find accessions containing sequences that align well to it. Here, each query read, and all sequences in an accession, are embedded by the LOCALE embedding model. The model is trained to translate a local sequence alignment objective into vector dot product distance over normalized vectors, so that well aligned sequences have a larger dot product. Each query read is labeled with a ground truth of matching accessions, and all indexed sequences are also labeled by which accession they come from.

To test the robustness of vector distance methods to sequence variation, the same set of query sequences is injected with random noise (simulating sequencing errors) across two noise profiles: 5% and 10% divergence. All embeddings are Float32 with D=768.

The mutated query sequences are published alongside the clean ones as queries_mut0.05.parquet and queries_mut0.10.parquet (queries_mut0.00.parquet is the clean set in the same layout). They were produced with mutation-simulator at SNP rate r with insertion and deletion rates r/10 each; the simulator's fasta and VCF for every rate are kept under mutations/. Every row keeps its query_id and ground-truth columns, so any method can be evaluated on identical mutated reads.

Note: the mut0.05/ and mut0.10/ query vectors under embeddings/locale/ were embedded from an earlier, in-process mutation draw of the same reads and are not embeddings of the sequences in queries_mut0.05.parquet / queries_mut0.10.parquet. The clean mut0.00/ vectors match queries.parquet / queries_mut0.00.parquet.

Quickstart

The repository is hosted on Hugging Face, and will need the HF CLI in order to download. The CLI can be downloaded with

curl -LsSf https://hf.co/cli/install.sh | bash

Then download the dataset. This will download roughly 30GB of data.

hf download rsynk/locale-benchmark-sra50 --repo-type dataset --local-dir .

To read one .fbin file, you can run

import numpy as np
n, d = np.fromfile(path, dtype=np.uint32, count=2)
x = np.memmap(path, dtype=np.float32, mode="r", offset=8, shape=(int(n), int(d)))

To read in all vectors, ensure to add the shard start offset for proper row indexing when streaming in with:

from pathlib import Path
start = 0
for shard in sorted(Path("embeddings/locale/base").glob("embeddings-*.fbin")):
    n, d = np.fromfile(shard, dtype=np.uint32, count=2)
    x = np.memmap(shard, dtype=np.float32, mode="r", offset=8, shape=(int(n), int(d)))
    index.add_with_ids(x, np.arange(start, start + int(n)))   # global row ids
    start += int(n)

All vectors are normalized, so IP/cosine/L2 rank identically. To score the results of your method, construct a my_res.bin file matching the gt.bin format with the following:

# ids: (500, k) uint32 global row indices, best first
# dists: (500, k) float32 scores, same ordering
with open("my_res.bin", "wb") as f:
    np.array(ids.shape, dtype=np.uint32).tofile(f)   # nq, k
    ids.astype(np.uint32).tofile(f)
    dists.astype(np.float32).tofile(f)

There should be a total of 500 result rows.

Calculate scores with:

python score.py my_res.bin --bundle embeddings/locale --mut 0.00

The score file produces

• —Vector recall @ k against gt.bin
• —accession recall / R-precision / AUPRC = downstream biological metric, computed after collapsing chunk hits to accessions by max score

Ground truth was constructed via exact brute force over vectors with k=100.

Exact-search Baseline

The following gives the downstream biological metrics computed by the ground truth nearest neighbors (note recall is perfect, since we compute with ground truth)

File Layout

locale-benchmark-sra50/
├── README.md                      this card
├── score.py                       evaluation script (numpy + pyarrow only)
├── accs.txt                       50 SRA accession IDs, one per line
├── queries.parquet                query_id, query_sequence, contig_accession (labels)
├── queries_mut0.00.parquet        queries.parquet + mutation_rate column (clean)
├── queries_mut0.05.parquet        same rows, query_sequence mutated: SNP 5%, ins 0.5%, del 0.5%
├── queries_mut0.10.parquet        same rows, query_sequence mutated: SNP 10%, ins 1%, del 1%
├── mutations/                     mutation-simulator fasta + VCF per rate (provenance)
└── embeddings/
    └── locale/                    the bundle -- pass this path as --bundle
        ├── manifest.json          n, dim, num_shards, rows_per_shard 
        ├── meta.parquet           srr_id, start_row, num_rows -- base row -> accession
        ├── base/                  9,688,220 x 768 float32, unit-norm  (30 GB)
        │   ├── embeddings-00000.fbin      325,520 rows   954 MiB
        │   ├── embeddings-00001.fbin      325,520 rows   954 MiB
        │   ├── ...                        (30 shards total)
        │   └── embeddings-00029.fbin      248,140 rows   727 MiB
        ├── mut0.00/               clean queries 
        │   ├── query.fbin                 500 x 768 float32
        │   ├── query_meta.parquet         query_id, start_row, num_rows
        │   └── gt.bin                     exact top-100 neighbors
        ├── mut0.05/               5% mutated queries  (same three files)
        └── mut0.10/               10% mutated queries (same three files)

File formats:

• —.fbin, uint32 n, uint32 d, then float32[n*d] row major
• —gt.bin / result files: uint32 nq, uint32 k, uint32 ids [nqk], float32 dists[nqk]
• —base is sharded, concatenate in filename order to get logical row index, per-shard headers hold shard row count. Ensure to add the shard start offset for proper ID mapping
• —meta.parquet: srrid, startrow, num_rows, maps global embedding row index to accessions
• —querymeta.parquet: queryid, startrow, numrows - one row per chunk, queries > 256 bp are split

License

The data is licensed under Creative Commons cc-by-4.0, and code is licensed under MIT.

The underlying sequence data comes from the NCBI Sequence Read Archive, which places no restrictions on redistribution.

Citation

For questions, reach out to ryansynk@umd.edu. If you find our work useful, please cite here:

@article {Synk2026.05.12.724581,
	author = {Synk, Ryan P. and Pandey, Prashant and Sahinalp, S. Cenk and Duraiswami, Ramani},
	title = {LOCALE: Local-Alignment Embeddings for Noise-Robust DNA Search at SRA Scale},
	elocation-id = {2026.05.12.724581},
	year = {2026},
	doi = {10.64898/2026.05.12.724581},
	publisher = {Cold Spring Harbor Laboratory},
	URL = {https://www.biorxiv.org/content/early/2026/05/14/2026.05.12.724581},
	eprint = {https://www.biorxiv.org/content/early/2026/05/14/2026.05.12.724581.full.pdf},
	journal = {bioRxiv}
}