HuggingFaceBio/genomic-niah

1---2license: apache-2.03task_categories:4  - text-generation5tags:6  - dna7  - genomic8  - long-context9  - retrieval10  - benchmark11  - niah12configs:13  - config_name: niah_4k14    data_files:15      - split: test16        path: data/niah_4k/test-*.parquet17  - config_name: niah_8k18    data_files:19      - split: test20        path: data/niah_8k/test-*.parquet21  - config_name: niah_16k22    data_files:23      - split: test24        path: data/niah_16k/test-*.parquet25  - config_name: niah_32k26    data_files:27      - split: test28        path: data/niah_32k/test-*.parquet29  - config_name: niah_64k30    data_files:31      - split: test32        path: data/niah_64k/test-*.parquet33  - config_name: niah_128k34    data_files:35      - split: test36        path: data/niah_128k/test-*.parquet37  - config_name: niah_neardup_d4_4k38    data_files:39      - split: test40        path: data/niah_neardup_d4_4k/test-*.parquet41  - config_name: niah_neardup_d4_8k42    data_files:43      - split: test44        path: data/niah_neardup_d4_8k/test-*.parquet45  - config_name: niah_neardup_d4_16k46    data_files:47      - split: test48        path: data/niah_neardup_d4_16k/test-*.parquet49  - config_name: niah_neardup_d4_32k50    data_files:51      - split: test52        path: data/niah_neardup_d4_32k/test-*.parquet53  - config_name: niah_neardup_d4_64k54    data_files:55      - split: test56        path: data/niah_neardup_d4_64k/test-*.parquet57  - config_name: niah_neardup_d4_128k58    data_files:59      - split: test60        path: data/niah_neardup_d4_128k/test-*.parquet61  - config_name: niah_neardup_d2_4k62    data_files:63      - split: test64        path: data/niah_neardup_d2_4k/test-*.parquet65  - config_name: niah_neardup_d2_8k66    data_files:67      - split: test68        path: data/niah_neardup_d2_8k/test-*.parquet69  - config_name: niah_neardup_d2_16k70    data_files:71      - split: test72        path: data/niah_neardup_d2_16k/test-*.parquet73  - config_name: niah_neardup_d2_32k74    data_files:75      - split: test76        path: data/niah_neardup_d2_32k/test-*.parquet77  - config_name: niah_neardup_d2_64k78    data_files:79      - split: test80        path: data/niah_neardup_d2_64k/test-*.parquet81  - config_name: niah_neardup_d2_128k82    data_files:83      - split: test84        path: data/niah_neardup_d2_128k/test-*.parquet85  - config_name: niah_neardup_d1_4k86    data_files:87      - split: test88        path: data/niah_neardup_d1_4k/test-*.parquet89  - config_name: niah_neardup_d1_8k90    data_files:91      - split: test92        path: data/niah_neardup_d1_8k/test-*.parquet93  - config_name: niah_neardup_d1_16k94    data_files:95      - split: test96        path: data/niah_neardup_d1_16k/test-*.parquet97  - config_name: niah_neardup_d1_32k98    data_files:99      - split: test100        path: data/niah_neardup_d1_32k/test-*.parquet101  - config_name: niah_neardup_d1_64k102    data_files:103      - split: test104        path: data/niah_neardup_d1_64k/test-*.parquet105  - config_name: niah_neardup_d1_128k106    data_files:107      - split: test108        path: data/niah_neardup_d1_128k/test-*.parquet109---110 111# Genomic-NIAH112 113**A long-context retrieval benchmark for genomic language models.**114 115Genomic-NIAH measures whether a DNA model actually *uses* its long context. Each example plants a random (KEY, VALUE) DNA pair inside a real-genome haystack and asks the model to recover VALUE given the haystack followed by KEY. Because both KEY and VALUE are uncorrelated with the surrounding sequence, the only way to succeed is to retrieve the planted pair at long range — turning the abstract question "does the model use distal context" into a concrete generative task with a per-example accuracy. This benchmark is inspired from [NIAH](https://github.com/gkamradt/LLMTest_NeedleInAHaystack) and [RULER](https://arxiv.org/abs/2404.06654), popular long context benchmarks for natural language.116 117The benchmark provides 4 tasks of varying difficulty, and 24 sub-configurations on a **4 × 6 grid** of task difficulty (plain retrieval and three near-duplicate variants) × context length (from 4k to 128k 6-mer tokens ≈ 24,000 DNA base pairs – 786,000 DNA base pairs). Each sub-configuration contains 500 examples, stratified across five needle depths and four eukaryotic kingdoms.118 119## Table of contents120 1211. [Quick start](#quick-start)1222. [Tasks](#tasks)1233. [Context lengths](#context-lengths)1244. [Sampling design](#sampling-design)1255. [Schema](#schema)1266. [Evaluation metrics](#evaluation-metrics)1277. [Reference results (Carbon-3B)](#reference-results-carbon-3b)1288. [How the benchmark is built](#how-the-benchmark-is-built)129 130## Quick start131 132```python133from datasets import load_dataset134 135# Plain retrieval at 16 k tokens (≈ 98 kbp)136ds = load_dataset("hf-carbon/genomic-niah", "niah_16k")137 138# Hardest near-duplicate variant (KEY′ differs by 1 bp) at 64 k tokens (≈ 384 kbp)139ds = load_dataset("hf-carbon/genomic-niah", "niah_neardup_d1_64k")140```141 142For each example, the `prompt` field already contains the full input to feed the model. The expected response is in `value` (for `gen_exact_match`) or can be discriminated against `negative_value` (for `ll_correct`). See [Evaluation metrics](#evaluation-metrics).143 144## Tasks145 146| Config prefix | Description | Difficulty |147|---|---|---|148| `niah` | Insert a single (KEY, VALUE) pair into the haystack, query KEY → expect VALUE. The plain retrieval cliff diagnostic. | Easy once the model handles the relevant context length. |149| `niah_neardup_d4` | Plain retrieval + **eight near-duplicate distractor pairs**. Each distractor key differs from the target KEY by 4 bp (83 % key identity). | Moderate. Stresses long-range exact matching. |150| `niah_neardup_d2` | Same as above, distractor distance Δ = 2 bp (92 % key identity). | Hard. Separates extension methods well in our experiments. |151| `niah_neardup_d1` | Same, Δ = 1 bp (96 % key identity). | Discrimination test. Used to characterise fine-grained retrieval. |152 153The near-duplicate variants are inspired by RULER's multi-key task, but they use *near-duplicate* rather than fully random distractors. This is the relevant regime for DNA: real genomes contain many similar-but-not-identical motifs, so a useful long-context DNA model must be able to discriminate sequences that differ at only a few positions.154 155## Context lengths156 157Six lengths, given in 6-mer tokens (the Carbon convention) and in base pairs (1 token ≈ 6 bp):158 159| Tokens | bp | Haystack construction |160|---:|---:|---|161| 4 k | ≈ 24 kbp | Contiguous |162| 8 k | ≈ 49 kbp | Contiguous |163| 16 k | ≈ 98 kbp | Contiguous |164| 32 k | ≈ 197 kbp | Stitched-within-record |165| 64 k | ≈ 393 kbp | Stitched-within-record |166| 128 k | ≈ 786 kbp | Stitched-within-record |167 168**Why two haystack modes.** OpenGenome2 records are pre-chunked at 128 KiB of contiguous ACGT, with breaks introduced by `N`s, soft-masked repeats, and metadata. For context lengths ≤ 16 k tokens, single contiguous ACGT runs of one OG2 record are sufficient; the `haystack_mode` field is `"real"`. For 32 k tokens and above, contiguous runs are rarely long enough, so we stitch together all ACGT runs from a single record into one sequence (ambiguity-coded and soft-masked positions removed); `haystack_mode` is `"real_stitched"`. This matches how DNA language models are trained, preserves chromosomal locality, and ensures the haystack remains real genomic sequence even at the longest evaluated contexts. **The benchmark never falls back to random A/C/G/T padding** — real haystacks are essential to its diagnostic value.169 170## Sampling design171 172Each of the 24 configurations contains **n = 500 retrieval examples**, stratified as a 5 × 4 grid:173 174- **Five needle depths.** The (KEY, VALUE) needle is inserted at one of five depths within the haystack: 10 %, 25 %, 50 %, 75 %, or 90 % of haystack length. 100 examples per depth. This exposes asymmetric attention patterns — for instance, models that retrieve well from the end but poorly from the middle.175- **Four eukaryotic kingdoms.** Examples are stratified across `animalia`, `fungi`, `plantae`, and `protista` (25 examples per depth-kingdom cell). The kingdom split matches the multi-kingdom composition of typical DNA-model training corpora. Chromista is excluded because it is not in the CARBON-0 training mixture and has the highest fragmentation rate under the ACGT filter.176 177Total benchmark size: 4 tasks × 6 contexts × 500 examples = **12 000 retrieval examples**.178 179## Schema180 181Each row contains:182 183| Column | Type | Description |184|---|---|---|185| `uid` | `string` | Deterministic SHA1 identifier for this row. |186| `source` | `string` | Eukaryotic kingdom: `animalia` / `fungi` / `plantae` / `protista`. |187| `haystack_mode` | `string` | `real` (contiguous ACGT run) or `real_stitched` (concatenated ACGT runs from one record). |188| `context_length_tokens` | `int` | Length of the haystack in 6-mer tokens. |189| `context_length_bp` | `int` | Length of `positive_sequence` in base pairs. |190| `depth` | `int` | Needle insertion depth as a percentage (10, 25, 50, 75, 90). |191| `condition` | `string` | `retrieval` (only condition in this release). |192| `key` | `string` | The needle KEY: a random 24 bp DNA string. |193| `value` | `string` | The needle VALUE: a random 24 bp DNA string. |194| `negative_value` | `string` | `value` with every nucleotide mutated to a different base. Used for `ll_correct`. |195| `prompt` | `string` | The model input: `<dna>` + haystack with embedded (KEY, VALUE) + trailing KEY. |196| `positive_sequence` | `string` | `prompt + value` (the correct full continuation). |197| `negative_sequence` | `string` | `prompt + negative_value` (used for likelihood discrimination). |198| `target_insert_start_bp`, `target_insert_end_bp` | `int` | Needle position. Available for `niah` only. |199| `distractor_keys`, `distractor_values` | `string` | Comma-separated lists of the 8 near-duplicate distractor pairs. Available for `niah_neardup_*` only. |200 201## Evaluation metrics202 203The official evaluation script lives in the Carbon repository. See [`evaluation/genome_niah_eval.py`](https://github.com/huggingface/carbon/tree/main/evaluation#5-genome-niah-long-context-retrieval) for the command-line interface, supported backends, and reproducible run instructions.204 205Genomic-NIAH supports two metrics, computed per example:206 207- **`gen_exact_match` (recommended headline metric).** Greedy decoding with the next-token logit space restricted to DNA tokens. The decoded VALUE must match the ground-truth `value` exactly. This is the strict, fully-generative retrieval accuracy.208- **`ll_correct` (complementary).** The log-likelihood of `positive_sequence` (`prompt + value`) is higher than the log-likelihood of `negative_sequence` (`prompt + negative_value`). This is a softer probe: the model only needs to prefer the correct VALUE over a fully-mutated alternative, not to generate it. Useful for diagnosing whether retrieval failures are decoding failures or attention failures.209 210We treat `gen_exact_match` as the task metric. `ll_correct` is informative but easier to satisfy; it should never be reported in isolation.211 212## Reference results (Carbon-3B)213 214We report reference Genomic-NIAH results on [Carbon-3B](https://huggingface.co/hf-carbon/Carbon-3B) and [Carbon-8B](https://huggingface.co/hf-carbon/Carbon-8B) to illustrate how the benchmark behaves across difficulties and context lengths. We evaluate two inference settings: Carbon until the native boundary **32k context**, and the same model under **YaRN 4×** (up to 64 k tokens, ≈ 384 kbp and 128k/131k=786kbp).215 216| Native (32 k) | YaRN 4× (up to 64 k) |217|:---:|:---:|218| ![Carbon-3B native, 32 k context, gen_exact_match across the 4×6 grid](assets/modelcard_carbon3b_native.png) | ![Carbon-3B YaRN 4×, up to 64 k context, gen_exact_match across the 4×6 grid](assets/modelcard_carbon3b_yarn4x.png) |219 220| Native (32 k) | YaRN 4× (up to 131 k) |221|:---:|:---:|222| ![Carbon-8B native, 32 k context, gen_exact_match across the 4×6 grid](https://cdn-uploads.huggingface.co/production/uploads/61c141342aac764ce1654e43/69WQVaZyjqxn4vJXs-GJG.png) | ![Carbon-8B YaRN 4×, up to 64 k context, gen_exact_match across the 4×6 grid](https://cdn-uploads.huggingface.co/production/uploads/61c141342aac764ce1654e43/RnOxhdp37rjrFZd1dqONq.png) |223 224 225 226**Reading the heatmaps.**227 228- The `niah_neardup_*` variants are harder than `niah`, and the performance drops with longer contexts.229- Carbon-3B retrieves reliably up to its 32k native boundary. **YaRN 4×** recovers most of the loss at the 32 k → 64 k boundary and extends the usable retrieval range from ≈ 197 kbp to ≈ 384 kbp; beyond 64 k tokens (i.e., at 128 k), retrieval degrades sharply.230 231## How the benchmark is built232 233- **Haystacks** are drawn from the OpenGenome2 NCBI-eukaryotic *test* and *valid* splits across all eight batches: [`arcinstitute/opengenome2 → midtraining_specific/ncbi_eukaryotic_genomes/`](https://huggingface.co/datasets/arcinstitute/opengenome2/tree/main/json/midtraining_specific/ncbi_eukaryotic_genomes).234- **ACGT filter:** ambiguity-coded and soft-masked positions are removed. For ctx ≤ 16 k tokens we use the longest contiguous ACGT run of a single record; for ctx ≥ 32 k tokens we stitch all ACGT runs from one record into a single sequence, preserving record-level (chromosome-level) locality.235- **Needles:** KEY and VALUE are independent random 24 bp DNA strings, sampled per-example with a deterministic seed derived from the row index.236- **Distractor keys** (near-duplicate variants only): obtained by substituting exactly Δ randomly chosen positions of KEY to a different base, for $\Delta \in \{1, 2, 4\}$. Each example has 8 distractor pairs with the same Δ.237- **Insertion:** the (KEY, VALUE) pair (and each distractor pair, if any) is inserted as a literal substring at the configured depth. The pairs are placed at distinct positions; for near-duplicate variants, distractors are placed before and after the target proportionally so that no depth bias is introduced by distractor placement.238 239## License240 241Apache 2.0.242 243## Citation244 245```246@article{allal2026carbon,247  title={Carbon: Decoding the Language of Life},248  author={Allal, Loubna Ben and Li, Qiuyi and Fiusco, Maurizio and Tunstall, Lewis and Rasul, Kashif and Beeching, Ed and Aubakirova, Dana and Pati{\~n}o, Carlos and Frere, Thibaud and Lozhkov, Anton and others},249  journal={bioRxiv},250  pages={2026--05},251  year={2026},252  publisher={Cold Spring Harbor Laboratory}253}254```