JG1310/mgnify-evo2-amr-shortread-eval-v0

MGnify × Evo 2 short-read AMR detection eval (v0)

A small reproducibility benchmark for AMR detection on simulated short reads using the Evo 2 DNA foundation model.

What's here

• —`reads.jsonl` — 100 records (50 AMR-positive, 50 matched-negative)
• —`embeddings/<read_id>.npz` — Evo 2 layer-26 activations per read

Source

All reads were simulated (MiSeq-like, 301 bp, paired-end, ART simulator) from two CDSs in MGnify MAG MGYG000307615 (chicken-gut catalogue):

• —Positives: MGYG000307615_01006 — abc-f, an ABC-F family ribosomal protection protein conferring MACROLIDE resistance. 57.11% protein identity to the AMRFinderPlus reference (a divergent allele, not a near-copy).
• —Negatives: MGYG000307615_00395 — its matched-CDS negative from the same MAG, length+GC-bracket-matched non-AMR control.

The MAG was held out (val split) of our v1 AMR linear probe — so the probe checkpoint we provide as a reference baseline was never trained on these sequences.

Schema

reads.jsonl

One JSON record per line. Fields:

• —read_id: original FASTQ header (e.g. MGYG000307615_01006_AMR_0_0/1)
• —safe_read_id: filesystem-safe version (slashes replaced with underscores)
• —label: 1 = AMR-positive, 0 = matched-negative
• —label_name: human-readable label
• —sequence: 301 bp DNA forward-strand string
• —seq_len: token count after Evo 2 tokenisation (= 301)
• —source_*: provenance — locustag, genesymbol, label_class, identity, MAG, etc.
• —v1_probe_*_logit: reference logits from our v1 AMR linear probe (mean / median / max over per-token logits)

embeddings/<safe_read_id>.npz

Numpy archive with:

• —layer26_activations_bf16 — uint16 array, shape [seq_len, 4096] (bf16 bit-pattern reinterpreted as uint16; numpy doesn't natively support bf16)
• —layer26_dtype — literal string "bfloat16"
• —source_layer_index — int 26
• —source_layer_name — "blocks-26" (the residual-stream output of block 26)
• —seq_len, hidden_size, model_name ("evo2_7b_262k")
• —read_id, label — for cross-reference with reads.jsonl

Decoding the embeddings

import numpy as np
import torch
import json

# Load metadata
with open("reads.jsonl") as f:
    records = [json.loads(line) for line in f]

# Load one embedding
d = np.load(f"embeddings/{records[0]['safe_read_id']}.npz", allow_pickle=False)
acts = torch.from_numpy(d["layer26_activations_bf16"]).view(torch.bfloat16).float()
# → shape [seq_len, 4096]

Reference baseline

Our v1 AMR linear probe (Linear(4096, 1), trained on full-length 5 kb extracts across other MAGs) achieved ROC-AUC = 1.0000 on these 100 reads using both mean-logit and max-logit aggregation. Per-read logits stored in reads.jsonl.

Caveats — read this before quoting numbers

This is a v0 sanity-check / proof-of-concept benchmark, not a production AMR-detection benchmark.

1. 1.Single CDS pair. All positives come from one gene (abc-f), all negatives from one matched non-AMR CDS. Class diversity = 0.
2. 2.AUC = 1.0 is on this specific easy pair — not representative of real short-read AMR-detection difficulty.
3. 3.Read sampling bias. Reads were taken from the first ~2500 of each FASTQ file (rather than randomly across the full 500k), so within-class diversity is artificially narrow.
4. 4.Simulated reads. ART-simulated MiSeq reads with realistic substitution error model, but no chimeras / contamination / low-complexity artefacts that real metagenomes have.

A meaningful benchmark would need multiple CDS pairs across multiple AMR classes, identity bins, and MAGs.

Citation / provenance

• —MGnify catalogue: chicken-gut species representatives
• —Evo 2 model: Arc Institute, 7B-262k variant
• —Layer 26 = the residual-stream hook target Goodfire's published SAE was trained against