fraQtl/mistral-nemo-instruct-2407-kv-sidecars

fraQtl Membrane Sidecar — Mistral-Nemo-Instruct-2407

What this is: the fraQtl calibration sidecars for Mistral-Nemo-Instruct-2407 (12B, 40 layers). Built by the fraQtl sidecar factory with zero kernel changes — the same membrane runtime, receipts below.

Receipts (llama.cpp membrane runtime, 1×A100-80GB, Q4KM weights)

Decode tok/s, single sequence:

<!-- 32K row restored 2026-08-20 from exclusive-mode receipt (needle 1/1, run fc-01M0FY3B9W69NX8AWXK8Z1Y9FF) — INFRA: re-stamp this row addition. Legacy-prototype column stays out of the public card (in receipts). -->

Concurrent capacity ("users" = N parallel sequences via -np N, each with an independent KV stream and a full 32,832-token context allocation; exact command line in every receipt JSON):

Effective context — what we measured

We publish Nemo receipts to 32K. At 128K, needle retrieval fails on this model under EVERY configuration we tested — including llama.cpp q8_0 KV and stock upstream llama.cpp with uncompressed fp16 KV (both 0/1). The model's advertised 128K window exceeds its effective retrieval range; this is a property of the model, control-verified, not of the compression. The membrane matched fp16 retrieval behavior exactly at every context.

Additional disclosure: isolated single-needle misses at high load — q8_0 dropped 1/16 at 16 users; membrane dropped 1/28 at 28 users (and passed 32/32 at 32 users). Both arms show occasional high-load misses on this model; per-cell counts are in the receipts.

How to run

Same prebuilt runtime and flags as the Qwen3-4B sidecar card, with this model's sidecar files (this repo). Runtime binaries (Linux x86_64, CUDA, SM80+SM90, manifest-sha'd): fraQtl/fraqtl-membrane-llamacpp-runtime

More from fraQtl

The on-device lane — calibration-aware Hi-Fi GGUFs with KLD receipts: org page. The same membrane in vLLM: nine concurrent ≈128K users on one A100 (134.1 tok/s, 9/9 retrieval) — receipt.