fraQtl/mistral-7b-instruct-v0.3-kv-sidecars

Mistral-7B-Instruct-v0.3 — fraQtl KV-Cache Compression Sidecars

Model Overview

Description

The fraQtl KV-cache compression sidecars for Mistral-7B-Instruct-v0.3 enable serving Mistral-7B-Instruct-v0.3 in vLLM with 2.4× the KV-cache capacity of fp16 (1.2× of fp8-KV) on NVIDIA A100, at ≈95% of fp16 batch-1 decode speed at 8K and parity at 32K — retrieval verified via needle-in-a-haystack grids at every published point.

The sidecars are calibrated per-layer, per-KV-head eigenbasis artifacts consumed by the fraqtl runtime (a prebuilt binary vLLM attention backend). Base model weights are not modified or redistributed.

[image]

License/Terms of Use

Sidecars and receipts: Apache 2.0. Base model weights download from mistralai/Mistral-7B-Instruct-v0.3 under its own license. The fraqtl runtime wheel is proprietary, free to install and run for verification and evaluation (license).

Deployment Geography

Global.

Use Case

Teams serving LLMs with vLLM who need more concurrent users per GPU, longer contexts within a fixed memory budget, or lower KV-cache cost per token.

Release Date

Hugging Face 07/2026 via this repository. Companion model: Qwen3-4B-Instruct-2507 kit (includes 128K-context and concurrency receipts).

References

• —Runtime wheel + build provenance: fraQtl/fraqtl-sm80-runtime
• —Original three-arm receipts: `receipts/` · Independent reproduction: `receipts/repro_2026_07_03/`

Model Architecture

Artifact type: Calibrated KV-cache eigenbasis sidecars (K + V) consumed by a compressed-page attention runtime for vLLM.

Compression architecture: fraQtl inserts a compression membrane between the model and vLLM's paged KV cache. On write, K and V are stored in a compressed page format: a calibrated protected subspace kept at high precision (rank 16 of 128 for K, rank 32 of 128 for V, per layer per KV head — the sidecars in this repository) plus an INT4 tail on the remaining dimensions. Logical rank stays 128; nothing is truncated. On read, the attention kernel consumes compressed pages directly at tensor-core speed — no decompress-then-attend step, which is why capacity gains do not cost decode bandwidth. Recipe label: sm80_k16prot_k112i4_v32prot_v96i4_rv128.

How the protected subspaces are calibrated is not published. The sidecars are the calibrated artifacts — sufficient to run and verify every number on this page.

Input / Output

• —Input: text (any workload the base model supports)
• —Output: text
• —Context length: receipted at 8K and 32K (Mistral-7B's native window). No 128K claims for this model: it collapses under YaRN extension past 32K — fp16 fails the retrieval gate too (receipted). For 128K receipts see the Qwen3-4B kit.

Software Integration

Supported runtime engine: vLLM 0.20.2 (torch 2.11.0 — the receipt-validated stack)

Supported hardware microarchitecture: NVIDIA Ampere (SM80 / A100). Hopper/Blackwell not yet supported.

Operating system: Linux

Integration uses vLLM's standard out-of-tree plugin mechanism (vllm.general_plugins entry point): pip install the wheel and the backend registers in every engine worker process.

Calibration Dataset

• —Dataset: wikitext-2-raw-v1
• —Data collection method: Automated
• —Properties: K sidecar: 16 sequences × 1024 tokens (test split). V sidecar: 128 sequences, 65,536 total tokens. Per-layer, per-KV-head. No task data, no benchmark data, no retrieval-test content.

Usage

Reproduce every number on this page with one command (≈$12 of rented A100 time):

pip install modal && modal setup
modal secret create huggingface HF_TOKEN=hf_...
modal run fraqtl_repro_receipts.py            # mistral is the default model

Prints the three-arm table with per-arm needle-in-a-haystack results. Docker fallback: --local. Script: `fraqtl_repro_receipts.py`.

Evaluation

Three arms, same command, same GPU (A100-80GB), retrieval gate per arm: fraQtl D2 vs fp16 (stock vLLM) vs fp8-KV (kv_cache_dtype=fp8, the strongest available baseline). CUDA graphs on, prefix caching off, batch 1.

Batch-1 decode and KV capacity

The 32K three-arm receipt JSONs are in `receipts/`.

Retrieval verification

Needle-in-a-haystack passkey grids, 7 depths × 3 keys per context, exact-match gated: 21/21 per arm at 8K and 32K — 126/126 total.

Independent reproduction

All six cells were re-run (2026-07-03) through the exact public path in this repository — prebuilt wheel + these sidecars, no internal source: all cells NIAH PASS; fp16/fp8 KV pools byte-identical to the receipts; D2 decode 85.51 @8K / 78.24 @32K and D2 pools within 2% of the receipts. Receipt JSONs: `receipts/repro_2026_07_03/`. Decode speeds vary a few percent run-to-run with host clocks; pools and retrieval results are deterministic.

Why the pool is slightly smaller at 32K than at 8K

The pool is what remains after vLLM reserves working memory at engine init: 0.85 × 80 GB − weights − peak activation scratch − CUDA-graph memory. The activation scratch is profiled against the largest possible prefill, which grows with max_model_len — so a 32K engine holds back more scratch than an 8K engine, for every arm (fp16 shrinks 428K→408K the same way). Page compression is unchanged; the ratios are context-independent.

Evaluation methodology notes

1. 1.Retrieval = NIAH passkey grids, exact-match gated. We never say "lossless".
2. 2.fp8-KV is the strongest available baseline and appears in every table.
3. 3.Batch and context are stated on every number; batch-1 short-context decode is weight-bound — parity is the ceiling for every KV method there.
4. 4.2026-07-03: a batch-routing bug (mixed prefill+decode steps) dropped needles at batch ≥ 2; root-caused, fixed, re-receipted green. Pre-fix multi-user results were never published.

Model Limitations

• —Hardware: SM80 (A100) only; the runtime binary will not run on H100 or consumer GPUs.
• —Context: 8K–32K for this model (base-model YaRN limitation, see Input/Output). 128K and concurrency receipts live on the Qwen3-4B kit.
• —Compression is lossy by construction; the evidence standard is retrieval verification, not bit-identity. Behavior on tasks other than retrieval is not separately benchmarked here.
• —The base model's own limitations and biases are inherited unchanged (weights are not modified).

Files

More Models

Same recipe, receipt format, and measurement standards per model: Qwen3-4B-Instruct-2507 (live, with 128K + concurrency receipts). Next: Llama-3.1-8B, DeepSeek-R1-Distill-Llama-8B, Mistral-Nemo. Request a model by opening a discussion — calibration is one factory run.

More from fraQtl

The on-device lane — calibration-aware Hi-Fi GGUFs with KLD receipts: Gemma-4-E2B (runs an offline phone agent) · Llama-3.1-8B + Llama-3.2-1B draft pair · Qwen3.6-35B MoE. Same standard per artifact: pinned provenance, measured numbers, losses disclosed. Org page: huggingface.co/fraQtl.