Datasetpublic

fraQtl/Mistral-7B-v0.3-fraqtl-sidecars

fraQtl D1 — Mistral-7B-v0.3 KV-cache compression sidecars This dataset hosts the precomputed V and K eigenbasis sidecars used by the fraQtl D1 reproduction bundle for Mistral-7B-Instruct-v0.3 at 128K context. What's here File Size sha256 sidecar_real_u_mistral-7b-instruct-v0.3.bin 8.4 MB 82186441d1810bba28c084a085b895ddd3530e4564e50c0b0c68e37fe69ce58e mistral-7b-instruct-v0.3-k.fraqtl-k-eigenbasis.bin 8.4 MB… See the full description on the dataset page: https://huggingface.co/datasets/fraQtl/Mistral-7B-v0.3-fraqtl-sidecars.

sourceHugging Facemitupdated 4mo agoView on Hugging Face

0likes21downloads

Dataset Card

fraQtl D1 — Mistral-7B-v0.3 KV-cache compression sidecars

This dataset hosts the precomputed V and K eigenbasis sidecars used by the fraQtl D1 reproduction bundle for Mistral-7B-Instruct-v0.3 at 128K context.

What's here

File	Size	sha256
`sidecar_real_u_mistral-7b-instruct-v0.3.bin`	8.4 MB	`82186441d1810bba28c084a085b895ddd3530e4564e50c0b0c68e37fe69ce58e`
`mistral-7b-instruct-v0.3-k.fraqtl-k-eigenbasis.bin`	8.4 MB	`85ed2893b372a4194386a5ae9f2895ac9c3eeae02faae8979c3d104706886faf`

How to use

These sidecars are consumed by the fraQtl D1 patch against llama.cpp. Full reproduction recipe + patch lives in the GitHub repo:

→ https://github.com/fraqtl-ai/fraqtl-mistral-d1

Short version:

bash

huggingface-cli download fraQtl/Mistral-7B-v0.3-fraqtl-sidecars \
  sidecar_real_u_mistral-7b-instruct-v0.3.bin \
  mistral-7b-instruct-v0.3-k.fraqtl-k-eigenbasis.bin \
  --local-dir ./sidecars --repo-type dataset

Then point the patched llama-completion --fraqtl-eigenbasis and --fraqtl-k-eigenbasis flags at those files. The receipts in the GitHub repo lock the exact CLI flags + expected VRAM/NIAH numbers.

The result

Same Q4KM weights, same llama.cpp Q4KM kernel path. Only the KV cache treatment varies. NIAH = 5-fact retrieval at 128K context, scored 0–5.

Run	KV cache	Live VRAM peak	NIAH
baseline	fp16	22,657 MiB	5 / 5
`llama.cpp --cache-type-k q8_0`	Q8_0	15,437 MiB	1 / 5
`llama.cpp --cache-type-k q4_0`	Q4_0	11,287 MiB	0 / 5
fraQtl D1 (these sidecars)	sidecar + sink/residual	13,261 MiB	5 / 5

→ fraQtl D1 is the only Q4-class KV configuration that holds NIAH at 128K on this base model. Both in-tree llama.cpp KV-quant options fail at this scale.

License

MIT. The sidecars are released under the MIT license. The fraQtl calibrator that generates these sidecars is private — DM contact@fraqtl.ai for calibration on other base models.

Contact

—🌐 fraqtl.ai
—📬 contact@fraqtl.ai
—🧪 Free diagnostic: pip install fraqtl-diagnostic