kleinnner/camus-10-kv-cache-quantization

We attempted KV cache quantization to Q4 — and documented why it fails on the qwen2vl architecture. KV Cache Quantization Attempt: type_k/type_v on qwen2vl Architecture The Problem The KV cache consumes significant memory bandwidth during autoregressive generation. On CPU-bound systems, memory bandwidth is the primary bottleneck. Quantizing the KV cache from FP16 to Q4 theoretically halves memory bandwidth requirements. What We Built We attempted… See the full description on the dataset page: https://huggingface.co/datasets/kleinnner/camus-10-kv-cache-quantization.