kv-cache-quantization
kvcache-quantization-logs-qwen7bcamus-10-kv-cache-quantization
We attempted KV cache quantization to Q4 — and documented why it fails on the qwen2vl architecture.
KV Cache Quantization Attempt: type_k/type_v on qwen2vl Architecture
The Problem
The KV cache consumes significant memory bandwidth during autoregressive generation. On CPU-bound systems, memory bandwidth is the primary bottleneck. Quantizing the KV cache from FP16 to Q4 theoretically halves memory bandwidth requirements.
What We Built
We attempted… See the full description on the dataset page: https://huggingface.co/datasets/Anticloud/camus-10-kv-cache-quantization.camus-10-kv-cache-quantization
We attempted KV cache quantization to Q4 — and documented why it fails on the qwen2vl architecture.
KV Cache Quantization Attempt: type_k/type_v on qwen2vl Architecture
The Problem
The KV cache consumes significant memory bandwidth during autoregressive generation. On CPU-bound systems, memory bandwidth is the primary bottleneck. Quantizing the KV cache from FP16 to Q4 theoretically halves memory bandwidth requirements.
What We Built
We attempted… See the full description on the dataset page: https://huggingface.co/datasets/kleinnner/camus-10-kv-cache-quantization.