KVarN Unlocks 5x More Context For Agentic LLMs Without Accuracy Loss
KVarN is a new KV-cache quantization method that expands context capacity for large language models. It compresses keys and values to 4-bit and 2-bit without calibration, yielding up to 3-5x more context and roughly 1.3x the throughput of FP16. It targets agentic and long-context workloads where memory bottlenecks are common.
Huawei released KVarN as a vLLM fork under Apache 2.0. Other compression methods trade speed or accuracy for memory, but KVarN retains both. That makes it viable for production serving with many long-context requests.
Designed for agentic and long-context workloads
Key Features
- Up to 3-5x more KV-cache capacity.
- Up to ~1.3x throughput versus FP16.
- FP16-level accuracy without calibration data.
- One flag to enable in vLLM.
- Supports MLA, hybrid, and speculative decoding.
- 4-bit keys, 2-bit values compression preset.
- Compatible with weight-quantized model checkpoints.
Local LLM users can now handle much longer prompts and conversations without adding GPUs. Teams using vLLM for private inference gain memory savings instantly, with no model changes. Agentic applications with multi-step reasoning benefit from maintained speed and full-precision quality.
How KVarN avoids accuracy loss
The method applies a Hadamard rotation and iterative variance normalization to spread out channel outliers, preventing error buildup during decoding. Built on vLLM 0.23.0 with Triton kernels, it requires no calibration data or model retraining. The 4-bit/2-bit preset matched FP16 accuracy on AIME25 and MATH500, beating TurboQuant at low bit-widths.
"KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy." — Source: GitHub
Project Links
