ExLlamaV3 Supercharges Home AI with Triple-Speed DFlash Decoding
ExLlamaV3 is an inference library that lets you run large language models on consumer graphics cards. It introduces the EXL3 quantization format, which compresses models to very low bitrates while keeping them usable. The library also includes speculative decoding, an OpenAI-compatible API, and support for multimodal models.
The project is maintained by Turboderp, a prolific open-source developer known for pushing the limits of local AI. Recent updates have added DFlash speculative decoding, which can multiply token generation speed by up to three times in coding and agentic tasks. Alongside this, model support has expanded to include Gemma 4, Qwen 3.5, and over 30 other architectures.
Recent updates bring speed gains and new model support
Key Features
- EXL3 quantization format for extreme compression.
- DFlash speculative decoding triples token speed.
- Tensor and expert parallelism across GPUs.
- OpenAI-compatible server with continuous batching.
- Broad model support, including Gemma 4.
- Multimodal handling for vision-language models.
- Cache quantization from 2-bit to 8-bit.
This tool targets anyone running advanced language models on their own hardware, especially with an RTX 4090 or similar GPU. Small teams can host a private AI server without cloud costs, and privacy-focused professionals can keep data local. Hobbyists can explore cutting-edge models that usually need expensive data-center hardware.
Developer notes and future plans
LoRA support and AMD ROCm compatibility are still in development, and Ampere GPU performance at low bitrates needs work. Updates come rapidly, with DFlash and model optimizations arriving in the past few days. Issues are best reported on GitHub, and the Discord is available for community support.
"Llama-3.1-70B-EXL3 is coherent at 1.6 bpw." — Source: GitHub
Project Links
