Unsloth Shrinks Gemma 4 Into Local Gemma-4-12B-It-GGUF Package

    
        By vramkickedin    
     | 
    
            June 15, 2026 at 1:36 pm        
    
     | 
    
        2 min read

Google DeepMind recently released Gemma 4, and the Gemma-4-12B-It-GGUF version from Unsloth makes the 12 billion parameter model ready for local use. This quantized format shrinks the model files dramatically, allowing it to run smoothly on prosumer GPUs and high-end workstations. The 12B variant is notable because it uses a unified, encoder-free architecture that handles text, images, and audio together without needing separate processing pipelines.

Unsloth took the official Google model and converted it to the GGUF format, which is essential for running large language models on local machines with limited VRAM. They also verified that the model works correctly with advanced features like tool-use and speculative decoding support. The result is a highly capable multimodal AI that fits within the hardware budget of serious hobbyists and small professional teams.

Handles text, vision, and audio natively

Model highlights

Unified architecture processes text, images, and audio.
256k token context window for large documents.
Runs efficiently on prosumer consumer GPUs.
Supports function calling for agent workflows.
Multilingual support for over 140 languages.
Built-in reasoning mode with adjustable thinking.
Available in multiple quantizations.

This model offers a strong alternative for privacy-conscious professionals who process sensitive documents and need a multimodal solution they control. Small agencies can deploy it internally to handle tasks like image analysis, speech transcription, and text generation without sending data to cloud APIs. Developers and power users can explore its function calling capabilities to build agent workflows that coordinate across vision, audio, and text inputs.

What to know before using it

The model requires a recent build of the llama.cpp software to run, and for best results with audio, you should use a 16kHz mono WAV file. On the Hugging Face page, Unsloth included sample code for text, vision, and audio prompting so you can verify everything works without guesswork. Performance falls between Google's smaller E4B model and the larger 26B and 31B offerings, making this a practical middle ground for local inference.