MiniCPM-V-4.6-OrangePi Boots Full Multimodal AI On A Sub-100 Dollar Board
A new engine brings the MiniCPM-V-4.6 vision-language model to a $100 edge board. The MiniCPM-V-4.6-OrangePi project is a from-scratch C++ inference engine that runs the full 4.6B-parameter multimodal model on the Ascend 310B NPU inside an Orange Pi AIPro 20T. Text and image chat both execute entirely on the NPU through a single subprocess, leaving Python only to handle tokenization and image pre-processing. No torch_npu dependency touches the hot inference path.
Developer lvyufeng built this release to deliver a completely local, offline AI experience on low-cost hardware. The engine ports the entire SigLIP vision tower plus the hybrid-attention language model to C++ and AscendC, validated end-to-end against the official Hugging Face reference. Doing so removes the need for a discrete GPU and makes the board a self-contained multimodal system.
Fully offline multimodal AI on Orange Pi
What the engine delivers
- Custom C++/AscendC engine for the NPU.
- Text and image chat run fully on NPU.
- No torch_npu dependency during inference.
- SigLIP vision tower ported and validated.
- Multi-slice support for high-resolution images.
- Gradio web UI with token-by-token streaming.
- Per-conversation prefix caching for reuse.
- Decode speed of 5.9 tokens per second.
Hobbyists, privacy-conscious professionals, and small agencies can all benefit from this setup. It turns a sub-$100 board into a quiet, always-available AI that can answer questions about images and documents without sending any data to the cloud. The engine’s single-process design and minimal dependencies also make it a practical base for custom offline tools.
Developer notes and roadmap
The engine currently runs only single-batch greedy decode, and multi-turn chat latency grows because the existing chat template forces a partial cache rebuild after each turn. Future plans tackle speed and features head-on: weight quantization (int8/int4) is expected to roughly double or triple the current 5.9 tokens/s, true multi-image support is queued, and beam search / top-p sampling remain on the list. The vision tower matches the CPU reference within a max absolute difference of 0.0098, confirming the NPU implementation is reliable.
“Three rounds of cube-unit / custom-kernel work took single-batch decode from 2.88 → 5.90 tokens/s (~2×) on the full 24-layer hybrid linear/full attention LM.” — Source: GitHub
Project Links
