This project ports MiniCPM-V-4.6 (GPTQ INT4 quantized) to Ascend 310B1 edge NPUs via the [llama.cpp](https://github.com/ggml-org/llama.cpp) CANN backend.
| Files | Last commit | Last update |
|---|---|---|
| 5 months ago | ||
| 5 months ago | ||
| 5 months ago | ||
| 5 months ago | ||
| 5 months ago | ||
| 5 months ago | ||
| 5 months ago | ||
| 5 months ago | ||
| 5 months ago | ||
| 5 months ago | ||
| 5 months ago | ||
| 5 months ago |
🚀 MiniCPM-V on Ascend NPU
Run MiniCPM-V-4.6 — a state-of-the-art multimodal LLM — on Huawei Ascend 310B1 edge NPUs with custom-operator acceleration.
✨ What This Is
A complete, self-contained deployment stack for serving MiniCPM-V-4.6 (GPTQ INT4 quantized) on Huawei Ascend 310B1 edge NPUs via the llama.cpp CANN backend.
| Component | Description |
|---|---|
| 🔧 12 Custom Ascend C Kernels | GPTQ INT4 dequant, fused linear, vision matmul, Q6_K matvec/dequant, GatedDeltaNet |
| 🩹 310B1 CANN Patches | Fixes for 15+ runtime compatibility issues in llama.cpp |
| 📐 CANN Header Shims | Declarations for operators missing from CANN 9.0.0 on 310B1 |
| 📦 Bundled asc-devkit 9.0.0 | Ascend C build framework — stripped, no separate install needed |
| 🐳 Deployment Tooling | Docker, systemd, tuned launch scripts with 30+ optimized env vars |
The custom operators replace slow CPU fallbacks with NPU-accelerated kernels. The production decode kernel achieves a 9.7x speedup over the naive implementation — from ~208 ms down to ~21.5 ms.
📊 Performance
| Workload | Throughput / Latency |
|---|---|
| 💬 Text decode | ~5.6 tok/s |
| 📝 Text-only (max_tokens=80) | ~17–21 s end-to-end |
| 🖼️ Image + text (640×480, max_tokens=80) | ~21–36 s end-to-end |
| 💾 NPU memory usage | ~10.6 / 11.6 GB |
🏗️ Architecture
User Request (OpenAI-compatible API)
│
▼
┌─────────────────────────────────┐
│ 🦙 llama-server (CANN) │
│ MiniCPM-V-4.6 GGUF │ ← model weights (~6 GB download)
└────────────┬────────────────────┘
│
┌──────┴──────┐
▼ ▼
┌───────────┐ ┌──────────────────┐
│ CANN ops │ │ 🔧 Custom Ops │
│ (builtin) │ │ libgptq_*.so │
└─────┬─────┘ └────────┬─────────┘
│ │
▼ ▼
┌──────────────────────────────────┐
│ CANN 9.0.0 Runtime │
├──────────────────────────────────┤
│ ⚡ Ascend 310B1 NPU (11.6 GB) │
└──────────────────────────────────┘
🔧 Custom Kernels
All 12 kernels in a single file: custom-ops/gptq_dequant_down_proj/src/gptq_dequant_down_proj.cpp
| Kernel | Purpose | Speed |
|---|---|---|
⚡ gptq_linear_down_proj_m1_ntile8_packk_sumx_opt |
Production decode (packK + sum_x) | ~21.5 ms |
🔄 gptq_dequant_down_proj |
Generic GPTQ INT4 → FP16 dequant | — |
🔗 gptq_linear |
Naive fused GPTQ Linear | ~207 ms |
👁️ vision_matmul_f16_f32 |
CLIP tiled matmul (4×4) | — |
🧊 vision_matmul_f16_f16_cube |
Dual-F16 cube matmul (vision) | — |
💨 q6_k_dequant_f16 |
Q6_K stream dequant | saves ~950 MB |
🧪 q6_k_matvec_m1_f32 |
Direct Q6_K matvec (experimental) | — |
🧠 gated_delta_net_m1_f32 |
Mamba/DeltaNet state update | — |
📖 Full operator docs: custom-ops/gptq_dequant_down_proj/README.md
📁 Repository Layout
minicpm-v-ascend/
├── 📦 asc-devkit/ # Ascend C build framework (bundled)
├── 🔧 custom-ops/gptq_dequant_down_proj/ # Custom kernels & Python wrappers
├── 🩹 patches/llama.cpp/ # 310B1 patches for llama.cpp
├── 📐 cann-shim/aclnnop/ # Missing CANN headers
├── 🚀 deployment/ # Launch scripts, Docker, systemd
├── 📜 scripts/ # setup.sh (one-shot build), profiler
├── 📂 models/ # Model download instructions
├── 📖 docs/ # Porting notes & quickstart
└── 🧪 test/ # E2E test & sample request
⚡ Quick Start
📖 Detailed instructions: docs/quickstart.md (EN) | docs/quickstart_zh.md (中文)
git clone https://github.com/HHhzy0903/MiniCPM-V-Ascend-NPU.git
cd MiniCPM-V-Ascend-NPU
# 1️⃣ Download models (~6 GB, one time)
huggingface-cli download openbmb/MiniCPM-V-4_6-gguf \
MiniCPM-V-4_6-Q4_K_M.gguf mmproj-model-f16.gguf \
--local-dir models/MiniCPM-V-4.6-gguf
# 2️⃣ Build everything
bash scripts/setup.sh all
# 3️⃣ Start serving
bash deployment/run.sh
# → http://localhost:1234/v1/chat/completions
🧠 Key Design Decisions
🎯 Three-Tier 310B1 Strategy
CANN 9.0.0 on 310B1 lacks kernel binaries for RmsNorm, Softplus, Im2col, RoPE, FusedInferAttention, and more. We use:
- ✅ Shape-whitelist CANN paths — re-enable only verified operator shapes
- ↩️ CPU fallback — for operators with no 310B1 binary
- 🔧 Custom Ascend C kernels — for performance-critical paths (MatMul, dequant, DeltaNet)
🔬 Dual-F16 Cube MatMul
CLIP vision projections (F32 × F16) lose semantic accuracy with naive cast. We decompose F32 input into two F16 components:
x_hi = fp16(x) # high 16 bits
x_lo = fp16(x - fp16(x_hi)) # residual
y = W @ x_hi + W @ x_lo # preserves F32 precision
Controlled by GGML_CANN_310B_CUSTOM_MATMUL_CUBE_DUAL=on.
⚠️ Known Limitations
| Issue | Details |
|---|---|
| 🐌 Decode bottleneck | Logits projection Q6_K[1024×248094] × F32[1024×1] at ~27 ms/token. Needs tiled/parallel quantized matvec. |
| 🖼️ CLIP attention | Batched attention blocks (72×1024×16) on CPU. Needs custom batched operator. |
| 💾 Memory | F16 weight caches consume ~950 MB. Stream dequant trades speed for memory. |
📚 Related Docs
| Document | Content |
|---|---|
| 📖 Porting Notes | Full 310B1 compatibility history (27 sections) |
| 🚀 Quickstart (EN) | Detailed build & run instructions |
| 🚀 Quickstart (中文) | Chinese build & run guide |
| 🐳 Docker | Containerized deployment |
🔗 Links
Introduction
This project ports MiniCPM-V-4.6 (GPTQ INT4 quantized) to Ascend 310B1 edge NPUs via the [llama.cpp](https://github.com/ggml-org/llama.cpp) CANN backend.
Customize your domain