MiniCPM-V-Ascend-NPU:基于昇腾NPU的MiniCPM-V多模态大模型部署项目

This project ports MiniCPM-V-4.6 (GPTQ INT4 quantized) to Ascend 310B1 edge NPUs via the [llama.cpp](https://github.com/ggml-org/llama.cpp) CANN backend.

Branch1Tags0
FilesLast commitLast update
5 months ago
5 months ago
5 months ago
5 months ago
5 months ago
5 months ago
5 months ago
5 months ago
5 months ago
5 months ago
5 months ago
5 months ago

🚀 MiniCPM-V on Ascend NPU

🇨🇳 中文版本

Run MiniCPM-V-4.6 — a state-of-the-art multimodal LLM — on Huawei Ascend 310B1 edge NPUs with custom-operator acceleration.

Ascend CANN Python License

✨ What This Is

A complete, self-contained deployment stack for serving MiniCPM-V-4.6 (GPTQ INT4 quantized) on Huawei Ascend 310B1 edge NPUs via the llama.cpp CANN backend.

Component Description
🔧 12 Custom Ascend C Kernels GPTQ INT4 dequant, fused linear, vision matmul, Q6_K matvec/dequant, GatedDeltaNet
🩹 310B1 CANN Patches Fixes for 15+ runtime compatibility issues in llama.cpp
📐 CANN Header Shims Declarations for operators missing from CANN 9.0.0 on 310B1
📦 Bundled asc-devkit 9.0.0 Ascend C build framework — stripped, no separate install needed
🐳 Deployment Tooling Docker, systemd, tuned launch scripts with 30+ optimized env vars

The custom operators replace slow CPU fallbacks with NPU-accelerated kernels. The production decode kernel achieves a 9.7x speedup over the naive implementation — from ~208 ms down to ~21.5 ms.

📊 Performance

Workload Throughput / Latency
💬 Text decode ~5.6 tok/s
📝 Text-only (max_tokens=80) ~17–21 s end-to-end
🖼️ Image + text (640×480, max_tokens=80) ~21–36 s end-to-end
💾 NPU memory usage ~10.6 / 11.6 GB

🏗️ Architecture

User Request (OpenAI-compatible API)
              │
              ▼
┌─────────────────────────────────┐
│    🦙 llama-server (CANN)        │
│    MiniCPM-V-4.6 GGUF            │  ← model weights (~6 GB download)
└────────────┬────────────────────┘
             │
      ┌──────┴──────┐
      ▼              ▼
┌───────────┐  ┌──────────────────┐
│ CANN ops  │  │ 🔧 Custom Ops     │
│ (builtin) │  │ libgptq_*.so     │
└─────┬─────┘  └────────┬─────────┘
      │                 │
      ▼                 ▼
┌──────────────────────────────────┐
│     CANN 9.0.0 Runtime            │
├──────────────────────────────────┤
│  ⚡ Ascend 310B1 NPU (11.6 GB)   │
└──────────────────────────────────┘

🔧 Custom Kernels

All 12 kernels in a single file: custom-ops/gptq_dequant_down_proj/src/gptq_dequant_down_proj.cpp

Kernel Purpose Speed
⚡ gptq_linear_down_proj_m1_ntile8_packk_sumx_opt Production decode (packK + sum_x) ~21.5 ms
🔄 gptq_dequant_down_proj Generic GPTQ INT4 → FP16 dequant —
🔗 gptq_linear Naive fused GPTQ Linear ~207 ms
👁️ vision_matmul_f16_f32 CLIP tiled matmul (4×4) —
🧊 vision_matmul_f16_f16_cube Dual-F16 cube matmul (vision) —
💨 q6_k_dequant_f16 Q6_K stream dequant saves ~950 MB
🧪 q6_k_matvec_m1_f32 Direct Q6_K matvec (experimental) —
🧠 gated_delta_net_m1_f32 Mamba/DeltaNet state update —

📖 Full operator docs: custom-ops/gptq_dequant_down_proj/README.md

📁 Repository Layout

minicpm-v-ascend/
├── 📦 asc-devkit/                         # Ascend C build framework (bundled)
├── 🔧 custom-ops/gptq_dequant_down_proj/  # Custom kernels & Python wrappers
├── 🩹 patches/llama.cpp/                   # 310B1 patches for llama.cpp
├── 📐 cann-shim/aclnnop/                   # Missing CANN headers
├── 🚀 deployment/                          # Launch scripts, Docker, systemd
├── 📜 scripts/                             # setup.sh (one-shot build), profiler
├── 📂 models/                              # Model download instructions
├── 📖 docs/                                # Porting notes & quickstart
└── 🧪 test/                                # E2E test & sample request

⚡ Quick Start

📖 Detailed instructions: docs/quickstart.md (EN) | docs/quickstart_zh.md (中文)

git clone https://github.com/HHhzy0903/MiniCPM-V-Ascend-NPU.git
cd MiniCPM-V-Ascend-NPU

# 1️⃣ Download models (~6 GB, one time)
huggingface-cli download openbmb/MiniCPM-V-4_6-gguf \
  MiniCPM-V-4_6-Q4_K_M.gguf mmproj-model-f16.gguf \
  --local-dir models/MiniCPM-V-4.6-gguf

# 2️⃣ Build everything
bash scripts/setup.sh all

# 3️⃣ Start serving
bash deployment/run.sh
# → http://localhost:1234/v1/chat/completions

🧠 Key Design Decisions

🎯 Three-Tier 310B1 Strategy

CANN 9.0.0 on 310B1 lacks kernel binaries for RmsNorm, Softplus, Im2col, RoPE, FusedInferAttention, and more. We use:

  1. ✅ Shape-whitelist CANN paths — re-enable only verified operator shapes
  2. ↩️ CPU fallback — for operators with no 310B1 binary
  3. 🔧 Custom Ascend C kernels — for performance-critical paths (MatMul, dequant, DeltaNet)

🔬 Dual-F16 Cube MatMul

CLIP vision projections (F32 × F16) lose semantic accuracy with naive cast. We decompose F32 input into two F16 components:

x_hi = fp16(x)                 # high 16 bits
x_lo = fp16(x - fp16(x_hi))    # residual
y = W @ x_hi + W @ x_lo        # preserves F32 precision

Controlled by GGML_CANN_310B_CUSTOM_MATMUL_CUBE_DUAL=on.

⚠️ Known Limitations

Issue Details
🐌 Decode bottleneck Logits projection Q6_K[1024×248094] × F32[1024×1] at ~27 ms/token. Needs tiled/parallel quantized matvec.
🖼️ CLIP attention Batched attention blocks (72×1024×16) on CPU. Needs custom batched operator.
💾 Memory F16 weight caches consume ~950 MB. Stream dequant trades speed for memory.
Document Content
📖 Porting Notes Full 310B1 compatibility history (27 sections)
🚀 Quickstart (EN) Detailed build & run instructions
🚀 Quickstart (中文) Chinese build & run guide
🐳 Docker Containerized deployment

Introduction

This project ports MiniCPM-V-4.6 (GPTQ INT4 quantized) to Ascend 310B1 edge NPUs via the [llama.cpp](https://github.com/ggml-org/llama.cpp) CANN backend.

Customize your domain