MultimodalSDK:基于昇腾生态的多模态大模型推理预处理加速工具

MultimodalSDK

Branch6Tags4
FilesLast commitLast update
3 months ago
5 months ago
2 months ago
7 days ago
3 months ago
3 months ago
2 months ago
3 months ago
2 months ago
5 months ago
2 months ago
4 months ago
3 months ago
5 months ago
2 months ago
3 months ago
3 months ago
5 months ago
3 months ago
4 months ago
7 days ago
5 months ago
4 months ago
5 months ago
4 months ago

Multimodal SDK

Latest Updates

  • [Dec. 30, 2025]: 🚀 Multimodal SDK is released as open source.

Introduction

Multimodal SDK is an LLM inference preprocessing acceleration toolkit optimized for Ascend devices. It provides high-performance interfaces for processing large volumes of multimodal data.
- It includes common preprocessing operations such as image and video loading and decoding, resizing, and cropping.
- It supports conversion between multiple open-source data structures and accelerator library data structures, which makes it easy to use and port.

For details, see Introduction.

Zread     DeepWiki

Directory Structure

├── build_script
│   └── build.sh
├── script
│   ├── help.info
│   ├── install.sh
│   ├── set_env.sh
│   └── uninstall.sh
├── setup.py
├── source
│   └── mm
│       ├── acc
│       │   ├── _impl
│       │   │   └── __init__.py
│       │   ├── __init__.py
│       │   └── wrapper
│       │       ├── data_type.py
│       │       ├── image_wrapper.py
│       │       ├── __init__.py
│       │       ├── tensor_wrapper.py
│       │       ├── util.py
│       │       └── video_wrapper.py
│       ├── adapter
│       │   ├── __init__.py
│       │   ├── internvl2_preprocessor.py
│       │   └── qwen2_vl_preprocessor.py
│       ├── comm
│       │   ├── __init__.py
│       │   └── log.py
│       ├── core
│       │   └── __init__.py
│       ├── __init__.py
│       └── patcher
│           ├── __init__.py
│           └── vllm
│               ├── image_patcher.py
│               ├── __init__.py
│               ├── internvl2_image_processor_patcher.py
│               ├── qwen2_vl_image_processor_patcher.py
│               └── video_patcher.py
└── test
    ├── assets
    │   ├── dog_1920_1080.jpg
    │   ├── dog_1920_1080.png
    │   └── test_aac.mp4
    ├── test_demo.py
    ├── test_image.py
    ├── test_internvl2_preprocessor.py
    ├── test_log.py
    ├── test_qwen2_vl_preprocessor.py
    ├── test_tensor.py
    └── test_video.py

Version Description

The release notes for Multimodal SDK include software version compatibility and the feature changes in each version. See the following table.

Product Version
Ascend HDK 25.5.0
CANN 8.5.0

For details, see Release Notes.

Environment Deployment

This section describes how to install Multimodal SDK. For details, see Installation Guide.

Build Process

This section uses the CANN 8.5.0 compatible release as an example to describe how to build Multimodal SDK from source. The NPU driver, firmware, and CANN package can be downloaded from the Ascend Community website.

  1. Download the build dependencies.

    Note: Building requires Python 3.11. Ensure that this version is installed in the environment.

    # Download the source code
    git clone https://gitcode.com/Ascend/MultimodalSDK.git
    # Enter the project root directory
    cd MultimodalSDK
    
    # Download the makeself dependency into the project root directory. The script automatically applies patches and builds it. The project uses a customized version of makeself for packaging, so you need to download makeself v2.5 and the corresponding patch
    git clone -b v2.5.0.x https://gitcode.com/cann-src-third-party/makeself.git makeself_patch
    git clone -b release-2.5.0 https://gitcode.com/gh_mirrors/ma/makeself.git
    

    The project requires several open-source components. Download the following source code:

    # AccSDK dependency
    cd MultimodalSDK/AccSDK
    wget https://mindcluster.obs.cn-north-4.myhuaweicloud.com/opensource.tar.gz
    # Acc_data dependency
    cd MultimodalSDK/AccSDK/acc_data/3rdparty/pybind
    git clone -b v2.13.6 https://gitcode.com/GitHub_Trending/py/pybind11.git
    cd MultimodalSDK/AccSDK/acc_data/3rdparty/gtest
    git clone -b release-1.11.0 https://gitcode.com/GitHub_Trending/go/googletest.git
    
  2. Run the build.

    Run the following commands to build the project:

    source /path/to/Ascend/ascend-toolkit/set_env.sh
    bash MultimodalSDK/build_script/build_merge.sh
    
  3. The generated run package is located in MultimodalSDK/output as Ascend-mindxsdk-multimodal_${SDK_VERSION}_linux-aarch64.run.

  4. Run the test cases.

    First install lcov 2.0 to collect test coverage and generate a visual report.

    apt update
    apt install -y libcapture-tiny-perl libdatetime-perl libtimedate-perl
    wget https://github.com/linux-test-project/lcov/releases/download/v2.0/lcov-2.0.tar.gz
    tar -xzf lcov-2.0.tar.gz && cd lcov-2.0
    make install
    

    Then run the following command to run the test cases.

    bash MultimodalSDK/build_script/build_merge.sh test
    

Quick Start

Multimodal SDK provides a set of high-performance, easy-to-use interfaces. For details, see Quick Start and Examples and Guidance.

Features

Multimodal SDK provides high-performance interfaces optimized for Ascend devices for LLM inference preprocessing. It includes common preprocessing operations such as image and video loading and decoding, resizing, and cropping. It supports conversion between multiple open-source data structures and accelerator library data structures for ease of use and portability.

API Reference

For the API reference, see the following documents.

Python API Reference

Adapter

Patcher

Function Reference

FAQ

Issue Symptom

Even after lzma is installed, calling torchvision still reports that the lzma module is missing.

Solution

Install the lzma module.

pip install backports.lzma

Go to the Python library directory. The following example uses Python 3.11.4.

cd /xx/xx/python-3.11.4/lib/python3.11

Modify lzma.py as follows.

from _lzma import *
from _lzma import _encode_filter_properties, _decode_filter_properties

Replace it with the following content.

from backports.lzma import *
from backports.lzma import _encode_filter_properties, _decode_filter_properties

Security Statement

  • When you use APIs to read a file, ensure that the file owner is you and that the permissions are no greater than 640 to avoid privilege escalation and other security issues. Software or programs downloaded from external sources may be risky. You are responsible for ensuring their security.
  • Communication matrix: Multimodal SDK does not actively open or depend on any ports. Therefore, it does not involve a communication matrix.
  • Public network addresses: The URLs in the Multimodal SDK installation package are removed after installation and are not accessed. Therefore, they do not pose a risk.

For details, see Security Statement and Appendix.

Disclaimer

  • This repository contains multiple development branches. These branches may include unfinished, experimental, or untested features. Before official release, these branches should not be used in any production environment or in projects that depend on critical business systems. Ensure that you use our official release version to guarantee code stability and security. The project and its contributors are not responsible for any issues, losses, or data corruption caused by using development branches.
  • For the official release, see https://gitcode.com/ascend/MultimodalSDK/releases.

License

Multimodal SDK is licensed under Apache 2.0. The corresponding license text is available in LICENSE.

The documents in the docs directory of Multimodal SDK are licensed under CC-BY 4.0. For details, see LICENSE.

Suggestions and Communication

Everyone is welcome to contribute to the community. If you have any questions or suggestions, submit an issue, and we will reply as soon as possible. Thank you for your support.

Introduction

MultimodalSDK

Customize your domain