MultimodalSDK
| Files | Last commit | Last update |
|---|---|---|
| 3 months ago | ||
| 5 months ago | ||
| 2 months ago | ||
| 7 days ago | ||
| 3 months ago | ||
| 3 months ago | ||
| 2 months ago | ||
| 3 months ago | ||
| 2 months ago | ||
| 5 months ago | ||
| 2 months ago | ||
| 4 months ago | ||
| 3 months ago | ||
| 5 months ago | ||
| 2 months ago | ||
| 3 months ago | ||
| 3 months ago | ||
| 5 months ago | ||
| 3 months ago | ||
| 4 months ago | ||
| 7 days ago | ||
| 5 months ago | ||
| 4 months ago | ||
| 5 months ago | ||
| 4 months ago |
Multimodal SDK
- Latest Updates
- Introduction
- Directory Structure
- Version Description
- Environment Deployment
- Build Process
- Quick Start
- Features
- API Reference
- FAQ
- Security Statement
- Disclaimer
- License
- Suggestions and Communication
Latest Updates
- [Dec. 30, 2025]: 🚀 Multimodal SDK is released as open source.
Introduction
Multimodal SDK is an LLM inference preprocessing acceleration toolkit optimized for Ascend devices. It provides high-performance interfaces for processing large volumes of multimodal data.
- It includes common preprocessing operations such as image and video loading and decoding, resizing, and cropping.
- It supports conversion between multiple open-source data structures and accelerator library data structures, which makes it easy to use and port.
For details, see Introduction.
Directory Structure
├── build_script
│ └── build.sh
├── script
│ ├── help.info
│ ├── install.sh
│ ├── set_env.sh
│ └── uninstall.sh
├── setup.py
├── source
│ └── mm
│ ├── acc
│ │ ├── _impl
│ │ │ └── __init__.py
│ │ ├── __init__.py
│ │ └── wrapper
│ │ ├── data_type.py
│ │ ├── image_wrapper.py
│ │ ├── __init__.py
│ │ ├── tensor_wrapper.py
│ │ ├── util.py
│ │ └── video_wrapper.py
│ ├── adapter
│ │ ├── __init__.py
│ │ ├── internvl2_preprocessor.py
│ │ └── qwen2_vl_preprocessor.py
│ ├── comm
│ │ ├── __init__.py
│ │ └── log.py
│ ├── core
│ │ └── __init__.py
│ ├── __init__.py
│ └── patcher
│ ├── __init__.py
│ └── vllm
│ ├── image_patcher.py
│ ├── __init__.py
│ ├── internvl2_image_processor_patcher.py
│ ├── qwen2_vl_image_processor_patcher.py
│ └── video_patcher.py
└── test
├── assets
│ ├── dog_1920_1080.jpg
│ ├── dog_1920_1080.png
│ └── test_aac.mp4
├── test_demo.py
├── test_image.py
├── test_internvl2_preprocessor.py
├── test_log.py
├── test_qwen2_vl_preprocessor.py
├── test_tensor.py
└── test_video.py
Version Description
The release notes for Multimodal SDK include software version compatibility and the feature changes in each version. See the following table.
| Product | Version |
|---|---|
| Ascend HDK | 25.5.0 |
| CANN | 8.5.0 |
For details, see Release Notes.
Environment Deployment
This section describes how to install Multimodal SDK. For details, see Installation Guide.
Build Process
This section uses the CANN 8.5.0 compatible release as an example to describe how to build Multimodal SDK from source. The NPU driver, firmware, and CANN package can be downloaded from the Ascend Community website.
-
Download the build dependencies.
Note: Building requires Python 3.11. Ensure that this version is installed in the environment.
# Download the source code git clone https://gitcode.com/Ascend/MultimodalSDK.git # Enter the project root directory cd MultimodalSDK # Download the makeself dependency into the project root directory. The script automatically applies patches and builds it. The project uses a customized version of makeself for packaging, so you need to download makeself v2.5 and the corresponding patch git clone -b v2.5.0.x https://gitcode.com/cann-src-third-party/makeself.git makeself_patch git clone -b release-2.5.0 https://gitcode.com/gh_mirrors/ma/makeself.gitThe project requires several open-source components. Download the following source code:
# AccSDK dependency cd MultimodalSDK/AccSDK wget https://mindcluster.obs.cn-north-4.myhuaweicloud.com/opensource.tar.gz # Acc_data dependency cd MultimodalSDK/AccSDK/acc_data/3rdparty/pybind git clone -b v2.13.6 https://gitcode.com/GitHub_Trending/py/pybind11.git cd MultimodalSDK/AccSDK/acc_data/3rdparty/gtest git clone -b release-1.11.0 https://gitcode.com/GitHub_Trending/go/googletest.git -
Run the build.
Run the following commands to build the project:
source /path/to/Ascend/ascend-toolkit/set_env.sh bash MultimodalSDK/build_script/build_merge.sh -
The generated run package is located in
MultimodalSDK/outputasAscend-mindxsdk-multimodal_${SDK_VERSION}_linux-aarch64.run. -
Run the test cases.
First install
lcov2.0 to collect test coverage and generate a visual report.apt update apt install -y libcapture-tiny-perl libdatetime-perl libtimedate-perl wget https://github.com/linux-test-project/lcov/releases/download/v2.0/lcov-2.0.tar.gz tar -xzf lcov-2.0.tar.gz && cd lcov-2.0 make installThen run the following command to run the test cases.
bash MultimodalSDK/build_script/build_merge.sh test
Quick Start
Multimodal SDK provides a set of high-performance, easy-to-use interfaces. For details, see Quick Start and Examples and Guidance.
Features
Multimodal SDK provides high-performance interfaces optimized for Ascend devices for LLM inference preprocessing. It includes common preprocessing operations such as image and video loading and decoding, resizing, and cropping. It supports conversion between multiple open-source data structures and accelerator library data structures for ease of use and portability.
API Reference
For the API reference, see the following documents.
FAQ
Issue Symptom
Even after lzma is installed, calling torchvision still reports that the lzma module is missing.
Solution
Install the lzma module.
pip install backports.lzma
Go to the Python library directory. The following example uses Python 3.11.4.
cd /xx/xx/python-3.11.4/lib/python3.11
Modify lzma.py as follows.
from _lzma import *
from _lzma import _encode_filter_properties, _decode_filter_properties
Replace it with the following content.
from backports.lzma import *
from backports.lzma import _encode_filter_properties, _decode_filter_properties
Security Statement
- When you use APIs to read a file, ensure that the file owner is you and that the permissions are no greater than 640 to avoid privilege escalation and other security issues. Software or programs downloaded from external sources may be risky. You are responsible for ensuring their security.
- Communication matrix: Multimodal SDK does not actively open or depend on any ports. Therefore, it does not involve a communication matrix.
- Public network addresses: The URLs in the Multimodal SDK installation package are removed after installation and are not accessed. Therefore, they do not pose a risk.
For details, see Security Statement and Appendix.
Disclaimer
- This repository contains multiple development branches. These branches may include unfinished, experimental, or untested features. Before official release, these branches should not be used in any production environment or in projects that depend on critical business systems. Ensure that you use our official release version to guarantee code stability and security. The project and its contributors are not responsible for any issues, losses, or data corruption caused by using development branches.
- For the official release, see https://gitcode.com/ascend/MultimodalSDK/releases.
License
Multimodal SDK is licensed under Apache 2.0. The corresponding license text is available in LICENSE.
The documents in the docs directory of Multimodal SDK are licensed under CC-BY 4.0. For details, see LICENSE.
Suggestions and Communication
Everyone is welcome to contribute to the community. If you have any questions or suggestions, submit an issue, and we will reply as soon as possible. Thank you for your support.