已合并
chore: add VS Code devcontainer configuration. #2911
xubin创建于 7月31日
chore: add VS Code devcontainer configuration. #2911
已合并
共 10 个文件变更+380-158
| @@ -0,0 +1,65 @@ | |||
| 1 | +# MindSpeed-MM Dev Container | ||
| 2 | + | ||
| 3 | +在 VS Code 中一键启动 MindSpeed-MM 开发环境。 | ||
| 4 | + | ||
| 5 | +## 工作模式 | ||
| 6 | + | ||
| 7 | +[devcontainer.json](./devcontainer.json) 默认基于预先构建好的镜像(`image` 字段),拉取即用,无需本地构建。 | ||
| 8 | + | ||
| 9 | +> 配置文件中填写的 `image` 仅为示例,请根据实际需求选择所需镜像。镜像获取方式与更多信息请查看 [docker/OVERVIEW.zh.md](../docker/OVERVIEW.zh.md)。 | ||
| 10 | + | ||
| 11 | +如需自行修改构建参数,请在 `devcontainer.json` 中: | ||
| 12 | + | ||
| 13 | +1. 注释掉 `image` 行 | ||
| 14 | +2. 取消注释 `build` 块与 `initializeCommand` 行 | ||
| 15 | + | ||
| 16 | +即可切换为从 [../docker/Dockerfile](../docker/Dockerfile) 本地构建的模式。 | ||
| 17 | + | ||
| 18 | +## 前置要求 | ||
| 19 | + | ||
| 20 | +- [VS Code](https://code.visualstudio.com/) + [Dev Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) 扩展 | ||
| 21 | +- 本地安装 Docker | ||
| 22 | +- 宿主机已安装昇腾 NPU 驱动,用于容器内访问 NPU | ||
| 23 | + | ||
| 24 | +## 使用方式 | ||
| 25 | + | ||
| 26 | +1. 用 VS Code 打开本项目根目录 | ||
是不是需要先ssh到服务器? ![]() ![]() | |||
| 27 | +2. 按 `F1` 打开命令面板,执行 **Dev Containers: Reopen in Container** | ||
| 28 | +3. 首次启动会拉取预构建镜像(默认模式)或本地构建镜像(自建模式,约 20-40 分钟,取决于网络),完成后进入开发环境 | ||
| 29 | + | ||
| 30 | +> 自建模式下,`initializeCommand` 会调用 [prepare-context.sh](./prepare-context.sh),自动下载 Miniconda 并准备 Dockerfile 所需的构建上下文文件,无需手动操作。 | ||
| 31 | + | ||
| 32 | +## 构建参数调整(自建模式) | ||
| 33 | + | ||
| 34 | +编辑 [devcontainer.json](./devcontainer.json) 的 `build.args` 即可调整构建参数: | ||
| 35 | +如果参数只是一个示例,请根据实际需求填写。 | ||
| 36 | + | ||
| 37 | +| 参数 | 默认值 | 说明 | | ||
| 38 | +| --- | --- | --- | | ||
| 39 | +| `NPU_TYPE` | `a3` | NPU 型号:`910b` 或 `a3` | | ||
| 40 | +| `OS` | `openeuler24.03` | 操作系统:`openeuler24.03` 或 `ubuntu22.04` | | ||
| 41 | +| `BASE_IMAGE_VERSION` | `9.0.0` | CANN 基础镜像版本 | | ||
| 42 | +| `TORCH_VERSION` | `2.7.1` | PyTorch 版本(在线安装) | | ||
| 43 | +| `TORCH_NPU_VERSION` | `2.7.1.post8` | torch_npu 版本(在线安装) | | ||
| 44 | +| `MINDSPEED_MM_BRANCH` | `26.1.0` | MindSpeed-MM 分支 | | ||
| 45 | + | ||
| 46 | +## NPU 设备配置 | ||
| 47 | + | ||
| 48 | +默认挂载 `/dev/davinci0` 与 `/dev/davinci1`。如需使用其他卡号,修改 `devcontainer.json` 中 `runArgs` 的 `--device=/dev/davinci2` 索引,或新增 `--device=/dev/davinci2` 等。 | ||
| 49 | + | ||
| 50 | +宿主机无 NPU 时,注释掉 `runArgs` 与 `mounts` 即可作为纯代码编辑环境使用。 | ||
| 51 | + | ||
| 52 | +## 工作区说明 | ||
| 53 | + | ||
| 54 | +- 本地源码通过 bind mount 挂载到容器 `/workspace/MindSpeed-MM`,编辑实时生效 | ||
| 55 | +- 容器创建后自动执行 `pip install -e . --no-deps` 以可编辑模式安装项目,并配置 pre-commit | ||
| 56 | +- 默认 Python 解释器:`/opt/conda/bin/python`(conda base 环境) | ||
| 57 | + | ||
| 58 | +## 常见问题排查 | ||
| 59 | + | ||
| 60 | +- **Miniconda 下载超时(自建模式)** | ||
| 61 | + 手动下载 [Miniconda3-py311_26.1.1-1-Linux-${ARCH}.sh](https://repo.anaconda.com/miniconda/) 放到 `docker/downloads/` 目录后重新构建 | ||
| 62 | + | ||
| 63 | +- **decord 依赖下载失败(仅 ARM 架构需要 & 自建模式)** | ||
| 64 | + decord 编译依赖较多且常因网络问题下载失败,可用浏览器手动下载后放入 `docker/decord_deps/` 目录,再重新执行 **Reopen in Container**。下载链接参考 [docker/common/download_decord_deps.sh](../docker/common/download_decord_deps.sh)。 | ||
| 65 | + 完成后 `docker/decord_deps/` 目录结构应包含上述全部文件与目录。x86_64 架构无需下载这些依赖(decord 在镜像构建阶段一已通过 pip 安装)。 | ||
| @@ -0,0 +1,85 @@ | |||
| 1 | +{ | ||
| 2 | + "name": "MindSpeed-MM Dev", | ||
| 3 | + "image": "swr.cn-south-1.myhuaweicloud.com/ascendhub/mindspeed-mm:26.0.0-a3-openeuler24.03-py3.11-aarch64", | ||
| 4 | + // // ----- build 模式(从 Dockerfile 构建时取消注释,注释上方 image 行) ----- | ||
| 5 | + // "build": { | ||
| 6 | + // "dockerfile": "../docker/Dockerfile", | ||
| 7 | + // "context": "../docker", | ||
| 8 | + // // 构建时使用 host 网络,使构建容器可通过 127.0.0.1:6268 访问宿主机代理 | ||
| 9 | + // "options": ["--network=host"], | ||
| 10 | + // "args": { | ||
| 11 | + // // PyTorch / torch_npu 版本(在线安装) | ||
| 12 | + // "TORCH_VERSION": "2.7.1", | ||
| 13 | + // "TORCH_NPU_VERSION": "2.7.1.post8", | ||
| 14 | + // // MindSpeed-MM 分支(dev 镜像会克隆该分支;工作区挂载会覆盖克隆内容) | ||
| 15 | + // "MINDSPEED_MM_BRANCH": "26.1.0", | ||
| 16 | + // // 构建开发镜像(非 CI 多版本镜像) | ||
| 17 | + // "BUILD_CI": "false", | ||
| 18 | + // // CANN 基础镜像版本 | ||
| 19 | + // "BASE_IMAGE": "swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11", | ||
| 20 | + // // 以下参数对应 prepare-context.sh 暂存到上下文中的文件/目录名 | ||
| 21 | + // "MINICONDA_SH": "miniconda.sh", | ||
| 22 | + // "DECORD_SCRIPT": "install_decord_on_arm.sh", | ||
| 23 | + // "DECORD_DEPS_DIR": "decord_deps", | ||
| 24 | + // "TORCH_WHL_DIR": "torch_wheels" | ||
| 25 | + // } | ||
| 26 | + // }, | ||
| 27 | + // // build 模式专用:构建前在宿主机执行,下载 Miniconda、暂存辅助脚本 | ||
| 28 | + // "initializeCommand": "bash .devcontainer/prepare-context.sh", | ||
| 29 | + // 将本地源码挂载到镜像默认工作目录,覆盖构建时克隆的仓库,实现实时编辑 | ||
| 30 | + "workspaceMount": "source=${localWorkspaceFolder},target=/workspace/MindSpeed-MM,type=bind,consistency=cached", | ||
| 31 | + "workspaceFolder": "/workspace/MindSpeed-MM", | ||
| 32 | + "remoteUser": "root", | ||
| 33 | + // 昇腾 NPU 设备访问。请按宿主机实际卡号调整 /dev/davinci0 索引。 | ||
| 34 | + // 若宿主机无 NPU,注释掉 runArgs 与 mounts 即可作为纯代码编辑环境使用。 | ||
| 35 | + "runArgs": [ | ||
| 36 | + "--network=host", | ||
| 37 | + "--device=/dev/davinci0", | ||
| 38 | + "--device=/dev/davinci1", | ||
| 39 | + "--device=/dev/davinci_manager", | ||
| 40 | + "--device=/dev/devmm_svm", | ||
| 41 | + "--device=/dev/hisi_hdc", | ||
| 42 | + "--ipc=host" | ||
| 43 | + ], | ||
| 44 | + "mounts": [ | ||
| 45 | + "source=/usr/local/dcmi,target=/usr/local/dcmi,type=bind", | ||
| 46 | + "source=/usr/local/bin/npu-smi,target=/usr/local/bin/npu-smi,type=bind", | ||
| 47 | + "source=/usr/local/Ascend/driver/lib64,target=/usr/local/Ascend/driver/lib64,type=bind", | ||
| 48 | + "source=/usr/local/Ascend/driver/version.info,target=/usr/local/Ascend/driver/version.info,type=bind", | ||
| 49 | + "source=/etc/ascend_install.info,target=/etc/ascend_install.info,type=bind" | ||
| 50 | + ], | ||
| 51 | + // 容器创建后:以可编辑模式安装项目(依赖已在镜像中预装),并配置 pre-commit | ||
| 52 | + "postCreateCommand": "pip install -e . --no-deps && pip install pre-commit && pre-commit install", | ||
| 53 | + "customizations": { | ||
| 54 | + "vscode": { | ||
| 55 | + "extensions": [ | ||
| 56 | + // Python 核心 | ||
| 57 | + "ms-python.python", | ||
| 58 | + "ms-python.vscode-pylance", | ||
| 59 | + "ms-python.debugpy", | ||
| 60 | + // Ruff(linting + formatting) | ||
| 61 | + "charliermarsh.ruff", | ||
| 62 | + // 通用 | ||
| 63 | + "eamodio.gitlens", | ||
| 64 | + "yzhang.markdown-all-in-one", | ||
| 65 | + "tamasfe.even-better-toml", | ||
| 66 | + "redhat.vscode-yaml" | ||
| 67 | + ], | ||
| 68 | + "settings": { | ||
| 69 | + "python.defaultInterpreterPath": "/opt/conda/bin/python", | ||
| 70 | + "python.terminal.activateEnvironment": true, | ||
| 71 | + "python.analysis.typeCheckingMode": "basic", | ||
| 72 | + "python.analysis.autoImportCompletions": true, | ||
| 73 | + "editor.formatOnSave": true, | ||
| 74 | + "editor.rulers": [120], | ||
| 75 | + "files.trimTrailingWhitespace": true, | ||
| 76 | + "files.insertFinalNewline": true, | ||
| 77 | + "yaml.schemas": {}, | ||
| 78 | + "search.exclude": { | ||
| 79 | + "**/kernel_meta_*": true, | ||
| 80 | + "**/.cache": true | ||
| 81 | + } | ||
| 82 | + } | ||
| 83 | + } | ||
| 84 | + } | ||
| 85 | +} | ||
| @@ -0,0 +1,88 @@ | |||
| 1 | +#!/bin/bash | ||
| 2 | +set -e | ||
| 3 | + | ||
| 4 | +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| 5 | +PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)" | ||
| 6 | +DOCKER_DIR="${PROJECT_ROOT}/docker" | ||
| 7 | +COMMON_DIR="${DOCKER_DIR}/common" | ||
| 8 | +MODEL_INSTALL_DIR="${DOCKER_DIR}/scripts/model_install" | ||
| 9 | + | ||
| 10 | +# 可通过环境变量覆盖默认值 | ||
| 11 | +OS="${OS:-openeuler24.03}" | ||
| 12 | + | ||
| 13 | +# 根据 OS 推导 OS_FAMILY 与软件源配置脚本 | ||
| 14 | +case "$OS" in | ||
| 15 | + openeuler*) OS_FAMILY="openeuler"; REPO_SCRIPT="configure_yum_repo.sh" ;; | ||
| 16 | + ubuntu*) OS_FAMILY="ubuntu"; REPO_SCRIPT="configure_apt_repo.sh" ;; | ||
| 17 | + *) echo "ERROR: 不支持的操作系统: $OS(可选: openeuler24.03 / ubuntu22.04)"; exit 1 ;; | ||
| 18 | +esac | ||
| 19 | + | ||
| 20 | +# 检测 CPU 架构 | ||
| 21 | +ARCH="$(uname -m)" | ||
| 22 | +if [ "$ARCH" != "x86_64" ] && [ "$ARCH" != "aarch64" ]; then | ||
| 23 | + echo "ERROR: 不支持的 CPU 架构: $ARCH(仅支持 x86_64 / aarch64)" | ||
| 24 | + exit 1 | ||
| 25 | +fi | ||
| 26 | + | ||
| 27 | +echo "==========================================" | ||
| 28 | +echo "准备 devcontainer 构建上下文" | ||
| 29 | +echo " OS: ${OS}" | ||
| 30 | +echo " OS_FAMILY: ${OS_FAMILY}" | ||
| 31 | +echo " 架构: ${ARCH}" | ||
| 32 | +echo " 上下文目录: ${DOCKER_DIR}" | ||
| 33 | +echo "==========================================" | ||
| 34 | + | ||
| 35 | +cd "${DOCKER_DIR}" | ||
| 36 | + | ||
| 37 | +# 1. 下载并暂存 Miniconda 安装器(统一命名为 miniconda.sh,便于构建参数固定) | ||
| 38 | +MINICONDA_FILE="Miniconda3-py311_26.1.1-1-Linux-${ARCH}.sh" | ||
| 39 | +if [ ! -f "miniconda.sh" ]; then | ||
| 40 | + if [ -f "downloads/${MINICONDA_FILE}" ]; then | ||
| 41 | + echo ">>> 复用已下载的 Miniconda: downloads/${MINICONDA_FILE}" | ||
| 42 | + else | ||
| 43 | + echo ">>> 下载 Miniconda (${ARCH})..." | ||
| 44 | + bash "${COMMON_DIR}/download_miniconda.sh" "${DOCKER_DIR}/downloads" "${ARCH}" | ||
| 45 | + fi | ||
| 46 | + cp "downloads/${MINICONDA_FILE}" "miniconda.sh" | ||
| 47 | +else | ||
| 48 | + echo ">>> Miniconda 已存在: miniconda.sh" | ||
| 49 | +fi | ||
| 50 | + | ||
| 51 | +# 2. 暂存 common_functions.sh | ||
| 52 | +cp "${COMMON_DIR}/common_functions.sh" "common_functions.sh" | ||
| 53 | + | ||
| 54 | +# 3. 暂存软件源配置脚本为 configure_repo.sh(Dockerfile 固定 COPY 该名称) | ||
| 55 | +cp "${COMMON_DIR}/${REPO_SCRIPT}" "configure_repo.sh" | ||
| 56 | + | ||
| 57 | +# 4. 暂存模型环境安装脚本到 install_scripts/ | ||
| 58 | +mkdir -p "install_scripts" | ||
| 59 | +for script in "${MODEL_INSTALL_DIR}"/install_*.sh; do | ||
| 60 | + [ -f "$script" ] && cp "$script" "install_scripts/" | ||
| 61 | +done | ||
| 62 | + | ||
| 63 | +# 5. 暂存 decord 安装脚本(ARM 构建需要;x86 构建会被跳过,但文件必须存在) | ||
| 64 | +cp "${COMMON_DIR}/install_decord_on_arm.sh" "install_decord_on_arm.sh" | ||
| 65 | + | ||
| 66 | +# 6. 创建 torch_wheels 占位目录(空目录 = 在线安装 PyTorch) | ||
| 67 | +mkdir -p "torch_wheels" | ||
| 68 | +touch "torch_wheels/.placeholder" | ||
| 69 | + | ||
| 70 | +# 7. 创建 decord_deps 目录 | ||
| 71 | +# - x86_64: decord 已在阶段一通过 pip 安装,仅创建占位目录满足 COPY 路径 | ||
| 72 | +# - aarch64: 下载 decord 源码编译依赖 | ||
| 73 | +if [ "$ARCH" = "aarch64" ]; then | ||
| 74 | + if [ ! -d "decord_deps" ] || [ -z "$(ls -A decord_deps 2>/dev/null)" ]; then | ||
| 75 | + echo ">>> 下载 decord 编译依赖(ARM)..." | ||
| 76 | + rm -rf "decord_deps" | ||
| 77 | + bash "${COMMON_DIR}/download_decord_deps.sh" "${DOCKER_DIR}/decord_deps" | ||
| 78 | + else | ||
| 79 | + echo ">>> decord 依赖已存在: decord_deps" | ||
| 80 | + fi | ||
| 81 | +else | ||
| 82 | + mkdir -p "decord_deps" | ||
| 83 | + touch "decord_deps/.placeholder" | ||
| 84 | +fi | ||
| 85 | + | ||
| 86 | +echo "==========================================" | ||
| 87 | +echo "构建上下文准备完成" | ||
| 88 | +echo "==========================================" | ||
| @@ -158,4 +158,15 @@ mindspeed | |||
| 158 | /tests/st/run_logs/ | 158 | /tests/st/run_logs/ |
| 159 | 159 | ||
| 160 | fusion_result.json | 160 | fusion_result.json |
| 161 | -cache_dir/ | 161 | +cache_dir/ |
| 162 | + | ||
| 163 | +# devcontainer: Build context artifacts staged to docker/ via .devcontainer/prepare-context.sh | ||
| 164 | +/docker/miniconda.sh | ||
| 165 | +/docker/Miniconda3-*.sh | ||
| 166 | +/docker/common_functions.sh | ||
| 167 | +/docker/configure_repo.sh | ||
| 168 | +/docker/install_decord_on_arm.sh | ||
| 169 | +/docker/install_scripts/ | ||
| 170 | +/docker/torch_wheels/ | ||
| 171 | +/docker/decord_deps/ | ||
| 172 | +/docker/downloads/ | ||
| @@ -19,6 +19,7 @@ repos: | |||
| 19 | - id: check-merge-conflict | 19 | - id: check-merge-conflict |
| 20 | - id: detect-private-key | 20 | - id: detect-private-key |
| 21 | - id: check-json | 21 | - id: check-json |
| 22 | + exclude: \.devcontainer/devcontainer\.json$ | ||
| 22 | 23 | ||
| 23 | # -------------------------- Python 核心检查 -------------------------- | 24 | # -------------------------- Python 核心检查 -------------------------- |
| 24 | # codespell | 25 | # codespell |
| @@ -17,20 +17,15 @@ | |||
| 17 | # then produce the final image. | 17 | # then produce the final image. |
| 18 | # ============================================ | 18 | # ============================================ |
| 19 | 19 | ||
| 20 | -# --- Base image arguments --- | 20 | +# BASE_IMAGE: Full base image URL; when empty, falls back to the hardcoded default below. |
| 21 | -# OS: Operating system type, e.g. openeuler24.03 or ubuntu22.04 | 21 | +# NOTE: Do not use nested ${VAR} inside ${BASE_IMAGE:-default} — Docker's :- parser |
| 22 | -ARG OS=openeuler24.03 | 22 | +# stops at the first '}' and produces a malformed image name. |
| 23 | -# BASE_IMAGE_VERSION: CANN base image version tag | ||
| 24 | -ARG BASE_IMAGE_VERSION=9.0.0-beta.2 | ||
| 25 | -# NPU_TYPE: Ascend NPU model (910b / a3 / etc.) | ||
| 26 | -ARG NPU_TYPE=910b | ||
| 27 | -# BASE_IMAGE: Full base image URL; the default splicing rule is used when empty | ||
| 28 | ARG BASE_IMAGE="" | 23 | ARG BASE_IMAGE="" |
| 29 | 24 | ||
| 30 | # Stage 1 (base): Build the base runtime environment. | 25 | # Stage 1 (base): Build the base runtime environment. |
| 31 | # Pulls the CANN base image from the Huawei Cloud Ascend registry by default; | 26 | # Pulls the CANN base image from the Huawei Cloud Ascend registry by default; |
| 32 | # comes with CANN and Python 3.11 pre-installed. | 27 | # comes with CANN and Python 3.11 pre-installed. |
| 33 | -FROM ${BASE_IMAGE:-swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:${BASE_IMAGE_VERSION}-${NPU_TYPE}-${OS}-py3.11} AS base | 28 | +FROM ${BASE_IMAGE:-swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11} AS base |
| 34 | 29 | ||
| 35 | # Switch to root (the CANN base image defaults to a non-root user) | 30 | # Switch to root (the CANN base image defaults to a non-root user) |
| 36 | USER root | 31 | USER root |
| @@ -40,11 +35,9 @@ USER root | |||
| 40 | ARG MINICONDA_SH | 35 | ARG MINICONDA_SH |
| 41 | # TORCH_VERSION / TORCH_NPU_VERSION: PyTorch and torch_npu versions | 36 | # TORCH_VERSION / TORCH_NPU_VERSION: PyTorch and torch_npu versions |
| 42 | ARG TORCH_VERSION=2.7.1 | 37 | ARG TORCH_VERSION=2.7.1 |
| 43 | -ARG TORCH_NPU_VERSION=2.7.1 | 38 | +ARG TORCH_NPU_VERSION=2.7.1.post8 |
| 44 | # TORCH_WHL_DIR: Directory containing local PyTorch .whl files (optional, for offline install) | 39 | # TORCH_WHL_DIR: Directory containing local PyTorch .whl files (optional, for offline install) |
| 45 | ARG TORCH_WHL_DIR=torch_wheels | 40 | ARG TORCH_WHL_DIR=torch_wheels |
| 46 | -# OS_FAMILY: Operating system family (openeuler / ubuntu), used to select the package manager | ||
| 47 | -ARG OS_FAMILY | ||
| 48 | 41 | ||
| 49 | # Detect CPU architecture and print info (x86_64 / aarch64) | 42 | # Detect CPU architecture and print info (x86_64 / aarch64) |
| 50 | RUN ARCH=$(uname -m) && \ | 43 | RUN ARCH=$(uname -m) && \ |
| @@ -60,18 +53,6 @@ RUN ARCH=$(uname -m) && \ | |||
| 60 | fi && \ | 53 | fi && \ |
| 61 | echo "==========================================" | 54 | echo "==========================================" |
| 62 | 55 | ||
| 63 | -# Configure DNS resolution (ensures the container can access the public internet). | ||
| 64 | -# Uses: domestic DNS (114) -> Google DNS (8.8.8.8) -> Alibaba DNS (223.5.5.5) | ||
| 65 | -RUN if [ -f /etc/resolv.conf ]; then \ | ||
| 66 | - echo "Existing resolv.conf:" && cat /etc/resolv.conf; \ | ||
| 67 | - else \ | ||
| 68 | - echo "No resolv.conf found, creating one..."; \ | ||
| 69 | - fi && \ | ||
| 70 | - echo "nameserver 114.114.114.114" > /etc/resolv.conf && \ | ||
| 71 | - echo "nameserver 8.8.8.8" >> /etc/resolv.conf && \ | ||
| 72 | - echo "nameserver 223.5.5.5" >> /etc/resolv.conf && \ | ||
| 73 | - echo "Updated resolv.conf:" && cat /etc/resolv.conf | ||
| 74 | - | ||
| 75 | # Configure system package repositories (via external script for yum/apt sources) | 56 | # Configure system package repositories (via external script for yum/apt sources) |
| 76 | COPY configure_repo.sh /tmp/configure_repo.sh | 57 | COPY configure_repo.sh /tmp/configure_repo.sh |
| 77 | RUN chmod +x /tmp/configure_repo.sh && \ | 58 | RUN chmod +x /tmp/configure_repo.sh && \ |
| @@ -82,8 +63,16 @@ RUN chmod +x /tmp/configure_repo.sh && \ | |||
| 82 | # - Build toolchain: gcc/g++, make, cmake (for compiling C++ extensions like decord) | 63 | # - Build toolchain: gcc/g++, make, cmake (for compiling C++ extensions like decord) |
| 83 | # - Network tools: git, wget, curl | 64 | # - Network tools: git, wget, curl |
| 84 | # - Media processing: ffmpeg and related dev libraries (for video decoding) | 65 | # - Media processing: ffmpeg and related dev libraries (for video decoding) |
| 85 | -RUN echo "Installing system dependencies..." && \ | 66 | +# OS family is detected from /etc/os-release (ID field), so no build arg is needed. |
| 86 | - if [ "$OS_FAMILY" = "openeuler" ]; then \ | 67 | +RUN . /etc/os-release && \ |
| 68 | + DETECTED_OS_FAMILY="" && \ | ||
| 69 | + ID_LOWER=$(echo "$ID" | tr '[:upper:]' '[:lower:]') && \ | ||
| 70 | + case "$ID_LOWER" in \ | ||
| 71 | + *openeuler*) DETECTED_OS_FAMILY="openeuler" ;; \ | ||
| 72 | + *ubuntu*) DETECTED_OS_FAMILY="ubuntu" ;; \ | ||
| 73 | + esac && \ | ||
| 74 | + echo "Installing system dependencies (ID=$ID, OS_FAMILY=$DETECTED_OS_FAMILY)..." && \ | ||
| 75 | + if [ "$DETECTED_OS_FAMILY" = "openeuler" ]; then \ | ||
| 87 | yum install -y \ | 76 | yum install -y \ |
| 88 | git \ | 77 | git \ |
| 89 | iproute \ | 78 | iproute \ |
| @@ -96,7 +85,7 @@ RUN echo "Installing system dependencies..." && \ | |||
| 96 | cmake \ | 85 | cmake \ |
| 97 | ffmpeg-devel \ | 86 | ffmpeg-devel \ |
| 98 | && yum clean all; \ | 87 | && yum clean all; \ |
| 99 | - elif [ "$OS_FAMILY" = "ubuntu" ]; then \ | 88 | + elif [ "$DETECTED_OS_FAMILY" = "ubuntu" ]; then \ |
| 100 | apt-get install -y \ | 89 | apt-get install -y \ |
| 101 | git \ | 90 | git \ |
| 102 | iproute2 \ | 91 | iproute2 \ |
| @@ -114,8 +103,7 @@ RUN echo "Installing system dependencies..." && \ | |||
| 114 | libswscale-dev \ | 103 | libswscale-dev \ |
| 115 | && apt-get clean; \ | 104 | && apt-get clean; \ |
| 116 | else \ | 105 | else \ |
| 117 | - echo "ERROR: Unsupported OS: $OS_FAMILY"; \ | 106 | + echo "ERROR: Unsupported OS ID: $ID" && exit 1; \ |
| 118 | - exit 1; \ | ||
| 119 | fi | 107 | fi |
| 120 | 108 | ||
| 121 | # Install Miniconda (for creating and managing Python virtual environments). | 109 | # Install Miniconda (for creating and managing Python virtual environments). |
| @@ -66,7 +66,7 @@ Taking `v26.1.0-cann9.0.0-torch_npu2.7.1-a3-openeuler24.03-py3.11-aarch64` as an | |||
| 66 | | Python version | `py3.11` | Python 3.11 | | 66 | | Python version | `py3.11` | Python 3.11 | |
| 67 | | architecture | `aarch64` | ARM64 architecture | | 67 | | architecture | `aarch64` | ARM64 architecture | |
| 68 | 68 | ||
| 69 | -> The CANN version and TorchNPU version are derived from the base image and the `--torch-npu-version` parameter at build time. | 69 | +> The CANN version, NPU type, OS, and Python version are all derived from the base image tag. The TorchNPU version comes from the `--torch-npu-version` parameter at build time. |
| 70 | 70 | ||
| 71 | Dockerfile naming follows the template: `Dockerfile[.{chip info}.{OS}.{other fields}]` | 71 | Dockerfile naming follows the template: `Dockerfile[.{chip info}.{OS}.{other fields}]` |
| 72 | 72 | ||
| @@ -181,12 +181,11 @@ The image includes the following pre-configured environments: | |||
| 181 | 181 | ||
| 182 | ### Build Script Parameter Description | 182 | ### Build Script Parameter Description |
| 183 | 183 | ||
| 184 | -The build script `build.sh` supports multiple parameter configurations. The following default values are examples only; please adjust according to actual needs. The default value `9.0.0` for `--base-image-version` is an example only. **It is strongly recommended to explicitly specify the base image matching your target environment via `--base-image` or `--base-image-version`.** | 184 | +The build script `build.sh` supports multiple parameter configurations. The CANN version, NPU type, OS, and Python version are **only** obtained by parsing the `--base-image` tag — they cannot be specified manually. |
| 185 | 185 | ||
| 186 | -| Parameter | Description | Default (Example) | | 186 | +| Parameter | Description | Default | |
| 187 | | ------ | ------ | ------------ | | 187 | | ------ | ------ | ------------ | |
| 188 | -| `-t, --npu-type` | NPU type: A3 or 910B (auto-detected from `--base-image` if not specified) | None (required) | | 188 | +| `--base-image` | **Required.** Full base image name. CANN version, NPU type, OS, and Python version are auto-detected from the image tag | None (required) | |
| 189 | -| `-o, --os` | Operating system: openeuler24.03 or ubuntu22.04 (auto-detected from `--base-image` if not specified) | openeuler24.03 | | ||
| 190 | | `-v, --version` | MindSpeed MM version identifier, also used as Git branch name and script directory selection basis | 26.1.0 | | 189 | | `-v, --version` | MindSpeed MM version identifier, also used as Git branch name and script directory selection basis | 26.1.0 | |
| 191 | | `-m, --miniconda` | Miniconda installer path (auto-downloaded if not specified) | None (auto-download) | | 190 | | `-m, --miniconda` | Miniconda installer path (auto-downloaded if not specified) | None (auto-download) | |
| 192 | | `-d, --decord-deps` | decord dependencies directory path (auto-downloaded for ARM if not specified) | None (auto-download) | | 191 | | `-d, --decord-deps` | decord dependencies directory path (auto-downloaded for ARM if not specified) | None (auto-download) | |
| @@ -200,8 +199,6 @@ The build script `build.sh` supports multiple parameter configurations. The foll | |||
| 200 | | `--torch-npu-whl` | torch-npu.whl file path (offline installation) | None | | 199 | | `--torch-npu-whl` | torch-npu.whl file path (offline installation) | None | |
| 201 | | `--torchvision-whl` | torchvision.whl file path (optional, offline installation) | None | | 200 | | `--torchvision-whl` | torchvision.whl file path (optional, offline installation) | None | |
| 202 | | `--torchaudio-whl` | torchaudio.whl file path (optional, offline installation) | None | | 201 | | `--torchaudio-whl` | torchaudio.whl file path (optional, offline installation) | None | |
| 203 | -| `--base-image-version` | Base image CANN version (example value; recommended to specify explicitly) | 9.0.0 | | ||
| 204 | -| `--base-image` | Complete base image name (higher priority than `--base-image-version`; recommended) | None | | ||
| 205 | | `--build-ci` | Build CI image with multi-version conda environments (skip verl + MindSpeed-MM clone) | None | | 202 | | `--build-ci` | Build CI image with multi-version conda environments (skip verl + MindSpeed-MM clone) | None | |
| 206 | | `--cleanup-on-fail` | Clean up dangling images/containers on build failure | None | | 203 | | `--cleanup-on-fail` | Clean up dangling images/containers on build failure | None | |
| 207 | 204 | ||
| @@ -210,28 +207,22 @@ The build script `build.sh` supports multiple parameter configurations. The foll | |||
| 210 | ```bash | 207 | ```bash |
| 211 | cd docker | 208 | cd docker |
| 212 | 209 | ||
| 213 | -# Build A3 + openEuler image (default) | 210 | +# Build A3 + openEuler image |
| 214 | -bash build.sh -t A3 | 211 | +bash build.sh --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11 |
| 215 | - | ||
| 216 | -# Build A2 + openEuler image | ||
| 217 | -bash build.sh -t 910B | ||
| 218 | 212 | ||
| 219 | # Build A3 + Ubuntu image | 213 | # Build A3 + Ubuntu image |
| 220 | -bash build.sh -t A3 -o ubuntu22.04 | 214 | +bash build.sh --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-ubuntu22.04-py3.11 |
| 221 | 215 | ||
| 222 | # Build with specified PyTorch version | 216 | # Build with specified PyTorch version |
| 223 | -bash build.sh -t A3 --torch-version 2.7.1 --torch-npu-version 2.7.1 | 217 | +bash build.sh --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11 --torch-version 2.7.1 --torch-npu-version 2.7.1.post4 |
| 224 | 218 | ||
| 225 | # Build with offline .whl files | 219 | # Build with offline .whl files |
| 226 | -bash build.sh -t A3 \ | 220 | +bash build.sh --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11 \ |
| 227 | --torch-whl /path/to/torch-2.7.1+cpu-cp311-cp311-linux_x86_64.whl \ | 221 | --torch-whl /path/to/torch-2.7.1+cpu-cp311-cp311-linux_x86_64.whl \ |
| 228 | --torch-npu-whl /path/to/torch_npu-2.7.1-cp311-cp311-linux_x86_64.whl | 222 | --torch-npu-whl /path/to/torch_npu-2.7.1-cp311-cp311-linux_x86_64.whl |
| 229 | 223 | ||
| 230 | -# Build with specified base image version | ||
| 231 | -bash build.sh -t A3 --base-image-version 9.0.0 | ||
| 232 | - | ||
| 233 | # Build with specified version | 224 | # Build with specified version |
| 234 | -bash build.sh -t A3 -v 26.1.0 | 225 | +bash build.sh --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11 -v 26.1.0 |
| 235 | ``` | 226 | ``` |
| 236 | 227 | ||
| 237 | ### Automatic Download Function Description | 228 | ### Automatic Download Function Description |
| @@ -246,12 +237,13 @@ The build script supports automatic download of the following resources. Please | |||
| 246 | 237 | ||
| 247 | ### Automatic Base Image Recognition | 238 | ### Automatic Base Image Recognition |
| 248 | 239 | ||
| 249 | -The build script automatically recognizes key information from the base image name: | 240 | +The build script automatically recognizes key information from the base image tag (the `--base-image` parameter is required): |
| 250 | 241 | ||
| 251 | -1. **NPU type recognition**: Recognizes the NPU type (e.g., `a3`) from the image tag | 242 | +1. **CANN version recognition**: Extracts the CANN version number (e.g., `9.0.0`) from the image tag |
| 252 | -2. **Operating system recognition**: Recognizes `openeuler24.03` or `ubuntu22.04` from the image tag | 243 | +2. **NPU type recognition**: Recognizes the NPU type (e.g., `a3`) from the image tag |
| 253 | -3. **CANN version recognition**: Extracts the CANN version number (e.g., `9.0.0`) from the image tag | 244 | +3. **Operating system recognition**: Recognizes `openeuler24.03` or `ubuntu22.04` from the image tag |
| 254 | -4. **Automatic image tag generation**: Automatically generates image tags conforming to naming rules based on recognized information | 245 | +4. **Python version recognition**: Extracts the Python version (e.g., `3.11`) from the `py<x.y>` field in the image tag |
| 246 | +5. **Automatic image tag generation**: Automatically generates image tags conforming to naming rules based on recognized information | ||
| 255 | 247 | ||
| 256 | ### Best Practice Example | 248 | ### Best Practice Example |
| 257 | 249 | ||
| @@ -262,7 +254,7 @@ The following example shows how to build a MindSpeed MM image using a custom bas | |||
| 262 | SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | 254 | SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" |
| 263 | 255 | ||
| 264 | # Custom configuration | 256 | # Custom configuration |
| 265 | -BASE_IMAGE="swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-910b-openeuler24.03-py3.11" | 257 | +BASE_IMAGE="swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11" |
| 266 | TORCH_VERSION="2.7.1" | 258 | TORCH_VERSION="2.7.1" |
| 267 | TORCH_NPU_VERSION="2.7.1.post6" | 259 | TORCH_NPU_VERSION="2.7.1.post6" |
| 268 | MINDSPEED_MM_VERSION="26.0.0" | 260 | MINDSPEED_MM_VERSION="26.0.0" |
| @@ -278,7 +270,7 @@ bash "${SCRIPT_DIR}/build.sh" \ | |||
| 278 | 270 | ||
| 279 | **Key Feature Description:** | 271 | **Key Feature Description:** |
| 280 | 272 | ||
| 281 | -1. **Automatic recognition**: The script automatically recognizes the NPU type, operating system (openeuler24.03), and CANN version (9.0.0) from `BASE_IMAGE`. If `BASE_IMAGE` doesn't exist in the system, it will be automatically pulled. | 273 | +1. **Automatic recognition**: The script automatically recognizes the CANN version (9.0.0), NPU type (a3), operating system (openeuler24.03), and Python version (3.11) from `BASE_IMAGE`. If `BASE_IMAGE` doesn't exist in the system, it will be automatically pulled. |
| 282 | 2. **Automatic tag generation**: Automatically generates image tags conforming to the naming rules based on recognition results. | 274 | 2. **Automatic tag generation**: Automatically generates image tags conforming to the naming rules based on recognition results. |
| 283 | 3. **Automatic download**: If the Miniconda installer or decord dependencies are not available locally, the script will automatically download them | 275 | 3. **Automatic download**: If the Miniconda installer or decord dependencies are not available locally, the script will automatically download them |
| 284 | 4. **Failure cleanup**: The `--cleanup-on-fail` parameter ensures cleanup of dangling resources if the build fails | 276 | 4. **Failure cleanup**: The `--cleanup-on-fail` parameter ensures cleanup of dangling resources if the build fails |
| @@ -375,7 +367,7 @@ Build an image containing the new model environment using the following command: | |||
| 375 | 367 | ||
| 376 | ```bash | 368 | ```bash |
| 377 | # Use the -v parameter to specify the version number, and the build script will automatically find the corresponding installation script | 369 | # Use the -v parameter to specify the version number, and the build script will automatically find the corresponding installation script |
| 378 | -bash build.sh -t A3 -v 26.1.0 | 370 | +bash build.sh --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11 -v 26.1.0 |
| 379 | ``` | 371 | ``` |
| 380 | 372 | ||
| 381 | #### 5. Update Documentation | 373 | #### 5. Update Documentation |
| @@ -66,7 +66,7 @@ MindSpeed MM 镜像基于 Ubuntu 22.04 和 openEuler 24.03 两种操作系统, | |||
| 66 | | Python版本 | `py3.11` | Python 3.11 | | 66 | | Python版本 | `py3.11` | Python 3.11 | |
| 67 | | 架构类型 | `aarch64` | ARM64 架构 | | 67 | | 架构类型 | `aarch64` | ARM64 架构 | |
| 68 | 68 | ||
| 69 | -> CANN 版本与 TorchNPU 版本来源于构建时的基础镜像和 `--torch-npu-version` 参数。 | 69 | +> CANN 版本、NPU 类型、操作系统和 Python 版本均来源于构建时基础镜像的 tag,TorchNPU 版本来源于 `--torch-npu-version` 参数。 |
| 70 | 70 | ||
| 71 | Dockerfile 命名遵循模板:`Dockerfile[.{芯片信息}.{操作系统}.{其他字段}]` | 71 | Dockerfile 命名遵循模板:`Dockerfile[.{芯片信息}.{操作系统}.{其他字段}]` |
| 72 | 72 | ||
| @@ -181,12 +181,11 @@ docker run -it --rm \ | |||
| 181 | 181 | ||
| 182 | ### 构建脚本参数说明 | 182 | ### 构建脚本参数说明 |
| 183 | 183 | ||
| 184 | -构建脚本 `build.sh` 支持多种参数配置,以下默认值仅为示例,请根据实际需求调整。其中 `--base-image-version` 的默认值 `9.0.0` 仅为示例,**强烈建议通过 `--base-image` 或 `--base-image-version` 显式指定与目标环境匹配的基础镜像**。 | 184 | +构建脚本 `build.sh` 支持多种参数配置。其中 CANN 版本、NPU 类型、操作系统和 Python 版本**只能**通过解析 `--base-image` 的 tag 获取,不支持手动指定。 |
| 185 | 185 | ||
| 186 | -| 参数 | 说明 | 默认值(示例) | | 186 | +| 参数 | 说明 | 默认值 | |
| 187 | | ------ | ------ | ------------ | | 187 | | ------ | ------ | ------------ | |
| 188 | -| `-t, --npu-type` | NPU 类型:A3 或 910B(未指定时可从 `--base-image` 自动识别) | 无(必需) | | 188 | +| `--base-image` | **必选。** 完整基础镜像名称,CANN 版本、NPU 类型、操作系统和 Python 版本均从镜像 tag 自动识别 | 无(必需) | |
| 189 | -| `-o, --os` | 操作系统:openeuler24.03 或 ubuntu22.04(未指定时可从 `--base-image` 自动识别) | openeuler24.03 | | ||
| 190 | | `-v, --version` | MindSpeed MM 版本标识,同时作为 Git 分支名称和脚本目录选择依据 | 26.1.0 | | 189 | | `-v, --version` | MindSpeed MM 版本标识,同时作为 Git 分支名称和脚本目录选择依据 | 26.1.0 | |
| 191 | | `-m, --miniconda` | Miniconda 安装器路径(未指定时自动下载) | 无(自动下载) | | 190 | | `-m, --miniconda` | Miniconda 安装器路径(未指定时自动下载) | 无(自动下载) | |
| 192 | | `-d, --decord-deps` | decord 依赖目录路径(ARM 架构下未指定时自动下载) | 无(自动下载) | | 191 | | `-d, --decord-deps` | decord 依赖目录路径(ARM 架构下未指定时自动下载) | 无(自动下载) | |
| @@ -200,8 +199,6 @@ docker run -it --rm \ | |||
| 200 | | `--torch-npu-whl` | torch-npu.whl 文件路径(离线安装) | 无 | | 199 | | `--torch-npu-whl` | torch-npu.whl 文件路径(离线安装) | 无 | |
| 201 | | `--torchvision-whl` | torchvision.whl 文件路径(可选,离线安装) | 无 | | 200 | | `--torchvision-whl` | torchvision.whl 文件路径(可选,离线安装) | 无 | |
| 202 | | `--torchaudio-whl` | torchaudio.whl 文件路径(可选,离线安装) | 无 | | 201 | | `--torchaudio-whl` | torchaudio.whl 文件路径(可选,离线安装) | 无 | |
| 203 | -| `--base-image-version` | 基础镜像 CANN 版本(示例值,建议显式指定) | 9.0.0 | | ||
| 204 | -| `--base-image` | 完整基础镜像名称(优先级高于 `--base-image-version`,推荐使用) | 无 | | ||
| 205 | | `--build-ci` | 构建 CI 镜像,包含多版本 conda 环境(跳过 verl + MindSpeed-MM 克隆) | 无 | | 202 | | `--build-ci` | 构建 CI 镜像,包含多版本 conda 环境(跳过 verl + MindSpeed-MM 克隆) | 无 | |
| 206 | | `--cleanup-on-fail` | 构建失败时清理悬空镜像/容器 | 无 | | 203 | | `--cleanup-on-fail` | 构建失败时清理悬空镜像/容器 | 无 | |
| 207 | 204 | ||
| @@ -210,28 +207,22 @@ docker run -it --rm \ | |||
| 210 | ```bash | 207 | ```bash |
| 211 | cd docker | 208 | cd docker |
| 212 | 209 | ||
| 213 | -# 构建 A3 + openEuler 镜像(默认) | 210 | +# 构建 A3 + openEuler 镜像 |
| 214 | -bash build.sh -t A3 | 211 | +bash build.sh --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11 |
| 215 | - | ||
| 216 | -# 构建 A2 + openEuler 镜像 | ||
| 217 | -bash build.sh -t 910B | ||
| 218 | 212 | ||
| 219 | # 构建 A3 + Ubuntu 镜像 | 213 | # 构建 A3 + Ubuntu 镜像 |
| 220 | -bash build.sh -t A3 -o ubuntu22.04 | 214 | +bash build.sh --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-ubuntu22.04-py3.11 |
| 221 | 215 | ||
| 222 | # 指定 PyTorch 版本构建 | 216 | # 指定 PyTorch 版本构建 |
| 223 | -bash build.sh -t A3 --torch-version 2.7.1 --torch-npu-version 2.7.1 | 217 | +bash build.sh --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11 --torch-version 2.7.1 --torch-npu-version 2.7.1.post4 |
| 224 | 218 | ||
| 225 | # 使用离线 .whl 文件构建 | 219 | # 使用离线 .whl 文件构建 |
| 226 | -bash build.sh -t A3 \ | 220 | +bash build.sh --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11 \ |
| 227 | --torch-whl /path/to/torch-2.7.1+cpu-cp311-cp311-linux_x86_64.whl \ | 221 | --torch-whl /path/to/torch-2.7.1+cpu-cp311-cp311-linux_x86_64.whl \ |
| 228 | --torch-npu-whl /path/to/torch_npu-2.7.1-cp311-cp311-linux_x86_64.whl | 222 | --torch-npu-whl /path/to/torch_npu-2.7.1-cp311-cp311-linux_x86_64.whl |
| 229 | 223 | ||
| 230 | -# 指定基础镜像版本构建 | ||
| 231 | -bash build.sh -t A3 --base-image-version 9.0.0 | ||
| 232 | - | ||
| 233 | # 指定版本构建 | 224 | # 指定版本构建 |
| 234 | -bash build.sh -t A3 -v 26.1.0 | 225 | +bash build.sh --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11 -v 26.1.0 |
| 235 | ``` | 226 | ``` |
| 236 | 227 | ||
| 237 | ### 自动下载功能说明 | 228 | ### 自动下载功能说明 |
| @@ -246,12 +237,13 @@ bash build.sh -t A3 -v 26.1.0 | |||
| 246 | 237 | ||
| 247 | ### 自动识别基础镜像 | 238 | ### 自动识别基础镜像 |
| 248 | 239 | ||
| 249 | -构建脚本会自动识别基础镜像名称中的关键信息: | 240 | +构建脚本会自动从基础镜像 tag 中识别关键信息(`--base-image` 参数为必选): |
| 250 | 241 | ||
| 251 | -1. **NPU 类型识别**:从镜像 tag 中识别NPU类型(如`a3`) | 242 | +1. **CANN 版本识别**:从镜像 tag 中提取 CANN 版本号(如 `9.0.0`) |
| 252 | -2. **操作系统识别**:从镜像 tag 中识别 `openeuler24.03` 或 `ubuntu22.04` | 243 | +2. **NPU 类型识别**:从镜像 tag 中识别 NPU 类型(如 `a3`) |
| 253 | -3. **CANN 版本识别**:从镜像 tag 中提取 CANN 版本号(如 `9.0.0`) | 244 | +3. **操作系统识别**:从镜像 tag 中识别 `openeuler24.03` 或 `ubuntu22.04` |
| 254 | -4. **自动生成镜像 tag**:基于识别到的信息自动生成符合命名规则的镜像 tag | 245 | +4. **Python 版本识别**:从镜像 tag 中的 `py<x.y>` 字段提取 Python 版本(如 `3.11`) |
| 246 | +5. **自动生成镜像 tag**:基于识别到的信息自动生成符合命名规则的镜像 tag | ||
| 255 | 247 | ||
| 256 | ### 最佳实践示例 | 248 | ### 最佳实践示例 |
| 257 | 249 | ||
| @@ -262,7 +254,7 @@ bash build.sh -t A3 -v 26.1.0 | |||
| 262 | SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | 254 | SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" |
| 263 | 255 | ||
| 264 | # 自定义配置 | 256 | # 自定义配置 |
| 265 | -BASE_IMAGE="swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-910b-openeuler24.03-py3.11" | 257 | +BASE_IMAGE="swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11" |
| 266 | TORCH_VERSION="2.7.1" | 258 | TORCH_VERSION="2.7.1" |
| 267 | TORCH_NPU_VERSION="2.7.1.post6" | 259 | TORCH_NPU_VERSION="2.7.1.post6" |
| 268 | MINDSPEED_MM_VERSION="26.0.0" | 260 | MINDSPEED_MM_VERSION="26.0.0" |
| @@ -278,7 +270,7 @@ bash "${SCRIPT_DIR}/build.sh" \ | |||
| 278 | 270 | ||
| 279 | **关键特性说明:** | 271 | **关键特性说明:** |
| 280 | 272 | ||
| 281 | -1. **自动识别**:脚本会自动从 `BASE_IMAGE` 中识别 NPU 类型、操作系统(openeuler24.03)和 CANN 版本(9.0.0)。如果`BASE_IMAGE`在系统中不存在,会自动拉取。 | 273 | +1. **自动识别**:脚本会自动从 `BASE_IMAGE` 中识别 CANN 版本(9.0.0)、NPU 类型(a3)、操作系统(openeuler24.03)和 Python 版本(3.11)。如果`BASE_IMAGE`在系统中不存在,会自动拉取。 |
| 282 | 2. **自动生成 tag**:基于识别结果自动生成符合命名规则的镜像 tag。 | 274 | 2. **自动生成 tag**:基于识别结果自动生成符合命名规则的镜像 tag。 |
| 283 | 3. **自动下载**:如果本地没有 Miniconda 安装器或 decord 依赖,脚本会自动下载 | 275 | 3. **自动下载**:如果本地没有 Miniconda 安装器或 decord 依赖,脚本会自动下载 |
| 284 | 4. **失败清理**:`--cleanup-on-fail` 参数确保构建失败时清理悬空资源 | 276 | 4. **失败清理**:`--cleanup-on-fail` 参数确保构建失败时清理悬空资源 |
| @@ -375,7 +367,7 @@ echo "==========================================" | |||
| 375 | 367 | ||
| 376 | ```bash | 368 | ```bash |
| 377 | # 使用 -v 参数指定版本号,构建脚本会自动找到对应的安装脚本 | 369 | # 使用 -v 参数指定版本号,构建脚本会自动找到对应的安装脚本 |
| 378 | -bash build.sh -t A3 -v 26.1.0 | 370 | +bash build.sh --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11 -v 26.1.0 |
| 379 | ``` | 371 | ``` |
| 380 | 372 | ||
| 381 | #### 5. 更新文档 | 373 | #### 5. 更新文档 |
| @@ -26,7 +26,9 @@ cleanup_dangling() { | |||
| 26 | # Dockerfile naming: Dockerfile (unified, supports all NPU types and OS) | 26 | # Dockerfile naming: Dockerfile (unified, supports all NPU types and OS) |
| 27 | # Image tag naming: {version}-cann{cann_version}-torch_npu{torch_npu_version}-{chip}-{os}-py{python_version}-{architecture} | 27 | # Image tag naming: {version}-cann{cann_version}-torch_npu{torch_npu_version}-{chip}-{os}-py{python_version}-{architecture} |
| 28 | # Usage: | 28 | # Usage: |
| 29 | -# bash build.sh -t A3 [-m /path/to/miniconda.sh] [-o ubuntu22.04] | 29 | +# bash build.sh --base-image <base-image> [-m /path/to/miniconda.sh] |
| 30 | +# CANN version, NPU type, OS and Python version are auto-detected from the | ||
| 31 | +# base image tag and cannot be specified manually. | ||
| 30 | # ============================================ | 32 | # ============================================ |
| 31 | 33 | ||
| 32 | set -e | 34 | set -e |
| @@ -42,7 +44,8 @@ Usage: $0 [OPTIONS] | |||
| 42 | Build MindSpeed MM Docker Image | 44 | Build MindSpeed MM Docker Image |
| 43 | 45 | ||
| 44 | Required: | 46 | Required: |
| 45 | - -t, --npu-type TYPE NPU type: A3 or 910B (auto-detected from --base-image if not specified) | 47 | + --base-image IMAGE Full base image name (e.g. swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-910b-openeuler24.03-py3.11) |
| 48 | + CANN version, NPU type, OS and Python version are auto-detected from the image tag. | ||
| 46 | 49 | ||
| 47 | Optional: | 50 | Optional: |
| 48 | -m, --miniconda PATH Miniconda installer path (auto-download if not specified) | 51 | -m, --miniconda PATH Miniconda installer path (auto-download if not specified) |
| @@ -51,7 +54,6 @@ Optional: | |||
| 51 | -i, --image-name NAME Image name (default: mindspeed-mm:v{version}-cann{cann_ver}-torch_npu{torch_npu_ver}-{chip}-{os}-py{py_ver}-{arch}) | 54 | -i, --image-name NAME Image name (default: mindspeed-mm:v{version}-cann{cann_ver}-torch_npu{torch_npu_ver}-{chip}-{os}-py{py_ver}-{arch}) |
| 52 | --tag TAG Custom image tag (overrides the default tag; keeps repo name 'mindspeed-mm') | 55 | --tag TAG Custom image tag (overrides the default tag; keeps repo name 'mindspeed-mm') |
| 53 | -n, --no-cache Build without cache | 56 | -n, --no-cache Build without cache |
| 54 | - -o, --os OS OS: openeuler24.03 or ubuntu22.04 (auto-detected from --base-image if not specified) | ||
| 55 | -v, --version VERSION MindSpeed MM version (default: 26.1.0, determines model install scripts) | 57 | -v, --version VERSION MindSpeed MM version (default: 26.1.0, determines model install scripts) |
| 56 | --torch-version VER PyTorch version (default: 2.7.1, for online install) | 58 | --torch-version VER PyTorch version (default: 2.7.1, for online install) |
| 57 | --torch-npu-version VER torch-npu version (default: 2.7.1, for online install) | 59 | --torch-npu-version VER torch-npu version (default: 2.7.1, for online install) |
| @@ -59,14 +61,11 @@ Optional: | |||
| 59 | --torch-npu-whl PATH torch-npu .whl file path (offline install) | 61 | --torch-npu-whl PATH torch-npu .whl file path (offline install) |
| 60 | --torchvision-whl PATH torchvision .whl file path (optional, offline install) | 62 | --torchvision-whl PATH torchvision .whl file path (optional, offline install) |
| 61 | --torchaudio-whl PATH torchaudio .whl file path (optional, offline install) | 63 | --torchaudio-whl PATH torchaudio .whl file path (optional, offline install) |
| 62 | - --base-image-version VER Base image CANN version (default: 9.0.0, example only; recommended to specify via --base-image or --base-image-version) | ||
| 63 | - --base-image IMAGE Full base image name (higher priority than --base-image-version; recommended) | ||
| 64 | --build-ci Build CI image with multi-version conda environments (skip verl + MindSpeed-MM clone) | 64 | --build-ci Build CI image with multi-version conda environments (skip verl + MindSpeed-MM clone) |
| 65 | --cleanup-on-fail Clean up dangling images/containers if build fails | 65 | --cleanup-on-fail Clean up dangling images/containers if build fails |
| 66 | -h, --help Show help | 66 | -h, --help Show help |
| 67 | 67 | ||
| 68 | Dockerfile naming convention: Dockerfile (unified, supports all NPU types and OS) | 68 | Dockerfile naming convention: Dockerfile (unified, supports all NPU types and OS) |
| 69 | - NPU type and OS are passed as build arguments | ||
| 70 | 69 | ||
| 71 | Image tag naming convention: {version}-cann{cann_version}-torch_npu{torch_npu_version}-{chip}-{os}-py{python_version}-{architecture}[-ci] | 70 | Image tag naming convention: {version}-cann{cann_version}-torch_npu{torch_npu_version}-{chip}-{os}-py{python_version}-{architecture}[-ci] |
| 72 | e.g. v26.0.0-cann9.0.0-torch_npu2.7.1-a3-openeuler24.03-py3.11-x86_64 | 71 | e.g. v26.0.0-cann9.0.0-torch_npu2.7.1-a3-openeuler24.03-py3.11-x86_64 |
| @@ -74,51 +73,39 @@ Image tag naming convention: {version}-cann{cann_version}-torch_npu{torch_npu_ve | |||
| 74 | v26.1.0-cann9.0.0-torch_npu2.7.1-910b-openeuler24.03-py3.11-aarch64-ci (CI build via --build-ci) | 73 | v26.1.0-cann9.0.0-torch_npu2.7.1-910b-openeuler24.03-py3.11-aarch64-ci (CI build via --build-ci) |
| 75 | 74 | ||
| 76 | Examples: | 75 | Examples: |
| 77 | - bash $0 -t A3 | 76 | + bash $0 --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11 |
| 78 | - bash $0 -t A3 -o ubuntu22.04 | 77 | + bash $0 --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-910b-openeuler24.03-py3.11 --torch-version 2.7.1 --torch-npu-version 2.7.1 |
| 79 | - bash $0 --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-beta.2-910b-openeuler24.03-py3.11 | 78 | + bash $0 --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11 -i myproject/mindspeed-mm:v1.0 |
| 80 | - bash $0 -t 910B --torch-version 2.7.1 --torch-npu-version 2.7.1 | 79 | + bash $0 --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11 --torch-whl /path/to/torch.whl --torch-npu-whl /path/to/torch_npu.whl |
| 81 | - bash $0 -t A3 --base-image-version 9.0.0 | 80 | + bash $0 --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-910b-openeuler24.03-py3.11 --build-ci # Build CI image (tag auto-appends '-ci') |
| 82 | - bash $0 -t A3 -i myproject/mindspeed-mm:v1.0 | ||
| 83 | - bash $0 -t A3 --torch-whl /path/to/torch.whl --torch-npu-whl /path/to/torch_npu.whl | ||
| 84 | - bash $0 -t 910B --build-ci # Build CI image (tag auto-appends '-ci') | ||
| 85 | EOF | 81 | EOF |
| 86 | } | 82 | } |
| 87 | 83 | ||
| 88 | -NPU_TYPE="" | ||
| 89 | MINICONDA_PATH="" | 84 | MINICONDA_PATH="" |
| 90 | DECORD_DEPS_PATH="" | 85 | DECORD_DEPS_PATH="" |
| 91 | DECORD_SCRIPT_PATH="" | 86 | DECORD_SCRIPT_PATH="" |
| 92 | IMAGE_NAME="" | 87 | IMAGE_NAME="" |
| 93 | IMAGE_TAG="" | 88 | IMAGE_TAG="" |
| 94 | NO_CACHE="" | 89 | NO_CACHE="" |
| 95 | -OS="openeuler24.03" | ||
| 96 | TORCH_VERSION="2.7.1" | 90 | TORCH_VERSION="2.7.1" |
| 97 | TORCH_NPU_VERSION="2.7.1" | 91 | TORCH_NPU_VERSION="2.7.1" |
| 98 | TORCH_WHL_PATH="" | 92 | TORCH_WHL_PATH="" |
| 99 | TORCH_NPU_WHL_PATH="" | 93 | TORCH_NPU_WHL_PATH="" |
| 100 | TORCHVISION_WHL_PATH="" | 94 | TORCHVISION_WHL_PATH="" |
| 101 | TORCHAUDIO_WHL_PATH="" | 95 | TORCHAUDIO_WHL_PATH="" |
| 102 | -# Default value is an example only. It is recommended to explicitly specify the | 96 | +BASE_IMAGE="" |
| 103 | -# base image via --base-image or --base-image-version to match your target environment. | ||
| 104 | -BASE_IMAGE_VERSION="9.0.0" | ||
| 105 | MINDSPEED_MM_VERSION="26.1.0" | 97 | MINDSPEED_MM_VERSION="26.1.0" |
| 106 | -PYTHON_VERSION="3.11" | ||
| 107 | -NPU_TYPE_EXPLICIT=false | ||
| 108 | -OS_EXPLICIT=false | ||
| 109 | CLEANUP_ON_FAIL=false | 98 | CLEANUP_ON_FAIL=false |
| 110 | BUILD_CI=false | 99 | BUILD_CI=false |
| 111 | 100 | ||
| 112 | while [[ $# -gt 0 ]]; do | 101 | while [[ $# -gt 0 ]]; do |
| 113 | case $1 in | 102 | case $1 in |
| 114 | - -t|--npu-type) NPU_TYPE="$2"; NPU_TYPE_EXPLICIT=true; shift 2 ;; | ||
| 115 | -m|--miniconda) MINICONDA_PATH="$2"; shift 2 ;; | 103 | -m|--miniconda) MINICONDA_PATH="$2"; shift 2 ;; |
| 116 | -d|--decord-deps) DECORD_DEPS_PATH="$2"; shift 2 ;; | 104 | -d|--decord-deps) DECORD_DEPS_PATH="$2"; shift 2 ;; |
| 117 | -s|--decord-script) DECORD_SCRIPT_PATH="$2"; shift 2 ;; | 105 | -s|--decord-script) DECORD_SCRIPT_PATH="$2"; shift 2 ;; |
| 118 | -i|--image-name) IMAGE_NAME="$2"; shift 2 ;; | 106 | -i|--image-name) IMAGE_NAME="$2"; shift 2 ;; |
| 119 | --tag) IMAGE_TAG="$2"; shift 2 ;; | 107 | --tag) IMAGE_TAG="$2"; shift 2 ;; |
| 120 | -n|--no-cache) NO_CACHE="--no-cache"; shift ;; | 108 | -n|--no-cache) NO_CACHE="--no-cache"; shift ;; |
| 121 | - -o|--os) OS="$2"; OS_EXPLICIT=true; shift 2 ;; | ||
| 122 | -v|--version) MINDSPEED_MM_VERSION="$2"; shift 2 ;; | 109 | -v|--version) MINDSPEED_MM_VERSION="$2"; shift 2 ;; |
| 123 | --torch-version) TORCH_VERSION="$2"; shift 2 ;; | 110 | --torch-version) TORCH_VERSION="$2"; shift 2 ;; |
| 124 | --torch-npu-version) TORCH_NPU_VERSION="$2"; shift 2 ;; | 111 | --torch-npu-version) TORCH_NPU_VERSION="$2"; shift 2 ;; |
| @@ -126,7 +113,6 @@ while [[ $# -gt 0 ]]; do | |||
| 126 | --torch-npu-whl) TORCH_NPU_WHL_PATH="$2"; shift 2 ;; | 113 | --torch-npu-whl) TORCH_NPU_WHL_PATH="$2"; shift 2 ;; |
| 127 | --torchvision-whl) TORCHVISION_WHL_PATH="$2"; shift 2 ;; | 114 | --torchvision-whl) TORCHVISION_WHL_PATH="$2"; shift 2 ;; |
| 128 | --torchaudio-whl) TORCHAUDIO_WHL_PATH="$2"; shift 2 ;; | 115 | --torchaudio-whl) TORCHAUDIO_WHL_PATH="$2"; shift 2 ;; |
| 129 | - --base-image-version) BASE_IMAGE_VERSION="$2"; shift 2 ;; | ||
| 130 | --base-image) BASE_IMAGE="$2"; shift 2 ;; | 116 | --base-image) BASE_IMAGE="$2"; shift 2 ;; |
| 131 | --build-ci) BUILD_CI=true; shift ;; | 117 | --build-ci) BUILD_CI=true; shift ;; |
| 132 | --cleanup-on-fail) CLEANUP_ON_FAIL=true; shift ;; | 118 | --cleanup-on-fail) CLEANUP_ON_FAIL=true; shift ;; |
| @@ -172,6 +158,12 @@ parse_base_image_tag() { | |||
| 172 | detected_cann="${tag_lower%%-*}" | 158 | detected_cann="${tag_lower%%-*}" |
| 173 | fi | 159 | fi |
| 174 | 160 | ||
| 161 | + # Python version: match 'py<x.y>' field in the tag (e.g. py3.11 -> 3.11) | ||
| 162 | + local detected_python="" | ||
| 163 | + if [[ "$tag_lower" =~ py([0-9]+\.[0-9]+) ]]; then | ||
| 164 | + detected_python="${BASH_REMATCH[1]}" | ||
| 165 | + fi | ||
| 166 | + | ||
| 175 | if [ -n "$detected_npu" ]; then | 167 | if [ -n "$detected_npu" ]; then |
| 176 | DETECTED_NPU_TYPE="$detected_npu" | 168 | DETECTED_NPU_TYPE="$detected_npu" |
| 177 | echo ">>> Auto-detected NPU type from base image: $detected_npu" | 169 | echo ">>> Auto-detected NPU type from base image: $detected_npu" |
| @@ -187,34 +179,57 @@ parse_base_image_tag() { | |||
| 187 | echo ">>> Auto-detected OS from base image: $detected_os" | 179 | echo ">>> Auto-detected OS from base image: $detected_os" |
| 188 | fi | 180 | fi |
| 189 | 181 | ||
| 182 | + if [ -n "$detected_python" ]; then | ||
| 183 | + DETECTED_PYTHON_VERSION="$detected_python" | ||
| 184 | + echo ">>> Auto-detected Python version from base image: $detected_python" | ||
| 185 | + fi | ||
| 186 | + | ||
| 190 | return 0 | 187 | return 0 |
| 191 | } | 188 | } |
| 192 | 189 | ||
| 193 | DETECTED_NPU_TYPE="" | 190 | DETECTED_NPU_TYPE="" |
| 194 | DETECTED_OS="" | 191 | DETECTED_OS="" |
| 195 | DETECTED_CANN_VERSION="" | 192 | DETECTED_CANN_VERSION="" |
| 193 | +DETECTED_PYTHON_VERSION="" | ||
| 196 | 194 | ||
| 197 | -if [ -n "$BASE_IMAGE" ]; then | 195 | +if [ -z "$BASE_IMAGE" ]; then |
| 198 | - echo ">>> Auto-detecting NPU type, OS, and CANN version from base image..." | 196 | + echo "Error: --base-image is required. CANN version, NPU type, OS and Python version are auto-detected from the base image tag." |
| 199 | - parse_base_image_tag "$BASE_IMAGE" | ||
| 200 | - | ||
| 201 | - if [ "$NPU_TYPE_EXPLICIT" = false ] && [ -n "$DETECTED_NPU_TYPE" ]; then | ||
| 202 | - NPU_TYPE="$DETECTED_NPU_TYPE" | ||
| 203 | - echo ">>> Using auto-detected NPU type: $NPU_TYPE" | ||
| 204 | - fi | ||
| 205 | - | ||
| 206 | - if [ "$OS_EXPLICIT" = false ] && [ -n "$DETECTED_OS" ]; then | ||
| 207 | - OS="$DETECTED_OS" | ||
| 208 | - echo ">>> Using auto-detected OS: $OS" | ||
| 209 | - fi | ||
| 210 | -fi | ||
| 211 | - | ||
| 212 | -if [ -z "$NPU_TYPE" ]; then | ||
| 213 | - echo "Error: NPU type is required (-t or --npu-type) or provide --base-image for auto-detection" | ||
| 214 | show_help | 197 | show_help |
| 215 | exit 1 | 198 | exit 1 |
| 216 | fi | 199 | fi |
| 217 | 200 | ||
| 201 | +echo ">>> Auto-detecting NPU type, OS, CANN version and Python version from base image..." | ||
| 202 | +parse_base_image_tag "$BASE_IMAGE" | ||
| 203 | + | ||
| 204 | +if [ -z "$DETECTED_NPU_TYPE" ]; then | ||
| 205 | + echo "Error: Failed to detect NPU type from base image tag (expected '910b' or 'a3')." | ||
| 206 | + exit 1 | ||
| 207 | +fi | ||
| 208 | +NPU_TYPE="$DETECTED_NPU_TYPE" | ||
| 209 | + | ||
| 210 | +if [ -z "$DETECTED_OS" ]; then | ||
| 211 | + echo "Error: Failed to detect OS from base image tag (expected 'openeuler24.03' or 'ubuntu22.04')." | ||
| 212 | + exit 1 | ||
| 213 | +fi | ||
| 214 | +OS="$DETECTED_OS" | ||
| 215 | + | ||
| 216 | +if [ -z "$DETECTED_CANN_VERSION" ]; then | ||
| 217 | + echo "Error: Failed to detect CANN version from base image tag." | ||
| 218 | + exit 1 | ||
| 219 | +fi | ||
| 220 | +CANN_VERSION="$DETECTED_CANN_VERSION" | ||
| 221 | + | ||
| 222 | +if [ -z "$DETECTED_PYTHON_VERSION" ]; then | ||
| 223 | + echo "Error: Failed to detect Python version from base image tag (expected 'py<x.y>' field, e.g. py3.11)." | ||
| 224 | + exit 1 | ||
| 225 | +fi | ||
| 226 | +PYTHON_VERSION="$DETECTED_PYTHON_VERSION" | ||
| 227 | + | ||
| 228 | +echo ">>> Using detected NPU type: $NPU_TYPE" | ||
| 229 | +echo ">>> Using detected OS: $OS" | ||
| 230 | +echo ">>> Using detected CANN version: $CANN_VERSION" | ||
| 231 | +echo ">>> Using detected Python: $PYTHON_VERSION" | ||
| 232 | + | ||
| 218 | NPU_TYPE=$(echo "$NPU_TYPE" | tr '[:lower:]' '[:upper:]') | 233 | NPU_TYPE=$(echo "$NPU_TYPE" | tr '[:lower:]' '[:upper:]') |
| 219 | NPU_TYPE_LOWER=$(echo "$NPU_TYPE" | tr '[:upper:]' '[:lower:]') | 234 | NPU_TYPE_LOWER=$(echo "$NPU_TYPE" | tr '[:upper:]' '[:lower:]') |
| 220 | OS=$(echo "$OS" | tr '[:upper:]' '[:lower:]') | 235 | OS=$(echo "$OS" | tr '[:upper:]' '[:lower:]') |
| @@ -319,15 +334,6 @@ if [[ "$MINICONDA_NAME" == *"aarch64"* ]]; then | |||
| 319 | ARCH_NAME="aarch64" | 334 | ARCH_NAME="aarch64" |
| 320 | fi | 335 | fi |
| 321 | 336 | ||
| 322 | -# Determine CANN version for the image tag: | ||
| 323 | -# - From the base image tag (auto-detected) when --base-image is provided | ||
| 324 | -# - From --base-image-version otherwise | ||
| 325 | -if [ -n "$BASE_IMAGE" ] && [ -n "$DETECTED_CANN_VERSION" ]; then | ||
| 326 | - CANN_VERSION="$DETECTED_CANN_VERSION" | ||
| 327 | -else | ||
| 328 | - CANN_VERSION="$BASE_IMAGE_VERSION" | ||
| 329 | -fi | ||
| 330 | - | ||
| 331 | if [ -n "$IMAGE_NAME" ] && [ -n "$IMAGE_TAG" ]; then | 337 | if [ -n "$IMAGE_NAME" ] && [ -n "$IMAGE_TAG" ]; then |
| 332 | echo "Warning: both --image-name and --tag provided; --image-name takes precedence (--tag ignored)" | 338 | echo "Warning: both --image-name and --tag provided; --image-name takes precedence (--tag ignored)" |
| 333 | fi | 339 | fi |
| @@ -404,14 +410,15 @@ fi | |||
| 404 | echo "==========================================" | 410 | echo "==========================================" |
| 405 | echo "Build Configuration" | 411 | echo "Build Configuration" |
| 406 | echo "==========================================" | 412 | echo "==========================================" |
| 413 | +echo "Base Image: ${BASE_IMAGE}" | ||
| 407 | echo "NPU Type: ${NPU_TYPE}" | 414 | echo "NPU Type: ${NPU_TYPE}" |
| 408 | echo "OS: ${OS}" | 415 | echo "OS: ${OS}" |
| 416 | +echo "CANN Version: ${CANN_VERSION}" | ||
| 417 | +echo "Python Version: ${PYTHON_VERSION}" | ||
| 409 | echo "CPU Architecture: ${ARCH_NAME}" | 418 | echo "CPU Architecture: ${ARCH_NAME}" |
| 410 | echo "Dockerfile: ${DOCKERFILE}" | 419 | echo "Dockerfile: ${DOCKERFILE}" |
| 411 | echo "Image Name: ${IMAGE_NAME}" | 420 | echo "Image Name: ${IMAGE_NAME}" |
| 412 | echo "Image Tag: ${IMAGE_TAG:-(auto)}" | 421 | echo "Image Tag: ${IMAGE_TAG:-(auto)}" |
| 413 | -echo "Base Image Version: ${BASE_IMAGE_VERSION}" | ||
| 414 | -echo "CANN Version: ${CANN_VERSION}" | ||
| 415 | echo "PyTorch Version: ${TORCH_VERSION}" | 422 | echo "PyTorch Version: ${TORCH_VERSION}" |
| 416 | echo "torch-npu Version: ${TORCH_NPU_VERSION}" | 423 | echo "torch-npu Version: ${TORCH_NPU_VERSION}" |
| 417 | echo "MindSpeed MM Ver: ${MINDSPEED_MM_VERSION}" | 424 | echo "MindSpeed MM Ver: ${MINDSPEED_MM_VERSION}" |
| @@ -434,21 +441,19 @@ echo "Build CI: ${BUILD_CI}" | |||
| 434 | echo "No Cache: ${NO_CACHE:-No}" | 441 | echo "No Cache: ${NO_CACHE:-No}" |
| 435 | echo "==========================================" | 442 | echo "==========================================" |
| 436 | 443 | ||
| 437 | -if [ -n "$BASE_IMAGE" ]; then | 444 | +echo "" |
| 438 | - echo "" | 445 | +echo ">>> Checking if base image exists..." |
| 439 | - echo ">>> Checking if base image exists..." | 446 | +if ! docker image inspect "$BASE_IMAGE" > /dev/null 2>&1; then |
| 440 | - if ! docker image inspect "$BASE_IMAGE" > /dev/null 2>&1; then | 447 | + echo ">>> Base image not found, pulling: ${BASE_IMAGE}" |
| 441 | - echo ">>> Base image not found, pulling: ${BASE_IMAGE}" | 448 | + docker pull "$BASE_IMAGE" |
| 442 | - docker pull "$BASE_IMAGE" | 449 | + if [ $? -ne 0 ]; then |
| 443 | - if [ $? -ne 0 ]; then | 450 | + echo "Error: Failed to pull base image" |
| 444 | - echo "Error: Failed to pull base image" | 451 | + exit 1 |
| 445 | - exit 1 | ||
| 446 | - fi | ||
| 447 | - else | ||
| 448 | - echo ">>> Base image already exists: ${BASE_IMAGE}" | ||
| 449 | fi | 452 | fi |
| 450 | - echo "" | 453 | +else |
| 454 | + echo ">>> Base image already exists: ${BASE_IMAGE}" | ||
| 451 | fi | 455 | fi |
| 456 | +echo "" | ||
| 452 | 457 | ||
| 453 | cd "$SCRIPT_DIR" | 458 | cd "$SCRIPT_DIR" |
| 454 | 459 | ||
| @@ -506,9 +511,7 @@ fi | |||
| 506 | 511 | ||
| 507 | BUILD_ARGS="--build-arg MINICONDA_SH=${MINICONDA_NAME}" | 512 | BUILD_ARGS="--build-arg MINICONDA_SH=${MINICONDA_NAME}" |
| 508 | BUILD_ARGS="$BUILD_ARGS --build-arg DECORD_SCRIPT=${DECORD_SCRIPT_NAME}" | 513 | BUILD_ARGS="$BUILD_ARGS --build-arg DECORD_SCRIPT=${DECORD_SCRIPT_NAME}" |
| 509 | -BUILD_ARGS="$BUILD_ARGS --build-arg OS=${OS}" | ||
| 510 | BUILD_ARGS="$BUILD_ARGS --build-arg OS_FAMILY=${OS_FAMILY}" | 514 | BUILD_ARGS="$BUILD_ARGS --build-arg OS_FAMILY=${OS_FAMILY}" |
| 511 | -BUILD_ARGS="$BUILD_ARGS --build-arg NPU_TYPE=${NPU_TYPE_LOWER}" | ||
| 512 | BUILD_ARGS="$BUILD_ARGS --build-arg TORCH_VERSION=${TORCH_VERSION}" | 515 | BUILD_ARGS="$BUILD_ARGS --build-arg TORCH_VERSION=${TORCH_VERSION}" |
| 513 | BUILD_ARGS="$BUILD_ARGS --build-arg TORCH_NPU_VERSION=${TORCH_NPU_VERSION}" | 516 | BUILD_ARGS="$BUILD_ARGS --build-arg TORCH_NPU_VERSION=${TORCH_NPU_VERSION}" |
| 514 | BUILD_ARGS="$BUILD_ARGS --build-arg TORCH_WHL_DIR=torch_wheels" | 517 | BUILD_ARGS="$BUILD_ARGS --build-arg TORCH_WHL_DIR=torch_wheels" |
| @@ -519,11 +522,8 @@ if [ "$BUILD_CI" = true ]; then | |||
| 519 | BUILD_ARGS="$BUILD_ARGS --build-arg FINAL_WORKDIR=/workspace" | 522 | BUILD_ARGS="$BUILD_ARGS --build-arg FINAL_WORKDIR=/workspace" |
| 520 | fi | 523 | fi |
| 521 | 524 | ||
| 522 | -if [ -n "$BASE_IMAGE" ]; then | 525 | +# Always pass the full BASE_IMAGE to the Dockerfile (required, auto-detected values come from its tag). |
| 523 | - BUILD_ARGS="$BUILD_ARGS --build-arg BASE_IMAGE=${BASE_IMAGE}" | 526 | +BUILD_ARGS="$BUILD_ARGS --build-arg BASE_IMAGE=${BASE_IMAGE}" |
| 524 | -else | ||
| 525 | - BUILD_ARGS="$BUILD_ARGS --build-arg BASE_IMAGE_VERSION=${BASE_IMAGE_VERSION}" | ||
| 526 | -fi | ||
| 527 | 527 | ||
| 528 | BUILD_ARGS="$BUILD_ARGS --build-arg DECORD_DEPS_DIR=${DECORD_DEPS_NAME}" | 528 | BUILD_ARGS="$BUILD_ARGS --build-arg DECORD_DEPS_DIR=${DECORD_DEPS_NAME}" |
| 529 | 529 | ||
| @@ -89,4 +89,4 @@ echo "" | |||
| 89 | echo "Next steps:" | 89 | echo "Next steps:" |
| 90 | echo " 1. Verify the file: bash ${MINICONDA_FILE} --help" | 90 | echo " 1. Verify the file: bash ${MINICONDA_FILE} --help" |
| 91 | echo " 2. Use it to build Docker image:" | 91 | echo " 2. Use it to build Docker image:" |
| 92 | -echo " bash build.sh -t A3 -m ${OUTPUT_DIR}/${MINICONDA_FILE}" | 92 | +echo " bash build.sh --base-image swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.0.0-a3-openeuler24.03-py3.11 -m ${OUTPUT_DIR}/${MINICONDA_FILE}" |


建议补充说明这个功能的背景与利弊